Method and device for distinguishing identity in classroom conversation, equipment and storage medium

Abstract

The invention discloses a method, device and equipment for distinguishing identities in classroom dialogues and a storage medium. The method comprises the following steps: acquiring a classroom audio file; calling a preset voice recognition model to perform voice recognition on the classroom audio file to obtain text content in the classroom audio file; performing sentence segmentation on the text content by using a trained punctuation prediction model based on deep learning to obtain sentences of the voice text; calculating the voiceprint score of each sentence of the voice text; calculating the total visual score of each sentence of the voice text; and distinguishing whether each sentence of the voice text belongs to a teacher or a student based on the voiceprint score and the total visual score. According to the method, the identity in the classroom conversation can be distinguished simply, conveniently and effectively, the technical blank of a method for distinguishing the identity in the classroom conversation is filled, and a technical basis is provided for subsequent classroom analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of multimedia processing, and more particularly, to a method, apparatus, device and storage medium for distinguishing identities in classroom conversations. Background Art

[0002] With the development of multimedia technology, the digitization and electronicization of classrooms can bring greater benefits to teaching. Currently, extracting the speech text of each speaker in the classroom can be used for tasks such as teaching content analysis and teaching evaluation. However, before using the speech text of each speaker in the classroom, the text content must be identified to determine which sentences in the text are from the teacher and which are from the students.

[0003] However, in related technologies, methods for distinguishing identities in classroom conversations cannot accurately distinguish between teachers and students. Summary of the Invention

[0004] The present invention aims to solve the technical problem in the prior art that the methods cannot accurately distinguish identities in classroom conversations, and provides a method, device, equipment and storage medium for distinguishing identities in classroom conversations. The technical solution adopted by the present invention is as follows.

[0005] In a first aspect, the present invention provides a method for distinguishing identities in a classroom conversation, comprising:

[0006] Get the class audio files;

[0007] Calling a preset speech recognition model to perform speech recognition on the classroom audio file to obtain the text content in the classroom audio file;

[0008] Segment the text content using a trained deep learning-based punctuation prediction model to obtain individual sentences of the speech text;

[0009] Calculate the voiceprint score of each sentence of the speech text;

[0010] Calculate the total visual score of each sentence of the speech text;

[0011] Distinguishing, based on the voiceprint score and the visual total score, whether each sentence of the speech text belongs to the teacher or the student;

[0012] The process of calculating the voiceprint score of each sentence of the speech text includes:

[0013] Obtain the speech fragments corresponding to each sentence of the speech text;

[0014] Use the trained voiceprint model to calculate the voiceprint features of the speech segments corresponding to each sentence of the speech text;

[0015] Obtain the teacher's voiceprint characteristics;

[0016] Comparing the teacher's voiceprint features with the voiceprint features of the speech segments corresponding to each sentence, and obtaining the teacher's voiceprint score of each speech segment to determine whether it is the teacher's voiceprint;

[0017] Determining the voiceprint score of the sentence of the speech text corresponding to each speech segment according to the teacher's voiceprint score;

[0018] The process of calculating the total visual score of each sentence of the speech text includes:

[0019] Get the classroom video file corresponding to the classroom audio file;

[0020] Through the trained lip-speech matching model, the matching degree between each sentence of the speech text and the teacher's lip shape in the classroom video file is analyzed to obtain the first visual score of each sentence in the speech text as to whether it is the teacher's sentence;

[0021] Based on the first visual score, a total visual score of each sentence of the speech text is obtained.

[0022] In one embodiment, the process of calculating the total visual score of each sentence of the speech text further includes:

[0023] Obtain the text content of the teaching PPT through the trained image text recognition model;

[0024] Performing a matching analysis on each sentence of the voice text and the text content of the teaching PPT to obtain a second visual score of each sentence of the voice text as to whether it is a teacher's sentence;

[0025] The process of deriving the total visual score of each sentence of the speech text based on the first visual score is also based on the second visual score.

[0026] In one embodiment, the method for distinguishing identities in a classroom conversation further includes:

[0027] Calculate the contextual semantic score of each sentence in the speech text;

[0028] The process of distinguishing whether each sentence of the speech text belongs to the teacher or the student based on the voiceprint score and the visual total score further distinguishes whether each sentence of the speech text belongs to the teacher or the student based on the contextual semantic score;

[0029] The process of calculating the contextual semantic score of each sentence of the speech text includes:

[0030] Each sentence of the speech text is fed into the trained large language model, which then calculates the probability of each sentence being the teacher's sentence based on the actual order of the sentences in the speech text and through contextual semantic analysis.

[0031] According to the probability that each sentence is a teacher sentence, the contextual semantic score of each sentence in the speech text is obtained.

[0032] In one embodiment, before the process of calling a preset speech recognition model to perform speech recognition on the classroom audio file to obtain the text content in the classroom audio file, the process further includes:

[0033] Feeding the classroom audio file into a trained voice activation detection model to segment the audio clips containing speech content;

[0034] The process of calling the preset speech recognition model to perform speech recognition on the classroom audio file is to call the preset speech recognition model to perform speech recognition on the audio segment containing speech content.

[0035] In one embodiment, the process of using the trained deep learning-based punctuation prediction model to segment the text content into sentences to obtain individual sentences of the speech text includes:

[0036] Adding punctuation marks to the text content using a trained deep learning-based punctuation prediction model;

[0037] Obtain the pause duration between the audio data corresponding to the preceding sentence and the audio data corresponding to the following sentence for each punctuation mark;

[0038] Determine whether the pause interval duration is less than the minimum interval duration corresponding to the punctuation mark. If so, merge the sentence before and after the punctuation mark into one sentence and discard the punctuation mark, wherein each punctuation mark type has its own corresponding minimum interval duration;

[0039] The text content is segmented into sentences using the added punctuation marks to obtain individual sentences of the speech text.

[0040] In one embodiment, before the process of obtaining the speech segments corresponding to each sentence of the speech text, the process further includes:

[0041] Identify the audio time points at which text characters are aligned for sentence segmentation in the text content;

[0042] The classroom audio file is segmented according to each of the audio time points to obtain the voice segments corresponding to each sentence of the voice text.

[0043] In one embodiment, the process of comparing the teacher's voiceprint features with the voiceprint features of the speech segments corresponding to each sentence to obtain the teacher's voiceprint score for each speech segment regarding whether it is the teacher's voiceprint includes:

[0044] Perform unsupervised cluster analysis on the voiceprint features of the speech clips to obtain N potential voiceprint feature clusters of different speakers, and obtain the voiceprint feature cluster center corresponding to each speaker;

[0045] According to the teacher's voiceprint characteristics, confirm the teacher's voiceprint cluster and the teacher's voiceprint cluster center;

[0046] By comparing the teacher's voiceprint cluster center with the voiceprint feature cluster centers of each speaker, the teacher's voiceprint score of each voice segment is obtained to determine whether it is the teacher's voiceprint.

[0047] In one embodiment, before the process of obtaining the teacher's voiceprint features, the process further includes:

[0048] Each sentence of the speech text is input into the trained large language model. The large language model analyzes the context and obtains the sentence most likely to be the teacher's identity;

[0049] Obtaining the audio corresponding to the sentence that is most likely to be the teacher's identity in the classroom audio data, that is, the teacher audio clip;

[0050] The teacher audio clip is input as a trained voiceprint model, and the voiceprint model calculates the voiceprint features of the teacher audio clip as the teacher voiceprint and saves it.

[0051] In a second aspect, the present invention provides a device for detecting teacher questions, comprising:

[0052] Acquisition module, used to obtain classroom audio files;

[0053] An extraction module is used to call a preset speech recognition model to perform speech recognition on the classroom audio file to obtain the text content in the classroom audio file;

[0054] A segmentation module is used to segment the text content into sentences using a trained deep learning-based punctuation prediction model to obtain individual sentences of the speech text;

[0055] The first analysis module is used to calculate the voiceprint score of each sentence of the speech text;

[0056] The second analysis module is used to calculate the total visual score of each sentence of the speech text;

[0057] a judgment module, configured to distinguish whether each sentence of the speech text belongs to the teacher or the student based on the voiceprint score and the visual total score;

[0058] The process of calculating the voiceprint score of each sentence of the speech text by the first analysis module includes:

[0059] Obtain the speech fragments corresponding to each sentence of the speech text;

[0060] Use the trained voiceprint model to calculate the voiceprint features of the speech segments corresponding to each sentence of the speech text;

[0061] Obtain the teacher's voiceprint characteristics;

[0062] Comparing the teacher's voiceprint features with the voiceprint features of the speech segments corresponding to each sentence, and obtaining the teacher's voiceprint score of each speech segment to determine whether it is the teacher's voiceprint;

[0063] Determining the voiceprint score of the sentence of the speech text corresponding to each speech segment according to the teacher's voiceprint score;

[0064] The process of calculating the visual total score of each sentence of the speech text by the second analysis module includes:

[0065] Get the classroom video file corresponding to the classroom audio file;

[0066] Through the trained lip-speech matching model, the matching degree between each sentence of the audio text and the teacher's lip shape in the classroom video file is analyzed to obtain the first visual score of each sentence in the audio text as to whether it is the teacher's sentence;

[0067] Based on the first visual score, a total visual score of each sentence of the speech text is obtained.

[0068] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned methods when executing the program.

[0069] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the program implements the method of any of the above-mentioned embodiments when executed by a processor.

[0070] In this paper, the voice text is analyzed using voiceprint, visual, and semantic information to determine whether each sentence is the teacher's, thereby distinguishing the identities of students in classroom conversations. This method can simply, conveniently, and effectively distinguish identities in classroom conversations, filling the technical gap in methods for distinguishing identities in classroom conversations and providing a technical foundation for subsequent classroom analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a schematic diagram of the overall process of an implementation method of Example 1 of the present invention.

[0072] Figure 2 It is a step-by-step flow chart of an implementation method of Example 1 of the present invention.

[0073] Figure 3 This is another step-by-step flowchart of an implementation method of Example 1 of the present invention.

[0074] Figure 4 It is a schematic diagram of the overall structure of an implementation method of Example 2 of the present invention. DETAILED DESCRIPTION

[0075] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0076] It should be noted that the terms "first, second, ..." used in the embodiments of the present invention are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that the terms "first, second, ..." may interchangeably represent a specific order or precedence, where permitted. It should be understood that the objects distinguished by "first, second, ..." may be interchangeable, where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein.

[0077] Example 1

[0078] See Figure 1 , Figure 1 This is a flowchart of a method for distinguishing identities in a classroom conversation provided by Example 1 of the present invention. The method includes steps S110, S120, S130, S140, S150, and S160. It should be noted that steps S110, S120, S130, S150, and S160 are merely reference numerals for the purpose of clarifying the embodiment and the appendix. Figure 1 The corresponding relationship does not limit the order of the steps in this embodiment.

[0079] Step S110, obtaining classroom audio files;

[0080] Step S120, calling a preset speech recognition model to perform speech recognition on the classroom audio file to obtain the text content in the classroom audio file;

[0081] Step S130, using the trained deep learning-based punctuation prediction model to segment the text content into sentences to obtain individual sentences of the speech text;

[0082] Step S140, calculating the voiceprint score of each sentence of the speech text;

[0083] Step S150, calculating the total visual score of each sentence in the speech text;

[0084] Step S160: Distinguish whether each sentence in the speech text belongs to the teacher or the student based on the voiceprint score and the visual total score.

[0085] Generally speaking, microphones are arranged in the classroom to obtain the audio data stream in the classroom. In step S110, this audio data stream is obtained.

[0086] In step S120, the audio containing speech content is converted into corresponding text content through the trained automatic speech recognition model.

[0087] In one embodiment, before executing step S120, certain preparations need to be done. Specifically, before step S120, the process also includes: step S119.

[0088] Step S119: feeding the classroom audio file into a trained voice activation detection model to segment the audio clips containing voice content;

[0089] The process of calling the preset speech recognition model to perform speech recognition on the classroom audio file in step S120 is to call the preset speech recognition model to perform speech recognition on the audio segment containing speech content.

[0090] Step S119 uses a trained speech activation detection model to segment the audio data recorded in the classroom into audio segments containing speech content. The trained speech activation detection model can remove silent audio segments that do not contain speech, which can reduce misjudgments as well as computing power consumption and computational delays.

[0091] Returning to the main thread of this method for distinguishing identities in classroom conversations, in step S130, the speech is segmented to obtain individual speech text sentences. Since each segmented speech text is a sentence, and typically a sentence only contains a single speaker, this improves the reliability of identity distinction.

[0092] In one embodiment, the process of step S130 includes: step S131 and step S132.

[0093] Step S131, adding punctuation marks to the text content using a trained deep learning-based punctuation prediction model;

[0094] Step S132, obtaining the pause interval between the audio data corresponding to the preceding sentence and the audio data corresponding to the following sentence of each punctuation mark;

[0095] Step S133: determining whether the pause interval duration is less than the minimum interval duration corresponding to the punctuation mark; if so, merging the sentence preceding and following the punctuation mark into one sentence and discarding the punctuation mark, wherein each punctuation mark type has its own corresponding minimum interval duration;

[0096] Step S134 , segmenting the text content into sentences using the added punctuation marks to obtain individual sentences of the speech text.

[0097] In this embodiment, the text content is segmented into sentences using punctuation marks.

[0098] After adding punctuation to the text content using a trained deep learning-based punctuation prediction model, the pause interval T between the end of the previous paragraph and the beginning of the next paragraph is calculated. In addition, different minimum interval times are set for different punctuation types (,?.!;,). Punctuation marks shorter than the pause interval are discarded, and the two adjacent paragraphs are merged into one sentence.

[0099] Doing so can avoid more colloquial communication, where there are too many modal particles that cause sentences to be cut into pieces, and too much punctuation that makes the text difficult to understand.

[0100] Let's go back to the main line of the method of distinguishing identities in classroom conversations. Since this method is to distinguish identities in classroom conversations, although the identities in the classroom include teachers and students, for actual classroom scenes and the actual requirements of using classroom texts for further analysis, it is only necessary to distinguish the identities of teachers and students. All students can be counted in the same category and given a unified identity label.

[0101] Based on the above ideas, the voiceprint score obtained in step S140 and the visual total score obtained in step S150 are both based on the scores obtained by distinguishing the two roles of teacher and student. For example, it can be set that the higher the voiceprint score and the visual total score, the more likely it is that the sentence is a teacher's sentence. In this way, in step S160, as long as the total score of the two is higher than a preset threshold, it can be considered as a teacher's sentence, otherwise it is a student's sentence; of course, it can also be reversed, so that the higher the voiceprint score and the visual total score, the more likely it is that the sentence is a student's sentence. In this way, in step S160, as long as the total score of the two is higher than a preset threshold, it can be considered as a student's sentence, otherwise it is a teacher's sentence.

[0102] Specifically, if Figure 2 As shown, in step S140, the voiceprint score of each sentence of the speech text is calculated, which specifically includes: step S141, step S142, step S143, step S144 and step S145.

[0103] Step S141, obtaining the speech segments corresponding to each sentence of the speech text;

[0104] Step S142: using the trained voiceprint model to calculate the voiceprint features of the speech segments corresponding to each sentence of the speech text;

[0105] Step S143, obtaining the teacher's voiceprint features;

[0106] Step S144, comparing the teacher's voiceprint features with the voiceprint features of the speech segments corresponding to each sentence, and obtaining the teacher's voiceprint score of each speech segment to determine whether it is the teacher's voiceprint;

[0107] Step S145: derive the voiceprint score of the sentence of the speech text corresponding to each speech segment according to the teacher's voiceprint score.

[0108] Each sentence of the speech text has a corresponding speech segment. In step S141, the speech segments corresponding to each of these sentences are obtained.

[0109] In one embodiment, before step S141 , the process includes: step S210 and step S220 .

[0110] Step S210, identifying the audio time points at which the text characters are aligned for sentence segmentation of the text content;

[0111] Step S220 , segmenting the classroom audio file according to each of the audio time points to obtain voice segments corresponding to each sentence of the voice text.

[0112] While step S131 uses the added punctuation marks to segment the text content into sentences, it can also use the trained deep learning-based punctuation prediction model to predict the punctuation marks of the text content corresponding to the speech and the audio time points corresponding to the punctuation marks, and segment the corresponding whole sentence speech audio data through the audio time points corresponding to the punctuation marks.

[0113] In step S142, the trained voiceprint model is used to calculate the voiceprint features of the speech segments corresponding to each sentence of the speech text.

[0114] In step S143, only the teacher's voiceprint features are obtained. In steps S144 and S155, only the teacher's voiceprint features are compared with the voiceprint features of the speech segments corresponding to each sentence to find out the sentences belonging to the teacher's speech text. Then the remaining sentences that do not belong to the teacher's speech text naturally belong to the students.

[0115] In one embodiment, the process of step S144 includes: step S1441, step S1442 and step S1443.

[0116] Step S1441: Perform unsupervised cluster analysis on the voiceprint features of the speech segment to obtain N potential voiceprint feature clusters of different speakers, and obtain the voiceprint feature cluster center corresponding to each speaker;

[0117] Step S1442: confirming the teacher's voiceprint cluster and the teacher's voiceprint cluster center based on the teacher's voiceprint characteristics;

[0118] Step S1443 , by comparing the teacher's voiceprint cluster center with the voiceprint feature cluster centers of each speaker, a teacher's voiceprint score of each voice segment is obtained to determine whether it is the teacher's voiceprint.

[0119] In step S1441, a spectral clustering algorithm is applied to perform unsupervised cluster analysis on the voiceprint features, obtaining 1, 2, 3, ...N potential voiceprint clusters of different speakers and corresponding voiceprint feature clusters. The voiceprint features within each cluster are averaged to determine the voiceprint cluster center of the corresponding cluster. Using spectral clustering to determine the number of potential speakers improves upon the conventional method of using a registered teacher's voiceprint and each sentence's voiceprint to calculate distance or similarity and then making a hard judgment based on empirical values.

[0120] In step S1442, the teacher's voiceprint cluster and the teacher's voiceprint cluster center are determined based on the teacher's voiceprint characteristics.

[0121] In step S1443, the cosine distance between the teacher's voiceprint cluster center and the voiceprint feature cluster centers of each speaker can be calculated. Cluster centers with a cosine distance less than a clustering preset value (the preset value can be an empirical value) are merged into the teacher's voiceprint cluster, and the remaining voiceprint clusters are marked as student voiceprint features, thereby obtaining a teacher voiceprint score for each voice segment regarding whether it is a teacher's voiceprint. The method of using saved teacher voiceprints to confirm teacher clusters and merging them based on the distance between each cluster center and the teacher cluster center can further reduce the problem of inaccurate voiceprint results caused by too small a cluster center segmentation interval in spectral clustering.

[0122] In step S145, based on the teacher voiceprint scores of each speech segment, the voiceprint scores of the sentences of the speech text corresponding to these speech segments are obtained. For example, when the voiceprint features of the speech segment belong to the teacher cluster, the voiceprint score (val_spk) of the sentence of the speech text corresponding to the speech segment is: a*val_cos, otherwise it is (1-a)*val_cos, and a is a preset constant.

[0123] Specifically, if Figure 3 As shown, in step S150 , the process of calculating the total visual score of each sentence of the speech text includes: step S151 , step S152 and step S153 .

[0124] Step S151, obtaining a classroom video file corresponding to the classroom audio file;

[0125] Step S152: Using the trained lip-speech matching model, the matching degree between each sentence in the speech text and the teacher's lip shape in the classroom video file is analyzed to obtain a first visual score of each sentence in the speech text as to whether it is a teacher's sentence;

[0126] Step S153: derive a total visual score of each sentence in the speech text based on the first visual score.

[0127] The classroom is generally equipped with microphones and cameras for recording, so the audio and video data streams in the classroom can be obtained. The classroom audio file is obtained in step S110, and the corresponding classroom video file is obtained in step S151.

[0128] Through the images of the classroom video file, we can see the teacher's mouth shape when pronouncing the words. At this time, we can use the trained lip-speech matching model to analyze the matching degree between each sentence in the speech text and the teacher's mouth shape in the classroom video file, thereby obtaining the first visual score vis_1 of each sentence in the speech text as to whether it is the teacher's sentence.

[0129] Finally, the total visual score val_vis of each sentence of the speech text is obtained based on the first visual score vis_1.

[0130] In one embodiment, the process of calculating the total visual score of each sentence of the speech text in step S150 further includes: step S154, step S155 and step S156.

[0131] Step S154, obtaining the text content of the teaching PPT through the trained image text recognition model;

[0132] Step S155, performing a matching analysis between each sentence of the voice text and the text content of the teaching PPT, and obtaining a second visual score of each sentence of the voice text as to whether it is a teacher's sentence;

[0133] Step S156, the process of deriving the total visual score of each sentence of the speech text based on the first visual score is also based on the second visual score.

[0134] Teachers generally use teaching PowerPoint presentations when giving lectures. Step S154 uses the trained image text recognition model to obtain the text content of the teaching PowerPoint presentation. It should be noted that this method does not limit how to obtain the teaching PowerPoint presentation. The presentation can be obtained by capturing it with a classroom camera or by using a PowerPoint file.

[0135] The teacher's speech and explanation are generally related to the PPT, so the content of the explanation sentence and the text content of the PPT should be relatively corresponding. In step S155, the matching degree of each sentence of the voice text and the text content of the teaching PPT is analyzed to obtain the second visual score vis_2 of each sentence of the voice text as to whether it is the teacher's sentence.

[0136] At this time, the total visual score val_vis in step S156 is val_vis=v1*vis_1+v2*vis_2, where the weighted values ​​v1 and v2 are determined by actual experimental analysis.

[0137] After obtaining the voiceprint score and the total visual score, in step S160, specifically, the comprehensive score (val_syn) = voiceprint score (val_spk) + total visual score (val_vis) can be obtained, and sentences with a comprehensive score val_syn greater than the experience judgment score val_just are regarded as sentences spoken by the teacher, while sentences with val_syn less than the experience judgment score val_just are regarded as sentences spoken by the student, so as to distinguish whether each sentence in the voice text in the final classroom conversation belongs to the teacher or the student.

[0138] In one embodiment, the method for distinguishing identities in classroom conversations further includes: step S170.

[0139] Step S170, calculating the contextual semantic score of each sentence in the speech text;

[0140] The process of step S160 of distinguishing whether each sentence of the speech text belongs to the teacher or the student based on the voiceprint score and the visual total score also distinguishes whether each sentence of the speech text belongs to the teacher or the student based on the contextual semantic score.

[0141] The process of calculating the contextual semantic score of each sentence of the speech text includes: step S171 and step S172.

[0142] Step S171: Each sentence of the speech text is fed into the trained large language model, and the large language model calculates the probability of each sentence being a teacher sentence through contextual semantic analysis based on the actual order of each sentence in the speech text;

[0143] Step S172: derive the contextual semantic score of each sentence in the speech text based on the probability that each sentence is a teacher sentence.

[0144] This embodiment adds a dimension of evaluation to the comprehensive score (val_syn), namely, the contextual semantic score (val_sem).

[0145] Specifically, after all the segmented classroom sentences are marked with corresponding serial numbers, they are sent to the trained large language model, and the trained large language model is instructed to output the probability val_tpr between 0 and 1 that each sentence corresponding to the serial number is the teacher's sentence through context semantics, and val_sem = val_tpr to obtain the corresponding context semantic score.

[0146] Finally, let the comprehensive score (val_syn) = voiceprint score (val_spk) + visual total score (val_vis) + context semantic score (val_sem), and regard the sentences with the comprehensive score val_syn greater than the experience judgment score val_just as the sentences spoken by the teacher, and regard the sentences with val_syn less than the experience judgment score val_just as the sentences spoken by the student, so as to distinguish the identities of the teacher and the student in the final classroom conversation.

[0147] This method uses voiceprint, visual, and semantic analysis to determine whether each sentence in the speech text is the teacher's, thereby distinguishing the identities of those involved in classroom conversations. This method can simply, conveniently, and effectively distinguish identities in classroom conversations, filling a technical gap in methods for distinguishing identities in classroom conversations and providing a technical foundation for subsequent classroom analysis.

[0148] In one embodiment, the process of obtaining the teacher's voiceprint features in step S143 includes: step S310, step S320 and step S330.

[0149] Step S310: Input each sentence of the speech text into the trained large language model. The large language model analyzes the context and obtains the sentence that is most likely to be the teacher's identity;

[0150] Step S320, obtaining the audio corresponding to the sentence that is most likely to be the teacher's identity in the classroom audio data, that is, the teacher audio clip;

[0151] Step S330: input the teacher audio clip as a trained voiceprint model, and use the voiceprint model to calculate the voiceprint features of the teacher audio clip as the teacher's voiceprint and save it.

[0152] This embodiment automatically acquires the teacher's voice data in the classroom. This can be done by using a microphone with an ID tag at the teacher's desk to confirm that the teacher is the one teaching the class. This data can then be directly used to calculate the teacher's voiceprint. Alternatively, the teacher's voice data can be obtained by aligning it with the video captured by the camera, using the teacher's lip movements to obtain corresponding voice segments, or by determining when no students stand up to answer questions to obtain voice segments. These segments can then be used as data for calculating the teacher's voiceprint.

[0153] Different from the above-mentioned method of automatically obtaining the teacher's voice data in the classroom, this embodiment converts the speech into text through a trained automatic speech recognition model and sends it to a large model, which confirms which content is the teacher's voice.

[0154] In this embodiment, the first step is to obtain classroom audio data, and perform voice activation detection on the classroom audio data to extract segments containing voice; the second step is to use the trained automatic speech recognition model to identify the text content consistent with the voice content and the audio time points aligned with the text characters; the third step is to add punctuation marks to the identified text content by the trained deep learning-based punctuation prediction model, and perform sentence segmentation of the voice text, and record the audio time points aligned with the text characters at the segmentation points: s(1), s(2), ..., s(n); the fourth step is to use the segmented The speech text sentences are added with serial numbers consistent with the order of the segmentation points: 1, 2, ..., n, and input into the trained large language model. The large language model analyzes the context and finds the sentence that is most likely to be the teacher's identity, and outputs the serial number of the corresponding sentence: for example, k; the fifth step is to obtain the serial number of the sentence that is most likely to be the teacher's identity through the large language model analysis: k, and index the audio time point of the text at the corresponding segmentation: s(k); the sixth step is to intercept s(k-1) to s(k) of the original audio as the input of the trained voiceprint model, and the voiceprint model calculates the voiceprint features of the corresponding audio segment and saves it as the teacher's voiceprint. Of course, some of the above processes are repeated with the previous steps S110 to S130, so they can be simplified to steps S310 to S330.

[0155] This embodiment fully utilizes the resources used between step S110 and step S130 , does not require additional site layout, and does not require excessive use of other resources, thereby simplifying the entire process.

[0156] Example 2

[0157] Corresponding to the method of embodiment 1, as Figure 4 As shown, the present invention also provides a device 4 for detecting teacher questions, including: an acquisition module 410, an extraction module 420, a segmentation module 430, a first analysis module 440, a second analysis module 450, and a judgment module 460.

[0158] An acquisition module 410 is used to acquire classroom audio files;

[0159] An extraction module 420 is configured to call a preset speech recognition model to perform speech recognition on the classroom audio file to obtain text content in the classroom audio file;

[0160] A segmentation module 430 is configured to segment the text content into sentences using a trained deep learning-based punctuation prediction model to obtain individual sentences of the speech text;

[0161] The first analysis module 440 is used to calculate the voiceprint score of each sentence of the speech text;

[0162] The second analysis module 450 is used to calculate the total visual score of each sentence in the speech text;

[0163] A judgment module 460 is configured to distinguish whether each sentence in the speech text belongs to the teacher or the student based on the voiceprint score and the visual total score;

[0164] The process of calculating the voiceprint score of each sentence of the speech text by the first analysis module includes:

[0165] Obtain the speech fragments corresponding to each sentence of the speech text;

[0166] Use the trained voiceprint model to calculate the voiceprint features of the speech segments corresponding to each sentence of the speech text;

[0167] Obtain the teacher's voiceprint characteristics;

[0168] Comparing the teacher's voiceprint features with the voiceprint features of the speech segments corresponding to each sentence, and obtaining the teacher's voiceprint score of each speech segment to determine whether it is the teacher's voiceprint;

[0169] Determining the voiceprint score of the sentence of the speech text corresponding to each speech segment according to the teacher's voiceprint score;

[0170] The process of calculating the visual total score of each sentence of the speech text by the second analysis module includes:

[0171] Get the classroom video file corresponding to the classroom audio file;

[0172] Through the trained lip-speech matching model, the matching degree between each sentence of the audio text and the teacher's lip shape in the classroom video file is analyzed to obtain the first visual score of each sentence in the audio text as to whether it is the teacher's sentence;

[0173] Based on the first visual score, a total visual score of each sentence of the speech text is obtained.

[0174] In one embodiment, the process of calculating the total visual score of each sentence of the speech text by the second analysis module further includes:

[0175] Obtain the text content of the teaching PPT through the trained image text recognition model;

[0176] Performing a matching analysis on each sentence of the voice text and the text content of the teaching PPT to obtain a second visual score of each sentence of the voice text as to whether it is a teacher's sentence;

[0177] The process of deriving the total visual score of each sentence of the speech text based on the first visual score is also based on the second visual score.

[0178] In one embodiment, the invention further comprises: a third analysis module, configured to calculate the contextual semantic score of each sentence of the speech text;

[0179] The judgment module further distinguishes whether each sentence of the speech text belongs to the teacher or the student based on the context semantics;

[0180] The process of calculating the contextual semantic score of each sentence in the speech text by the third analysis module includes:

[0181] Each sentence of the speech text is fed into the trained large language model, which then calculates the probability of each sentence being the teacher's sentence based on the actual order of the sentences in the speech text and through contextual semantic analysis.

[0182] According to the probability that each sentence is a teacher sentence, the contextual semantic score of each sentence in the speech text is obtained.

[0183] In one embodiment, before the extraction module calls a preset speech recognition model to perform speech recognition on the classroom audio file to obtain the text content in the classroom audio file, the process further includes:

[0184] Feeding the classroom audio file into a trained voice activation detection model to segment the audio clips containing speech content;

[0185] The process of calling the preset speech recognition model to perform speech recognition on the classroom audio file is to call the preset speech recognition model to perform speech recognition on the audio segment containing speech content.

[0186] In one embodiment, the segmentation module uses a trained deep learning-based punctuation prediction model to segment the text content into sentences to obtain individual sentences of the speech text, including:

[0187] Adding punctuation marks to the text content using a trained deep learning-based punctuation prediction model;

[0188] Obtain the pause duration between the audio data corresponding to the preceding sentence and the audio data corresponding to the following sentence for each punctuation mark;

[0189] Determine whether the pause interval duration is less than the minimum interval duration corresponding to the punctuation mark. If so, merge the sentence before and after the punctuation mark into one sentence and discard the punctuation mark, wherein each punctuation mark type has its own corresponding minimum interval duration;

[0190] The text content is segmented into sentences using the added punctuation marks to obtain individual sentences of the speech text.

[0191] In one embodiment, before the process of obtaining the speech segments corresponding to each sentence of the speech text, the process further includes:

[0192] Identify the audio time points at which text characters are aligned for sentence segmentation in the text content;

[0193] The classroom audio file is segmented according to each of the audio time points to obtain the voice segments corresponding to each sentence of the voice text.

[0194] In one embodiment, the process of comparing the teacher's voiceprint features with the voiceprint features of the speech segments corresponding to each sentence to obtain the teacher's voiceprint score for each speech segment regarding whether it is the teacher's voiceprint includes:

[0195] Perform unsupervised cluster analysis on the voiceprint features of the speech clips to obtain N potential voiceprint feature clusters of different speakers, and obtain the voiceprint feature cluster center corresponding to each speaker;

[0196] According to the teacher's voiceprint characteristics, confirm the teacher's voiceprint cluster and the teacher's voiceprint cluster center;

[0197] By comparing the teacher's voiceprint cluster center with the voiceprint feature cluster centers of each speaker, the teacher's voiceprint score of each voice segment is obtained to determine whether it is the teacher's voiceprint.

[0198] In one embodiment, before the process of obtaining the teacher's voiceprint features, the process further includes:

[0199] Each sentence of the speech text is input into the trained large language model. The large language model analyzes the context and obtains the sentence most likely to be the teacher's identity;

[0200] Obtaining the audio corresponding to the sentence that is most likely to be the teacher's identity in the classroom audio data, that is, the teacher audio clip;

[0201] The teacher audio clip is input as a trained voiceprint model, and the voiceprint model calculates the voiceprint features of the teacher audio clip as the teacher voiceprint and saves it.

[0202] This device uses voiceprint, visual, and semantic analysis to determine whether each sentence in the audio text is the teacher's, thereby distinguishing the identities of those involved in classroom conversations. This device can easily, conveniently, and effectively distinguish identities in classroom conversations, filling a gap in the technology for distinguishing identities in classroom conversations and providing a technical foundation for subsequent classroom analysis.

[0203] Example 3

[0204] An embodiment of the present invention further provides a storage medium storing computer instructions, which, when executed by a processor, implement the method for distinguishing identities in classroom conversations according to any of the above embodiments.

[0205] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, random access memory (RAM), read-only memory (ROM), magnetic disk or optical disk, etc. Various media that can store program codes.

[0206] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, RAM, ROM, magnetic disks or optical disks.

[0207] Corresponding to the above-mentioned computer storage medium, in one embodiment, a computer device is also provided, which includes a memory, an encoder, and a computer program stored in the memory and runnable on the encoder, wherein when the encoder executes the program, it implements any one of the methods for distinguishing identities in classroom conversations in the above-mentioned embodiments.

[0208] The computer device uses voiceprint, visual, and semantic analysis to determine whether each sentence in the audio text is the teacher's, thereby distinguishing the identities of those involved in classroom conversations. This simple, convenient, and effective method for distinguishing identities in classroom conversations fills a technological gap and provides a foundation for future classroom analysis.

[0209] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0210] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A method for distinguishing identities in classroom conversations, characterized in that include: Get the class audio files; Calling a preset speech recognition model to perform speech recognition on the classroom audio file to obtain the text content in the classroom audio file; Segment the text content using a trained deep learning-based punctuation prediction model to obtain individual sentences of the speech text; Calculate the voiceprint score of each sentence of the speech text; Calculate the total visual score of each sentence of the speech text; Distinguishing, based on the voiceprint score and the visual total score, whether each sentence of the speech text belongs to the teacher or the student; The process of calculating the voiceprint score of each sentence of the speech text includes: Obtain the speech fragments corresponding to each sentence of the speech text; Use the trained voiceprint model to calculate the voiceprint features of the speech segments corresponding to each sentence of the speech text; Obtain the teacher's voiceprint characteristics; Comparing the teacher's voiceprint features with the voiceprint features of the speech segments corresponding to each sentence, and obtaining the teacher's voiceprint score of each speech segment to determine whether it is the teacher's voiceprint; Determining the voiceprint score of the sentence of the speech text corresponding to each speech segment according to the teacher's voiceprint score; The process of calculating the total visual score of each sentence of the speech text includes: Get the classroom video file corresponding to the classroom audio file; Through the trained lip-speech matching model, the matching degree between each sentence of the audio text and the teacher's lip shape in the classroom video file is analyzed to obtain the first visual score of each sentence in the audio text as to whether it is the teacher's sentence; Based on the first visual score, a total visual score of each sentence of the speech text is obtained.

2. The method for distinguishing identities in classroom conversations according to claim 1, characterized in that: The process of calculating the visual total score of each sentence of the speech text also includes: Obtain the text content of the teaching PPT through the trained image text recognition model; Performing a matching analysis on each sentence of the voice text and the text content of the teaching PPT to obtain a second visual score of each sentence of the voice text as to whether it is a teacher's sentence; The process of deriving the total visual score of each sentence of the speech text based on the first visual score is also based on the second visual score.

3. The method for distinguishing identities in classroom conversations according to claim 1, characterized in that: Also includes: Calculate the contextual semantic score of each sentence in the speech text; The process of distinguishing whether each sentence of the speech text belongs to the teacher or the student based on the voiceprint score and the visual total score further distinguishes whether each sentence of the speech text belongs to the teacher or the student based on the contextual semantic score; The process of calculating the contextual semantic score of each sentence of the speech text includes: Each sentence of the speech text is fed into the trained large language model, which then calculates the probability of each sentence being the teacher's sentence based on the actual order of the sentences in the speech text and through contextual semantic analysis. According to the probability that each sentence is a teacher sentence, the contextual semantic score of each sentence in the speech text is obtained.

4. The method for distinguishing identities in classroom conversations according to claim 1, characterized in that: Before the process of calling a preset speech recognition model to perform speech recognition on the classroom audio file to obtain the text content in the classroom audio file, the process further includes: Feeding the classroom audio file into a trained voice activation detection model to segment the audio clips containing speech content; The process of calling the preset speech recognition model to perform speech recognition on the classroom audio file is to call the preset speech recognition model to perform speech recognition on the audio segment containing speech content.

5. The method for distinguishing identities in classroom conversations according to claim 1, characterized in that: The process of using the trained deep learning-based punctuation prediction model to segment the text content into sentences to obtain individual sentences of the speech text includes: Adding punctuation marks to the text content using a trained deep learning-based punctuation prediction model; Obtain the pause duration between the audio data corresponding to the preceding sentence and the audio data corresponding to the following sentence for each punctuation mark; Determine whether the pause interval duration is less than the minimum interval duration corresponding to the punctuation mark. If so, merge the sentence before and after the punctuation mark into one sentence and discard the punctuation mark, wherein each punctuation mark type has its own corresponding minimum interval duration; The text content is segmented into sentences using the added punctuation marks to obtain individual sentences of the speech text.

6. The method for distinguishing identities in classroom conversations according to claim 1, characterized in that: Before the process of obtaining the speech segments corresponding to the respective sentences of the speech text, the process further includes: Identify the audio time points at which text characters are aligned for sentence segmentation in the text content; The classroom audio file is segmented according to each of the audio time points to obtain the voice segments corresponding to each sentence of the voice text.

7. The method for distinguishing identities in classroom conversations according to claim 1, characterized in that: The process of comparing the teacher's voiceprint features with the voiceprint features of the speech segments corresponding to each sentence to obtain the teacher's voiceprint score of each speech segment regarding whether it is the teacher's voiceprint includes: Perform unsupervised cluster analysis on the voiceprint features of the speech clips to obtain N potential voiceprint feature clusters of different speakers, and obtain the voiceprint feature cluster center corresponding to each speaker; According to the teacher's voiceprint characteristics, confirm the teacher's voiceprint cluster and the teacher's voiceprint cluster center; By comparing the teacher's voiceprint cluster center with the voiceprint feature cluster centers of each speaker, the teacher's voiceprint score of each voice segment is obtained to determine whether it is the teacher's voiceprint.

8. The method for distinguishing identities in classroom conversations according to claim 1, characterized in that: Before the process of obtaining the teacher's voiceprint features, the process also includes: Each sentence of the speech text is input into the trained large language model. The large language model analyzes the context and obtains the sentence most likely to be the teacher's identity; Obtaining the audio corresponding to the sentence that is most likely to be the teacher's identity in the classroom audio data, that is, the teacher audio clip; The teacher audio clip is input as a trained voiceprint model, and the voiceprint model calculates the voiceprint features of the teacher audio clip as the teacher voiceprint and saves it.

9. A device for detecting teacher questions, characterized in that: include: Acquisition module, used to obtain classroom audio files; An extraction module is used to call a preset speech recognition model to perform speech recognition on the classroom audio file to obtain the text content in the classroom audio file; A segmentation module is used to segment the text content into sentences using a trained deep learning-based punctuation prediction model to obtain individual sentences of the speech text; The first analysis module is used to calculate the voiceprint score of each sentence of the speech text; The second analysis module is used to calculate the total visual score of each sentence of the speech text; a judgment module, configured to distinguish whether each sentence of the speech text belongs to the teacher or the student based on the voiceprint score and the visual total score; The process of calculating the voiceprint score of each sentence of the speech text by the first analysis module includes: Obtain the speech fragments corresponding to each sentence of the speech text; Use the trained voiceprint model to calculate the voiceprint features of the speech segments corresponding to each sentence of the speech text; Obtain the teacher's voiceprint characteristics; Comparing the teacher's voiceprint features with the voiceprint features of the speech segments corresponding to each sentence, and obtaining the teacher's voiceprint score of each speech segment to determine whether it is the teacher's voiceprint; Determining the voiceprint score of the sentence of the speech text corresponding to each speech segment according to the teacher's voiceprint score; The process of calculating the visual total score of each sentence of the speech text by the second analysis module includes: Get the classroom video file corresponding to the classroom audio file; Through the trained lip-speech matching model, the matching degree between each sentence of the audio text and the teacher's lip shape in the classroom video file is analyzed to obtain the first visual score of each sentence in the audio text as to whether it is the teacher's sentence; Based on the first visual score, a total visual score of each sentence of the speech text is obtained.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images