A classroom teacher-student discourse log generation method and system fusing sentence-level multi-modal information

Abstract

The application provides a classroom teacher-student dialogue log generation method and system fusing sentence-level multi-modal information, and relates to the technical field of intelligent classrooms, comprising: pre-segmenting obtained classroom audio of a whole class to obtain each sub-audio and performing speech transcription and word-level time alignment to obtain a transcription text sequence and a word-level time sequence; obtaining a whole-class sentence-level record according to the end punctuation of the word-level time sequence and the transcription text sequence; performing speaker log processing on the classroom audio of the whole class to obtain speaker log segment information; cutting each sentence according to the whole-class sentence-level record to obtain a sentence audio segment, and performing teacher voiceprint verification to obtain a teacher matching score of each sentence; fusing the speaker log segment information and the teacher matching score to obtain an initial speaker identity of each sentence, and performing identity correction on a sentence with an uncertain initial speaker identity to obtain a final classroom teacher-student dialogue log.

Description

Technical Field

[0001] This invention relates to the field of intelligent classroom technology, and in particular to a method and system for generating classroom teacher-student discourse logs that integrates sentence-level multimodal information. Background Technology

[0002] Classroom audio and video recordings are important sources of information for classroom teaching data processing. Effectively identifying and transcribing the speech of teachers and students in the classroom, and further forming structured discourse records with speaker identification, has always been an important direction for classroom teaching analysis and educational research.

[0003] Currently, existing speech recognition and speaker separation and recognition technologies have achieved good application results in scenarios such as meetings, interviews, phone calls, and training. For example, products and models such as iFlytek Hearing, Tongyi Hearing, CAM++, and Pyannote SpeakerDiarization 3.1 can provide speaker logging functions; models such as Whisper, Mimo-ASR, Fun-ASR, and Paraformer can provide speech recognition functions; in addition, the development of multimodal large language models has also promoted the extension of speech recognition technology to deeper applications. However, these related products or models still have significant limitations in classroom scenarios.

[0004] The classroom setting presents unique complexities. Frequent shifts in speaking between teachers and students, short and scattered student speeches, often weak student responses interspersed with pauses, interruptions, speech overlap, environmental noise, and limitations of recording equipment all contribute to the problem. When general speaker logging technology is directly applied to the classroom, speaker identification errors or omissions frequently occur. While speech recognition can transcribe classroom discourse, it lacks speaker identification information, and the transcribed content is presented in whole segments, failing to reflect the classroom interaction process and thus hindering the needs of classroom teaching analysis. Although multimodal large language models can provide relatively accurate speaker and content information, they lack accurate timestamps, making it difficult to support refined segmentation and analysis of classroom events.

[0005] Furthermore, classroom audio recordings and videos are typically quite long, generally ranging from 30 to 45 minutes. When directly processing long audio segments, a common approach in existing speech technologies is to segment the audio at fixed time points before performing separate recognition and transcription. This approach can easily result in segmentation points falling in the middle of sentences, affecting subsequent time alignment and speaker identification. Another approach is to use Voice Activity Detection (VAD) for segmentation. However, in classroom scenarios, teacher-student interactions often involve pauses, speech overlap, and weak far-field speech signals, making VAD prone to issues such as lost speech activity, leading to the omission of key utterances.

[0006] From the perspective of speaker identification, each individual identification method has its own shortcomings. Speaker log methods can obtain the number of speakers in the classroom, but cannot determine the specific identity of the speakers; teacher voiceprint verification methods can determine whether a sentence is similar to the voice characteristics of a specific teacher, but may still misidentify sentences with insufficient voiceprint information, such as short sentences, noisy sentences, or sentences with overlapping speech; methods that identify speakers based on the semantic recognition of spoken text are also prone to identification errors in classroom scenarios because similar texts may come from both teachers and students. Therefore, there is an urgent need for a method that can integrate multi-modal feature information to improve the accuracy and robustness of speaker identification in classroom scenarios.

[0007] Existing research on speaker recognition and teacher-student interaction analysis in classroom settings shows that teacher-student dialogues in the classroom environment are often presented in sentence-level discourse. The short duration of these sentence-level interactions and the difficulty in capturing student feature vectors significantly increase the difficulty of speaker recognition in the classroom; relying solely on semantic feature information from a single modality is often insufficient to accurately determine the speaker's identity. While existing speech recognition technologies can transcribe classroom discourse, their output is presented in whole paragraphs, failing to reflect the fine-grained structure of classroom interactions.

[0008] In summary, existing technologies still have many shortcomings in processing long-form classroom audio transcription, separating teacher and student discourse identities, and forming complete and effective classroom speaker log records, making it difficult to meet the actual needs of classroom teaching analysis and educational research. Therefore, it is necessary to design a method and system for generating classroom teacher and student discourse logs that can integrate sentence-level multimodal information to overcome the limitations of existing technologies. Summary of the Invention

[0009] To address the aforementioned problems, this invention provides a method and system for generating classroom teacher-student discourse logs that integrates sentence-level multimodal information. When dealing with long audio recordings in the classroom, especially in complex classroom scenarios such as frequent teacher-student speech transitions, short and weak student speech, speech overlap, and noise interference, this invention solves the technical problems of recognition errors, information loss, and output formats unsuitable for teaching analysis in existing methods. It achieves precise time positioning of teacher-student discourse, accurate speaker identification, and automatic generation of structured sentence-level logs.

[0010] This invention provides a method for generating classroom teacher-student discourse logs that integrates sentence-level multimodal information, the method comprising: Get the full lecture audio; The entire class audio is pre-segmented to obtain multiple sub-audio files; and each sub-audio file is transcribed and aligned with word-level time to obtain a transcribed text sequence and a word-level time sequence. Based on the word-level time sequence and the ending punctuation marks of the transcribed text sequence, sentence-level discourse records are generated and spliced ​​together to obtain the sentence-level record of the entire lesson; Speaker log processing is performed on the entire class audio to obtain speaker log segment information; Based on the sentence-level record of the entire lesson, the audio segments of each sentence are obtained, and the teacher's voiceprint is verified for each sentence audio segment to obtain the teacher matching score for each sentence. By integrating the speaker log fragment information with the teacher matching score, speaker identity is identified for sentence-level discourse to obtain the initial speaker identity for each sentence; By integrating the speaker log fragment information, the teacher matching score, and the text and contextual semantic information in the sentence-level discourse records, the identities of sentences whose initial speaker identities were identified as uncertain are corrected to obtain the final classroom teacher-student discourse log.

[0011] Furthermore, the pre-segmentation process includes: determining the theoretical cutting point based on the preset maximum segmentation duration, setting an extended search interval on both sides of the theoretical cutting point, and finding the lowest energy point within the extended search interval based on the total short-time energy as the actual cutting point for audio segmentation.

[0012] Furthermore, audio segmentation is performed by finding the lowest energy point within the extended search interval based on the sum of short-time energy, which serves as the actual segmentation point. Specifically, this includes: Within the extended search interval, calculate the sum of short-time energy within each sliding window. , ,in, As the starting point of the window, For the index of the sampling points within the window, The length of the sliding window; Select Minimum window position The sampling point with the smallest absolute amplitude within the candidate low-energy window is selected as the actual cutting point.

[0013] Furthermore, sentence-level discourse records are generated, specifically including: dividing the original sentences according to the end punctuation marks of the character-level time sequence and the transcribed text sequence, and mapping the character-level time information to each sentence; for sentences that lack end punctuation marks and whose duration exceeds a preset threshold, supplementary sentences are generated according to the pause gaps between adjacent characters.

[0014] Furthermore, by piecing together the data, we obtain a sentence-level record of the entire lesson, specifically including: Obtain the starting offset of each sub-audio element within the overall audio of the lesson; Add the corresponding start offset to the local start time and local end time of each sentence in the sentence-level speech record generated from each sub-audio to obtain the global start time and global end time of each sentence; All sub-audio sentence-level utterance records are sorted and renumbered according to the global start time to obtain the sentence-level records of the entire lesson.

[0015] Furthermore, teacher voiceprint verification is performed on audio at each sentence level, specifically including: A teacher voiceprint database is established and teacher voiceprint feature vectors are extracted. Based on the start and end times of each sentence in the whole class's sentence-level record, sentence audio segments are cut out from the whole class audio. The speaker embedding vector of the sentence audio segment is extracted and its similarity is calculated with the teacher voiceprint feature vector to obtain the teacher matching score.

[0016] Furthermore, by integrating the speaker log fragment information with the teacher matching score, speaker identification is performed on sentence-level utterances to obtain the initial speaker identity for each sentence, specifically including: Calculate the temporal overlap between each sentence and the speaker's log segment information. Determine the speaker clustering information corresponding to the sentence; among which, the first The time interval for each sentence is , No. The time range of the speaker's log segment information is: ; From the sentences whose teacher matching scores are higher than the first preset threshold and whose time overlap is normal, the cumulative duration of different speaker clustering results is counted, and the clustering result with the largest cumulative duration is determined as the teacher speaker cluster; When a sentence's teacher matching score is higher than the second preset threshold and belongs to the teacher speaker cluster, it is identified as a teacher; When a sentence's teacher matching score is below the third preset threshold and it does not belong to the teacher speaker cluster, it is identified as a student. In all other cases, the identity is identified as uncertain.

[0017] Furthermore, the speaker clustering information corresponding to the sentence is determined, specifically including: Calculate the main overlap ratio between the sentence and the speaker's log fragment information. and dominant proportion ,in, The length of the sentence duration interval. ; When the main overlap ratio Not lower than the first overlap threshold and the dominant proportion If the value is not lower than the second overlap threshold, the speaker clustering result corresponding to the sentence is considered reliable.

[0018] Furthermore, for sentences where the initial speaker identity is identified as uncertain, identity correction is performed, specifically including: Construct structured input information that includes sentence number, sentence text, sentence context semantic information, start time, end time, current speaker identity information, speaker clustering results, teacher matching score, and time quality; The structured input information is input into a language model, which corrects the identity of sentences identified as having uncertain identities based on the phonological and semantic cues in the structured input.

[0019] On the other hand, the present invention also provides a classroom teacher-student discourse log generation system that integrates sentence-level multimodal information, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the above-described methods.

[0020] Overall, this invention provides a method and system for generating classroom teacher-student discourse logs that integrates sentence-level multimodal information, and compared with existing technologies, it can achieve the following beneficial effects: (1) This invention constructs a complete end-to-end classroom teacher-student discourse log generation scheme by sequentially executing steps such as audio pre-segmentation and transcription alignment, sentence-level discourse record generation, speaker log processing, teacher voiceprint verification, multi-source speech feature fusion recognition, and speech-semantic feature fusion correction. Compared with existing technologies, this invention can simultaneously achieve accurate sentence-level time positioning, accurate speaker identification, and structured log output in long audio classroom scenarios, significantly improving the automation level and data quality of classroom discourse analysis.

[0021] (2) This invention employs an audio pre-segmentation method based on low-energy boundaries, which determines the actual segmentation point near the theoretical segmentation point through short-time energy search. Compared with fixed-duration segmentation and VAD segmentation, this invention effectively avoids the problem of a sentence being truncated due to the segmentation point falling in the middle of the sentence, and also prevents the loss of speech activity detection due to speech pauses or weak signals, thus ensuring the integrity and continuity of the speech content in the subsequent speech transcription and word-level time alignment process.

[0022] (3) This invention performs sentence-level segmentation by combining punctuation marks and pauses between characters, and then concatenates the sentence-level records of each segment according to global time after offset correction. Compared with the existing technology that directly outputs the entire text, this invention can generate sentence-level discourse records with precise timestamps, fully preserving the fine-grained structure of classroom interactions, and providing directly usable structured data for teaching event segmentation and classroom behavior analysis.

[0023] (4) This invention achieves effective fusion of speaker log temporal clustering information and voiceprint feature information by calculating the temporal overlap relationship between sentences and speaker log segments and combining it with teacher voiceprint matching scores for comprehensive judgment. Compared with methods that rely solely on speaker logs or voiceprint verification, this invention can significantly reduce recognition errors caused by students' short speeches, overlapping speech, or noise interference in classroom scenarios, and improve the accuracy of distinguishing between teachers and students.

[0024] (5) This invention identifies sentences with uncertain identities when information is insufficient, rather than forcibly assigning incorrect labels, and then makes targeted corrections in subsequent steps. This strategy of retaining first and then correcting effectively avoids generating a large number of incorrect judgments on short sentences, boundary sentences, or sentences with insufficient voiceprint features. While ensuring the credibility of the overall results, it leaves room for accurate correction at the semantic level. In addition, a language model is further introduced. By fusing sentence-level speech feature information with discourse text and its contextual semantic feature information, the identity of sentences initially identified as having uncertain identities is corrected. It can effectively distinguish similar expressions that teachers and students may share with the help of contextual semantic information. It is especially suitable for scenarios with insufficient speech features, such as short student responses and choral reading, and significantly improves the completeness and accuracy of the final log. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0026] Figure 1This is a flowchart illustrating the method and system for generating classroom teacher-student discourse logs that integrates sentence-level multimodal information, as provided by this invention. Figure 2 This is a schematic diagram illustrating the principle of a method and system for generating classroom teacher-student discourse logs that integrates sentence-level multimodal information, as provided by this invention. Figure 3 This is a pre-segmentation diagram of a classroom teacher-student discourse log generation method and system that integrates sentence-level multimodal information provided by the present invention; Figure 4 This is a schematic diagram of the speech transcription and character-level time alignment of a classroom teacher-student speech log generation method and system that integrates sentence-level multimodal information provided by the present invention; Figure 5 This is a schematic diagram of a classroom teacher-student discourse log generation method and system that integrates sentence-level multimodal information, provided by the present invention, for teacher voiceprint verification and sentence-level speaker identity recognition. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0028] It should be noted that, in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method, step, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to the method, step, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of additional identical elements in the method, step, or apparatus that includes the element.

[0029] To address the technical problems of existing methods, such as recognition errors, missing information, and output formats unsuitable for teaching analysis, this invention proposes a method and system for generating classroom teacher-student discourse logs that integrates sentence-level multimodal information. By performing low-energy boundary pre-segmentation, speech transcription, and word-level time alignment on long classroom audio recordings to generate sentence-level discourse records, the method then integrates speaker log clustering results, teacher matching scores, and text semantic information to perform multi-level identity recognition and correction. Finally, it outputs sentence-level classroom teacher-student discourse logs containing accurate timestamps and speaker identities.

[0030] like Figure 1 and Figure 2As shown, the method specifically includes: S100: Get the audio of the entire lesson.

[0031] Extracting the entire lesson's audio from a recorded classroom video, or directly reading the classroom audio file. Preprocessing the entire lesson's audio involves standardizing the audio to obtain a standard audio format with mono, fixed sampling rate, and fixed encoding, which will then be used for subsequent processing.

[0032] Specifically, this embodiment extracts audio from a 40-minute recorded video of a Chinese classroom. ffmpeg is used to standardize the audio, converting the output audio into a mono, 16kHz sampling rate, PCM S16LE encoded WAV format to ensure input consistency for subsequent speech recognition, timestamp generation, and classroom speaker identification processes.

[0033] S200: Pre-segment the audio of the entire class to obtain multiple sub-audio files; and perform speech transcription and word-level time alignment on each sub-audio file to obtain the transcribed text sequence and word-level time sequence.

[0034] It should be noted that the entire lecture audio is first pre-segmented using low-energy boundaries to obtain sub-audio segments that meet the input requirements of the ASR (Automatic Speech Recognition) model and are free of anomalies. Pre-segmentation is one of the keys to solving long audio problems. It does not involve fixed-time cutting or using complex VADs. Instead, it first calculates a theoretical cut point, and then finds a point near this point with the lowest short-time energy that is closest to silence as the actual cut point. In this way, each segment will not break up a sentence or lose any content.

[0035] Preferably, for classroom audio segments that exceed the preset processing limit, pre-segmentation processing is only performed during speech transcription and word-level time alignment.

[0036] As one embodiment, the pre-segmentation process includes: determining the theoretical cut point based on a preset maximum segmentation duration; setting extended search intervals on both sides of the theoretical cut point; and finding the lowest energy point within the extended search interval based on the sum of short-time energy as the actual cut point for audio segmentation. This low-energy boundary pre-segmentation process solves the problems of sentence truncation caused by fixed segmentation of long audio and content loss due to VAD (Voice-Activated Discourse), ensuring the integrity of sentence-level discourse.

[0037] If the discrete audio signal is Sampling rate The target maximum segment duration is The theoretical tangent point corresponds to the sample position. for: ; in, This represents the floor function, used to round down. The corresponding continuous values ​​are converted into integer sample positions.

[0038] An extended search interval is set on both sides of the theoretical tangent point, and the extension time is denoted as . The search range is: Within this extended search interval, for lengths of Sliding window calculation of short-time total energy .

[0039] Furthermore, based on the sum of short-time energy, the lowest energy point within the extended search interval is found as the actual segmentation point for audio segmentation. Specifically, this includes: calculating the sum of short-time energy within the sliding window of each segment within the extended search interval. , ,in, As the starting point of the window, For the index of the sampling points within the window, The length of the sliding window; select Minimum window position The sample point with the smallest absolute amplitude within the candidate low-energy window is selected as the actual split point.

[0040] Each sub-audio segment is connected end-to-end without overlap or gaps. Sub-audio segments with excessively short endings are padded with minimum duration to meet the input requirements of subsequent ASR models.

[0041] Then, the ASR model is used to transcribe the sub-audio of each pre-segment into speech, resulting in a transcribed text sequence of each segment containing text and punctuation.

[0042] Specifically, the Transcription of an audio clip for: ,in, Indicates the first Each character or word.

[0043] Finally, a forced alignment model (such as Qwen3-ForcedAligner) is used to precisely match each word in the transcribed text sequence with its start and end times in the audio, resulting in a word-level time sequence.

[0044] As an example, character-level time alignment uses a non-autoregressive forced alignment model to align each character or word in the transcribed text with its corresponding position in the audio, and outputs the start and end times of each character or word.

[0045] For example, using the Forced Aligner non-autoregressive forced alignment model, each audio segment is first segmented into frames, and Mel-spectral features are extracted as frame-level audio feature representations. Simultaneously, each character or word in the transcribed text sequence is temporally encoded to obtain a text temporal encoding representation. Then, using the Viterbi decoding algorithm, under the condition of satisfying text order constraints, the audio feature representation and the text temporal encoding representation are optimally matched in the time dimension to determine the continuous audio frame interval corresponding to each character or word. Finally, based on the time information of each frame, the start and end times of each character or word are output, forming a word-level time sequence. .

[0046] Specifically, the Word-level time series of audio segments The calculation method is as follows: ; in, Indicates the first A character or word element, Indicates the time of the character or word. Indicates the start time of the character or word. Indicates the end time of the character or word.

[0047] In a specific implementation process, in order to adapt to the processing requirements of longer pre-segmentation, each sub-audio can still be further divided into smaller internal blocks according to the low-energy boundary, and then the internal blocks are transcribed and aligned separately, and finally the results are spliced ​​within the segment.

[0048] For example, based on the audio file of the entire lesson, a long audio pre-segmentation step is first introduced in the speech-to-text front-end to avoid boundary errors caused by model input length limitations and fixed audio segmentation. Figure 3 As shown, the entire audio signal is The total sample size is Sampling rate The target maximum segment duration is .when At that time, pre-segmentation processing is performed. Specifically, A value of 285s is chosen, which is less than the single-segment upper limit of 300s, reserving space for subsequent time alignment. As mentioned earlier, an extended search interval is constructed near the theoretical tangent point to find the low-energy boundary based on the short-time absolute amplitude accumulation. Extended duration. The time interval is 5 seconds, and the sliding window length W is 100 ms. Furthermore, for excessively short segments after segmentation, zero-value padding can be performed at the end to meet the minimum input requirements of the subsequent speech recognition module. Then, based on each pre-segmented audio sub-segment, speech is transcribed using the Qwen3-ASR speech recognition model to obtain a transcribed text sequence. The transcribed text sequence and the corresponding audio segments are then uniformly input into the Qwen3-ForcedAligner-0.6B forced alignment model. Figure 4 As shown, the model performs time-stamped encoding on the input transcribed text sequence and normalizes the audio segments. The audio feature representation and the text time-encoded representation are input into the forced alignment model, and the start and end times of each character or word are output to form a word-level time series.

[0049] This invention employs a non-autoregressive forced alignment model to achieve character-level time alignment, accurately associating each character in the transcribed text with its corresponding position in the audio, and outputting the start and end times of each character. Compared to traditional whole-segment text time annotation methods, this invention provides a fine-grained temporal information foundation for subsequent sentence-level speech segmentation and speaker identification, significantly improving the accuracy and reliability of timestamps.

[0050] S300: Generate sentence-level discourse records based on the word-level time sequence and the punctuation marks at the end of the transcribed text sequence, and then splice them together to obtain the sentence-level record of the entire lesson.

[0051] As an example, generating sentence-level discourse records specifically includes: dividing the original sentences according to the end punctuation marks of the character-level time sequence and the transcribed text sequence, and mapping the character-level time information to each sentence; for sentences lacking end punctuation marks or whose duration exceeds a preset threshold, supplementing the sentences according to the pause gaps between adjacent characters.

[0052] Specifically, firstly, the character-level time series is associated with the character positions in the transcribed text sequence, so that each character corresponds to its start and end times; then, the text is divided into original sentences according to the ending punctuation marks in the transcribed text sequence, and the start and end times of each original sentence are determined according to the start and end times associated with each character; that is, each sentence is cut out according to semantic boundary markers such as periods, question marks, exclamation marks, and semicolons; the start time of each original sentence is the start time of the first character in the sentence, and the end time is the end time of the last character in the sentence.

[0053] Furthermore, for sentences lacking a closing punctuation mark and whose duration exceeds a preset threshold, supplementary clauses are created at the positions where the time interval between adjacent characters in the character-level time series exceeds a preset pause threshold; this ensures that each sentence has a reasonable duration and semantic integrity. In other words, for sentences lacking a closing punctuation mark or whose duration exceeds a preset threshold, the time interval between adjacent characters is calculated one by one; when there is an adjacent character pair with a time interval exceeding the preset pause threshold, the sentence is segmented at the position between the adjacent character pair, dividing the original sentence into multiple clause segments; after segmentation, the start time of each clause segment is the start time of the first character in that clause segment, and the end time is the end time of the last character in that clause segment.

[0054] For example, if the first The local start time and local time of each candidate sentence are respectively and Then its duration for: When a sentence lacks a closing punctuation mark and When the threshold is exceeded, the sentence is segmented by finding a larger pause.

[0055] Finally, the time quality of each sentence is checked to identify cases such as consecutive zero timestamps, consecutive identical timestamps, non-monotonic increasing times, or abnormal durations. Sentences with the above problems are marked as abnormal status markers and their reasons. Finally, sentence-level discourse records containing sentence number, sentence text, sentence start time, sentence end time, and abnormal status markers and their reasons are generated.

[0056] Specifically, identifying abnormal situations includes: the presence of three or more consecutive characters with start and end times of zero; the presence of three or more consecutive characters with identical start or end times; the presence of a character whose start time is later than the start time of the next character; or the total duration of the sentence exceeding a preset maximum sentence duration threshold. If a sentence exhibits any of these abnormal situations, it is marked as a time-abnormal sentence, and its time status is set to abnormal; otherwise, the time status of the sentence is set to normal.

[0057] By adding the starting offset of the corresponding segment to the sentence-level discourse records of all segments in the whole lesson, concatenating them in global time order, and renumbering them, we obtain the sentence-level records of the whole lesson, which include sentence number, sentence text, sentence start time, and sentence end time.

[0058] As an example, the entire lesson's sentence-level record is obtained by piecing it together, specifically including: Obtain the starting offset of each sub-audio element within the overall audio of the lesson; Add the corresponding start offset to the local start time and local end time of each sentence in the sentence-level speech record generated from each sub-audio to obtain the global start time and global end time of each sentence; All sub-audio sentence-level utterance records are sorted and renumbered according to the global start time to obtain the sentence-level records of the entire lesson.

[0059] As an example, if the first The initial offset of each pre-segmented audio segment throughout the lesson is: Then the global time of any sentence within this segment can be expressed as: ; ;in, , Indicates the local start time and local end time of a sentence. , This indicates the global start and end times of the sentence within the entire lesson. Then, all sentences are sorted by... By sorting and numbering, the sentence-level record of the entire lesson is obtained.

[0060] S400: Perform speaker log processing on the entire class audio to obtain speaker log segment information.

[0061] It should be noted that this step can be processed in parallel with S300. For example, an existing mature speaker log system model (such as CAM++) can be used to process the entire class audio separately to obtain speaker log segment information. Specifically, the speaker log segment information includes anonymous speaker identifiers (such as speaker 1, speaker 2) and their corresponding time information.

[0062] Furthermore, the speaker's log fragment information set is as follows: : ; in, Indicates the first The start time of a speaker's segment. Indicates the first The end time of a speaker's segment. This indicates the speaker clustering result number to which the segment belongs. This indicates the total number of anonymous speaker segments in the entire class.

[0063] In a specific implementation, in a classroom setting, the speaker log processing presets the number of clusters to at least two, corresponding to the speaker categories of teachers and at least one student. That is, the speaker identities in the classroom are fixed as two categories: teachers and students, and the number of clusters is limited to at least two during the speaker separation phase.

[0064] For example, speaker log processing is specifically designed for the complete audio of an entire class session. It employs a speaker log system model based on CAM++ and clustering to process the entire class audio. Specifically, this includes: first, detecting speech activity in the audio, removing silence and non-human voice segments to obtain candidate speech segments; then, extracting speaker embedding features from each candidate speech segment; next, calculating the speaker feature similarity between segments and clustering based on the similarity, grouping segments belonging to the same speaker into one category; finally, post-processing the clustering results, including transition point localization and segment smoothing, outputting speaker log segment information containing start time, end time, and anonymous speaker identifier sequence number.

[0065] S500: Based on the sentence-level record of the whole lesson, the audio segments of each sentence are obtained, and the teacher's voiceprint is verified for each sentence audio segment to obtain the teacher matching score for each sentence.

[0066] In other words, a voiceprint database containing only the teacher's voiceprint is created; based on sentence-level speech records, audio segments of each sentence are extracted from the entire class audio; finally, the voiceprint features of each sentence are extracted and compared with the teacher's voiceprint database to obtain a teacher matching score. The higher the score, the greater the likelihood that the sentence was spoken by the teacher.

[0067] As an example, teacher voiceprint verification is performed on each sentence-level audio file, specifically including: establishing a teacher voiceprint database and extracting teacher voiceprint feature vectors; based on the start and end times of each sentence in the whole class's sentence-level record, extracting sentence audio segments from the whole class audio; extracting the speaker embedding vector of the sentence audio segment, calculating the similarity with the teacher voiceprint feature vector, and obtaining the teacher matching score.

[0068] Furthermore, a teacher voiceprint database containing only the current classroom voiceprint is established, and a teacher voiceprint feature model is created for subsequent teacher voiceprint matching. For each sentence with normal time quality, based on the start and end times of each sentence in the entire class's sentence-level record, combined with the left and right constraints of adjacent sentences with normal time quality, padding is added to both sides of the current sentence, and minimum duration, maximum duration, and adjacency boundary constraints are applied. The audio segment of that sentence is then extracted from the entire class audio. For each sentence audio segment, a speaker embedding vector is extracted using a trained speaker embedding model, and its similarity is calculated with the teacher voiceprint feature vector obtained from the teacher voiceprint feature model to obtain the teacher matching score.

[0069] It should be noted that the trained speaker embedding model can use the publicly available pre-trained CAM++ model, which can receive 16kHz mono audio and output a 192-dimensional fixed-length speaker embedding vector. The network structure, loss function and training method of the model itself are all existing technologies. This invention directly loads its publicly available weights for inference and does not train or fine-tune the model parameters.

[0070] The teacher voiceprint feature model can be constructed as follows: several audio segments confirmed to be spoken by the teacher are manually extracted from the classroom audio of this lesson or audio of the teacher reading aloud is collected before class, and uniformly resampled to 16kHz mono; the embedding vector of each audio segment is extracted by using a pre-trained speaker embedding model; the arithmetic mean of all embedding vectors is calculated to obtain the mean feature vector of the teacher's voiceprint.

[0071] As an example, if the first The speaker embedding vector of the sentence is The teacher voiceprint feature vector in the teacher voiceprint database is: Then the teacher matching score It can be represented as: ; in, Representing vectors sum vector The dot product; , Representing vectors respectively sum vector The Euclidean norm, It represents the product of the norms of two vectors; This represents the cosine similarity between the audio segment of the sentence and the teacher's voiceprint. The larger the value, the closer the speaker features of the audio segment of the sentence are to the teacher's voiceprint features, thus obtaining the teacher matching score for each sentence.

[0072] For example, to provide independent feature information about the teacher's speaker identity, a teacher voiceprint database is established. The root directory of the teacher voiceprint database contains only one teacher subdirectory, which stores the teacher's registration audio. Subsequently, based on the sentence-level recording results of the entire lesson, for each sentence with normal time quality, the audio of the sentence is trimmed according to the sentence's time range and the boundaries of adjacent normal sentences, resulting in sentence audio segments. During trimming, 0.2s of padding is added to both sides of the sentence. The minimum duration of the sentence audio is 1s, and the maximum duration is 4s. Finally, for the trimmed sentence audio segments, a trained speaker embedding model is used to extract speaker embedding vectors, and the cosine similarity is calculated with the teacher voiceprint feature vectors in the teacher voiceprint database to obtain the teacher matching score.

[0073] S600: By integrating speaker log fragment information with teacher matching scores, speaker identity is identified for sentence-level discourse to obtain the initial speaker identity for each sentence.

[0074] Specifically, the speaker log clustering information and teacher voiceprint information of each sentence are integrated, and the maximum likelihood estimation of the teacher speaker clustering is used to match and calculate with the sentence-level discourse record information to identify the speaker identity of the sentence.

[0075] As one example, by integrating speaker log fragment information with teacher matching scores, speaker identification is performed on sentence-level discourse to obtain the initial speaker identity for each sentence, specifically including: Calculate the temporal overlap between each sentence and the speaker's log segment information. Determine the speaker clustering information corresponding to the sentence; among which, the first The time interval for each sentence is , No. The time range of the speaker's log segment information is: ; From sentences where the teacher matching score is higher than the first preset threshold and the time overlap is normal, the cumulative duration of different speaker clustering results is counted, and the clustering result with the largest cumulative duration is determined as the teacher speaker cluster; When a sentence's teacher matching score is higher than the second preset threshold and belongs to the teacher speaker cluster, it is identified as a teacher; When the teacher matching score of a sentence is lower than the third preset threshold and does not belong to the teacher speaker cluster, it is identified as a student; In all other cases, the identity is identified as uncertain.

[0076] Furthermore, the speaker clustering information corresponding to the sentence is determined, specifically including: Calculate the main overlap ratio between the sentence and the speaker's log fragment information. and dominant proportion ,in, The length of the sentence duration interval. ; When the main overlap ratio Not lower than the first overlap threshold and the dominant proportion If the value is not lower than the second overlap threshold, the speaker clustering result corresponding to the sentence is considered reliable.

[0077] For example, the first threshold is 0.6 and the second threshold is 0.75.

[0078] Furthermore, by integrating the speaker log features from speaker log fragments with the voiceprint features from teacher matching scores, the cumulative duration of sentences corresponding to different speaker clustering results is calculated from sentences with normal time quality and teacher matching scores not lower than the first preset threshold. The speaker clustering result with the largest cumulative duration is then determined as the teacher speaker cluster.

[0079] If the set of high-confidence sentences that meet the conditions is , No. The speaker clustering result of the sentence is Then the teacher speaker clustering model for: ; in, This means selecting the cluster number with the largest cumulative duration from all candidate speaker cluster results as the teacher speaker cluster.

[0080] In other words, if a sentence highly overlaps with an anonymous speaker, that speaker's label (e.g., Speaker 1) is assigned to that sentence. Then, from sentences with high teacher matching scores, the corresponding anonymous speaker labels are counted; the label with the longest total duration is preliminarily identified as a teacher. Finally, combining voiceprint scores and cluster labels, a comprehensive judgment is made for each sentence: if the sentence has normal time quality, a high voiceprint score, and belongs to the teacher cluster, the speaker is labeled as a teacher; if the sentence has normal time quality, a low voiceprint score, and does not belong to the teacher cluster, the speaker is labeled as a student; other situations or insufficient information to determine the identity are labeled as uncertain.

[0081] For example, such as Figure 5 As shown, multi-source speech features are fused to identify the speaker identity at the sentence level. For each sentence, its temporal overlap with speaker log fragment information is calculated, and the primary overlap ratio and dominant ratio are further obtained. When both the primary overlap ratio and dominant ratio reach a set threshold, it proves that the speaker clustering result information of the sentence is usable. From sentences with normal timing and teacher matching scores higher than the first preset threshold, the teacher speaker clustering results are inferred. Subsequently, the obtained voiceprint feature information and anonymous speaker identity features are fused to complement and integrate information, and speaker identity is identified for each sentence: when the teacher matching score is higher than 0.53 and corresponds to the teacher speaker clustering result, it is identified as a teacher; when the teacher matching score is lower than 0.45 and does not correspond to the teacher speaker clustering result, it is identified as a student; otherwise, it is identified as uncertain.

[0082] S700: It integrates speaker log fragment information, teacher matching scores, and text and contextual semantic information in sentence-level discourse records, and corrects the identity of sentences whose initial speaker identity is uncertain, to obtain the final classroom teacher-student discourse log.

[0083] As an example, identity correction is performed on sentences whose initial speaker identity is identified as uncertain. Specifically, this includes: constructing structured input information containing sentence number, sentence text, sentence context semantic information, start time, end time, current speaker identity information, speaker clustering results, teacher matching score, and time quality; inputting the structured input information into a language model, and the language model performing identity correction on sentences identified as having uncertain identities based on the phonological and semantic cues in the structured input.

[0084] In other words, this invention, for sentences marked as uncertain, no longer relies solely on phonological information but introduces semantic information. It constructs a structured prompt containing multimodal information such as sentence text, contextual semantic information, start time, end time, current speaker identity information, speaker clustering results, teacher matching score, and time quality. This prompt is then input into a language model, which, combined with existing phonological cues, determines who spoke the sentence. For example, a simple "Yes!" can be inferred from a teacher's question as a student's answer. After correction, a classroom discourse log containing sentence number, sentence text, start time, end time, and corrected speaker identity label is obtained, and the output log undergoes a consistency check.

[0085] The language model can employ publicly available pre-trained large language models, such as Qwen2.5-7B-Instruct or ChatGLM3-6B. The language model only allows modification of uncertain sentence identities, without altering already defined sentence identities or any textual or temporal information, thus ensuring the stability and reliability of the results.

[0086] This invention does not force a judgment when information is insufficient, but marks it as uncertain and uses a stronger language model to process it, thus avoiding a large number of errors on difficult samples and improving the overall credibility of the final result.

[0087] On the other hand, the present invention also provides a classroom teacher-student discourse log generation system that integrates sentence-level multimodal information, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the above-described methods.

[0088] Specifically, the system includes: The audio acquisition and preprocessing module is used to acquire the audio of the entire lesson; The pre-segmentation, transcription, and alignment module is used to pre-segment the audio of the entire class to obtain multiple sub-audio files; and to perform speech transcription and word-level time alignment on each sub-audio file to obtain the transcribed text sequence and word-level time sequence. The whole lesson sentence-level record generation module is used to generate sentence-level discourse records based on the word-level time sequence and the ending punctuation marks of the transcribed text sequence, and then splice them together to obtain the whole lesson sentence-level record; The speaker log processing module is used to process the speaker log of the entire class audio to obtain speaker log segment information; The teacher voiceprint verification module is used to cut the audio segments of each sentence from the sentence-level records of the whole lesson, and to verify the teacher voiceprint of each sentence audio segment to obtain the teacher matching score for each sentence. The classroom speaker identification module is used to integrate speaker log fragment information with teacher matching scores to identify the speaker identity of sentence-level discourse and obtain the initial speaker identity for each sentence. The classroom speaker identity correction module is used to correct the identity of sentences whose initial speaker identity was identified as uncertain. The results output module is used to output the final classroom teacher and student discourse log.

[0089] The specific implementation methods of each module are consistent with the descriptions of the corresponding steps in the above methods, and will not be repeated here.

[0090] To investigate how this invention helps in achieving speaker identification, speech transcription, and speech timing in classroom settings, thereby promoting teaching reflection and educational research, related experiments were conducted. Primary school Chinese language classroom recordings were obtained through the Central Radio and Television University's intelligent training data platform. The method and system for generating classroom teacher-student speech logs integrating sentence-level multimodal information were applied to the audio and video files of multiple lessons. The comparative results of a single classroom audio recording using four recognition methods are shown in Tables 1-4.

[0091] Table 1 Comparison results of four identification methods

[0092] As shown in Table 1, this invention significantly outperforms existing methods in various metrics, including speaker log error rate, speech accuracy, macro-average F1 score, teacher F1 score, and student F1 score. Specifically, the speaker log error rate is only 3.98%, far lower than other methods; the student F1 score reaches 87.23%, more than double that of the best existing method, indicating that this invention has significant advantages in challenging scenarios such as short student speeches and weak speech.

[0093] Table 2. Speaker identification results of four identification methods in student response sentences.

[0094] As can be seen from Table 2, in scenarios such as students' brief responses and choral reading, existing methods generally misjudge student speeches as teacher speeches or directly omit them. However, the method proposed in this invention can accurately identify the speaker of all student responses, effectively solving the identification problems of student responses, choral responses and brief responses in classroom scenarios.

[0095] Table 3. Results of Classroom Speaker Logs Obtained Through General Listening Comprehension

[0096] Table 4. Classroom speaker log results obtained by this invention

[0097] Tables 3 and 4 show the classroom speaker log results generated by this invention. It can be seen that the log output by this invention includes sentence number (Sentence_id), speaker role (Role), start time (Start_time), end time (End_time), and text content (text), forming a complete structured classroom teacher-student discourse log; it can clearly reproduce the classroom teacher-student interactive speaking process, providing data and technical support for classroom discourse analysis, teaching event segmentation, and classroom teaching behavior recognition.

[0098] In summary, this invention proposes a method and system for generating classroom teacher-student discourse logs that integrates sentence-level multimodal information. First, the entire audio of the lesson is pre-segmented based on low-energy boundaries to avoid sentence truncation and content loss. Then, each segment undergoes speech-to-text transcription and character-level time alignment. Sentence-level discourse records with precise timestamps are generated based on punctuation marks and pauses between characters. These are then concatenated globally to obtain the entire lesson's sentence-level record. Regarding speaker identification, on one hand, the entire classroom audio is processed into speaker logs to obtain an anonymous speaker segment sequence. On the other hand, each sentence undergoes teacher voiceprint verification to obtain a teacher matching score. By calculating the temporal overlap between sentences and speaker log segments and combining this with the teacher matching score, multi-source speech feature information is integrated to identify teachers, students, and uncertain identities. For sentences identified as having uncertain identities, a structured input containing sentence text, contextual semantics, and speech feature prompts is further constructed. Identity correction is performed using a language model, ultimately outputting the classroom teacher-student discourse log. This invention effectively improves the accuracy of speaker identification in classroom scenarios by multi-level fusion of speech and semantic features, providing reliable data support for classroom discourse analysis, teaching event segmentation, and teaching behavior research.

[0099] It should be noted that, for the sake of simplicity, the foregoing embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0100] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0101] In the several embodiments provided in this application, it should be understood that the disclosed methods or systems can be implemented in other ways. For example, the embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0102] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0103] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0104] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0105] Those skilled in the art will understand that all or part of the circuits in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0106] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0107] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0108] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating classroom teacher-student discourse logs that integrates sentence-level multimodal information, characterized in that, The method includes: Get the full lecture audio; The entire class audio is pre-segmented to obtain multiple sub-audio files; and each sub-audio file is transcribed and aligned with word-level time to obtain a transcribed text sequence and a word-level time sequence. Based on the word-level time sequence and the ending punctuation marks of the transcribed text sequence, sentence-level discourse records are generated and spliced ​​together to obtain the sentence-level record of the entire lesson; Speaker log processing is performed on the entire class audio to obtain speaker log segment information; Based on the sentence-level record of the entire lesson, the audio segments of each sentence are obtained, and the teacher's voiceprint is verified for each sentence audio segment to obtain the teacher matching score for each sentence. By integrating the speaker log fragment information with the teacher matching score, speaker identity is identified for sentence-level discourse to obtain the initial speaker identity for each sentence; By integrating the speaker log fragment information, the teacher matching score, and the text and contextual semantic information in the sentence-level discourse records, the identities of sentences whose initial speaker identities were identified as uncertain are corrected to obtain the final classroom teacher-student discourse log.

2. The method for generating classroom teacher-student discourse logs that integrates sentence-level multimodal information according to claim 1, characterized in that, The pre-segmentation process includes: determining the theoretical cutting point based on the preset maximum segmentation duration, setting an extended search interval on both sides of the theoretical cutting point, and finding the lowest energy point within the extended search interval based on the total short-time energy as the actual cutting point for audio segmentation.

3. The method for generating classroom teacher-student discourse logs integrating sentence-level multimodal information according to claim 2, characterized in that, Audio segmentation is performed by finding the lowest energy point within the extended search interval based on the sum of short-time energy, which is then used as the actual segmentation point. Specifically, this includes: Within the extended search interval, calculate the sum of short-time energy within each sliding window. , ,in, As the starting point of the window, For the index of the sampling points within the window, The length of the sliding window; Select Minimum window position The sampling point with the smallest absolute amplitude within the candidate low-energy window is selected as the actual cutting point.

4. The method for generating classroom teacher-student discourse logs that integrates sentence-level multimodal information according to claim 1, characterized in that, The process of generating sentence-level discourse records includes: dividing the original sentences based on the end punctuation marks of the character-level time sequence and the transcribed text sequence, and mapping the character-level time information to each sentence; for sentences that lack end punctuation marks and whose duration exceeds a preset threshold, supplementing the sentences based on the pause gaps between adjacent characters.

5. The method for generating classroom teacher-student discourse logs that integrates sentence-level multimodal information according to claim 4, characterized in that, By piecing together the information, we obtain a sentence-level record of the entire lesson, specifically including: Obtain the starting offset of each sub-audio element within the overall audio of the lesson; Add the corresponding start offset to the local start time and local end time of each sentence in the sentence-level speech record generated from each sub-audio to obtain the global start time and global end time of each sentence; All sub-audio sentence-level utterance records are sorted and renumbered according to the global start time to obtain the sentence-level records of the entire lesson.

6. The method for generating classroom teacher-student discourse logs by integrating sentence-level multimodal information according to claim 1, characterized in that, Teacher voiceprint verification was performed on audio at each sentence level, specifically including: A teacher voiceprint database is established and teacher voiceprint feature vectors are extracted. Based on the start and end times of each sentence in the whole class's sentence-level record, sentence audio segments are cut out from the whole class audio. The speaker embedding vector of the sentence audio segment is extracted and its similarity is calculated with the teacher voiceprint feature vector to obtain the teacher matching score.

7. The method for generating classroom teacher-student discourse logs integrating sentence-level multimodal information according to claim 1, characterized in that, By integrating the speaker log fragment information with the teacher matching score, speaker identification is performed on sentence-level utterances to obtain the initial speaker identity for each sentence, specifically including: Calculate the temporal overlap between each sentence and the speaker's log segment information. Determine the speaker clustering information corresponding to the sentence; among which, the first The time interval for each sentence is , No. The time range of the speaker's log segment information is: ; From the sentences whose teacher matching scores are higher than the first preset threshold and whose time overlap is normal, the cumulative duration of different speaker clustering results is counted, and the clustering result with the largest cumulative duration is determined as the teacher speaker cluster; When a sentence's teacher matching score is higher than the second preset threshold and belongs to the teacher speaker cluster, it is identified as a teacher; When a sentence's teacher matching score is below the third preset threshold and it does not belong to the teacher speaker cluster, it is identified as a student. In all other cases, the identity is identified as uncertain.

8. The method for generating classroom teacher-student discourse logs integrating sentence-level multimodal information according to claim 7, characterized in that, Determine the speaker clustering information corresponding to the sentence, specifically including: Calculate the main overlap ratio between the sentence and the speaker's log fragment information. and dominant proportion ,in, The length of the sentence duration interval. ; When the main overlap ratio Not lower than the first overlap threshold and the dominant proportion If the value is not lower than the second overlap threshold, the speaker clustering result corresponding to the sentence is considered reliable.

9. The method for generating classroom teacher-student discourse logs integrating sentence-level multimodal information according to claim 1, characterized in that, For sentences whose initial speaker identity is identified as uncertain, identity correction is performed, specifically including: Construct structured input information that includes sentence number, sentence text, sentence context semantic information, start time, end time, current speaker identity information, speaker clustering results, teacher matching score, and time quality; The structured input information is input into a language model, which corrects the identity of sentences identified as having uncertain identities based on the phonological and semantic cues in the structured input.

10. A classroom teacher-student discourse log generation system integrating sentence-level multimodal information, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

Images