A teaching behavior analysis method and device, an electronic device, and a storage medium

Abstract

This disclosure provides a method, apparatus, electronic device, and storage medium for analyzing teaching behavior. The method includes: acquiring a speech signal in a target teaching scenario; performing text-to-text conversion on the speech signal; and performing voiceprint processing on the speech signal; decomposing the speech-to-text conversion content obtained from the text conversion based on the voiceprint feature information of at least one speaker obtained from the voiceprint processing, determining the speech-to-text conversion content of each speaker within a corresponding speaking time period; and performing teaching behavior analysis based on the speech-to-text conversion content of each speaker to determine the teaching behavior analysis result in the target teaching scenario. This disclosure can realize teaching behavior analysis for various roles such as teachers and students in various teaching scenarios, and is sufficient to adapt to the needs of various complex application scenarios.

Description

Technical Field

[0001] This disclosure relates to the field of speech processing technology, and more specifically, to a teaching behavior analysis method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of the internet, online education has become increasingly popular. Online education and research allow for flexible learning without time or location restrictions, which can greatly enhance one's skills.

[0003] In this process, students learn in the classroom, and teachers teach relevant teaching content. Inevitably, there will be student speeches, teacher-student interactions, and free discussions in the classroom. The addition of these elements significantly increases the complexity of teaching behavior analysis (such as the coverage of teaching knowledge points displayed on the smart blackboard, the level of teacher-student interaction, students' expression ability, etc.). This greatly limits teachers' ability to fully understand their own teaching situation and makes it impossible to provide the most suitable analysis results for students to improve their own classroom performance. Summary of the Invention

[0004] This disclosure provides at least one teaching behavior analysis method, apparatus, electronic device, and storage medium to meet the application needs of more complex teaching scenarios.

[0005] In a first aspect, embodiments of this disclosure provide a method for analyzing teaching behavior, including:

[0006] Acquire audio signals in the target teaching scenario;

[0007] The speech signal is subjected to text conversion processing to determine the speech-to-text conversion content; and the speech signal is subjected to voiceprint processing to determine the voiceprint feature information of at least one speaker in the target teaching scenario.

[0008] Based on the voiceprint feature information of the at least one speaker, the speech-to-text conversion content is decomposed to determine the speech-to-text conversion content of each speaker within the corresponding speaking time period;

[0009] Based on the speech-to-text conversion content of each speaker within the corresponding speaking time period, the teaching behavior analysis results are determined for the target teaching scenario.

[0010] In one possible implementation, when the target teaching scenario includes a teacher-led instruction scenario, determining the teaching behavior analysis results under the target teaching scenario includes:

[0011] Key words were extracted from the audio-text conversion of teachers during their corresponding speaking time periods to identify key knowledge points; and teachers' teaching behavior data were statistically analyzed during their corresponding speaking time periods.

[0012] The quality of a teacher's instruction is evaluated based on the key knowledge points and / or the teaching behavior data.

[0013] The quality of the lecture is determined as the result of the teaching behavior analysis.

[0014] In one possible implementation, when the target teaching scenario includes a teacher-student question-and-answer scenario, determining the teaching behavior analysis results under the target teaching scenario includes:

[0015] The quality of responses is analyzed based on the audio-text conversion of students during their respective speaking time periods to determine the data for response quality analysis; and the interactive behavior data of teachers and students during their respective speaking time periods is also statistically analyzed.

[0016] The level of interaction activity between teachers and students is assessed based on the answer quality analysis data and / or the interaction behavior data.

[0017] The level of interactive activity is determined as the result of the teaching behavior analysis.

[0018] In one possible implementation, when the target teaching scenario includes a student group discussion scenario, determining the teaching behavior analysis results under the target teaching scenario includes:

[0019] Based on the audio-text conversion of each student in the group during their respective speaking time period, the discussion behavior data of each student during their respective speaking time period is statistically analyzed.

[0020] The adequacy of student discussions was assessed based on the aforementioned discussion behavior data.

[0021] The sufficiency of the discussion is determined as the result of the teaching behavior analysis.

[0022] In one possible implementation, when the target teaching scenario includes a student presentation scenario, determining the teaching behavior analysis results under the target teaching scenario includes:

[0023] Keyword extraction is performed based on the audio-text conversion of students during their corresponding speaking time periods to determine key content to be displayed; and student display behavior data is collected during their corresponding speaking time periods.

[0024] Based on the key display content and / or the display behavior data, learning suggestions for students are determined.

[0025] The learning suggestions are identified as the results of the teaching behavior analysis.

[0026] In one possible implementation, the text conversion processing of the speech signal includes:

[0027] The speech signal is converted from time domain to frequency domain based on the speech feature extraction model to determine the speech feature information.

[0028] The speech feature information is sequentially input into the acoustic model and the language model to determine the predicted text sequence content;

[0029] The speech-to-text conversion content is determined based on a preset search method and the predicted text sequence content.

[0030] In one possible implementation, the voiceprint processing of the speech signal includes:

[0031] The speech signal is divided into at least one speech segment and at least one non-speech segment;

[0032] The non-speech segments are filtered out, and for the target speech segment in at least one speech segment, the voiceprint feature information of at least one speaker in the target speech segment is determined based on the speech embedding extraction model.

[0033] In one possible implementation, the method further includes:

[0034] Speaker clustering is performed based on the voiceprint feature information determined for each of the aforementioned speech segments to determine the identifier of each speaker and the corresponding speaking time period;

[0035] The process of decomposing the speech-to-text conversion content based on the voiceprint feature information of at least one speaker includes:

[0036] For at least one target speaker, determine from the speech-to-text conversion content the target speaker's identifier and corresponding speaking time period match; and...

[0037] The target speech-to-text conversion content is determined as the speech-to-text conversion content of the target speaker within the corresponding speaking time period.

[0038] In one possible implementation, when multiple speakers speak during the same speaking time period, determining the identifier of each speaker and the corresponding speaking time period includes:

[0039] Speaker clustering is performed based on the voiceprint feature information determined for each of the aforementioned speech segments, and the clustering result for each speaker is determined.

[0040] The clustering results of each speaker are input into the speech overlap detection model to determine the identifiers of multiple speakers corresponding to the same speaking time period.

[0041] In one possible implementation, the method further includes:

[0042] Extract contextual semantic information from the speech-to-text conversion content;

[0043] The speaker's identifier and corresponding speaking time period are verified based on the contextual semantic information.

[0044] In one possible implementation, the target teaching scenario is determined according to one of the following methods:

[0045] In response to a trigger operation on the scene selection button, the teaching scene selected under the trigger operation is determined as the target teaching scene;

[0046] Acquire surrounding teaching scene data, perform data analysis on the surrounding teaching scene data, and determine the corresponding target teaching scene.

[0047] In one possible implementation, the voiceprint processing of the speech signal includes:

[0048] Obtain pre-set role information for the target teaching scenario;

[0049] The character information is used as additional information, and voiceprint processing is performed based on the additional information and the voice signal.

[0050] In one possible implementation, the method further includes:

[0051] In response to a request to display teaching behavior triggered on the target teaching interface, the analysis results of teaching behavior in the target teaching scenario are displayed on the target teaching interface.

[0052] Secondly, this disclosure also provides a teaching behavior analysis device, comprising:

[0053] The acquisition module is used to acquire the voice signal in the target teaching scenario;

[0054] The processing module is used to perform text conversion processing on the speech signal to determine the speech-to-text conversion content; and to perform voiceprint processing on the speech signal to determine the voiceprint feature information of at least one speaker in the target teaching scenario.

[0055] The decomposition module is used to decompose the speech-to-text conversion content based on the voiceprint feature information of the at least one speaker, and determine the speech-to-text conversion content of each speaker in the corresponding speaking time period;

[0056] The analysis module is used to analyze the teaching behavior based on the speech-to-text conversion content of each speaker within the corresponding speaking time period, and to determine the teaching behavior analysis results under the target teaching scenario.

[0057] Thirdly, this disclosure also provides an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the teaching behavior analysis method as described in any one of the first aspects and various embodiments thereof is performed.

[0058] Fourthly, this disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the teaching behavior analysis method as described in any one of the first aspects and its various embodiments.

[0059] Using the aforementioned teaching behavior analysis methods, devices, electronic equipment, and storage media, upon acquiring the speech signal from the target teaching scenario, the system can perform text-to-text conversion to determine the speech-to-text conversion content, and simultaneously perform voiceprint processing to determine the voiceprint feature information of at least one speaker. Then, based on the voiceprint feature information of at least one speaker, the speech-to-text conversion content is decomposed, and teaching behavior analysis is performed based on the decomposed speech-to-text conversion content for each speaker within their corresponding speaking time period. This approach combines the transcription function corresponding to text conversion with the speaker separation function corresponding to voiceprint processing, making the speech-to-text conversion content for each speaker within their corresponding speaking time period more accurate and easier to display. Furthermore, it enables teaching behavior analysis for various roles such as teachers and students in various teaching scenarios, sufficiently adaptable to the needs of various complex application scenarios. For example, it allows teachers to comprehensively understand their teaching performance, and it can provide the most suitable learning suggestions for different students.

[0060] Other advantages of this disclosure will be explained in more detail in conjunction with the following description and accompanying drawings.

[0061] It should be understood that the above description is merely an overview of the technical solution of this disclosure, so as to provide a general understanding of the technical means of this disclosure and to implement it in accordance with the contents of the specification. In order to make the above and other objects, features and advantages of this disclosure more apparent and understandable, specific embodiments of this disclosure are illustrated below. Attached Figure Description

[0062] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. The accompanying drawings are incorporated in and constitute a part of this specification. These drawings illustrate embodiments conforming to this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure. It should be understood that the drawings only illustrate certain embodiments of this disclosure and should not be considered as a limitation on the scope of protection. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. Furthermore, the same reference numerals denote the same components throughout the drawings. In the drawings:

[0063] Figure 1 A flowchart of a teaching behavior analysis method provided by an embodiment of this disclosure is shown;

[0064] Figure 2 An example diagram of a teaching interface is shown in the teaching behavior analysis method provided in this embodiment of the present disclosure;

[0065] Figure 3 A schematic diagram of a teaching behavior analysis device provided in an embodiment of this disclosure is shown;

[0066] Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present disclosure is shown. Detailed Implementation

[0067] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0068] In the description of embodiments disclosed herein, it should be understood that terms such as “comprising” or “having” are intended to indicate the presence of the disclosed features, figures, steps, behaviors, components, portions or combinations thereof in this specification, and do not exclude the possibility of the presence of one or more other features, figures, steps, behaviors, components, portions or combinations thereof.

[0069] Unless otherwise stated, " / " means "or". For example, A / B can mean A or B. In this article, "and / or" is merely a way of describing the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A alone, A and B at the same time, and B alone.

[0070] The terms "first," "second," etc., are used only for ease of description to distinguish identical or similar technical features and should not be construed as indicating or implying the relative importance or number of these technical features. Therefore, a feature defined by "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, the term "multiple" means two or more.

[0071] Research has found that students learn in the classroom, and teachers teach relevant teaching content. Inevitably, there will be student speeches, teacher-student interactions, and free discussions in the classroom. The addition of these elements significantly increases the complexity of teaching behavior analysis (such as the coverage of teaching knowledge points displayed on the smart blackboard, the level of teacher-student interaction, and students' expression abilities). This greatly limits teachers' ability to fully understand their own teaching situation and makes it impossible to provide the most suitable analysis results for students to improve their own classroom performance.

[0072] Some teaching behavior analysis solutions based on speech-to-text (STT) and speaker recognition (SR) technologies have been provided in related technologies, but they still have some technical shortcomings, such as high demand for computing resources, dependence on training data, limitations in real-time processing capabilities, complexity of use, insufficient adaptability to specific scenarios, relatively little community support and documentation, and functional limitations in certain audio analysis tasks. These shortcomings can be summarized as follows:

[0073] (1) High computational resource consumption: The training and inference processes require high computational power, which may not be suitable for environments with limited resources.

[0074] (2) Sensitive to data quality: Model performance depends on high-quality and diverse training data. Insufficient or imbalanced data will affect the recognition effect.

[0075] (3) Poor adaptability to specific scenarios: In specific languages, accents or environments, the model may require additional training or fine-tuning to achieve the desired effect.

[0076] (4) Insufficient community support and documentation: Compared with other mature libraries, such as speaker recognition technology represented by Pyannote, there may be less documentation and community support, making it difficult for users to find solutions when they encounter problems.

[0077] To at least partially address one or more of the aforementioned problems and other potential issues, this disclosure provides a more robust and adaptable method, apparatus, electronic device, and storage medium for analyzing instructional behavior in a variety of complex scenarios.

[0078] To clearly illustrate the embodiments of this disclosure, some concepts that may appear in subsequent embodiments will be introduced first.

[0079] Speech-to-text (STT) technology converts speech signals into text information. Advances in this technology are attributed to the development of deep learning and Natural Language Processing (NLP), particularly the application of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). Through training on a large number of speech signals, the STT system can recognize different accents, speaking speeds, and intonations, thereby improving recognition accuracy.

[0080] Speaker recognition technology is used to identify and verify the identity of a speaker. This technology is generally divided into two categories: speaker identification (identifying the speaker) and speaker verification (confirming the speaker's identity). By analyzing a speaker's voiceprint characteristics, SR systems can accurately identify different speakers in various environments. This is particularly important for analyzing teacher-student interactions in online education, as it helps the system differentiate between different participants, thus enabling more precise behavioral analysis.

[0081] Pyannote is a Python library focused on audio processing and speaker recognition. It utilizes deep learning techniques (such as convolutional neural networks and recurrent neural networks) to automatically extract audio features. It provides various pre-trained models, supporting functions such as speaker activity detection, speaker separation, and speaker recognition, making it suitable for multi-speaker scenarios, such as meeting transcripts and interview analysis. Its core component is the Pipeline, which encapsulates the entire audio processing workflow, including feature extraction and model inference.

[0082] To facilitate understanding of this embodiment, a detailed description of the teaching behavior analysis method disclosed in this disclosure is provided first. The executing entity of the teaching behavior analysis method provided in this disclosure is generally an electronic device with a certain computing capability. This electronic device may include, for example, a terminal device, a server, or other processing devices. The terminal device may be a user equipment (UE), a mobile device, a user terminal, a personal digital assistant (PDA), a handheld device, a wearable device, etc. In some possible implementations, this teaching behavior analysis method can be implemented by a processor calling computer-readable instructions stored in memory.

[0083] See Figure 1The flowchart illustrates the teaching behavior analysis method provided in this embodiment, which includes the following steps S101 to S104:

[0084] S101: Acquire the speech signal in the target teaching scenario;

[0085] S102: Perform text conversion processing on the speech signal to determine the content of the speech-to-text conversion; and perform voiceprint processing on the speech signal to determine the voiceprint feature information of at least one speaker in the target teaching scenario.

[0086] S103: Decompose the speech-to-text conversion content based on the voiceprint feature information of at least one speaker, and determine the speech-to-text conversion content of each speaker in the corresponding speaking time period;

[0087] S104: Analyze teaching behavior based on the speech-to-text conversion content of each speaker within the corresponding speaking time period, and determine the teaching behavior analysis results under the target teaching scenario.

[0088] To facilitate understanding of the teaching behavior analysis method provided in this disclosure, the application scenarios of the method will be briefly introduced first. This teaching behavior analysis method can be applied to various teaching scenarios that require teaching behavior analysis, including but not limited to teacher lectures, teacher-student Q&A scenarios, student group discussions, and student presentations.

[0089] It should be noted that different teaching scenarios often correspond to different roles, and may also involve different numbers of roles. For example, in a teacher-led lecture scenario, there is usually only one role, namely the teacher; while in a teacher-student Q&A scenario, there are usually multiple roles, with both the teacher and students participating simultaneously. Therefore, different role types need to be identified for different application scenarios. In addition, it is necessary to further determine the teaching behavior analysis strategies to be adopted in different application scenarios. For example, in a teacher-led lecture scenario, data such as the teacher's lecture duration and speaking speed need to be paid special attention to; similarly, in a teacher-student Q&A scenario, data such as the frequency of interaction needs to be paid special attention to.

[0090] Since related technologies cannot support the needs of teaching behavior analysis in various application scenarios, especially complex scenarios, this disclosure provides a technical solution for teaching behavior analysis that combines speech-to-text technology (such as Whisper) and speaker separation technology (such as Pyannote). This solution can perform real-time analysis and recording to generate high-quality (playback) data for the classroom. On the other hand, it can quantify participation and interaction effects. For example, by using speaker separation and speaking time statistics, the depth of teacher-student interaction and student participation can be quantified. It can also support personalized improvement, such as providing improvement suggestions to teachers and students based on automated analysis results to further improve teaching quality.

[0091] Here, the speech signal in the target teaching scenario is first acquired. Then, the speech signal undergoes both text-to-text conversion and voiceprint processing. Based on the voiceprint feature information of at least one speaker obtained from the voiceprint processing, the speech-to-text conversion content obtained from the text conversion is decomposed. Finally, teaching behavior analysis is performed based on the decomposed speech-to-text conversion content of each speaker within their corresponding speaking time period. It is evident that this embodiment combines the transcription function corresponding to text conversion with the speaker separation function corresponding to voiceprint processing, making the speech-to-text conversion content for each speaker within their corresponding speaking time period more accurate and easier to display. Furthermore, it can realize teaching behavior analysis for various roles such as teachers and students in various teaching scenarios, thus adapting to the needs of various complex application scenarios.

[0092] In practical applications, the embodiments of this disclosure can achieve transcription through Whisper technology and speaker separation through Pyannote technology. In other words, the embodiments of this disclosure implement a teaching behavior analysis scheme that combines two functions. The specific combination principle will be explained from the following aspects.

[0093] At the algorithm fusion level, the embodiments of this disclosure integrate speech recognition and speaker separation at the algorithm level. For example, the same feature extraction module can be used to share audio preprocessing; in the process of Whisper speech transcription, the speaker embedding model of Pyannote can be accessed in real time to output the separated transcription results; and the transcribed text and speaker tags can be jointly optimized after the recognition stage.

[0094] At the scene fusion level, the embodiments of this disclosure optimize the processing flow for specific teaching scenarios, such as teacher-student Q&A and discussions. For example, in discussion scenarios, Pyannote is used first to separate speakers, while in lecture scenarios, Whisper is relied upon for recognition.

[0095] At the system integration level, the embodiments of this disclosure integrate at the technical framework level. For example, a unified Application Programming Interface (API) or microservices are built to modularize the functions of Whisper and Pyannote; another example is implementing configurable process logic and dynamically selecting appropriate modules according to scenario requirements, etc.

[0096] In practical applications combining Whisper and Pyannote, various technical challenges arise. Data-level challenges include: Whisper and Pyannote may have different time windows or sampling rate requirements, necessitating consistent input data; insufficient high-quality labeled data for specific scenarios (such as classroom discussions) can impact model performance; model-level challenges: both Whisper and Pyannote are computationally intensive models, potentially leading to performance bottlenecks under high concurrency; they may also use different frameworks or technology stacks (such as PyTorch and TensorFlow), requiring cross-model collaboration and increasing integration difficulty; challenges related to the complexity of scene switching: automatically determining whether a lesson is lecturing, Q&A, or discussion in complex teaching scenarios requires additional detection algorithms; furthermore, switching between scenarios necessitates calling different models or adjusting parameters, potentially complicating the processing flow; and user experience challenges: output results must simultaneously achieve high accuracy and high readability (coherent subtitles, clear character identification), placing higher demands on post-processing.

[0097] This disclosure provides corresponding technical means to address the various technical challenges mentioned above. For example, by effectively extracting and representing speaker features, consistency is maintained under different environments and conditions; furthermore, the model's recognition ability in noisy environments is improved, ensuring accurate speaker recognition even under background noise or other interference; in scenarios with multiple speaker interactions, the voices of different speakers are correctly and accurately distinguished and recognized, especially in cases of overlapping speech; the model is optimized to support real-time speaker recognition, reducing latency and improving response speed; furthermore, through further technical development, the model can quickly adapt to new speakers or environments, reducing dependence on large amounts of labeled data; the efficiency and quality of training data labeling are improved, ensuring the effectiveness of model training; the interpretability of the model is enhanced, enabling users to understand the model's decision-making process, especially in cases of misidentification; and the model's recognition ability in different language and cultural contexts is improved, ensuring its broad applicability across languages ​​and cultures.

[0098] In practical applications, the teaching behavior analysis is implemented here by combining the Pyannote-Whisper (hereinafter referred to as PW) architecture, which provides functions such as Automatic Speech Recognition (ASR), speaker separation, timestamp annotation and multilingual support.

[0099] During ASR processing, Faster Whisper is used for speech recognition to obtain speech-to-text results (i.e., the speech-to-text conversion content obtained by processing the speech signal into text); during speaker separation, pyannote.audio is used to separate speakers and obtain speaker identifiers for each time period; then, the ASR results and speaker separation results are merged to obtain transcribed text with speaker information.

[0100] In this embodiment, the final transcribed text can be further customized for different roles and teaching scenarios, making it more applicable.

[0101] The following section will further elaborate on the teaching behavior analysis process provided in the embodiments of this disclosure, using specific teaching scenarios as examples.

[0102] In a teacher-led instruction scenario, the following steps can be used to analyze teaching behaviors:

[0103] Step 1: Extract keywords from the audio-text conversion of the teacher's speech during the corresponding speaking time period to determine the key knowledge points; and collect data on the teacher's teaching behavior during the corresponding speaking time period.

[0104] Step 2: Evaluate the quality of the teacher's instruction based on key knowledge points and / or teaching behavior data;

[0105] Step 3: Determine the quality of instruction as the result of teaching behavior analysis.

[0106] In practical applications, embodiments of this disclosure can utilize Whisper for audio-to-text (ASR) to extract lecture content and generate a transcript; use Pyannote for speaker separation to ensure the teacher's lecture is isolated from other background noise; and use Natural Language Processing (NLP) tools to extract keywords or generate summaries from the transcript, outputting key knowledge points that can aid in teaching review. Furthermore, statistical data on lecture behavior, such as lecture duration and speaking speed, can be used to comprehensively evaluate the quality of the lecture, facilitating a more comprehensive understanding of teaching effectiveness for the teacher.

[0107] In teacher-student question-and-answer scenarios, the following steps can be used to analyze teaching behaviors:

[0108] Step 1: Analyze the quality of students' responses based on the audio-text conversion content during their respective speaking time periods, and determine the data for response quality analysis; and collect data on the interactive behavior of teachers and students during their respective speaking time periods.

[0109] Step 2: Assess the level of interaction activity between teachers and students based on the analysis data of response quality and / or interaction behavior data;

[0110] Step 3: Determine the level of interactive activity as the result of the teaching behavior analysis.

[0111] In practical applications, embodiments of this disclosure can utilize Pyannote to distinguish between the voices of teachers and students; use Whisper to transcribe question-and-answer content and differentiate between questions and answers through contextual analysis. Furthermore, it can analyze the quality of students' answers and their language expression abilities, and collect data on teacher-student interaction behaviors, such as interaction frequency, thereby assessing the level of activity in teacher-student interaction.

[0112] In student group discussion scenarios, the following steps can be used to analyze teaching behaviors:

[0113] Step 1: Based on the audio-to-text conversion of each student's speech during their respective speaking time period, collect data on each student's discussion behavior during their respective speaking time period;

[0114] Step 2: Assess the adequacy of student discussions based on discussion behavior data;

[0115] Step 3: Determine the sufficiency of the discussion as a result of the teaching behavior analysis.

[0116] In practical applications, Pyannote can be used for multi-speaker separation, marking each student's speaking segments. It can also identify each participant in the group discussion and their speaking content. By using discussion behavior data among students, such as discussion duration, the sufficiency of the discussion can be evaluated, further supporting the application needs in this scenario.

[0117] In student presentation scenarios, the following steps can be used to analyze teaching behaviors:

[0118] Step 1: Extract keywords based on the audio-text conversion of students during the corresponding speaking time period to determine key content to be displayed; and collect data on students' display behavior during the corresponding speaking time period.

[0119] Step 2: Determine learning suggestions for students based on key presentation content and / or presentation behavior data;

[0120] Step 3: Confirm the learning suggestions as the results of the teaching behavior analysis.

[0121] In practical applications, the embodiments of this disclosure can utilize NLP tools to perform further semantic analysis on text content, assess logical clarity, language expression ability, and key point coverage, etc. In addition, Pyannote can monitor background noise or other students interrupting, and collect behavioral data such as display / speaking time to provide the most suitable learning suggestions for students.

[0122] Considering the crucial role of transcription and speaker separation functions in realizing the teaching behavior analysis method in the embodiments of this disclosure, the following will elaborate on the following two aspects.

[0123] In a first aspect, the embodiments of this disclosure can perform text conversion processing to achieve transcription function according to the following steps:

[0124] Step 1: Perform time-domain to frequency-domain transformation on the speech signal based on the speech feature extraction model to determine the speech feature information;

[0125] Step 2: Input the speech feature information into the acoustic model and language model in sequence to determine the predicted text sequence content;

[0126] Step 3: Determine the speech-to-text conversion content based on the preset search method and the predicted text sequence content.

[0127] The text conversion process here mainly includes several core steps such as speech feature extraction, acoustic model and language model processing, and decoding.

[0128] In practical applications, speech feature extraction methods such as Mel Frequency Cepstrum Coefficient (MFCC) or Log-Mel Spectrogram are used to extract features, transforming the original speech signal into frequency domain features (e.g., time-spectrum graphs) to provide standardized feature representations for the model input. Then, acoustic models based on Transformer, such as Whisper, can map feature vectors to basic speech units (e.g., phonemes), and language models like Whisper-LargeV2 infer the most reasonable text sequence from the context. Finally, search algorithms (such as Viterbi or Beam Search) are used in conjunction with the acoustic and language models to generate the most probable text sequence as the speech-to-text conversion content.

[0129] Secondly, the embodiments of this disclosure can perform voiceprint processing, speaker clustering, etc., according to the following steps to realize speaker functions:

[0130] Step 1: Divide the speech signal into at least one speech segment and at least one non-speech segment;

[0131] Step 2: Filter out non-speech segments, and for the target speech segment in at least one speech segment, determine the voiceprint feature information of at least one speaker in the target speech segment based on the speech embedding extraction model.

[0132] Step 3: Based on the voiceprint feature information determined for each speech segment, perform speaker clustering to determine the identifier of each speaker and the corresponding speaking time period.

[0133] Speaker segmentation here refers to the process of identifying different speaker segments in an audio file. Its implementation mainly includes the following steps: Voice Activity Detection (VAD), speech embedding extraction, speaker clustering, time alignment, and segmentation optimization.

[0134] First, neural networks (such as RNNs or CNNs) are used to identify speech segments and non-speech segments (such as silence and noise) from audio. Specifically, RNNs or CNNs can be used to process frequency domain features and output the probability of speech activities. The probability values ​​are then converted into time segments of speech and non-speech activities through energy thresholding methods (such as thresholding), which has higher robustness.

[0135] Then, a pre-trained speech embedding extraction model (such as x-vector or d-vector) is used to extract high-dimensional embedding vectors from the audio, which serve as the speaker embedding (corresponding to voiceprint feature information) for each speech segment, thereby distinguishing different speakers. Here, the embedding representation can be information such as the pronunciation pattern, speech rate, and pitch of the captured speech signal.

[0136] Then, based on K-means and DBSCAN clustering algorithms, the speech embeddings are grouped by speaker to determine the identifier of each speaker and their corresponding speaking time period. In practical applications, audio can be sliced ​​using time windows (usually 1-2 seconds), a feature vector is generated for each slice, and then similar speech embeddings are classified into the same speaker.

[0137] Specifically, unsupervised clustering methods such as K-means, DBSCAN, or probabilistic linear discriminant analysis (PLDA), or supervised clustering based on deep learning (such as LSTM and Transformer), can be used to optimize the embedded vector sequence, thereby assigning each time segment to a speaker label. Dynamic programming or sequence alignment algorithms can be used to optimize the segmentation, and the Viterbi algorithm can be combined to infer the optimal path to determine the clustering result.

[0138] Furthermore, the initial clustering results can be time-aligned to ensure accurate boundaries for each speech segment. Dynamic programming algorithms can also be used to time-align the clustering results to further ensure that each speech segment belongs to the correct speaker.

[0139] In practical applications, overlapping speech segments may exist. Embodiments of this disclosure can introduce a speech overlap detection model to label which time segments contain multiple speakers. Specifically, a multi-output neural network is used to label time segments where multiple speakers coexist, and combined with the aforementioned feature space embedding method, overlapping speech segments are classified to different speakers.

[0140] To further optimize speaker separation, embodiments of this disclosure can also combine speech embeddings in the audio with transcribed text to optimize speaker separation through contextual semantic information. For example, transcribed text provided by Whisper can be used as additional input to help eliminate the effects of background noise and unclear pronunciation in the audio.

[0141] Furthermore, this method allows for the alignment of audio timestamps with the transcribed text, improving the segmentation of speaker boundaries. For example, by combining pauses and semantic transition points in the text, the timing of transitions between different speakers can be detected more accurately.

[0142] Pyannote's primary task is not transcribing speech content, but rather analyzing the structure of audio (such as speaker identity and activity). It typically requires an input audio sampling rate of 16kHz, necessitates resampling audio at different sampling rates, and also requires audio amplitude normalization to reduce the impact of recording equipment and environmental noise.

[0143] To achieve a combination of transcriptional and speaker separation functions, embodiments of this disclosure can determine the speech-to-text conversion content corresponding to each speaker by following these steps:

[0144] Step 1: For at least one target speaker, determine the target speech-to-text content that matches the target speaker's identifier and corresponding speaking time period from the speech-to-text conversion content; and,

[0145] Step 2: Determine the target speech-to-text content as the speech-to-text content of the target speaker within the corresponding speaking time period.

[0146] In practical applications, this disclosure combines OpenAI's Whisper speech recognition model with Pyannote.audio's speaker separation technology through a PW architecture, providing an efficient and accurate solution for audio transcription and multi-person dialogue analysis. This solution utilizes the Whisper model to achieve high-quality speech-to-text conversion while using Pyannote.audio technology to identify and separate different speakers. It mainly includes the following aspects:

[0147] 1) Modular architecture: Build a modular framework and dynamically call the functional modules of Whisper and PyAnnote according to the scenario.

[0148] 2) Scene switching: Analyze the characteristics of the audio (such as speech rate, tone, number of speakers) to determine the frequency and content of the speakers in the subtitles, and determine whether it is a lecture, discussion or question and answer scene.

[0149] 3) Distributed processing: Pyannote's speech separation, VAD, and Whisper's speech recognition are distributed across multiple nodes to improve concurrency capabilities.

[0150] 4) API service: The speech recognition and analysis functions are encapsulated as APIs, which can be called by different types of classroom applications.

[0151] Here, to further understand the teaching behavior analysis method provided in the embodiments of this disclosure, a complete implementation step will be provided below:

[0152] The first step is audio input: input the audio file into the system.

[0153] The second step is speech activity detection: using a pre-trained model (such as inputting the audio source into the VAD model) to detect speech activity in the audio, corresponding to non-silent segments, and identifying the parts containing speech.

[0154] The third step is speaker embedding: feature extraction is performed on the detected speech activity, and the speech signal is converted into an embedding vector that can be used for recognition.

[0155] The fourth step is speaker clustering: clustering algorithms are used to cluster the extracted embedding vectors to distinguish different speakers.

[0156] The fifth step is to output the speaker recognition results: output the final speaker recognition results, including the time period and recognition label for each speaker.

[0157] As can be seen, compared with the prior art, the teaching behavior analysis method provided in this disclosure mainly has the following advantages:

[0158] Firstly, higher speech recognition accuracy: Whisper's robustness to noisy environments and overlapping speech makes it perform well in complex scenarios.

[0159] Secondly, deep integration capability: Whisper's language information and PyAnnote's speaker information complement each other, improving the overall recognition effect.

[0160] Third, the advantage of time alignment: the consistency of time stamps between the two greatly reduces the problem of misalignment.

[0161] Fourth, multimodal scalability: combining semantic, acoustic and contextual information enhances the comprehensiveness of speaker recognition.

[0162] To further optimize the analysis of teaching behavior in various teaching scenarios, role information can be pre-set for the target teaching scenario, such as the role of teacher and its voiceprint characteristics, and the role of student and its voiceprint characteristics, etc. The role information is used as additional information, and voiceprint processing is performed based on the additional information and the voice signal.

[0163] In practical applications related to teaching scenarios, the following technical improvements are still needed:

[0164] Firstly, it addresses the specific needs of recording and broadcasting systems, such as improving speaker recognition accuracy through predefined roles. Secondly, it covers multiple scenarios, including various interactive formats such as teacher lectures, teacher-student Q&A, and group discussions. Thirdly, it enables long-duration audio processing, mainly considering that recording and broadcasting systems need to process long-duration, continuously recorded audio. Here, through functional integration, it implements speaker recognition algorithms optimized for specific scenarios (such as classrooms). It segments long-duration audio to reduce cumulative errors and can dynamically adapt to scene changes (such as switching from teacher lectures to student discussions).

[0165] Secondly, regarding long-duration audio processing, the existing Pyannote, as an offline processing tool, is suitable for speaker separation of short audio clips (such as 1-2 minutes). However, when processing long audio clips, it is necessary to segment them into smaller segments and cluster them segment by segment, which easily leads to segmentation errors and accumulated errors. Based on this, the embodiments of this disclosure mainly make the following improvements:

[0166] 1) Long audio files are segmented using a sliding window while retaining contextual information. This contextual information helps ensure consistency of speaker labels between segments.

[0167] 2) Divide the audio into multiple segments, perform preliminary clustering on each segment, and then recursively optimize the clustering results to generate globally consistent speaker labels.

[0168] 3) Use time-series-based models (such as LSTM and Transformer) to capture long-range dependencies in audio. Specifically, this could involve combining timestamp information to optimize speaker separation in long audio clips.

[0169] Thirdly, regarding adaptation to dynamic scenarios, the existing Pyannote leans towards static scenario analysis and has limited support for dynamic scenarios (such as alternating teacher-student question-and-answer sessions). Based on this, the embodiments disclosed in this publication mainly make the following improvements:

[0170] 1) Introduce a scene detection module to automatically distinguish different stages such as lecturing, Q&A, and discussion by analyzing data from surrounding teaching scenes.

[0171] 2) Switch speaker recognition strategies in different scenarios (e.g., focus more on the main speaker or multi-speaker interaction).

[0172] It should be noted that, in addition to automatically determining the scene, the embodiments of this disclosure can also determine the teaching scene selected under the trigger operation of the scene selection button as the target teaching scene, and other methods can also be used to determine the teaching scene, without making specific limitations here.

[0173] In order to further improve the applicability of the teaching behavior analysis method provided in this embodiment, a target teaching interface can also be set up, and the results of teaching behavior analysis can be displayed through relevant trigger operations on the target teaching interface.

[0174] like Figure 2 The image shown is a teaching interface for a recorded lesson, as exemplified by an embodiment of this disclosure. This interface can display teaching behavior analysis results, such as teaching sequence distribution, teaching time allocation, and student evaluations. Furthermore, it can display courseware index content for more efficient course playback.

[0175] In practical applications, it can identify different speakers in the classroom and determine various teaching scenarios such as teacher lecturing, teacher-student Q&A, and student presentations based on the interactions and behaviors between speakers. It can also use semantic recognition to set the teaching behaviors to be identified in the management backend and display them accordingly. This helps teachers understand student participation in real time, provide timely feedback, and enhance student engagement and interaction.

[0176] Furthermore, the teaching behavior analysis method provided in this disclosure has the following advantages in actual teaching scenarios:

[0177] Firstly, it enhances students' expression and communication skills: During discussions and answering questions, the speaker recognition function helps students better organize their language and express their views. By analyzing the audio signals during the discussion, students can review their own speeches, reflect on the effectiveness of their communication, and thus improve their expression and communication abilities.

[0178] Secondly, it improves teaching quality and efficiency: Teachers can use speech recognition and speaker separation technologies to transcribe and analyze student discussions, thereby more accurately assessing students' learning progress and comprehension levels. The combined application of these two technologies can save teachers time, allowing them to focus more on optimizing teaching content and providing personalized guidance to students.

[0179] Third, technological innovation and application promotion drive the development of educational technology: providing new solutions for the education field and promoting the progress and development of educational technology. This technological innovation can inspire more research and development, and promote the digital transformation of the education industry.

[0180] In the description of this specification, references to terms such as "some possible implementations," "some implementations," "example," "specific example," or "some examples" indicate that a specific feature, structure, material, or characteristic described in connection with that implementation or example is included in at least one implementation or example of this disclosure, and the aforementioned terms do not necessarily refer to the same implementation or example. Furthermore, the described specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more implementations or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different implementations or examples described in this specification, as well as the features of different implementations or examples.

[0181] Regarding the method flowcharts of embodiments of this disclosure, certain operations are described as different steps performed in a certain order. Such flowcharts are illustrative and not restrictive. Some steps described herein may be grouped together and performed in a single operation, or some steps may be divided into multiple sub-steps, and some steps may be performed in an order different from that shown herein. The various steps shown in the flowcharts may be implemented in any way by any circuit structure and / or tangible mechanism (e.g., software running on a computer device, hardware (e.g., logic functions implemented by a processor or chip), and / or any combination thereof).

[0182] Those skilled in the art will understand that in the methods described in the above specific embodiments, the order in which the steps are written does not imply a strict execution order, and the specific execution order of each step should be determined by its function and possible internal logic.

[0183] Based on the same inventive concept, this disclosure also provides a teaching behavior analysis device corresponding to the teaching behavior analysis method. Since the principle of the device in this disclosure for solving the problem is similar to the teaching behavior analysis method described above in this disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0184] Reference Figure 3 The diagram shown is a schematic of a teaching behavior analysis device provided in an embodiment of this disclosure. The device includes: an acquisition module 201, a processing module 202, a disassembly module 203, and an analysis module 204; wherein,

[0185] Acquisition module 201 is used to acquire the speech signal in the target teaching scenario;

[0186] Processing module 202 is used to perform text conversion processing on the speech signal to determine the content of the speech-to-text conversion; and to perform voiceprint processing on the speech signal to determine the voiceprint feature information of at least one speaker in the target teaching scenario.

[0187] The decomposition module 203 is used to decompose the speech-to-text conversion content based on the voiceprint feature information of at least one speaker, and to determine the speech-to-text conversion content of each speaker in the corresponding speaking time period.

[0188] The analysis module 204 is used to analyze teaching behavior based on the speech-to-text conversion content of each speaker within the corresponding speaking time period, and to determine the teaching behavior analysis results under the target teaching scenario.

[0189] This embodiment combines the transcription function corresponding to text conversion with the speaker separation function corresponding to voiceprint processing, making the speech-text conversion content for each speaker within the corresponding speaking time period more accurate and easier to display. Furthermore, it can realize the analysis of teaching behavior for various roles such as teachers and students in various teaching scenarios, which is sufficient to meet the needs of various complex application scenarios. For example, it can enable teachers to have a comprehensive understanding of their teaching situation, and can provide the most suitable learning suggestions for different students.

[0190] In one possible implementation, when the target teaching scenario includes a teacher-led instruction scenario, the analysis module 204 is specifically used to determine the teaching behavior analysis results in the target teaching scenario according to the following steps:

[0191] Key words were extracted from the audio-text conversion of teachers during their corresponding speaking time periods to identify key knowledge points; and teachers' teaching behavior data were statistically analyzed during their corresponding speaking time periods.

[0192] The quality of a teacher's instruction is assessed based on key knowledge points and / or teaching behavior data.

[0193] The quality of instruction was determined as a result of the teaching behavior analysis.

[0194] In one possible implementation, when the target teaching scenario includes a teacher-student question-and-answer scenario, the analysis module 204 is specifically used to determine the teaching behavior analysis results in the target teaching scenario according to the following steps:

[0195] The quality of responses is analyzed based on the audio-text conversion of students during their respective speaking time periods to determine the data for response quality analysis; and the interactive behavior data of teachers and students during their respective speaking time periods is also statistically analyzed.

[0196] The level of interaction activity between teachers and students is assessed based on the analysis data of response quality and / or interaction behavior data.

[0197] The level of interactive activity was determined as the result of the teaching behavior analysis.

[0198] In one possible implementation, when the target teaching scenario includes a student group discussion scenario, the analysis module 204 is specifically used to determine the teaching behavior analysis results in the target teaching scenario according to the following steps:

[0199] Based on the audio-text conversion of each student in the group during their respective speaking time period, the discussion behavior data of each student during their respective speaking time period is statistically analyzed.

[0200] The adequacy of student discussions is assessed based on discussion behavior data.

[0201] The sufficiency of the discussion was determined as a result of the teaching behavior analysis.

[0202] In one possible implementation, when the target teaching scenario includes a student presentation scenario, the analysis module 204 is specifically used to determine the teaching behavior analysis results under the target teaching scenario according to the following steps:

[0203] Keyword extraction is performed based on the audio-text conversion of students during their corresponding speaking time periods to determine key content to be displayed; and student display behavior data is collected during their corresponding speaking time periods.

[0204] Based on key content displayed and / or display behavior data, determine learning suggestions for students;

[0205] The learning suggestions were identified as part of the teaching behavior analysis results.

[0206] In one possible implementation, the processing module 202 is specifically configured to perform text conversion processing on the speech signal according to the following steps:

[0207] Based on the speech feature extraction model, the speech signal is converted from time domain to frequency domain to determine the speech feature information;

[0208] The speech feature information is sequentially input into the acoustic model and the language model to determine the predicted text sequence content;

[0209] The speech-to-text conversion content is determined based on the preset search method and the predicted text sequence content.

[0210] In one possible implementation, the processing module 202 is specifically configured to perform voiceprint processing on the speech signal according to the following steps:

[0211] Divide the speech signal into at least one speech segment and at least one non-speech segment;

[0212] Non-speech segments are filtered out, and for the target speech segment in at least one speech segment, the voiceprint feature information of at least one speaker in the target speech segment is determined based on the speech embedding extraction model.

[0213] In one possible implementation, the disassembly module 203 is specifically used to disassemble the speech-to-text conversion content according to the following steps:

[0214] Speaker clustering is performed based on the voiceprint feature information determined for each speech segment to identify each speaker and the corresponding speaking time period.

[0215] For at least one target speaker, determine the target speech-to-text content that matches the target speaker's identifier and corresponding speaking time period from the speech-to-text conversion content; and...

[0216] The target speech-to-text content is defined as the speech-to-text content of the target speaker during the corresponding speaking time period.

[0217] In one possible implementation, when multiple speakers speak during the same speaking time period, the disassembly module 203 is specifically used to determine the identifier of each speaker and the corresponding speaking time period according to the following steps:

[0218] Speaker clustering is performed based on the voiceprint feature information determined for each speech segment, and the clustering result for each speaker is determined.

[0219] The clustering results of each speaker are input into the speech overlap detection model to determine the identifiers of multiple speakers corresponding to the same speaking time period.

[0220] In one possible implementation, a verification module 205 is also included, for:

[0221] Extracting contextual semantic information from speech-to-text conversion content;

[0222] The identifier of the identified speaker and the corresponding speaking time period are verified based on contextual semantic information.

[0223] In one possible implementation, the acquisition module 201 is used to determine that the target teaching scenario is being performed in one of the following ways:

[0224] In response to the trigger operation of the scene selection button, the teaching scene selected under the trigger operation is determined as the target teaching scene;

[0225] Acquire data on surrounding teaching scenarios, perform data analysis on the surrounding teaching scenarios, and determine the corresponding target teaching scenarios.

[0226] In one possible implementation, the processing module 202 is specifically configured to perform voiceprint processing on the speech signal through the following steps:

[0227] Obtain pre-set role information for the target teaching scenario;

[0228] Character information is used as supplementary information, and voiceprint processing is performed based on the supplementary information and voice signals.

[0229] In one possible implementation, a display module 206 is also included, for:

[0230] In response to a request to display teaching behavior triggered on the target teaching interface, the analysis results of teaching behavior in the target teaching scenario are displayed on the target teaching interface.

[0231] It should be noted that the apparatus in this embodiment can implement the various processes of the aforementioned method and achieve the same effects and functions, which will not be elaborated here.

[0232] This disclosure also provides an electronic device, such as... Figure 4 The diagram shown is a schematic representation of an electronic device structure provided in this embodiment of the present disclosure, including: a processor 301, a memory 302, and a bus 303. The memory 302 stores machine-readable instructions executable by the processor 301 (e.g., ...). Figure 3 The device contains the execution instructions corresponding to the acquisition module 201, processing module 202, disassembly module 203, and analysis module 204. When the electronic device is running, the processor 301 and memory 302 communicate via bus 303. When machine-readable instructions are executed by the processor 301, the following processing is performed:

[0233] Acquire audio signals in the target teaching scenario;

[0234] The speech signal is processed into text to determine the content of the speech-to-text conversion; and the speech signal is processed into voiceprints to determine the voiceprint feature information of at least one speaker in the target teaching scenario.

[0235] The speech-to-text conversion content is decomposed based on the voiceprint feature information of at least one speaker to determine the speech-to-text conversion content of each speaker in the corresponding speaking time period.

[0236] Based on the speech-to-text conversion content of each speaker during the corresponding speaking time period, the teaching behavior analysis results are determined in the target teaching scenario.

[0237] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the teaching behavior analysis method described in the above-described method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0238] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the teaching behavior analysis method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0239] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0240] The various embodiments in this disclosure are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments. In particular, the description of the apparatus, device, and computer-readable storage medium embodiments is simplified because they are basically similar to the method embodiments, and the relevant parts can be referred to the description of the method embodiments.

[0241] The apparatus, device, and computer-readable storage medium provided in this disclosure correspond one-to-one with the method. Therefore, the apparatus, device, and computer-readable storage medium also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the apparatus, device, and computer-readable storage medium will not be repeated here.

[0242] Those skilled in the art will understand that embodiments of this disclosure can be implemented as methods and apparatus (devices or systems), or as computer-readable storage media. Therefore, this disclosure can be implemented entirely in hardware, entirely in software, or in a combination of software and hardware. Furthermore, this disclosure can be implemented as a computer-readable storage medium on one or more computer-readable storage media containing computer-usable program code (including, but not limited to, disk storage, read-only optical disc storage (CD-ROM), optical storage, etc.).

[0243] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices or systems), and computer-readable storage media according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to create a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0244] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article including instruction means, wherein the instruction means implement the functions specified in one or more flowcharts and / or one or more blocks in a block diagram.

[0245] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.

[0246] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0247] Memory can include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0248] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of a program, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory, read-only memory, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. Furthermore, although the operations of the methods of this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally, certain steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple sub-steps.

[0249] While the spirit and principles of this disclosure have been described above with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method for analyzing teaching behavior, characterized in that, include: Acquire audio signals in the target teaching scenario; The speech signal is subjected to text conversion processing to determine the speech-to-text conversion content; as well as, The speech signal is analyzed to identify the current teaching scenario type; wherein the teaching scenario type includes at least teacher-led lecture scenarios and teacher-student question-and-answer scenarios. Based on the identified teaching scenario type, different voiceprint processing objectives are adapted and executed, including: When the scene is identified as a teacher lecturing, voiceprint processing is performed, which mainly separates and extracts the teacher's voice signal and secondarily suppresses background noise. When the scenario is identified as a teacher-student question-and-answer session, voiceprint processing is performed to distinguish and separate the teacher's voice signal from that of at least one student. Based on the voiceprint processing target of the adaptive execution, the speech signal is processed to determine the voiceprint feature information of at least one speaker and its corresponding speaking time period. Based on the voiceprint feature information of the at least one speaker and its corresponding speaking time period, the speech-to-text conversion content is decomposed to determine the speech-to-text conversion content of each speaker in the corresponding speaking time period. Based on the identified teaching scenario type and the speech-to-text conversion content of each speaker during the corresponding speaking time period, teaching behavior analysis is performed to determine the teaching behavior analysis results.

2. The method according to claim 1, characterized in that, When the target teaching scenario includes a teacher-led instruction scenario, determining the teaching behavior analysis results under the target teaching scenario includes: Key words were extracted from the audio-text conversion of teachers during their corresponding speaking time periods to identify key knowledge points; and teachers' teaching behavior data were statistically analyzed during their corresponding speaking time periods. The quality of a teacher's instruction is evaluated based on the key knowledge points and / or the teaching behavior data. The quality of the lecture is determined as the result of the teaching behavior analysis.

3. The method according to claim 1, characterized in that, When the target teaching scenario includes a teacher-student question-and-answer scenario, determining the teaching behavior analysis results under the target teaching scenario includes: The quality of responses is analyzed based on the audio-text conversion of students during their respective speaking time periods to determine the data for response quality analysis; and the interactive behavior data of teachers and students during their respective speaking time periods is also statistically analyzed. The level of interaction activity between teachers and students is assessed based on the answer quality analysis data and / or the interaction behavior data. The level of interactive activity is determined as the result of the teaching behavior analysis.

4. The method according to claim 1, characterized in that, When the target teaching scenario includes a student group discussion scenario, determining the teaching behavior analysis results under the target teaching scenario includes: Based on the audio-text conversion of each student in the group during their respective speaking time period, the discussion behavior data of each student during their respective speaking time period is statistically analyzed. The adequacy of student discussions was assessed based on the aforementioned discussion behavior data. The sufficiency of the discussion is determined as the result of the teaching behavior analysis.

5. The method according to claim 1, characterized in that, When the target teaching scenario includes a student presentation scenario, determining the teaching behavior analysis results under the target teaching scenario includes: Keyword extraction is performed based on the audio-text conversion of students during their corresponding speaking time periods to determine key content to be displayed; and student display behavior data is collected during their corresponding speaking time periods. Based on the key display content and / or the display behavior data, learning suggestions for students are determined. The learning suggestions are identified as the results of the teaching behavior analysis.

6. The method according to any one of claims 1 to 5, characterized in that, The text conversion processing of the speech signal includes: The speech signal is converted from time domain to frequency domain based on the speech feature extraction model to determine the speech feature information. The speech feature information is sequentially input into the acoustic model and the language model to determine the predicted text sequence content; The speech-to-text conversion content is determined based on a preset search method and the predicted text sequence content.

7. The method according to any one of claims 1 to 5, characterized in that, The voiceprint processing of the speech signal includes: The speech signal is divided into at least one speech segment and at least one non-speech segment; The non-speech segments are filtered out, and for the target speech segment in at least one speech segment, the voiceprint feature information of at least one speaker in the target speech segment is determined based on the speech embedding extraction model.

8. The method according to claim 7, characterized in that, The method further includes: Speaker clustering is performed based on the voiceprint feature information determined for each of the aforementioned speech segments to determine the identifier of each speaker and the corresponding speaking time period; The process of decomposing the speech-to-text conversion content based on the voiceprint feature information of at least one speaker includes: For at least one target speaker, determine from the speech-to-text conversion content the target speaker's identifier and corresponding speaking time period match; and... The target speech-to-text conversion content is determined as the speech-to-text conversion content of the target speaker within the corresponding speaking time period.

9. The method according to claim 8, characterized in that, When multiple speakers speak within the same speaking time period, determining the identifier of each speaker and the corresponding speaking time period includes: Speaker clustering is performed based on the voiceprint feature information determined for each of the aforementioned speech segments, and the clustering result for each speaker is determined. The clustering results of each speaker are input into the speech overlap detection model to determine the identifiers of multiple speakers corresponding to the same speaking time period.

10. The method according to claim 8 or 9, characterized in that, The method further includes: Extract contextual semantic information from the speech-to-text conversion content; The speaker's identifier and corresponding speaking time period are verified based on the contextual semantic information.

11. The method according to any one of claims 1 to 5, characterized in that, The target teaching scenario is determined using one of the following methods: In response to a trigger operation on the scene selection button, the teaching scene selected under the trigger operation is determined as the target teaching scene; Acquire surrounding teaching scene data, perform data analysis on the surrounding teaching scene data, and determine the corresponding target teaching scene.

12. The method according to any one of claims 1 to 5, characterized in that, The voiceprint processing of the speech signal includes: Obtain pre-set role information for the target teaching scenario; The character information is used as additional information, and voiceprint processing is performed based on the additional information and the voice signal.

13. The method according to any one of claims 1 to 5, characterized in that, The method further includes: In response to a request to display teaching behavior triggered on the target teaching interface, the analysis results of teaching behavior in the target teaching scenario are displayed on the target teaching interface.

14. A teaching behavior analysis device, characterized in that, include: The acquisition module is used to acquire the voice signal in the target teaching scenario; The processing module is used to perform text conversion processing on the speech signal and determine the content of the speech-to-text conversion; as well as, The speech signal is analyzed to identify the current teaching scenario type; wherein the teaching scenario type includes at least teacher-led lecture scenarios and teacher-student question-and-answer scenarios. Based on the identified teaching scenario type, different voiceprint processing objectives are adapted and executed, including: When the scene is identified as a teacher lecturing, voiceprint processing is performed, which mainly separates and extracts the teacher's voice signal and secondarily suppresses background noise. When the scenario is identified as a teacher-student question-and-answer session, voiceprint processing is performed to distinguish and separate the teacher's voice signal from that of at least one student. Based on the voiceprint processing target of the adaptive execution, the speech signal is processed to determine the voiceprint feature information of at least one speaker and its corresponding speaking time period. The decomposition module is used to decompose the speech-to-text conversion content based on the voiceprint feature information of the at least one speaker and its corresponding speaking time period, and to determine the speech-to-text conversion content of each speaker in the corresponding speaking time period. The analysis module is used to analyze teaching behaviors corresponding to the identified teaching scenario type and the speech-to-text conversion content of each speaker during the corresponding speaking time period, and to determine the teaching behavior analysis results.

15. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the teaching behavior analysis method as described in any one of claims 1 to 13 is performed.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the teaching behavior analysis method as described in any one of claims 1 to 13.

Images