Voice-based group learning atmosphere analysis method and system

Abstract

The invention discloses a voice-based group learning atmosphere analysis method and system. The method comprises the following steps: constructing a target voice enhancement model and a target voiceprint recognition model; inputting the group voice data into a target voice enhancement model for data enhancement processing to obtain target voice enhancement information; inputting the group voice data into a target voiceprint recognition model for voiceprint recognition processing to obtain target voice character information corresponding to the group voice data; calculating the target voice enhancement information and the target voice character information to obtain a group learning participation degree; performing feature extraction processing on the target speech enhancement information to obtain target speech feature information; inputting the target voice feature information into a target emotion prediction model for emotion prediction to obtain target emotion information; and analyzing and processing the group learning participation degree and the target emotion information to obtain group learning atmosphere data. The method can obtain comprehensively evaluated group learning atmosphere data, and can be widely applied to the technical field of data processing.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and system for analyzing group learning atmosphere based on speech. Background Art

[0002] At present, group learning is an important teaching method, in which students complete learning tasks together in groups. In the traditional group learning process, there may be uneven participation among group members, which may even lead to conflicts and contradictions; for teachers, classroom evaluation is difficult. Therefore, it is difficult for teachers to understand the learning situation of each group in the class during the group learning teaching process.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The embodiment of the present application aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the main purpose of the embodiment of the present application is to propose a group learning atmosphere analysis method and system based on speech, which can obtain group learning atmosphere data for comprehensive evaluation.

[0005] To achieve the above purpose, one aspect of an embodiment of the present application proposes a method for analyzing a group learning atmosphere based on speech, the method comprising the following steps:

[0006] Obtain a speech information processing solution, and build a target speech enhancement model and a target voiceprint recognition model based on the speech information processing solution;

[0007] Obtain group speech data in teaching classrooms;

[0008] Inputting the group speech data into the target speech enhancement model for data enhancement processing to obtain target speech enhancement information;

[0009] Inputting the group voice data into the target voiceprint recognition model for voiceprint recognition processing to obtain target voice character information corresponding to the group voice data;

[0010] Calculating the target voice enhancement information and the target voice character information to obtain group learning participation;

[0011] Performing feature extraction processing on the target speech enhancement information to obtain target speech feature information;

[0012] Inputting the target speech feature information into a target emotion prediction model to perform emotion prediction to obtain target emotion information;

[0013] The group learning participation and the target emotion information are analyzed and processed to obtain group learning atmosphere data.

[0014] In some embodiments, the voice information processing scheme includes a local processing scheme, and the acquiring of the voice information processing scheme and the building of the target voice enhancement model and the target voiceprint recognition model based on the voice information processing scheme include:

[0015] Acquire the speech information processing scheme and the speech data to be trained;

[0016] Selecting a target speech processing scheme from the speech information processing schemes according to the teaching environment information;

[0017] If the target speech processing scheme is the local processing scheme in the speech information processing scheme, the target speech enhancement model and the target voiceprint recognition model are constructed according to the speech data to be trained.

[0018] In some embodiments, if the target speech processing scheme is the local processing scheme in the speech information processing scheme, constructing the target speech enhancement model and the target voiceprint recognition model according to the speech data to be trained includes:

[0019] If the target speech processing scheme is the local processing scheme in the speech information processing scheme, information is extracted from the speech data to be trained to obtain pure speech data;

[0020] Adding a reverberation effect to the clean voice data to obtain reverberation voice data;

[0021] Integrating the reverberant speech data and a plurality of target noise segments to obtain target training data;

[0022] Inputting the target training data into an initial speech enhancement model for training to obtain the target speech enhancement model;

[0023] If the target speech processing scheme is the local processing scheme in the speech information processing scheme, performing feature extraction processing on the speech data to be trained to obtain target training feature information;

[0024] The target training feature information is input into a linear time series model for training to obtain the target voiceprint recognition model; the target voiceprint recognition model is used to obtain the voiceprint information of the target voice character, and the voiceprint information is used to construct a voiceprint model library.

[0025] In some embodiments, inputting the target training feature information into a linear time series model for training to obtain the target voiceprint recognition model includes:

[0026] Inputting the target training feature information into the linear time series model for training to obtain an initial embedding vector;

[0027] Regularizing the initial embedding vector to obtain a target embedding vector;

[0028] Calculating the centroid of the target embedding vector to obtain the target vector centroid;

[0029] Constructing a similarity matrix based on the target embedding vector and the target vector centroid;

[0030] The linear time series model is trained based on the similarity matrix and the target voiceprint loss function to obtain the target voiceprint recognition model.

[0031] In some embodiments, the calculating the target voice enhancement information and the target voice character information to obtain the group learning participation degree includes:

[0032] Calculating the number of target active characters and the number of target voice characters in the target voice character information within the target time period;

[0033] The ratio of the number of target active characters to the number of target voice characters is used as the group learning participation.

[0034] In some embodiments, the performing feature extraction processing on the target speech enhancement information to obtain target speech feature information includes:

[0035] Performing audio feature extraction processing on the target speech enhancement information to obtain an audio feature vector;

[0036] Performing text feature extraction processing on the target speech enhancement information to obtain a text feature vector corresponding to the audio feature vector;

[0037] The audio feature vector and the text feature vector are concatenated to obtain the target speech feature information.

[0038] In some embodiments, the voice information processing solution includes a cloud processing solution, and the method further includes:

[0039] Selecting a target speech processing scheme from the speech information processing schemes according to the teaching environment information;

[0040] If the target voice processing solution is the cloud processing solution in the voice information processing solution, constructing a target cloud processing model based on the cloud processing solution;

[0041] Acquiring the group voice data in the teaching classroom;

[0042] Inputting the group voice data into the target cloud processing model for data processing to obtain target cloud voice information and the target voice character information corresponding to the group voice data;

[0043] Calculating the target cloud voice information and the target voice character information to obtain the group learning participation degree;

[0044] Perform feature extraction processing on the target cloud voice information to obtain the target voice feature information.

[0045] To achieve the above purpose, another aspect of the embodiment of the present application proposes a group learning atmosphere analysis system based on speech, and the system includes the following modules:

[0046] A model building module, used to obtain a speech information processing solution, and build a target speech enhancement model and a target voiceprint recognition model based on the speech information processing solution;

[0047] A group voice data acquisition module is used to acquire group voice data in a teaching classroom;

[0048] A data enhancement processing module, used for inputting the group speech data into the target speech enhancement model for data enhancement processing to obtain target speech enhancement information;

[0049] A voiceprint recognition processing module, used for inputting the group voice data into the target voiceprint recognition model for voiceprint recognition processing, and obtaining the target voice character information corresponding to the group voice data;

[0050] A group learning participation calculation module, used to calculate the target voice enhancement information and the target voice character information to obtain the group learning participation;

[0051] A feature extraction processing module is used to perform feature extraction processing on the target speech enhancement information to obtain target speech feature information;

[0052] An emotion information prediction module, used for inputting the target speech feature information into a target emotion prediction model to perform emotion prediction and obtain target emotion information;

[0053] The group learning atmosphere analysis module is used to analyze and process the group learning participation and the target emotion information to obtain group learning atmosphere data.

[0054] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0055] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0056] The embodiments of the present application include at least the following beneficial effects: The present application provides a method and system for analyzing group learning atmosphere based on speech, which obtains a speech information processing scheme, and constructs a target speech enhancement model and a target voiceprint recognition model based on the speech information processing scheme; obtains group speech data in a teaching classroom; inputs the group speech data into a target speech enhancement model for data enhancement processing to obtain target speech enhancement information; inputs the group speech data into a target voiceprint recognition model for voiceprint recognition processing to obtain target speech character information corresponding to the group speech data; calculates the target speech enhancement information and the target speech character information to obtain group learning participation; performs feature extraction processing on the target speech enhancement information to obtain target speech feature information; inputs the target speech feature information into a target emotion prediction model for emotion prediction to obtain target emotion information; analyzes and processes the group learning participation and target emotion information to obtain group learning atmosphere data. The embodiment of the present application performs enhancement processing on the group voice data through the target voice enhancement model, thereby improving the clarity and purity of the voice signal, thereby improving the accuracy of subsequent emotion prediction and group learning participation analysis; using the target voiceprint recognition model to perform voiceprint recognition on the group voice data can accurately identify the voice of each group member, providing the possibility for subsequent personalized feedback and intervention; by analyzing the group learning participation combined with the emotion prediction results, it is possible to obtain comprehensive evaluation of the group learning atmosphere data, so that group members and teachers can quickly capture the learning situation in the classroom. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flowchart of the steps of the speech-based group learning atmosphere analysis method provided in an embodiment of the present application;

[0058] Figure 2 It is a schematic diagram of a framework of a voiceprint recognition model training and reasoning process provided in an embodiment of the present application;

[0059] Figure 3 It is a schematic diagram of a framework of a speech enhancement process provided in an embodiment of the present application;

[0060] Figure 4 It is a schematic diagram of a framework of a group learning participation calculation process provided by an embodiment of the present application;

[0061] Figure 5 It is a structural diagram of a sentiment analysis and recognition module provided in an embodiment of the present application;

[0062] Figure 6 It is a schematic diagram of the interface content of a visualization module provided in an embodiment of the present application;

[0063] Figure 7 It is a flow chart of a method for analyzing group learning atmosphere based on speech provided in an embodiment of the present application;

[0064] Figure 8 It is a schematic diagram of a neural network application implementation of a speech-based group learning atmosphere analysis method provided in an embodiment of the present application;

[0065] Fig. 9 It is a structural diagram of a speech-based group learning atmosphere analysis system provided in an embodiment of the present application;

[0066] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0068] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0069] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0071] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0072] At present, group learning is an important teaching method, in which students complete learning tasks together in groups. In the traditional group learning process, there may be uneven participation among group members, which may even lead to conflicts and contradictions; for teachers, classroom evaluation is difficult. Therefore, it is difficult for teachers to understand the learning situation of each group in the class during the group learning teaching process.

[0073] In view of this, a speech-based group learning atmosphere analysis method and system are provided in an embodiment of the present application. The scheme obtains a speech information processing scheme, and constructs a target speech enhancement model and a target voiceprint recognition model based on the speech information processing scheme; obtains group speech data in a teaching classroom; inputs the group speech data into the target speech enhancement model for data enhancement processing to obtain target speech enhancement information; inputs the group speech data into the target voiceprint recognition model for voiceprint recognition processing to obtain target speech character information corresponding to the group speech data; calculates the target speech enhancement information and the target speech character information to obtain group learning participation; performs feature extraction processing on the target speech enhancement information to obtain target speech feature information; inputs the target speech feature information into the target emotion prediction model for emotion prediction to obtain target emotion information; analyzes and processes the group learning participation and the target emotion information to obtain group learning atmosphere data. The embodiment of the present application performs enhancement processing on the group voice data through the target voice enhancement model, thereby improving the clarity and purity of the voice signal, thereby improving the accuracy of subsequent emotion prediction and group learning participation analysis; using the target voiceprint recognition model to perform voiceprint recognition on the group voice data can accurately identify the voice of each group member, providing the possibility for subsequent personalized feedback and intervention; by analyzing the group learning participation combined with the emotion prediction results, it is possible to obtain comprehensive evaluation of the group learning atmosphere data, so that group members and teachers can quickly capture the learning situation in the classroom.

[0074] The speech-based group learning atmosphere analysis method provided in the embodiment of the present application relates to the field of data processing technology. The speech-based group learning atmosphere analysis method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network, content distribution network) and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in the blockchain network; the software can be an application that implements a speech-based group learning atmosphere analysis method, etc., but is not limited to the above forms.

[0075] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs (Personal Computers, personal computers), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0076] See also Figure 1 , Figure 1 is an optional step flow chart of the speech-based group learning atmosphere analysis method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S108.

[0077] Step S101, obtaining a speech information processing solution, and building a target speech enhancement model and a target voiceprint recognition model based on the speech information processing solution;

[0078] Optionally, the speech information processing scheme includes a local processing scheme and a cloud processing scheme. The local processing scheme is a scheme including a training data preprocessing process and a model building process. After the training data preprocessing and model building, the target speech enhancement model and the target voiceprint recognition model can be obtained. The target speech enhancement model and the target voiceprint recognition model will process the real-time teaching classroom speech data to obtain a group learning atmosphere scheme; the cloud processing scheme is to directly deploy a cloud processing model (cloud online speaker diarization model), and then process the real-time teaching classroom speech data according to the cloud processing model to obtain a group learning atmosphere scheme, wherein the cloud processing model can accurately detect the target speaker's speaking state in a complex audio environment through multi-layer feature extraction and sequence modeling. Specifically, if there is no condition or requirement for deploying a local voiceprint recognition model in the teaching implementation environment, there is no need to perform the preprocessing process of the training data, and directly enter the next implementation step S102; if there is a condition or requirement for deploying a local voiceprint recognition model in the teaching implementation environment, the training data needs to be preprocessed.

[0079] It should be noted that the cloud processing model is a mature model in the relevant technology, such as the TS-VAD model. The structure of the TS-VAD model is usually composed of an input feature extraction layer, a target speaker feature encoding layer, a speaker correlation analysis layer, and a VAD decision layer. It is easy to understand that for the selection of the cloud processing model, the TS-VAD model can be referred to for selecting the relevant model, and the embodiments of the present application are not limited to this. In addition, the specific implementation principle of the TS-VAD model can refer to the specific description of the relevant technical solution, and the embodiments of the present application will not be repeated here.

[0080] In some embodiments, step S101 may include: obtaining a speech information processing scheme and speech data to be trained; selecting a target speech processing scheme from the speech information processing scheme according to teaching environment information; if the target speech processing scheme is a local processing scheme in the speech information processing scheme, constructing a target speech enhancement model and a target voiceprint recognition model according to the speech data to be trained.

[0081] The speech data to be trained is a set of original speech data used to train the target speech enhancement model and the target voiceprint recognition model. The target speech enhancement model refers to a machine learning model used to improve the quality of speech signals. By analyzing and processing noisy speech signals, the noise and interference therein are removed, thereby restoring a clear speech signal. In a specific implementation, the target speech enhancement model is constructed based on the speech data to be trained, and is used to enhance the student's speech signal in real time in a group learning classroom, and improve the purity and clarity of the speech. The target voiceprint recognition model includes a voiceprint encoder, which is used to obtain the voiceprint information of the target voice character, and the voiceprint information is used to build a voiceprint model library. Specifically, the target voiceprint recognition model refers to a machine learning model used to identify the identities of different speakers. By analyzing and comparing the feature information in the speech signal, the unique voiceprint features representing the speaker's identity are extracted, and the identity is identified accordingly. In a specific implementation, the target voiceprint recognition model is also constructed based on the speech data to be trained, and is used to identify the speech of each group member in real time in a group learning classroom, and provide accurate feedback information for teachers to intervene and guide in a timely manner.

[0082] Among them, teaching environment information refers to various parameters and conditions related to a specific teaching environment, including but not limited to the size, layout, noise level, echo conditions, number of students, group division, etc. of the classroom.

[0083] The target voice processing solution is a voice information processing solution selected based on whether or not there is a condition or need to deploy a local voiceprint recognition model in the teaching implementation environment. The target voice processing solution can be a local processing solution or a cloud processing solution.

[0084] In some specific embodiments, if the target speech processing scheme is a local processing scheme in the speech information processing scheme, a target speech enhancement model and a target voiceprint recognition model are constructed according to the speech data to be trained, which may include: if the target speech processing scheme is a local processing scheme in the speech information processing scheme, information is extracted from the speech data to be trained to obtain pure speech data; reverberation effect is added to the pure speech data to obtain reverberation speech data; the reverberation speech data and several target noise fragments are integrated to obtain target training data; the target training data is input into the initial speech enhancement model for training to obtain the target speech enhancement model; if the target speech processing scheme is a local processing scheme in the speech information processing scheme, feature extraction is performed on the speech data to be trained to obtain target training feature information; the target training feature information is input into the linear time series model for training to obtain the target voiceprint recognition model; the target voiceprint recognition model is used to obtain the voiceprint information of the target voice character, and the voiceprint information is used to construct a voiceprint model library.

[0085] In some specific embodiments, inputting the target training feature information into a linear time series model for training to obtain a target voiceprint recognition model can include: inputting the target training feature information into a linear time series model for training to obtain an initial embedding vector; regularizing the initial embedding vector to obtain a target embedding vector; calculating the centroid of the target embedding vector to obtain the centroid of the target vector; constructing a similarity matrix based on the target embedding vector and the target vector centroid; training the linear time series model based on the similarity matrix and the target voiceprint loss function to obtain the target voiceprint recognition model.

[0086] Among them, pure speech data refers to speech data without noise, reverberation and other effects. Reverberation refers to the phenomenon that sound waves are superimposed on direct sound after being reflected by reflective surfaces (such as walls, ceilings, floors, etc.). The reverberation effect causes the sound to continue for a period of time during the propagation process, making the original sound more blurred and diffuse. It is easy to understand that reverberation speech data refers to speech signals processed on the basis of pure speech data by simulating the acoustic characteristics (and reverberation) in the teaching environment.

[0087] In the specific implementation, for the training of the target speech enhancement model, the use of speech clips with reverberation can improve the robustness and effect of the model in practical applications; and adding noise enables the model to learn to recognize and separate the speech components in the noise signal during the training process, thereby providing clearer and higher-quality speech output in practical applications.

[0088] The target noise segment refers to different types of noise samples used to mix with speech signals to generate noisy speech, such as white noise, traffic noise, etc. The target training data is a speech dataset used to train speech enhancement models or voiceprint recognition models, which includes pure speech data, reverberant speech data, and noisy speech data.

[0089] The initial speech enhancement model refers to the initial version of the speech enhancement model used at the beginning of the training phase. The initial speech enhancement model will be optimized through the training process to obtain the target speech enhancement model. The target speech enhancement model is the final speech enhancement model that can effectively remove noise and reverberation and improve speech quality after training and optimization.

[0090] Optionally, the target training feature information is a feature vector extracted from the speech data to be trained, and the target training feature information is used to train the voiceprint recognition model. Among them, the linear time series model is a machine learning model for voiceprint recognition, which can process time series data and learn the characteristics of the speaker therefrom. By inputting the target training feature information into the linear time series model for training, the target voiceprint recognition model can be obtained. For the target voice character, the target voice character can refer to a student or teacher in a teaching classroom. The voiceprint model library refers to a database containing voiceprint information corresponding to multiple speakers.

[0091] For the initial embedding vector, it is the initial vector extracted from the target training feature information during the training process of the voiceprint recognition model, which is used to represent the speaker's voiceprint features. By processing the initial embedding vector with L2 regularization, a target embedding vector with stronger representation and generalization capabilities can be obtained, which can more accurately reflect the speaker's identity characteristics. For the target vector centroid, it is the center point calculated in a group of target embedding vectors. The target vector centroid can be used to represent the overall characteristics or central trend of the group of vectors. In the voiceprint recognition task, the target vector centroid is used to compare with the target embedding vector to calculate the similarity between them and obtain a similarity matrix. The similarity matrix defines the cosine similarity between each embedding vector and all centroids.

[0092] Optionally, the target voiceprint loss function is defined based on the target embedding vector and the target vector centroid, and is used to train and optimize the linear time series model to obtain the target voiceprint recognition model.

[0093] In the specific implementation, the process of building the target voiceprint recognition model is as follows: first, a certain amount of speaker corpus data (voice data to be trained) is obtained, and the obtained corpus data is feature extracted to obtain corpus features (target training feature information); the obtained corpus features are processed by a linear time series model with selective state space (also known as a Mamba model) to obtain an embedding vector, and then the embedding vector is L2 regularized; the centroid of the embedding vectors of all speakers is calculated, and a similarity matrix is ​​constructed based on the embedding vectors and the vector centroids, wherein the similarity matrix defines the cosine similarity between each embedding vector and all centroids; then a voiceprint loss function is defined based on the embedding vectors and the vector centroids to train the linear time series model, and the training purpose is to make each embedding vector closer to the centroid of its real speaker and away from the centroids of other speakers; finally, the linear time series model is trained based on the similarity matrix and the loss function to obtain a voiceprint recognition model, which includes a student voiceprint encoder, which is used to obtain the student's voiceprint information, and multiple voiceprint information is used to construct a voiceprint information library.

[0094] In the specific implementation, the process of constructing the target speech enhancement model is as follows: first, information is extracted from the training speech data to obtain the student's pure speech data, reverberation effect is added to the pure speech data to obtain speech with reverberation, and training data is generated using speech with reverberation and different noise fragments; then, the convolutional layer is used to extract the features of the training data, and a dual-path recurrent neural network module is used to model the time and frequency dependencies of the training data, that is, the speech signal of the training data is modeled in two dimensions of time and frequency to effectively capture the time-frequency characteristics of the speech signal; then, a speech enhancement loss function is defined, and a loss function combining a negative signal-to-noise ratio (SNR) and a mean square error (MSE) is used to optimize the initial speech enhancement model; the training process updates the model parameters of the initial speech enhancement model through back propagation, and finally the target speech enhancement model is obtained, and the target speech enhancement model uses batch normalization and instantaneous layer normalization to process data.

[0095] In the embodiment of the present application, the implementation process of step S101 includes the following two steps (step A and step B):

[0096] Step A: Select a voice information processing solution based on the teaching environment information.

[0097] Specifically, if there is no condition or demand for deploying a local voiceprint recognition model in the teaching implementation environment (i.e., using a cloud processing solution), no preprocessing is required, and the next implementation step B is directly entered. If there is a condition or demand for deploying a local voiceprint recognition model in the teaching implementation environment (i.e., using a local processing solution), the following preprocessing process is performed, specifically: first, different corpus data of multiple speakers are obtained, and then the corpus data is feature extracted using Mel frequency cepstral coefficients, and the features extracted are input into the Mamba model for training to obtain a series of voiceprint information libraries.

[0098] For example, if there is a condition or requirement to deploy a local voiceprint recognition model in the teaching implementation environment, a data preprocessing process is required. Please refer to Figure 2 , Figure 2 : is a schematic diagram of a framework of a voiceprint recognition model training and reasoning process provided by an embodiment of the present application, wherein the data preprocessing process (i.e., the training phase of the voiceprint recognition model) is: Figure 2 As shown, first obtain the training speech information of K speakers, each speaker has S training speech information; then extract the features of the training speech information through the Mel frequency cepstral coefficient to obtain the feature vector, each feature vector x ij (1≤i≤K,1≤j≤S) represents the features extracted from the corpus data j of speaker i. Let f(x ij; w) represents the output of the neural network (Mamba model), where w represents all the parameters of the neural network, and the L2 normalized output of the neural network is the speaker voiceprint embedding vector, The embedding vector of the jth speech information of the i-th speaker is [e 1j ,…,e Mj ]. Similarity matrix S ij,k Defined as each embedding vector e ij With all the center points o k Scaled cosine similarity between: Where w and b are learnable parameters. The training goal is to make the corpus embedding of each speaker similar to the center point of all embeddings of the speaker, and away from the embedding center of other speakers. Each embedding vector e ih The loss can be defined as This loss function means pulling each embedding vector closer to its center point and away from all other center points. After training, the target voiceprint recognition model is obtained. The target voiceprint recognition model contains a student voiceprint encoder, which is used to obtain the student's voiceprint information. Multiple voiceprint information (such as Figure 2 The "voiceprint information A" and "voiceprint information B" in the voiceprint information are used to build a voiceprint information library.

[0099] After completing the training of the voiceprint recognition model and obtaining the target voiceprint recognition model, the inference phase of the voiceprint recognition model is as follows: Figure 2 As shown, first, feature extraction is performed on the test speech to obtain test speech feature information; then the extracted test speech feature information is input into the trained target voiceprint recognition model; in the target voiceprint recognition model, the test speech feature information is compared with the voiceprint features of known speakers stored in the model library, and scored / judged, that is, the score is given according to the similarity between the input features and the known voiceprint features, and the test speech is determined by comparison and calculation to determine which (or which) known speakers' voiceprint features are most matched; finally, according to the results of the scoring / judgment, the system determines the identity of the speaker to which the test speech belongs, and outputs the result.

[0100] Step B: Deploy a cloud-based online speaker diarization model or train a local speech enhancement model or a local voiceprint recognition model according to the voice information processing solution selected in step A. Specifically: if a non-local voiceprint recognition solution (i.e., a cloud-based processing solution) is selected, deploy a cloud-based online speaker diarization model; if a local voiceprint recognition solution (i.e., a local processing solution) is selected, perform local model training.

[0101] Exemplarily, if you choose to use a locally deployed voiceprint recognition solution (that is, a local processing solution), the process of local model training is as follows: first, collect clean speech data, add reverberation effects to the clean speech data to obtain speech with reverberation, and use speech with reverberation and different noise fragments to generate training data; in the training stage, randomly divide the clean audio into 5-second segments, and convolve the clean audio with room impulse responses (Room ImpulseResponse, RIRs for short) randomly selected from openSLR26 and openSLR28. Then, the noisy speech is generated by mixing the reverberated speech with the noise, and the signal-to-noise ratio (SNR) range is set between -5dB and 5dB; then the convolution layer is used to extract the features of the noisy speech, and the dual-path recurrent neural network module is used to model the time and frequency dependencies; among them, the training target is the complex ratio mask (Complex Ratio Mask, referred to as CRM), and the decoder outputs the real and imaginary parts of the complex ratio mask as two streams respectively; in the training stage, the learning target is optimized by signal approximation, that is, the spectrum of the noisy speech is multiplied by the estimated mask to obtain the enhanced spectrum, and the inverse STFT (Short-Time Fourier Transform) is used to reconstruct it into a time domain signal; then, the loss function is defined, and the model is optimized using a loss function that combines the negative signal-to-noise ratio (SNR) and the mean square error (MSE). The first loss function is the negative signal-to-noise ratio (SNR), which is defined as: The first loss function can constrain the amplitude of the output and avoid level shift between input and output; the second loss function combines the negative SNR and the mean square error (MSE) of the spectrum and is defined as: The MSE loss function includes the errors of the real part, imaginary part and amplitude, and ensures that it is at the same order of magnitude as the negative SNR by taking the logarithm; the training process updates the model parameters of the initial speech enhancement model through back propagation, and finally obtains the target speech enhancement model. The target speech enhancement model uses batch normalization and instantaneous layer normalization to process data.

[0102] See also Figure 3 , Figure 3 is a schematic diagram of a framework of a speech enhancement process provided in an embodiment of the present application; Figure 3 As shown in the figure, the framework of the speech enhancement process includes a spectrum decomposition module, an encoding module, a neural network module, a decoding module, and a spectrum reconstruction module. The functions of each module are as follows:

[0103] Spectral decomposition module: converts the time domain signal of the input speech into frequency domain representation through STFT, so that the model can process complex speech signals more conveniently;

[0104] Encoding module: extracts useful features from the noisy spectrum, reduces the dimension of the data, and retains important speech information;

[0105] Neural network module: Use a dual-path recurrent neural network module to capture and enhance complex features in speech signals, improving the performance of speech enhancement models;

[0106] Decoding module: restores the low-dimensional features output by the network to the size of the original spectrum in order to perform inverse short-time Fourier transform (iSTFT) and reconstruct the time domain signal;

[0107] Spectrum reconstruction module: The enhanced frequency domain features are resynthesized into time domain signals through iSTFT to ensure that the processed signals can be actually used and ultimately obtain enhanced speech data.

[0108] Step S102, obtaining group voice data in the teaching classroom;

[0109] Among them, group voice data refers to the collection of voice information generated by all participants (mainly teachers and students) in a teaching classroom. Group voice data can include the teacher’s teaching content, students’ questions, answers, discussions and other sound activities in the classroom.

[0110] In the embodiment of the present application, the general mode of group learning is group learning. For example, in a teaching classroom, group members of the same group sit together, an array microphone device is placed in the center of the table, and the microphone array is used to collect real-time voice to obtain group voice data in the teaching classroom.

[0111] Step S103, inputting the group speech data into the target speech enhancement model for data enhancement processing to obtain target speech enhancement information;

[0112] The target speech enhancement information refers to a speech signal with higher quality obtained after being processed by the target speech enhancement model.

[0113] Exemplarily, an array microphone is used to acquire speech data in the direction of each group member in real time, and the speech data in each direction is respectively input into a target speech enhancement model as input, and the speech enhancement information of the group members is acquired through processing by the target speech enhancement model.

[0114] Step S104, inputting the group voice data into the target voiceprint recognition model for voiceprint recognition processing to obtain target voice character information corresponding to the group voice data;

[0115] The target voice character information is the speaker identity information corresponding to the group voice data obtained by performing voiceprint recognition processing on the target voiceprint recognition model.

[0116] Step S105, calculating the target voice enhancement information and the target voice character information to obtain group learning participation;

[0117] In some embodiments, step S105 may include: calculating the number of target active characters and the number of target voice characters in the target voice character information within the target time period; and using the ratio of the number of target active characters to the number of target voice characters as the group learning participation.

[0118] The target time period refers to the time range considered when calculating group learning participation, which can be set according to actual needs, such as the length of a class, the length of a discussion session, etc. The setting of the target time period is conducive to focusing on analyzing student participation within a specific time period, thereby more accurately evaluating group learning participation.

[0119] For the number of target active characters, it is the number of students who actively participate in class discussions or speak in the target time period, identified by the target voice character information. The target active characters (students) usually speak frequently, ask questions or participate in discussions in class, showing high activity and participation. For the number of target voice characters, it is the number of all students participating in the class obtained from the target voice character information in the target time period.

[0120] Optionally, group learning participation can also be expressed by group activity. Group learning participation is an indicator to measure the overall participation and interaction of students in a teaching class, which is of great significance for subsequent analysis of group learning atmosphere.

[0121] In the specific implementation, the calculation of group learning participation is specifically to use the proportion of active group members to the total number of group members within a period of time as the result. Exemplarily, the analysis time window length T is set, and the group learning participation is calculated based on the group member voice information output by the speech enhancement model within the window (from tT to the current time t). Among them, the analysis process is carried out in real time, and each moment is the analysis window with the previous T time span. The analysis process can obtain a continuously changing group learning participation.

[0122] For example, see Figure 4 , Figure 4 is a schematic diagram of a group learning participation calculation process provided by an embodiment of the present application; Figure 4 As shown, the calculation process of group learning participation is as follows: input the group voice data into the voiceprint recognition model for voiceprint recognition processing to obtain the group member information corresponding to the group voice data; calculate the voice enhancement information and group member information output by the voice enhancement model to obtain the group learning participation.

[0123] Step S106, performing feature extraction processing on the target speech enhancement information to obtain target speech feature information;

[0124] In some embodiments, step S106 may include: performing audio feature extraction processing on the target speech enhancement information to obtain an audio feature vector; performing text feature extraction processing on the target speech enhancement information to obtain a text feature vector corresponding to the audio feature vector; and concatenating the audio feature vector and the text feature vector to obtain target speech feature information.

[0125] The audio feature vector is a numerical representation of the audio signal after feature extraction. The text feature extraction process is to extract text data from the audio signal, and then extract features that can represent the text content or semantic information from the text data, that is, to obtain a text feature vector. The text feature vector is a numerical representation of the text data after feature extraction. For the target speech feature information, it is the fused data obtained by splicing the audio feature vector and the text feature vector.

[0126] In other embodiments, it may also include: selecting a target voice processing scheme from voice information processing schemes based on teaching environment information; if the target voice processing scheme is a cloud processing scheme among voice information processing schemes, constructing a target cloud processing model based on the cloud processing scheme; obtaining group voice data in the teaching classroom; inputting the group voice data into the target cloud processing model for data processing to obtain target cloud voice information and target voice character information corresponding to the group voice data; calculating the target cloud voice information and the target voice character information to obtain group learning participation; performing feature extraction processing on the target cloud voice information to obtain target voice feature information.

[0127] It is easy to understand that if there is no condition or demand to deploy a local voiceprint recognition model in the teaching implementation environment, there is no need to perform data preprocessing, and the cloud-based online speaker diarization model can be directly deployed. Then, the real-time group voice data of the teaching classroom is input into the cloud-based online speaker diarization model for data processing to obtain the target cloud-based voice information and the target voice character information corresponding to the group voice data; the target cloud-based voice information and the target voice character information are calculated to obtain the group learning participation; the target cloud-based voice information is subjected to feature extraction processing to obtain the target voice feature information.

[0128] In a specific implementation, the audio feature vector in the target speech enhancement information is extracted, and the text feature vector corresponding to the target speech enhancement information is extracted. For example, the open source toolkit openSmile is used to extract LLDS (Low Level Descriptors) features, and then the acquired speech information is subjected to text recognition processing to obtain text data, and the text feature vector corresponding to the speech information is extracted from the text data.

[0129] Step S107, inputting the target speech feature information into a target emotion prediction model to perform emotion prediction to obtain target emotion information;

[0130] Optionally, the target emotion prediction model is a pre-trained model that is specifically used to receive speech feature information (such as a combination of audio feature vectors and text feature vectors) as input and output corresponding emotion prediction results. The target emotion prediction model can capture the emotional characteristics contained in the speech signal by learning and understanding a large amount of speech data with emotion annotations, and perform emotion prediction on new speech input accordingly.

[0131] The target emotion information is the output result obtained after the target emotion prediction model performs emotion prediction on the target speech feature information.

[0132] In the specific implementation, see Figure 5 , Figure 5 It is a structural schematic diagram of a sentiment analysis and recognition module provided in an embodiment of the present application. The sentiment analysis and recognition module includes a voice pickup unit, a feature extraction unit and an emotion detection unit. The voice pickup unit is used to pick up the voice information learned in group learning; the feature extraction unit is used to extract the audio feature vector in the voice information, and extract the text feature vector corresponding to the voice information; the emotion detection unit is used to detect the emotions of the group members in the group learning according to the audio feature vector and the text feature vector, and determine the emotion type. Specifically, the voice information output by the speech enhancement model is first obtained, and then the feature extraction of the voice information is performed, and then the extracted audio feature vector and text feature vector are input into the emotion prediction model for prediction, and the prediction result is obtained, and the student's emotion type is determined according to the prediction result. Among them, the student's emotion type can be summarized as positive emotion type, negative emotion type and neutral emotion type.

[0133] Step S108, analyzing and processing the group learning participation and the target emotion information to obtain group learning atmosphere data.

[0134] Among them, group learning atmosphere data is used to reflect the atmosphere of group (group) learning in teaching classes. Group learning atmosphere data is a multi-dimensional, quantitative evaluation result that integrates group learning participation and emotional information to comprehensively reflect the atmosphere of group learning in teaching classes. It is not only beneficial for teachers to understand students' performance in class, but also can provide strong support for teaching improvement. At the same time, group members can also timely understand their own learning status and make self-adjustments based on group learning atmosphere data.

[0135] For group learning group atmosphere analysis, an open source emotion prediction model is used to predict the emotion type of each group member. The group learning atmosphere of the group is obtained based on the proportion of members with positive emotion types, negative emotion types, and neutral emotion types in the total number of group members and the learning participation rate.

[0136] It should be noted that the emotion prediction model can refer to the implementation principles in the relevant technology. The embodiment of the present application does not limit the selected emotion prediction model, and the embodiment of the present application will not be described in detail here.

[0137] Exemplarily, the group discussion atmosphere is derived by combining the group learning participation and the emotional types of group members. In order to be intuitive and convenient for classroom observation and adjustment, the discussion atmosphere is set to only two levels, namely excellent and poor. When the group learning participation is lower than 50%, the discussion atmosphere is poor; when the group learning participation is greater than 50% and the number of positive emotion members plus neutral emotion members is less than the number of negative emotion members, the discussion atmosphere is also poor; when the group learning participation is greater than 50% and the number of positive emotion members plus neutral emotion members is greater than the number of negative emotion members, the discussion atmosphere is excellent. It should be noted that the atmosphere level can be increased and the corresponding calculation method can be adapted according to the actual situation, and the embodiments of the present application do not limit this.

[0138] It should be noted that after step S108, the following steps are also included: the data is visualized through a specific device to achieve real-time feedback in the classroom. Specifically, the group learning participation analysis results are mapped to a series of colors through a color LED (Light-Emitting Diode) device, and the changes in light are used to characterize the changes in the group participation in the group learning classroom, and the classroom group learning participation and group learning atmosphere analysis results are displayed in real time through a display device. The real-time feedback effect of group learning based on voice enables teachers to understand the classroom situation in real time and conveniently. Finally, teachers can adjust and intervene in classroom teaching through feedback results. Specifically, teachers can understand the teaching situation of the group learning classroom at any time according to the visual real-time feedback function, and intervene in the classroom in real time according to needs, for example, guide groups with low participation to discuss, understand the problems encountered by groups with poor group atmosphere and give suggestions, etc. At the same time, it can also enable group members to understand their own learning status and self-adjust according to the group learning atmosphere data in a timely manner.

[0139] See also Figure 6 , Figure 6 is a schematic diagram of the interface content of a visualization module provided in an embodiment of the present application, such as Figure 6 As shown, the feedback page content in the visualization module includes a color LED device ( Figure 6 The group learning participation analysis results are mapped to a series of colors, and the light changes are used to represent the changes in group participation in the group learning classroom. The display device also displays the classroom group learning participation and group learning atmosphere analysis results in real time. Figure 6 As shown, Figure 6 The yellow part indicates low group participation, and the green part indicates high group participation. When the yellow color gradually deepens to the green part, it means that the group participation transitions from low to high. Figure 6 As shown, Figure 6 The LED device in the image is green. According to the color change comparison standard, it means that the group participation is high. Combined with the emotional information, it can be obtained Figure 6 In the visualization interface, the content of "discussion atmosphere: excellent" also includes "number of participants in the discussion: N people" and "inactive members: Zhang San, Li Si". Through the visualization real-time feedback function, teachers can understand the teaching situation of group learning classes at any time and intervene in the class in real time according to needs.

[0140] Steps S101 to S108 shown in the embodiment of the present application are as follows: obtaining a speech information processing scheme, and constructing a target speech enhancement model and a target voiceprint recognition model based on the speech information processing scheme; obtaining group speech data in a teaching classroom; inputting the group speech data into the target speech enhancement model for data enhancement processing to obtain target speech enhancement information; inputting the group speech data into the target voiceprint recognition model for voiceprint recognition processing to obtain target speech character information corresponding to the group speech data; calculating the target speech enhancement information and the target speech character information to obtain group learning participation; performing feature extraction processing on the target speech enhancement information to obtain target speech feature information; inputting the target speech feature information into the target emotion prediction model for emotion prediction to obtain target emotion information; analyzing and processing the group learning participation and target emotion information to obtain group learning atmosphere data. The embodiment of the present application performs enhancement processing on the group voice data through the target voice enhancement model, thereby improving the clarity and purity of the voice signal, thereby improving the accuracy of subsequent emotion prediction and group learning participation analysis; using the target voiceprint recognition model to perform voiceprint recognition on the group voice data can accurately identify the voice of each group member, providing the possibility for subsequent personalized feedback and intervention; by analyzing the group learning participation combined with the emotion prediction results, it is possible to obtain comprehensive evaluation of the group learning atmosphere data, so that group members and teachers can quickly capture the learning situation in the classroom.

[0141] In order to explain the principle of the technical solution of the present invention in detail, the overall process of the present invention is described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and cannot be regarded as a limitation of the present invention.

[0142] See also Figure 7 , Figure 7 is a flow chart of the speech-based group learning atmosphere analysis method provided in the embodiment of the present application; please refer to Figure 8 , Figure 8 Schematic diagram of neural network application implementation of the speech-based group learning atmosphere analysis method provided in the embodiment of the present application; Figure 7 and Figure 8 , we can get the implementation process of the speech-based group learning atmosphere analysis method, such as Figure 7 As shown, the overall process of the speech-based group learning atmosphere analysis method includes the following six steps (step 701-step 706):

[0143] Step 701, selecting a voice information processing scheme according to the teaching environment information;

[0144] In the specific implementation, the voice information processing solution includes a local processing solution and a cloud processing solution. The local processing solution is a solution that includes a training data preprocessing process and a model building process. After training data preprocessing and model building, the target speech enhancement model and the target voiceprint recognition model can be obtained. The target speech enhancement model and the target voiceprint recognition model will process the real-time teaching classroom voice data to obtain a group learning atmosphere solution; the cloud processing solution is to directly deploy a cloud processing model (cloud online speaker diarization model), and then process the real-time teaching classroom voice data according to the cloud processing model to obtain a group learning atmosphere solution.

[0145] Specifically, Figure 8 As shown, first, it is necessary to select a voice information processing solution. It is necessary to determine whether the teaching implementation environment has the conditions or requirements for deploying a local voiceprint recognition model. If there are no conditions or requirements for deploying a local voiceprint recognition model in the teaching implementation environment (i.e., a cloud processing solution is adopted), no preprocessing is required, and the next implementation step 702 is directly entered. If there are conditions or requirements for deploying a local voiceprint recognition model in the teaching implementation environment (i.e., a local processing solution is adopted), a data preprocessing process is required. The data preprocessing process is specifically as follows: first, pre-record the student voice information, then use the Mel frequency cepstrum coefficient to extract the features of the student voice information, input the extracted voice features into the Mamba model for training, and obtain a voiceprint recognition model. The voiceprint recognition model includes a student voiceprint encoder, which is used to obtain the student's voiceprint information. Multiple voiceprint information is used to construct a voiceprint information library.

[0146] Step 702: deploy a cloud-based online speaker diarization model or train a local speech enhancement model or a local voiceprint recognition model according to the selected type of speech information processing solution;

[0147] According to the selection of the voice information processing solution in step 701, deploy the cloud online speaker diarization model or train the local voice enhancement model and the local voiceprint recognition model. Specifically: if you choose to use a non-local voiceprint recognition solution (i.e., use a cloud processing solution), deploy the cloud online speaker diarization model; if you choose to use a local voiceprint recognition solution (i.e., use a local processing solution), train the local voice enhancement model and the local voiceprint recognition model.

[0148] Step 703, obtaining real-time classroom speech, and processing the real-time speech using a cloud processing model or using a local speech enhancement model or a local voiceprint recognition model for speech enhancement processing or voiceprint recognition processing according to the selected type of speech information processing solution, to obtain processed speech information;

[0149] Specifically, Figure 8 As shown, in a teaching classroom, members of the same student group sit together, and an array microphone device is placed in the center of the table. The microphone array is used to collect real-time voice to obtain real-time group voice data in the teaching classroom.

[0150] like Figure 8 As shown, if a local processing solution is adopted, the real-time group voice data in the teaching classroom is input into the local voice enhancement model as input data, and the voice enhancement information of the group members is obtained after being processed by the local voice enhancement model; at the same time, the group voice data is input into the local voiceprint recognition model for voiceprint recognition processing to obtain the voice character information corresponding to the group voice data.

[0151] like Figure 8 As shown, if a cloud processing solution is adopted, the real-time group voice data in the teaching classroom is input into the cloud online speaker diarization model for data processing to obtain the cloud voice information and the target voice character information corresponding to the group voice data.

[0152] Step 704, based on the voice information output in step 703, use an algorithm to calculate the group learning participation, perform sentiment analysis on the group learning group members, and calculate the group learning atmosphere by combining the group learning participation and sentiment analysis results;

[0153] Specifically, the calculation of group learning participation is done by using the proportion of active group members to the total number of group members over a period of time as the result.

[0154] like Figure 8 As shown in the figure, if the cloud processing solution is adopted, the cloud voice information obtained by data processing of the cloud online speaker diarization model and the target voice character information corresponding to the group voice data are calculated to obtain the group learning participation. The cloud voice information is processed by audio feature extraction and text feature extraction to obtain voice feature information.

[0155] like Figure 8 As shown, if the local processing solution is adopted, the group learning participation is calculated using the speech enhancement information of the group members output by the speech enhancement model and the voice character information obtained by the voiceprint recognition model. The speech enhancement information is processed by audio feature extraction and text feature extraction to obtain speech feature information.

[0156] For the group learning group atmosphere analysis, an open source emotion prediction model is used to predict the emotion of the voice feature information obtained from each scheme, and the emotion type of each group member is predicted. According to the proportion of members with positive emotion type, negative emotion type and neutral emotion type in the total group members, combined with the learning participation, the group learning atmosphere of the group is obtained.

[0157] Step 705, visually displaying the data through a specific device to achieve real-time feedback in the classroom;

[0158] Specifically, Figure 8 As shown, the group learning participation analysis results are mapped to a series of colors through a color LED device, and the light changes are used to represent the changes in group participation in the group learning class. The display device also displays the classroom group learning participation and group learning atmosphere analysis results in real time. The real-time feedback effect of voice-based group learning allows teachers to understand the classroom situation in real time and conveniently.

[0159] Step 706: The teacher adjusts and intervenes in classroom teaching based on the feedback results.

[0160] Specifically, Figure 8 As shown, teachers can adjust and intervene in classroom teaching through feedback results. Specifically, teachers can understand the teaching situation of group learning classes at any time based on the visual real-time feedback function, and intervene in the classroom in real time according to needs. For example, they can guide groups with low participation to discuss, understand the problems encountered by groups with poor group atmosphere and give suggestions, etc. At the same time, it can also enable group members to understand their own learning status in time and adjust themselves according to the group learning atmosphere data.

[0161] It should be pointed out that this embodiment only briefly illustrates the general process of the speech-based group learning atmosphere analysis method. The detailed description of each step can refer to the relevant content in the aforementioned embodiment and will not be repeated here. It can be understood that the present invention is not limited to this.

[0162] The embodiment of the present application obtains a speech information processing scheme, and constructs a target speech enhancement model and a target voiceprint recognition model based on the speech information processing scheme; obtains group speech data in a teaching classroom; inputs the group speech data into the target speech enhancement model for data enhancement processing to obtain target speech enhancement information; inputs the group speech data into the target voiceprint recognition model for voiceprint recognition processing to obtain target speech character information corresponding to the group speech data; calculates the target speech enhancement information and the target speech character information to obtain group learning participation; performs feature extraction processing on the target speech enhancement information to obtain target speech feature information; inputs the target speech feature information into the target emotion prediction model for emotion prediction to obtain target emotion information; analyzes and processes the group learning participation and the target emotion information to obtain group learning atmosphere data. The embodiment of the present application enhances the group voice data through the target voice enhancement model, thereby improving the clarity and purity of the voice signal, thereby improving the accuracy of subsequent emotion prediction and group learning participation analysis; using the target voiceprint recognition model to perform voiceprint recognition on the group voice data, it is possible to accurately identify the voice of each group member, providing the possibility for subsequent personalized feedback and intervention; by analyzing the group learning participation combined with the emotion prediction results, it is possible to obtain a comprehensive evaluation of the group learning atmosphere data, so that group members and teachers can quickly capture the learning situation in the classroom. At the same time, it enables group members to timely understand their own learning status and self-adjust according to the group learning atmosphere data. At the same time, teachers can also quickly capture dynamic changes in the classroom and take timely intervention measures, which can assist teachers in optimizing teaching strategies and help improve classroom teaching results.

[0163] See also Fig. 9 The embodiment of the present application also provides a speech-based group learning atmosphere analysis system 900, which can implement the above-mentioned speech-based group learning atmosphere analysis method. The speech-based group learning atmosphere analysis system includes the following modules:

[0164] The model building module 901 is used to obtain a speech information processing scheme, and build a target speech enhancement model and a target voiceprint recognition model based on the speech information processing scheme;

[0165] A group voice data acquisition module 902 is used to acquire group voice data in a teaching classroom;

[0166] A data enhancement processing module 903 is used to input the group speech data into the target speech enhancement model for data enhancement processing to obtain target speech enhancement information;

[0167] The voiceprint recognition processing module 904 is used to input the group voice data into the target voiceprint recognition model for voiceprint recognition processing to obtain the target voice character information corresponding to the group voice data;

[0168] A group learning participation calculation module 905 is used to calculate the target voice enhancement information and the target voice character information to obtain the group learning participation;

[0169] A feature extraction processing module 906 is used to perform feature extraction processing on the target speech enhancement information to obtain target speech feature information;

[0170] Emotion information prediction module 907, used for inputting the target speech feature information into a target emotion prediction model to perform emotion prediction and obtain target emotion information;

[0171] The group learning atmosphere analysis module 908 is used to analyze and process the group learning participation and the target emotion information to obtain group learning atmosphere data.

[0172] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0173] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned voice-based group learning atmosphere analysis method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0174] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0175] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0176] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC for short), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0177] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the speech-based group learning atmosphere analysis method of the embodiment of this application;

[0178] Input / output interface 1003, used to implement information input and output;

[0179] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0180] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0181] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0182] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned speech-based group learning atmosphere analysis method is implemented.

[0183] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0184] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0185] The speech-based group learning atmosphere analysis method and speech-based group learning atmosphere analysis system provided in the embodiments of the present application obtain a speech information processing scheme, and construct a target speech enhancement model and a target voiceprint recognition model based on the speech information processing scheme; obtain group speech data in a teaching classroom; input the group speech data into the target speech enhancement model for data enhancement processing to obtain target speech enhancement information; input the group speech data into the target voiceprint recognition model for voiceprint recognition processing to obtain target speech character information corresponding to the group speech data; calculate the target speech enhancement information and the target speech character information to obtain group learning participation; perform feature extraction processing on the target speech enhancement information to obtain target speech feature information; input the target speech feature information into the target emotion prediction model for emotion prediction to obtain target emotion information; analyze and process the group learning participation and target emotion information to obtain group learning atmosphere data. The embodiment of the present application performs enhancement processing on the group voice data through the target voice enhancement model, thereby improving the clarity and purity of the voice signal, thereby improving the accuracy of subsequent emotion prediction and group learning participation analysis; using the target voiceprint recognition model to perform voiceprint recognition on the group voice data can accurately identify the voice of each group member, providing the possibility for subsequent personalized feedback and intervention; by analyzing the group learning participation combined with the emotion prediction results, it is possible to obtain comprehensive evaluation of the group learning atmosphere data, so that group members and teachers can quickly capture the learning situation in the classroom.

[0186] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0187] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0188] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0189] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0190] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0191] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0192] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0193] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0194] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0195] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0196] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A speech-based group learning atmosphere analysis method, characterized in that: The method comprises the following steps: Obtain a speech information processing solution, and build a target speech enhancement model and a target voiceprint recognition model based on the speech information processing solution; Obtain group speech data in teaching classrooms; Inputting the group speech data into the target speech enhancement model for data enhancement processing to obtain target speech enhancement information; Inputting the group voice data into the target voiceprint recognition model for voiceprint recognition processing to obtain target voice character information corresponding to the group voice data; Calculating the target voice enhancement information and the target voice character information to obtain group learning participation; Performing feature extraction processing on the target speech enhancement information to obtain target speech feature information; Inputting the target speech feature information into a target emotion prediction model to perform emotion prediction to obtain target emotion information; The group learning participation and the target emotion information are analyzed and processed to obtain group learning atmosphere data.

2. The method according to claim 1, characterized in that The speech information processing scheme includes a local processing scheme, the speech information processing scheme is acquired, and a target speech enhancement model and a target voiceprint recognition model are constructed based on the speech information processing scheme, including: Acquire the speech information processing scheme and the speech data to be trained; Selecting a target speech processing scheme from the speech information processing schemes according to the teaching environment information; If the target speech processing scheme is the local processing scheme in the speech information processing scheme, the target speech enhancement model and the target voiceprint recognition model are constructed according to the speech data to be trained.

3. The method according to claim 2, characterized in that If the target speech processing scheme is the local processing scheme in the speech information processing scheme, constructing the target speech enhancement model and the target voiceprint recognition model according to the speech data to be trained includes: If the target speech processing scheme is the local processing scheme in the speech information processing scheme, information is extracted from the speech data to be trained to obtain pure speech data; Adding a reverberation effect to the clean voice data to obtain reverberation voice data; Integrating the reverberant speech data and a plurality of target noise segments to obtain target training data; Inputting the target training data into an initial speech enhancement model for training to obtain the target speech enhancement model; If the target speech processing scheme is the local processing scheme in the speech information processing scheme, performing feature extraction processing on the speech data to be trained to obtain target training feature information; The target training feature information is input into a linear time series model for training to obtain the target voiceprint recognition model; the target voiceprint recognition model is used to obtain the voiceprint information of the target voice character, and the voiceprint information is used to construct a voiceprint model library.

4. The method according to claim 3, characterized in that The step of inputting the target training feature information into a linear time series model for training to obtain the target voiceprint recognition model includes: Inputting the target training feature information into the linear time series model for training to obtain an initial embedding vector; Regularizing the initial embedding vector to obtain a target embedding vector; Calculating the centroid of the target embedding vector to obtain the target vector centroid; Constructing a similarity matrix based on the target embedding vector and the target vector centroid; The linear time series model is trained based on the similarity matrix and the target voiceprint loss function to obtain the target voiceprint recognition model.

5. The method according to claim 1, characterized in that The step of calculating the target voice enhancement information and the target voice character information to obtain group learning participation includes: Calculating the number of target active characters and the number of target voice characters in the target voice character information within the target time period; The ratio of the number of target active characters to the number of target voice characters is used as the group learning participation.

6. The method according to claim 1, characterized in that The step of performing feature extraction processing on the target speech enhancement information to obtain target speech feature information includes: Performing audio feature extraction processing on the target speech enhancement information to obtain an audio feature vector; Performing text feature extraction processing on the target speech enhancement information to obtain a text feature vector corresponding to the audio feature vector; The audio feature vector and the text feature vector are concatenated to obtain the target speech feature information.

7. The method according to claim 1, characterized in that The voice information processing solution includes a cloud processing solution, and the method further includes: Selecting a target speech processing scheme from the speech information processing schemes according to the teaching environment information; If the target voice processing solution is the cloud processing solution in the voice information processing solution, constructing a target cloud processing model based on the cloud processing solution; Acquiring the group voice data in the teaching classroom; Inputting the group voice data into the target cloud processing model for data processing to obtain target cloud voice information and the target voice character information corresponding to the group voice data; Calculating the target cloud voice information and the target voice character information to obtain the group learning participation degree; Perform feature extraction processing on the target cloud voice information to obtain the target voice feature information.

8. The speech-based group learning atmosphere analysis system is characterized by: The system includes the following modules: A model building module, used to obtain a speech information processing solution, and build a target speech enhancement model and a target voiceprint recognition model based on the speech information processing solution; A group voice data acquisition module is used to acquire group voice data in a teaching classroom; A data enhancement processing module, used for inputting the group speech data into the target speech enhancement model for data enhancement processing to obtain target speech enhancement information; A voiceprint recognition processing module, used for inputting the group voice data into the target voiceprint recognition model for voiceprint recognition processing, and obtaining the target voice character information corresponding to the group voice data; A group learning participation calculation module, used to calculate the target voice enhancement information and the target voice character information to obtain the group learning participation; A feature extraction processing module is used to perform feature extraction processing on the target speech enhancement information to obtain target speech feature information; An emotion information prediction module, used for inputting the target speech feature information into a target emotion prediction model to perform emotion prediction and obtain target emotion information; The group learning atmosphere analysis module is used to analyze and process the group learning participation and the target emotion information to obtain group learning atmosphere data.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images