A coarse-grained sound event detection method in classroom based on audio and video feature fusion

Abstract

The present invention belongs to the technical field of smart classrooms, and specifically relates to a classroom coarse-grained sound event detection method based on audio and video feature fusion, including: using a video information processing model to perform face detection on video data frame by frame, and extract all mouth state information in each frame; performing human posture detection on video data frame by frame, and extracting all posture information of all people in each frame; splicing all mouth state information and all posture information of all people according to time series as video action features; using an audio information processing model to extract audio features from audio data frame by frame, and converting audio data into text to extract text features frame by frame; splicing audio features and text features according to time series as audio information features; based on video action features and audio information features, using feature fusion and classification models, outputting the detection and classification results of the speaking role in each frame, and obtaining coarse-grained sound event detection results. The present invention can improve the accuracy of classroom sound event detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of smart classrooms, and more specifically, relates to a classroom coarse-grained sound event detection method based on audio and video feature fusion. Background Art

[0002] Classroom activity detection has always been a hot topic, and experts and scholars have been conducting research in this area. By analyzing the behavior of students and teachers in class and making corresponding adjustments to the content after class, both teachers' teaching skills and students' learning efficiency can be improved.

[0003] To achieve classroom activity detection, high-quality and detailed classroom activity records are essential. This requires determining whether someone is speaking and the identity of the speaker, and recording the start and end time of the teacher and student's speech in the classroom. In general, it is a coarse-grained sound event detection for the classroom. However, unless someone records everything that happens in the classroom, or the teacher and students each wear independent sound collection devices, it is very difficult to obtain separate activity records of the teacher and students in the classroom. However, these two are obviously impossible to achieve. Often, only one or two sound collection devices can be provided, which contain a mixture of all sounds in the classroom scenario.

[0004] Some studies provide classroom sound event detection based on audio. When the quality of the collected audio is poor, the classroom environment is more complex, or the teacher's voice may be very close to the voice of some students, the accuracy of classroom sound event detection will be affected. At the same time, in some situations, such as when teachers and students communicate, the teacher and students switch very quickly, which brings great challenges to classroom sound event detection based only on audio. It is often difficult to detect the event switching point well, resulting in some wrong divisions. Summary of the invention

[0005] In view of the defects and improvement needs of the prior art, the present invention provides a method for detecting coarse-grained sound events in the classroom based on audio and video feature fusion, which aims to improve the detection accuracy of coarse-grained sound events in the classroom.

[0006] To achieve the above object, according to one aspect of the present invention, a method for detecting coarse-grained sound events in a classroom based on audio and video feature fusion is provided, comprising:

[0007] Acquire video and audio data generated in class;

[0008] Using the constructed video information processing model, face detection is performed frame by frame on the video data to extract the state information of all mouths in each frame; human posture detection is performed frame by frame on the video data to extract the posture information of all persons in each frame; the state information of all mouths and the posture information of all persons are spliced ​​in time series as video action features;

[0009] The constructed audio information processing model is used to extract audio features from the audio data frame by frame, and the audio data is converted into text, and text features are extracted from the text frame by frame; the audio features and the text features are spliced ​​in time series as audio information features;

[0010] Based on the video action features and the audio information features, the constructed feature fusion and classification model based on the attention mechanism is adopted to output the detection and classification results of the speaking role in each frame, so as to obtain the coarse-grained sound event detection results in the classroom. Each coarse-grained sound event includes the start and end time of the event and its corresponding speaking role. The speaking roles are divided into three categories: teacher, student and mixed.

[0011] Furthermore, the part of the video information processing model used to extract the status information of all mouths in each frame is constructed by training based on the MTCNN algorithm.

[0012] Furthermore, the state information of each mouth is composed of four key points: left and right mouth corners and upper and lower lips.

[0013] Furthermore, when a mask is recognized, the status information of the corresponding mouth is identified using the mask information.

[0014] Furthermore, the part of the video information processing model used to extract posture information of all persons in each frame is constructed by training based on the AlphaPose algorithm.

[0015] Further, the video data includes teacher video data and student video data;

[0016] The video action features obtained based on the teacher video data and the video action features obtained based on the student video data are spliced ​​in time series as the overall video action features, which are used to be input into the feature fusion and classification model together with the audio information features.

[0017] Further, the teacher's posture information extracted based on the teacher video data is posture information composed of 15 key points of the head, neck, left and right shoulders, left and right elbows, left and right hands, left and right ankles, left and right knees, left and right hips and torso;

[0018] The student's posture information extracted based on the student video data is posture information composed of eight key points: head, neck, left and right shoulders, left and right elbows, and left and right hands.

[0019] Furthermore, the audio feature is a Mel-frequency cepstral coefficient.

[0020] Further, the method of splicing the audio features and the text features according to the time series is:

[0021] The audio features and the text features are respectively input into the CNN feature extraction network, the feature extraction results are aligned in time series and then spliced, and then input into the RNN network to obtain the audio information features combined with context information.

[0022] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the above-mentioned method for detecting coarse-grained sound events in a classroom based on the fusion of audio and video features.

[0023] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0024] (1) The present invention is based on the data of multiple modes produced by the actual classroom environment, including audio and video, and it is also proposed to convert audio into text and perform text feature extraction, which is equivalent to proposing to combine the data of three modes to perform a classroom coarse-grained sound event detection method. Specifically, by extracting and analyzing the features of classroom audio and video information, three features, audio features, text features and video action features, are used. This multi-modal fusion method is used to perform classroom coarse-grained sound event detection, which effectively avoids the problem of poor segmentation results when encountering poor audio quality, complex classroom environment, and the teacher's voice may be very close to the voice of some students in the audio mode alone, thereby improving the accuracy of classroom coarse-grained sound event detection.

[0025] (2) The present invention proposes that video data includes teacher video data and student video data, and clearly collects video data related to teacher actions and video data related to student actions, so as to improve the accuracy of coarse-grained sound event detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic block diagram of a method for detecting coarse-grained sound events in a classroom based on audio and video feature fusion provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0028] Embodiment 1

[0029] A coarse-grained classroom sound event detection method based on audio and video feature fusion, such as Figure 1 As shown, including:

[0030] Acquire video and audio data generated in class;

[0031] Using the constructed video information processing model, face detection is performed frame by frame on the video data to extract the state information of all mouths in each frame; human posture detection is performed frame by frame on the video data to extract the posture information of all persons in each frame; the state information of all mouths and the posture information of all persons are spliced ​​in time series as video action features;

[0032] The constructed audio information processing model is used to extract audio features from the audio data frame by frame, and the audio data is converted into text, and text features are extracted from the text frame by frame; the audio features and the text features are concatenated according to the time series to serve as audio information features;

[0033] Based on the above video action features and the above audio information features, the constructed feature fusion and classification model based on the attention mechanism is adopted to output the detection and classification results of the speaking role in each frame, so as to obtain the coarse-grained sound event detection results in the classroom. Each coarse-grained sound event includes the start and end time of the event and its corresponding speaking role. The speaking roles are divided into three categories: teacher, student and mixed.

[0034] The classroom coarse-grained sound event detection model in this embodiment finally divides the sound event detection results into three categories: teacher, student and Babble (i.e., mixed mode, speaking together), and outputs the classroom coarse-grained sound event detection results, including the start and end time of each event and its corresponding speaking role. This embodiment combines mouth features, text features and action features to jointly determine the speaking role.

[0035] The retinaface, MTCNN and other algorithms are used to detect the faces of teachers and students in the classroom; at the same time, the facial feature point algorithm (such as Dlib, MTCNN) is used to extract the facial key points of the detected face, which can be used as a preferred implementation. Here, the MTCNN algorithm is used for video feature extraction (i.e., facial key points of the face), which can complete the two tasks of face detection and key point extraction at the same time. If retinaface is selected for face detection, Dlib and other algorithms will be required for facial key point extraction later.

[0036] MTCNN consists of three layers of network structures: P-Net for quickly generating candidate windows, R-Net for high-precision candidate window filtering and selection, and O-Net for generating the final bounding box and facial key points. P-Net is a fully convolutional network that undergoes three layers of convolution with a convolution kernel size of 3*3. Three fully connected layers are used to output whether there is a face in the area, and at the same time, multiple possible bounding boxes of faces and the corresponding facial key points are output, and these areas are input into R-Net for further processing. Compared with P-Net, R-Net has an additional fully connected layer of dimension 128 after the last convolution layer, which is used to refine the input and discard most of the wrong inputs. Finally, three fully connected layers are used to output the three pieces of information of bounding boxes and facial key points respectively. The O-Net network is more complex. The information transmitted from R-Net undergoes 4 convolutions and 3 maximum poolings, passes through a fully connected layer of dimension 256, and then uses three fully connected layers to output the three pieces of information of bounding boxes and facial key points respectively.

[0037] Some of the earliest face algorithms and corresponding data sets were labeled as left and right eyes, nose, left and right corners of the mouth, and these five key points were used as the key points of the face for face recognition and detection. This is not applicable to the scenario of this embodiment. The scenario of this embodiment requires mouth information features and whether a mask is worn, so the key point information is modified and the key point of whether a mask is worn is added. That is, because in this example, it is more necessary to focus on the mouth features to detect whether to speak, it is preferred to modify the feature point information extraction target to the four key points of the left and right corners of the mouth and the upper and lower lips. At the same time, considering that there may be students or teachers wearing masks, when the mask is identified during feature point extraction, the mask information is output as the mouth key point, and the mask information can be marked with a feature map, for example.

[0038] Regarding the acquisition of posture information, a human posture algorithm (such as AlphaPose) is used to extract human key points. Preferably, the AlphaPose algorithm is used, which is a multi-person posture estimation method. It adopts a top-down detection method, first detects the bounding box of each person in the image, and then independently detects the posture in each human body bounding box. It mainly includes two networks: (1) Symmetric spatial transformation network, including a spatial transformation network STN, a spatial inverse transformation network SDTN, and a single-person posture estimation SPPE, and SPPE is located between STN and SDTN. Then a parallel Parallel SPPE branch is connected from STN to optimize the network. Parallel SPPE mainly determines whether the posture extracted by STN is located at the center position, back-propagates the error to STN, and updates the STN network weight. Parallel SPPE does not participate in the output and is only used to optimize the STN network. (2) Parameterized posture maximum suppression network. Human body positioning will inevitably produce redundant detection frames, that is, a detection target has multiple detection frames, so redundant posture detection will also be generated. The posture distance measurement is used to compare the similarity of the postures to remove similar postures.

[0039] In order to collect clearer and more useful motion features, preferably, the above-mentioned video data includes teacher video data and student video data; illustratively, a camera can be set at the front of the classroom to collect student video data, and a camera can be set at the back of the classroom to collect teacher video data.

[0040] The video action features obtained based on the teacher's video data and the video action features obtained based on the student's video data are spliced ​​in time series as the overall video action features, which are used to input into the feature fusion and classification model together with the above-mentioned audio information features.

[0041] As a preferred embodiment, the teacher's posture information extracted based on the teacher video data is posture information composed of 15 key points of the head, neck, left and right shoulders, left and right elbows, left and right hands, left and right ankles, left and right knees, left and right hips and torso;

[0042] The student's posture information extracted based on the student video data is composed of eight key points: head, neck, left and right shoulders, left and right elbows, and left and right hands.

[0043] Regarding the extraction of text features and audio features. The classroom audio data is transcribed into text data. Preferably, the whisper, kalid and other algorithms can be used. The transcription result contains the timestamp corresponding to the text content, which inputs the text data into the Bert model. The Bert model uses a multi-layer bidirectional Transformer as a feature extractor to extract word vector features. The extracted features can contain context information, where the word vector features are obtained by combining the vocabulary embedding vector and position embedding of the input text. ti =w i +p i embedding , x ti Represents the word vector feature of the i-th word, w i is the vocabulary embedding vector corresponding to the i-th word, p i embedding is the position embedding vector corresponding to the i-th word, and the final word vector feature Xt = [x t1 ,x t2 …x tn ].

[0044] Use audio processing tools to extract audio features Xs such as MFCC and LLD. Preferably, this example chooses to extract Mel-frequency cepstral coefficients, referred to as MFCC features. First, x(n) represents the original audio signal, which is preprocessed to remove silent segments, reduce noise, pre-emphasize, etc. The preprocessed audio signal is divided into overlapping frames with a frame length of N. Preferably, a 50% ratio is used for audio signal overlap (to avoid excessive changes between two adjacent frames, so there will be an overlapping area between two adjacent frames). Apply a window function w(n) to the audio signal of each frame, that is, s(n) = x(n)*w(n), where s(n) is the windowed signal. Perform a fast Fourier transform (FFT) on the windowed signal, that is, S(k) = FFT[s(n)], where S(k) is the frequency domain signal and k represents the frequency index. Calculate the energy spectrum E(k) of each frame, that is, the square of the amplitude of the frequency domain signal: E(k) = |S(k)|^2. Design a set of Mel filters that are evenly distributed on the Mel scale. Filter the energy spectrum through this set of filters to obtain the output H of the filter bank. m (k), i.e. Where m represents the filter index, H m (k′,m) is the response of the mth filter at frequency index k′. Performing a logarithmic operation on the output of the filter bank yields the logarithmic energy spectrum M m (k) = log(H m (k)), perform discrete cosine transform (DCT) on the logarithmic energy spectrum to obtain the cepstrum coefficient Xs m :

[0045]

[0046] Among them, Xs m is the mth cepstral coefficient.

[0047] Finally, we get Xs m It can be used as the MFCC feature vector of the audio signal.

[0048] As a preferred implementation, the method of splicing the above audio features and the above text features in time series is:

[0049] The above audio features and the above text features are respectively input into the CNN feature extraction network, the feature extraction results are aligned in time series and then spliced, and then input into the RNN network to obtain the audio information features combined with context information.

[0050] The constructed feature fusion and classification model based on the attention mechanism is used to output the detection and classification results of the speaking role in each frame. The video action feature Xv and the audio information feature Xa are input into the fusion model with the attention mechanism, and then fused and input into the fully connected layer to obtain the integrated classroom sound event detection results. The audio information feature is used as the key and the video action feature is used as the query feature to calculate the correlation s between Xai and Xvi. i , Xai represents the audio information feature at the i-th moment, Xvi represents the video action feature at the i-th moment, s i =Vtanh(W·Xvi+U·Xai), where W and U are weight parameters, tanh is the activation function, and V is the weight parameter of multimodal fusion. α i is the normalized attention coefficient, and the final fused feature can be expressed as Yi = α i *Xai+(1-α i )*Xvi.

[0051] Finally, the fused feature Yi is input into the fully connected layer network for classification to obtain the coarse-grained sound event detection results in the classroom.

[0052] Regarding the construction of the video information processing model, the audio information processing model, and the feature fusion and classification model based on the attention mechanism, the three can be trained independently, among which:

[0053] The loss function L of the part of the video information processing model used to extract the state information of all mouths in each frame mouth It is expressed as:

[0054]

[0055] In this formula, i refers to the number of mouth key points, e is the number of mouth key points, t represents time, indicating the number of frames, and l bce represents the cross entropy loss, p i,t Indicates the key point p at the mouth i The real state corresponding to time t is, Represents the mouth key point p i The predicted probability at time t.

[0056] The loss function L of the part of the video information processing model used to extract the posture information of all people in each frame posture It is expressed as:

[0057]

[0058] In this formula, i refers to the number of human key points, g is the number of human key points, t represents time, indicating the number of frames, and a i,t Indicates that at the posture key point a i The real state corresponding to time t is, Represents the key point a i The predicted probability at time t.

[0059] The loss function L of the part of the audio information processing model used to extract text features frame by frame word It is expressed as:

[0060]

[0061] In this formula, W represents the total vocabulary of the speech transcription text, q i Represents the true result of the i-th word in the speech transcription text result, Represents the prediction result of the i-th word in the speech transcription text result.

[0062] The loss function L used to fuse the above video action features and the above audio information features in the feature fusion and classification model is fusion It is expressed as:

[0063]

[0064] In this formula, i refers to the type of sound detection label, z is the number of sound detection labels, t represents time, indicating the frame, and y i,t Represents the classroom sound event detection result y in the feature fusion model i The actual state corresponding to time t Classroom sound event detection results i The predicted probability at time t.

[0065] For real label annotation, in video annotation, when teachers or students wear masks, the masks are annotated as key points of the mouth. Through manual annotation, the start and end time points of each event and their corresponding labels are marked. Student annotation does not distinguish between different students, and the unified label is student. When the sound is extremely short, no label is added to it. According to the above annotation rules, a well-annotated classroom audio and video dataset is obtained.

[0066] In general, the present invention adopts a method of using multiple modal data fusion to perform coarse-grained classroom sound event detection, adding teacher and student video modal features, detecting the current teacher and student mouth state and body posture, and overcoming the problem of affecting the accuracy of classroom sound event detection when only using audio when encountering poor audio quality, more complex classroom environment, and the teacher's voice may be very close to the voice of some students, because at this time the video modality can better reflect the current classroom sound event. At the same time, the use of audio and video fusion can avoid the problem of being unable to perform classroom sound event detection when encountering students and teachers wearing masks when using only video modality.

[0067] Embodiment 2

[0068] A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute a method for detecting coarse-grained sound events in a classroom based on audio and video feature fusion as described above.

[0069] The relevant technical solution is the same as that in Embodiment 1 and will not be described in detail here.

[0070] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting coarse-grained sound events in a classroom based on audio and video feature fusion, characterized in that: include: Acquire video and audio data generated in class; Using the constructed video information processing model, face detection is performed frame by frame on the video data to extract the status information of all mouths in each frame; Performing human posture detection on the video data frame by frame, extracting posture information of all persons in each frame; splicing the state information of all mouths and the posture information of all persons in a time series as video action features; The constructed audio information processing model is used to extract audio features from the audio data frame by frame, and the audio data is converted into text, and text features are extracted from the text frame by frame; splicing the audio features and the text features according to a time series as audio information features; Based on the video action features and the audio information features, the constructed feature fusion and classification model based on the attention mechanism is adopted to output the detection and classification results of the speaking role in each frame, so as to obtain the coarse-grained sound event detection results in the classroom. Each coarse-grained sound event includes the start and end time of the event and its corresponding speaking role. The speaking roles are divided into three categories: teacher, student and mixed. Specifically, the video action feature Xv and the audio information feature Xa are input into the fusion model with the attention mechanism, and after fusion, they are input into the fully connected layer to obtain the integrated classroom sound event detection results. Specifically, the audio information feature is used as the key and the video action feature is used as the query feature, and the correlation s between Xai and Xvi is calculated. i , Xai represents the audio information feature at the i-th moment, Xvi represents the video action feature at the i-th moment, s i =Vtanh(W·Xvi+U·Xai), where W and U are weight parameters, tanh is the activation function, and V is the weight parameter of multimodal fusion. α i is the normalized attention coefficient, and the final fused feature can be expressed as Yi = α i *Xai+(1-α i )*Xvi; Finally, the fusion feature Yi is input into the fully connected layer network for classification to obtain the coarse-grained sound event detection result of the classroom; The video data includes teacher video data and student video data; Then, the video action features obtained based on the teacher video data and the video action features obtained based on the student video data are spliced ​​in time series as the total video action features, which are used to be input into the feature fusion and classification model together with the audio information features; The teacher's posture information extracted based on the teacher video data is posture information composed of 15 key points, namely, head, neck, left and right shoulders, left and right elbows, left and right hands, left and right ankles, left and right knees, left and right hips and torso; The student's posture information extracted based on the student video data is posture information composed of eight key points: head, neck, left and right shoulders, left and right elbows, and left and right hands.

2. The method for detecting coarse-grained classroom sound events according to claim 1, characterized in that: The part of the video information processing model used to extract the status information of all mouths in each frame is constructed by training based on the MTCNN algorithm.

3. The method for detecting coarse-grained classroom sound events according to claim 1 or 2, characterized in that: The state information of each mouth is composed of four key points: the left and right corners of the mouth and the upper and lower lips.

4. The method for detecting coarse-grained classroom sound events according to claim 1 or 2, characterized in that: When a mask is recognized, the status information of the corresponding mouth is marked with the mask information.

5. The method for detecting coarse-grained classroom sound events according to claim 1, characterized in that: The part of the video information processing model used to extract the posture information of all persons in each frame is constructed by training based on the AlphaPose algorithm.

6. The method for detecting coarse-grained classroom sound events according to claim 1, characterized in that: The audio feature is a Mel-frequency cepstral coefficient.

7. The method for detecting coarse-grained classroom sound events according to claim 1, characterized in that: The method of splicing the audio features and the text features according to the time series is: The audio features and the text features are respectively input into the CNN feature extraction network, the feature extraction results are aligned in time series and then spliced, and then input into the RNN network to obtain the audio information features combined with context information.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by the processor, the device where the storage medium is located is controlled to execute a method for detecting coarse-grained sound events in a classroom based on audio and video feature fusion as described in any one of claims 1 to 7.

Images