Class-in and class-out state determination method, electronic equipment and storage medium

Abstract

The invention provides a class starting and ending state determination method, electronic equipment and a storage medium. The method comprises the following steps: acquiring image data and audio data, wherein the acquisition time of the image data is the same as that of the audio data; respectively extracting features of the image data and the audio data to obtain initial image features and initial audio features; performing feature conversion on the initial image features and the initial audio features to unify the dimensions of the initial image features and the initial audio features into the same dimension; performing feature fusion on the target image feature and the target audio feature to obtain a target fusion feature, the target image feature being an initial image feature after feature conversion, and the target audio feature being an initial audio feature after feature conversion; and based on the target fusion feature, determining a class-on and class-off state classification result. According to the method, the class starting and ending states can be accurately determined in real time.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method for determining the status of entering or leaving a class, an electronic device, and a storage medium. Background Art

[0002] A smart classroom is a modern teaching place based on emerging information technologies such as the Internet of Things, cloud computing, and big data. It provides convenient and efficient services for teaching activities through smart devices and software, integrates modern technology into the teaching process, and aims to improve students' independent thinking and learning abilities.

[0003] Determining the status of starting and ending classes is an important part of smart classrooms. Its purpose is to classify the status of starting and ending classes in the classroom through the real-time collected data. The accuracy of the classification results of the status of starting and ending classes directly affects the subsequent intelligent allocation of classroom resources, the activation and deactivation of teaching auxiliary tools, and the automatic recording of teacher and student behaviors. Therefore, how to accurately and real-time determine the status of starting and ending classes with limited information has become an urgent problem to be solved in the construction of smart classrooms.

[0004] In view of this, this application is hereby filed. Summary of the invention

[0005] In view of the above problems, the present application is proposed. According to one aspect of the present application, a method for determining the status of entering or leaving a get out of class is provided, comprising: Acquire image data and audio data, where the image data and the audio data are collected at the same time; Extracting features of the image data and the audio data respectively to obtain initial image features and initial audio features; Performing feature conversion on the initial image feature and the initial audio feature respectively, so as to unify the dimensions of the initial image feature and the initial audio feature into the same dimension; Performing feature fusion on the target image feature and the target audio feature to obtain a target fusion feature, wherein the target image feature is an initial image feature after feature conversion, and the target audio feature is an initial audio feature after feature conversion; Based on the target fusion features, the classification result of the class entry and exit status is determined.

[0006] Exemplarily, the performing feature conversion on the initial image feature and the initial audio feature respectively includes: Processing the initial image features using a first feature processing operation to eliminate spatial dimensions in the initial image features; The initial audio features are processed using a second feature processing operation to eliminate a time dimension in the initial audio features.

[0007] Exemplarily, the first feature processing operation includes: Performing two-dimensional global average pooling compression on the initial image features to compress the spatial dimension of the initial image features; The compressed initial image features are subjected to dimensionality reduction processing to eliminate the spatial dimensions in the initial image features.

[0008] Exemplarily, the second feature processing operation includes: Performing one-dimensional global average pooling compression on the initial audio features to compress the time dimension of the initial audio features; The compressed initial audio features are subjected to dimensionality reduction processing to eliminate the audio dimensions in the initial audio features.

[0009] Exemplarily, the performing feature conversion on the initial image features and the initial audio features respectively further includes: Compressing the number of channels of the initial image feature after the first feature processing operation to a first number of channels, where the first number of channels is the same as the number of channels of the initial audio feature; or, Compressing the number of channels of the initial audio feature after the second feature processing operation to a second number of channels, where the second number of channels is the same as the number of channels of the initial image feature; or, Compressing the number of channels of the initial image feature after the first feature processing operation to a third number of channels; The number of channels of the initial audio feature after the second feature processing operation is compressed to the third number of channels.

[0010] Exemplarily, the target image feature and the target audio feature both include a first dimension for representing a batch and a second dimension for representing a number of channels; and the step of fusing the target image feature with the target audio feature includes: Adding a dimension to the target image feature and the target audio feature respectively, so that the target image feature and the target audio feature both have a third dimension; splicing the target image feature and the target audio feature having the third dimension in the third dimension to obtain a first fusion feature; Processing the first fused feature through a first transposition operation to exchange the second dimension and the third dimension of the first fused feature; Processing the first dimension of the first fused feature processed by the first transposition operation through a batch normalization operation; Processing the first fused features processed by the batch normalization operation through a second transposition operation to restore the dimension of the first fused features; Processing the first fused features processed by the second transposition operation through a one-dimensional convolution operation to eliminate the third dimension of the first fused features; Adding the second fusion feature to the initial image feature and the initial fusion feature to obtain the target fusion feature; The second fusion feature is the first fusion feature processed by the one-dimensional convolution operation.

[0011] Exemplarily, determining the classification result of the class entry and exit status based on the target fusion feature includes: The target fusion features are input into the prediction model to obtain the classification result of the get out of class entry and exit status.

[0012] Exemplarily, the step of inputting the target fusion feature into a prediction model to obtain a classification result of the class entry and exit status includes: using the prediction model to perform the following operations: Based on the target fusion features, output the probability of belonging to the class status and the probability of belonging to the get out of class status; The state with a higher probability between the class state and the get out of class ending state is determined as the class ending and class ending state classification result.

[0013] According to another aspect of the present application, an electronic device is provided, including a processor and a memory, wherein a computer program is stored in the memory, and the processor is used to execute the computer program to implement the above method.

[0014] According to another aspect of the present application, a computer-readable storage medium is provided, storing a computer program / instruction, and the computer program / instruction implements the above method when executed by a processor.

[0015] In the above technical solution, the target fusion feature is obtained by fusing the features in the image data and the audio data, and the target fusion feature is used to determine the classification result of the get out of class status. This method can make full use of the complementary characteristics of image and audio data, overcome the limitations of the single modality method, improve the perception ability and generalization performance of the changes in the get out of class status, and thus more accurately determine the classification results of the get out of class status. In short, this solution can solve the problems of low accuracy and lack of real-time performance in the automatic discrimination of the get out of class status in the prior art. The classification results of the get out of class status determined by this solution can provide reliable support for subsequent functions such as the intelligent allocation of smart classroom resources and the automatic activation and deactivation of teaching auxiliary tools, which helps to improve the user experience.

[0016] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0018] Figure 1 A schematic flow chart showing a method for determining the state of starting or ending a get out of class according to an embodiment of the present application is shown; Figure 2 A schematic block diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present application more obvious, the example embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application described in the present application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present application.

[0020] It can be understood that the classification result of the get out of class status is used to represent the state of the current scene (class status or get out of class end status). As mentioned above, the accuracy of the classification result of the get out of class status directly affects the subsequent intelligent allocation of classroom resources, the activation and deactivation of teaching auxiliary tools, and the automatic recording of teacher and student behaviors. In the related art, the classification results of the get out of class status are mainly determined by the following three methods: image-based method, audio-based method and rule-based method. Among them, the image-based method mainly judges the get out of class status based on the input image data. This method has problems such as poor model generalization ability and poor perception of classroom status changes. The audio-based method mainly judges the get out of class status based on the input audio data. This method has poor anti-interference ability, low fault tolerance and insufficient robustness. The rule-based method mainly obtains the initial classification results by using the input image data and audio data respectively, and then synthesizes the two initial classification results according to the preset decision scheme to obtain the final get out of class status classification results. This method is heavily dependent on the quality of the artificially designed strategy and cannot be achieved end-to-end. In short, the accuracy of the methods for determining the status of classes in related technologies is poor. How to accurately and in real time determine the status of classes with limited information is still an urgent problem to be solved in the construction of smart classrooms. In view of this, the present application provides a method for determining the status of classes, an electronic device and a storage medium. The method can integrate image data and audio data to more accurately determine the status of classes, thereby providing an accurate basis for various functions in smart classrooms (such as intelligent allocation of classroom resources, activation and deactivation of teaching auxiliary tools, and automatic recording of teacher and student behaviors, etc.), thereby helping to improve user experience. The method, electronic device and storage medium are described in detail below.

[0021] According to one aspect of an embodiment of the present application, a method for determining a class entry or exit status is provided. Figure 1 FIG. 2 is a schematic flow chart of a method for determining the class entry and exit status according to an embodiment of the present application. Figure 1 As shown, the method may include the following steps S110, S120, S130, S140 and S150.

[0022] In step S110, image data and audio data are acquired, and the acquisition time of the image data and the audio data is the same.

[0023] Optionally, the image data can be obtained by any existing or future developed image acquisition device installed in the smart classroom. For example, the device can be a camera. The image data can be a frame of real-time acquired image or a certain video frame in a video.

[0024] Optionally, the audio data can be obtained through any existing or future developed audio acquisition device installed in the smart classroom. For example, the device can be a microphone, a recorder, etc.

[0025] In this article, the acquisition time of image data and audio data is the same, that is, the acquisition time of image data is the same as the end time of audio data acquisition. Among them, the acquisition time of audio data is the preset time. For example, if the preset time is 3s and the acquisition time is the 6th second, the image data is the image acquired at the 6th second (when the image data is a video frame, it is the video frame corresponding to the 6th second), and the audio data is the audio acquired from the 3rd second to the 6th second.

[0026] In some embodiments, an image acquisition device in a smart classroom may be used to acquire image data, and at the same time, an audio acquisition device may be used to acquire audio data, thereby obtaining image data and corresponding audio data.

[0027] In step S120, features of the image data and audio data are extracted respectively to obtain initial image features and initial audio features.

[0028] After obtaining the image data and audio data, the features of the two can be extracted so that the status of class entry and exit can be determined based on the features in the subsequent steps. For example, for image data, one or more of directional gradient histograms, convolutional neural networks, autoencoders, generative adversarial networks, residual networks, etc. can be used to extract image features. For audio data, one or more of mel frequency cepstral coefficients, linear frequency cepstral coefficients, recurrent neural networks, temporal convolutional networks, etc. can be used to extract audio features. Those skilled in the art can understand the specific implementation methods for extracting features of image data and audio data, which will not be described in detail.

[0029] In some embodiments, extracting features from image data may include the following steps: processing the image data using a backbone network of a ResNet18 model to obtain initial image features. Input into the backbone network of the ResNet18 model to extract the image feature vector (i.e., initial image features). ResNet18 can maintain good performance even with a high number of parameters. This solution uses the backbone network of the ResNet18 model to extract features from image data, which can ensure good feature extraction performance.

[0030] In a specific embodiment, the encoding process of a single residual block (ResBlock) in the backbone network of the ResNet18 model can be expressed by the following formula: ; ; ; ; In the formula, Represents input data; and Respectively represent the weight matrix and bias vector of the first convolutional layer of ResBlock; Represents the intermediate feature map after the first convolution operation; ReLU represents the ReLU activation function, which can map nonlinearity into the feature map to increase the nonlinearity of the feature; Represents the feature map after being processed by the activation function; and They represent the weight matrix and bias vector in the second convolutional layer in ResBlock respectively; is the feature map obtained after the second convolution operation; is the output feature of the residual block. In this embodiment, the input feature map is connected through residual linking. And the convolution feature Add together to generate the final output features of ResBlock In this process, the convolution operation is represented as *, that is, a local weighted sum operation is performed on the input features. It can be understood that for a single residual block, the input of the residual block is the output of the previous residual block, the input data of the first residual block is the image data, and the output features of the last residual block are the initial image features.

[0031] In some embodiments, extracting features of audio data may include the following steps: using the ECAPA-TDNN (Emphasized Channel Attention, Propagation and Aggregation in TDNN) model to analyze the audio data. Processing to obtain initial audio features .

[0032] In a specific embodiment, the ECAPA-TDNN model can process audio data in the following way: To process: First, a basic feature extraction is performed through a one-dimensional convolution, as shown in the following formula: ; in represents a one-dimensional convolution, is the feature vector obtained by initial feature extraction (i.e., extraction through one-dimensional convolution).

[0033] Then, the residual block Enhanced feature extraction is performed as follows: ; ; ; in, It is a convolutional module that combines residual connections and attention mechanisms to improve the ability of feature representation. This step uses three convolutional modules in sequence to extract features, and uses residual connections to add input features to extracted features each time to alleviate the degradation of deep networks and the problem of small gradients.

[0034] In addition, in order to improve the model's ability to extract important features, the SE attention mechanism can also be used to improve the representation ability. Its structure is shown in the following formula: ; in, is the channel in the SE module The weight of This means that the input feature is at position and Channel The value on . and are the height and width of the feature map, respectively. Represents the sigmoid activation function. After that, the SE module performs nonlinear transformation through two layers of fully connected networks, and uses activation functions to adjust channel weights. Finally, it performs weighted operations on each pass to improve the model representation ability. The specific formula is as follows: ; ; In the formula, and are the fully connected matrix weights for dimensionality reduction and dimensionality increase, respectively. Represents the output feature map of the reallocated channel weights. The SE module calculates the importance of each channel through global average pooling, adjusts the channel weights in the feature map through the fully connected layer and activation function, and finally applies these weights to the feature map to ensure the classification accuracy of the model.

[0035] To improve the classification accuracy, the ECAPA-TDNN model then concatenates three sets of features at different levels in the channel dimension to fuse the three sets of features, and extracts the input features through attention statistical pooling (ASP) to more effectively capture the long-term changes in speaker characteristics. Finally, a fully connected layer is used to map out Audio features. This process can be expressed by the following formula: ; ; In the formula, Represents the concatenation of feature maps along the channel dimension. is the feature after splicing and fusion. and They represent the attention statistical pooling and fully connected layer feature mapping operations respectively.

[0036] The above briefly describes the steps of processing audio data by the ECAPA-TDNN model. Those skilled in the art can understand the specific implementation method of extracting audio features through the ECAPA-TDNN model, which is not described in detail.

[0037] In step S130 , feature conversion is performed on the initial image features and the initial audio features respectively, so as to unify the dimensions of the initial image features and the initial audio features into the same dimension.

[0038] It can be understood that there are differences in the data formats of the initial image features and the initial audio features. The initial image features have spatial dimensions, while the initial audio features have temporal dimensions. In this example, the dimensions of the two features are unified into the same dimension through feature transformation in advance (i.e., the different dimensions in the two features are eliminated and only the same dimensions are retained), so that the two features can fully complement each other, which is conducive to the fusion of the features of the two modalities in the subsequent steps.

[0039] In step S140, the target image feature and the target audio feature are feature fused to obtain a target fused feature, wherein the target image feature is the initial image feature after feature conversion, and the target audio feature is the initial audio feature after feature conversion.

[0040] After the initial features of the two modalities are converted, the target image features and the target audio features obtained after the conversion can be directly fused. The feature fusion method includes but is not limited to splicing, element-by-element addition, bilinear pooling, and fusing two features using a deep learning model, etc. This example does not limit this.

[0041] In step S150, based on the target fusion features, the classification result of the get out of class entry and exit status is determined.

[0042] After obtaining the target fusion feature, the classification result of the get out of class status can be determined based on the target fusion feature, that is, whether the current scene in the smart classroom is in the class status or the get out of class status. In some embodiments, the classification result of the get out of class status can be determined by methods such as support vector machines, decision trees, naive Bayes classifiers, pre-trained neural networks, etc., which are not limited in this example.

[0043] In the above technical solution, the target fusion feature is obtained by fusing the features in the image data and the audio data, and the target fusion feature is used to determine the classification result of the get out of class status. This method can make full use of the complementary characteristics of image and audio data, overcome the limitations of the single modality method, improve the perception ability and generalization performance of the changes in the get out of class status, and thus more accurately determine the classification results of the get out of class status. In short, this solution can solve the problems of low accuracy and lack of real-time performance in the automatic discrimination of the get out of class status in the prior art. The classification results of the get out of class status determined by this solution can provide reliable support for subsequent functions such as the intelligent allocation of smart classroom resources and the automatic activation and deactivation of teaching auxiliary tools, which helps to improve the user experience.

[0044] Exemplarily, before extracting features of the image data and the audio data respectively in step S120, the method may further include the following steps: preprocessing the image data and the audio data respectively. Step S120 specifically includes the following steps: extracting features of the preprocessed image data and the preprocessed audio data respectively.

[0045] Optionally, the image data is preprocessed in a manner including, but not limited to, normalization, standardization, color conversion, data enhancement, and the like. In a specific embodiment, the image data (i.e., a frame of image) may be first standardized and formatted, the image may be converted into a tensor format, and the pixel values ​​may be normalized to the range of [0, 1]. Next, the image is adjusted to a uniform size to ensure input consistency and computational efficiency. Finally, the RGB channels are standardized according to a preset mean and standard deviation, which helps the image meet the requirements for the image in subsequent feature extraction steps. For example, when the backbone network of the ResNet18 model is used to extract image features, the above-mentioned preprocessing method may be used to make the processed image data consistent with the distribution of the expected input of the ResNet18 model.

[0046] Optionally, when a neural network model such as a ResNet18 model is used to extract image features, the model can be trained in advance using a large number of samples. During training, the samples can be processed by data enhancement methods such as random cropping and horizontal flipping, which can enable the trained model to have good feature extraction capabilities for image data collected in different scenarios.

[0047] Optionally, the audio data is preprocessed in a manner including, but not limited to, filtering, denoising, normalization, and the like. In a specific embodiment, first, the audio data is uniformly sampled and formatted to ensure the consistency of the processing process. Next, the audio data is converted into a spectrogram using a short-time Fourier transform (STFT), and further converted into a Mel spectrogram through a Mel filter. This process effectively extracts key features in the audio, simulates the human ear's perception of frequency, and reduces noise interference. Finally, the Mel spectrogram is subjected to necessary normalization and compression to reduce computational complexity while retaining core audio features.

[0048] The above scheme can make the image data and audio data meet the needs of subsequent processing by preprocessing the image data and audio data, and help reduce the computational complexity in subsequent steps and improve computational efficiency.

[0049] Exemplarily, step S130, performing feature conversion on the initial image features and the initial audio features respectively, may include the following steps S131 and S132.

[0050] In step S131, the initial image features are processed using a first feature processing operation to eliminate the spatial dimension in the initial image features.

[0051] In step S132, the initial audio features are processed using a second feature processing operation to eliminate the time dimension in the initial audio features.

[0052] As described above, the difference between the initial image feature and the initial audio feature is that the initial image feature has a spatial dimension, while the initial audio feature has a temporal dimension. In the solution of this example, the spatial dimension and the temporal dimension in the initial image feature are eliminated by the first feature processing operation and the second feature processing operation, respectively, so that the features of the two modalities can be easily merged.

[0053] Optionally, the first feature processing operation includes, but is not limited to, one or more of vector flattening, feature extraction, global pooling, dimensionality reduction, etc., which is not limited in this example.

[0054] Optionally, the second feature processing operation includes but is not limited to one or more of mean pooling, maximum pooling, dimensionality reduction, etc., which is not limited in this example.

[0055] In the above technical solution, the spatial dimension in the initial image feature is eliminated through the first feature processing operation, and the time dimension in the initial audio feature is eliminated through the second feature processing operation. This can unify the data format of the features of the two modalities, facilitating feature fusion in subsequent steps.

[0056] Exemplarily, the first feature processing operation includes: performing two-dimensional global average pooling compression on the initial image features to compress the spatial dimensions of the initial image features; and performing dimensionality reduction processing on the compressed initial image features to eliminate the spatial dimensions in the initial image features.

[0057] It can be understood that the initial image features ,in, is the number of channels of the initial image features, and Represent the height and width (i.e., spatial dimension) of the initial image features. In this example, two-dimensional global average pooling compression and dimensionality reduction can be used to obtain For ease of description, the initial image features are expressed as [C, H, W]. In this example, [C, H, W] is converted to [C, 1, 1] through two-dimensional global average pooling compression. Then, [C] is obtained through dimensionality reduction processing. In this way, the spatial dimension of the initial image features is eliminated, which can facilitate feature fusion in subsequent steps.

[0058] Exemplarily, the second feature processing operation includes: performing one-dimensional global average pooling compression on the initial audio features to compress the time dimension of the initial audio features; and performing dimensionality reduction processing on the compressed initial audio features to eliminate the audio dimension in the initial audio features.

[0059] It can be understood that the initial audio characteristics , where is the number of channels of the initial audio feature and represents the time dimension of the initial audio feature. In this example, through one-dimensional global average pooling compression and dimensionality reduction processing, we can obtain For ease of description, the initial audio feature is expressed as [C, T]. In this example, [C, T] is converted to [C, 1] through one-dimensional global average pooling compression. Then, [C, 1] is converted to [C] through dimensionality reduction processing. In this way, the time dimension of the initial audio feature is eliminated, which can facilitate feature fusion in subsequent steps.

[0060] Before fusing the features of the two modes, the number of channels of the two modes may be compressed to be consistent in order to perform feature fusion. For example, the number of channels of one mode may be directly compressed to be consistent with the number of channels of the other mode. For another example, the number of channels of the two modes may be compressed to a specific number of channels. The channel number compression process is described below with several embodiments.

[0061] In one implementation, feature conversion is performed on the initial image features and the initial audio features respectively, and the following steps may also be included: compressing the number of channels of the initial image features after the first feature processing operation to a first number of channels, where the first number of channels is the same as the number of channels of the initial audio features.

[0062] In another implementation, feature conversion is performed on the initial image features and the initial audio features respectively, and the following steps may also be included: compressing the number of channels of the initial audio features after the second feature processing operation to a second number of channels, and the second number of channels is the same as the number of channels of the initial image features.

[0063] In another implementation, the feature conversion is performed on the initial image feature and the initial audio feature respectively, and the following steps may be further included: compressing the number of channels of the initial image feature after the first feature processing operation to a third number of channels; compressing the number of channels of the initial audio feature after the second feature processing operation to a third number of channels. The third number of channels may be selected according to actual needs, for example, may be 192.

[0064] Optionally, the method of compressing the number of channels in the above includes but is not limited to using a convolution layer, a multi-layer perceptron, etc. For example, the third number of channels is 192. The number of channels of the initial image feature after the first feature processing operation and the number of channels of the initial audio feature after the second feature processing operation can be mapped and compressed to 192 by the multi-layer perceptron.

[0065] The above technical solution can maintain the data consistency of the features of the two modalities and avoid information imbalance by compressing the number of channels of the two modalities to be consistent before fusing the features of the two modalities, thereby helping to better fuse the features of the two modalities.

[0066] Exemplarily, the target image feature and the target audio feature both include a first dimension for representing a batch and a second dimension for representing the number of channels; the target image feature and the target audio feature are feature-fused, including: adding a dimension to the target image feature and the target audio feature respectively, so that the target image feature and the target audio feature both have a third dimension; splicing the target image feature and the target audio feature with the third dimension on the third dimension to obtain a first fused feature; processing the first fused feature through a first transposition operation to exchange the second dimension of the first fused feature with the third dimension; processing the first dimension of the first fused feature processed by the first transposition operation through a batch normalization operation; processing the first fused feature processed by the batch normalization operation through a second transposition operation to restore the dimension of the first fused feature; processing the first fused feature processed by the second transposition operation through a one-dimensional convolution operation to eliminate the third dimension of the first fused feature; adding the second fused feature to the initial image feature and the initial fused feature to obtain the target fused feature; wherein the second fused feature is the first fused feature processed by the one-dimensional convolution operation.

[0067] For ease of description, the solution of this example is described below through a specific embodiment. In this embodiment, the target image feature is , the target audio feature is First, and Add a dimension to each and concatenate in the third dimension to obtain , and then Transpose (i.e. swap the second and third dimensions) to obtain . Then Batch normalization is performed on the first dimension to stabilize the features of the two modalities and then transposed back to , then a one-dimensional convolution is performed to reduce the dimension and merge the dimensions to obtain Finally, and as well as Add together to get the fusion result The process can be expressed by the following formula: ; ; ; in, Represents a splicing operation, represents the batch normalization operation, Represents a one-dimensional convolution operation.

[0068] The above technical solution can effectively combine image features and audio features. The fused features can provide a more accurate basis for determining the status of entering and leaving get out of class in subsequent steps, which helps to improve the accuracy of classification results.

[0069] Exemplarily, based on the target fusion feature, determining the classification result of the class entry and exit status includes: inputting the target fusion feature into a prediction model to obtain the classification result of the class entry and exit status. The prediction model includes but is not limited to a vector machine, a decision tree, a naive Bayes classifier, a pre-trained neural network, etc., which is not limited in this example.

[0070] The above technical solution can improve the calculation efficiency by using the prediction model to determine the classification results of the get out of class status, and help avoid the interference of human factors and ensure the accuracy of the results. Exemplarily, the target fusion feature is input into the prediction model to obtain the classification result of the in-class and out-of-get out of class status, including: using the prediction model to perform the following operations: based on the target fusion feature, output the probability of belonging to the in-class state and the probability of belonging to the out-of-get out of class state; determine the state with a higher probability between the in-class state and the out-of-get out of class state as the classification result of the in-class and out-of-get out of class status.

[0071] In the solution of this example, the probability of being in class and the probability of being off class can be output in advance based on the target fusion features, and then the state with a higher probability is selected as the classification result of the on-class and off-get out of class states. In a specific embodiment, the process can be expressed by the following formula: ; in, Indicates the classification results of the status of entering and leaving get out of class; Express i conduct Operation, that is, finding the index that maximizes the input function i In other words, this function is used to find the subscript of the category with the highest probability (categories include class status and off-get out of class status) i ; Indicates that the output is converted into a probability distribution. The output result of this function is between [0,1], and the sum of the probabilities of all categories output by this function is 1 (that is, the sum of the probability of belonging to the class state and the probability of belonging to the get out of class state is 1); fc 1 and fc 2 represents two fully connected layers.

[0072] in, The specific formula of the function is as follows: ; In the above formula, Indicates i The input value of each category (i.e., the output value of the class status or the get out of class ending status obtained through two fully connected forward propagations), e is the natural logarithm.

[0073] In the above technical solution, the probability of belonging to the class state and the probability of belonging to the get out of class end state can be determined respectively according to the target fusion feature, and then the state with a larger probability is selected as the classification result of the class start and end state. This method can determine the classification result of the class start and end state relatively quickly and accurately, and can meet the requirements of the smart classroom for the real-time and accuracy of the class start and end state determination.

[0074] The effectiveness of the method of the present application (which may be referred to as a multimodal method) is verified below through a specific example.

[0075] In this embodiment, the multimodal method includes the following steps: first, image data and audio data are acquired and preprocessed. Then, the backbone network of the ResNet18 model is used to extract the initial image features of the image data, and the ECAPA-TDNN is used to extract the initial audio features of the audio data. After obtaining the initial image features and the initial audio features, the data formats of the features of the two modalities are unified (i.e., the spatial dimension of the initial image features and the temporal dimension of the initial audio features are eliminated), and the number of channels is compressed to obtain the target image features. and target audio features Next, the target image feature and the target audio feature are fused to obtain the target fusion feature. The target fusion feature is input into the prediction model to obtain the classification result of the get out of class entry and exit status. The specific implementation method of each step in this embodiment has been described in detail above and will not be repeated here.

[0076] In this embodiment, 9000 pairs of data (including data corresponding to the class status and data corresponding to the get out of class status) are collected in advance, and each pair of data includes image data and corresponding audio data. The image-based methods in the prior art (including methods for identifying images using the ResNet18 model to determine the classification results, methods for identifying images using the ResNet50 model to determine the classification results, methods for identifying images using the Swin Transformer-tiny model to determine the classification results, and methods for identifying images using the VIT-B16 model to determine the classification results), audio-based methods (methods for identifying audio using the ECAPA-TDNN model to determine the classification results, methods for identifying audio using the CAM++ model to determine the classification results, and methods for identifying audio using the ERes2Net model to determine the classification results) and the multimodal method of the present application are used to identify these data, and analyze the accuracy of the recognition results (i.e., the classification accuracy in the table below). The results are shown in Table 1 below.

[0077] Table 1 Classification results of entering and leaving get out of class

[0078] It can be seen from the above table that compared with the prior art, the method of the present application has higher classification accuracy.

[0079] According to another aspect of the embodiments of the present application, an electronic device is also provided. Figure 2 FIG. 1 is a schematic block diagram of an electronic device according to an embodiment of the present application. Figure 2 As shown, the electronic device 200 includes: a processor 210 and a memory 220. The memory 220 stores a computer program, and the processor 210 is used to execute the computer program to implement the above method.

[0080] According to another aspect of the embodiment of the present application, a computer-readable storage medium is also provided. The storage medium stores a computer program / instruction, and the computer program / instruction implements the above method when executed by the processor. The storage medium may include, for example, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0081] A person skilled in the art can easily understand the implementation structure, working principle and beneficial effects of the electronic device and the computer-readable storage medium by reading the above method, and will not be described in detail for the sake of brevity.

[0082] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present application to this. Those of ordinary skill in the art may make various changes and modifications therein without departing from the scope and spirit of the present application. All these changes and modifications are intended to be included within the scope of the present application as required by the appended claims.

[0083] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0084] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0085] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0086] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various application aspects, in the description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present application should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the corresponding claims, the application point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0087] Those skilled in the art will understand that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstract and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise explicitly stated, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0088] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0089] The various component embodiments of the present application may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all functions of some modules in the electronic device according to the embodiment of the present application. The present application may also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present application may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0090] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0091] The above is only a specific implementation or description of a specific implementation of the present application, and the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for determining the status of starting and ending classes, characterized in that: include: Acquire image data and audio data, where the image data and the audio data are collected at the same time; Extracting features of the image data and the audio data respectively to obtain initial image features and initial audio features; Performing feature conversion on the initial image feature and the initial audio feature respectively, so as to unify the dimensions of the initial image feature and the initial audio feature into the same dimension; Performing feature fusion on the target image feature and the target audio feature to obtain a target fusion feature, wherein the target image feature is an initial image feature after feature conversion, and the target audio feature is an initial audio feature after feature conversion; Based on the target fusion features, the classification result of the class entry and exit status is determined.

2. The method according to claim 1, characterized in that The performing feature conversion on the initial image feature and the initial audio feature respectively comprises: Processing the initial image features using a first feature processing operation to eliminate spatial dimensions in the initial image features; The initial audio features are processed using a second feature processing operation to eliminate a time dimension in the initial audio features.

3. The method according to claim 2, characterized in that The first feature processing operation includes: Performing two-dimensional global average pooling compression on the initial image features to compress the spatial dimension of the initial image features; The compressed initial image features are subjected to dimensionality reduction processing to eliminate the spatial dimensions in the initial image features.

4. The method according to claim 2, characterized in that: The second feature processing operation includes: Performing one-dimensional global average pooling compression on the initial audio features to compress the time dimension of the initial audio features; The compressed initial audio features are subjected to dimensionality reduction processing to eliminate the audio dimensions in the initial audio features.

5. The method according to any one of claims 2 to 4, characterized in that: The performing feature conversion on the initial image features and the initial audio features respectively further includes: Compressing the number of channels of the initial image feature after the first feature processing operation to a first number of channels, where the first number of channels is the same as the number of channels of the initial audio feature; or, Compressing the number of channels of the initial audio feature after the second feature processing operation to a second number of channels, where the second number of channels is the same as the number of channels of the initial image feature; or, Compressing the number of channels of the initial image feature after the first feature processing operation to a third number of channels; The number of channels of the initial audio feature after the second feature processing operation is compressed to the third number of channels.

6. The method according to claim 1, characterized in that The target image feature and the target audio feature both include a first dimension for representing a batch and a second dimension for representing the number of channels; and the step of fusing the target image feature with the target audio feature includes: Adding a dimension to the target image feature and the target audio feature respectively, so that the target image feature and the target audio feature both have a third dimension; splicing the target image feature and the target audio feature having the third dimension in the third dimension to obtain a first fusion feature; Processing the first fused feature through a first transposition operation to exchange the second dimension and the third dimension of the first fused feature; Processing the first dimension of the first fused feature processed by the first transposition operation through a batch normalization operation; Processing the first fused features processed by the batch normalization operation through a second transposition operation to restore the dimension of the first fused features; Processing the first fused features processed by the second transposition operation through a one-dimensional convolution operation to eliminate the third dimension of the first fused features; Adding the second fusion feature to the initial image feature and the initial fusion feature to obtain the target fusion feature; The second fusion feature is the first fusion feature processed by the one-dimensional convolution operation.

7. The method according to claim 1, characterized in that Determining the classification result of the class entry and exit status based on the target fusion feature includes: The target fusion features are input into the prediction model to obtain the classification result of the get out of class entry and exit status.

8. The method according to claim 7, characterized in that The step of inputting the target fusion feature into a prediction model to obtain a classification result of the class entry and exit status includes: using the prediction model to perform the following operations: Based on the target fusion features, output the probability of belonging to the class status and the probability of belonging to the get out of class status; The state with a higher probability between the class state and the get out of class ending state is determined as the class ending and class ending state classification result.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: A computer program / instruction is stored, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images