Image classification method and device, computer device and storage medium

Abstract

The application relates to an image classification method and device, computer equipment and a storage medium. The method comprises the following steps: identifying an expression classification result and a limb classification result of a multi-target object in a to-be-classified image through a dual-modal classification model, obtaining a learning state of the target object based on a decision result of the expression classification result and the limb classification result of the target object, identifying a sentiment attitude of the target object from multiple dimensions, improving the emotional sensitivity of the sentiment attitude to the target object, and determining a learning effect corresponding to the to-be-classified image according to the similarity between an actual probability distribution value corresponding to each learning state and a preset probability distribution value, so that the real learning situation of students can be effectively reflected.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image classification method, apparatus, computer device, and storage medium. Background Technology

[0002] Online teaching, due to the lack of training opportunities and experimental environments, results in a significant gap between its actual effectiveness and that of in-person teaching. Therefore, improving the interactivity, monitoring, and feedback capabilities of online teaching plays a crucial role in enhancing the overall effectiveness of online education.

[0003] Existing methods for improving the interactivity, supervision, and feedback capabilities of online teaching generally employ artificial intelligence image recognition to collect and classify students' facial expressions, and use deep learning models to complete facial emotion recognition tasks. Based on the facial emotion recognition results, classroom satisfaction can be obtained more efficiently and objectively.

[0004] However, training models based solely on facial expression recognition are insufficient to accurately describe students' learning status, thus failing to effectively reflect students' true learning situation. Summary of the Invention

[0005] Therefore, it is necessary to provide an image classification method, device, computer equipment, and storage medium that can intelligently identify students' emotional states through multimodal behavior, addressing the aforementioned technical problems.

[0006] Firstly, this application provides an image classification method, the method comprising:

[0007] Obtain facial expression and body images of multiple target objects in the image to be classified;

[0008] The facial expression image is input into the first classification model of the pre-trained bimodal classification model to obtain the facial expression classification result;

[0009] The limb image is input into the second classification model of the pre-trained bimodal classification model to obtain the limb classification result;

[0010] The facial expression classification results and body classification results are fused together, and the decision-maker of the bimodal classification model is used to obtain the learning state of multiple target objects in the image to be classified, as well as the actual probability distribution value corresponding to each learning state.

[0011] The learning effect of the image to be classified is determined based on the similarity between the actual probability distribution value and the preset probability distribution value corresponding to each learning state.

[0012] In one embodiment, the facial expression image is input into the first classification model of a pre-trained bimodal classification model to obtain the facial expression classification result, including:

[0013] The facial expression image is input into the first classification model of the pre-trained bimodal classification model. The first classification model slices the facial expression image to obtain multiple feature maps.

[0014] Multiple feature maps are input into the attention module of the first classification model. After dimensionality reduction processing by the attention module, the first data matrix of the first dimension is obtained.

[0015] The first data matrix is ​​input into the attention module of the first classification model. After dimensionality reduction processing by the attention module, a second data matrix with a second dimension is obtained; the second dimension is smaller than the first dimension.

[0016] After performing convolution, normalization, activation, and pooling operations on the second data matrix in sequence, a third data matrix with a third dimension is obtained; the third dimension is smaller than the second dimension.

[0017] The first, second, and third data matrices are input into the fully connected layer of the first classification model to obtain the expression classification results.

[0018] In one embodiment, the first data matrix is ​​input into the attention module of the first classification model. After dimensionality reduction processing by the attention module, a second data matrix with a second dimension is obtained, including:

[0019] The first data matrix is ​​input into the attention module of the first classification model. The first data matrix is ​​then subjected to convolution, normalization and activation operations in sequence through the convolutional layer in the attention module to obtain the first convolution result.

[0020] After performing convolution, residual processing, and convolution operations sequentially on the first convolution result, the second convolution result is obtained.

[0021] Perform a convolution operation on the first convolution result to obtain the third convolution result;

[0022] The results of the second and third convolutions are concatenated to obtain a concatenated matrix;

[0023] The concatenated matrix is ​​then subjected to normalization, activation, and attention feature extraction processes in sequence to obtain the second data matrix of the second dimension.

[0024] In one embodiment, the learning effect is the actual classroom difficulty level corresponding to the image to be classified. The learning effect of the image to be classified is determined based on the similarity between the actual probability distribution value corresponding to each learning state and the preset probability distribution value, including:

[0025] Determine the estimated difficulty level of the classroom corresponding to the image to be classified;

[0026] Based on at least one prediction model, obtain the preset probability distribution value corresponding to the predicted difficulty level of the classroom;

[0027] Calculate the similarity between the actual probability distribution value and the preset probability distribution value corresponding to each learning state of the image to be classified;

[0028] If the similarity is less than the preset threshold, the estimated difficulty level of the classroom will be used as the actual difficulty level of the classroom corresponding to the image to be classified.

[0029] In one embodiment, the prediction model includes at least one prediction module, which, based on the at least one prediction model, obtains a preset probability distribution value corresponding to the predicted difficulty level of the classroom, including:

[0030] Obtain multiple prediction models that match the difficulty level of the classroom prediction, as well as multiple preset probability distribution values ​​corresponding to the multiple prediction models;

[0031] Calculate the clustering distance between the actual probability distribution value corresponding to each learning state of the image to be classified and the preset probability distribution values ​​of multiple prediction models. Based on the prediction model corresponding to the smallest clustering distance, determine the preset probability distribution value corresponding to the image to be classified.

[0032] In one embodiment, the method further includes:

[0033] If the similarity is greater than the preset threshold, the step of determining the estimated difficulty level of the class corresponding to the image to be classified is repeated until the similarity is less than the preset threshold.

[0034] In one embodiment, the method further includes:

[0035] Obtain training samples. Each training sample includes an expression sample, the first expression classification result of the labeled expression sample, a body sample, the first body classification result of the labeled body sample, and the first decision result corresponding to the labeled expression sample and body sample.

[0036] The facial expression sample is input into the first classification model to obtain the second facial expression classification result of the first classification model. Based on the difference between the second facial expression classification result and the first facial expression classification result, the parameters of the first classification model are adjusted.

[0037] The limb samples are input into the second classification model to obtain the second limb classification result of the second classification model. Based on the difference between the second limb classification result and the first limb classification result, the parameters of the second classification model are adjusted.

[0038] The second facial expression classification result and the second limb classification result are input into the decision-maker of the bimodal classification model to obtain the second decision result output by the decision-maker. Based on the difference between the second decision result and the first decision result, the parameters of the decision-maker are adjusted to complete one training cycle.

[0039] The training process is iterated multiple times. When the loss function meets the target value, the training stops, and the trained bimodal classification model is obtained.

[0040] Secondly, this application also provides an image classification device. The device includes:

[0041] The acquisition module is used to acquire facial expression images and body images of multiple target objects in the image to be classified;

[0042] The facial expression classification module is used to input facial expression images into the first classification model of a pre-trained bimodal classification model to obtain facial expression classification results;

[0043] The limb classification module is used to input limb images into the second classification model of the pre-trained bimodal classification model to obtain limb classification results;

[0044] The fusion decision module is used to fuse the facial expression classification results and the body classification results. It obtains the learning state of multiple target objects in the image to be classified and the actual probability distribution value corresponding to each learning state through the decision-maker of the bimodal classification model.

[0045] The determination module is used to determine the learning effect of the image to be classified based on the similarity between the actual probability distribution value corresponding to each learning state and the preset probability distribution value.

[0046] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0047] Obtain facial expression and body images of multiple target objects in the image to be classified;

[0048] The facial expression image is input into the first classification model of the pre-trained bimodal classification model to obtain the facial expression classification result;

[0049] The limb image is input into the second classification model of the pre-trained bimodal classification model to obtain the limb classification result;

[0050] The facial expression classification results and body classification results are fused together, and the decision-maker of the bimodal classification model is used to obtain the learning state of multiple target objects in the image to be classified, as well as the actual probability distribution value corresponding to each learning state.

[0051] The learning effect of the image to be classified is determined based on the similarity between the actual probability distribution value and the preset probability distribution value corresponding to each learning state.

[0052] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0053] Obtain facial expression and body images of multiple target objects in the image to be classified;

[0054] The facial expression image is input into the first classification model of the pre-trained bimodal classification model to obtain the facial expression classification result;

[0055] The limb image is input into the second classification model of the pre-trained bimodal classification model to obtain the limb classification result;

[0056] The facial expression classification results and body classification results are fused together, and the decision-maker of the bimodal classification model is used to obtain the learning state of multiple target objects in the image to be classified, as well as the actual probability distribution value corresponding to each learning state.

[0057] The learning effect of the image to be classified is determined based on the similarity between the actual probability distribution value and the preset probability distribution value corresponding to each learning state.

[0058] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0059] Obtain facial expression and body images of multiple target objects in the image to be classified;

[0060] The facial expression image is input into the first classification model of the pre-trained bimodal classification model to obtain the facial expression classification result;

[0061] The limb image is input into the second classification model of the pre-trained bimodal classification model to obtain the limb classification result;

[0062] The facial expression classification results and body classification results are fused together, and the decision-maker of the bimodal classification model is used to obtain the learning state of multiple target objects in the image to be classified, as well as the actual probability distribution value corresponding to each learning state.

[0063] The learning effect of the image to be classified is determined based on the similarity between the actual probability distribution value and the preset probability distribution value corresponding to each learning state.

[0064] The aforementioned image classification method, apparatus, computer equipment, and storage medium identify the facial expression and body classification results of multiple target objects in the image to be classified through a bimodal classification model. Based on the decision results of the facial expression and body classification results of the target objects, the learning state of the target objects is obtained. The emotional attitude of the target objects is identified from multiple dimensions, and the sensitivity of emotional attitude to the target objects is improved. Based on the similarity between the actual probability distribution value corresponding to each learning state and the preset probability distribution value, the learning effect corresponding to the image to be classified is determined, which can effectively reflect the students' real learning situation. Attached Figure Description

[0065] Figure 1 This is a diagram illustrating the application environment of an image classification method in one embodiment;

[0066] Figure 2 This is a flowchart illustrating an image classification method in one embodiment;

[0067] Figure 3 This is a schematic diagram of the structure of a bimodal classification model in one embodiment;

[0068] Figure 4 This is a flowchart illustrating the process of obtaining facial expression classification results in another embodiment;

[0069] Figure 5 This is a schematic diagram of the slicing operation in the first classification module of one embodiment;

[0070] Figure 6 This is a data processing flowchart for the first classification model in one embodiment;

[0071] Figure 7 This is a flowchart of the attention module processing in the first classification model in one embodiment;

[0072] Figure 8 This is a schematic diagram of the process for obtaining the second data matrix of the second dimension in one embodiment;

[0073] Figure 9 This is a flowchart illustrating the process of determining the learning effect of an image to be classified in one embodiment.

[0074] Figure 10 This is a schematic diagram of the process for obtaining the preset probability distribution value corresponding to the estimated difficulty level of the classroom in one embodiment;

[0075] Figure 11 This is a schematic diagram of the training process of a bimodal classification model in one embodiment;

[0076] Figure 12 This is a structural block diagram of an image classification device in one embodiment;

[0077] Figure 13 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0079] The image classification method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 acquires facial expression images and body images of multiple target objects in an image to be classified; the facial expression images are input into the first classification model of a pre-trained bimodal classification model to obtain facial expression classification results; the body images are input into the second classification model of a pre-trained bimodal classification model to obtain body classification results; the facial expression classification results and body classification results are fused, and the decision-maker of the bimodal classification model obtains the learning state of multiple target objects in the image to be classified, as well as the actual probability distribution value corresponding to each learning state; based on the similarity between the actual probability distribution value corresponding to each learning state and the preset probability distribution value, the learning effect corresponding to the image to be classified is determined. Terminal 102 communicates with server 104 via a network. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart TVs, smart in-vehicle devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster composed of multiple servers.

[0080] In one embodiment, such as Figure 2 As shown, an image classification method is provided, which can be applied to... Figure 1 Taking the terminal in the example, the explanation includes the following steps:

[0081] Step 202: Obtain facial expression images and body images of multiple target objects in the image to be classified.

[0082] The images to be classified refer to images containing students' facial expressions and upper limb movements during online or offline classes. The target object refers to the object to be identified in the images to be classified; for example, the target object could be a person in the images to be classified. Facial expression images include expressions such as raised eyebrows, furrowed brows, squinting eyes, tight corners of the mouth, upturned corners of the mouth, downturned corners of the mouth, raised cheeks, and dimples; body images include actions such as looking up, looking down, looking left and right, and lying down.

[0083] Optionally, the terminal can use a camera or the terminal's screenshot function to capture images taken by the same camera during online or offline learning, filter the images to remove those that do not include the target object's facial expressions and upper limb movements, adjust the size of the filtered images to a uniform size, and use the adjusted images as images to be classified.

[0084] Step 204: Input the facial expression image into the first classification model of the pre-trained bimodal classification model to obtain the facial expression classification result.

[0085] The bimodal classification model refers to a learning model capable of processing and understanding multimodal information. This multimodal information can be two different objects in an image, or it can be multimodal learning between images, audio, video, and semantics. The structure of the bimodal classification model in this embodiment is as follows: Figure 3 As shown, multimodal information refers to the facial expressions and upper limb movements of a target object in the same image.

[0086] The first classification model is used to identify the category of facial expressions in the image to be classified. The expression classification result is the emotion category mapped to the facial expressions in the image to be classified. For example, confusion is mainly manifested by lowered eyebrows and tense eyelids. If the classification probability threshold corresponding to the facial expression in the image to be classified exceeds a set value (initial value = 0.4), the emotion category mapped to the facial expression is considered to be confusion. Fuzzy inference algorithm is used to map the facial expressions classified in the facial expression image to six emotions: "focused", "joyful", "bored", "distracted", "confused", and "frustrated".

[0087] The first classification model employs a multi-object recognition model, utilizing a grid division method and an adaptive anchor box algorithm to determine the size of the target box used to identify facial expressions of the target objects in the image to be classified, and to identify facial expressions within target boxes of different sizes.

[0088] Optionally, the terminal inputs the facial expression image from the image to be classified into the first classification model of the pre-trained bimodal classification model. The first classification model recognizes the facial expression image to obtain the facial expression. Based on the mapping relationship between facial expression and emotion, the facial expression classification result of the facial expression image in the image to be classified is determined.

[0089] Step 206: Input the limb image into the second classification model of the pre-trained bimodal classification model to obtain the limb classification result.

[0090] The second classification model is used to identify the category of the limb images in the image to be classified. The limb classification result is the category to which the limb images in the image to be classified belong.

[0091] Optionally, the terminal inputs the limb image from the image to be classified into the second classification model of the pre-trained bimodal classification model. The second classification model recognizes the limb image and obtains the limb classification result.

[0092] Step 208: The facial expression classification results and body classification results are fused together. The decision-maker of the bimodal classification model is used to obtain the learning state of multiple target objects in the image to be classified, as well as the actual probability distribution value corresponding to each learning state.

[0093] In real-world scenarios, a learner's facial expressions can only be fully observed when the learner is in a "head-up" state. Furthermore, the learner's learning state can be easily determined when their body movements include looking down, glancing around, or lying down. Therefore, this embodiment only performs the final decision fusion for the "head-up" situation. The learning states of multiple target objects in the image to be classified, obtained through the decision-maker of the bimodal classification model, include: attentive listening, lack of interest, daydreaming, confusion, not understanding, drowsiness, fatigue, sleeping, whispering, and uncertainty. The decision-maker of the bimodal classification model can employ fuzzy inference algorithms or pattern recognition algorithms. Pattern recognition algorithms include methods such as support vectors, nearest neighbors, tree sets, and neural networks.

[0094] The actual probability distribution value corresponding to each learning state refers to the proportion of each learning state in the total number of target objects in the image to be classified. For example, if there are 100 target objects in the image to be classified, and 70 target objects are in the learning state of attentive listening, 10 target objects are in the learning state of not understanding, 5 target objects are in the learning state of being sleepy, and 15 target objects are in the learning state of uncertainty, then the actual probability distribution value of attentive listening is 70%, the actual probability distribution value of not understanding is 10%, the actual probability distribution value of being sleepy is 5%, and the actual probability distribution value of uncertainty is 15%.

[0095] Optionally, the terminal fuses the facial expression classification results and body classification results using a fuzzy inference algorithm or a pattern recognition algorithm, obtains the learning states of multiple target objects in the image to be classified through the decision-maker of the bimodal classification model, and determines the actual probability distribution value corresponding to each learning state based on the proportion of each learning state to the total number of target objects.

[0096] Step 210: Determine the learning effect of the image to be classified based on the similarity between the actual probability distribution value and the preset probability distribution value corresponding to each learning state.

[0097] The preset probability distribution value refers to the expected probability distribution value of the learning states of multiple target objects in the image to be classified. The more similar the actual probability distribution value corresponding to each learning state is to the preset probability distribution value, the better the current learning effect is; the less similar the actual probability distribution value corresponding to each learning state is to the preset probability distribution value, the worse the current learning effect is, and the teacher needs to adjust the teaching plan.

[0098] Learning effectiveness reflects the efficiency of teachers' instruction and the efficiency with which students absorb the course content. It can be assessed using parameters such as student satisfaction with the course content, their level of understanding, or the difficulty level of the content.

[0099] Optionally, the terminal determines the preset probability distribution value of the current teaching content, calculates the similarity between the actual probability distribution value corresponding to each learning state and the preset probability distribution value. If the similarity is less than the preset value, it indicates that the current learning effect is better; if the actual probability distribution value corresponding to each learning state is less similar to the preset probability distribution value, it indicates that the current learning effect is worse, and the teacher needs to adjust the teaching plan.

[0100] In the above image classification method, the facial expression classification result and body classification result of multiple target objects in the image to be classified are identified by a bimodal classification model. Based on the decision result of the facial expression classification result and body classification result of the target object, the learning state of the target object is obtained. The emotional attitude of the target object is identified from multiple dimensions, and the emotional sensitivity of the target object is improved. Based on the similarity between the actual probability distribution value corresponding to each learning state and the preset probability distribution value, the learning effect corresponding to the image to be classified is determined, which can effectively reflect the student's real learning situation.

[0101] In one embodiment, the structures of the first classification model and the second classification model can be the same or different. Since upper body posture recognition is relatively simple, a simple convolutional neural network model is directly used to intelligently identify four behaviors: "looking up," "looking down," "looking left and right," and "lying down." Therefore, this embodiment only describes the classification process of the first classification model. Specifically, as... Figure 4 As shown, the facial expression image is input into the first classification model of a pre-trained bimodal classification model to obtain the facial expression classification result, including:

[0102] Step 402: Input the facial expression image into the first classification model of the pre-trained bimodal classification model. The first classification model slices the facial expression image to obtain multiple feature maps.

[0103] The slicing process involves taking a value from every other pixel in an image, similar to neighbor-to-neighbor downsampling. This results in four complementary images that are similar in appearance without information loss. The stitched image, compared to the original RGB three-channel mode, becomes a 12-channel image. Finally, the resulting image undergoes a convolution operation to obtain a double-downsampled feature map without information loss. For example, ... Figure 5 As shown, a 4×4×3 image slice becomes a 2×2×12 feature map. For example, ... Figure 6 As shown, the original 608×608×3 image is sliced ​​to obtain four 304×304×12 feature maps. After a convolution operation with 32 kernels, it finally becomes a 304×304×32 feature map. The initial values ​​of the convolution kernels are random, for example, the preset kernel size is 3×3.

[0104] Optionally, the terminal inputs a 608×608×3 facial expression image into the first classification model of the pre-trained bimodal classification model. The facial expression is sliced ​​using the slicing structure of the first classification model to obtain four 304×304×12 feature maps. After a convolution operation with 32 convolution kernels, it finally becomes a 304×304×32 feature map.

[0105] Step 404: Input multiple feature maps into the attention module of the first classification model. After dimensionality reduction processing by the attention module, the first data matrix of the first dimension is obtained.

[0106] In the process of identifying feature maps using the first classification model, the large size of the feature matrix leads to a large computational load and a long training time for the first classification model. Therefore, to solve the above problems, this embodiment reduces the computational load of the first classification model through dimensionality reduction processing of the attention module, thereby improving the accuracy of classification and clustering algorithms and avoiding the curse of dimensionality.

[0107] Attention mechanism is a mechanism that focuses on local information. In image recognition, it increases the weight of the features of interest in an image to obtain important information. In this embodiment, attention mechanism target detection algorithm is added to the first classification model. Attention factor amplifies the feature values ​​of facial expressions (such as eyes, eyebrows and corners of mouth) and classifies the students' facial expressions by performing neural network image recognition.

[0108] Since the number of dimensionality reduction iterations is related to the size of the convolutional kernel and the stride of the attention module, the target dimensionality reduction amount is determined based on the difference between the dimension of the feature map and the first data matrix required for the first dimension; the target number of dimensionality reduction iterations is determined based on the convolutional kernel and stride of the attention module. For example, the convolutional kernel size of the attention module is typically 3×3, the stride is 2, the input facial expression image dimension is 608×608, and the first dimension is 76×76; therefore, the target number of dimensionality reduction iterations is 2. Figure 6 As shown, two attention modules are needed to perform two dimensionality reduction processes on the input feature map. The feature map with a dimension of 304×304 is reduced to a feature map with a dimension of 152×152, and then the feature map with a dimension of 152×152 is reduced to a feature map with a dimension of 76×76, thus obtaining the first data matrix with a dimension of 76×76.

[0109] Optionally, the terminal inputs multiple feature maps into the attention module of the first classification model, determines the target dimensionality reduction number based on the convolution kernel and stride size of the attention module, determines the number of attention modules in the first classification model based on the target dimensionality reduction number, and obtains the first data matrix of the first dimension after multiple dimensionality reduction processing of the feature maps through multiple attention modules.

[0110] Step 406: Input the first data matrix into the attention module of the first classification model. After dimensionality reduction processing by the attention module, a second data matrix with a second dimension is obtained; the second dimension is smaller than the first dimension.

[0111] In order to improve the recognition accuracy of the first classification model and avoid overfitting, this embodiment, based on the first data matrix of the first dimension, also obtains a second data matrix of the second dimension, thus recognizing the classification results of the feature maps from multiple dimensions. Figure 6 For example, assuming the second dimension is 38×38, an attention module is needed to perform a dimensionality reduction process, which can reduce the first data matrix with a dimension of 76×76 to the second data matrix with a dimension of 38×38.

[0112] Optionally, the terminal determines the target number of dimensionality reductions based on the dimensionality difference between the second dimension and the first dimension, determines the number of attention modules based on the target number of dimensionality reductions, inputs the first data matrix of the first dimension into the attention module of the first classification model, and obtains the second data matrix of the second dimension after dimensionality reduction processing by the attention module.

[0113] Step 408: After performing convolution, normalization, activation and pooling operations on the second data matrix in sequence, a third data matrix with a third dimension is obtained; the third dimension is smaller than the second dimension.

[0114] Performing convolution operations on the second data matrix sequentially can reduce dimensionality. The three feature map sizes typically used for prediction are 19×19×255, 38×38×255, and 76×76×255, where 255 represents the matrix thickness of the feature map corresponding to 80 categories. Figure 6 For example, the second data matrix of the second dimension is reduced to a third data matrix of 19×19.

[0115] Optionally, the terminal performs convolution, normalization, activation, and pooling operations on the second data matrix in sequence to obtain a third data matrix with a third dimension.

[0116] Step 410: Input the first data matrix, the second data matrix, and the third data matrix into the fully connected layer of the first classification model to obtain the expression classification result.

[0117] The fully connected layer connects the first, second, and third data matrices to obtain a concatenated matrix. Based on the classification results of the feature map identified by the concatenated matrix, the recognition accuracy of the first classification model is improved, and overfitting of the first classification model is avoided.

[0118] Optionally, the terminal inputs the first data matrix, the second data matrix, and the third data matrix of the first classification model into the fully connected layer of the first classification model to obtain a concatenated matrix, and recognizes the expression classification result mapped by the concatenated matrix.

[0119] In this embodiment, on the one hand, an attention module is introduced into the first classification model. The attention module amplifies the feature values ​​of the facial expression images in the images to be classified, thereby improving the feature extraction efficiency and facilitating the first classification model to recognize facial expressions in the images to be classified. On the other hand, the facial expression images are subjected to multiple dimensionality reduction processes to obtain first, second, and third data matrices of different dimensions. Based on the concatenation matrix of the first, second, and third data matrices, the facial expression classification result mapped by the concatenation matrix is ​​identified. This can reduce the computational load of the first classification model, identify the classification results of the feature maps from multiple dimensions, improve the recognition accuracy of the first classification model, and avoid overfitting of the first classification model.

[0120] In one embodiment, the normalization processing and activation functions of the convolutional computation units in the attention module are used frequently, but excessive normalization and activation are not necessary in practice, resulting in a large computational load for the first classification module. Therefore, to solve the above problem, this embodiment improves the structure of the attention module, and the processing flow of the attention module is as follows: Figure 7As shown, using multiple convolutional calculations followed by unified normalization and activation reduces the number of normalization and activation operations. Since the attention modules in this embodiment have the same structure, only the structure of one attention module in the first classification model will be described here. Specifically, as... Figure 8 As shown, the first data matrix is ​​input into the attention module of the first classification model. After dimensionality reduction processing by the attention module, a second data matrix with a second dimension is obtained, including:

[0121] Step 802: Input the first data matrix into the attention module of the first classification model. The first data matrix is ​​then subjected to convolution, normalization and activation operations by the convolutional layer in the attention module to obtain the first convolution result.

[0122] In this model, convolution, normalization, and activation operations are the smallest computational units, which can reduce dimensionality. After processing by these smallest computational units, the first data matrix is ​​reduced from the first dimension to the second dimension. The result of the first convolution is a data matrix with the second dimension.

[0123] Optionally, the terminal inputs the first data matrix into the attention module of the first classification model, and performs convolution, normalization and activation operations on the first data matrix in sequence through the convolutional layer in the attention module to obtain the first convolution result.

[0124] Step 804: Perform convolution operation, residual processing and convolution operation on the first convolution result in sequence to obtain the second convolution result.

[0125] Residual processing can alleviate the gradient explosion and vanishing problems caused by network deepening. Convolutional operations, residual processing, and convolutional operations are handled by the first convolutional layer, the residual component, and the second convolutional layer of the attention module, respectively. The first convolutional layer has a 3×3 kernel and a stride of 1, ensuring that the dimensionality of the first convolution result remains unchanged after processing. The second convolutional layer has a 3×3 kernel and a stride of 2, which helps to reduce dimensionality.

[0126] Optionally, the terminal inputs the first convolution result into the attention module, and the attention module performs convolution operation, residual processing and convolution operation on the first convolution result in sequence to obtain the second convolution result.

[0127] Step 806: Perform a convolution operation on the first convolution result to obtain the third convolution result.

[0128] The third convolution result serves to concatenate with the second convolution result. This concatenated structure allows for the reuse of a large amount of gradient information, which is beneficial for network learning. The third convolution result is obtained by performing a convolution operation on the first convolution result by the third convolutional layer of the attention module.

[0129] Optionally, the terminal inputs the first convolution result into the third convolutional layer of the attention module of the first classification model, and performs a convolution operation on the first convolution result through the third convolutional layer to obtain the third convolution result.

[0130] Step 808: Concatenate the second and third convolution results to obtain a concatenated matrix.

[0131] The cascaded structure allows for the reuse of a large amount of gradient information, which is beneficial for network learning. In this embodiment, the second and third convolutional results are cascaded through a fully connected layer in the attention module.

[0132] Optionally, the terminal inputs the second convolution result and the third convolution result into the fully connected layer of the attention module of the first classification module, and concatenates the second convolution result and the third convolution result through the fully connected layer to obtain a concatenated matrix.

[0133] Step 810: Normalize, activate, and extract attention features from the concatenated matrix in sequence to obtain the second data matrix of the second dimension.

[0134] In this embodiment, the first convolution result is processed through three convolutional layers before normalization and activation. Using multiple convolutional calculations followed by unified normalization and activation reduces the number of normalization and activation operations. Activation can be performed using an activation function, such as the LeakyReLU activation function or the Mish activation function.

[0135] Attention feature extraction can be processed using attention factors. For example, the SEnet attention mechanism can be used to amplify the feature values ​​of facial expressions (such as eyes, eyebrows, and corners of the mouth).

[0136] In this embodiment, after performing a convolution operation on the first convolution result, residual processing is performed immediately instead of normalization and activation operations. The normalization and activation operations are placed at the end of the attention module structure. By performing unified normalization and activation after multiple convolution calculations, the number of normalization and activation operations can be reduced.

[0137] In one embodiment, if the learning outcome corresponds to the actual classroom difficulty level of the image to be classified, then as follows: Figure 9 As shown, the learning effect of the image to be classified is determined based on the similarity between the actual probability distribution value corresponding to each learning state and the preset probability distribution value, including:

[0138] Step 902: Determine the estimated difficulty level of the classroom corresponding to the image to be classified.

[0139] The estimated difficulty level in class can be the difficulty level estimated by the teacher based on the teaching content, or the difficulty level determined by the terminal based on the difficulty coefficient of the knowledge points in the teaching content.

[0140] Step 904: Based on at least one prediction model, obtain the preset probability distribution value corresponding to the predicted difficulty level of the classroom.

[0141] Each estimated difficulty level in the classroom corresponds to a pre-defined overall probability distribution. For example, if the estimated difficulty level is set to 10 levels, with level 1 being the easiest and level 10 being the most difficult, then the corresponding pre-defined overall probability distribution could be: 22% of the sampled points are attentive, 46% are confused / did not understand, 23% are sleepy / tired / sleep, 5% are uninterested / daydreaming, 1% are whispering, and 3% are uncertain.

[0142] The preset probability distribution value can be determined based on the prediction value of one prediction model or based on the prediction values ​​of multiple prediction models. If the preset probability distribution value is determined based on the prediction values ​​of multiple prediction models, then the prediction value corresponding to the smallest clustering distance between the prediction values ​​of multiple prediction models and the actual probability distribution value is selected as the final preset probability distribution value.

[0143] Different difficulty levels are used as training samples to classify images. Difficulty levels are labeled on the training samples. The prediction model recognizes the facial expressions of the target objects on the samples. Based on the facial expressions of the target objects, the learning state corresponding to the facial expressions is determined, and the probability distribution value corresponding to the learning state is calculated. The difficulty level corresponding to the training sample is output based on the probability distribution value. If the error value between the difficulty level and the difficulty level labeled on the sample is less than a preset value, one training cycle is completed. If the error value between the difficulty level and the difficulty level labeled on the sample is greater than the preset value, the parameters of the prediction model are adjusted, and the training process is iterated multiple times until the number of iterations meets the preset number and the difficulty level output by the prediction model is the same as the difficulty level labeled on the sample. At this point, training stops, and the trained prediction model is obtained.

[0144] Optionally, the terminal determines the classroom prediction difficulty level corresponding to the image to be classified, and obtains the preset probability distribution value corresponding to the classroom prediction difficulty level through at least one prediction model.

[0145] Step 906: Calculate the similarity between the actual probability distribution value and the preset probability distribution value corresponding to each learning state of the image to be classified.

[0146] The similarity between the actual probability distribution value and the preset probability distribution value can be determined by calculating the distance between them. For example, the k-means clustering algorithm can be used to calculate the clustering distance between the actual and preset probability distribution values. The closer the clustering distance is to 0, the more similar the actual and preset probability distribution values ​​are, and the closer the actual difficulty level of the image to be classified in the classroom is to the estimated difficulty level. In other words, the learning effect of the image to be classified is closer to the actual situation.

[0147] Optionally, the terminal uses a clustering algorithm to calculate the similarity between the actual probability distribution value and the preset probability distribution value corresponding to each learning state of the image to be classified.

[0148] Step 908: If the similarity is less than the preset threshold, the estimated difficulty level of the classroom is taken as the actual difficulty level of the classroom corresponding to the image to be classified.

[0149] If the similarity is less than a preset threshold, it means that the actual difficulty level of the classroom is closer to the estimated difficulty level. For example, if the estimated difficulty level of the classroom is level 10, and the similarity is less than the preset threshold, then the estimated difficulty level of the classroom, i.e., level 10, will be taken as the actual difficulty level of the classroom corresponding to the image to be classified.

[0150] In some embodiments, if the similarity is greater than a preset threshold, the step of determining the estimated difficulty level of the classroom corresponding to the image to be classified is repeated until the similarity is less than the preset threshold.

[0151] If the similarity exceeds a preset threshold, it indicates a significant discrepancy between the actual and estimated difficulty levels of the lesson, suggesting low student engagement with the current content and requiring teacher adjustments. In this embodiment, to assess the learning outcomes of the image to be classified, the estimated difficulty level of the image needs to be redefined. The actual difficulty level is determined based on the similarity between the actual probability distribution value and the preset probability distribution value for each learning state of the image.

[0152] Optionally, the terminal determines the learning effect of the image to be classified based on the relationship between the actual probability distribution value corresponding to each learning state of the image to be classified and the preset probability distribution value and the preset threshold. If the similarity is less than the preset threshold, the estimated classroom difficulty level is taken as the actual classroom difficulty level of the image to be classified. If the similarity is greater than the preset threshold, the step of determining the estimated classroom difficulty level of the image to be classified is repeated until the similarity is less than the preset threshold.

[0153] In this embodiment, based on the relationship between the similarity between the actual probability distribution value and the preset probability distribution value of each learning state of the image to be classified and the preset threshold, the actual classroom difficulty level of the image to be classified is determined, and a mapping relationship between the learning state and the actual classroom difficulty level is established. Based on the mapping relationship, the learning effect of the image to be classified can be evaluated from multiple dimensions of the learning platform and the actual classroom difficulty level, so that the learning effect can effectively reflect the student's real learning situation.

[0154] In one embodiment, the prediction model includes at least one prediction module, that is, obtaining a preset probability distribution value corresponding to the difficulty level of classroom prediction through multiple prediction models. Then, based on at least one prediction model, the preset probability distribution value corresponding to the difficulty level of classroom prediction is obtained, such as... Figure 10 As shown, it includes the following steps:

[0155] Step 1002: Obtain multiple prediction models that match the predicted difficulty level of the class, as well as multiple preset probability distribution values ​​corresponding to the multiple prediction models.

[0156] In this embodiment, each predicted difficulty level of a class corresponds to at least one prediction model, which outputs a preset probability distribution value of the learning state under the corresponding predicted difficulty level. The preset probability distribution value can be determined based on the prediction value of one prediction model or on the prediction values ​​of multiple prediction models. This embodiment uses the prediction values ​​of multiple prediction models to determine the preset probability distribution value matching the predicted difficulty level of the class. Each prediction model can be trained using different models, and the preset probability distribution value output by each prediction model is different.

[0157] Optionally, the terminal obtains multiple prediction models for predicting the learning status under the current predicted difficulty level of the class, based on the predicted difficulty level of the class, and obtains the preset probability distribution values ​​corresponding to the multiple prediction models.

[0158] Step 1004: Calculate the clustering distance between the actual probability distribution value corresponding to each learning state of the image to be classified and the preset probability distribution value of multiple prediction models. Based on the prediction model corresponding to the smallest clustering distance, determine the preset probability distribution value corresponding to the image to be classified.

[0159] If the preset probability distribution value is determined based on the predicted values ​​of multiple prediction models, then the predicted value corresponding to the smallest clustering distance between the predicted values ​​of multiple prediction models and the actual probability distribution value is selected as the final preset probability distribution value.

[0160] The clustering distance between the actual probability distribution value and the preset probability distribution values ​​of multiple prediction models can be calculated using the following algorithm:

[0161]

[0162] Among them, a i Indicates weight; x i y represents the actual probability distribution value; i denoted by ; k represents the number of learning states corresponding to the image to be classified; H(z) represents the clustering distance.

[0163] For example, the actual probability distribution of the learning state corresponding to the image to be classified is 27% attentive, 41% confused / did not understand, 15% sleepy / fatigued / sleeping, 6% uninterested / distracted, 1% whispering, and 11% uncertain. The preset probability distribution of the learning state corresponding to the image to be classified is 22% attentive, 46% confused / did not understand, 23% sleepy / fatigued / sleeping, 5% uninterested / distracted, 1% whispering, and 3% uncertain. If the weight is set to 1, the clustering distance between the actual probability distribution and the preset probability distribution is:

[0164]

[0165] Using the same principle, the clustering distance between the actual probability distribution value corresponding to each learning state of the image to be classified and the preset probability distribution values ​​of multiple prediction models is calculated, and the predicted value corresponding to the smallest clustering distance is selected as the final preset probability distribution value.

[0166] Optionally, the terminal calculates the clustering distance between the actual probability distribution value corresponding to each learning state of the image to be classified and the preset probability distribution values ​​of multiple prediction models, and selects the predicted value corresponding to the smallest clustering distance as the final preset probability distribution value corresponding to the image to be classified.

[0167] In this embodiment, based on the clustering distance between the predicted values ​​of multiple prediction models and the actual probability distribution values, the predicted value corresponding to the smallest clustering distance is selected as the final preset probability distribution value. By selecting the classroom prediction difficulty level that is closest to the actual classroom difficulty level through the above method, the calculation accuracy of the similarity between the actual probability distribution value and the preset probability distribution value corresponding to each learning state of the image to be classified can be improved, so that the determined actual classroom difficulty level can effectively reflect the students' real learning situation.

[0168] In one instance, such as Figure 11 As shown, the training process of the bimodal classification model includes the following steps:

[0169] Step 1102: Obtain training samples. Each training sample includes an expression sample, the first expression classification result of the labeled expression sample, a body sample, the first body classification result of the labeled body sample, and the first decision result corresponding to the labeled expression sample and body sample.

[0170] The training samples can be images of teachers during lectures, including students' facial expressions and upper limb movements.

[0171] Optionally, the terminal uses image regions in the training samples that include students' facial expressions as expression samples, and labels the first expression classification result of the expression samples in the training samples. It also uses image regions in the training samples that include students' upper limb movements as upper limb samples, and labels the first limb classification result of the limb samples in the training samples. Based on the first expression classification result and the first limb classification result labeled in the training samples, the terminal determines the first decision result of the current training sample.

[0172] Step 1104: Input the facial expression sample into the first classification model, obtain the second facial expression classification result of the first classification model, and adjust the parameters of the first classification model according to the difference between the second facial expression classification result and the first facial expression classification result.

[0173] The first classification model categorizes facial expressions into six types: focused, joyful, annoyed, distracted, confused, and frustrated. The second expression classification result includes at least one of these six expressions. To avoid overfitting in the first classification model, this embodiment randomly removes 10% of the expression samples.

[0174] The difference between the second expression classification result and the first expression classification result is determined by a loss function. The weights of the first classification model are updated using the loss function. Each node of the first classification model is assigned a random weight and a bias value. Based on the single iteration output of the first classification model, the error value between the single iteration output and the input annotation information is determined. The error value and the gradient of the cost function are fed back to the first classification model to update its weights, thereby reducing the error value in subsequent iterations.

[0175] Optionally, the terminal inputs the expression sample into the first classification model, obtains the second expression classification result of the first classification model, determines the error value between the second expression classification result and the first expression classification result according to the loss function, and feeds back the error value and the gradient of the cost function to the first classification model to update the weights of the first classification model.

[0176] Step 1106: Input the limb sample into the second classification model, obtain the second limb classification result of the second classification model, and adjust the parameters of the second classification model according to the difference between the second limb classification result and the first limb classification result.

[0177] The second classification model categorizes upper limb movements into four types: head tilting, head tilting, looking left and right, and lying down. In other words, the second facial expression classification result includes at least one of these four movements. To avoid overfitting in the second classification model, this embodiment randomly removes 10% of the limb samples.

[0178] The training process for the second classification model is the same as that for the first classification model, so it will not be repeated here.

[0179] Step 1108: Input the second facial expression classification result and the second limb classification result into the decision-maker of the bimodal classification model, obtain the second decision result output by the decision-maker, and adjust the parameters of the decision-maker according to the difference between the second decision result and the first decision result to complete one training.

[0180] The training process for the decision-maker is the same as that for the first classification model, so it will not be repeated here.

[0181] Step 1110: Iterate the training process multiple times. When the loss function meets the target value, stop training to obtain the trained bimodal classification model.

[0182] Optionally, the training process can be iterated multiple times until the loss functions of the first classification model, the second classification model, and the decision maker all meet the target value. At this point, the training ends, and a trained bimodal classification model can be obtained.

[0183] In this embodiment, the first classification model, the second classification model, and the decision maker form a residual network structure. During training, since the bimodal classification model is a residual network structure, the training efficiency and feature extraction accuracy of the network are improved.

[0184] In one embodiment, an image classification method is provided, specifically including the following steps:

[0185] Step 1: Obtain facial expression images and body images of multiple target objects in the image to be classified.

[0186] Step 2: Input the facial expression image into the first classification model of the pre-trained bimodal classification model. The first classification model performs slice processing on the facial expression image to obtain multiple feature maps.

[0187] Step 3: Input multiple feature maps into the attention module of the first classification model. After dimensionality reduction processing by the attention module, the first data matrix of the first dimension is obtained.

[0188] Step 4: Input the first data matrix into the attention module of the first classification model. The first data matrix is ​​then subjected to convolution, normalization and activation operations by the convolutional layer in the attention module to obtain the first convolution result.

[0189] Step 5: Perform convolution, residual processing, and convolution operations sequentially on the first convolution result to obtain the second convolution result.

[0190] Step 6: Perform a convolution operation on the first convolution result to obtain the third convolution result.

[0191] Step 7: Concatenate the second and third convolution results to obtain a concatenated matrix.

[0192] Step 8: Perform normalization, activation and attention feature extraction on the concatenated matrix in sequence to obtain the second data matrix of the second dimension.

[0193] Step 9: Perform convolution, normalization, activation and pooling operations on the second data matrix in sequence to obtain the third data matrix with a third dimension; the third dimension is smaller than the second dimension.

[0194] Step 10: Input the first data matrix, the second data matrix, and the third data matrix into the fully connected layer of the first classification model to obtain the expression classification result.

[0195] Step 11: Input the limb image into the second classification model of the pre-trained bimodal classification model to obtain the limb classification result.

[0196] Step 12: The facial expression classification results and body classification results are fused together. The decision-maker of the bimodal classification model is used to obtain the learning state of multiple target objects in the image to be classified, as well as the actual probability distribution value corresponding to each learning state.

[0197] Step 13: Determine the estimated difficulty level of the classroom corresponding to the image to be classified.

[0198] Step 14: Obtain multiple prediction models that match the predicted difficulty level of the class, as well as multiple preset probability distribution values ​​corresponding to the multiple prediction models.

[0199] Step 15: Calculate the clustering distance between the actual probability distribution value corresponding to each learning state of the image to be classified and the preset probability distribution values ​​of the multiple prediction models. Based on the prediction model corresponding to the smallest clustering distance, determine the preset probability distribution value corresponding to the image to be classified.

[0200] Step 16: Calculate the similarity between the actual probability distribution value and the preset probability distribution value corresponding to each learning state of the image to be classified.

[0201] Step 17: If the similarity is less than the preset threshold, the estimated difficulty level of the classroom is taken as the actual difficulty level of the classroom corresponding to the image to be classified.

[0202] Step 18: If the similarity is greater than the preset threshold, the step of determining the estimated difficulty level of the classroom corresponding to the image to be classified is repeated until the similarity is less than the preset threshold.

[0203] In this embodiment, the facial expressions and body movements of the target objects in the image to be classified are used to fuse the actual classroom difficulty level corresponding to the image to be classified, and the computational load of the bimodal classification model is reduced without reducing the recognition accuracy of the bimodal classification model.

[0204] In some embodiments, a typical online education platform includes a permission management module, a facial recognition authentication module, a virtual background synthesis module, and a classroom evaluation module. This embodiment provides a scenario where an image classification method is applied to the classroom evaluation module of an online education platform. The functions of each module of the online education platform are as follows:

[0205] The access control module is used to create users, who are divided into three roles: teachers, students, and administrators. Teachers have the ability to create, delete, modify, and query their own classes, and to add, delete, modify, and query students for the classes they manage. Students have course selection permissions. Administrators have the highest level of system privileges and can have full access to both classes and students.

[0206] The facial recognition authentication module provides a baseline for facial recognition matching in identity authentication based on recent facial photos uploaded by students. This serves as a service for attendance tracking, preventing issues such as proxy attendance and automatically implementing the roll call function.

[0207] The virtual background compositing module is divided into four sub-modules: image collection, image processing, image compositing, and graphic background rendering. The functions of each sub-module are as follows:

[0208] The image acquisition submodule updates and collects images captured by the camera every certain period of time (e.g., every 30 seconds or 60 seconds), and processes and merges them according to the classroom seating chart, which can effectively reduce the amount of data processed in the images.

[0209] The image processing submodule addresses the issue of inconsistent image composition. Since different camera distances and student sizes result in varying head sizes for different students, processing is required. The image processing manager employs two methods: first, it preprocesses the images from students' cameras within a specified distance range; second, it enlarges or reduces the size of images acquired within this range to ensure facial sizes are roughly similar, thus preventing inconsistencies in the composite image.

[0210] The image compositing submodule merges images collected by students' cameras according to a specific strategy and then adds a virtual background. Teachers can view the merged group photos in real-time, processed according to the classroom seating chart or other strategies, and can switch to split-screen mode by clicking a tab. The image merging strategy can be based on the uploaded classroom seating chart, or by sorting by student ID, automatic generation, or other methods. The strategy is implemented through a strategy manager. For faces where no image is recognized, blank images are stitched together, allowing for immediate identification of absent students.

[0211] The image background presentation submodule is used to present the virtual background synthesized by the graphics compositing submodule. It can also switch to split-screen mode, displaying the unprocessed camera image. Clicking on a student in the split-screen page can cause page shaking or pop-ups, providing timely reminders to students who are not paying attention and improving learning interactivity. An alarm is triggered on the teacher's side if a student's image fails to appear in the image multiple times consecutively. This can be done through image recognition. Virtual backgrounds include templates and can also be uploaded. For example, desks and decorations can be added.

[0212] The classroom assessment module performs quasi-random sampling of student camera images processed by the image processing submodule in the virtual background synthesis module. Images from the same camera are acquired for the same course and labeled with the same ID. Two copies of each image are processed. Channel 1 is used for background synthesis, and Channel 2 is used for image recognition in the course assessment. Channel 2 places images with the same ID into an image processing filter for filtering. Each student's sampled camera image must include facial expressions; images without expressions are discarded. 10% of the remaining images are randomly discarded. Preprocessing, including size reduction, results in a training set of 608*608 pixels, which is then stored. This facilitates subsequent emotion classification in image recognition and is used to identify reference points for each student's face. For example, popular facial recognition datasets use 68 sampling points. Triangulation is performed to obtain geometric features, and a multi-target recognition algorithm is used to recognize facial expressions and upper limb movements.

[0213] In this embodiment, the system intelligently identifies the facial expressions and upper limb postures of students in the camera's image. The dual-modal modeling makes decisions to identify students' emotions on the one hand, and uses intelligent algorithms to score and objectively evaluate the course on the other.

[0214] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0215] Based on the same inventive concept, this application also provides an image classification apparatus for implementing the image classification method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more image classification apparatus embodiments provided below can be found in the limitations of the image classification method described above, and will not be repeated here.

[0216] In one embodiment, such as Figure 12 As shown, an image classification device is provided, comprising:

[0217] The acquisition module 100 is used to acquire facial expression images and body images of multiple target objects in the image to be classified.

[0218] The expression classification module 200 is used to input expression images into the first classification model of a pre-trained bimodal classification model to obtain expression classification results;

[0219] The limb classification module 300 is used to input limb images into the second classification model of the pre-trained bimodal classification model to obtain limb classification results;

[0220] The fusion decision module 400 is used to fuse the facial expression classification results and the body classification results. It obtains the learning state of multiple target objects in the image to be classified and the actual probability distribution value corresponding to each learning state through the decision-maker of the bimodal classification model.

[0221] The determination module 500 is used to determine the learning effect of the image to be classified based on the similarity between the actual probability distribution value and the preset probability distribution value corresponding to each learning state.

[0222] In one embodiment, the expression classification module 200 is further configured to: input the expression image into the first classification model of the pre-trained bimodal classification model, and perform slicing processing on the expression image through the first classification model to obtain multiple feature maps;

[0223] Multiple feature maps are input into the attention module of the first classification model. After dimensionality reduction processing by the attention module, the first data matrix of the first dimension is obtained.

[0224] The first data matrix is ​​input into the attention module of the first classification model. After dimensionality reduction processing by the attention module, a second data matrix with a second dimension is obtained; the second dimension is smaller than the first dimension.

[0225] After performing convolution, normalization, activation, and pooling operations on the second data matrix in sequence, a third data matrix with a third dimension is obtained; the third dimension is smaller than the second dimension.

[0226] The first, second, and third data matrices are input into the fully connected layer of the first classification model to obtain the expression classification results.

[0227] In one embodiment, the expression classification module 200 is further configured to: input the first data matrix into the attention module of the first classification model, and perform convolution, normalization and activation operations on the first data matrix sequentially through the convolutional layer in the attention module to obtain the first convolution result;

[0228] After performing convolution, residual processing, and convolution operations sequentially on the first convolution result, the second convolution result is obtained.

[0229] Perform a convolution operation on the first convolution result to obtain the third convolution result;

[0230] The results of the second and third convolutions are concatenated to obtain a concatenated matrix;

[0231] The concatenated matrix is ​​then subjected to normalization, activation, and attention feature extraction processes in sequence to obtain the second data matrix of the second dimension.

[0232] In one embodiment, the learning effect is the actual classroom difficulty level corresponding to the image to be classified, and the determining module 500 is further used to: determine the estimated classroom difficulty level corresponding to the image to be classified.

[0233] Based on at least one prediction model, obtain the preset probability distribution value corresponding to the predicted difficulty level of the classroom;

[0234] Calculate the similarity between the actual probability distribution value and the preset probability distribution value corresponding to each learning state of the image to be classified;

[0235] If the similarity is less than the preset threshold, the estimated difficulty level of the classroom will be used as the actual difficulty level of the classroom corresponding to the image to be classified.

[0236] In one embodiment, the prediction model includes at least one prediction module, and the determination module 500 is further configured to:

[0237] Obtain multiple prediction models that match the difficulty level of the classroom prediction, as well as multiple preset probability distribution values ​​corresponding to the multiple prediction models;

[0238] Calculate the clustering distance between the actual probability distribution value corresponding to each learning state of the image to be classified and the preset probability distribution values ​​of multiple prediction models. Based on the prediction model corresponding to the smallest clustering distance, determine the preset probability distribution value corresponding to the image to be classified.

[0239] In one embodiment, the determining module 500 is further configured to: if the similarity is greater than a preset threshold, re-execute the step of determining the estimated difficulty level of the classroom corresponding to the image to be classified until the similarity is less than the preset threshold.

[0240] In one embodiment, the device further includes a training module 600, which is used to acquire training samples. Each training sample includes an expression sample, a first expression classification result labeled with the expression sample, a body sample, a first body classification result labeled with the body sample, and a first decision result corresponding to the labeled expression sample and body sample.

[0241] The facial expression sample is input into the first classification model to obtain the second facial expression classification result of the first classification model. Based on the difference between the second facial expression classification result and the first facial expression classification result, the parameters of the first classification model are adjusted.

[0242] The limb samples are input into the second classification model to obtain the second limb classification result of the second classification model. Based on the difference between the second limb classification result and the first limb classification result, the parameters of the second classification model are adjusted.

[0243] The second facial expression classification result and the second limb classification result are input into the decision-maker of the bimodal classification model to obtain the second decision result output by the decision-maker. Based on the difference between the second decision result and the first decision result, the parameters of the decision-maker are adjusted to complete one training cycle.

[0244] The training process is iterated multiple times. When the loss function meets the target value, the training stops, and the trained bimodal classification model is obtained.

[0245] Each module in the aforementioned image classification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0246] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 13 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements an image classification method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0247] Those skilled in the art will understand that Figure 13 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0248] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0249] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0250] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0251] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0252] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0253] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0254] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image classification method, characterized in that, The method includes: Obtain facial expression and body images of multiple target objects in the image to be classified; The facial expression image is input into the first classification model of the pre-trained bimodal classification model to obtain the facial expression classification result; The limb image is input into the second classification model of the pre-trained bimodal classification model to obtain the limb classification result; The facial expression classification result and the body classification result are fused together, and the decision-maker of the bimodal classification model is used to obtain the learning state of multiple target objects in the image to be classified, as well as the actual probability distribution value corresponding to each learning state. The learning effect of the image to be classified is determined based on the clustering distance between the actual probability distribution value and the preset probability distribution value corresponding to each learning state. The learning effect is the actual classroom difficulty level corresponding to the image to be classified. Determining the learning effect corresponding to the image to be classified based on the clustering distance between the actual probability distribution value and the preset probability distribution value corresponding to each learning state includes: Determine the estimated classroom difficulty level corresponding to the image to be classified; Based on at least one prediction model, obtain the preset probability distribution value corresponding to the predicted difficulty level of the classroom; Calculate the clustering distance between the actual probability distribution value and the preset probability distribution value corresponding to each learning state of the image to be classified; If the clustering distance is less than a preset threshold, the estimated classroom difficulty level is taken as the actual classroom difficulty level corresponding to the image to be classified.

2. The method according to claim 1, characterized in that, The step of inputting the facial expression image into the first classification model of a pre-trained bimodal classification model to obtain the facial expression classification result includes: The facial expression image is input into the first classification model of a pre-trained bimodal classification model. The first classification model then slices the facial expression image to obtain multiple feature maps. Multiple feature maps are input into the attention module of the first classification model. After dimensionality reduction processing by the attention module, a first data matrix with a first dimension is obtained. The first data matrix is ​​input into the attention module of the first classification model. After dimensionality reduction processing by the attention module, a second data matrix with a second dimension is obtained; the second dimension is smaller than the first dimension. After performing convolution, normalization, activation, and pooling operations sequentially on the second data matrix, a third data matrix with a third dimension is obtained; the third dimension is smaller than the second dimension. The first data matrix, the second data matrix, and the third data matrix are respectively input into the fully connected layer of the first classification model to obtain the expression classification result.

3. The method according to claim 2, characterized in that, The step of inputting the first data matrix into the attention module of the first classification model, and obtaining a second data matrix with a second dimension after dimensionality reduction processing by the attention module, includes: The first data matrix is ​​input into the attention module of the first classification model. The first data matrix is ​​then subjected to convolution, normalization and activation operations in sequence through the convolutional layer in the attention module to obtain the first convolution result. After performing convolution, residual processing, and convolution operations sequentially on the first convolution result, the second convolution result is obtained. Perform a convolution operation on the first convolution result to obtain the third convolution result; The second convolution result and the third convolution result are concatenated to obtain a concatenated matrix; The cascaded matrix is ​​then subjected to normalization, activation, and attention feature extraction processes in sequence to obtain the second data matrix of the second dimension.

4. The method according to claim 1, characterized in that, The prediction model includes at least one prediction module. The step of obtaining the preset probability distribution value corresponding to the predicted difficulty level of the classroom based on the at least one prediction model includes: Obtain multiple prediction models that match the predicted difficulty level of the class, and multiple preset probability distribution values ​​corresponding to the multiple prediction models; Calculate the clustering distance between the actual probability distribution value corresponding to each learning state of the image to be classified and the preset probability distribution values ​​of multiple prediction models. Based on the prediction model corresponding to the smallest clustering distance, determine the preset probability distribution value corresponding to the image to be classified.

5. The method according to claim 1, characterized in that, The method further includes: If the clustering distance is greater than a preset threshold, the step of determining the estimated difficulty level of the classroom corresponding to the image to be classified is repeated until the clustering distance is less than the preset threshold.

6. The method according to claim 1, characterized in that, The method further includes: Acquire training samples, each training sample including an expression sample, a first expression classification result annotating the expression sample, a body sample, a first body classification result annotating the body sample, and a first decision result annotating the expression sample and the body sample; The facial expression sample is input into the first classification model to obtain the second facial expression classification result of the first classification model. Based on the difference between the second facial expression classification result and the first facial expression classification result, the parameters of the first classification model are adjusted. The limb sample is input into the second classification model to obtain the second limb classification result of the second classification model. Based on the difference between the second limb classification result and the first limb classification result, the parameters of the second classification model are adjusted. The second facial expression classification result and the second limb classification result are input into the decision-maker of the bimodal classification model to obtain the second decision result output by the decision-maker. Based on the difference between the second decision result and the first decision result, the parameters of the decision-maker are adjusted to complete one training cycle. The training process is iterated multiple times. When the loss function meets the target value, the training stops, and the trained bimodal classification model is obtained.

7. An image classification device, characterized in that, The device includes: The acquisition module is used to acquire facial expression images and body images of multiple target objects in the image to be classified; The facial expression classification module is used to input the facial expression image into the first classification model of the pre-trained bimodal classification model to obtain the facial expression classification result; The limb classification module is used to input the limb image into the second classification model of the pre-trained bimodal classification model to obtain the limb classification result; The fusion decision module is used to fuse the expression classification result and the body classification result, and obtain the learning state of multiple target objects in the image to be classified, as well as the actual probability distribution value corresponding to each learning state, through the decision-maker of the bimodal classification model. The determination module is used to determine the learning effect of the image to be classified based on the clustering distance between the actual probability distribution value and the preset probability distribution value corresponding to each learning state. The learning effect is the actual classroom difficulty level corresponding to the image to be classified. The determining module is also used to determine the estimated difficulty level of the classroom corresponding to the image to be classified; Based on at least one prediction model, obtain the preset probability distribution value corresponding to the predicted difficulty level of the classroom; Calculate the clustering distance between the actual probability distribution value and the preset probability distribution value corresponding to each learning state of the image to be classified; If the clustering distance is less than a preset threshold, the estimated classroom difficulty level is taken as the actual classroom difficulty level corresponding to the image to be classified.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images