Classroom behavior identification method and device, computer equipment and storage medium

Abstract

The embodiment of the invention discloses a classroom behavior identification method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining classroom video and audio data collected by a plurality of video and audio collection devices in a teaching process, the classroom video and audio data at least comprising each classroom image collected by each video and audio collection device; the classroom image is input into multiple pre-trained artificial intelligence models for behavior prediction, multiple first prediction results of each person in the classroom image are obtained, and the prediction results are used for reflecting classroom behavior types of the persons; determining imaging distortion characteristics of the video and audio acquisition device for each person; and performing prediction result optimization processing by using the imaging distortion features and the plurality of first prediction results to obtain a target prediction result of each person. Through the above mode, not only are the respective model advantages of various pre-trained artificial intelligence models integrated, but also the influence of imaging distortion on the prediction result is considered, so that the accuracy of the obtained target prediction result is higher.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for identifying classroom behavior, computer equipment and storage medium. Background Art

[0002] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Artificial Intelligence is a very broad science, including robotics, speech recognition, image recognition, natural language processing, expert systems, machine learning, computer vision, etc.

[0003] The application of AI in education is becoming more and more common, especially in classroom behavior management. The combination of AI and classroom can bring many benefits, such as real-time feedback: AI systems can monitor students' behavior and engagement in real time and provide teachers with instant feedback, which enables teachers to adjust teaching strategies in time to better attract students' attention and improve classroom management efficiency.

[0004] Although the application of AI in predicting classroom behavior has great potential, it also has some drawbacks and challenges. Classroom teaching behavior is very complex, including speech, body movements, facial expressions and other factors. AI systems need to be able to accurately identify and classify these behaviors, which is a technical challenge; and AI systems may produce unfair behavior predictions due to biases in input data or algorithm design, which may lead to unfair treatment of some students. Therefore, the accuracy of AI in predicting classroom behavior still needs to be improved. Summary of the invention

[0005] The main purpose of the present invention is to provide a method and device for identifying classroom behavior, a computer device and a storage medium, which can solve the problem in the prior art that the accuracy of AI in predicting classroom behavior needs to be improved.

[0006] To achieve the above object, the present invention provides a method for identifying classroom behavior in a first aspect, the method comprising:

[0007] Acquire classroom audio and video data collected by multiple audio and video collection devices during the teaching process, wherein the classroom audio and video data at least includes each classroom image collected by each audio and video collection device;

[0008] Inputting the classroom image into a plurality of pre-trained artificial intelligence models for behavior prediction, respectively, to obtain a plurality of first prediction results for each character in the classroom image, wherein the prediction results are used to reflect the classroom behavior type of the character;

[0009] Determining imaging distortion characteristics of each person by the video and audio acquisition device;

[0010] The imaging distortion feature and the plurality of first prediction results are used to perform prediction result optimization processing to obtain a target prediction result for each person.

[0011] In a feasible implementation, the video and audio acquisition device includes at least a camera, and the step of determining the imaging distortion characteristics of each person by the video and audio acquisition device includes:

[0012] Utilizing the first position coordinates of each person in the world coordinate system and the second position coordinates of the camera in the world coordinate system, each target azimuth angle between each person and each camera is determined, and the imaging distortion feature includes the target azimuth angle.

[0013] In a feasible implementation, the using the imaging distortion feature and the plurality of the first prediction results to perform prediction result optimization processing to obtain a target prediction result for each person includes:

[0014] Determining a target distortion type for each of the cameras;

[0015] Determine a target deformation coefficient list corresponding to the target deformation type by using a preset database of deformation types and deformation coefficient lists and the target deformation type, wherein the deformation coefficient list includes a corresponding relationship between the deformation coefficient and the azimuth under the distortion type;

[0016] Searching in the target deformation coefficient list based on the target azimuth angle, obtaining a target deformation coefficient corresponding to the azimuth angle that is the same as the target azimuth angle;

[0017] The target deformation coefficient is used as a weight to perform prediction result optimization processing on the multiple first prediction results of each character to obtain the target prediction result of each character.

[0018] In a feasible implementation, the first prediction result includes a first prediction label of a classroom behavior type and the credibility of the first prediction label, and the target deformation coefficient is used as a weight to perform prediction result optimization processing on multiple first prediction results of each character to obtain a target prediction result of each character, including:

[0019] Taking the target azimuth as a clustering condition, clustering the plurality of the first prediction results for each character respectively, to obtain a target cluster set corresponding to each target azimuth of each character, wherein the target cluster set includes each second prediction result corresponding to the target azimuth;

[0020] Using the credibility as a weight, respectively perform weighted decision processing on the second prediction labels included in the second prediction results of each target cluster set to obtain a third prediction label after weighted decision;

[0021] The target deformation coefficient is used as a weight to perform weighted decision processing on each third prediction label of each character to obtain a final prediction label of each character, and the target prediction result includes the final prediction label.

[0022] In a feasible implementation, the artificial intelligence model includes at least a Transformer model based on a self-attention mechanism, a three-dimensional convolutional neural network, a recurrent neural network, and a convolutional neural network. Then, the classroom image is respectively input into a plurality of pre-trained artificial intelligence models for behavior prediction, and a plurality of first prediction results for each character in the classroom image are obtained, including:

[0023] The classroom image is respectively input into the trained Transformer model based on the self-attention mechanism, the three-dimensional convolutional neural network, the recurrent neural network and the convolutional neural network for behavior prediction to obtain multiple first prediction results for each character in the classroom image.

[0024] In a feasible implementation, the method further includes:

[0025] Acquire training sample data, wherein the training sample data includes at least a number of sample classroom images and true labels of classroom behavior types corresponding to the sample classroom images;

[0026] Input the sample classroom images into the initial convolutional neural network for training to obtain the fourth predicted label for each character;

[0027] Determine a loss value of the initial convolutional neural network using the fourth predicted label, the true label, and a preset loss function;

[0028] If the loss value is lower than a preset loss threshold, it is determined that the initial convolutional neural network has converged, and a trained convolutional neural network is obtained.

[0029] In one feasible implementation, the loss function is the following mathematical expression:

[0030]

[0031] In the formula, N is the total number of samples, M is the total number of classroom behavior types, and IOU i Represents the ratio of the intersection and union of the predicted box and the true box of sample i, D i is the distance between the predicted box and the center point of the real box of sample i, L iis the diagonal length of the minimum enclosing rectangle of the predicted box and the real box of sample i, v i is the similarity between the aspect ratio of the predicted box and the real box of sample i, and α is v i The impact factor, y ic is an indicator variable used to indicate the authenticity of the classroom behavior type c of the fourth predicted label of sample i. If the classroom behavior type c is the same as the classroom behavior type of the true label of sample i, then y ic The value is 1, otherwise it is 0. ic Represents the predicted probability that observation sample i belongs to classroom behavior type c.

[0032] To achieve the above object, the second aspect of the present invention provides a device for identifying classroom behavior, the device comprising:

[0033] Data acquisition module: used to acquire classroom audio and video data collected by multiple audio and video acquisition devices during the teaching process, wherein the classroom audio and video data at least includes each classroom image collected by each audio and video acquisition device;

[0034] Behavior prediction module: used for inputting the classroom image into a plurality of pre-trained artificial intelligence models for behavior prediction, and obtaining a plurality of first prediction results for each character in the classroom image, wherein the prediction results are used to reflect the classroom behavior type of the character;

[0035] Distortion determination module: used to determine the imaging distortion characteristics of each person by the video and audio acquisition device;

[0036] Result optimization module: used to optimize the prediction results by using the imaging distortion features and the plurality of the first prediction results to obtain the target prediction result for each person.

[0037] To achieve the above-mentioned purpose, the third aspect of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps shown in the first aspect and any feasible implementation method.

[0038] To achieve the above-mentioned purpose, the fourth aspect of the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps shown in the first aspect and any feasible implementation method.

[0039] The embodiments of the present invention have the following beneficial effects:

[0040] The present invention provides a method for identifying classroom behavior, the method comprising: obtaining classroom audio and video data collected by multiple audio and video acquisition devices during the teaching process, the classroom audio and video data at least including each classroom image collected by each audio and video acquisition device; inputting the classroom images into multiple pre-trained artificial intelligence models for behavior prediction, and obtaining multiple first prediction results for each character in the classroom image, the prediction results are used to reflect the classroom behavior type of the character; determining the imaging distortion characteristics of the audio and video acquisition device for each character; using the imaging distortion characteristics and multiple first prediction results to optimize the prediction results, and obtain the target prediction results of each character. Through the above method, the classroom behavior prediction results and imaging distortion characteristics of multiple pre-trained artificial intelligence models can be combined to optimize the prediction results, and the optimized target prediction results of each character can be obtained. Not only the model advantages of multiple pre-trained artificial intelligence models are combined, but also the influence of imaging distortion on the prediction results is taken into account, so that the accuracy of the target prediction results obtained is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] in:

[0043] Figure 1 is a flow chart of a method for identifying classroom behavior in an embodiment of the present invention;

[0044] Figure 2 is another flow chart of a method for identifying classroom behavior in an embodiment of the present invention;

[0045] Figure 3 is a structural block diagram of a classroom behavior recognition device in an embodiment of the present invention;

[0046] Figure 4 4 is a structural block diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] See also Figure 1, Figure 1 Flow chart of a method for identifying classroom behavior in an embodiment of the present invention. The method can be applied to both a terminal and a server. The terminal can be a desktop terminal or a mobile terminal. The mobile terminal can be at least one of a mobile phone, a tablet computer, a laptop computer, etc. The server can be implemented by an independent server or a server cluster composed of multiple servers. This embodiment is described by applying to a terminal as an example. Figure 1 The method shown includes the following steps:

[0049] 101. Acquire classroom audio and video data collected by multiple audio and video collection devices during the teaching process, wherein the classroom audio and video data at least includes each classroom image collected by each audio and video collection device;

[0050] It should be noted that in order to realize intelligent monitoring of students' classroom behavior in the classroom during the teaching process, it is necessary to arrange audio and video acquisition equipment in the classroom in advance. The audio and video acquisition equipment can be an integrated image acquisition equipment and audio acquisition equipment, or a split image acquisition equipment and audio acquisition equipment. The present application takes an integrated image acquisition equipment and audio acquisition equipment as an example, wherein the integrated image acquisition equipment and audio acquisition equipment can be monitoring. The number of monitoring in the present application can be multiple, at least two, and each monitoring is dispersedly arranged in different positions in the classroom to realize all-round and multi-angle monitoring of students. Among them, the audio and video acquisition equipment is used to collect classroom audio and video data during the teaching process, including classroom images and audio during the teaching process. The image can be divided frame by frame to obtain classroom images, and classroom behavior recognition can be realized based on the classroom images. It can be understood that each camera can capture classroom images. Therefore, in the present application, multi-angle classroom images can be collected at the same time, and correspondingly, the multi-faceted behavior performance of each student can be obtained at the same time, so as to perform subsequent classroom behavior recognition with higher accuracy.

[0051] 102. Input the classroom image into a plurality of pre-trained artificial intelligence models for behavior prediction, and obtain a plurality of first prediction results for each character in the classroom image, wherein the prediction results are used to reflect the classroom behavior type of the character;

[0052] Furthermore, after obtaining the above-mentioned classroom image, the classroom image is input into a pre-trained artificial intelligence model for behavior prediction, and multiple first prediction results for each character in the classroom image are obtained, wherein the prediction results are used to reflect the classroom behavior type of the character, wherein classroom behavior refers to the performance of students in the classroom, including their learning attitude, participation, compliance with classroom rules and respect for others, and classroom behavior types include but are not limited to listening, daydreaming, standing and turning heads, etc. Among them, the prediction results include the prediction label of the classroom behavior type and the credibility of the prediction label. Among them, the prediction label can be the above-mentioned listening, daydreaming, standing and turning heads, etc. behavior types, which are not limited here.

[0053] It should be noted that the terms "first", "second", etc. in the description of this application are used to distinguish different objects rather than to describe a specific order.

[0054] Since different artificial intelligence models have different advantages and disadvantages, this application uses multiple artificial intelligence models for prediction, which can utilize the strengths of each artificial intelligence model. In a feasible implementation method, the artificial intelligence model includes at least a Transformer model based on a self-attention mechanism, a three-dimensional convolutional neural network, a recurrent neural network, and a convolutional neural network, etc.

[0055] Among them, the Transformer model based on self-attention mechanism, three-dimensional convolutional neural network (3D CNN), recurrent neural network (RNN) and convolutional neural network (CNN) are all important models in the field of deep learning. 1) The Transformer model based on self-attention mechanism is a deep learning model based on self-attention mechanism. The core of the Transformer model is the multi-head self-attention mechanism, which allows the model to consider the information of the entire sequence when processing each element, thereby capturing the long-distance dependencies within the sequence. This mechanism enables the model to process sequence data in parallel and improves computational efficiency; 2) The three-dimensional convolutional neural network (3D CNN) is an extension of the convolutional neural network. It adds a depth dimension on the basis of two-dimensional convolution, enabling it to process three-dimensional data, such as videos or three-dimensional medical images. 3D CNN extracts features by sliding convolution kernels in three dimensions, which enables it to capture spatial and temporal patterns and is very suitable for video classification and 3D image segmentation tasks; 3) Recurrent Neural Network (RNN) is a neural network suitable for processing sequence data. It can remember previous information and use it for the calculation of current output. The hidden layer nodes of RNN have connections not only from the input layer, but also from the previous time step, which enables it to maintain contextual information in the sequence. 4) Convolutional Neural Network (CNN) is a deep learning model commonly used in image processing. It extracts local features of images through convolution layers and reduces the spatial size of features through pooling layers. CNN can identify features such as edges and textures in images through learned convolution kernels, and combine features at multiple levels to ultimately achieve recognition and classification of image content.

[0056] The classroom images are then input into a variety of pre-trained artificial intelligence models for behavior prediction, and multiple first prediction results for each character in the classroom image are obtained. Specifically, the classroom images are input into the trained Transformer model based on the self-attention mechanism, the three-dimensional convolutional neural network, the recurrent neural network and the convolutional neural network for behavior prediction, and multiple first prediction results for each character in the classroom image are obtained.

[0057] It is understandable that the cameras are distributed everywhere, so the cameras have multiple shooting angles, so each character has multiple classroom images shot from multiple angles, and each classroom image corresponds to multiple first prediction results of multiple models, that is, there are S cameras and M artificial intelligence models. There are S*M first prediction results for character 1. For example, the cameras are set in the east, south, west and north directions of the classroom, that is, S=4, and there are the above four artificial intelligence models, that is, M=4, so there are 16 first prediction results for character 1.

[0058] Among them, the above models are all pre-trained using pre-collected training sample data, and the training sample data at least includes several sample classroom images and the real labels of the classroom behavior types corresponding to the sample classroom images.

[0059] Taking the training of convolutional neural networks as an example, in order to improve the prediction accuracy of convolutional neural networks, the convolutional neural network in this application at least includes a target detection network and a deep separable convolutional neural network, wherein the output of the target detection network is the input of the deep separable convolutional neural network, the sample classroom image is input into the target detection network, the target detection network is used to identify the human targets in the sample classroom image, so the output of the target detection network is the human region in the sample classroom image, the deep separable convolutional neural network is used to predict the classroom behavior of the human targets in the sample classroom image, the human region includes the face and the body, so that the deep separable convolutional neural network can perform behavior recognition through facial features and body features, the facial features include but are not limited to the physical structure of the face, facial expressions, face shape and features of various parts of the face, the body features include but are not limited to human posture features, etc. With the convolutional neural network trained in this way, the classroom image can be input into the trained target detection network for target detection, and the human targets in the classroom image can be identified. Further, the human targets in the classroom image are input into the trained deep separable convolutional neural network for behavior prediction, and the prediction results of the classroom behavior of the characters in the classroom image are obtained.

[0060] Among them, the network composition of the target detection network and the deep separable convolutional neural network can refer to the network composition of the existing target detection network and the deep separable convolutional neural network, which will not be described in detail. Furthermore, the training of the convolutional neural network including the target detection network and the deep separable convolutional neural network can refer to the following processes A01 to A04:

[0061] A01. Obtain training sample data, wherein the training sample data includes at least a number of sample classroom images and real labels of classroom behavior types corresponding to the sample classroom images;

[0062] It can be understood that if the convolutional neural network is to learn the corresponding classroom behavior, the corresponding sample data needs to be used as sample data to train the convolutional neural network, wherein the training sample data includes at least several sample classroom images and the real labels of the classroom behavior types corresponding to the sample classroom images.

[0063] A02. Input the sample classroom image into the initial convolutional neural network for training to obtain the fourth predicted label of each character;

[0064] A03. Determine the loss value of the initial convolutional neural network using the fourth predicted label, the true label and a preset loss function;

[0065] A04. If the loss value is lower than a preset loss threshold, it is determined that the initial convolutional neural network has converged to obtain a trained convolutional neural network.

[0066] Furthermore, the sample classroom images are input into the initial convolutional neural network for training to obtain the fourth prediction label for each character. The initial convolutional neural network is the original convolutional neural network that has not yet learned the classroom behavior characteristics, and then by comparing the predicted situation with the actual situation, the initial convolutional neural network gradually learns the classroom behavior characteristics, and obtains a trained convolutional neural network that can identify classroom behavior. Among them, the loss value of the initial convolutional neural network is determined using the fourth prediction label, the real label and the preset loss function. The loss value is an indicator used in machine learning to measure the difference between the model prediction value and the true value. The loss function is a mathematical function that takes the model's predicted value and true value as input, and outputs a non-negative real number to represent the loss value. The smaller the loss value, the closer the model's prediction result is to the true value, and the better the performance of the model.

[0067] In one feasible implementation, the loss function is the following mathematical expression:

[0068]

[0069] In the formula, is the loss calculation part of the target detection network, is the loss calculation part of the deep separable convolutional neural network, where N is the total number of samples, M is the total number of classroom behavior types, and IOU i Represents the ratio of the intersection and union of the predicted box and the true box of sample i, D i is the distance between the predicted box and the center point of the real box of sample i, L i is the diagonal length of the minimum enclosing rectangle of the predicted box and the real box of sample i, v i is the similarity between the aspect ratio of the predicted box and the real box of sample i, and α is v i The impact factor, y ic is an indicator variable used to indicate the authenticity of the classroom behavior type c of the fourth predicted label of sample i. If the classroom behavior type c is the same as the classroom behavior type of the true label of sample i, then y ic The value is 1, otherwise it is 0. icRepresents the predicted probability that the observed sample i belongs to the classroom behavior type c, which is also called credibility. Sample i is a sample classroom image, and the true box is the smallest enclosing rectangle of the selected person. The target detection network outputs the predicted box of the person. The sample classroom image is pre-labeled with the true box of the person and the true classroom behavior label, so as to calculate the corresponding loss value during training.

[0070] It should be noted that the loss value of the convolutional neural network is obtained by combining the loss values ​​of the two networks. Among them, the output of the target detection network is used as the input of the deep separable convolutional neural network. The target detection network determines the accuracy of character recognition, which indirectly affects the accuracy of the classroom behavior prediction of the deep separable convolutional neural network. If the target detection network does not converge, then the deep separable convolutional neural network will inevitably be difficult to converge; and if the target detection network converges, the deep separable convolutional neural network may not converge at the same time. Therefore, the RAND() function is used to take a random number between the loss values ​​of the two networks. As long as any network has not converged, the deep separable convolutional neural network will not converge at the same time. If the target detection network does not converge, the randomly obtained loss value is not converged, and if the target detection network does not converge, the deep separable convolutional neural network will inevitably be difficult to converge, and the randomly obtained loss value is still not converged; if the target detection network converges and the deep separable convolutional neural network does not converge, then the randomly obtained loss value is still not converged. In this way, only when both networks converge, a converged random loss value will be obtained between the two converged loss values, that is, the obtained random number must be a converged random number. Therefore, the final loss value of the convolutional neural network can be calculated by the above loss function to evaluate whether the initial convolutional neural network converges.

[0071] Furthermore, if the loss value loss is lower than the preset loss threshold, it is determined that the initial convolutional neural network has converged and a trained convolutional neural network is obtained; otherwise, if the loss value loss is greater than the preset loss threshold, it is determined that it has not converged. After further adjusting and optimizing the model parameters of the initial convolutional neural network, return to step A02 to continue training the initial convolutional neural network until convergence.

[0072] 103. Determine the imaging distortion characteristics of each person by the video and audio acquisition device;

[0073] 104. Utilize the imaging distortion feature and the plurality of first prediction results to perform prediction result optimization processing to obtain a target prediction result for each person.

[0074] Furthermore, the audio and video acquisition device is fixedly installed, and the same audio and video acquisition device will produce different imaging distortions for each character, which affects the accuracy of the characters' expressions and postures in the classroom image, and further affects the first prediction result. Therefore, in order to further improve the accuracy of the prediction results, the present application determines the imaging distortion characteristics of the audio and video acquisition device for each character; uses the imaging distortion characteristics and multiple first prediction results to optimize the prediction results to obtain the target prediction results for each character, and the imaging distortion characteristics are used to reflect the degree of imaging distortion, wherein the optimization processing of the prediction results includes but is not limited to weighted decision-making, which will not be elaborated here.

[0075] Among them, the audio data can also be input into the natural language processing AI model to predict classroom behavior, obtain new prediction results of classroom behavior, and make further decisions based on the target prediction results and the new prediction results.

[0076] The present invention provides a method for identifying classroom behavior, the method comprising: obtaining classroom audio and video data collected by multiple audio and video acquisition devices during the teaching process, the classroom audio and video data at least including each classroom image collected by each audio and video acquisition device; inputting the classroom images into multiple pre-trained artificial intelligence models for behavior prediction, and obtaining multiple first prediction results for each character in the classroom image, the prediction results are used to reflect the classroom behavior type of the character; determining the imaging distortion characteristics of the audio and video acquisition device for each character; using the imaging distortion characteristics and multiple first prediction results to optimize the prediction results, and obtain the target prediction results of each character. Through the above method, the classroom behavior prediction results and imaging distortion characteristics of multiple pre-trained artificial intelligence models can be combined to optimize the prediction results, and the optimized target prediction results of each character can be obtained. Not only the model advantages of multiple pre-trained artificial intelligence models are combined, but also the influence of imaging distortion on the prediction results is taken into account, so that the accuracy of the target prediction results obtained is higher.

[0077] See also Figure 2 , Figure 2 Another flow chart of a method for identifying classroom behavior in an embodiment of the present invention is shown in FIG. Figure 2 The method shown includes the following steps:

[0078] 201. Acquire classroom audio and video data collected by multiple audio and video collection devices during the teaching process, wherein the classroom audio and video data at least includes each classroom image collected by each audio and video collection device;

[0079] 202. Input the classroom image into a plurality of pre-trained artificial intelligence models for behavior prediction, and obtain a plurality of first prediction results for each character in the classroom image, wherein the prediction results are used to reflect the classroom behavior type of the character;

[0080] 203. Determine the imaging distortion characteristics of each person by the video and audio acquisition device;

[0081] It should be noted that steps 201 to 203 are Figure 1 The contents of steps 101 to 103 are similar, and are not described here to avoid repetition. For details, please refer to Figure 1 The contents of steps 101 to 103 are shown.

[0082] It can be understood that the audio and video acquisition device includes at least a camera, and step 203 can specifically be to use the first position coordinates of each person in the world coordinate system and the second position coordinates of the camera in the world coordinate system to determine the target azimuth angles between each person and each of the cameras, and the imaging distortion characteristics include the target azimuth angles.

[0083] In this embodiment, an electronic tag is pre-set on the desk of each person. The electronic tag contains the position coordinates of the desk in the world coordinate system. The first position coordinates of the person are acquired by identifying the electronic tag of each person's desk. The position coordinates of the person's desk are used as the position coordinates of the person. Then, the second position coordinates of the camera in the world coordinate system are used to determine the target azimuths between each person and the camera. Since the direction of the camera is fixed, the position of the person in the camera can be known through two coordinates.

[0084] 204. Determine a target distortion type of each camera;

[0085] Since different camera types may cause different types of distortion, the distortion types include at least barrel distortion and pincushion distortion. The camera type can be used to determine whether the target distortion type of the camera is barrel distortion or pincushion distortion.

[0086] 205. Determine a target deformation coefficient list corresponding to the target deformation type by using a preset database of deformation types and deformation coefficient lists and the target deformation type, wherein the deformation coefficient list includes a corresponding relationship between the deformation coefficient and the azimuth under the distortion type;

[0087] Then, the influence of the imaging is determined according to the distortion type. Specifically, the target deformation coefficient list corresponding to the target distortion type is determined by using a preset database of distortion types and deformation coefficient lists and the target distortion type. The database stores multiple distortion types and deformation coefficient lists of each distortion type. For example, the database stores deformation coefficient list 1 of barrel distortion and deformation coefficient list 2 of pincushion distortion. If the target distortion type is pincushion distortion, the target deformation coefficient list is deformation coefficient list 2 of pincushion distortion. The deformation coefficient list includes the corresponding relationship between the azimuth angle and the deformation coefficient under the distortion type, that is, the deformation coefficient list stores the corresponding relationship between the azimuth angle of the person relative to the lens and the deformation coefficient under the distortion type. The corresponding relationship can be obtained through the control variable experiment of the azimuth angle and the deformation situation between the person and each camera. In this way, the deformation situation under each azimuth angle under the distortion type of each camera can be obtained. According to the deformation situation, a unique deformation coefficient is assigned to each situation to characterize the deformation situation.

[0088] Among them, the target deformation coefficient is used to reflect the influence of the target azimuth on the deformation of the character imaging under the target distortion type. The deformation coefficient takes a value between 0 and 1. The higher the deformation coefficient, the greater the influence of the azimuth on the character imaging under this distortion type, and the more deformed the image; the lower the deformation coefficient, the smaller the influence of the azimuth on the character imaging under this distortion type, and the less deformed the image.

[0089] 206. Search the target deformation coefficient list based on the target azimuth to obtain a target deformation coefficient corresponding to the azimuth that is the same as the target azimuth;

[0090] 207. Use the target deformation coefficient as a weight to perform prediction result optimization processing on the multiple first prediction results of each character to obtain a target prediction result for each character.

[0091] After obtaining the target azimuth of the character, the corresponding target deformation coefficient can be searched in the target deformation coefficient list based on the target azimuth. Specifically, the target deformation coefficient list is searched based on the target azimuth to obtain the target deformation coefficient corresponding to the azimuth that is the same as the target azimuth, and then the target deformation coefficient is used as a weight to optimize the prediction results of the multiple first prediction results of each character to obtain the target prediction result of each character.

[0092] In a feasible implementation, the first prediction result includes a first prediction label of a classroom behavior type and the credibility of the first prediction label, and the target deformation coefficient is used as a weight to perform prediction result optimization processing on multiple first prediction results of each character to obtain a target prediction result of each character, including steps B01 to B03:

[0093] B01, clustering the plurality of first prediction results of each character respectively with the target azimuth as a clustering condition to obtain a target cluster set corresponding to each target azimuth of each character, wherein the target cluster set includes each second prediction result corresponding to the target azimuth;

[0094] First, cluster the first prediction results, and cluster the multiple first prediction results of each character with the target azimuth as the clustering condition to obtain a target clustering set corresponding to each target azimuth of each character, the target clustering set including each second prediction result corresponding to the target azimuth. Exemplarily, the cameras are set in 4 orientations in the classroom, i.e., S=4, and there are the above four artificial intelligence models, i.e., M=4. In this way, there are 16 first prediction results for character 1, wherein character 1 has target azimuth 1, target azimuth 2, target azimuth 3 and target azimuth 4 with the four cameras respectively, wherein target azimuth 1 is the azimuth between character 1 and camera 1, target azimuth 2 is the azimuth between character 1 and camera 2, target azimuth 3 is the azimuth between character 1 and camera 3, and target azimuth 4 is the azimuth between character 1 and camera 4. Cluster the first prediction results of character 1 with the target azimuth as the clustering condition to obtain four target clustering sets respectively. The target cluster sets with target azimuths include target cluster set 1 of target azimuth 1, target cluster set 2 of target azimuth 2, target cluster set 3 of target azimuth 3 and target cluster set 4 of target azimuth 4, wherein target cluster set 1 includes multiple artificial intelligence models that predict the classroom behavior of person 1 based on the classroom images taken by camera 1, and output respective second prediction results; target cluster set 2 includes multiple artificial intelligence models that predict the classroom behavior of person 1 based on the classroom images taken by camera 2, and output respective second prediction results; target cluster set 3 includes multiple artificial intelligence models that predict the classroom behavior of person 1 based on the classroom images taken by camera 3, and output respective second prediction results; target cluster set 4 includes multiple artificial intelligence models that predict the classroom behavior of person 1 based on the classroom images taken by camera 4, and output respective second prediction results, and so on. If there are i people, the number of target cluster sets is i*4.

[0095] B02, using the credibility as a weight, respectively performing weighted decision processing on the second prediction labels included in the second prediction results of each of the target cluster sets, to obtain a third prediction label after weighted decision;

[0096] After obtaining the target cluster set classified by target azimuth for each character, a weighted decision process can be performed based on the target cluster set. Specifically, the credibility is used as the weight, and the second prediction labels included in the second prediction results of each target cluster set are weighted decision-made to obtain the third prediction label after the weighted decision. It can be understood that the prediction result of the artificial intelligence model includes the prediction label and the prediction probability, and the prediction probability is the credibility of the prediction label. The weighted decision method can be to select the second prediction label with the highest credibility as the third prediction label after the weighted decision; if there are multiple highest second prediction labels, it can be determined whether the second prediction label refers to the same classroom behavior type. If it refers to the same classroom behavior type, a highest second prediction label is randomly selected as the third prediction label after the weighted decision. If it does not refer to the same classroom behavior type, the second prediction label with the largest number is selected from several second prediction labels with the highest credibility as the third prediction label after the weighted decision.

[0097] Continuing with the example that each character has four target cluster sets, at this time, there are four third prediction labels after weighted decision. Specifically, target cluster set 1 can select a second prediction label as the third prediction label, target cluster set 2 can select a second prediction label as the third prediction label, target cluster set 3 can select a second prediction label as the third prediction label, and target cluster set 4 can select a second prediction label as the third prediction label.

[0098] B03. Using the target deformation coefficient as a weight, a weighted decision process is performed on each third prediction label of each character to obtain a final prediction label of each character, and the target prediction result includes the final prediction label.

[0099] Furthermore, a weighted decision is made again through the target deformation coefficient, and a final prediction label is selected from multiple third prediction labels to obtain a target prediction result, wherein the weighted decision method can be to select the third prediction label with the lowest target deformation coefficient as the final prediction label after the weighted decision; if there are multiple lowest third prediction labels, it can be determined whether the third prediction label refers to the same classroom behavior type. If it refers to the same classroom behavior type, a lowest third prediction label is randomly selected as the final prediction label after the weighted decision; if it does not refer to the same classroom behavior type, the third prediction label with the largest number is selected from several third prediction labels with the lowest credibility as the final prediction label after the weighted decision.

[0100] The present invention provides a method for identifying classroom behavior, which includes: obtaining classroom audio and video data collected by multiple audio and video acquisition devices during the teaching process, the classroom audio and video data at least including each classroom image collected by each audio and video acquisition device; inputting the classroom images into multiple pre-trained artificial intelligence models for behavior prediction, and obtaining multiple first prediction results for each character in the classroom image, and the prediction results are used to reflect the classroom behavior type of the character; determining the imaging distortion characteristics of the audio and video acquisition device for each character, the imaging distortion characteristics including the target azimuth; determining the target distortion type of each camera; using a database of preset distortion types and deformation coefficient lists and the target distortion type, determining a target deformation coefficient list corresponding to the target distortion type, the deformation coefficient list including the corresponding relationship between the deformation coefficient and the azimuth under the distortion type; searching in the target deformation coefficient list based on the target azimuth to obtain the target deformation coefficient corresponding to the azimuth that is the same as the target azimuth; using the target deformation coefficient as a weight to optimize the prediction results of the multiple first prediction results of each character, and obtain the target prediction result of each character. Through the above method, the classroom behavior prediction results and imaging distortion characteristics of multiple pre-trained artificial intelligence models can be combined to optimize the prediction results and obtain optimized target prediction results for each character. It not only combines the model advantages of multiple pre-trained artificial intelligence models, but also takes into account the imaging distortion of the character caused by the azimuth angle between the character and the camera, and measures the impact of imaging distortion on the prediction results. The target prediction results obtained in this way are more accurate.

[0101] See also Figure 3 , Figure 3 FIG. 1 is a structural block diagram of a device for identifying classroom behavior in an embodiment of the present invention. Figure 3 The device shown comprises:

[0102] Data collection module 301: used to obtain classroom audio and video data collected by multiple audio and video collection devices during the teaching process, wherein the classroom audio and video data at least includes each classroom image collected by each audio and video collection device;

[0103] Behavior prediction module 302: used to input the classroom image into a plurality of pre-trained artificial intelligence models for behavior prediction, and obtain a plurality of first prediction results for each character in the classroom image, wherein the prediction results are used to reflect the classroom behavior type of the character;

[0104] Distortion determination module 303: used to determine the imaging distortion characteristics of each person by the video and audio acquisition device;

[0105] The result optimization module 304 is used to optimize the prediction results by using the imaging distortion features and the first prediction results to obtain the target prediction result for each person.

[0106] It should be noted that if Figure 3 The functions of each module in the device shown are Figure 1 The contents of each step in the method shown are similar. To avoid repetition, they are not described here. For details, please refer to Figure 1 The contents of each step in the method shown.

[0107] The present invention provides a device for identifying classroom behavior, the device comprising: a data acquisition module: used to obtain classroom audio and video data collected by multiple audio and video acquisition devices during the teaching process, the classroom audio and video data at least including each classroom image collected by each audio and video acquisition device; a behavior prediction module: used to input the classroom images into multiple pre-trained artificial intelligence models for behavior prediction, and obtain multiple first prediction results for each character in the classroom image, and the prediction results are used to reflect the classroom behavior type of the character; a distortion determination module: used to determine the imaging distortion characteristics of the audio and video acquisition device for each character; a result optimization module: used to optimize the prediction results using the imaging distortion characteristics and multiple first prediction results, and obtain the target prediction results of each character. Through the above method, the classroom behavior prediction results and imaging distortion characteristics of multiple pre-trained artificial intelligence models can be combined to optimize the prediction results, and the optimized target prediction results of each character can be obtained. Not only the model advantages of multiple pre-trained artificial intelligence models are combined, but also the influence of imaging distortion on the prediction results is taken into account, so that the accuracy of the target prediction results obtained is higher.

[0108] In a feasible implementation, the above device further includes:

[0109] Sample collection module: used to obtain training sample data, the training sample data at least includes a number of sample classroom images and real labels of classroom behavior types corresponding to the sample classroom images;

[0110] Model training module: used to input sample classroom images into the initial convolutional neural network for training to obtain the fourth predicted label of each character;

[0111] A loss determination module: used to determine the loss value of the initial convolutional neural network using the fourth predicted label, the true label and a preset loss function;

[0112] Convergence judgment module: used to determine that the initial convolutional neural network converges if the loss value is lower than a preset loss threshold, thereby obtaining a trained convolutional neural network.

[0113] The present invention also provides a device for identifying classroom behavior, the device includes: a sample collection module: used to obtain training sample data, the training sample data at least includes a number of sample classroom images and the real labels of the classroom behavior types corresponding to the sample classroom images; a model training module: used to input the sample classroom images into the initial convolutional neural network for training, and obtain the fourth predicted label of each character; a loss determination module: used to determine the loss value of the initial convolutional neural network using the fourth predicted label, the real label and the preset loss function; a convergence judgment module: used to determine that the initial convolutional neural network converges if the loss value is lower than the preset loss threshold, and obtain a trained convolutional neural network. By training the initial convolutional neural network in the above manner, judging whether the training has converged based on the loss value of the loss function, not only can a convolutional neural network that can predict classroom behavior be obtained, but also the convergence judgment accuracy is higher, thereby improving the training accuracy.

[0114] Figure 4 FIG. 1 shows an internal structure diagram of a computer device in an embodiment. The computer device may be a terminal or a server. Figure 4 As shown, the computer device includes a processor, a memory and a network interface connected via a system bus. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor can implement the above method. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor can implement the above method. Those skilled in the art can understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0115] In one embodiment, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the following steps: Figure 1 or Figure 2 The steps of the method are shown.

[0116] In one embodiment, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the processor performs the following steps: Figure 1 or Figure 2 The steps of the method are shown.

[0117] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0118] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0119] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for identifying classroom behavior, characterized in that: The method comprises: Acquire classroom audio and video data collected by multiple audio and video collection devices during the teaching process, wherein the classroom audio and video data at least includes each classroom image collected by each audio and video collection device; Inputting the classroom image into a plurality of pre-trained artificial intelligence models for behavior prediction, respectively, to obtain a plurality of first prediction results for each character in the classroom image, wherein the prediction results are used to reflect the classroom behavior type of the character; Determining imaging distortion characteristics of each person by the video and audio acquisition device; The imaging distortion feature and the plurality of first prediction results are used to perform prediction result optimization processing to obtain a target prediction result for each person.

2. The method according to claim 1, characterized in that: The video and audio acquisition device at least includes a camera, and the step of determining the imaging distortion characteristics of each person by the video and audio acquisition device includes: Utilizing the first position coordinates of each person in the world coordinate system and the second position coordinates of the camera in the world coordinate system, each target azimuth angle between each person and each camera is determined, and the imaging distortion feature includes the target azimuth angle.

3. The method according to claim 2, characterized in that: The using the imaging distortion feature and the plurality of the first prediction results to perform prediction result optimization processing to obtain a target prediction result for each person includes: Determining a target distortion type for each of the cameras; Determine a target deformation coefficient list corresponding to the target deformation type by using a preset database of deformation types and deformation coefficient lists and the target deformation type, wherein the deformation coefficient list includes a corresponding relationship between the deformation coefficient and the azimuth under the distortion type; Searching in the target deformation coefficient list based on the target azimuth angle, obtaining a target deformation coefficient corresponding to the azimuth angle that is the same as the target azimuth angle; The target deformation coefficient is used as a weight to perform prediction result optimization processing on the multiple first prediction results of each character to obtain the target prediction result of each character.

4. The method according to claim 3, characterized in that: The first prediction result includes a first prediction label of the classroom behavior type and the credibility of the first prediction label, and the target deformation coefficient is used as a weight to perform prediction result optimization processing on multiple first prediction results of each character to obtain a target prediction result of each character, including: Taking the target azimuth as a clustering condition, clustering the plurality of the first prediction results for each character respectively, to obtain a target cluster set corresponding to each target azimuth of each character, wherein the target cluster set includes each second prediction result corresponding to the target azimuth; Using the credibility as a weight, respectively perform weighted decision processing on the second prediction labels included in the second prediction results of each target cluster set to obtain a third prediction label after weighted decision; The target deformation coefficient is used as a weight to perform weighted decision processing on each third prediction label of each character to obtain a final prediction label of each character, and the target prediction result includes the final prediction label.

5. The method according to claim 1, characterized in that: The artificial intelligence model at least includes a Transformer model based on a self-attention mechanism, a three-dimensional convolutional neural network, a recurrent neural network, and a convolutional neural network. The classroom image is respectively input into a plurality of pre-trained artificial intelligence models for behavior prediction to obtain a plurality of first prediction results for each character in the classroom image, including: The classroom image is respectively input into the trained Transformer model based on the self-attention mechanism, the three-dimensional convolutional neural network, the recurrent neural network and the convolutional neural network for behavior prediction to obtain multiple first prediction results for each character in the classroom image.

6. The method according to claim 5, characterized in that: The method further comprises: Acquire training sample data, wherein the training sample data includes at least a number of sample classroom images and true labels of classroom behavior types corresponding to the sample classroom images; Input the sample classroom images into the initial convolutional neural network for training to obtain the fourth predicted label for each character; Determine a loss value of the initial convolutional neural network using the fourth predicted label, the true label, and a preset loss function; If the loss value is lower than a preset loss threshold, it is determined that the initial convolutional neural network has converged, and a trained convolutional neural network is obtained.

7. The method according to claim 6, characterized in that: The loss function is expressed as follows: In the formula, N is the total number of samples, M is the total number of classroom behavior types, and IOU i Represents the ratio of the intersection and union of the predicted box and the true box of sample i, D i is the distance between the predicted box and the center point of the real box of sample i, L i is the diagonal length of the minimum enclosing rectangle of the predicted box and the real box of sample i, v i is the similarity between the aspect ratio of the predicted box and the real box of sample i, and α is v i The impact factor, y ic is an indicator variable used to indicate the authenticity of the classroom behavior type c of the fourth predicted label of sample i. If the classroom behavior type c is the same as the classroom behavior type of the true label of sample i, then y ic The value is 1, otherwise it is 0. ic Represents the predicted probability that observation sample i belongs to classroom behavior type c.

8. A device for identifying classroom behavior, characterized in that: The device comprises: Data acquisition module: used to acquire classroom audio and video data collected by multiple audio and video acquisition devices during the teaching process, wherein the classroom audio and video data at least includes each classroom image collected by each audio and video acquisition device; Behavior prediction module: used for inputting the classroom image into a plurality of pre-trained artificial intelligence models for behavior prediction, and obtaining a plurality of first prediction results for each character in the classroom image, wherein the prediction results are used to reflect the classroom behavior type of the character; Distortion determination module: used to determine the imaging distortion characteristics of each person by the video and audio acquisition device; Result optimization module: used to optimize the prediction results by using the imaging distortion features and the plurality of the first prediction results to obtain the target prediction result for each person.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

10. A computer device comprising a memory and a processor, characterized in that: The memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

Images