A real-time human motion recognition method, computer device and readable medium in classroom

Abstract

The present application discloses a method for real-time classroom human motion recognition, a computer device and a readable medium, the method comprising: extracting RGB images and optical flow maps in real time from a classroom video to be analyzed; inputting the RGB image or the RGB image and the optical flow map into a trained hybrid convolutional neural network to obtain a human motion classification result in the classroom video; the hybrid convolutional neural network has a wide residual network structure, comprising a two-dimensional convolutional network and a three-dimensional convolutional network connected in parallel with the two-dimensional convolutional network, extracting a two-dimensional feature vector and a three-dimensional feature vector from the RGB image and the optical flow map respectively; recognizing human motion based on a feature map obtained by fusing the two-dimensional feature vector and the three-dimensional feature vector; the present invention combines a deep convolutional neural network, an RGB image and an optical flow map, and uses an efficient data extraction algorithm, which not only further improves the accuracy of human motion recognition, but also eliminates frame drops and freezes in the real-time video human motion recognition process.

Description

Technical Field

[0001] The present application relates to the field of educational information technology, and more specifically, to a real-time human motion recognition method in a classroom, a computer device, and a readable medium. Background Art

[0002] In recent years, the rapid development of technologies such as artificial intelligence and computer vision based on deep learning has accelerated the transformation of the field of educational informatization. Concepts such as "smart education" and "artificial intelligence + education" have been continuously proposed, and the use of machines to assist teaching is becoming a research focus and hotspot, which not only reduces the teaching pressure of teachers, but also helps teachers formulate teaching plans.

[0003] Classroom teaching is the main battlefield for school education. Student classrooms are also paying more and more attention to informatization and intelligence. The use of artificial intelligence technology to analyze and evaluate student classrooms is of great significance to education and teaching. Students' classroom behavior is the main component of classroom evaluation. By analyzing students' classroom behavior, we can not only understand students' learning status, but also warn teachers when students engage in dangerous non-learning behaviors.

[0004] Chinese patent application No. 201910674395.6 discloses a method for human behavior recognition in a student classroom based on target detection. This method can recognize common classroom behaviors such as sitting, writing, lying on the table, looking around, raising hands, and standing up, but ignores some important non-learning behaviors in the classroom; and this method uses a multi-channel 3D CNN network as a benchmark model for training and testing. Although it reduces the amount of calculation to a certain extent, it cannot perform real-time human action recognition.

[0005] Chinese patent application No. 201711200452.4 discloses a real-time human action recognition method. The method first obtains a depth image of a video, then extracts human skeleton data based on the depth image and normalizes it. The training data used needs to mark the starting position of the action in advance and set a set of key points. Although this method realizes real-time human action recognition, it requires additional extraction of human skeleton data, which increases the amount of calculation and complexity.

[0006] In summary, the existing human motion recognition methods for student classrooms have the following main shortcomings: 1. They mainly focus on students' learning behaviors, thus ignoring important non-learning behaviors; 2. The network model used cannot perform real-time human motion recognition; 3. They cannot make full use of video frame data, and the acquisition of data such as the human skeleton is relatively cumbersome. Summary of the invention

[0007] In view of at least one defect or improvement need of the prior art, the present invention provides a method for real-time human action recognition in a classroom, a computer device and a readable medium. The method uses a deep convolutional neural network as a benchmark model. The network adopts a wide residual structure and integrates two-dimensional convolution and three-dimensional convolution, which effectively reduces the amount of calculation. In addition, the method uses RGB video frames and optical flow maps as inputs, and does not require additional calculations.

[0008] To achieve the above object, according to a first aspect of the present invention, a method for real-time human motion recognition in a classroom is provided, which comprises:

[0009] Obtaining a classroom video to be analyzed, and extracting an RGB image and an optical flow map from the classroom video in real time;

[0010] Inputting the RGB image or the RGB image and the optical flow map into a trained hybrid convolutional neural network to obtain a human motion classification result in the classroom video; the human motion classification result is a predetermined learning behavior or a non-learning behavior;

[0011] The hybrid convolutional neural network has a wide residual network structure, including a two-dimensional convolutional network and a three-dimensional convolutional network connected in parallel with the two-dimensional convolutional network, which are respectively used to extract two-dimensional feature vectors and three-dimensional feature vectors from RGB images and optical flow maps; and recognize human body movements based on a feature map obtained by fusing the two-dimensional feature vector and the three-dimensional feature vector.

[0012] Preferably, in the above-mentioned real-time human action recognition method in the classroom, the RGB image and the optical flow map are serialized data of the same period;

[0013] Real-time extraction of RGB images and optical flow maps from the lecture video, including:

[0014] Use the opencv algorithm library to extract RGB images in real time;

[0015] The TV-L1 algorithm is used to extract the optical flow map of adjacent frames from the classroom video. The TV-L1 algorithm is as follows:

[0016]

[0017] Among them, I0 and I1 represent two adjacent RGB images, u:Ω→R 2 is the disparity map that needs to be obtained, represents the fidelity of the image data, represents the regularization term, and the range of λ weight is between data fidelity and regularization. The following formula can be obtained:

[0018]

[0019] The u that makes E reach the minimum value is the optical flow map of adjacent frames that needs to be obtained.

[0020] Preferably, the above-mentioned real-time human action recognition method in the classroom inputs the RGB image into a trained hybrid convolutional neural network to obtain the human action classification result in the classroom video, including:

[0021] Select N RGB images and add them to the first image sequence;

[0022] Extracting the first N / 2 RGB images from the first image sequence and adding them to the second image sequence; the second image sequence is an image sequence having N RGB images, and before each addition of N / 2 new RGB images, the first N / 2 RGB images in the sequence are deleted, so that the second image sequence maintains an image sequence of length N;

[0023] Inputting N RGB images in the second image sequence into the deep convolutional neural network to obtain candidate prediction results Pc;

[0024] Output the final prediction result P: = (P+Pc) / 2.

[0025] Preferably, the above-mentioned real-time human action recognition method in the classroom inputs the RGB image and the optical flow map into a trained hybrid convolutional neural network to obtain the human action classification results in the classroom video, including:

[0026] Add N RGB images to the first image sequence, and add M optical flow images to the first optical flow sequence;

[0027] Extracting the first N / 2 image sequences from the first image sequence and adding them to the second image sequence; extracting the first M / 2 optical flow sequences from the first optical flow sequence and adding them to the second optical flow sequence;

[0028] The second image sequence and the second optical flow sequence are an image sequence and an optical flow sequence having N RGB images and M optical flow maps, respectively, and the first N / 2 image sequences and M / 2 optical flow sequences are deleted before new sequence data is added each time, so that the second image sequence and the second optical flow sequence are always image sequences and optical flow sequences with lengths of N and M;

[0029] Inputting N RGB images in the second image sequence into the deep convolutional neural network to obtain a candidate prediction result Pc1, and inputting M optical flow images in the second optical flow sequence into the deep convolutional neural network to obtain a candidate prediction result Pc2;

[0030] Output the final prediction result P: = (P + Pc) / 2; wherein, Pc: = (x * Pc1 + y * Pc2), x and y are the weights of the candidate prediction results Pc1 and Pc2 respectively.

[0031] Preferably, in the above-mentioned real-time human motion recognition method in the classroom, in the mixed-dimensional convolutional neural network, the three-dimensional convolutional network serves as the residual connection part of the two-dimensional convolutional network, and the two-dimensional convolutional network and the three-dimensional convolutional network are combined in the following manner:

[0032] [B*S,C,H,W]→[B,C,S,H,W]

[0033] Among them, B represents batch size, S represents the number of input images, C, H, and W represent the number of channels and the height and width of the feature map respectively.

[0034] Preferably, in the above-mentioned classroom real-time human action recognition method, the calculation method of the deep convolutional neural network error is:

[0035]

[0036] Among them, l(x i ) represents the label value of the i-th input data, p(x i ) is the predicted value of the i-th input data after passing through the convolutional network, Loss(x i ) represents the loss function.

[0037] Preferably, the above-mentioned classroom real-time human action recognition method,

[0038] The N RGB images may be N consecutive images or N images with the same interval;

[0039] The M optical flow maps may be M continuous optical flow maps, or may be M optical flow maps with the same interval; the M optical flow maps include optical flow maps in both the X direction and the Y direction.

[0040] Preferably, the above-mentioned classroom real-time human motion recognition method also includes:

[0041] If the human motion classification result is a learning behavior, the human motion classification result is recorded in a log file;

[0042] If the human body movement classification result is a non-learning behavior, a warning signal is issued.

[0043] According to the second aspect of the present invention, there is also provided a computer device, which includes at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of any one of the above-mentioned methods for real-time human motion recognition in the classroom.

[0044] According to the third aspect of the present invention, there is also provided a computer-readable medium storing a computer program executable by a computer device, which, when the computer program is run on the computer device, enables the computer device to execute the steps of any one of the above-mentioned methods for real-time human motion recognition in the classroom.

[0045] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0046] (1) In the present invention, the deep convolutional neural network adopts a wide residual structure and a combination of two-dimensional convolution and three-dimensional convolution to extract two-dimensional feature vectors and three-dimensional feature vectors from RGB images and optical flow maps, respectively, and recognizes human movements based on the feature map obtained by fusing the two-dimensional feature vectors and the three-dimensional feature vectors; the network parameters are further reduced while ensuring the accuracy, so that it can realize real-time human movement recognition; the RGB image and the optical flow map are used as the input data of the deep convolutional neural network, and there is no need to additionally extract human skeleton data for calculation, thereby reducing the amount of calculation.

[0047] (2) The recognition of human body movements in videos is divided into learning behaviors and non-learning behaviors, which allows for a comprehensive and objective analysis of classroom videos and makes up for the shortcomings of simply analyzing learning behaviors. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A schematic diagram of a flow chart of a method for real-time human action recognition in a classroom based on RGB images and optical flow graphs provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of the network structure of a mixed-dimensional convolutional neural network provided in an embodiment of the present application;

[0051] Figure 3 An example schematic diagram of a wide residual network provided in an embodiment of the present application;

[0052] Figure 4 A schematic diagram of a real-time extraction algorithm for RGB images and optical flow maps provided in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of real-time human motion recognition feedback provided in an embodiment of the present application;

[0054] Figure 6 It is a logical block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0056] In the following description, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The following description provides multiple embodiments of the present invention, and different embodiments can be replaced or combined, so the present invention can also be considered to include all possible combinations of the same and / or different embodiments described. Therefore, if one embodiment includes features A, B, and C, and another embodiment includes features B and D, then the present invention should also be considered to include embodiments containing one or more of all other possible combinations of A, B, C, and D, even though the embodiment may not be clearly described in the following text.

[0057] The following description provides examples and does not limit the scope, applicability or examples set forth in the claims. Changes may be made to the functions and arrangements of the elements described without departing from the scope of the present invention. Various processes or components may be appropriately omitted, substituted or added to each example. For example, the described method may be performed in an order different from the described order, and various steps may be added, omitted or combined. In addition, the features described in some examples may be combined in other examples.

[0058] The technical idea of ​​the present invention is: collect classroom video data through cameras and other devices, use Opencv algorithm library and TV-L1 algorithm to obtain RGB images and optical flow map data of the video, set labels for a large amount of acquired data and input them as training samples into the constructed deep convolutional neural network for training, and save the trained model; specifically, human action labels are divided into learning behaviors and non-learning behaviors, learning behaviors include raising hands, standing up, writing, etc., and non-learning behaviors include throwing, falling, eating, etc. Among them, the deep convolutional neural network adopts a wide residual structure, and combines two-dimensional convolution with three-dimensional convolution. Then, RGB images and optical flow map data are extracted from the real-time classroom video to be analyzed and input into the trained model for real-time human action recognition. The recognition results are divided into two categories, one is learning behavior and the other is non-learning behavior. The present invention combines deep convolutional neural networks, RGB images and optical flow maps, and uses efficient data extraction algorithms, which not only further improves the accuracy of human action recognition, but also does not cause frame drops and freezes in the real-time video human action recognition process.

[0059] Figure 1 A flow chart of a method for real-time classroom human action recognition based on RGB images and optical flow graphs provided in this embodiment; see Figure 1 , the method comprises the following steps:

[0060] S1 obtains a classroom video to be analyzed, and extracts an RGB image and an optical flow map from the classroom video in real time;

[0061] In this embodiment, conventional cameras and other equipment are used to collect classroom video data, and the RGB images and optical flow maps extracted from the classroom video in real time are serialized data of the same time period.

[0062] Specifically, firstly, real-time video data is collected through a camera, and then each video frame is extracted through the Opencv algorithm library, and the optical flow map is extracted through the TV-L1 algorithm; the TV-L1 algorithm is as follows:

[0063]

[0064] Among them, I0 and I1 represent two adjacent RGB images, u:Ω→R 2 is the disparity map that needs to be obtained, represents the fidelity of the image data, represents the regularization term, and the range of λ weight is between data fidelity and regularization. The following formula can be obtained:

[0065]

[0066] The u that makes E take the minimum value is the optical flow map of the adjacent frames.

[0067] Optionally, the extracted RGB image and optical flow map are adjusted to a uniform size, such as 224*224; and the extracted RGB image and optical flow map are encoded in chronological order, and the encoding format is frame000001, frame0000002...

[0068] S2 inputs the RGB image or the RGB image and the optical flow map into a trained hybrid convolutional neural network, wherein the hybrid convolutional neural network is a deep convolutional neural network with a wide residual network structure, which includes a two-dimensional convolutional network and a three-dimensional convolutional network connected in parallel with the two-dimensional convolutional network, and is used to extract a two-dimensional feature vector and a three-dimensional feature vector from the RGB image and the optical flow map, respectively;

[0069] Based on the feature map obtained by fusing the two-dimensional feature vector and the three-dimensional feature vector, human body movements are recognized, and human body movement classification results in the classroom video are generated; the human body movement classification results are pre-defined learning behaviors or non-learning behaviors;

[0070] Specifically, it can be divided into two methods: using optical flow map data and not using optical flow map data;

[0071] (1) If optical flow data is not used:

[0072] Add N RGB images to the incoming_frames image sequence, where the N image sequences can be selected from N consecutive images or N images with the same interval;

[0073] Extract the first N / 2 image sequences from the incoming_frames image sequence and add the image sequences to working_frames;

[0074] Working_frames is an image sequence having N RGB images, and each time before a new image sequence is added to Working_frames, the first N / 2 image sequences are deleted, so that Working_frames is always an image sequence with a length of N;

[0075] Input the images in Working_frames into the deep convolutional neural network, extract the two-dimensional image feature vector from the N RGB images by the two-dimensional convolutional network in the deep convolutional neural network, extract the three-dimensional image feature vector from the N RGB images by the three-dimensional convolutional network, fuse the two-dimensional image feature vector and the three-dimensional image feature vector, perform classification based on the fused image feature vector, and obtain the candidate prediction result Pc;

[0076] The final prediction result P: = (P+Pc) / 2.

[0077] (2) If using optical flow data:

[0078] Add N RGB images to the Incoming_frames image sequence, and add M optical flow maps to the Incoming_flows optical flow sequence, wherein adjacent picture frames are selected from the Incoming_frames image sequence, and the N picture sequences may be selected from N consecutive pictures or N pictures with the same interval; the M optical flow maps may be M consecutive optical flow maps or M optical flow maps with the same interval; however, the M optical flow maps include optical flow maps in both the X direction and the Y direction;

[0079] Extract the first N / 2 and M / 2 image sequences and optical flow sequences from the incoming_frames image sequence and the incoming_flows respectively, and add these sequence data to working_frames and working_flows respectively;

[0080] The Working_frames and Working_flows are image sequences and optical flow sequences with N RGB images and M optical flow maps, respectively, and the first N / 2 and M / 2 image sequences and optical flow data are deleted before new sequence data is added to Working_frames and Working_flows each time, so that Working_frames and Working_flows are always image sequences and optical flow sequences with lengths of N and M;

[0081] Input the pictures in Working_frames into the deep convolutional neural network, the two-dimensional convolutional network in the deep convolutional neural network extracts two-dimensional image feature vectors from N RGB images, and the three-dimensional convolutional network extracts three-dimensional image feature vectors from N RGB images, fuses the two-dimensional image feature vectors and the three-dimensional image feature vectors, and classifies based on the fused image feature vectors to obtain the candidate prediction result Pc1;

[0082] The optical flow in Working_flows is input into the deep convolutional neural network, the two-dimensional convolutional network in the deep convolutional neural network extracts a two-dimensional image feature vector from the M optical flow maps, and the three-dimensional convolutional network extracts a three-dimensional image feature vector from the M optical flow maps, the two-dimensional image feature vector and the three-dimensional image feature vector are fused, and classification is performed based on the fused image feature vector to obtain a candidate prediction result Pc2;

[0083] Pc: = (x*Pc1+y*Pc2), x and y are the weights of the prediction results Pc1 and Pc2 respectively;

[0084] The final prediction result P: = (P+Pc) / 2.

[0085] In this embodiment, the input of the network can be an RGB image or an RGB image and an optical flow map. If a high accuracy requirement is required, both the RGB image and the optical flow map data can be used as input. If a high speed requirement is required for real-time action recognition, only the RGB image can be selected as input.

[0086] This scheme uses RGB video frames and optical flow maps as input, and fuses the features of RGB video frames and optical flow maps for human action recognition. It does not require additional extraction of human skeleton data, thus reducing the amount of calculation.

[0087] Figure 2 A schematic diagram of the network structure of a mixed-dimensional convolutional neural network provided in this embodiment; Figure 2 As shown in the figure, the three-dimensional convolutional network in the mixed-dimensional convolutional neural network serves as the residual connection part of the two-dimensional convolutional network. The two-dimensional convolutional network and the three-dimensional convolutional network are combined in the following way:

[0088] [B*S,C,H,W]→[B,C,S,H,W]

[0089] Among them, B represents batch size, S represents the number of input images, C, H, and W represent the number of channels and the height and width of the feature map respectively.

[0090] In a specific example, the two-dimensional convolutional network uses the BN-inception network as the base network, the three-dimensional convolutional network uses the 3DResNet18 network as the base network, and the three-dimensional convolutional network is used as the residual connection part of the two-dimensional convolutional network. In this way, the mixed-dimensional convolutional neural network uses two layers of residual connections in width to form a wide residual network structure; and due to shortcut connections, the residual network effectively avoids the degradation problem. The residual network structure of the mixed-dimensional convolutional neural network is expressed as follows:

[0091] F(x)=H(x)-x

[0092] Among them, F(x) represents the residual of the network, H(x) is the predicted value, and x represents the input of the convolution layer.

[0093] The hybrid dimensional convolutional neural network provided in this embodiment adopts a wide residual structure and integrates a two-dimensional convolutional network and a three-dimensional convolutional network, which effectively reduces the amount of calculation; while ensuring the accuracy, the network parameters are further reduced, so that it can realize real-time human motion recognition.

[0094] Figure 3 For a schematic diagram of the training process of the mixed-dimensional convolutional neural network provided in this embodiment, see Figure 3 , the training process includes:

[0095] Obtain training samples, which are RGB images and optical flow map data extracted from classroom videos, and each training sample has a human action label; specifically, the human action label is divided into learning behavior and non-learning behavior, learning behavior includes raising hands, standing up, writing, etc., and non-learning behavior includes throwing, falling, eating, etc.; in a specific example, when RGB images and optical flow maps are used at the same time, the training of RGB images and optical flow maps is performed independently. When training RGB images, the input of the deep convolutional neural network is 16 images randomly selected from the RGB image sequence; when training optical flow maps, the input is 6 optical flow images randomly selected from the optical flow map sequence, including 3 optical flow maps in the X direction and 3 in the Y direction. The extraction method of RGB images and optical flow maps can be found in the attached Figure 4 Taking RGB images as an example, the data input into the deep convolutional neural network comes from Working_figure and Incoming_figure respectively. The second half of the data in Working_figure is combined with the first half of the data in Incoming_figure and input into the deep convolutional neural network together. Among them, the RGB image and the optical flow map can be continuous or discontinuous.

[0096] The training samples and their corresponding human motion labels are input into a mixed-dimensional convolutional neural network, and the human motion prediction values ​​corresponding to the training samples are output through the mixed-dimensional convolutional neural network to be trained;

[0097] The loss function is calculated based on the human action label and the human action prediction value output by the mixed-dimensional convolutional neural network, and the model parameters of the mixed-dimensional convolutional neural network to be trained are reversely adjusted until the loss function is minimized to obtain a trained mixed-dimensional convolutional neural network model. The calculation method of the loss function is:

[0098]

[0099] Among them, l(x i ) represents the label value of the i-th input data, p(x i ) is the predicted value of the i-th input data after passing through the convolutional network, Loss(x i ) represents the loss function.

[0100] S3 outputs the real-time human motion recognition results in the classroom to help teachers manage and teach the classroom;

[0101] Figure 5 This is a schematic diagram of real-time human motion recognition feedback provided by this embodiment; see Figure 5 ,The results of human motion recognition can be divided into two categories, namely learning ,behavior and non-learning behavior. According to the final classification results of the deep ,convolutional neural network, the classroom human motion is analyzed. Specifically, when the result of human motion recognition is ,learning behavior, this behavior is recorded in the log to help teachers specify ,teaching plans. When the result of human motion recognition is non-learning ,behavior, a warning is issued to teachers to help them manage ,the classroom. Real-time human motion recognition and analysis of the classroom can ,not only help teachers better manage the classroom, but also provide support for the formulation of ,teacher’s teaching strategies.

[0102] It should be noted that although the operations of the methods of the embodiments of this specification are described in a specific order in the above-mentioned embodiments, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flowchart can change the order of execution. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step for execution, and / or one step can be decomposed into multiple steps for execution.

[0103] This embodiment also provides a computer device, such as Figure 6 As shown, it includes at least one processor and at least one memory, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the processor executes the steps of the above-mentioned classroom real-time human motion recognition method. In this embodiment, the types of processor and memory are not specifically limited. For example, the processor can be a microprocessor, a digital information processor, an on-chip programmable logic system, etc.; the memory can be a volatile memory, a non-volatile memory, or a combination thereof, etc.

[0104] The computer device may also communicate with one or more external devices (such as a keyboard, a pointing terminal, a display, etc.), may communicate with one or more terminals that enable a user to interact with the computer device, and / or communicate with any terminal that enables the computer device to communicate with one or more other computing terminals (such as a network card, a modem, etc.). Such communication may be performed through an input / output (I / O) interface. Furthermore, the computer device may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter.

[0105] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0106] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for real-time human action recognition in classroom, characterized in that: include: Obtaining a classroom video to be analyzed, and extracting an RGB image and an optical flow map from the classroom video in real time; The RGB image and the optical flow map are serialized data of the same period; wherein, the RGB image and the optical flow map are extracted from the classroom video in real time, including: Use the opencv algorithm library to extract RGB images in real time; The TV-L1 algorithm is used to extract the optical flow map of adjacent frames from the classroom video. The TV-L1 algorithm is as follows: Among them, I0 and I1 represent two adjacent RGB images, u:Ω→R 2 is the disparity map that needs to be obtained, represents the fidelity of the image data, represents the regularization term, and the range of λ weight is between data fidelity and regularization. The following formula can be obtained: The u that makes E reach the minimum value is the optical flow map of the adjacent frames that needs to be obtained; Input the RGB image or the RGB image and the optical flow map into a trained hybrid convolutional neural network to obtain a human action classification result in a classroom video; the human action classification result is a predetermined learning behavior or a non-learning behavior; wherein, input the RGB image and the optical flow map into a trained hybrid convolutional neural network to obtain a human action classification result in a classroom video, including: adding N RGB images to a first image sequence, adding M optical flow maps to the first optical flow sequence; extracting the first N / 2 image sequences from the first image sequence and adding them to a second image sequence; extracting the first M from the first optical flow sequence / 2 optical flow sequences and add them to the second optical flow sequence; the second image sequence and the second optical flow sequence are respectively an image sequence and an optical flow sequence having N RGB images and M optical flow maps, and the first N / 2 image sequences and M / 2 optical flow sequences are deleted before new sequence data is added each time, so that the second image sequence and the second optical flow sequence are always image sequences and optical flow sequences with lengths of N and M; the N RGB images in the second image sequence are input into a deep convolutional neural network to obtain a candidate prediction result Pc1, and the M optical flow maps in the second optical flow sequence are input into the deep convolutional neural network to obtain a candidate prediction result Pc2; Output the final prediction result P: = (P + Pc) / 2; wherein, Pc: = (x * Pc1 + y * Pc2), x, y are the weights of the candidate prediction results Pc1, Pc2 respectively; the hybrid convolutional neural network has a wide residual network structure, including a two-dimensional convolutional network and a three-dimensional convolutional network in parallel with the two-dimensional convolutional network, which are respectively used to extract two-dimensional feature vectors and three-dimensional feature vectors from RGB images and optical flow maps; recognize human body movements based on the feature map obtained by fusing the two-dimensional feature vector and the three-dimensional feature vector.

2. The method for real-time human motion recognition in a classroom as claimed in claim 1, characterized in that: The RGB image is input into the trained hybrid convolutional neural network to obtain the human action classification results in the classroom video, including: Select N RGB images and add them to the first image sequence; Extracting the first N / 2 RGB images from the first image sequence and adding them to the second image sequence; the second image sequence is an image sequence having N RGB images, and before each addition of N / 2 new RGB images, the first N / 2 RGB images in the sequence are deleted, so that the second image sequence maintains an image sequence of length N; Inputting N RGB images in the second image sequence into the deep convolutional neural network to obtain candidate prediction results Pc; Output the final prediction result P: = (P+Pc) / 2.

3. The real-time human motion recognition method in a classroom as claimed in claim 1, characterized in that: The mixed In the combined dimensional convolutional neural network, the three-dimensional convolutional network is used as the residual connection part of the two-dimensional convolutional network. The two-dimensional convolutional network and the three-dimensional convolutional network are combined in the following way: [B*S,C,H,W]→[B,C,S,H,W] Among them, B represents the batch size, S represents the number of input images, C, H, and W represent the number of channels and the height and width of the feature map, respectively.

4. The method for real-time human motion recognition in a classroom as claimed in claim 1, characterized in that: The calculation method of the deep convolutional neural network error is: Among them, l(x i ) represents the label value of the i-th input data, p(x i ) is the predicted value of the i-th input data after passing through the convolutional network, Loss(x i ) represents the loss function.

5. The method for real-time human motion recognition in a classroom as claimed in claim 1, characterized in that: The N RGB images may be N consecutive images or N images with the same interval; The M optical flow maps may be M continuous optical flow maps, or may be M optical flow maps with the same interval; the M optical flow maps include optical flow maps in both the X direction and the Y direction.

6. The method for real-time human motion recognition in a classroom as claimed in claim 1, characterized in that: Also includes: If the human motion classification result is a learning behavior, the human motion classification result is recorded in a log file; If the human body movement classification result is a non-learning behavior, a warning signal is issued.

7. A computer device, characterized in that: The method comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of the method according to any one of claims 1 to 6.

8. A computer-readable medium, characterized in that It stores a computer program executable by a computer device. When the computer program is run on the computer device, the computer device executes the steps of the method according to any one of claims 1 to 6.

Images