Teacher behavior recognition method and system based on multi-level semantic spatiotemporal graph convolution

Abstract

The present invention discloses a teacher behavior recognition method and system based on multi-level semantic spatiotemporal graph convolution. The method comprises: obtaining video frames containing teacher behavior information to be recognized, and extracting human skeleton sequence data from the video frames; inputting the skeleton sequence data into a pre-trained multi-level semantic spatiotemporal graph convolutional network; wherein the multi-level semantic spatiotemporal graph convolutional network includes several spatiotemporal graph convolutional layers and several semantic fusion layers; the several spatiotemporal graph convolutional layers and the several semantic fusion layers are arranged in series and at intervals, and the semantic fusion layers are arranged after the low-level spatiotemporal graph convolutional layers; the multi-level semantic spatiotemporal graph convolutional network recognizes the behavior information in the skeleton sequence data based on spatiotemporal features and high-level semantic features, and outputs the recognition results; based on the recognition results, the teacher behavior is analyzed to obtain different types of behavior information and display them. This method achieves accurate recognition of teacher behavior.

Description

Technical Field

[0001] The present invention belongs to the field of digital education technology, and specifically relates to a teacher behavior recognition method and system based on multi-level semantic spatiotemporal graph convolution. Background Art

[0002] With the continuous advancement of science and technology, digital education is becoming a key trend driving the development of education. In this era of change, big data and artificial intelligence technologies are offering new possibilities for education. Digital education has garnered particular attention. In the process of digital education, classroom teaching evaluation is particularly important. Classroom teaching evaluation primarily encompasses both teaching and learning. Traditional methods for evaluating teachers' teaching processes include on-site observation and surveillance. However, these methods suffer from low scoring efficiency and high subjectivity. A new computer vision-based method can more efficiently and accurately identify and analyze teachers' teaching behaviors. This innovative approach is expected to address the shortcomings of traditional evaluation methods, improve objectivity and efficiency, and further promote the development of digital education.

[0003] Action recognition is a crucial research area in computer vision. Its primary task is to identify and understand human actions from video or image sequences. Deep learning-based algorithms are currently the mainstream in this field. Deep learning-based action recognition can be categorized into RGB image-based and skeleton-based approaches. Compared to RGB images, skeleton data is more robust against complex backgrounds, variations in body scale, perspective, and motion speed. Furthermore, the development of sensors like the Microsoft Kinect and advanced human pose estimation algorithms has made it easier to acquire accurate 3D human skeleton data. Therefore, skeleton-based action recognition methods have gradually become the mainstream algorithm. Among current skeleton-based action recognition models, the spatiotemporal graph convolutional network (STN) uses skeleton sequence data as input, extracting spatial features using graph convolution and temporal features using one-dimensional convolution.

[0004] However, existing graph convolution-based skeleton behavior recognition methods mainly construct topological maps based on human skeleton points. When aggregating features, they only perform it at the relatively subtle level of skeleton points, and cannot enable information interaction between different body parts at a higher level, resulting in inaccurate recognition results.

[0005] In summary, traditional teaching evaluation methods have problems such as low scoring efficiency and strong subjectivity; and the existing skeleton behavior recognition method based on graph convolution cannot directly learn higher-level semantic features, which affects the accuracy of the recognition results. Summary of the Invention

[0006] In order to solve the above problems existing in the prior art, the present invention provides a teacher behavior recognition method and system based on multi-level semantic spatiotemporal graph convolution. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0007] In the first aspect, the present invention proposes a teacher behavior recognition method based on multi-level semantic spatiotemporal graph convolution, comprising:

[0008] Obtaining a video frame including information about the teacher's behavior to be identified, and extracting human skeleton sequence data from the video frame;

[0009] The skeleton sequence data is input into a pre-trained multi-level semantic spatiotemporal graph convolutional network; wherein the multi-level semantic spatiotemporal graph convolutional network includes several spatiotemporal graph convolutional layers and several semantic fusion layers; the several spatiotemporal graph convolutional layers and the several semantic fusion layers are arranged in series, and the semantic fusion layers are arranged after the low-level spatiotemporal graph convolutional layers; the spatiotemporal graph convolutional layers are used to extract the spatiotemporal features of the skeleton sequence data, and the semantic fusion layers are used to extract the high-level semantic features of the skeleton sequence data; the multi-level semantic spatiotemporal graph convolutional network recognizes the behavioral information in the skeleton sequence data based on the spatiotemporal features and the high-level semantic features, and outputs the recognition results;

[0010] Based on the recognition results, the teacher's behavior is analyzed, and different types of behavior information are obtained and displayed.

[0011] In a second aspect, the present invention provides a teacher behavior recognition system based on multi-level semantic spatiotemporal graph convolution, which is used to implement the method provided in the first aspect of the present invention, including:

[0012] A data acquisition module is used to acquire video frames including the teacher behavior information to be identified and extract human skeleton sequence data from the video frames;

[0013] The data processing module includes a trained multi-level semantic spatiotemporal graph convolutional network, which is used to identify behavioral information in skeleton sequence data based on spatiotemporal features and high-level semantic features, and output the recognition results;

[0014] The multi-level semantic spatiotemporal graph convolutional network includes several spatiotemporal graph convolutional layers and several semantic fusion layers. The spatiotemporal graph convolutional layers and the semantic fusion layers are arranged in series, and the semantic fusion layers are arranged after the low-level spatiotemporal graph convolutional layers. The spatiotemporal graph convolutional layers are used to extract the spatiotemporal features of the skeleton sequence data, and the semantic fusion layers are used to extract the high-level semantic features of the skeleton sequence data.

[0015] The display module is used to analyze the teacher's behavior based on the recognition results, obtain different types of behavior information and display them in real time.

[0016] Beneficial effects of the present invention:

[0017] The teacher behavior recognition method based on multi-level semantic spatiotemporal graph convolution proposed in the present invention constructs and trains a multi-level semantic spatiotemporal graph convolution network including a multi-level spatiotemporal graph convolution layer and a semantic fusion layer. By learning the spatiotemporal features and high-level semantic features of skeleton sequence data, it achieves accurate recognition of teacher behavior, avoids the problems of low scoring efficiency and strong subjectivity in scoring in traditional teaching evaluation methods, and improves the recognition accuracy.

[0018] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 1 is a flow chart of a teacher behavior recognition method based on multi-level semantic spatiotemporal graph convolution provided by an embodiment of the present invention;

[0020] Figure 2 Schematic diagram of the structure of a multi-level semantic spatiotemporal graph convolutional network provided by an embodiment of the present invention;

[0021] Figure 3 Schematic diagram of the structure of the spatiotemporal graph convolution layer provided by an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of the structure of the semantic fusion layer provided by an embodiment of the present invention;

[0023] Figure 5 is a schematic diagram of different limb positions of a human skeleton provided by an embodiment of the present invention;

[0024] Figure 6 1 is a structural block diagram of a teacher behavior recognition device based on multi-level semantic spatiotemporal graph convolution provided by an embodiment of the present invention;

[0025] Figure 7 This is an example diagram of a display interface provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0027] Example 1

[0028] See Figure 1 , Figure 1: This is a flow chart of a method for teacher behavior recognition based on multi-level semantic spatiotemporal graph convolution provided by an embodiment of the present invention. The method for teacher behavior recognition based on multi-level semantic spatiotemporal graph convolution provided by this embodiment specifically includes the following steps:

[0029] Step 1: Obtain a video frame containing the teacher behavior information to be identified and extract human skeleton sequence data from the video frame.

[0030] In this embodiment, the OpenPose algorithm can be used to extract skeleton data points from the video including the teacher behavior information to be identified.

[0031] Specifically, for each frame of the input video, 18 human joints are extracted as skeleton data. Each joint consists of 2D coordinates and their confidence scores. After obtaining the skeleton data for the Tth frame, a skeleton sequence of 3×T×18 dimensions is constructed, where 3 represents the 2D coordinates and confidence scores of the joints, 18 represents the number of joints, and T represents the number of video frames.

[0032] Step 2: Input the skeleton sequence data into the pre-trained multi-level semantic spatiotemporal graph convolutional network, identify the behavioral information in the skeleton sequence data based on spatiotemporal features and high-level semantic features, and output the recognition results.

[0033] In many cases, since human behavior is completed by the cooperation of various movements, the body parts can be considered as the smallest unit of action execution. For example, running is the most typical example in daily behavior, which requires the joint movement of the arms and legs; in the classroom behavior of teachers, writing on the blackboard requires the movement of the entire arm, not a certain joint. The human body can naturally be regarded as a hierarchical structure. The general cognition can divide the human body into five parts: torso, left arm, right arm, left leg, and right leg. In order to enable information interaction between skeleton point data and higher-level, different body parts, this embodiment designs a multi-level semantic spatiotemporal graph convolutional network that can directly learn high-level structural features, and trains it through a self-built dataset, so that the trained network can capture the multi-level semantic features of the skeleton data, thereby improving the recognition accuracy.

[0034] Specifically, the multi-level semantic spatiotemporal graph convolutional network designed in this embodiment includes several spatiotemporal graph convolutional layers and several semantic fusion layers; several spatiotemporal graph convolutional layers and several semantic fusion layers are spaced and arranged in series, and the semantic fusion layer is arranged after the low-level spatiotemporal graph convolutional layer; the spatiotemporal graph convolutional layer is used to extract the spatiotemporal features of the skeleton sequence data, and the semantic fusion layer is used to extract the high-level semantic features of the skeleton sequence data.

[0035] Optionally, as an implementation method, this embodiment designs a multi-level semantic spatiotemporal graph convolutional network including 9 spatiotemporal graph convolutional layers and 2 semantic fusion layers, and its structure is as follows: Figure 2 As shown in the figure, the 9 spatiotemporal graph convolution layers are denoted as L1 to L9, and the 2 semantic fusion layers are M1 and M2 respectively; the first spatiotemporal graph convolution layer L1 to the third spatiotemporal graph convolution layer L3 are connected in sequence; the first semantic fusion layer M1 is connected after the third spatiotemporal graph convolution layer L3; the fourth spatiotemporal graph convolution layer L4 is connected after the first semantic fusion layer M1; the second semantic fusion layer M2 is connected after the fourth spatiotemporal graph convolution layer L4; the fifth spatiotemporal graph convolution layer to the ninth spatiotemporal graph convolution layer L5 to L9 are connected after the second semantic fusion layer M2.

[0036] For further information, see Figure 3 , Figure 3 This is a schematic diagram of the structure of the spatiotemporal graph convolution layer provided by an embodiment of the present invention. The spatiotemporal graph convolution layer designed in this embodiment includes three modules: a graph convolution network (GCN), a temporal convolutional network (TCN), and a multi-scale adaptive feature aggregation module (MS-AA). The output data of the graph convolution module is used as the input data of the temporal convolution module and the multi-scale adaptive feature aggregation module respectively.

[0037] Specifically, each spatiotemporal convolutional layer processes the input data according to the following steps:

[0038] a) Use the graph convolution module to extract the spatial features of the input data and obtain the spatial feature map.

[0039] b) Input the spatial feature map into the temporal convolution module, so that the temporal convolution module extracts the temporal features of the input data and obtains a temporal feature map. At the same time, the spatial feature map is input into the multi-scale adaptive feature aggregation module. The spatial feature map is first spatially compressed using the pooling layer. Then, a local branch is used to generate a temporal sensitivity map based on the compressed spatial feature map, and a first feature map is generated based on the temporal sensitivity map. Then, an adaptive convolution kernel is generated based on the compressed spatial feature map using the global branch. Finally, the first feature map is convolved with the adaptive convolution kernel to obtain the output feature map of the multi-scale adaptive feature aggregation module.

[0040] c) Add the output feature map of the multi-scale adaptive feature aggregation module to the temporal feature map output by the temporal convolution module to obtain the output feature map of each level of spatiotemporal graph convolution layer, and use it as the input data of the next level of spatiotemporal graph convolution layer or semantic fusion layer.

[0041] It should be noted that the specific network structures of the graph convolution module, the temporal convolution module, and the multi-scale adaptive feature aggregation module can be implemented by referring to the existing technology, and this embodiment will not be described in detail here.

[0042] Furthermore, the semantic fusion layer designed in this embodiment mainly includes a pooling layer, two convolutional layers, a normalization layer, etc. For details, please refer to Figure 4 , Figure 4 This is a schematic diagram of the structure of the semantic fusion layer provided by an embodiment of the present invention. The semantic fusion layer designed in this embodiment extracts high-level semantic features according to the following steps:

[0043] a) Divide the input skeleton data feature map into multiple limb parts in the spatial dimension, and record the divided features as the first eigenvector.

[0044] Assume that the input skeleton data feature map is Where C represents the number of channels, T represents the number of frames, and V represents the number of nodes. According to the rules of limb division, the skeleton points are divided into N parts, where N is the number of limb parts. The feature of each limb part is the sum of the features of the skeleton points it contains. The feature map at this time is It can also be called the first eigenvector.

[0045] Optional, such as Figure 5 As shown, Figure 5 The figure is a schematic diagram illustrating different limb positions of a human skeleton provided by an embodiment of the present invention. In accordance with common knowledge, this embodiment can spatially divide the input skeleton data feature map into five limb parts: torso, left arm, right arm, left leg, and right leg. Furthermore, during implementation, those skilled in the art may further divide limb parts based on practical circumstances, and this embodiment does not specifically limit this.

[0046] b) Perform an average pooling operation on the first eigenvector, retaining the feature channel dimension and spatial dimension, to obtain the second eigenvector.

[0047] Specifically, since each sample contains only one action, the temporal changes of each limb part are not considered. The features of the limb parts in the skeleton sequence are uniformly extracted and average pooling operation is performed first, retaining the feature channel dimension C and spatial dimension N of the skeleton data feature map. The feature map at this time is It is recorded as the second eigenvector.

[0048] c) Perform channel dimension reduction and dimension increase operations on the second eigenvector through two convolutional layers to obtain the third eigenvector.

[0049] Specifically, we first reduce the number of channels to Then the number of channels is increased to C×N through the second convolutional layer, and the third eigenvector of (C×N)×1×1 dimension is obtained.

[0050] d) Use the Softmax function to normalize the third eigenvector to obtain the score of each limb part, which is recorded as the fourth eigenvector.

[0051] Specifically, the third eigenvector is first converted into a C×N×1-dimensional vector, and then the vector is normalized using the Softmax function. The new vector obtained is the score of N limb parts in the number of channels, which indicates the importance of each limb part to this sample action and is recorded as the fourth eigenvector.

[0052] e) reshaping the skeleton data feature map of the input semantic fusion layer based on the fourth eigenvector to obtain an output feature map of the semantic fusion layer.

[0053] First, the fourth eigenvector is expanded in the spatial dimension, and the score of each limb part is mapped one-to-one with the skeleton points contained in that limb part to obtain the fifth eigenvector. Then, the fifth eigenvector is replicated in the temporal dimension to obtain the weight map of the limb part. Finally, based on the weight map, the skeleton data feature map of the input semantic fusion layer is reshaped and superimposed with the skeleton data feature map of the input semantic fusion layer to serve as the output feature map of the semantic fusion layer.

[0054] Specifically, due to the previous pooling operation, the limb part weight map does not contain the time dimension T and needs to be expanded and replicated. Specifically, the spatial expansion is first performed based on the skeleton points contained in each limb part. The score of each limb part is mapped one-to-one to the skeleton points it contains to obtain a new (C, 1, V)-dimensional vector. This is then replicated in time to obtain the final limb part weight map.

[0055] After obtaining the weight map, the original input skeleton data feature map will be reshaped. The formula can be expressed as:

[0056] f out =δ(f in ⊙W p +f in );

[0057] Where, f out represents the output features of the semantic fusion layer, δ represents the activation function ReLu, f in represents the input features of the semantic fusion layer, W p represents the weight graph, and ⊙ represents the element-wise multiplication operation.

[0058] The reshaped feature map f in ⊙W p, which already has the feature representation of the limb part level. Finally, in order to retain the original skeleton point level features, the input feature f in It is superimposed with the reshaped feature map and serves as the output of the semantic fusion layer.

[0059] For further information, please see Figure 2 , where the multi-level semantic spatiotemporal graph convolutional network also includes a global average pooling layer (GAP), a fully connected layer (FC), and a Softmax layer connected in sequence to the last spatiotemporal graph convolutional layer; the entire multi-level semantic spatiotemporal graph convolutional network forms a 13-layer serial structure, which is: the first spatiotemporal convolutional layer L1→the second spatiotemporal convolutional layer L2→the third spatiotemporal convolutional layer L3→the first multi-level semantic fusion layer M1→the fourth spatiotemporal convolutional layer L4→the second multi-level semantic fusion layer M2→the fifth spatiotemporal convolutional layer L5→the sixth spatiotemporal convolutional layer L6→the seventh spatiotemporal convolutional layer L7→the eighth spatiotemporal convolutional layer L8→the ninth spatiotemporal convolutional layer L9→the global average pooling layer GAP→the fully connected layer FC→the Softmax layer.

[0060] Among them, the global average pooling layer averages each channel of the output feature map of the last spatiotemporal graph convolution layer, and uses each average as a neuron for classification tasks. The fully connected layer and the Softmax layer then map the probabilities of belonging to each category to determine the behavior recognition results.

[0061] The multi-level semantic spatiotemporal graph convolutional network constructed in this embodiment utilizes multiple spatiotemporal graph convolutional layers and multiple semantic fusion layers that are cascaded at intervals, and the semantic fusion layer is set after the low-level spatiotemporal graph convolutional layer. The reason why the semantic fusion layer is set after the low-level spatiotemporal graph convolutional layer of the network in this embodiment is that in the deep layer of the network, after multiple convolution operations, each skeleton point aggregates the features of its adjacent nodes, and a single skeleton point already has relatively rich features. At this time, simply dividing the limbs according to the predefined physical structure of the human body no longer has the expected semantic information. Therefore, setting the semantic fusion layer after the low-level spatiotemporal graph convolutional layer of the network can obtain high-level semantic information in the early learning process.

[0062] After constructing the multi-level semantic spatiotemporal graph convolutional network, this embodiment uses a self-made training dataset to train it. Specifically, the training dataset is constructed according to the following method:

[0063] a) Classify different teaching video contents based on teacher behavior categories and crop videos that include teacher behavior information.

[0064] First, teacher behavior can be divided into the following four categories: writing on the blackboard, multimedia presentation, standing still, and gesturing with both hands. Writing on the blackboard means the teacher writes down important points on the blackboard; multimedia presentation means the teacher explains the content on the blackboard or multimedia screen; standing still means the teacher stands still at the podium, waiting for student feedback; gesturing with both hands means the teacher stands at the podium and uses his hands and words to explain the teaching content to students.

[0065] After determining the teacher behavior categories, you can download videos from the Internet for classification and image cropping. For example, you can download public classroom teaching videos from MOOCs.

[0066] It should be noted that when trimming the video, since this embodiment only focuses on the teacher's own behavior and the students' classroom behavior is not considered for the time being, it is ensured that the teacher occupies the main part of the video and only the teacher appears, which facilitates the human posture algorithm to extract skeleton point data; at the same time, it is ensured that the interaction between the teacher and the blackboard or multimedia screen is fully reflected in the video.

[0067] b) The length of the video is trimmed to ensure that each segment contains only one action, thereby obtaining multiple video segments of a certain length.

[0068] Optionally, the embodiment can cut the video into 5-8 second video segments, and ensure that each segment contains only one action. After cutting, check each video segment to ensure that the teacher's actions are clear and can be correctly distinguished, and there is no blurring of different types of actions.

[0069] c) The image of each video clip is mirror-flipped to enhance the video clip, and the enhanced video clips are classified according to the teacher behavior category.

[0070] Specifically, by enhancing the video clips, the dataset can be expanded, thereby improving the generalization and robustness of the model when facing different samples. After completing the enhancement operation, each video clip is placed in a folder corresponding to the action category according to the teacher's behavior category, and the video resolution under each file is changed to 340×256, 30fps. Select the Openpose human pose estimation algorithm and start extracting key points to build a skeleton dataset. Select "COCO" as the human pose model, and the coordinates of 18 key points will be extracted for each frame. Ultimately, each video will correspond to a skeleton sequence, and then the corresponding label will be generated.

[0071] d) For different categories of video clips, extract the human skeleton data of each frame of video image, obtain the skeleton sequence data corresponding to each video clip, and generate corresponding labels; the set of skeleton sequence data corresponding to all video clips is used as the training dataset.

[0072] The training dataset constructed above is sent to the constructed multi-level semantic spatiotemporal graph convolutional network for training. The specific training methods and loss function settings can refer to existing similar network training methods, and finally an algorithm model for teacher behavior recognition is obtained.

[0073] The 3×T×18 skeleton sequence data extracted from the video to be identified in step 1 is input into the trained multi-level semantic spatiotemporal graph convolutional network, and the final teacher behavior recognition result is obtained after calculation.

[0074] In addition, to ensure recognition accuracy, the algorithm will update the recognition results every 5 frames.

[0075] Step 3: Analyze the teacher's behavior based on the recognition results, obtain different types of behavior information and display them.

[0076] First, the video frame to be recognized and the current recognition result are displayed in real time, and the recognition result is updated regularly.

[0077] At the same time, based on the recognition results, the duration of different behavior categories and the proportion of different behavior categories in the current video frame to be recognized are obtained and displayed.

[0078] The teacher behavior recognition method based on multi-level semantic spatiotemporal graph convolution proposed in the present invention constructs and trains a multi-level semantic spatiotemporal graph convolution network including a multi-level spatiotemporal graph convolution layer and a semantic fusion layer. By learning the spatiotemporal features and high-level semantic features of skeleton sequence data, it achieves accurate recognition of teacher behavior, avoids the problems of low scoring efficiency and strong subjectivity in scoring in traditional teaching evaluation methods, and improves the recognition accuracy.

[0079] Example 2

[0080] Based on the above embodiment 1 and the same inventive concept, this embodiment also provides a teacher behavior recognition system based on multi-level semantic spatiotemporal graph convolution. Figure 6 , Figure 6 This is a structural block diagram of a teacher behavior recognition device based on multi-level semantic spatiotemporal graph convolution provided by an embodiment of the present invention, including:

[0081] A data acquisition module is used to acquire video frames including the teacher behavior information to be identified and extract human skeleton sequence data from the video frames;

[0082] The data processing module includes a trained multi-level semantic spatiotemporal graph convolutional network, which is used to identify behavioral information in skeleton sequence data based on spatiotemporal features and high-level semantic features, and output the recognition results;

[0083] The multi-level semantic spatiotemporal graph convolutional network includes several spatiotemporal graph convolutional layers and several semantic fusion layers. The spatiotemporal graph convolutional layers and the semantic fusion layers are arranged in series, and the semantic fusion layers are arranged after the low-level spatiotemporal graph convolutional layers. The spatiotemporal graph convolutional layers are used to extract the spatiotemporal features of the skeleton sequence data, and the semantic fusion layers are used to extract the high-level semantic features of the skeleton sequence data.

[0084] The display module is used to analyze the teacher's behavior based on the recognition results, obtain different types of behavior information and display them in real time.

[0085] Specifically, the data acquisition module and the data processing module in this embodiment are implemented according to the method of the above-mentioned embodiment 1 when performing data extraction and calculation, and the detailed process is not repeated here.

[0086] Regarding the display module, this embodiment designs a display interface with three display areas. Figure 7 , Figure 7 This is an example diagram of the display interface provided by an embodiment of the present invention, wherein the first is the playback area of the original video, which displays the current teacher behavior in real time; the second is the display area of each behavior, which displays the duration of each behavior; the third is the analysis area of the video, which displays the proportion of each behavior category in the video after the video is played.

[0087] The system provided in this embodiment can implement the method of the above-mentioned embodiment 1, thereby, the system can also realize accurate identification of teacher behavior.

[0088] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0089] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art can understand and implement other changes to the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0090] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices (equipment), or computer products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware, which are all collectively referred to as "module" or "system" herein. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The computer program is stored / distributed in a suitable medium, provided together with other hardware or as a part of hardware, or other distribution forms can be adopted, such as by the Internet or other wired or wireless telecommunication systems.

[0091] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A teacher behavior recognition method based on multi-level semantic spatiotemporal graph convolution, characterized by: include: Acquire a video frame including information about the teacher's behavior to be identified, and extract human skeleton sequence data from the video frame; The skeleton sequence data is input into a pre-trained multi-level semantic spatiotemporal graph convolutional network; wherein the multi-level semantic spatiotemporal graph convolutional network includes a plurality of spatiotemporal graph convolutional layers and a plurality of semantic fusion layers; the plurality of spatiotemporal graph convolutional layers and the plurality of semantic fusion layers are arranged in series and at intervals, and the semantic fusion layer is arranged after the low-level spatiotemporal graph convolutional layer; the spatiotemporal graph convolutional layer is used to extract the spatiotemporal features of the skeleton sequence data, and the semantic fusion layer is used to extract the high-level semantic features of the skeleton sequence data; the multi-level semantic spatiotemporal graph convolutional network recognizes the behavioral information in the skeleton sequence data based on the spatiotemporal features and the high-level semantic features, and outputs the recognition result; Analyzing the teacher's behavior based on the recognition results, obtaining different types of behavior information and displaying them; The semantic fusion layer extracts high-level semantic features according to the following steps: The input skeleton data feature map is divided into multiple limb parts in the spatial dimension, and the features after division are recorded as the first feature vector; wherein the feature of each limb part is the representation of the sum of the features of the skeleton points contained therein; Performing an average pooling operation on the first eigenvector to retain the feature channel dimension and spatial dimension to obtain a second eigenvector; Performing channel dimension reduction and dimension increase operations on the second eigenvector through two convolutional layers to obtain a third eigenvector; Normalizing the third eigenvector using a Softmax function to obtain a score for each limb part, which is recorded as a fourth eigenvector; The skeleton data feature map input to the semantic fusion layer is reshaped based on the fourth feature vector to obtain an output feature map of the semantic fusion layer.

2. The teacher behavior recognition method based on multi-level semantic spatiotemporal graph convolution according to claim 1 is characterized in that: The multiple limb parts include five parts: trunk, left arm, right arm, left leg and right leg.

3. The teacher behavior recognition method based on multi-level semantic spatiotemporal graph convolution according to claim 2 is characterized in that: Reshaping the skeleton data feature map input to the semantic fusion layer based on the fourth feature vector to obtain an output feature map of the semantic fusion layer specifically includes: Expanding the fourth eigenvector in the spatial dimension, and making one-to-one correspondence between the score of each limb part and the skeleton points contained in the limb part, to obtain a fifth eigenvector; Copying the fifth eigenvector in the time dimension to obtain a weight map of the limb part; The skeleton data feature map input to the semantic fusion layer is reshaped based on the weight map, and the reshaped feature map is superimposed on the skeleton data feature map input to the semantic fusion layer to serve as the output feature map of the semantic fusion layer.

4. The teacher behavior recognition method based on multi-level semantic spatiotemporal graph convolution according to claim 1 is characterized in that The multi-level semantic spatiotemporal graph convolutional network specifically includes 9 spatiotemporal graph convolutional layers and 2 semantic fusion layers; Among them, the first layer of spatiotemporal graph convolution layer to the third layer of spatiotemporal graph convolution layer are connected in sequence; the first layer of semantic fusion layer is connected after the third layer of spatiotemporal graph convolution layer; the fourth layer of spatiotemporal graph convolution layer is connected after the first layer of semantic fusion layer; the second layer of semantic fusion layer is connected after the fourth layer of spatiotemporal graph convolution layer; the fifth layer of spatiotemporal graph convolution layer to the ninth layer of spatiotemporal graph convolution layer are connected after the second layer of semantic fusion layer.

5. The teacher behavior recognition method based on multi-level semantic spatiotemporal graph convolution according to claim 1 is characterized in that: The multi-level semantic spatiotemporal graph convolutional network also includes a global average pooling layer, a fully connected layer, and a Softmax layer sequentially connected to the last spatiotemporal graph convolutional layer; The global average pooling layer averages each channel of the output feature map of the last spatiotemporal graph convolution layer, and uses each average as a neuron to perform the classification task. The fully connected layer and the Softmax layer then map the probability of belonging to each category to determine the behavior recognition result.

6. The teacher behavior recognition method based on multi-level semantic spatiotemporal graph convolution according to claim 1 is characterized in that: Before training the multi-level semantic spatiotemporal graph convolutional network, a training dataset is constructed as follows: Classify different teaching video contents based on teacher behavior categories and crop videos containing teacher behavior information; The length of the video is trimmed to ensure that each segment contains only one action, thereby obtaining a plurality of video segments of a certain length; Mirror-flipping the image of each video clip to enhance the video clip, and classifying the enhanced video clips according to the teacher behavior category; For different categories of video clips, the human skeleton data of each frame of video image is extracted respectively, the skeleton sequence data corresponding to each video clip is obtained, and the corresponding label is generated; the set of skeleton sequence data corresponding to all video clips is used as the training data set.

7. The teacher behavior recognition method based on multi-level semantic spatiotemporal graph convolution according to claim 6 is characterized in that: The teacher behavior categories include writing on the blackboard, multimedia presentation, standing still, and gesturing with both hands.

8. The teacher behavior recognition method based on multi-level semantic spatiotemporal graph convolution according to claim 1 is characterized in that: The teacher's behavior is analyzed based on the recognition results, and different types of behavior information are obtained and displayed, specifically including: Real-time display of the video frame to be recognized and the current recognition results; At the same time, the duration of different behavior categories and the proportion of different behavior categories in the current video frame to be identified are obtained based on the recognition result and displayed.

9. A teacher behavior recognition system based on multi-level semantic spatiotemporal graph convolution, used to implement the method according to any one of claims 1 to 8, characterized in that: include: A data acquisition module is used to acquire video frames including teacher behavior information to be identified, and extract human skeleton sequence data from the video frames; a data processing module comprising a trained multi-level semantic spatiotemporal graph convolutional network, wherein the multi-level semantic spatiotemporal graph convolutional network is used to identify behavioral information in the skeleton sequence data based on spatiotemporal features and high-level semantic features, and output recognition results; The multi-level semantic spatiotemporal graph convolutional network includes several spatiotemporal graph convolutional layers and several semantic fusion layers; the several spatiotemporal graph convolutional layers and the several semantic fusion layers are arranged in series, and the semantic fusion layer is arranged after the low-level spatiotemporal graph convolutional layer; the spatiotemporal graph convolutional layer is used to extract the spatiotemporal features of the skeleton sequence data, and the semantic fusion layer is used to extract the high-level semantic features of the skeleton sequence data; The semantic fusion layer extracts high-level semantic features in the following steps: The input skeleton data feature map is divided into multiple limb parts in the spatial dimension, and the features after division are recorded as the first feature vector; wherein the feature of each limb part is the representation of the sum of the features of the skeleton points contained therein; Performing an average pooling operation on the first eigenvector to retain the feature channel dimension and spatial dimension to obtain a second eigenvector; Performing channel dimension reduction and dimension increase operations on the second eigenvector through two convolutional layers to obtain a third eigenvector; Normalizing the third eigenvector using a Softmax function to obtain a score for each limb part, which is recorded as a fourth eigenvector; Reshape the skeleton data feature map input to the semantic fusion layer based on the fourth feature vector to obtain an output feature map of the semantic fusion layer; The display module is used to analyze the teacher's behavior based on the recognition result, obtain different types of behavior information and display them in real time.

Images