Teaching posture recognition method, training method of posture classification network and related devices

Abstract

The invention relates to a teaching posture recognition method, a training method of a posture classification network, a related device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a teaching image to be identified; performing feature extraction on the teaching image by adopting a key point detection network to obtain an image semantic feature; the key point detection network is obtained based on human body data set training; performing posture classification on the image semantic features by adopting a posture classification network to obtain teaching postures in the teaching images; the posture classification network is obtained based on teaching posture data set training; the teaching posture data set is constructed based on a simulated teaching video in a simulated teaching scene and a real teaching video in a real teaching scene. By adopting the method, the teaching posture recognition accuracy in a teaching scene can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a teaching posture recognition method, a training method for a posture classification network, related devices, computer equipment, computer-readable storage media, and computer program products. Background Technology

[0002] A teacher's posture is a direct expression of their teaching mindset and emotional state, and an important component of their teaching skills. New teachers, lacking experience, often exhibit stiff postures or other unnecessary tense movements. Experienced teachers, on the other hand, typically show lower energy levels and a lack of necessary physical gestures. Therefore, identifying teachers' postures is crucial for improving teaching quality.

[0003] Traditional methods for recognizing teaching postures using conventional neural network models, such as support vector machines, cannot achieve the fine-grained classification of detailed limb postures. For example, traditional methods suffer from low accuracy in recognizing common teaching postures such as raising hands, pointing, and putting hands behind the back. Summary of the Invention

[0004] Therefore, it is necessary to provide a teaching posture recognition method, a posture classification network training method, related devices, computer equipment, computer-readable storage media, and computer program products to address the above-mentioned technical problems, which can improve the accuracy of teaching posture recognition in teaching scenarios.

[0005] This application provides a teaching posture recognition method, including:

[0006] Acquire the teaching image to be recognized;

[0007] A keypoint detection network is used to extract features from teaching images to obtain semantic features; the keypoint detection network is trained based on a human body dataset.

[0008] A pose classification network is used to classify the poses of images based on semantic features, thereby obtaining the teaching poses in the teaching images. The pose classification network is trained on a teaching pose dataset. The teaching pose dataset is constructed from simulated teaching videos in simulated teaching scenarios and real teaching videos in real teaching scenarios.

[0009] This application also provides a teaching posture recognition device, including:

[0010] The acquisition module is used to acquire the teaching images to be recognized;

[0011] The feature extraction module is used to extract features from teaching images using a keypoint detection network to obtain semantic features of the images; the keypoint detection network is trained based on a human body dataset.

[0012] The classification module is used to classify the semantic features of images into poses using a pose classification network to obtain the teaching poses in the teaching images. The pose classification network is trained based on the teaching pose dataset. The teaching pose dataset is constructed based on simulated teaching videos in simulated teaching scenarios and real teaching videos in real teaching scenarios.

[0013] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0014] Acquire the teaching image to be recognized;

[0015] A keypoint detection network is used to extract features from teaching images to obtain semantic features; the keypoint detection network is trained based on a human body dataset.

[0016] A pose classification network is used to classify the poses of images based on semantic features, thereby obtaining the teaching poses in the teaching images. The pose classification network is trained on a teaching pose dataset. The teaching pose dataset is constructed from simulated teaching videos in simulated teaching scenarios and real teaching videos in real teaching scenarios.

[0017] This application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, performs the following steps:

[0018] Acquire the teaching image to be recognized;

[0019] A keypoint detection network is used to extract features from teaching images to obtain semantic features; the keypoint detection network is trained based on a human body dataset.

[0020] A pose classification network is used to classify the poses of images based on semantic features, thereby obtaining the teaching poses in the teaching images. The pose classification network is trained on a teaching pose dataset. The teaching pose dataset is constructed from simulated teaching videos in simulated teaching scenarios and real teaching videos in real teaching scenarios.

[0021] This application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0022] Acquire the teaching image to be recognized;

[0023] A keypoint detection network is used to extract features from teaching images to obtain semantic features; the keypoint detection network is trained based on a human body dataset.

[0024] A pose classification network is used to classify the poses of images based on semantic features, thereby obtaining the teaching poses in the teaching images. The pose classification network is trained on a teaching pose dataset. The teaching pose dataset is constructed from simulated teaching videos in simulated teaching scenarios and real teaching videos in real teaching scenarios.

[0025] The aforementioned teaching posture recognition methods, devices, computer equipment, computer-readable storage media, and computer program products extract features from teaching images using a keypoint detection network trained on a human body dataset. This yields the image semantic features of the teaching images. Since these image semantic features contain high-level semantic information and a significant amount of human skeleton posture information, using a posture classification network to classify the image semantic features for posture classification is beneficial for improving the accuracy of posture classification. Furthermore, because the teaching posture dataset is constructed from simulated teaching videos in simulated teaching scenarios and real teaching videos in real teaching scenarios—meaning it includes teaching videos from both simulated and real teaching scenarios—the number of samples for posture recognition in teaching scenarios is greatly expanded. Therefore, using a posture classification network trained on the teaching posture dataset for teaching posture recognition further improves the accuracy of teaching posture recognition in teaching scenarios.

[0026] In one embodiment, the training steps of the pose classification network include:

[0027] Obtain the teaching posture dataset, which includes multiple teaching image samples;

[0028] Obtain the teaching pose annotation for each teaching image sample;

[0029] Target teaching image samples were selected from the teaching posture dataset, and key point detection network was used to extract features from the target teaching image samples to obtain image semantic feature samples;

[0030] A pose classification network to be trained is used to classify the pose of image semantic feature samples to obtain pose prediction results;

[0031] The model loss is calculated based on the teaching posture annotation and posture prediction results of the target teaching image samples.

[0032] If the model loss does not meet the training stopping condition, the network parameters of the pose classification network to be trained are adjusted based on the model loss. Then, based on the remaining teaching image samples in the teaching pose dataset, the pose classification network to be trained with the network parameters adjusted is continued to be trained until the calculated model loss reaches the training stopping condition, and the trained pose classification network is obtained.

[0033] In this embodiment, the teaching posture dataset includes multiple teaching image samples. Training the posture classification network based on the teaching posture dataset and teaching posture annotations is beneficial to improving the accuracy of the posture classification network in recognizing teaching postures in teaching scenarios. In addition, during the training process of the posture classification network, the network parameters of the key point detection network are frozen, and only the network parameters of the posture classification network to be trained are adjusted. In this way, the key point detection network can still retain the ability to output high-level semantic features and detect key points of human posture. At the same time, the posture classification network continuously learns its ability to recognize different teaching postures in teaching images during the training process. The trained posture classification network has high recognition accuracy for teaching postures in teaching scenarios.

[0034] In one embodiment, obtaining the teaching posture dataset includes:

[0035] Obtain teaching video samples; teaching video samples include simulated teaching videos collected in simulated teaching scenarios, and real teaching videos collected in real teaching scenarios;

[0036] A teaching posture dataset was constructed based on image frames from teaching video samples.

[0037] In this embodiment, by acquiring teaching video samples from simulated teaching scenarios and real teaching scenarios, the teaching posture dataset is expanded, providing sufficient data support for model training. At the same time, the teaching posture dataset is a dataset under teaching scenarios, which is conducive to training a posture classification model with high recognition accuracy for teaching postures under teaching scenarios.

[0038] In one embodiment, there are multiple real teaching videos, which are simultaneously filmed by recording devices placed in different locations on the same teaching scene.

[0039] Obtain the teaching pose annotation for each teaching image sample, including:

[0040] Select the target device from the recording devices placed in different locations, and use the real teaching video corresponding to the target device as the target teaching video;

[0041] For each image frame in the target teaching video, obtain the manual pose annotation of the current image frame, and obtain the target image frames in the other real teaching videos that were acquired at the same time as the current image frame.

[0042] The teaching pose annotations for each target image frame are determined to be the manual pose annotations for the current image frame.

[0043] In this embodiment, by manually annotating the target teaching video collected by the target device, the teaching posture annotation of the target image frame with the same collection time in other real teaching videos is automatically annotated with the corresponding manual posture annotation, realizing semi-automatic annotation of teaching posture, which is beneficial to improving the accuracy and efficiency of teaching posture annotation.

[0044] In one embodiment, the teaching posture recognition method further includes:

[0045] The head network of the key point detection network is used to decode the semantic features of the image to obtain the structural key points that represent the teaching posture;

[0046] Based on the location of structural key points in the teaching image, the structural key points are displayed in the teaching image.

[0047] In this embodiment, the semantic features of the image are decoded by the head network of the key point detection network to obtain the structural key points representing the teaching posture. Combined with the teaching posture output by the posture classification network, two results can be output simultaneously after one inference of the teaching image. The output results of the network can be used for subsequent posture tracking and key point structure visualization without the need to stack other detection models, thus improving the inference performance of the overall process.

[0048] In one embodiment, the teaching image contains multiple teaching poses; the teaching pose recognition method further includes:

[0049] Obtain a pre-built priority order table; the priority order table includes the priority order of each of the multiple preset teaching postures;

[0050] Look up the priority order table to determine the priority order of each teaching posture;

[0051] Based on priority order, the target teaching pose is determined from multiple teaching poses in the teaching image.

[0052] In this embodiment, the priority order of each teaching posture in the teaching image is determined by a pre-constructed priority order table, which is beneficial to identify the target teaching posture from multiple teaching postures, that is, to determine the teaching posture that best matches the teacher's current teaching scenario.

[0053] In one embodiment, the teaching image to be identified is an image frame from a video to be identified; the teaching posture recognition method further includes:

[0054] Obtain the teaching posture in each image frame of the video to be recognized;

[0055] Action analysis is performed on the teaching postures in each image frame to obtain the action analysis results; based on the teaching postures and action analysis results in each image frame, a teaching posture analysis report is generated and displayed.

[0056] In this embodiment, action analysis is performed on the teaching postures in each image frame of the video to be identified, and a teaching posture analysis report is generated and displayed based on the teaching postures and action analysis results in each image frame. In practical applications, recording devices record teachers' teaching videos, and this posture recognition method can be used after teaching to analyze the teaching postures in the video frame by frame to obtain corresponding teaching posture analysis reports. This is beneficial for summarizing the teacher's body posture expression throughout the teaching process, thereby assisting teachers in consciously conducting posture training.

[0057] This application provides a method for training a pose classification network, including:

[0058] Obtain the teaching posture dataset, which includes multiple teaching image samples;

[0059] Obtain the teaching pose annotation for each teaching image sample;

[0060] Target teaching image samples were selected from the teaching posture dataset, and key point detection network was used to extract features from the target teaching image samples to obtain image semantic feature samples;

[0061] A pose classification network to be trained is used to classify the pose of image semantic feature samples to obtain pose prediction results;

[0062] The model loss is calculated based on the teaching posture annotation and posture prediction results of the target teaching image samples.

[0063] If the model loss does not meet the training stopping condition, the network parameters of the pose classification network to be trained are adjusted based on the model loss. Then, based on the remaining teaching image samples in the teaching pose dataset, the pose classification network with adjusted network parameters is trained again until the calculated model loss reaches the training stopping condition, and the trained pose classification network is obtained.

[0064] This application also provides a training device for a pose classification network, comprising:

[0065] The sample acquisition module is used to acquire the teaching posture dataset, which includes multiple teaching image samples.

[0066] The annotation acquisition module is used to acquire the teaching posture annotations for each teaching image sample;

[0067] The feature sample extraction module is used to select target teaching image samples from the teaching posture dataset, and to extract features from the target teaching image samples using a key point detection network to obtain image semantic feature samples.

[0068] The prediction module is used to classify the pose of image semantic feature samples using the pose classification network to be trained, and obtain the pose prediction result.

[0069] The loss calculation module is used to calculate the model loss based on the teaching pose annotation and pose prediction results of the target teaching image samples.

[0070] The training module is used to adjust the network parameters of the pose classification network to be trained based on the model loss when the model loss does not meet the training stopping condition. Based on the remaining teaching image samples in the teaching pose dataset, the network parameter-adjusted pose classification network is trained again until the calculated model loss reaches the training stopping condition, thus obtaining the trained pose classification network.

[0071] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0072] Obtain the teaching posture dataset, which includes multiple teaching image samples;

[0073] Obtain the teaching pose annotation for each teaching image sample;

[0074] Target teaching image samples were selected from the teaching posture dataset, and key point detection network was used to extract features from the target teaching image samples to obtain image semantic feature samples;

[0075] A pose classification network to be trained is used to classify the pose of image semantic feature samples to obtain pose prediction results;

[0076] The model loss is calculated based on the teaching posture annotation and posture prediction results of the target teaching image samples.

[0077] If the model loss does not meet the training stopping condition, the network parameters of the pose classification network to be trained are adjusted based on the model loss. Then, based on the remaining teaching image samples in the teaching pose dataset, the pose classification network with adjusted network parameters is trained again until the calculated model loss reaches the training stopping condition, and the trained pose classification network is obtained.

[0078] This application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, performs the following steps:

[0079] Obtain the teaching posture dataset, which includes multiple teaching image samples;

[0080] Obtain the teaching pose annotation for each teaching image sample;

[0081] Target teaching image samples were selected from the teaching posture dataset, and key point detection network was used to extract features from the target teaching image samples to obtain image semantic feature samples;

[0082] A pose classification network to be trained is used to classify the pose of image semantic feature samples to obtain pose prediction results;

[0083] The model loss is calculated based on the teaching posture annotation and posture prediction results of the target teaching image samples.

[0084] If the model loss does not meet the training stopping condition, the network parameters of the pose classification network to be trained are adjusted based on the model loss. Then, based on the remaining teaching image samples in the teaching pose dataset, the pose classification network with adjusted network parameters is trained again until the calculated model loss reaches the training stopping condition, and the trained pose classification network is obtained.

[0085] This application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0086] Obtain the teaching posture dataset, which includes multiple teaching image samples;

[0087] Obtain the teaching pose annotation for each teaching image sample;

[0088] Target teaching image samples were selected from the teaching posture dataset, and key point detection network was used to extract features from the target teaching image samples to obtain image semantic feature samples;

[0089] A pose classification network to be trained is used to classify the pose of image semantic feature samples to obtain pose prediction results;

[0090] The model loss is calculated based on the teaching posture annotation and posture prediction results of the target teaching image samples.

[0091] If the model loss does not meet the training stopping condition, the network parameters of the pose classification network to be trained are adjusted based on the model loss. Then, based on the remaining teaching image samples in the teaching pose dataset, the pose classification network with adjusted network parameters is trained again until the calculated model loss reaches the training stopping condition, and the trained pose classification network is obtained.

[0092] The training method, apparatus, computer equipment, computer-readable storage medium, and computer program product of the aforementioned pose classification network, including a teaching pose dataset comprising multiple teaching image samples, and training the pose classification network based on the teaching pose dataset and teaching pose annotations, is beneficial for improving the accuracy of the pose classification network in recognizing teaching poses in teaching scenarios. Furthermore, during the training process of the pose classification network, the network parameters of the keypoint detection network are frozen, and only the network parameters of the pose classification network to be trained are adjusted. In this way, the keypoint detection network can still retain its ability to output high-level semantic features and detect key points of human poses. Simultaneously, the pose classification network continuously learns its ability to recognize different teaching poses in teaching images during training, resulting in a pose classification network with high accuracy in recognizing teaching poses in teaching scenarios. Attached Figure Description

[0093] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0094] Figure 1 This is an application environment diagram of the teaching posture recognition method in one embodiment;

[0095] Figure 2 This is a flowchart illustrating a teaching posture recognition method in one embodiment;

[0096] Figure 3 This is a flowchart illustrating the teaching posture recognition method in another embodiment;

[0097] Figure 4 This is a flowchart illustrating the teaching posture recognition method in yet another embodiment;

[0098] Figure 5 This is a schematic diagram of a teaching posture analysis report in one embodiment;

[0099] Figure 6 This is a flowchart illustrating the training method of a pose classification network in one embodiment;

[0100] Figure 7 This is a structural block diagram of a teaching posture recognition device in one embodiment;

[0101] Figure 8 This is a structural block diagram of a training device for a pose classification network in one embodiment;

[0102] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0103] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0104] Teachers typically use body language to aid in conveying information during classroom instruction. A teacher's posture is a direct expression of their teaching dynamism and emotional state, and is an important component of their teaching skills. Current research on teacher posture reveals that new teachers, lacking experience, often exhibit stiffness or other unnecessary tense movements. Meanwhile, experienced teachers often show lower energy levels and a lack of necessary physical gestures. Therefore, analyzing teachers' postures during the teaching process can provide posture training templates that can effectively improve negative postures such as stiffness and awkwardness caused by tension, lethargy, or other negative emotions during instruction. This is of great significance for improving teaching quality.

[0105] Traditional methods for recognizing teaching postures typically utilize 2D human keypoints to detect key points in the human body and then employ traditional machine learning methods, such as support vector machines, to classify the postures based on the keypoint locations and skeletal structural features. However, the accuracy of this approach depends heavily on the accuracy of keypoint detection, making it highly dependent on the placement angle and distance of the image acquisition device. This results in poor generalization performance. Consequently, traditional neural network methods cannot achieve the fine-grained classification of detailed limb postures, such as common teaching postures like raising a hand, pointing, or putting hands behind the back. In other words, traditional methods suffer from low accuracy in recognizing teaching postures. Furthermore, existing public datasets lack large-scale training data specifically for teacher postures and teaching scenarios, thus limiting the accuracy of neural network methods in recognizing teacher postures in teaching settings at the data level.

[0106] Based on this, this application proposes a teaching posture recognition method and a training method for a posture classification network. A keypoint detection network trained on a human body dataset is used to extract features from teaching images, obtaining image semantic features. Since these image semantic features contain high-level semantic information and a significant amount of human skeleton posture information, using a posture classification network to classify these image semantic features for posture classification improves the accuracy of posture classification. Furthermore, because the teaching posture dataset is constructed from simulated teaching videos in simulated teaching scenarios and real teaching videos in real teaching scenarios, it greatly expands the number of samples for posture recognition in teaching scenarios. Therefore, using a posture classification network trained on the teaching posture dataset for teaching posture recognition further improves the accuracy of teaching posture recognition in teaching scenarios.

[0107] The teaching posture recognition method provided in this application embodiment can be applied to, for example... Figure 1 The application environment shown is illustrated. Terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. This embodiment uses the method applied to terminal 102 as an example. It is understood that this method can also be applied to server 104, and can also be applied to a system including terminal 102 and server 104, and implemented through the interaction between terminal 102 and server 104. Terminal 102 acquires the teaching image to be identified; it uses a keypoint detection network to extract features from the teaching image to obtain image semantic features; the keypoint detection network is trained based on a human body dataset; it uses a pose classification network to classify the image semantic features to obtain the teaching pose in the teaching image; the pose classification network is trained based on a teaching pose dataset. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Headset devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0108] In one exemplary embodiment, such as Figure 2As shown, a teaching posture recognition method is provided, which can be applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps 202 to 206. Wherein:

[0109] Step 202: Obtain the teaching image to be identified.

[0110] Among them, teaching images refer to images of teachers in teaching scenarios. Teaching images can be images taken of teachers in real teaching scenarios, or teaching videos recorded in real teaching scenarios, from which images are extracted.

[0111] In some embodiments, teaching images can also be obtained by constructing a simulated teaching scenario using the same or different recording equipment as the recording equipment in a real teaching scenario, and by taking pictures or recording videos in the simulated teaching scenario.

[0112] Step 204: Use a keypoint detection network to extract features from the teaching images to obtain semantic features of the images; the keypoint detection network is trained based on a human dataset.

[0113] Among them, the keypoint detection network refers to the neural network that can extract key points of human pose. For example, the LitePose model (an efficient architecture model for 2D human pose estimation) which is fully trained on the large-scale human dataset COCO (Common Objects in Context) can be used as the keypoint detection model. The LitePose model is a lightweight network based on convolution, which not only ensures high-precision inference results, but also has good inference performance, which is conducive to improving the accuracy and efficiency of pose recognition in teaching.

[0114] like Figure 3 The diagram shows a flowchart of the teaching posture recognition method in another embodiment. In this method, image semantic features are extracted from the teaching image using the LitePose model. The LitePose model comprises a backbone network and a head network. The LitePose model uses MBV2 (MobileNetV2, a lightweight network architecture) as the backbone network. The terminal inputs the teaching image into the backbone network of the LitePose model, which then extracts features from the teaching image, resulting in the output image semantic features.

[0115] By using the backbone network of the trained LitePose model to extract features from teaching images, the semantic features of the images can include high-level speech information of the teaching images, as well as more human posture information.

[0116] Step 206: Use a pose classification network to classify the semantic features of the image to obtain the teaching poses in the teaching images; the pose classification network is trained based on the teaching pose dataset; the teaching pose dataset is constructed based on simulated teaching videos in simulated teaching scenarios and real teaching videos in real teaching scenarios.

[0117] The pose classification network is a neural network trained on a teaching pose dataset, which contains multiple teaching image samples from teaching scenarios. Teaching scenarios can include real teaching scenarios and simulated teaching scenarios. Simulated teaching scenarios are constructed by simulating real teaching scenarios using recording equipment that is the same as or different from that used in real teaching scenarios.

[0118] refer to Figure 3 The pose classification network consists of a bottleneck layer, an average pooling layer (AvgPool), and a fully connected layer (FC-Layer). The terminal inputs the semantic features of the image into the pose classification network. The bottleneck layer is a CNN (Convolutional Neural Networks) structure. The bottleneck layer further extracts relevant feature maps for classification. Then, the features are processed by the average pooling layer and the fully connected layer in sequence to regress the features to the classification of the teacher's pose. The output of the pose classification network is the teaching pose in the teaching image.

[0119] A teaching image may contain at least one teaching pose, and a pose classification network can detect at least one teaching pose in a teaching image at the same time.

[0120] In teaching scenarios, teaching images often contain various fixed teaching postures, such as raising hands, pointing, putting hands behind back, hands on hips, lowering hands, writing on the blackboard, touching the head, hugging elbows, and clasping hands. Traditional neural network methods cannot classify such detailed body postures in teaching images for recognition in teaching scenarios. The method proposed in this application uses a posture classification network to classify the image semantic features extracted by the key point detection network to obtain the teaching postures in teaching images. This method is used for teaching posture recognition in such teaching scenarios and has good recognition accuracy and precision.

[0121] In the aforementioned teaching posture recognition method, a keypoint detection network trained on a human body dataset is used to extract features from teaching images, resulting in image semantic features. Since these image semantic features contain high-level semantic information and a significant amount of human skeleton posture information, using a posture classification network to classify the image semantic features for posture classification is beneficial for improving the accuracy of posture classification. Furthermore, because the teaching posture dataset is constructed from simulated teaching videos in simulated teaching scenarios and real teaching videos in real teaching scenarios, it greatly expands the number of samples for posture recognition in teaching scenarios. Therefore, using a posture classification network trained on the teaching posture dataset for teaching posture recognition further improves the accuracy of teaching posture recognition in teaching scenarios.

[0122] In an exemplary embodiment, the training steps of the pose classification network include: acquiring a teaching pose dataset, which includes multiple teaching image samples; acquiring the teaching pose annotation for each teaching image sample; selecting a target teaching image sample from the teaching pose dataset, and using a keypoint detection network to extract features from the target teaching image sample to obtain image semantic feature samples; using the pose classification network to be trained to classify the image semantic feature samples to obtain pose prediction results; calculating the model loss based on the teaching pose annotation and pose prediction results of the target teaching image samples; and adjusting the network parameters of the pose classification network to be trained based on the model loss if the model loss does not meet the training stopping condition, and continuing to train the pose classification network to be trained with adjusted network parameters based on the remaining teaching image samples in the teaching pose dataset until the calculated model loss reaches the training stopping condition, thus obtaining a trained pose classification network.

[0123] The teaching posture dataset is a collection of teaching image samples. These teaching image samples refer to images of teachers in teaching scenarios, which can be images from real teaching scenarios or images from simulated teaching scenarios. Simulated teaching scenarios are constructed using recording equipment that is the same as or different from that used in real teaching scenarios.

[0124] Teaching posture annotation samples refer to annotations used to characterize the teaching postures contained in teaching image samples. Teaching posture annotation samples can include common teaching postures, such as raising hands, pointing, putting hands behind back, hands on hips, lowering hands, writing on the blackboard, touching the head, hugging elbows, clasping hands, and other postures.

[0125] The pose classification network to be trained can be a neural network model used for classification, see reference. Figure 3The pose classification network to be trained includes a bottleneck network layer, an average pooling layer, and a fully connected layer. The terminal uses multiple teaching image samples from the teaching pose dataset as training samples, and the teaching pose annotation of each teaching image sample as the training label. The pose classification network to be trained is trained multiple times, and the training stops when the stopping condition is met, resulting in the trained pose classification network.

[0126] The target teaching image sample is any teaching image sample selected from the teaching posture dataset. The training process using the target teaching image sample is illustrated as an example. A keypoint detection network is used to extract features from the target teaching image sample to obtain image semantic feature samples. A posture classification network to be trained is used to classify the image semantic feature samples to obtain posture prediction results. The model loss is calculated based on the difference between the teaching posture annotation and the posture prediction results of the target teaching image sample and the preset model function. The model is then trained based on the model loss. If the model loss does not meet the training stopping condition, the network parameters of the posture classification network to be trained are adjusted based on the model loss.

[0127] In some embodiments, the target teaching image sample may contain at least one teaching pose at different positions in the image. Therefore, the target teaching image sample corresponds to at least one teaching pose annotation at different positions. The pose classification network to be trained can simultaneously output multiple pose prediction results at different positions in the target teaching image sample. The difference between the teaching pose annotations and pose prediction results at the same position is obtained, the model loss is calculated, and the network parameters are adjusted to obtain a pose classification network that can detect all teaching poses contained in the teaching image at once.

[0128] The remaining instructional image samples are the instructional image samples in the instructional pose dataset, excluding the target instructional image samples. The terminal will continue to train the pose classification network with adjusted network parameters using these remaining instructional image samples until the calculated model loss reaches the training stopping condition, at which point the training will stop, resulting in a trained pose classification network. The trained pose classification model can be used for instructional pose recognition methods.

[0129] In this embodiment, the teaching posture dataset includes multiple teaching image samples. Training the posture classification network based on the teaching posture dataset and teaching posture annotations is beneficial to improving the accuracy of the posture classification network in recognizing teaching postures in teaching scenarios. In addition, during the training process of the posture classification network, the network parameters of the key point detection network are frozen, and only the network parameters of the posture classification network to be trained are adjusted. In this way, the key point detection network can still retain the ability to output high-level semantic features and detect key points of human posture. At the same time, the posture classification network continuously learns its ability to recognize different teaching postures in teaching images during the training process. The trained posture classification network has high recognition accuracy for teaching postures in teaching scenarios.

[0130] In one exemplary embodiment, obtaining a teaching posture dataset includes: obtaining teaching video samples; the teaching video samples include simulated teaching videos collected in simulated teaching scenarios and real teaching videos collected in real teaching scenarios; and constructing a teaching posture dataset based on image frames in the teaching video samples.

[0131] The authentic teaching videos are recorded using recording equipment in real teaching scenarios. Since the data size of authentic teaching videos is limited, a method is proposed to simulate real teaching scenarios and construct simulated teaching scenarios in order to expand the data scale and improve model performance.

[0132] In some embodiments, the same recording equipment as in a real teaching scenario can be used, and the recording equipment can be placed in the same position and orientation as in a real teaching scenario to construct a simulated teaching scenario. The teaching process of the teacher or a simulated teacher is recorded in this simulated teaching scenario to obtain a first simulated teaching video.

[0133] In other embodiments, a camera with no limitation on placement or orientation can be used to construct the simulated teaching scenario. The camera can be a different recording device than that used in a real teaching scenario; for example, it can be a Kinect (a depth camera based on structured light principles) or a regular RGB camera (color camera). The teaching process of the teacher or a simulated teacher is recorded in this simulated teaching scenario to obtain a second simulated teaching video. The simulated teaching video includes both a first and a second simulated teaching video.

[0134] When recording teaching video samples, teachers or simulated teachers should use a variety of teaching postures to simulate the teaching process.

[0135] The terminal can use image frames from teaching video samples as teaching image samples to construct a teaching posture dataset, or select target image frames from teaching video samples and use them as teaching image samples to construct a teaching posture dataset. Table 1 shows the data distribution table of the teaching posture dataset in one embodiment.

[0136] Table 1. Data Distribution of the Teaching Posture Dataset

[0137]

[0138] In this embodiment, by acquiring teaching video samples from simulated teaching scenarios and real teaching scenarios, the teaching posture dataset is expanded, providing sufficient data support for model training. At the same time, the teaching posture dataset is a dataset under teaching scenarios, which is conducive to training a posture classification model with high recognition accuracy for teaching postures under teaching scenarios.

[0139] In one exemplary embodiment, there are multiple real teaching videos, which are simultaneously captured by recording devices placed in different locations on the same teaching scene. Obtaining the teaching posture annotation for each teaching image sample includes: selecting a target device from the recording devices placed in different locations, and using the real teaching video corresponding to the target device as the target teaching video; for each image frame in the target teaching video, obtaining the manual posture annotation for the current image frame, and obtaining target image frames from the remaining real teaching videos that were captured at the same time as the current image frame; and determining that the teaching posture annotation for each target image frame is the manual posture annotation for the current image frame.

[0140] In addition, for the annotation of teaching postures in teaching image samples, manual annotation can be used. In order to improve the annotation efficiency when the number of samples is large, this application also proposes a semi-automatic annotation method. This semi-automatic annotation method can be used for the first simulated teaching video, the second simulated teaching video and the real teaching video to obtain the teaching posture annotations of each image frame.

[0141] Taking real teaching videos as an example, multiple real teaching videos are simultaneously filmed by recording devices placed in different locations, capturing the same teaching scene. In this way, multiple real teaching videos captured at the same time contain the same teaching posture from different angles.

[0142] The target device can be any recording device placed in different locations. After selecting the target device, for example, it could be a recording device positioned directly facing the teacher, capturing a frontal image of the teacher, while other devices capture side views of the teacher. For the target teaching video captured by the target device, each image frame can be manually labeled to obtain a corresponding manual pose annotation for each frame. The manual pose annotation is a label representing the type of teaching pose in the image frame.

[0143] The terminal acquires the manual pose annotations for each image frame in the target teaching video. For the remaining real teaching videos, teaching pose annotations are automatically added to target image frames that were captured at the same time as each image frame in the target teaching video. The target image frames with the same capture time have the same manual pose annotations as the corresponding image frames in the target teaching video. Table 2 shows a classification table of teaching poses in one embodiment.

[0144] Table 2 Classification of Teaching Postures

[0145]

[0146] For each image frame in the first and second simulated teaching videos, the above semi-automatic annotation method can also be used to obtain the teaching posture annotation for each image frame.

[0147] In this embodiment, by manually annotating the target teaching video collected by the target device, the teaching posture annotation of the target image frame with the same collection time in other real teaching videos is automatically annotated with the corresponding manual posture annotation, realizing semi-automatic annotation of teaching posture, which is beneficial to improving the accuracy and efficiency of teaching posture annotation.

[0148] In an exemplary embodiment, the teaching posture recognition method further includes: decoding the semantic features of the image using the head network of a key point detection network to obtain structural key points representing the teaching posture; and displaying the structural key points in the teaching image based on their positions in the teaching image.

[0149] The keypoint detection network's head network includes multiple deconvolutional layers and a final prediction layer, used to decode the semantic features of the input image to obtain structural keypoints representing the teaching pose. For example, the keypoint detection network can be the LitePose model. (Reference) Figure 3 The keypoint detection network in the image decodes the semantic features of the image and outputs structural keypoints. Using a keypoint detection network to extract structural keypoints improves the accuracy of keypoint extraction.

[0150] Since structural key points are crucial points representing teaching postures in teaching images, the terminal can acquire the location of each structural key point in the teaching image and display each structural key point within the teaching image. For example... Figure 4 The diagram shows a flowchart of a teaching posture recognition method in another embodiment. The key point detection network outputs structural key points representing teaching postures in the teaching image, and the posture classification network outputs the teaching posture in the teaching image. The terminal can connect the structural key points according to preset rules to obtain the delineation result of the teaching posture, and display this delineation result in the teaching image. The delineation result provides a more intuitive representation of the teaching posture in the teaching image, and through multiple consecutive teaching images, the changing process of the teaching posture can be visually demonstrated.

[0151] In this embodiment, the semantic features of the image are decoded by the head network of the key point detection network to obtain the structural key points representing the teaching posture. Combined with the teaching posture output by the posture classification network, two results can be output simultaneously after one inference of the teaching image. The output results of the network can be used for subsequent posture tracking and key point structure visualization without the need to stack other detection models, thus improving the inference performance of the overall process.

[0152] In an exemplary embodiment, there are multiple teaching poses in the teaching image; the teaching pose recognition method further includes: obtaining a pre-constructed priority order table; the priority order table includes the priority order corresponding to each of the multiple preset teaching poses; searching the priority order table to determine the priority order corresponding to each of the multiple teaching poses; and determining the target teaching pose among the multiple teaching poses in the teaching image based on the priority order.

[0153] In teaching scenarios, if there are multiple teaching postures in a teaching image, then often only one or more postures are primary, while the others are secondary. For example, if a teaching image contains both raised hands and folded elbows, then raised hands are the primary posture, and folded elbows are the secondary posture. Similarly, if a teaching image contains both hands in a teaching posture, then a pre-defined rule can be used to designate the posture of one hand as the primary posture.

[0154] This application proposes a pre-constructed priority order table, which stores the priority order of multiple teaching postures. The terminal can look up the priority order table to determine the priority order of the multiple teaching postures in the teaching image. The terminal can choose the highest priority posture, the teaching posture ranked in the top N, or the teaching posture ranked M as the target teaching posture, where N and M are positive integers.

[0155] The target teaching posture is the main teaching posture in the teaching image, and the target teaching posture is often more in line with the teacher's current teaching scenario.

[0156] In this embodiment, the priority order of each teaching posture in the teaching image is determined by a pre-constructed priority order table, which is beneficial to identify the target teaching posture from multiple teaching postures, that is, to determine the teaching posture that best matches the teacher's current teaching scenario.

[0157] In an exemplary embodiment, the teaching image to be identified is an image frame in the video to be identified; the teaching posture recognition method further includes: acquiring the teaching posture in each image frame of the video to be identified; performing motion analysis on the teaching posture in each image frame to obtain motion analysis results; and generating and displaying a teaching posture analysis report based on the teaching posture and motion analysis results in each image frame.

[0158] The teaching image to be identified can be a single image captured by an image acquisition device, or it can be an image frame extracted from the video to be identified. Taking the image frame in the video to be identified as the teaching image to be identified as an example, the image frames in the video to be identified have a temporal sequence. By analyzing the teaching posture in each image frame, a teaching posture analysis report is generated to show the changes in teaching posture throughout the entire video time period.

[0159] Motion analysis is primarily used to identify teaching postures that are irrelevant to conveying teaching information. The motion analysis results are used to indicate teaching postures in each image frame that are not related to conveying teaching information. The terminal can define certain teaching postures as irrelevant to conveying teaching information based on user needs, such as hands on hips or elbows crossed.

[0160] The terminal uses the aforementioned teaching posture recognition method to obtain the teaching posture in each image frame of the video to be recognized, and performs motion analysis on the teaching posture in each image frame to obtain the motion analysis results. For example, the motion analysis results may be: the teaching postures that are not related to conveying teaching information appear in this instance, namely, hands on hips and elbows crossed.

[0161] In some embodiments, the teaching posture analysis report may include the teaching posture in each image frame, as well as the motion analysis results. For example... Figure 5The diagram illustrates a teaching posture analysis report in one embodiment. The terminal's human-computer interface includes a teaching posture analysis result display section and a motion analysis result display section. The teaching posture analysis result display section shows various teaching postures appearing in the video to be identified, as well as the occurrence of each posture at different time points. The motion analysis result display section shows the motion analysis results. Additionally, the terminal can perform common posture analysis on the teaching postures in each image frame to obtain common posture analysis results. These results are added to the teaching posture analysis report and displayed in the teaching posture analysis result display section. For example, the terminal obtains the frequency of each teaching posture in the video to be identified, identifies the posture with the highest frequency, or the posture with more than a preset frequency, as the common posture, and obtains the common posture analysis results based on the common postures and the frequency of each teaching posture. For example, the common posture analysis results could be: "A total of 7 teaching postures were used in this study, among which the common posture is 'decentralization'."

[0162] In some embodiments, the terminal's human-computer interaction interface also includes a video playback section. Responding to user clicks on different teaching postures at different time points in the teaching posture analysis results display section, the terminal displays the position of the corresponding teaching posture in the teaching video sample and plays back the corresponding teaching video sample in the video playback section, which helps to better present teaching feedback to teachers. Additionally, the terminal can also respond to user selections at different time points in the motion analysis results display section and play back the teaching video sample at the corresponding time point in the video playback section.

[0163] In this embodiment, action analysis is performed on the teaching postures in each image frame of the video to be identified, and a teaching posture analysis report is generated and displayed based on the teaching postures and action analysis results in each image frame. In practical applications, recording devices record teachers' teaching videos, and this posture recognition method can be used after teaching to analyze the teaching postures in the video frame by frame to obtain corresponding teaching posture analysis reports. This is beneficial for summarizing the teacher's body posture expression throughout the teaching process, thereby assisting teachers in consciously conducting posture training.

[0164] In one exemplary embodiment, such as Figure 6 As shown, a training method for a pose classification network is provided, which can be applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 601 to 606. Wherein:

[0165] Step 601: Obtain the teaching posture dataset, which includes multiple teaching image samples.

[0166] Step 602: Obtain the teaching pose annotation for each teaching image sample.

[0167] Step 603: Select target teaching image samples from the teaching posture dataset, and use a key point detection network to extract features from the target teaching image samples to obtain image semantic feature samples.

[0168] Step 604: Use the pose classification network to be trained to classify the pose of the image semantic feature samples to obtain the pose prediction result.

[0169] Step 605: Calculate the model loss based on the teaching posture annotation and posture prediction results of the target teaching image samples.

[0170] Step 606: If the model loss does not meet the training stopping condition, adjust the network parameters of the pose classification network to be trained based on the model loss, and continue to train the pose classification network to be trained with the network parameters adjusted based on the remaining teaching image samples in the teaching pose dataset, until the calculated model loss reaches the training stopping condition, and then stop to obtain the trained pose classification network.

[0171] The teaching posture dataset is a collection of teaching image samples. These teaching image samples refer to images of teachers in teaching scenarios, which can be images from real teaching scenarios or images from simulated teaching scenarios. Simulated teaching scenarios are constructed using recording equipment that is the same as or different from that used in real teaching scenarios.

[0172] Teaching posture annotation samples refer to annotations used to characterize the teaching postures contained in teaching image samples. Teaching posture annotation samples can include common teaching postures, such as raising hands, pointing, putting hands behind back, hands on hips, lowering hands, writing on the blackboard, touching the head, hugging elbows, clasping hands, and other postures.

[0173] The pose classification network to be trained can be a neural network model used for classification, see reference. Figure 3 The pose classification network to be trained includes a bottleneck network layer, an average pooling layer, and a fully connected layer. The server uses multiple teaching image samples from the teaching pose dataset as training samples, and uses the teaching pose annotation of each teaching image sample as the training label. The pose classification network to be trained is trained multiple times, and stops when the stopping condition is met, thus obtaining the trained pose classification network.

[0174] The target teaching image sample is any teaching image sample selected from the teaching posture dataset. The training process using the target teaching image sample is illustrated as an example. A keypoint detection network is used to extract features from the target teaching image sample to obtain image semantic feature samples. A posture classification network to be trained is used to classify the image semantic feature samples to obtain posture prediction results. The model loss is calculated based on the difference between the teaching posture annotation and the posture prediction results of the target teaching image sample and the preset model function. The model is then trained based on the model loss. If the model loss does not meet the training stopping condition, the network parameters of the posture classification network to be trained are adjusted based on the model loss.

[0175] In some embodiments, the target teaching image sample may contain at least one teaching pose at different positions in the image. Therefore, the target teaching image sample corresponds to at least one teaching pose annotation at different positions. The pose classification network to be trained can simultaneously output multiple pose prediction results at different positions in the target teaching image sample. The difference between the teaching pose annotations and pose prediction results at the same position is obtained, the model loss is calculated, and the network parameters are adjusted to obtain a pose classification network that can detect all teaching poses contained in the teaching image at once.

[0176] The remaining teaching image samples are the teaching image samples in the teaching posture dataset excluding the target teaching image sample. The server will continue to train the training posture classification network with adjusted network parameters using the remaining teaching image samples until the calculated model loss reaches the training stopping condition, at which point the training will stop, resulting in the trained posture classification network. The trained posture classification model can be used in the above-mentioned teaching posture recognition method.

[0177] In the training method of the above-mentioned pose classification network, the teaching pose dataset includes multiple teaching image samples. Training the pose classification network based on the teaching pose dataset and teaching pose annotations is beneficial to improving the accuracy of the pose classification network in recognizing teaching poses in teaching scenarios. In addition, during the training process of the pose classification network, the network parameters of the key point detection network are frozen, and only the network parameters of the pose classification network to be trained are adjusted. In this way, the key point detection network can still retain the ability to output high-level semantic features and detect key points of human poses. At the same time, the pose classification network continuously learns its ability to recognize different teaching poses in teaching images during the training process. The trained pose classification network has high recognition accuracy for teaching poses in teaching scenarios.

[0178] To illustrate in detail the teaching posture recognition method and the training method and effect of the posture classification network in this scheme, a detailed embodiment is described below:

[0179] Taking the teaching posture recognition in a teaching scenario as an example, the key point detection model trained on a large-scale human body dataset is used to extract features from the teaching image to obtain the image semantic features; the constructed posture classification model is used to classify the image semantic features to obtain the teaching posture in the teaching image. The posture classification network is trained on the teaching posture dataset.

[0180] In one example, the keypoint detection model can employ a pre-trained LitePose model. The LitePose model's backbone network layer is used to extract semantic features from the teaching images, allowing these features to contain high-level semantic information while preserving more human pose information. Meanwhile, the LitePose model is a lightweight convolutional network, ensuring not only high-precision inference results but also good inference performance.

[0181] A pose classification network is employed to classify poses based on the semantic features of images. This network comprises a bottleneck layer, an average pooling layer, and a fully connected layer. The terminal inputs the semantic features of the image into the pose classification network. The bottleneck layer can be a CNN convolutional network structure. After passing through the bottleneck layer, relevant feature maps for classification are further extracted. These features are then processed sequentially through the average pooling layer and the fully connected layer to regress the features to the teacher's pose classification. The output of the pose classification network is the teaching pose in the teaching image. During the training of the pose classification network, a pre-trained LitePose model can be loaded, and the network parameters of the Backbone layer in the LitePose model can be frozen, preserving its ability to detect human keypoint features without updating. Simultaneously, the model loss is calculated by comparing the pose prediction results output by the pose classification network under training with the teaching pose annotations, allowing the network module to learn the ability to recognize different teaching poses of the teacher.

[0182] Because the keypoint detection network is not updated during network training, maintaining its ability to detect 2D human keypoints, the teaching images, after being processed by the keypoint detection network and the pose classification network, can simultaneously output structural keypoints representing the teaching pose and the teaching pose. The network output can then be used directly for subsequent teacher tracking and keypoint structure visualization without the need to overlay other detection models, thus improving the overall inference performance of the process.

[0183] To address the potential for teachers' hands to exhibit various hand gestures, a pose classification network was designed to provide multi-label output. This network can output all detected poses, and a post-processing stage explicitly defines unified action rules for final action determination. This reduces ambiguity in the network output and improves the model's inference accuracy. For example, a pre-constructed priority order table can be obtained, containing the priority order of multiple preset teaching poses. By searching the priority order table, the priority order of each teaching pose can be determined, and based on this priority order, the target teaching pose can be identified from among the multiple teaching poses in the teaching image.

[0184] In some embodiments, to match the multi-label output pattern, a binary cross-entropy function (such as BCELoss) can be used during training to calculate the multi-class multi-label loss, thereby updating the network parameters and optimizing the model's recognition ability.

[0185] The construction process of the teaching posture dataset is as follows: To train the posture classification network, a sufficiently large teaching posture dataset with labeled information needs to be constructed for training the posture classification network. Currently, there is no publicly available dataset specifically for action recognition of teacher teaching postures. Therefore, a first simulated teaching video with more than 20 people was collected using professional recording equipment; a second simulated teaching video with more than 10 people was collected using a Kinect camera / ordinary RGB camera; simultaneously, real teaching videos of more than 10 teachers in different classrooms were recorded and compiled using recording equipment in real teaching scenarios. The data from real and simulated teaching scenarios were semi-automatically labeled using various methods such as single-frame annotation and video annotation, and finally integrated into a teaching posture dataset with more than 126k images, providing sufficient data support for model training. This dataset is a classroom-based dataset with labeled teacher teaching posture classification.

[0186] In addition, the terminal can also acquire the teaching posture in each image frame of the video to be identified, perform motion analysis on the teaching posture in each image frame, and obtain motion analysis results; based on the teaching posture and motion analysis results in each image frame, a teaching posture analysis report is generated, and combined with the engineering display of the web front end, teachers can view the posture analysis report after class.

[0187] The aforementioned teaching posture recognition method and posture classification network training method extract features from teaching images through a keypoint detection network trained on a human body dataset, obtaining image semantic features. Since image semantic features contain high-level semantic information and a large amount of human skeleton posture information, using a posture classification network to classify image semantic features for posture is beneficial to improving the accuracy of posture classification. At the same time, since the teaching posture dataset is constructed based on simulated teaching videos in simulated teaching scenarios and real teaching videos in real teaching scenarios, that is, the teaching posture dataset contains teaching videos in simulated teaching scenarios and real teaching scenarios, it greatly expands the number of samples for posture recognition in teaching scenarios. Therefore, using a posture classification network trained on a teaching posture dataset for teaching posture recognition is further beneficial to improving the accuracy of teaching posture recognition in teaching scenarios. Furthermore, in terms of network design, feature extraction is performed based on a 2D human keypoint detection network pre-trained on a large-scale human dataset, and a pose classification network is constructed. This allows the extracted image semantic features to include more human skeleton pose information while taking into account the high-level semantic information of the teaching images. This is more conducive to the subsequent pose classification network's recognition of teaching poses, resulting in more accurate and stable pose recognition and higher overall inference performance. A special dataset specifically designed for teachers' teaching poses is constructed for training the pose classification network. Through simulated teaching scenarios and recorded classroom videos from real teaching scenarios, a semi-automatic annotation scheme is used to semi-automatically annotate the image frames in the videos and then manually review them. Finally, a dedicated dataset focusing on teachers' poses is generated, which is key to improving the accuracy and generalization performance of the neural network.

[0188] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0189] Based on the same inventive concept, this application also provides a teaching posture recognition device for implementing the teaching posture recognition method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more teaching posture recognition device embodiments provided below can be found in the limitations of the teaching posture recognition method described above, and will not be repeated here.

[0190] In one exemplary embodiment, such as Figure 7 As shown, a teaching posture recognition device 700 is provided, including: an acquisition module 720, a feature extraction module 740, and a classification module 760, wherein:

[0191] The acquisition module 720 is used to acquire the teaching image to be recognized;

[0192] The feature extraction module 740 is used to extract features from teaching images using a keypoint detection network to obtain semantic features of the images; the keypoint detection network is trained based on a human body dataset.

[0193] The classification module 760 is used to classify the semantic features of the image using a pose classification network to obtain the teaching poses in the teaching images; the pose classification network is trained based on the teaching pose dataset.

[0194] The aforementioned teaching posture recognition device extracts features from teaching images using a keypoint detection network trained on a human body dataset. This yields the image semantic features of the teaching images. Since these semantic features contain high-level semantic information and a significant amount of human skeleton posture information, using a posture classification network to classify the image semantic features for posture classification improves the accuracy of posture classification. Furthermore, because the teaching posture dataset is constructed from simulated teaching videos in simulated teaching scenarios and real teaching videos in real teaching scenarios, it greatly expands the sample size for posture recognition in teaching scenarios. Therefore, using a posture classification network trained on the teaching posture dataset for teaching posture recognition further enhances the accuracy of teaching posture recognition in teaching scenarios.

[0195] In one embodiment, in training the pose classification network, the classification module 760 is further configured to: acquire a teaching pose dataset, which includes multiple teaching image samples; acquire the teaching pose annotation for each teaching image sample; select a target teaching image sample from the teaching pose dataset, and use a keypoint detection network to extract features from the target teaching image sample to obtain image semantic feature samples; use the pose classification network to be trained to classify the image semantic feature samples to obtain pose prediction results; calculate the model loss based on the teaching pose annotation and pose prediction results of the target teaching image samples; if the model loss does not meet the training stopping condition, adjust the network parameters of the pose classification network to be trained based on the model loss, and continue training the pose classification network to be trained with adjusted network parameters based on the remaining teaching image samples in the teaching pose dataset, until the calculated model loss reaches the training stopping condition, and the trained pose classification network is obtained.

[0196] In one embodiment, the classification module 760 is further configured to: acquire teaching posture datasets; the teaching video samples include simulated teaching videos collected in simulated teaching scenarios and real teaching videos collected in real teaching scenarios; and construct teaching posture datasets based on image frames in the teaching video samples.

[0197] In one embodiment, there are multiple real teaching videos, which are simultaneously captured by recording devices placed in different locations on the same teaching scene. The classification module 760 is further configured to: obtain the teaching posture annotation for each teaching image sample; select a target device from the recording devices placed in different locations; and use the real teaching video corresponding to the target device as the target teaching video; for each image frame in the target teaching video, obtain the manual posture annotation of the current image frame, and obtain target image frames from the remaining real teaching videos that were captured at the same time as the current image frame; and determine that the teaching posture annotation of each target image frame is the manual posture annotation of the current image frame.

[0198] In one embodiment, the teaching posture recognition transpose further includes a decoding module, which is used to: decode the semantic features of the image using the head network of the key point detection network to obtain the structural key points representing the teaching posture; and display the structural key points in the teaching image based on the position of the structural key points in the teaching image.

[0199] In one embodiment, there are multiple teaching postures in the teaching image; the teaching posture recognition device 700 further includes a sorting module, which is used to: obtain a pre-constructed priority order table; the priority order table includes the priority order corresponding to each of the multiple preset teaching postures; search the priority order table to determine the priority order corresponding to each of the multiple teaching postures; and determine the target teaching posture among the multiple teaching postures in the teaching image based on the priority order.

[0200] In one embodiment, the teaching image to be identified is an image frame in the video to be identified; the teaching posture recognition device 700 further includes a display module, which is used to: acquire the teaching posture in each image frame of the video to be identified; perform motion analysis on the teaching posture in each image frame to obtain motion analysis results; and generate and display a teaching posture analysis report based on the teaching posture and motion analysis results in each image frame.

[0201] Based on the same inventive concept, this application also provides a training apparatus for a pose classification network to implement the training method for the pose classification network described above. The solution provided by this apparatus is similar to the implementation described in the above method. Therefore, the specific limitations of one or more pose classification network training apparatus embodiments provided below can be found in the limitations of the pose classification network training method described above, and will not be repeated here.

[0202] In one exemplary embodiment, such as Figure 8 As shown, a training device 800 for a pose classification network is provided, including: a sample acquisition module 810, a label acquisition module 820, a feature sample extraction module 830, a prediction module 840, a loss calculation module 850, and a training module 860, wherein:

[0203] The sample acquisition module 810 is used to acquire the teaching posture dataset, which includes multiple teaching image samples.

[0204] The annotation acquisition module 820 is used to acquire the teaching posture annotations for each teaching image sample.

[0205] The feature sample extraction module 830 is used to select target teaching image samples from the teaching posture dataset, and to extract features from the target teaching image samples using a key point detection network to obtain image semantic feature samples.

[0206] The prediction module 840 is used to perform pose classification on image semantic feature samples using the pose classification network to be trained, and obtain pose prediction results.

[0207] The loss calculation module 850 is used to calculate the model loss based on the teaching posture annotation and posture prediction results of the target teaching image samples.

[0208] The training module 860 is used to adjust the network parameters of the pose classification network to be trained based on the model loss when the model loss does not meet the training stopping condition. Based on the remaining teaching image samples in the teaching pose dataset, the training module continues to train the pose classification network with adjusted network parameters until the calculated model loss reaches the training stopping condition, thus obtaining the trained pose classification network.

[0209] The training device for the aforementioned pose classification network uses a teaching pose dataset comprising multiple teaching image samples. Training the pose classification network based on this dataset and its annotations improves the accuracy of the network in recognizing teaching poses in teaching scenarios. Furthermore, during training, the network parameters of the keypoint detection network are frozen, and only the parameters of the pose classification network being trained are adjusted. This allows the keypoint detection network to retain its ability to output high-level semantic features and detect key points in human poses. Simultaneously, the pose classification network continuously learns its ability to recognize different teaching poses in teaching images during training, resulting in a highly accurate recognition system for teaching poses in teaching scenarios.

[0210] Each module in the aforementioned teaching posture recognition device and posture classification network training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0211] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a training method for a teaching posture recognition and posture classification network. The display unit of the computer device is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0212] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0213] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0214] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0215] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0216] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0217] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0218] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0219] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A teaching posture recognition method, characterized by, The method comprises: obtaining a teaching image to be recognized; using a key point detection network to perform feature extraction on the teaching image to obtain image semantic features; the key point detection network is trained based on a human body data set; using a posture classification network to perform posture classification on the image semantic features to obtain a teaching posture in the teaching image; the posture classification network is trained based on a teaching posture data set; the teaching posture data set is constructed based on simulated teaching videos in a simulated teaching scene and real teaching videos in a real teaching scene.

2. The method of claim 1, wherein, The training steps of the posture classification network comprise: obtaining a teaching posture data set, the teaching posture data set comprising a plurality of teaching image samples; obtaining a teaching posture label of each teaching image sample; selecting a target teaching image sample from the teaching posture data set, and using the key point detection network to perform feature extraction on the target teaching image sample to obtain an image semantic feature sample; using a to-be-trained posture classification network to perform posture classification on the image semantic feature sample to obtain a posture prediction result; based on the teaching posture label of the target teaching image sample and the posture prediction result, calculating a model loss; if the model loss does not satisfy a training stop condition, adjusting network parameters of the to-be-trained posture classification network based on the model loss, and continuing to train the to-be-trained posture classification network with the adjusted network parameters based on the remaining teaching image samples in the teaching posture data set until the calculated model loss reaches the training stop condition, and stopping to obtain a trained posture classification network.

3. The method of claim 2, wherein, The teaching posture data set is obtained by: obtaining a teaching video sample; the teaching video sample comprises simulated teaching videos collected in a simulated teaching scene and real teaching videos collected in a real teaching scene; constructing a teaching posture data set based on image frames in the teaching video sample.

4. The method of claim 3, wherein, There are a plurality of real teaching videos, and the plurality of real teaching videos are obtained by simultaneously shooting the same teaching scene by recording and broadcasting devices placed at different positions; The teaching posture label of each teaching image sample is obtained by: selecting a target device from the recording and broadcasting devices placed at different positions, and taking the real teaching video corresponding to the target device as a target teaching video; for each image frame in the target teaching video, obtaining an artificial posture label of a current image frame, and obtaining target image frames in the remaining real teaching videos that have the same collection time as the current image frame; determining that the teaching posture labels of the target image frames are all the artificial posture label of the current image frame.

5. The method of claim 1, wherein, The method further comprises: using a head network of the key point detection network to decode the image semantic features to obtain structural key points representing the teaching posture; based on the positions of the structural key points in the teaching image, displaying the structural key points in the teaching image.

6. The method of claim 1, wherein, There are a plurality of teaching postures in the teaching image; the method further comprises: obtaining a pre-constructed priority order table; the priority order table comprises a priority order corresponding to each of a plurality of preset teaching postures; The priority order table is searched to determine respective priority orders of the plurality of teaching postures; A target teaching posture is determined from the plurality of teaching postures in the teaching image based on the priority orders.

7. The method of claim 1, wherein, The teaching image to be recognized is an image frame in a video to be recognized; the method further comprises: A teaching posture in each image frame in the video to be recognized is obtained; Action analysis is performed on the teaching posture in each image frame to obtain an action analysis result; and a teaching posture analysis report is generated and displayed based on the teaching posture in each image frame and the action analysis result.

8. A method for training a pose classification network, the method comprising: The method comprises: A teaching posture dataset is obtained, the teaching posture dataset comprising a plurality of teaching image samples; A teaching posture label of each teaching image sample is obtained; A target teaching image sample is selected from the teaching posture dataset, and a key point detection network is used to extract features of the target teaching image sample to obtain an image semantic feature sample; A posture classification network to be trained is used to classify the posture of the image semantic feature sample to obtain a posture prediction result; Based on the teaching posture label of the target teaching image sample and the posture prediction result, a model loss is calculated; If the model loss does not satisfy a training stop condition, the network parameters of the posture classification network to be trained are adjusted based on the model loss, and the posture classification network to be trained with the adjusted network parameters is further trained based on the remaining teaching image samples in the teaching posture dataset until the model loss calculated reaches the training stop condition, and the training of the posture classification network to be trained is stopped, thereby obtaining a trained posture classification network.

9. A teaching posture recognition apparatus, characterized by comprising: The device comprises: An obtaining module is configured to obtain a teaching image to be recognized; A feature extraction module is configured to use a key point detection network to extract features of the teaching image to obtain an image semantic feature, wherein the key point detection network is trained based on a human body dataset; A classification module is configured to use a posture classification network to classify the posture of the image semantic feature to obtain a teaching posture in the teaching image, wherein the posture classification network is trained based on a teaching posture dataset, and the teaching posture dataset is constructed based on a simulated teaching video in a simulated teaching scene and a real teaching video in a real teaching scene.

10. A training device for a pose classification network, characterized in that, The device comprises: A sample obtaining module is configured to obtain a teaching posture dataset, wherein the teaching posture dataset comprises a plurality of teaching image samples; A label obtaining module is configured to obtain a teaching posture label of each teaching image sample; A feature sample extraction module is configured to select a target teaching image sample from the teaching posture dataset, and use a key point detection network to extract features of the target teaching image sample to obtain an image semantic feature sample; A prediction module is configured to use a posture classification network to be trained to classify the posture of the image semantic feature sample to obtain a posture prediction result; A loss calculation module is configured to calculate a model loss based on the teaching posture label of the target teaching image sample and the posture prediction result. The training module is configured to, in a case where the model loss does not satisfy a training stop condition, adjust network parameters of the to-be-trained gesture classification network based on the model loss, and continue training the to-be-trained gesture classification network with the adjusted network parameters based on the rest of the teaching image samples in the teaching gesture data set until the model loss calculated reaches the training stop condition, and stop, thereby obtaining a trained gesture classification network. 11.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-10. The processor implements the steps of the teaching gesture recognition method according to any one of claims 1 to 7 or the steps of the training method of the gesture classification network according to claim 8 when executing the computer program.

12. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the teaching gesture recognition method according to any one of claims 1 to 7 or the steps of the training method of the gesture classification network according to claim 8.

13. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the teaching gesture recognition method according to any one of claims 1 to 7 or the steps of the training method of the gesture classification network according to claim 8. The computer program, when executed by the processor, implements the steps of the teaching gesture recognition method according to any one of claims 1 to 7 or the steps of the training method of the gesture classification network according to claim 8.

Images