A gesture recognition method, system, device and medium

Abstract

The present invention provides a posture recognition method, system, device and medium, comprising: performing self-supervised learning on unlabeled posture sequence images to generate a pre-trained model; inputting labeled posture sequence images into the pre-trained model for iterative training, and adjusting the parameters of the pre-trained model during the iterative training process to generate a posture recognition model for identifying the posture of a target object. The present invention can improve the model's ability to represent human posture and the performance of pedestrian behavior recognition tasks by using a large amount of unlabeled data when there is less labeled data. Moreover, the self-supervised comparative learning method proposed in the present invention effectively utilizes the person ID information and time-series frame information in the human image data, especially by using the same person ID and adjacent time-series images as negative example pairs, which can directly ignore the portrait information within the same person ID and focus on the changes in human posture.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a gesture recognition method, system, device and medium. Background Art

[0002] Pedestrian posture recognition is a key task in computer vision and video surveillance, and has garnered significant attention in recent years. Currently, the mainstream approach to pedestrian posture recognition is to use deep neural networks to learn representations of human images. While end-to-end modeling based on deep neural networks has achieved relatively good results in general recognition tasks, there is still room for improvement in human posture representation tasks, which involve more complex content, advanced semantics, and spatial variability. Furthermore, the representational capabilities of neural networks rely heavily on large-scale annotated data. Due to the scarcity of abnormal posture data in surveillance scenarios, trained models often struggle to achieve good representational and generalization capabilities. Summary of the Invention

[0003] In view of the above-mentioned shortcomings of the prior art, an object of the present invention is to provide a gesture recognition method, system, device and medium for solving the problems existing in the prior art.

[0004] To achieve the above and other related purposes, the present invention provides a gesture recognition method, which includes the following steps:

[0005] Perform self-supervised learning on unlabeled pose sequence images to generate a pre-trained model;

[0006] The labeled posture sequence pictures are input into the pre-training model for iterative training, and the parameters of the pre-training model are adjusted during the iterative training process to generate a posture recognition model for recognizing the posture of the target object.

[0007] Optionally, the process of performing self-supervised learning on unlabeled pose sequence images to generate a pre-trained model includes:

[0008] Perform object detection and frame processing on the target video to generate multi-frame unlabeled posture sequence images;

[0009] Obtain the object number and the picture frame number of each frame of the unlabeled posture sequence picture, and form the unlabeled posture sequence pictures of the positive example pair and the unlabeled posture sequence pictures of the negative example pair according to the object number and the picture frame number;

[0010] The unlabeled posture sequence images corresponding to all positive example pairs and the unlabeled posture sequence images corresponding to the negative example pairs are used as training data, and self-supervised learning is performed on the training data to generate the pre-training model.

[0011] Optionally, the process of forming an unlabeled pose sequence picture of a positive example pair and an unlabeled pose sequence picture of a negative example pair according to the object number and the picture frame number includes:

[0012] Perform data augmentation on each frame of the original unlabeled pose sequence image to obtain an enhanced image corresponding to the original unlabeled pose sequence image, and associate the enhanced image with the original unlabeled pose sequence image to form a positive example pair of unlabeled pose sequence images;

[0013] Obtain unlabeled pose sequence images with the same object number and adjacent picture frame numbers, and associate the unlabeled pose sequence images with the same object number and adjacent picture frame numbers to form unlabeled pose sequence images of negative example pairs.

[0014] Optionally, the method further comprises: performing self-supervised learning on the training data using a convolutional neural network to obtain pre-trained weights corresponding to the convolutional neural network;

[0015] The pre-training weights are combined with the network structure corresponding to the convolutional neural network to generate the pre-training model.

[0016] Optionally, when adjusting the parameters of the pre-trained model during the iterative training process, it includes adjusting the pre-trained weights in the pre-trained model.

[0017] Optionally, the method of performing data enhancement on the original unlabeled posture sequence images includes at least one of the following: adjusting the temporal distance of the images, randomly cropping the original unlabeled posture sequence images, color jittering the original unlabeled posture sequence images, randomly flipping the original unlabeled posture sequence images, randomly changing the aspect ratio of the original unlabeled posture sequence images, and randomly adding noise to the original unlabeled posture sequence images.

[0018] Optionally, when performing data enhancement on the original unlabeled posture sequence images, it also includes adjusting the intensity of the data enhancement; wherein, the method of adjusting the intensity of the data enhancement includes at least one of the following: increasing the range of random cropping of the original unlabeled posture sequence images, increasing the intensity of color jittering of the original unlabeled posture sequence images, increasing the intensity of random flipping of the original unlabeled posture sequence images, increasing the intensity of randomly changing the aspect ratio of the original unlabeled posture sequence images, and increasing the intensity of randomly adding noise to the original unlabeled posture sequence images.

[0019] Optionally, the method further includes: obtaining all unlabeled posture sequence images containing the same object number, and recording them as images to be processed;

[0020] Randomly select two frames of unlabeled posture sequence pictures from the pictures to be processed for comparison, and determine whether the posture change range of the two selected unlabeled posture sequence pictures is greater than a preset range;

[0021] If it is greater than or equal to the preset range, the two selected frames of unlabeled posture sequence images are retained; if it is less than the preset range, only one of the selected frames of unlabeled posture sequence images is included.

[0022] The present invention also provides a gesture recognition system, the system comprising:

[0023] Self-supervised training module, used to perform self-supervised learning on unlabeled pose sequence images to generate a pre-trained model;

[0024] The adjustment training module is used to input the labeled posture sequence pictures into the pre-training model for iterative training, and adjust the parameters of the pre-training model during the iterative training process to generate a posture recognition model for identifying the posture of the target object.

[0025] The present invention also provides a computer device, comprising:

[0026] one or more processors; and

[0027] A computer-readable medium storing instructions, when the one or more processors execute the instructions, causes the device to perform any of the methods described above.

[0028] The present invention also provides a computer-readable medium having instructions stored thereon. When the instructions are executed by one or more processors, the device executes any one of the methods described above.

[0029] As described above, the present invention provides a posture recognition method, system, device, and medium, which have the following beneficial effects: the present invention first performs self-supervised learning on unlabeled posture sequence images to generate a pre-trained model; then, labeled posture sequence images are input into the pre-trained model for iterative training, and during the iterative training process, parameters of the pre-trained model are adjusted to generate a posture recognition model for identifying the posture of a target object. Therefore, the present invention can first generate a pre-trained model by performing self-supervised learning on unlabeled posture sequence images, and then adjust the pre-trained model using labeled posture sequence images to generate a posture recognition model for identifying the posture of a target object. This is equivalent to first training a model based on unlabeled sequence images and then adjusting the trained model using labeled posture sequence images. Training with unlabeled sequence images can reduce the amount of data annotation; at the same time, training and adjusting with labeled posture sequence images also increases the model's representational and generalization capabilities. Therefore, the present invention can enhance the model's representational capabilities for human posture by utilizing a large amount of unlabeled data, even when there is relatively little labeled data, thereby improving performance in pedestrian behavior recognition tasks. In addition, the self-supervised comparative learning method proposed in the present invention effectively utilizes the person ID information and time-series frame information in the human image data. In particular, by taking pictures with the same person ID and adjacent time sequences as negative pairs, the portrait information within the same person ID can be directly ignored, and the focus can be placed on the changes in human posture. Moreover, by improving the model's ability to represent human posture through large-scale unlabeled data, the model's representation ability and accuracy in pedestrian behavior recognition tasks can also be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of a flow chart of an object posture recognition method provided by an embodiment;

[0031] Figure 2 A schematic diagram of a process for generating a pre-training model according to an embodiment;

[0032] Figure 3 A schematic flow chart of an object posture recognition method provided in another embodiment;

[0033] Figure 4 A schematic diagram of the hardware structure of an object gesture recognition system provided by an embodiment;

[0034] Figure 5 A schematic diagram of the hardware structure of a terminal device provided in one embodiment;

[0035] Figure 6 A schematic diagram of the hardware structure of a terminal device provided in another embodiment.

[0036] Component number description

[0037] M10 self-supervised training module

[0038] M20 Conditioning Training Module

[0039] 1100 Input Devices

[0040] 1101 First Processor

[0041] 1102 Output Devices

[0042] 1103 First Memory

[0043] 1104 Communication Bus

[0044] 1200 Processing Components

[0045] 1201 Second Processor

[0046] 1202 Second Memory

[0047] 1203 Communication Components

[0048] 1204 Power Supply Assembly

[0049] 1205 Multimedia Components

[0050] 1206 Audio Components

[0051] 1207 Input / Output Interface

[0052] 1208 sensor assembly DETAILED DESCRIPTION

[0053] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0054] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0055] The current common approach for pedestrian posture recognition tasks is to input a single or sequential human body detection image into a neural network for feature extraction and classification. However, in monitoring scenarios, there may be multiple angles, heights, and lighting conditions, as well as the relatively broad semantics of pedestrian postures. This results in large differences in image features for the same posture definition, posing certain challenges to the model's representation learning. The inventors discovered that a model with an attention mechanism can better learn the fine-grained features of human posture, but the introduction of more parameters often requires more training data to prevent the model from overfitting. Abnormal human postures, such as fighting, climbing, and sleeping, are relatively rare in monitoring scenarios, making it difficult to optimize model parameters and focus on the semantic features of human posture.

[0056] Therefore, this embodiment provides a gesture recognition method. Figure 1 and Figure 2 As shown, the method includes the following steps:

[0057] S100, performing self-supervised learning on unlabeled pose sequence images to generate a pre-trained model. Specifically, the process of generating the pre-trained model includes:

[0058] S110, performing object detection and frame processing on the target video to generate a multi-frame unlabeled posture sequence image;

[0059] S120, obtaining an object number and a picture frame number of each frame of the unlabeled posture sequence picture, and forming an unlabeled posture sequence picture of a positive example pair and an unlabeled posture sequence picture of a negative example pair according to the object number and the picture frame number;

[0060] S130 , using the unlabeled posture sequence images corresponding to all positive example pairs and the unlabeled posture sequence images corresponding to the negative example pairs as training data, and performing self-supervised learning on the training data to generate the pre-training model.

[0061] S200: Inputting a sequence of labeled pose images into the pre-trained model for iterative training, and adjusting parameters of the pre-trained model during the iterative training process to generate a pose recognition model for recognizing the pose of a target object. In this embodiment, the target object includes, but is not limited to, a pedestrian in a video or a pedestrian in a single-frame image.

[0062] In an exemplary implementation, the process of forming a positive pair of unlabeled pose sequence images according to the object number and the picture frame number, and forming a negative pair of unlabeled pose sequence images according to the object number and the picture frame number includes:

[0063] Data enhancement is performed on each frame of the original unlabeled pose sequence image, an enhanced image corresponding to the original unlabeled pose sequence image is obtained, and the enhanced image and the original unlabeled pose sequence image are associated to form a positive unlabeled pose sequence image pair; unlabeled pose sequence images with the same object number and adjacent image frame number are obtained, and unlabeled pose sequence images with the same object number and adjacent image frame number are associated to form a negative unlabeled pose sequence image pair. The data enhancement method for the original unlabeled pose sequence image includes at least one of the following: adjusting the temporal distance of the image, randomly cropping the original unlabeled pose sequence image, color jittering the original unlabeled pose sequence image, randomly flipping the original unlabeled pose sequence image, randomly changing the aspect ratio of the original unlabeled pose sequence image, and randomly adding noise to the original unlabeled pose sequence image. This embodiment can also dynamically adjust the temporal distance of the image and the data enhancement strength according to the training progress. As an example, as the training progresses, the sampling spacing of the human body sequence image can be adjusted from large to small, and the degree of data enhancement can be adjusted from weak to strong. Among them, when adjusting the intensity of data enhancement in this embodiment, the methods adopted include but are not limited to one of the following: increasing the range of random cropping of the original unlabeled posture sequence pictures, increasing the intensity of color jittering of the original unlabeled posture sequence pictures, increasing the intensity of random flipping of the original unlabeled posture sequence pictures, increasing the intensity of randomly changing the aspect ratio of the original unlabeled posture sequence pictures, and increasing the intensity of randomly adding noise to the original unlabeled posture sequence pictures.

[0064] In an exemplary embodiment, the method further includes: using a convolutional neural network to perform self-supervised learning on the training data to obtain pre-trained weights corresponding to the convolutional neural network; combining the pre-trained weights with the network structure corresponding to the convolutional neural network to generate the pre-trained model. Wherein, when adjusting the parameters of the pre-trained model during the iterative training process, it includes but is not limited to adjusting the pre-trained weights in the pre-trained model. Wherein, when training and generating the pre-trained model and the behavior recognition model, the training epoch can be set according to actual conditions, and the specific value of the epoch is not limited in this embodiment. As an example, for example, when generating the pre-trained model, the training epoch can be 600; when generating the behavior recognition model, the training epoch can be 30. At the same time, when the training epoch of this embodiment reaches the corresponding number, the training process can be terminated and then the corresponding model can be generated.

[0065] In an exemplary implementation, the method further includes: obtaining all unlabeled posture sequence images containing the same object number, recording them as images to be processed; arbitrarily selecting two frames of unlabeled posture sequence images from the images to be processed for comparison, and determining whether the posture change range of the selected two frames of unlabeled posture sequence images is greater than a preset range; if it is greater than or equal to the preset range, retaining the two selected frames of unlabeled posture sequence images; if it is less than the preset range, only including one of the selected frames of unlabeled posture sequence images. This embodiment deduplicates images with the same object number or the same character ID by comparing the posture change range, and then filters out images with no obvious posture change. As an example, the method of comparing the posture change range in this embodiment can use the difference hash algorithm DHash, or other methods that can identify image differences.

[0066] In addition, traditional self-supervised contrastive learning treats each image as an instance, and different data enhancements of the same image are positive pairs, and different instances are negative pairs. Since the feature differences between human IDs are greater than the differences in posture changes within adjacent sequences, it is easy for the model to learn the non-posture features of the human body. At the same time, general self-supervised learning methods all constrain the consistency of the feature vectors of an instance after different data enhancements, and perform characterization learning on unlabeled data. These methods do not use the correlation and temporal relationship of the IDs of human images, and are not targeted at the characterization of human posture. Therefore, in another exemplary embodiment, such as Figure 3As shown, this embodiment also provides a posture recognition method, comprising the following steps: detecting and tracking pedestrians in a video to generate a sequence of human images; then deduplicating the sequence of human images; then using unlabeled human image sequences for self-supervised contrastive learning to train pre-trained weights; and finally fine-tuning the pre-trained weights on labeled data. Specifically, a large video containing pedestrians is detected and tracked, and the images, including person IDs and frame numbers, are saved to generate unlabeled human image sequence data. Images associated with each person ID are deduplicated using DHash or other methods, filtering out images with no significant posture changes. The unlabeled human image sequence data is then trained using the Moco v2 self-supervised contrastive learning algorithm. The order in which data is input to the model is modified so that each human image sequence is distributed into different batches based on frame number. This allows each image to serve as a positive example against its own image using different data augmentation methods and as a negative example against other poses in the same sequence in the memory bank, effectively contrasting different poses within the same person ID sequence. Moco v2 is a momentum-based self-supervised contrastive learning framework proposed by Facebook AI Research in 2020. Memory bank refers to the feature encoding vector generated for the input image. The method for self-supervised contrastive learning using unlabeled human image sequences is described in a course. The course learning method used in this embodiment is a training method in deep learning, where the difficulty of the task increases as the model trains. In addition to the momentum-based parameter updates and Moco v2 with a memory bank module, the self-supervised learning in this embodiment also includes SwAV, which clusters samples with symmetric structures and parameters and then distinguishes each cluster; BYOL, which has a symmetric structure and asymmetric parameters and does not use negative examples; and SimSiam, which uses a Siamese network by exchanging encoders and constraining feature vector consistency. Finally, the pre-trained weights obtained from self-supervised contrastive learning are fine-tuned using labeled pedestrian behavior data. Therefore, this embodiment addresses existing problems by using large-scale unlabeled human image sequences for self-supervised contrastive learning to obtain pre-trained model weights, which can then be fine-tuned on labeled data to improve the model's ability to learn and represent human posture. Furthermore, this embodiment uses images of people with the same person ID but different poses as negative pairs, allowing the model to more easily capture pose differences between people with the same ID. In this embodiment, the unlabeled human image sequence can include images of people dancing, while the labeled human image sequence can include images of people climbing and / or climbing over a railing.

[0067] In summary, the present invention provides a posture recognition method that first performs self-supervised learning on unlabeled posture sequence images to generate a pre-trained model; then, labeled posture sequence images are input into the pre-trained model for iterative training, and during the iterative training process, parameters of the pre-trained model are adjusted to generate a posture recognition model for identifying the posture of a target object. Thus, the present invention can first generate a pre-trained model by performing self-supervised learning on unlabeled posture sequence images, and then adjust the pre-trained model using labeled posture sequence images to generate a posture recognition model for identifying the posture of a target object. This is equivalent to the present invention first training a model based on unlabeled sequence images and then adjusting the trained model using labeled posture sequence images; training with unlabeled sequence images can reduce the amount of data annotation; at the same time, training and adjustment with labeled posture sequence images also increases the model's representational and generalization capabilities. Therefore, the present invention can improve the model's representational capabilities for human posture and enhance the performance of pedestrian behavior recognition tasks by using a large amount of unlabeled data when there is less labeled data. In addition, the self-supervised comparative learning method proposed in the present invention effectively utilizes the person ID information and time-series frame information in the human image data, especially taking the pictures with the same person ID and adjacent time sequences as negative pairs, which can directly ignore the portrait information within the same person ID and focus on the changes in human posture; and through large-scale unlabeled data, the model's ability to represent human posture is improved, and the model's representation ability and accuracy in pedestrian behavior recognition tasks can also be improved. Therefore, the present invention uses large-scale unlabeled human posture data through self-supervised comparative learning. In the case of less labeled data, it can improve model performance and reduce the difficulty of data acquisition and labeling. Moreover, this method utilizes the feature differences within the human sequence and between different sequences, and through a carefully designed data enhancement method, by comparing the posture changes between human sequences, it can enhance the model's learning of human posture features. And the model training process in this method can refer to the course learning method, so that the data enhancement intensity and the similarity between sequences can be dynamically adjusted during the training process, making the model training easier to converge and improving training efficiency.

[0068] like Figure 4 As shown, the present invention also provides a gesture recognition system, the system comprising:

[0069] The self-supervised training module M10 is used to perform self-supervised learning on unlabeled posture sequence images to generate a pre-training model.

[0070] Specifically, the process of generating a pre-trained model includes:

[0071] Perform object detection and frame processing on the target video to generate multi-frame unlabeled posture sequence images;

[0072] Obtain the object number and the picture frame number of each frame of the unlabeled posture sequence picture, and form the unlabeled posture sequence pictures of the positive example pair and the unlabeled posture sequence pictures of the negative example pair according to the object number and the picture frame number;

[0073] The unlabeled posture sequence images corresponding to all positive example pairs and the unlabeled posture sequence images corresponding to the negative example pairs are used as training data, and self-supervised learning is performed on the training data to generate the pre-training model.

[0074] The adjustment training module M20 is configured to input a sequence of labeled pose images into the pre-trained model for iterative training, and to adjust parameters of the pre-trained model during the iterative training process to generate a pose recognition model for identifying the pose of a target object. In this embodiment, the target object includes, but is not limited to, a pedestrian in a video or a single-frame image.

[0075] In an exemplary implementation, the process of forming a positive pair of unlabeled pose sequence images according to the object number and the picture frame number, and forming a negative pair of unlabeled pose sequence images according to the object number and the picture frame number includes:

[0076] Data enhancement is performed on each frame of the original unlabeled pose sequence image, an enhanced image corresponding to the original unlabeled pose sequence image is obtained, and the enhanced image and the original unlabeled pose sequence image are associated to form a positive unlabeled pose sequence image pair; unlabeled pose sequence images with the same object number and adjacent image frame number are obtained, and unlabeled pose sequence images with the same object number and adjacent image frame number are associated to form a negative unlabeled pose sequence image pair. The data enhancement method for the original unlabeled pose sequence image includes at least one of the following: adjusting the temporal distance of the image, randomly cropping the original unlabeled pose sequence image, color jittering the original unlabeled pose sequence image, randomly flipping the original unlabeled pose sequence image, randomly changing the aspect ratio of the original unlabeled pose sequence image, and randomly adding noise to the original unlabeled pose sequence image. This embodiment can also dynamically adjust the temporal distance of the image and the data enhancement strength according to the training progress. As an example, as the training progresses, the sampling spacing of the human body sequence image can be adjusted from large to small, and the degree of data enhancement can be adjusted from weak to strong. Among them, when adjusting the intensity of data enhancement in this embodiment, the methods adopted include but are not limited to one of the following: increasing the range of random cropping of the original unlabeled posture sequence pictures, increasing the intensity of color jittering of the original unlabeled posture sequence pictures, increasing the intensity of random flipping of the original unlabeled posture sequence pictures, increasing the intensity of randomly changing the aspect ratio of the original unlabeled posture sequence pictures, and increasing the intensity of randomly adding noise to the original unlabeled posture sequence pictures.

[0077] In an exemplary embodiment, the system further includes: using a convolutional neural network to perform self-supervised learning on the training data to obtain pre-trained weights corresponding to the convolutional neural network; combining the pre-trained weights with the network structure corresponding to the convolutional neural network to generate the pre-trained model. Particularly, when adjusting the parameters of the pre-trained model during the iterative training process, it includes but is not limited to adjusting the pre-trained weights in the pre-trained model. Particularly, when training and generating the pre-trained model and the behavior recognition model, the training epoch can be set according to actual conditions, and the present embodiment does not impose any restrictions on the specific value of the epoch. As an example, for example, when generating the pre-trained model, the training epoch can be 600; when generating the behavior recognition model, the training epoch can be 30. At the same time, when the training epoch of the present embodiment reaches the corresponding number, the training process can be terminated, and then the corresponding model can be generated.

[0078] In an exemplary embodiment, the system further includes: obtaining all unlabeled posture sequence images containing the same object number, which are recorded as images to be processed; arbitrarily selecting two frames of unlabeled posture sequence images from the images to be processed for comparison, and determining whether the posture change range of the two selected unlabeled posture sequence images is greater than a preset range; if it is greater than or equal to the preset range, retaining the two selected unlabeled posture sequence images; if it is less than the preset range, only including one of the selected unlabeled posture sequence images. This embodiment deduplicates images with the same object number or the same character ID by comparing the posture change range, and then filters out images with no obvious posture change. As an example, the method of comparing the posture change range in this embodiment can use the difference hash algorithm DHash, or other methods that can identify image differences.

[0079] Furthermore, traditional self-supervised contrastive learning treats each image as an instance, with different data augmentations of the same image serving as positive pairs and different instances serving as negative pairs. Because the feature differences between person IDs are greater than the differences in pose changes within adjacent sequences, the model is more susceptible to learning non-pose features of the person. Furthermore, common self-supervised learning methods constrain the consistency of the feature vectors of an instance after different data augmentations, and perform representation learning on unlabeled data. These methods fail to utilize the correlation and temporal relationships between IDs of human images and are therefore not targeted at representing human poses. Therefore, in another exemplary embodiment, this embodiment also provides a pose recognition system that performs the following steps: detecting and tracking pedestrians in a video to generate a sequence of human images; then deduplicating the sequence of human images; then performing self-supervised contrastive learning on the unlabeled sequence of human images to train pre-trained weights; and finally, fine-tuning the pre-trained weights on labeled data. Specifically, large-scale videos containing pedestrians are detected and tracked, and images containing person IDs and frame numbers are saved to generate unlabeled sequence of human images. Use DHash or other methods to deduplicate the images under each character ID and filter out images with no obvious changes in posture. Then use the Moco v2 self-supervised contrastive learning algorithm to train the above-mentioned unlabeled human sequence data. Modify the order of data input into the model so that each human sequence image is distributed to different batches according to the frame number, so that each image can be a positive example pair with its own image in different data enhancement methods, and a negative example with other postures in the same sequence in the Memory bank, forming an effective comparison of different postures in the same character ID sequence. Among them, Moco v2 is a momentum-based self-supervised contrastive learning framework proposed by Facebook AI Research in 2020; Memory bank refers to the feature encoding vector generated for the input image. Among them, the method of using unlabeled human sequence images for self-supervised contrastive learning refers to course learning. The course learning method in this embodiment is a training method in deep learning. As the model is trained, the difficulty of the task will change from easy to difficult. In addition to the Moco v2 with momentum update parameters and a memory bank module, the self-supervised learning in this embodiment also includes SwAV, which clusters various samples with symmetrical structures and parameters and then distinguishes each cluster; BYOL, which has a symmetrical structure and asymmetric parameters and does not use negative examples; and SimSiam, which uses a twin network by exchanging encoders and constraining feature vector consistency. Finally, the pre-trained weights obtained through self-supervised contrastive learning are fine-tuned using labeled pedestrian behavior data. Therefore, this embodiment addresses existing issues by using large-scale unlabeled human image sequences for self-supervised contrastive learning to obtain pre-trained model weights, which are then fine-tuned on labeled data to improve the model's ability to learn and represent human posture.Furthermore, this embodiment uses images of people with the same person ID but different poses as negative pairs, allowing the model to more easily capture pose differences between people with the same ID. In this embodiment, the unlabeled human image sequence can include images of people dancing, while the labeled human image sequence can include images of people climbing and / or climbing over a railing.

[0080] In another exemplary embodiment, a gesture recognition system is further provided, comprising:

[0081] The human body detection and tracking module is used to detect and track pedestrians in the video and obtain sequence detection images of the human body.

[0082] The similar image deduplication module is used to perform similarity deduplication on pedestrian static images of the same sequence and filter out images with small posture changes.

[0083] The human posture representation learning module is used to compare and learn the posture changes of adjacent time series of the same human ID in a self-supervised manner, so that the model ignores pedestrian feature information and focuses more on learning the human posture.

[0084] The course-based data augmentation module dynamically adjusts the temporal distance of images and the strength of data augmentation based on the training progress. In this method, the module applies five data transformation methods to the model training: random cropping, color jittering, random flipping, random aspect ratio, and random noise.

[0085] In summary, the present invention provides a posture recognition system that first performs self-supervised learning on unlabeled posture sequence images to generate a pre-trained model; then, labeled posture sequence images are input into the pre-trained model for iterative training, and during the iterative training process, parameters of the pre-trained model are adjusted to generate a posture recognition model for identifying the posture of a target object. Thus, the present invention can first generate a pre-trained model by performing self-supervised learning on unlabeled posture sequence images, and then adjust the pre-trained model using labeled posture sequence images to generate a posture recognition model for identifying the posture of a target object. This is equivalent to the present invention first training a model based on unlabeled sequence images and then adjusting the trained model using labeled posture sequence images; while training with unlabeled sequence images can reduce the amount of data annotation, training and adjusting with labeled posture sequence images also increases the model's representational and generalization capabilities. Therefore, the present invention can improve the model's representational capabilities for human posture and enhance the performance of pedestrian behavior recognition tasks by using a large amount of unlabeled data when there is less labeled data. In addition, the self-supervised comparative learning method proposed in the present invention effectively utilizes the person ID information and time-series frame information in the human image data, especially taking the pictures with the same person ID and adjacent time sequences as negative pairs, which can directly ignore the portrait information within the same person ID and focus on the changes in human posture; and through large-scale unlabeled data, the model's ability to represent human posture is improved, and the model's representation ability and accuracy in pedestrian behavior recognition tasks can also be improved. Therefore, the present invention uses large-scale unlabeled human posture data through self-supervised comparative learning. In the case of less labeled data, it can improve model performance and reduce the difficulty of data acquisition and labeling. Moreover, this system utilizes the feature differences within the human sequence and between different sequences, and through a carefully designed data enhancement method, by comparing the posture changes between human sequences, it can enhance the model's learning of human posture features. In addition, the model training process in this system can refer to the course learning system, so that the data enhancement intensity and the similarity between sequences can be dynamically adjusted during the training process, making the model training easier to converge and improving training efficiency.

[0086] The present application also provides a computer device, which may include: one or more processors; and one or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, enable the device to execute Figure 1In practical applications, the device can be used as a terminal device or as a server. Examples of terminal devices may include: smartphones, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, car computers, desktop computers, set-top boxes, smart TVs, wearable devices, etc. The embodiments of this application do not limit the specific devices.

[0087] The present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute the embodiment of the present application. Figure 1 Instructions for the steps involved in a data processing method.

[0088] Figure 5 A schematic diagram of the hardware structure of a terminal device provided in one embodiment of the present application. As shown in the figure, the terminal device may include: an input device 1100, a first processor 1101, an output device 1102, a first memory 1103, and at least one communication bus 1104. Communication bus 1104 is used to achieve communication connections between components. First memory 1103 may include high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk storage device. Various programs can be stored in first memory 1103 to perform various processing functions and implement the method steps of this embodiment.

[0089] Optionally, the first processor 1101 may be implemented as, for example, a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components, and the processor 1101 is coupled to the input device 1100 and the output device 1102 via a wired or wireless connection.

[0090] Optionally, the input device 1100 may include multiple input devices, such as at least one of a user interface for a user, a device interface for a device, a software programmable interface, a camera, and a sensor. Optionally, the device interface for a device may be a wired interface for data transmission between devices, or a hardware plug-in interface for data transmission between devices (such as a USB interface, a serial port, etc.); optionally, the user interface for a user may be, for example, a user-oriented control button, a voice input device for receiving voice input, and a touch sensing device for receiving user touch input (such as a touch screen or touchpad with touch sensing function); optionally, the software programmable interface may be, for example, an entry for a user to edit or modify a program, such as an input pin interface or input interface of a chip; the output device 1102 may include output devices such as a display and a speaker.

[0091] In this embodiment, the processor of the terminal device includes functions for executing each module of the voice recognition device in each device. The specific functions and technical effects can be referred to the above embodiments and will not be repeated here.

[0092] Figure 6 A schematic diagram of the hardware structure of a terminal device provided for another embodiment of the present application. Figure 6 Yes Figure 5 A specific embodiment in the implementation process. As shown in the figure, the terminal device of this embodiment may include a second processor 1201 and a second memory 1202.

[0093] The second processor 1201 executes the computer program code stored in the second memory 1202 to implement the above embodiment. Figure 1 The method.

[0094] The second memory 1202 is configured to store various types of data to support operations on the terminal device. Examples of such data include instructions for any application or method operating on the terminal device, such as messages, images, and videos. The second memory 1202 may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0095] Optionally, the second processor 1201 is provided in the processing component 1200. The terminal device may further include: a communication component 1203, a power component 1204, a multimedia component 1205, an audio component 1206, an input / output interface 1207, and / or a sensor component 1208. The specific components included in the terminal device are set according to actual needs and are not limited in this embodiment.

[0096] The processing component 1200 generally controls the overall operation of the terminal device. The processing component 1200 may include one or more second processors 1201 to execute instructions to complete the above Figure 1 In addition, the processing component 1200 may include one or more modules to facilitate interaction between the processing component 1200 and other components. For example, the processing component 1200 may include a multimedia module to facilitate interaction between the multimedia component 1205 and the processing component 1200.

[0097] The power supply component 1204 provides power to various components of the terminal device. The power supply component 1204 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the terminal device.

[0098] The multimedia component 1205 includes a display screen that provides an output interface between the terminal device and the user. In some embodiments, the display screen may include a liquid crystal display (LCD) and a touch panel (TP). If the display screen includes a touch panel, the display screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0099] The audio component 1206 is configured to output and / or input voice signals. For example, the audio component 1206 includes a microphone (MIC), which is configured to receive external voice signals when the terminal device is in an operating mode, such as a voice recognition mode. The received voice signals can be further stored in the second memory 1202 or transmitted via the communication component 1203. In some embodiments, the audio component 1206 also includes a speaker for outputting voice signals.

[0100] The input / output interface 1207 provides an interface between the processing component 1200 and peripheral interface modules, which may be click wheels, buttons, etc. These buttons may include but are not limited to: volume buttons, start buttons, and lock buttons.

[0101] Sensor component 1208 includes one or more sensors for providing various status assessments for the terminal device. For example, sensor component 1208 can detect the open / closed state of the terminal device, the relative positioning of components, and the presence or absence of user contact with the terminal device. Sensor component 1208 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact, including detecting the distance between the user and the terminal device. In some embodiments, sensor component 1208 may also include a camera, etc.

[0102] The communication component 1203 is configured to facilitate wired or wireless communication between the terminal device and other devices. The terminal device can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In one embodiment, the terminal device may include a SIM card slot for inserting a SIM card, allowing the terminal device to log into a GPRS network and establish communication with a server via the Internet.

[0103] From the above, we can see that Figure 6 The communication component 1203, audio component 1206, input / output interface 1207, and sensor component 1208 involved in the embodiment can all be used as Figure 5 Implementation of the input device in the embodiment.

[0104] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A gesture recognition method, characterized in that: The method comprises the following steps: The method performs self-supervised learning on unlabeled posture sequence pictures to generate a pre-trained model; the method includes: performing object detection and frame processing on the target video, saving the object number and picture frame number, and generating multiple frames of unlabeled posture sequence pictures; obtaining the object number and picture frame number of each frame of the unlabeled posture sequence picture, and performing data enhancement on each frame of the original unlabeled posture sequence picture, obtaining the enhanced picture corresponding to the original unlabeled posture sequence picture, and associating the enhanced picture with the original unlabeled posture sequence picture to form a positive pair of unlabeled posture sequence pictures; obtaining unlabeled posture sequence pictures with the same object number and adjacent picture frame number, and encoding the object The unlabeled posture sequence pictures with the same number and adjacent picture frame numbers are associated to form unlabeled posture sequence pictures of negative example pairs; the unlabeled posture sequence pictures corresponding to all positive example pairs and the unlabeled posture sequence pictures corresponding to the negative example pairs are used as training data, and self-supervised learning is performed on the training data, and the input order of the unlabeled posture sequence pictures is modified during the self-supervised learning process, and each unlabeled posture sequence picture is made to be a positive example pair with similar pictures of different data augmentation methods and a negative example with other postures in the feature coding vector sequence, and different postures in the same object number sequence are effectively compared to generate the pre-trained model; The labeled posture sequence pictures are input into the pre-training model for iterative training, and the parameters of the pre-training model are adjusted during the iterative training process to generate a posture recognition model for recognizing the posture of the target object.

2. The gesture recognition method according to claim 1, characterized in that: The method further comprises: Performing self-supervised learning on the training data using a convolutional neural network to obtain pre-trained weights corresponding to the convolutional neural network; The pre-training weights are combined with the network structure corresponding to the convolutional neural network to generate the pre-training model.

3. The gesture recognition method according to claim 2, characterized in that: When adjusting the parameters of the pre-training model during the iterative training process, it includes adjusting the pre-training weights in the pre-training model.

4. The gesture recognition method according to claim 1, wherein: The method of performing data enhancement on the original unlabeled posture sequence images includes at least one of the following: adjusting the temporal distance of the images, randomly cropping the original unlabeled posture sequence images, color jittering the original unlabeled posture sequence images, randomly flipping the original unlabeled posture sequence images, randomly changing the aspect ratio of the original unlabeled posture sequence images, and randomly adding noise to the original unlabeled posture sequence images.

5. The gesture recognition method according to claim 4, characterized in that: When performing data enhancement on the original unlabeled posture sequence images, it also includes adjusting the intensity of the data enhancement; wherein, the method of adjusting the intensity of the data enhancement includes at least one of the following: increasing the range of random cropping of the original unlabeled posture sequence images, increasing the intensity of color jittering of the original unlabeled posture sequence images, increasing the intensity of random flipping of the original unlabeled posture sequence images, increasing the intensity of randomly changing the aspect ratio of the original unlabeled posture sequence images, and increasing the intensity of randomly adding noise to the original unlabeled posture sequence images.

6. The gesture recognition method according to claim 1, characterized in that: The method further comprises: Get all unlabeled pose sequence images containing the same object number and record them as images to be processed; Randomly select two frames of unlabeled posture sequence pictures from the pictures to be processed for comparison, and determine whether the posture change range of the two selected unlabeled posture sequence pictures is greater than a preset range; If it is greater than or equal to the preset range, the two selected frames of unlabeled posture sequence images are retained; if it is less than the preset range, only one of the selected frames of unlabeled posture sequence images is included.

7. A gesture recognition system, characterized in that: The system includes: The self-supervised training module is used to perform self-supervised learning on unlabeled posture sequence pictures and generate a pre-training model; it includes: performing object detection and frame processing on the target video, saving the object number and picture frame number, and generating multiple frames of unlabeled posture sequence pictures; obtaining the object number and picture frame number of each frame of the unlabeled posture sequence picture, and performing data enhancement on each frame of the original unlabeled posture sequence picture, obtaining an enhanced picture corresponding to the original unlabeled posture sequence picture, and associating the enhanced picture with the original unlabeled posture sequence picture to form a positive example pair of unlabeled posture sequence pictures; obtaining unlabeled posture sequence pictures with the same object number and adjacent picture frame numbers, and associating unlabeled pose sequence images with the same object number and adjacent picture frame number to form unlabeled pose sequence images of negative example pairs; using the unlabeled pose sequence images corresponding to all positive example pairs and the unlabeled pose sequence images corresponding to the negative example pairs as training data, and performing self-supervised learning on the training data, and modifying the input order of the unlabeled pose sequence images during the self-supervised learning process, and making each unlabeled pose sequence image a positive example pair with similar images of different data augmentation methods and a negative example with other poses in the feature encoding vector sequence, and effectively comparing different poses in the same object number sequence to generate the pre-trained model; The adjustment training module is used to input the labeled posture sequence pictures into the pre-training model for iterative training, and adjust the parameters of the pre-training model during the iterative training process to generate a posture recognition model for identifying the posture of the target object.

8. A computer device, characterized in that: include: one or more processors; and A computer-readable medium storing instructions, which, when the one or more processors execute the instructions, causes the device to perform the method according to any one of claims 1 to 6.

9. A computer-readable medium, characterized in that Instructions are stored thereon, which, when executed by one or more processors, cause the device to perform the method according to any one of claims 1 to 6.

Images