Posture recognition method, action recognition method and model training method

Abstract

The invention provides a posture recognition method, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, virtual reality, large models and the like, and can be applied to scenes such as meta universe, content generation based on artificial intelligence, virtual digital humans and the like. The posture recognition method comprises the steps of performing feature extraction on key point position information of a target object in a to-be-recognized image to obtain key point features; using the key point features to update attitude features to obtain target features, the attitude features including image features obtained by processing the to-be-identified image; and performing posture recognition on the target features to obtain a posture recognition result of the target object. The invention further provides an action recognition method, a model training method and device, electronic equipment, a storage medium and a computer program product.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, especially to technical fields such as computer vision, deep learning, virtual reality, and large models, and can be applied to scenarios such as the metaverse, content generation based on artificial intelligence, and virtual digital humans. It specifically relates to posture recognition methods, action recognition methods, model training methods, devices, electronic devices, storage media, and computer program products. Background Art

[0002] Posture recognition is a key technology in the field of computer vision. Its core is to analyze human postures in images or videos through algorithms, and then recognize and understand human movements and behaviors. It can be widely used in motion capture, sports training, rehabilitation training, human-computer interaction, video surveillance, virtual reality and other scenarios. How to make the posture recognition results more reasonable and accurate is still a direction worth exploring. Summary of the invention

[0003] The present disclosure provides a posture recognition method, an action recognition method, a model training method, an apparatus, an electronic device, a storage medium, and a computer program product.

[0004] According to one aspect of the present disclosure, a gesture recognition method is provided, comprising: extracting features of key point position information of a target object in an image to be recognized to obtain key point features; updating gesture features using the key point features to obtain target features, wherein the gesture features include image features obtained by processing the image to be recognized; and performing gesture recognition on the target features to obtain a gesture recognition result of the target object.

[0005] According to another aspect of the present disclosure, a motion recognition method is provided, comprising: determining a posture recognition result of a target object of each of a plurality of images to be recognized, wherein the posture recognition result is obtained using the above-mentioned posture recognition method; and obtaining a motion recognition result of the target object based on the plurality of posture recognition results.

[0006] According to another aspect of the present disclosure, a training method for a deep learning model is provided, comprising: extracting features of key point position information of a sample target object in a sample image to obtain sample key point features; updating sample posture features using the sample key point features to obtain sample target features, wherein the sample posture features include sample image features obtained by processing the sample image; performing posture recognition on the sample target features to obtain a sample posture recognition result of the sample target object; and training a deep learning model using the sample posture recognition result and a label of the sample image, wherein the label of the sample image represents the true posture of the sample target object.

[0007] According to another aspect of the present disclosure, a gesture recognition device is provided, including: a first feature extraction module, used to extract features of key point position information of a target object in an image to be recognized, and obtain key point features; a first update module, used to update gesture features using key point features, and obtain target features, wherein the gesture features include image features obtained by processing the image to be recognized; and a first recognition module, used to perform gesture recognition on the target features, and obtain a gesture recognition result of the target object.

[0008] According to another aspect of the present disclosure, there is provided an action recognition device, comprising: a first determination module, used to determine a posture recognition result of a target object of each of a plurality of images to be recognized, wherein the posture recognition result is obtained using the above-mentioned posture recognition device; and a second determination module, used to obtain an action recognition result of the target object based on the plurality of posture recognition results.

[0009] According to another aspect of the present disclosure, a training device for a deep learning model is provided, comprising: a sample feature extraction module, used to perform feature extraction on key point position information of a sample target object in a sample image to obtain sample key point features; a sample update module, used to update sample posture features using the sample key point features to obtain sample target features, wherein the sample posture features include sample image features obtained by processing the sample image; a sample recognition module, used to perform posture recognition on the sample target features to obtain a sample posture recognition result of the sample target object; and a training module, used to train the deep learning model using the sample posture recognition result and the label of the sample image, wherein the label of the sample image represents the true posture of the sample target object.

[0010] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0011] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method as described above.

[0012] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method described above when executed by a processor.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0015] Figure 1 An exemplary system architecture to which the gesture recognition method and device according to an embodiment of the present disclosure can be applied is schematically shown;

[0016] Figure 2 A flowchart of a gesture recognition method according to an embodiment of the present disclosure is schematically shown;

[0017] Figure 3 The flowchart of the gesture recognition method according to the embodiment of the present disclosure is schematically shown;

[0018] Figure 4 A schematic diagram of a flow chart of a gesture recognition method according to another embodiment of the present disclosure is schematically shown;

[0019] Figure 5 The flowchart of the action recognition method according to the embodiment of the present disclosure is schematically shown;

[0020] Figure 6 A flowchart of a training method for a deep learning model according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 7 A block diagram of a gesture recognition device according to an embodiment of the present disclosure is schematically shown;

[0022] Figure 8 A block diagram of a motion recognition device according to an embodiment of the present disclosure is schematically shown;

[0023] Fig. 9 A block diagram schematically shows a training device for a deep learning model according to an embodiment of the present disclosure; and

[0024] Fig.10 A block diagram of an electronic device suitable for implementing a gesture recognition method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] Figure 1 An exemplary system architecture to which the gesture recognition method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.

[0027] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, an exemplary system architecture to which the gesture recognition method and apparatus can be applied may include a terminal device, but the terminal device may implement the gesture recognition method and apparatus provided by the embodiments of the present disclosure without interacting with a server.

[0028] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0029] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0030] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0031] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for the content browsed by the user using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device.

[0032] It should be noted that the gesture recognition method provided in the embodiment of the present disclosure can generally be executed by the terminal device 101, 102, or 103. Accordingly, the gesture recognition apparatus provided in the embodiment of the present disclosure can also be set in the terminal device 101, 102, or 103.

[0033] Alternatively, the gesture recognition method provided in the embodiment of the present disclosure may also be generally performed by the server 105. Accordingly, the gesture recognition device provided in the embodiment of the present disclosure may generally be arranged in the server 105. The gesture recognition method provided in the embodiment of the present disclosure may also be performed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the gesture recognition device provided in the embodiment of the present disclosure may also be arranged in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0034] For example, in a human-computer interaction scenario, the terminal devices 101, 102, and 103 can obtain an action video of a target object, and then send the obtained action video to the server 105, which deframes the action video, obtains an image to be identified, extracts features of key point position information of the target object in the image to be identified, obtains key point features, updates posture features using key point features, obtains target features, performs posture recognition on the target features, and determines the posture of the target object. Alternatively, a server or server cluster that can communicate with the terminal devices 101, 102, 103 and / or the server 105 recognizes the image to be identified and determines the posture of the target object.

[0035] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0036] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0037] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0038] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0039] Figure 2 The flowchart of the gesture recognition method according to the embodiment of the present disclosure is schematically shown.

[0040] like Figure 2 As shown, the method includes operations S210 to S230.

[0041] In operation S210, feature extraction is performed on key point position information of a target object in an image to be recognized to obtain key point features.

[0042] In operation S220, the posture feature is updated using the key point feature to obtain a target feature, wherein the posture feature includes an image feature obtained by processing the image to be recognized.

[0043] In operation S230, gesture recognition is performed on the target feature to obtain a gesture recognition result of the target object.

[0044] According to an embodiment of the present disclosure, an image to be identified may include a target object whose posture is to be identified. The target object may include a human body, an animal body, an object, and a mechanical body, etc. The image to be identified may be processed using the posture recognition method provided in the embodiment of the present disclosure to obtain a posture recognition result about the target object. Based on the posture recognition result of the target object, the computer can accurately understand the intention and action of the target object. Therefore, the posture recognition method provided in the embodiment of the present disclosure can be applied to scenes such as motion capture, sports training, rehabilitation training, human-computer interaction, video monitoring, virtual reality, etc.

[0045] According to an embodiment of the present disclosure, a key point may be a feature point that can represent a unique form or posture of a target object. The number of key points may include multiple key points, and the number of predetermined key points may be determined according to the type of the target object.

[0046] For example, if the target object is a human body, the key points may include joints of the head, neck, shoulders, elbows, wrists, hips, knees, ankles, etc. The posture of the human body can be determined based on multiple joints. The actions of the human body within a predetermined period of time, such as standing, walking, running, waving, etc., can be determined based on posture sequences at multiple times.

[0047] According to an embodiment of the present disclosure, the key point feature may be a feature vector of the key point three-dimensional coordinates extracted from the key point position information. However, it is not limited thereto. It may also be a feature vector related to the key point position, type, etc. extracted from the key point position information.

[0048] According to an embodiment of the present disclosure, the posture feature may be an image feature obtained by processing the image to be recognized, but is not limited thereto. The posture feature may also be a feature obtained by fusing the image feature and at least a portion of key point features.

[0049] According to the embodiments of the present disclosure, the key point features are used to update the posture features, which can be achieved by fusing the key point feature sequence of all the key points of the target object with the image features to update the posture features. However, it is not limited to this. The key point features of each key point of the target object can also be sequentially fused with the posture features in a predetermined order to update the posture features.

[0050] According to an embodiment of the present disclosure, the image features obtained by processing the image to be recognized can be processed by a visual pre-training model. The visual pre-training model may include CAE (Context Autoencoder), and CAE may be pre-trained by a self-supervised training method. The self-supervised training method may include: training using a large amount of unlabeled image data. CAE can extract rich image features from the image to be recognized. However, the present disclosure is not limited to this, and other image feature extraction networks can also be used, as long as rich image features can be extracted from the image to be recognized.

[0051] According to an embodiment of the present disclosure, a pre-trained deep learning model can be used to extract features from key point position information, such as a convolutional neural network (CNN). Through multi-layer convolution and pooling operations, CNN can automatically learn and extract key point features from key point position information. The present disclosure is not limited to this. A pre-trained large model can also be used to extract features from key point position information. It can also be used to extract features from key point position information through a multi-modal large model, which can process data in multiple modalities including images, videos, text, etc., and can perform cross-modal generation and interaction between these modalities.

[0052] According to an embodiment of the present disclosure, the key point position information may be three-dimensional coordinate information (X, Y, Z) in a three-dimensional pixel space, where X may represent the horizontal coordinate in the three-dimensional pixel space, Y may represent the vertical coordinate in the three-dimensional pixel space, and Z may represent the depth coordinate in the three-dimensional pixel space. However, this is not limited thereto. Any information that can represent three-dimensional coordinate information may be used.

[0053] According to an embodiment of the present disclosure, by extracting features of key point position information, local detail information of the target object is captured, and then the key point features are used to update image features containing global information, so that the target features contain both global contextual information and highlight key local details, thereby improving the robustness and accuracy of gesture recognition.

[0054] Figure 3 The figure schematically shows a flow chart of a gesture recognition method according to an embodiment of the present disclosure.

[0055] like Figure 3 As shown, the image to be identified 320 is sequentially input into the image feature extraction network M310 such as CAE and the feature mapping network M330 to obtain the posture feature 340. The posture feature 340 is input into the large model M350 to output a key point position information 360. The key point position information 360 is input into the key point feature mapping network M370 such as MLP to perform feature mapping, and the key point feature 380 can be obtained. The posture feature 340 is updated using the key point feature 380 to obtain the target feature for identifying the next key point position information, and the target feature is also the new posture feature, thus completing the first cycle. Then, according to the predetermined spatial association relationship, the key point features of the key point position information of the target object are sequentially used to update the posture feature until all the key point position information of the target object is output, and the posture recognition result of the target object is determined based on all the key point position information of the target object, that is, the posture recognition of the image to be identified is completed.

[0056] In conjunction with specific embodiments, Figure 2 The gesture recognition method shown is further explained.

[0057] According to the embodiments of the present disclosure, Figure 2 Before the operation S220 shown, the gesture recognition method may further include: processing the image to be recognized to obtain image features.

[0058] According to an embodiment of the present disclosure, processing the image to be identified to obtain image features may include: splitting the image to be identified to obtain a set of image blocks; and extracting features from the set of image blocks to obtain image features.

[0059] According to an embodiment of the present disclosure, by splitting an input image to be recognized into multiple smaller image blocks, and performing parallel feature extraction on the multiple smaller image blocks, the efficiency of extracting image features can be improved.

[0060] According to another embodiment of the present disclosure, processing the image to be identified to obtain image features may also include: extracting features from the image to be identified to obtain initial image features, and performing vector dimension conversion on the initial image features to obtain image features that match the key point feature dimensions.

[0061] For example, the image to be identified can be split first to obtain a sequence of image blocks. The image block sequence is input into the CAE model to extract k initial image features that correspond to the image block sequence one by one, and then the MLP (Multi-Layer Perceptron) visual feature mapping network is used to map the k initial image features extracted by CAE to the same dimension as the key point features to obtain image features. However, this is not limited to this. Linear functions can also be used to convert the vector dimension of the initial image features to obtain image features that match the dimension of the key point features.

[0062] According to the embodiments of the present disclosure, by performing vector dimension conversion on the initial image features, the dimensions of the image features and the key point features can be aligned. The aligned vectors have a uniform format and size, which helps to reduce update problems caused by dimensional mismatch, improves the success rate of updating posture features containing image features using key point features, and provides a basis for subsequent posture recognition results based on target features.

[0063] According to the embodiments of the present disclosure, Figure 2 The operation S220 shown, using the key point feature to update the posture feature to obtain the target feature, may include: concatenating the key point feature and the posture feature to obtain the target feature.

[0064] According to the embodiments of the present disclosure, the key point features and the image features are spliced, and the splicing method can be a simple connection, that is, combining two feature vectors in a certain order. However, it is not limited to this. It can also be a vector multiplication of the key point features and the image features to obtain the target features. As long as the key point features and the posture features can be integrated, it can be used.

[0065] According to the embodiments of the present disclosure, by splicing key point features with posture features, the target features can retain the global information of the original features while incorporating the local detail information of the key points, thereby enhancing the expressiveness of the features.

[0066] According to an optional embodiment of the present disclosure, splicing key point features and posture features to obtain target features may include: splicing image features of the image to be identified and the first key point features to obtain target features for identifying the next key point.

[0067] According to an embodiment of the present disclosure, the first key point feature of the first key point can be obtained by extracting features from the first key point position information of the first key point. The first key point position information is obtained by performing posture recognition on the image features of the image to be identified. The first key point can be determined from multiple key points of the target object based on a predetermined spatial association relationship. Taking the target object as a person as an example, the predetermined spatial association relationship may include the spatial association relationship between the joints of the limbs and body of the target object as key points. Based on the predetermined spatial association relationship, the elbow joint point of the right hand can be used as the first key point, and the shoulder joint point of the right hand can be used as the second key point, that is, the next key point. However, it is not limited to this, and the predetermined spatial association relationship and the first key point can be adjusted according to actual needs.

[0068] The first key point feature can be obtained by first inputting the image feature into the large model, outputting the first key point position information of the first key point, and then inputting the first key point position information into the key point feature mapping network to obtain the first key point feature.

[0069] According to an embodiment of the present disclosure, the large model may include InternLM-XComposer2, but is not limited thereto, as long as it is a model that can utilize target features to obtain key point position information.

[0070] According to an embodiment of the present disclosure, the image feature and the first key point feature are concatenated to obtain the target feature for identifying the position information of the next key point.

[0071] According to an embodiment of the present disclosure, since there is a correlation between the key points of the target object, by splicing the first key point feature with the image feature, it is possible to easily identify the key point position information of the next key point associated with the first key point feature, thereby improving the accuracy of subsequent key point recognition.

[0072] According to an embodiment of the present disclosure, extracting features from key point position information of a target object in an image to be identified to obtain key point features may include: extracting features from key point position information of a next key point to obtain key point features of the next key point.

[0073] According to an embodiment of the present disclosure, feature extraction is performed on the key point position information of the next key point, which can be achieved by inputting the target feature obtained based on the current key point feature into the large model, outputting the next key point position information, and then obtaining the key point feature of the next key point through the key point mapping network.

[0074] For example, k n-dimensional vectors (image features) and the first key point feature can be concatenated to obtain k+1 n-dimensional vectors, and the k+1 n-dimensional vectors are used as target features for identifying the next key point. The k+1 n-dimensional vectors are input into the large model to output the second key point position information, and the second key point position information is input into the key point feature mapping network to obtain the second key point feature.

[0075] By looping the above operations, the third, fourth, and i-th key point features can be obtained in turn.

[0076] According to an embodiment of the present disclosure, by performing feature extraction on the next key point position information, the next key point feature is obtained, thereby providing a basis for subsequent feature splicing and posture recognition.

[0077] According to an embodiment of the present disclosure, the key point feature may include a current key point feature. The posture feature may include an image feature and a key point feature subsequence.

[0078] The key point features and the posture features are concatenated to obtain the target features, which may include: concatenating the image features, the key point feature subsequence and the current key point features to obtain the target features for identifying the next key point.

[0079] According to an embodiment of the present disclosure, the target object may include multiple key points with a predetermined spatial association relationship, and the key point feature subsequence represents a feature set of some key points in the multiple key points. The next key point is determined using the current key point corresponding to the current key point feature and the predetermined spatial association relationship.

[0080] According to an embodiment of the present disclosure, the predetermined spatial association relationship may be a relative position relationship between key points in space. When the target object is a fixed structure, the spatial association relationship is usually known, that is, the predetermined spatial association relationship. By combining the current key point and the predetermined spatial association relationship, the next key point can be determined from multiple key points of the target object.

[0081] For example, the various skeletal joints of the human body, such as the head, neck, shoulders, elbows, wrists, hips, knees, and ankles, have a fixed connection relationship. When the current key point is the neck joint, the next key point is the shoulder joint.

[0082] According to an embodiment of the present disclosure, a key point feature subsequence may be an i-key point feature sequence consisting of the first key point feature, the second key point feature, and the i-th key point feature obtained in sequence. A posture feature may be a (k+i) vector, and a (k+i+1) vector may be obtained by concatenating the current i+1-th key point feature with the (k+i)-th vector, and a target feature for identifying the i+2-th key point may be obtained.

[0083] According to the embodiments of the present disclosure, by continuously updating the posture features, that is, adding new key point features to the key point feature subsequence, and using the current key point features and the predetermined spatial association relationship to determine the next key point, it is possible to realize the recognition of continuous key points of the target object, maintain the accuracy of the key point position, and thereby improve the effectiveness and accuracy of extracting posture information.

[0084] According to the embodiments of the present disclosure, Figure 2 The operation S230 shown is to perform gesture recognition on the target feature to obtain a gesture recognition result of the target object, including performing gesture recognition on the target feature to obtain key point position information of the next key point of the target object. Based on the key point position information of each of the multiple key points, the gesture recognition result of the target object is obtained.

[0085] According to an embodiment of the present disclosure, after obtaining the key point position information of each key point, the key point position information can be mapped to the posture model of the target object to obtain the overall skeleton of the target object. The overall posture of the target object can be inferred through the relative position, distance and angle relationship between multiple key points.

[0086] The gesture recognition result can also be compared and classified with the predetermined gesture category of the target object to determine the gesture category to which the target object belongs. The gesture recognition result can be output in the form of text labels, image annotations, or three-dimensional models according to the needs and preferences of the user.

[0087] According to the embodiments of the present disclosure, the overall skeleton of the target object can be composed through the key point position information of each of the multiple key points, providing an accurate description of the posture of the target object, making the posture estimation result more accurate.

[0088] According to the embodiments of the present disclosure, the above operations can be repeatedly performed in a loop: feature extraction is performed on the key point position information of the current key point of the target object in the image to be identified to obtain the current key point feature. The current key point feature is used to update the posture feature using the image feature and the key point feature subsequence as the target feature to obtain the target feature for identifying the next key point. The target feature is input into the large model to obtain the key point position information of the next key point. In order to determine the posture recognition result of the target object based on the key point position information of all the key points of the target object.

[0089] According to the embodiments of the present disclosure, the posture recognition method trained by the above-mentioned training can be adapted to the general large model, so that the posture recognition method provided by the embodiments of the present disclosure can be integrated with the existing technology while utilizing the spatial position correlation relationship between multiple key points of the target object for posture recognition, thereby improving the scope of application and universality.

[0090] However, it is not limited to this. Image features can also be used to obtain key point position information of all key points of the target object. Feature extraction is performed on the key point position information of all key points to obtain a key point feature sequence of all key points of the target object. The key point feature sequence of all key points is used as a key point feature and is concatenated with the posture feature to obtain the target feature. This avoids the operation of generating key point features in a loop. For a detailed explanation, please refer to Figure 4 .

[0091] Figure 4 The flowchart of a gesture recognition method according to another embodiment of the present disclosure is schematically shown.

[0092] like Figure 4 As shown, the image to be identified 410 can be input into the image feature extraction network M420 to extract image features and obtain posture features 430. The posture features 430 are input into the large model M440 to output all the key point position information of the target object. All the key point position information is mapped through the key point feature mapping network M460 to obtain a key point feature sequence 450. The posture features 430 and the key point sequence features 450 are spliced ​​to obtain the target features 470. The target features 470 are input into the large model M440 to obtain all the key point position information 490 of the target object.

[0093] In conjunction with specific embodiments, Figure 2 The illustrated embodiment is further explained.

[0094] According to the embodiments of the present disclosure, Figure 2 The operation S210 shown, extracting features from key point position information of the target object in the image to be recognized to obtain key point features, may include: extracting features from key point position information of each of a plurality of key points of the target object to obtain a key point feature sequence.

[0095] According to an embodiment of the present disclosure, the image features of the image to be identified can be extracted through an image feature extraction network. After the image features are input into the large model, the key point position information of all key points of the target object is output at one time, and the key point position information is mapped into a key point feature sequence through a key point mapping network. The present disclosure is not limited to this, and all key point features can also be extracted at one time through other deep learning models.

[0096] According to the embodiments of the present disclosure, feature extraction is performed on the key point position information of each of the multiple key points. In the process of feature extraction, not only the features of the key point position information of each of the multiple key points can be extracted, but also the features of the relative distance, angle, direction, etc. between the multiple key points can be extracted.

[0097] According to the embodiments of the present disclosure, by extracting the position information of each of a plurality of key points of a target object at one time and generating a key point feature sequence, a rapid analysis of the posture of the target object can be achieved.

[0098] According to an embodiment of the present disclosure, splicing key point features and posture features to obtain target features may include: splicing a key point feature sequence and an image feature to obtain the target feature. The key point feature sequence represents a feature set of multiple key points of a target object with a predetermined spatial association relationship.

[0099] According to an embodiment of the present disclosure, the key point feature sequence is spliced ​​with the image feature, and the splicing method can be a simple connection, that is, combining the two feature vectors in a certain order. It can also be a more complex fusion method, for example, fusing the two feature vectors by weighted summation, attention mechanism, etc.

[0100] According to the embodiments of the present disclosure, by splicing the key point sequence features and image features of the target object, the spliced ​​target features include all global features and local detail features of the key points, thereby improving the accuracy of gesture recognition.

[0101] According to the embodiments of the present disclosure, Figure 2 The operation S230 shown, performing gesture recognition on the target feature to obtain the gesture recognition result of the target object, includes: performing gesture recognition on the target feature to obtain key point position information of each of the multiple key points of the target object. Based on the key point position information of each of the multiple key points, the gesture recognition result of the target object is obtained.

[0102] According to the embodiments of the present disclosure, the target features obtained by splicing the key point feature sequence and the image features can be input into the large model, and the key point position information of each of the multiple key points can be output.

[0103] According to an embodiment of the present disclosure, after obtaining the key point position information of each key point, the key point position information can be mapped to the posture model of the target object to obtain the overall skeleton of the target object, and the overall posture of the target object can be inferred through the relative position, distance and angle relationship between the key points.

[0104] The gesture recognition result can also be compared and classified with the predetermined gesture category of the target object to determine the gesture category to which the target object belongs. The gesture recognition result can be output in the form of text labels, image annotations, or three-dimensional models according to the needs and preferences of the user.

[0105] According to the embodiments of the present disclosure, the overall skeleton or shape model of the target object is constructed through the key point position information of multiple key points, which can capture the subtle changes of these key points, thereby comprehensively and accurately representing the posture of the target object while improving the efficiency of posture recognition.

[0106] According to an optional embodiment of the present disclosure, a posture recognition result of the target object is obtained based on the key point position information of each of the multiple key points, including: when it is determined that the number of key points with known position information meets a predetermined number threshold, a posture recognition result of the target object is obtained based on the key point position information of each of the multiple key points.

[0107] According to an embodiment of the present disclosure, when it is determined that the number of key points of known position information does not meet a predetermined number threshold, the key point features and posture features of the current key point may be concatenated to obtain a target feature for identifying the next key point.

[0108] According to the embodiments of the present disclosure, the predetermined number threshold can be determined according to the specific task requirements. Only when the number of key points reaches the predetermined number threshold, the key point position information required for gesture recognition is determined to be complete, so that gesture recognition can be performed. Using the predetermined number threshold, it can be preliminarily determined whether the known key point position information used for gesture recognition results is accurate. However, it is not limited to this. It can also be used as a stop condition for the number of cycles. When it is determined that the number of key points with known position information meets the predetermined number threshold, the operation of determining the target feature is stopped, and the key point position information of each of the multiple key points is directly used to obtain the gesture recognition result of the target object.

[0109] According to the embodiments of the present disclosure, by setting a predetermined number threshold, the integrity and accuracy of key point detection during gesture recognition can be ensured, which helps to improve the accuracy of gesture recognition and avoid erroneous recognition caused by missing key points or misdetection.

[0110] According to another optional embodiment of the present disclosure, obtaining a gesture recognition result of the target object based on the position information of each of the multiple key points may also include: when it is determined that the spatial distance between adjacent key points satisfies a predetermined spatial distance threshold, obtaining a gesture recognition result of the target object based on the key point position information of each of the multiple key points.

[0111] According to an embodiment of the present disclosure, the predetermined spatial distance threshold may be determined based on statistical data, or may be determined based on empirical values ​​or model predictions. The spatial distance between adjacent key points may be characterized by Euclidean distance or Manhattan distance, for example, by using the key point features of each of two adjacent key points to perform a similarity comparison to obtain the spatial distance. However, this is not limited to this. It may also be characterized by position values. For example, the spatial distance between two adjacent key points may be determined using the key point position information of each of two adjacent key points.

[0112] By comparing the spatial distance between adjacent key points with a predetermined spatial distance threshold, it is possible to determine whether the spatial distance between adjacent key points is reasonable. Only when the spatial distance between adjacent key points is within the threshold range, it can be determined that the spatial relationship between adjacent key points is reasonable, and gesture recognition can be performed.

[0113] According to an embodiment of the present disclosure, when the spatial distance between adjacent key points is greater than a predetermined spatial distance threshold, the operation may be performed again until key point data satisfying the predetermined spatial distance threshold is obtained.

[0114] According to an embodiment of the present disclosure, by setting a predetermined spatial distance threshold, some abnormal spatial distances caused by key point detection errors or noise interference can be excluded, thereby improving the accuracy of gesture recognition.

[0115] According to the embodiments of the present disclosure, Figure 2 The operation S220 shown, using key point features to update posture features to obtain target features, may include: when it is determined that the total data volume of key point features and posture features is greater than a predetermined data volume threshold, using key point features to update lightweight posture features to obtain target features.

[0116] According to an embodiment of the present disclosure, the predetermined data volume threshold may be set according to model resources, model processing capabilities, or actual needs.

[0117] According to the embodiments of the present disclosure, lightweight posture features are obtained by lightweight processing the posture features. The lightweight processing of the posture features may be to lightweight process only the image features in the posture features, to lightweight process only the key point features in the posture features, or to lightweight process both the image features and the key point features in the posture features.

[0118] According to the embodiments of the present disclosure, due to the large amount of overall data, the overall processing efficiency may be low. The posture features after lightweight processing can retain key information while reducing the amount of data. Therefore, the processing efficiency is improved while ensuring the recognition accuracy.

[0119] According to an embodiment of the present disclosure, the key point feature is used to update the posture feature to obtain the target feature, and the method further includes: performing lightweight processing on the posture feature to obtain the lightweight posture feature. The lightweight processing includes at least one of the following: downsampling and cropping.

[0120] According to an embodiment of the present disclosure, downsampling is a method of reducing the sampling rate or resolution of data by reducing the number of data points. A uniform sampling method can be used to select data points at fixed intervals in the original data. A non-uniform sampling method can also be used to dynamically adjust the sampling interval according to the local characteristics of the data.

[0121] According to an embodiment of the present disclosure, cropping is a method of reducing the size of data by removing unimportant parts of the data. Unimportant or redundant data in image features or key point features can be first determined, and then the data in that part is deleted, leaving only key or useful features.

[0122] You can choose a suitable method from downsampling and cropping or use a combination of the two methods according to specific needs and data characteristics.

[0123] According to the embodiments of the present disclosure, by lightweight processing of posture features, it is possible to improve computing efficiency, reduce resource consumption, and at the same time, maintain or improve the performance of the model as much as possible.

[0124] In relevant examples, a deep learning model can be used to directly learn the 3D posture knowledge of the human body in the image. For example, the image to be recognized is input into the deep learning model to obtain the 3D position information of the key points. This can be achieved through an end-to-end deep learning model.

[0125] In another related example, a deep learning model can also be used to learn the 2D posture knowledge of the human body in the image, and based on the 2D posture knowledge, the human body is reconstructed in 3D to obtain the 3D position information of the key points. For example, a 2D posture estimation model is first used to predict the 2D position information of the key points of the human body in the image to be identified, and then the 2D position information of the key points of the human body is used to estimate the 3D posture to obtain the 3D position information of the key points. The 2D posture estimation model can include convolutional neural networks (CNNs) and specific architectures, such as Hourglass Network, Mask R-CNN, etc.

[0126] Compared with the above-mentioned related examples, the posture recognition method provided by the embodiment of the present disclosure can effectively utilize the structural characteristics of the human body itself. For example, the human body is a skeleton formed by a series of joints, and has sequence features from the root node to the leaf node. Therefore, the target feature is obtained by using image features and a key point feature sequence constructed based on the key point features. The target feature can contain the global information of the image while containing the sequence correlation features of multiple human joints, thereby making the posture recognition result obtained based on the target feature accurate and effective.

[0127] Figure 5 The flowchart of the action recognition method according to the embodiment of the present disclosure is schematically shown.

[0128] The method includes: operations S510~S520.

[0129] In operation S510 , a gesture recognition result of a target object of each of a plurality of images to be recognized is determined.

[0130] In operation S520, based on the plurality of posture recognition results, an action recognition result of the target object is obtained.

[0131] According to an embodiment of the present disclosure, the gesture recognition result of the target object is obtained by the gesture recognition method provided by the embodiment of the present disclosure.

[0132] The multiple images to be identified can be obtained by de-framing the target video or by taking multiple pictures continuously. The action of the target object can be inferred by the posture changes of the target object at different time points. The action category performed by the target object can be determined based on the motion trajectory, motion speed, and bone angle of the key points of the target object.

[0133] According to the embodiments of the present disclosure, multiple posture recognition results are applied to action recognition. By capturing the continuous changes in the posture of the target object, the complete action sequence of the target object is identified, which can be widely used in scenarios such as motion capture, sports training, rehabilitation training, human-computer interaction, video surveillance, virtual reality, etc.

[0134] Figure 6 A flowchart of a deep learning model training method according to an embodiment of the present disclosure is schematically shown.

[0135] The method includes: operations S610 to S640.

[0136] In operation S610, feature extraction is performed on key point position information of a sample target object in a sample image to obtain sample key point features.

[0137] In operation S620, the sample pose feature is updated using the sample key point feature to obtain the sample target feature. The sample pose feature includes the sample image feature obtained by processing the sample image.

[0138] In operation S630, gesture recognition is performed on the sample target feature to obtain a sample gesture recognition result of the sample target object.

[0139] In operation S640, a deep learning model is trained using the sample pose recognition results and the labels of the sample images. The labels of the sample images represent the true poses of the sample target objects.

[0140] According to the embodiments of the present disclosure, the sample key point features and the sample image features can be spliced ​​to obtain the sample target features, and the sample target features are used as the input data of the deep learning model, and the sample key point position information is used as the output data. The output sample key point position information is mapped to the sample key point features, and the sample posture features are continuously updated and the sample target features are subjected to posture recognition through the key point features until all the sample key point position information of the sample target object is output.

[0141] According to the embodiments of the present disclosure, by continuously splicing sample key point features with sample posture features, a sample key point feature sequence can be formed, so that the deep learning model can learn the sample key point context knowledge from the sample key point feature sequence, thereby enabling the model to perform posture recognition more accurately.

[0142] During the training process, a conventional loss function may be used, and the present disclosure does not impose any restrictions on this. The network parameters of the model are continuously adjusted to minimize the difference between the predicted sample posture recognition results and the true posture labels. Through a large number of iterative training, the model gradually learns to accurately recognize the posture of the target object from the image.

[0143] According to the embodiments of the present disclosure, by fusing sample key point features into sample image features, the deep learning model can fully learn the key point sequence knowledge, thereby improving the richness and diversity of the knowledge learned by the deep learning model, and further enabling the trained deep learning model to accurately identify the posture of the target object in the image to be identified.

[0144] According to the embodiments of the present disclosure, Figure 6 The operation S610 shown is to extract features from key point position information of a sample target object in a sample image to obtain sample key point features, including: extracting features from labels of the sample image to obtain a sample key point feature sequence, and obtaining sample key point features using a mask matrix.

[0145] According to an embodiment of the present disclosure, the mask matrix may be a matrix of the same dimension as the key point feature sequence, which may be used to indicate which key points are visible. For invisible key points, the corresponding elements in the mask matrix may be set to 0 or a smaller value. For visible key points, they may be set to 1 or a larger value.

[0146] According to an embodiment of the present disclosure, in order to simulate causal relationships in natural language processing, the mask matrix can limit the model's access to future information, thereby forcing the model to pay more attention to current and past contextual information, thereby improving the model's performance on generation tasks.

[0147] The mask matrix can also be used to select specific key point features from the key point feature sequence for subsequent processing. For example, if you are only interested in certain parts of the target object (such as hands or faces), you can select the key point features of these parts by setting the corresponding mask matrix.

[0148] According to the embodiments of the present disclosure, by applying feature extraction and mask matrix, the label information of sample images can be more effectively used to train the deep learning model, thereby improving the posture recognition accuracy, robustness and generalization ability of the model.

[0149] According to an embodiment of the present disclosure, the sample image features obtained by processing the sample image further include: extracting features from the sample image to obtain sample initial image features. Performing vector dimension conversion on the sample initial image features to obtain sample image features that match the sample key point feature dimensions.

[0150] According to an embodiment of the present disclosure, sample key point features are used to update sample posture features to obtain sample target features, including: concatenating sample key point features and sample posture features to obtain sample target features.

[0151] According to an embodiment of the present disclosure, sample key point features and sample posture features are spliced ​​to obtain sample target features, including: splicing sample image features and first sample key point features of a sample image to obtain sample target features for identifying the next sample key point.

[0152] According to an embodiment of the present disclosure, feature extraction is performed on sample key point position information of a sample target object in a sample image to obtain sample key point features, including: feature extraction is performed on sample key point position information of a next sample key point to obtain sample key point features of the next sample key point.

[0153] According to an embodiment of the present disclosure, the sample posture feature includes a sample image feature and a sample key point feature subsequence. The sample key point feature includes a current sample key point feature. The sample key point feature and the sample posture feature are spliced ​​to obtain a sample target feature, including splicing the sample image feature, the sample key point feature subsequence and the current sample key point feature to obtain a sample target feature for identifying the next sample key point. The sample target object includes a plurality of sample key points having a predetermined spatial association relationship, and the sample key point feature subsequence represents a feature set of some sample key points among the plurality of sample key points. The next sample key point is determined using the current sample key point corresponding to the current sample key point feature and a predetermined spatial association relationship.

[0154] According to an embodiment of the present disclosure, performing gesture recognition on sample target features to obtain a sample gesture recognition result of the sample target object includes performing gesture recognition on the sample target features to obtain sample key point position information of the next sample key point of the sample target object. Based on the sample key point position information of each of the multiple sample key points, the sample gesture recognition result of the sample target object is obtained.

[0155] According to an embodiment of the present disclosure, feature extraction is performed on sample key point position information of a sample target object in a sample image to obtain sample key point features, including: feature extraction is performed on sample key point position information of each of multiple sample key points of the sample target object to obtain a sample key point feature sequence.

[0156] According to an embodiment of the present disclosure, sample key point features and sample posture features are spliced ​​to obtain sample target features, including: splicing a sample key point feature sequence and a sample image feature to obtain a sample target feature, wherein the sample key point feature sequence represents a feature set of multiple sample key points of a sample target object having a predetermined spatial association relationship.

[0157] According to an embodiment of the present disclosure, posture recognition is performed on sample target features to obtain a sample posture recognition result of the sample target object, including: performing posture recognition on the sample target features to obtain sample key point position information of each of multiple sample key points of the sample target object; and obtaining a sample posture recognition result of the sample target object based on the sample key point position information of each of the multiple sample key points.

[0158] According to an embodiment of the present disclosure, a sample gesture recognition result of a sample target object is obtained based on the sample key point position information of each of a plurality of sample key points, including: when it is determined that the number of sample key points with known position information meets a predetermined number threshold, a sample gesture recognition result of the sample target object is obtained based on the sample key point position information of each of the plurality of sample key points.

[0159] According to an embodiment of the present disclosure, a sample gesture recognition result of a sample target object is obtained based on the position information of each of a plurality of sample key points, including: when it is determined that the spatial distance between adjacent sample key points satisfies a predetermined spatial distance threshold, a sample gesture recognition result of the sample target object is obtained based on the sample key point position information of each of the plurality of sample key points.

[0160] According to an embodiment of the present disclosure, sample key point features are used to update sample posture features to obtain sample target features, including: when it is determined that the total data volume of sample key point features and sample posture features is greater than a predetermined data volume threshold, the sample key point features are used to update lightweight sample posture features to obtain sample target features.

[0161] According to an embodiment of the present disclosure, the lightweight sample posture feature is obtained by performing at least one of the following lightweight processing on the sample posture feature: downsampling, cropping.

[0162] Figure 7 The block diagram of a gesture recognition device according to an embodiment of the present disclosure is schematically shown.

[0163] like Figure 7 As shown, the gesture recognition device includes a first feature extraction module 710 , a first updating module 720 and a first recognition module 730 .

[0164] The first feature extraction module 710 is used to extract features from key point position information of the target object in the image to be identified to obtain key point features.

[0165] The first updating module 720 is used to update the posture feature by using the key point feature to obtain the target feature, wherein the posture feature includes the image feature obtained by processing the image to be recognized.

[0166] The first recognition module 730 is used to perform posture recognition on the target feature to obtain a posture recognition result of the target object.

[0167] According to an embodiment of the present disclosure, the first update module includes a splicing submodule.

[0168] The splicing submodule is used to splice the key point features and posture features to obtain the target features.

[0169] According to an embodiment of the present disclosure, the posture feature further includes a key point feature subsequence. The key point feature includes a current key point feature.

[0170] The splicing submodule includes a first splicing unit.

[0171] The first splicing unit is used to splice the image features, the key point feature subsequence and the current key point feature to obtain the target feature for identifying the next key point, wherein the target object includes multiple key points with a predetermined spatial association relationship, the key point feature subsequence represents a feature set of some key points among the multiple key points, and the next key point is determined by using the current key point corresponding to the current key point feature and the predetermined spatial association relationship.

[0172] According to an embodiment of the present disclosure, the key point feature includes a first key point feature of a first key point. The splicing submodule also includes a second splicing unit.

[0173] The second splicing unit is used to splice the image feature of the image to be identified and the first key point feature to obtain a target feature for identifying the next key point. The first key point is determined from multiple key points based on a predetermined spatial association relationship.

[0174] According to an embodiment of the present disclosure, the first recognition module includes a key point recognition submodule and a posture determination submodule.

[0175] The key point recognition submodule is used to perform posture recognition on the target feature and obtain the key point position information of the next key point of the target object.

[0176] The posture determination submodule is used to obtain the posture recognition result of the target object based on the key point position information of each of the multiple key points.

[0177] According to an embodiment of the present disclosure, the first feature extraction module includes a first extraction submodule.

[0178] The first extraction submodule is used to extract features from the key point position information of the next key point to obtain the key point features of the next key point.

[0179] According to an embodiment of the present disclosure, the splicing submodule includes a third splicing unit.

[0180] The third splicing unit is used to splice the key point feature sequence and the image feature to obtain the target feature, wherein the key point feature sequence represents a feature set of multiple key points with a predetermined spatial association relationship of the target object.

[0181] According to an embodiment of the present disclosure, the first recognition module further includes a key point recognition submodule and a posture determination submodule.

[0182] The key point recognition submodule is used to perform posture recognition on the target features and obtain the key point position information of each of the multiple key points of the target object.

[0183] The posture determination submodule is used to obtain the posture recognition result of the target object based on the key point position information of each of the multiple key points.

[0184] According to an embodiment of the present disclosure, the posture determination submodule includes a first posture determination unit.

[0185] The first posture determination unit is used to obtain a posture recognition result of the target object based on the key point position information of each of the multiple key points when it is determined that the number of key points with known position information meets a predetermined number threshold.

[0186] According to an embodiment of the present disclosure, the posture determination submodule includes a second posture determination unit.

[0187] The second posture determination unit is used to obtain a posture recognition result of the target object based on the key point position information of each of the multiple key points when it is determined that the spatial distance between adjacent key points meets a predetermined spatial distance threshold.

[0188] According to an embodiment of the present disclosure, the first feature extraction module also includes a second extraction submodule.

[0189] The second extraction submodule is used to extract features from key point position information of each of the multiple key points of the target object to obtain a key point feature sequence.

[0190] According to an embodiment of the present disclosure, a second feature extraction module and a vector conversion module are also included.

[0191] The second feature extraction module is used to extract features from the image to be identified to obtain initial image features.

[0192] The vector conversion module is used to convert the vector dimension of the initial image features to obtain image features that match the key point feature dimensions.

[0193] According to an embodiment of the present disclosure, the first update module includes a first update submodule.

[0194] The first updating submodule is used to update the lightweight posture feature with the key point feature to obtain the target feature when it is determined that the total data amount of the key point feature and the posture feature is greater than a predetermined data amount threshold.

[0195] According to an embodiment of the present disclosure, the lightweight posture feature is obtained by performing at least one of the following lightweight processing on the posture feature: downsampling, cropping.

[0196] Figure 8 The block diagram of the motion recognition device according to the embodiment of the present disclosure is schematically shown.

[0197] like Figure 8 As shown, the motion recognition device 800 includes a first determination module 810 and a second determination module 820 .

[0198] The first determination module 810 is used to determine the gesture recognition results of the target objects of the multiple images to be recognized, wherein the gesture recognition results are obtained by using the above-mentioned gesture recognition device.

[0199] The second determination module 820 is used to obtain a motion recognition result of the target object based on the multiple posture recognition results.

[0200] Fig. 9 A block diagram of a deep learning model training device according to an embodiment of the present disclosure is schematically shown.

[0201] like Fig. 9 As shown, the training device 900 of the deep learning model includes a sample feature extraction module 910, a sample update module 920, a sample identification module 930 and a training module 940.

[0202] The sample feature extraction module 910 is used to extract features from key point position information of the sample target object in the sample image to obtain sample key point features.

[0203] The sample updating module 920 is used to update the sample posture feature by using the sample key point feature to obtain the sample target feature, wherein the sample posture feature includes the sample image feature obtained by processing the sample image.

[0204] The sample recognition module 930 is used to perform posture recognition on the sample target features to obtain a sample posture recognition result of the sample target object.

[0205] The training module 940 is used to train a deep learning model using the sample posture recognition results and the labels of the sample images, wherein the labels of the sample images represent the real posture of the sample target object.

[0206] According to an embodiment of the present disclosure, the sample feature extraction module includes a sample extraction submodule and a sample key point determination submodule.

[0207] The sample extraction submodule is used to extract features from the labels of the sample images to obtain a sample key point feature sequence.

[0208] The sample key point determination submodule is used to obtain the sample key point features using the mask matrix.

[0209] Fig.10A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0210] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0211] A number of components in the device 1000 are connected to an input / output (I / O) interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0212] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as gesture recognition methods. For example, in some embodiments, the gesture recognition method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the gesture recognition method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the gesture recognition method in any other appropriate manner (e.g., by means of firmware).

[0213] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0214] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0215] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0216] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0217] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0218] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0219] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0220] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A gesture recognition method, comprising: Feature extraction is performed on the key point position information of the target object in the image to be identified to obtain key point features; Using the key point features to update the posture features to obtain target features, wherein the posture features include image features obtained by processing the image to be identified; and Performing posture recognition on the target feature to obtain a posture recognition result of the target object.

2. The method according to claim 1, wherein: The updating of posture features by using the key point features, Get the target features, including: The key point features and the posture features are concatenated to obtain the target features.

3. The method according to claim 2, wherein: The posture feature also includes a key point feature subsequence, and the key point feature includes a current key point feature. The step of combining the key point feature and the posture feature to obtain the target feature includes: The image features, the key point feature subsequence and the current key point feature are concatenated to obtain a target feature for identifying a next key point, wherein the target object includes a plurality of key points having a predetermined spatial association relationship, the key point feature subsequence represents a feature set of some key points among the plurality of key points, and the next key point is determined using the current key point corresponding to the current key point feature and the predetermined spatial association relationship.

4. The method according to claim 3, wherein: The key point feature comprises a first key point feature of a first key point, The step of combining the key point feature and the posture feature to obtain the target feature includes: The image features of the image to be identified and the first key point features are concatenated to obtain target features for identifying a next key point, wherein the first key point is determined from the multiple key points based on the predetermined spatial association relationship.

5. The method according to claim 3 or 4, wherein: The performing posture recognition on the target feature to obtain a posture recognition result of the target object includes: Performing posture recognition on the target feature to obtain key point position information of the next key point of the target object; and Based on the key point position information of each of the plurality of key points, a gesture recognition result of the target object is obtained.

6. The method according to any one of claims 3 to 5, wherein: The feature extraction of key point position information of the target object in the image to be identified to obtain key point features includes: Feature extraction is performed on the key point position information of the next key point to obtain the key point feature of the next key point.

7. The method according to claim 2, wherein: The step of combining the key point feature and the posture feature to obtain the target feature includes: The key point feature sequence and the image feature are concatenated to obtain the target feature, wherein the key point feature sequence represents a feature set of multiple key points of the target object having a predetermined spatial association relationship.

8. The method according to claim 7, wherein: The performing posture recognition on the target feature to obtain a posture recognition result of the target object includes: Performing posture recognition on the target feature to obtain key point position information of each of the multiple key points of the target object; and Based on the key point position information of each of the plurality of key points, a gesture recognition result of the target object is obtained.

9. The method according to claim 5 or 8, wherein: The step of obtaining the gesture recognition result of the target object based on the key point position information of each of the plurality of key points includes: When it is determined that the number of key points with known position information meets a predetermined number threshold, a gesture recognition result of the target object is obtained based on the key point position information of each of the plurality of key points.

10. The method according to claim 5 or 8, wherein: The step of obtaining the gesture recognition result of the target object based on the position information of each of the plurality of key points includes: When it is determined that the spatial distance between adjacent key points meets a predetermined spatial distance threshold, a gesture recognition result of the target object is obtained based on key point position information of each of the plurality of key points.

11. The method according to any one of claims 7 to 9, wherein: The feature extraction of key point position information of the target object in the image to be identified to obtain key point features includes: Feature extraction is performed on key point position information of each of the multiple key points of the target object to obtain the key point feature sequence.

12. The method according to any one of claims 1 to 11, further comprising: Extracting features of the image to be identified to obtain initial image features; as well as The initial image features are vector-dimensionally transformed to obtain the image features that match the key point feature dimensions.

13. The method according to any one of claims 1 to 12, wherein: The step of updating the posture feature using the key point feature to obtain the target feature includes: When it is determined that the sum of the data amounts of the key point features and the posture features is greater than a predetermined data amount threshold, the key point features are used to update the lightweight posture features to obtain the target features.

14. The method according to claim 13, wherein: The lightweight posture feature is obtained by performing at least one of the following lightweight processing on the posture feature: Downsampling and cropping.

15. A method for motion recognition, comprising: Determine a gesture recognition result of a target object in each of a plurality of images to be recognized, wherein the gesture recognition result is obtained by using the method according to any one of claims 1 to 14; and Based on the multiple gesture recognition results, an action recognition result of the target object is obtained.

16. A method for training a deep learning model, comprising: Extracting features of key point position information of sample target objects in sample images to obtain sample key point features; Using the sample key point features to update sample posture features to obtain sample target features, wherein the sample posture features include the sample image features obtained by processing the sample image; Performing posture recognition on the sample target feature to obtain a sample posture recognition result of the sample target object; and The deep learning model is trained using the sample posture recognition results and the labels of the sample images, wherein the labels of the sample images represent the true posture of the sample target object.

17. The method according to claim 16, wherein: The step of extracting features from key point position information of the sample target object in the sample image to obtain sample key point features includes: Extracting features from the labels of the sample images to obtain a sample key point feature sequence; and The mask matrix is ​​used to obtain the key point features of the sample.

18. A gesture recognition device, comprising: The first feature extraction module is used to extract the key point position information of the target object in the image to be identified to obtain the key point features; A first updating module, configured to update a posture feature using the key point feature to obtain a target feature, wherein the posture feature includes an image feature obtained by processing the image to be identified; as well as The first recognition module is used to perform posture recognition on the target feature to obtain a posture recognition result of the target object.

19. The device according to claim 18, wherein: The first update module includes: The splicing submodule is used to splice the key point features and the posture features to obtain the target features.

20. The device according to claim 19, wherein The posture feature also includes a key point feature subsequence, and the key point feature includes a current key point feature. The splicing submodule includes: A first splicing unit is used to splice the image feature, the key point feature subsequence and the current key point feature to obtain a target feature for identifying a next key point, wherein the target object includes multiple key points with a predetermined spatial association relationship, the key point feature subsequence represents a feature set of some key points among the multiple key points, and the next key point is determined by using the current key point corresponding to the current key point feature and the predetermined spatial association relationship.

21. The device according to claim 20, wherein: The key point feature includes a first key point feature of a first key point, and the splicing submodule further includes: A second stitching unit is used to stitch the image features of the image to be identified and the first key point features to obtain a target feature for identifying a next key point, wherein the first key point is determined from the multiple key points based on the predetermined spatial association relationship.

22. The device according to claim 20 or 21, wherein: The first identification module comprises: A key point recognition submodule, used to perform posture recognition on the target feature to obtain key point position information of the next key point of the target object; and The posture determination submodule is used to obtain the posture recognition result of the target object based on the key point position information of each of the multiple key points.

23. The device according to any one of claims 20 to 22, wherein: The first feature extraction module comprises: The first extraction submodule is used to perform feature extraction on the key point position information of the next key point to obtain the key point feature of the next key point.

24. The device according to claim 19, wherein The splicing submodule includes: The third splicing unit is used to splice the key point feature sequence and the image feature to obtain the target feature, wherein the key point feature sequence represents a feature set of multiple key points of the target object with a predetermined spatial association relationship.

25. The device according to claim 24, wherein: The first identification module also includes: A key point recognition submodule, used to perform posture recognition on the target feature to obtain key point position information of each of the multiple key points of the target object; and The posture determination submodule is used to obtain the posture recognition result of the target object based on the key point position information of each of the multiple key points.

26. The device according to claim 22 or 25, wherein: The posture determination submodule comprises: The first posture determination unit is used to obtain the posture recognition result of the target object based on the key point position information of each of the key points when it is determined that the number of key points with known position information meets a predetermined number threshold.

27. The device according to claim 22 or 25, wherein: The posture determination submodule comprises: The second posture determination unit is used to obtain the posture recognition result of the target object based on the key point position information of each of the multiple key points when it is determined that the spatial distance between adjacent key points meets a predetermined spatial distance threshold.

28. The device according to any one of claims 24 to 26, wherein: The first feature extraction module also includes: The second extraction submodule is used to extract features from the key point position information of each of the multiple key points of the target object to obtain the key point feature sequence.

29. The apparatus according to any one of claims 18 to 28, further comprising: A second feature extraction module is used to extract features from the image to be identified to obtain initial image features; as well as The vector conversion module is used to perform vector dimension conversion on the initial image features to obtain the image features that match the key point feature dimensions.

30. The device according to any one of claims 18 to 29, wherein: The first update module includes: The first updating submodule is used to update the lightweight posture feature with the key point feature to obtain the target feature when it is determined that the total data amount of the key point feature and the posture feature is greater than a predetermined data amount threshold.

31. The device according to claim 30, wherein The lightweight posture feature is obtained by performing at least one of the following lightweight processing on the posture feature: Downsampling and cropping.

32. A motion recognition device, comprising: A first determination module, configured to determine a gesture recognition result of a target object of each of a plurality of images to be recognized, wherein the gesture recognition result is obtained by using the apparatus according to any one of claims 18 to 31; and The second determination module is used to obtain the action recognition result of the target object based on the multiple posture recognition results.

33. A training device for a deep learning model, comprising: A sample feature extraction module is used to extract the key point position information of the sample target object in the sample image to obtain the sample key point features; A sample updating module, used for updating a sample posture feature by using the sample key point feature to obtain a sample target feature, wherein the sample posture feature includes the sample image feature obtained by processing the sample image; A sample recognition module, used to perform posture recognition on the sample target feature to obtain a sample posture recognition result of the sample target object; and A training module is used to train the deep learning model using the sample posture recognition results and the labels of the sample images, wherein the labels of the sample images represent the true posture of the sample target object.

34. The device according to claim 33, wherein The sample feature extraction module comprises: A sample extraction submodule, used to extract features from the labels of the sample images to obtain a sample key point feature sequence; and The sample key point determination submodule is used to obtain the sample key point features using a mask matrix.

35. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 17.

36. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-17.

37. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 17.

Images