Posture recognition method and device, equipment and storage medium

Abstract

The invention relates to a posture recognition method, device and equipment and a storage medium, and the method comprises the steps: responding to a received to-be-recognized target video, extracting a target skeleton point of a person in each frame of image of the target video, the target skeleton point comprising a skeleton point of a preset human body part of the person, and obtaining a target skeleton point of the person in each frame of image of the target video; the preset human body part at least comprises a hand part; dividing the target skeleton points into a plurality of groups, wherein each group of skeleton points corresponds to the same preset human body part; respectively carrying out centralization processing on each group of skeleton points to obtain a plurality of groups of centralized skeleton points; and obtaining a target recognition result corresponding to the target video based on the centralized multiple groups of skeleton points and a pre-trained target posture recognition model. According to the invention, the robustness of gesture recognition under different conditions can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a gesture recognition method, device, equipment and storage medium. Background Art

[0002] Gesture recognition aims to convert the input video into a sequence of sign language words in a consistent order. The gesture recognition method in the related art includes extracting spatiotemporal information from the input video and converting it into discriminative features, and then establishing the correspondence between the features and the sign language word labels.

[0003] However, the recognition methods in related technologies have problems such as large model calculation amount and poor robustness in complex scenarios, which greatly limits the practical application of the recognition methods. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the embodiments of the present disclosure provide a gesture recognition method, device, equipment and storage medium to solve the defects in the related art.

[0005] According to a first aspect of an embodiment of the present disclosure, a gesture recognition method is provided, the method comprising:

[0006] In response to receiving a target video to be identified, extracting target skeleton points of a person in each frame of the target video, wherein the target skeleton points include skeleton points of preset human body parts of the person, and the preset human body parts include at least a hand;

[0007] Dividing the target skeleton points into multiple groups, each group of skeleton points corresponds to the same preset human body part;

[0008] Performing centralization processing on each group of skeleton points respectively to obtain multiple groups of centralized skeleton points;

[0009] Based on the centralized multiple groups of skeleton points and the pre-trained target posture recognition model, a target recognition result corresponding to the target video is obtained.

[0010] In some embodiments, the target gesture recognition model includes a first single-stream network with skeleton points as input, the first single-stream network including a first set of specific graph convolutional networks, a first context module, and a first complementary regularization module;

[0011] The obtaining of the target recognition result corresponding to the target video based on the centralized multiple groups of skeleton points and the pre-trained target posture recognition model includes:

[0012] Determining a first skeleton point feature based on the first specific graph convolutional network set and the centralized multiple groups of skeleton points;

[0013] Determine two second skeleton point features at different viewing angles based on the two currently generated complementary first masks and the first skeleton point features;

[0014] Determine a first target feature based on the first context module and the second skeleton point features from the two different perspectives;

[0015] The target recognition result is determined based on the first target feature.

[0016] In some embodiments, determining the first skeleton point feature based on the first specific graph convolutional network set and the centralized multiple groups of skeleton points includes:

[0017] Mapping the centralized multiple groups of skeleton points to a first linear space through a preset first fully connected layer to obtain each group of mapped skeleton points;

[0018] Based on each specific graph convolutional network set in the first specific graph convolutional network set, extract features from each group of skeleton points of the mapping to obtain features of each group of skeleton points;

[0019] Each group of skeleton point features is spliced, and the spliced ​​skeleton point features are fused through a preset second fully connected layer to obtain the first skeleton point features.

[0020] In some embodiments, the first context module includes a first one-dimensional convolutional neural network and a first bidirectional long short-term memory network layer, the output of the first one-dimensional convolutional neural network is used as the input of the first bidirectional long short-term memory network layer, and the first target feature includes a first modeling feature and a second modeling feature;

[0021] The determining of the first target feature based on the first context module and the second skeleton point features from two different perspectives includes:

[0022] The second skeleton point feature is sequentially passed through the first one-dimensional convolutional neural network and the first bidirectional long short-term memory network layer to obtain the first modeling feature and the second modeling feature.

[0023] In some embodiments, the target gesture recognition model further includes a first main classifier and a first connectionist temporal classifier;

[0024] The determining the target recognition result based on the first target feature includes:

[0025] determining a first prediction probability based on the first target feature and the first main classifier;

[0026] The target recognition result is determined based on the first prediction probability and the first connectionist temporal classifier.

[0027] In some embodiments, the target posture recognition model further includes a second single-stream network that takes motion information of skeleton points as input, the second single-stream network including a second set of specific graph convolutional networks, a second context module, and a second complementary regularization module;

[0028] The method further comprises:

[0029] Extracting target motion information of the target skeleton points;

[0030] Determine a first motion information feature based on the second set of specific graph convolutional networks and the target motion information;

[0031] Determine two second motion information features of different viewing angles based on the two currently generated complementary second masks and the first motion information features;

[0032] Determine a second target feature based on the second context module and the second motion information features of the two different perspectives;

[0033] The determining the target recognition result based on the first target feature includes:

[0034] The target recognition result is determined based on the first target feature and the second target feature.

[0035] In some embodiments, determining the first motion information feature based on the second set of specific graph convolutional networks and the target motion information includes:

[0036] Dividing the target motion information into multiple groups, the skeleton points corresponding to each group of motion information belong to the same preset human body part;

[0037] Centralizing each group of motion information to obtain centralized motion information;

[0038] The first motion information feature is determined based on the second set of specific graph convolutional network sets and each set of centralized motion information.

[0039] In some embodiments, determining the first motion information feature based on the second set of specific graph convolutional network sets and each set of centralized motion information includes:

[0040] Mapping each group of the centralized motion information to the second linear space through a preset third fully connected layer to obtain each group of mapped motion information;

[0041] Based on each group of specific graph convolutional networks in the second group of specific graph convolutional network sets, extract features from each group of motion information of the mapping to obtain each group of motion information features;

[0042] Each group of motion information features is spliced, and the spliced ​​motion information is fused through a preset fourth fully connected layer to obtain the first motion information feature.

[0043] In some embodiments, the second context module includes a second one-dimensional convolutional neural network and a second bidirectional long short-term memory network layer, the output of the second one-dimensional convolutional neural network is used as the input of the second bidirectional long short-term memory network layer, and the second target feature includes a third modeling feature and a fourth modeling feature;

[0044] The determining the second target feature based on the second context module and the second motion information features of the two different perspectives includes:

[0045] The second motion information feature is sequentially passed through the second one-dimensional convolutional neural network and the second bidirectional long short-term memory network layer to obtain the third modeling feature and the fourth modeling feature.

[0046] In some embodiments, determining the target recognition result based on the first target feature and the second target feature includes:

[0047] Stepwise fusion of the first skeleton point feature, the first modeling feature, and the second modeling feature with the first motion information feature, the third modeling feature, and the fourth modeling feature to obtain a stepwise fusion feature;

[0048] Processing the step-by-step fusion features based on two randomly generated complementary third masks to obtain target fusion features from two different perspectives;

[0049] Determine a third target feature based on the third context module and the target fusion feature;

[0050] The target recognition result is determined based on the third target feature.

[0051] In some embodiments, the stepwise fusion of the first skeleton point feature, the first modeling feature, and the second modeling feature with the first motion information feature, the third modeling feature, and the fourth modeling feature to obtain the stepwise fusion feature includes:

[0052] Based on a first preset fusion method, the first skeleton point feature and the first motion information feature are fused to obtain a first fusion feature, wherein the first preset fusion method includes a series connection method;

[0053] Based on a second preset fusion method, the first modeling feature and the third modeling feature are fused to obtain a second fused feature, wherein the second preset fusion method includes an element addition method;

[0054] Based on the first preset fusion mode, the first fusion feature and the second fusion feature are fused to obtain a third fusion feature;

[0055] Based on the second preset fusion mode, the second modeling feature is fused with the fourth modeling feature to obtain a fourth fusion feature;

[0056] Based on the first preset fusion mode, the third fusion feature and the fourth fusion feature are fused to obtain the step-by-step fusion feature.

[0057] In some embodiments, the third context module includes a third one-dimensional convolutional neural network and a third bidirectional long short-term memory network layer, the output of the third one-dimensional convolutional neural network is used as the input of the third bidirectional long short-term memory network layer, and the third target feature includes a fifth modeling feature and a sixth modeling feature;

[0058] The determining a third target feature based on the third context module and the target fusion feature includes:

[0059] The target fusion feature is sequentially passed through the third one-dimensional convolutional neural network and the third bidirectional long short-term memory network layer to obtain the fifth modeling feature and the sixth modeling feature.

[0060] In some embodiments, the target gesture recognition model further includes a second main classifier and a second connectionist temporal classifier;

[0061] The determining the target recognition result based on the third target feature includes:

[0062] determining a second prediction probability based on the third target feature and the second main classifier;

[0063] The target recognition result is determined based on the second prediction probability and the second connectionist temporal classifier.

[0064] In some embodiments, the method further comprises training the target gesture recognition model based on:

[0065] In response to acquiring a sample video, extracting sample skeleton points of a person in each frame of the sample video, wherein the sample skeleton points include skeleton points of a preset human body part of the person, and the preset human body part includes at least a hand;

[0066] Dividing the sample skeleton points into multiple groups, each group of sample skeleton points belongs to the same preset human body part;

[0067] Performing centralization processing on each group of sample skeleton points respectively to obtain centralized groups of sample skeleton points;

[0068] The pre-constructed model to be trained is trained based on each group of centralized sample skeleton points to obtain the target posture recognition model.

[0069] According to a second aspect of an embodiment of the present disclosure, a gesture recognition device is provided, the device comprising:

[0070] A skeleton point extraction module, configured to extract target skeleton points of a person in each frame of the target video in response to receiving the target video to be identified, wherein the target skeleton points include skeleton points of preset human body parts of the person, and the preset human body parts include at least hands;

[0071] A skeleton point division module, used for dividing the target skeleton points into multiple groups, each group of skeleton points corresponding to the same preset human body part;

[0072] A centralization processing module, used to perform centralization processing on each group of skeleton points to obtain multiple groups of centralized skeleton points;

[0073] The recognition result acquisition module is used to obtain the target recognition result corresponding to the target video based on the centralized multiple groups of skeleton points and the pre-trained target posture recognition model.

[0074] In some embodiments, the target gesture recognition model includes a first single-stream network with skeleton points as input, the first single-stream network including a first set of specific graph convolutional networks, a first context module, and a first complementary regularization module;

[0075] The recognition result acquisition module includes:

[0076] A first feature determining unit, configured to determine a first skeleton point feature based on the first specific graph convolutional network set and the centralized multiple groups of skeleton points;

[0077] A second feature determining unit, configured to determine second skeleton point features of two different viewing angles based on the two currently generated complementary first masks and the first skeleton point features;

[0078] A first target determining unit, configured to determine a first target feature based on the first context module and the second skeleton point features from the two different perspectives;

[0079] A recognition result determination unit is used to determine the target recognition result based on the first target feature.

[0080] In some embodiments, the first feature determination unit is further configured to:

[0081] Mapping the centralized multiple groups of skeleton points to a first linear space through a preset first fully connected layer to obtain each group of mapped skeleton points;

[0082] Based on each specific graph convolutional network set in the first specific graph convolutional network set, extract features from each group of skeleton points of the mapping to obtain features of each group of skeleton points;

[0083] Each group of skeleton point features is spliced, and the spliced ​​skeleton point features are fused through a preset second fully connected layer to obtain the first skeleton point features.

[0084] In some embodiments, the first context module includes a first one-dimensional convolutional neural network and a first bidirectional long short-term memory network layer, the output of the first one-dimensional convolutional neural network is used as the input of the first bidirectional long short-term memory network layer, and the first target feature includes a first modeling feature and a second modeling feature;

[0085] The first target determination unit is also used to sequentially pass the second skeleton point feature through the first one-dimensional convolutional neural network and the first bidirectional long short-term memory network layer to obtain the first modeling feature and the second modeling feature.

[0086] In some embodiments, the target gesture recognition model further includes a first main classifier and a first connectionist temporal classifier;

[0087] The recognition result determination unit is also used for:

[0088] determining a first prediction probability based on the first target feature and the first main classifier;

[0089] The target recognition result is determined based on the first prediction probability and the first connectionist temporal classifier.

[0090] In some embodiments, the target posture recognition model further includes a second single-stream network that takes motion information of skeleton points as input, the second single-stream network including a second set of specific graph convolutional networks, a second context module, and a second complementary regularization module;

[0091] The recognition result acquisition module also includes a second target determination unit;

[0092] The second target determination unit is further used for:

[0093] Extracting target motion information of the target skeleton points;

[0094] Determine a first motion information feature based on the second set of specific graph convolutional networks and the target motion information;

[0095] Determine two second motion information features of different viewing angles based on the two currently generated complementary second masks and the first motion information features;

[0096] Determine a second target feature based on the second context module and the second motion information features of the two different perspectives;

[0097] The recognition result determination unit is further configured to determine the target recognition result based on the first target feature and the second target feature.

[0098] In some embodiments, the second target determination unit is further configured to:

[0099] Dividing the target motion information into multiple groups, the skeleton points corresponding to each group of motion information belong to the same preset human body part;

[0100] Centralizing each group of motion information to obtain centralized motion information;

[0101] The first motion information feature is determined based on the second set of specific graph convolutional network sets and each set of centralized motion information.

[0102] In some embodiments, the second target determination unit is further configured to:

[0103] Mapping each group of the centralized motion information to the second linear space through a preset third fully connected layer to obtain each group of mapped motion information;

[0104] Based on each group of specific graph convolutional networks in the second group of specific graph convolutional network sets, extract features from each group of motion information of the mapping to obtain each group of motion information features;

[0105] Each group of motion information features is spliced, and the spliced ​​motion information is fused through a preset fourth fully connected layer to obtain the first motion information feature.

[0106] In some embodiments, the second context module includes a second one-dimensional convolutional neural network and a second bidirectional long short-term memory network layer, the output of the second one-dimensional convolutional neural network is used as the input of the second bidirectional long short-term memory network layer, and the second target feature includes a third modeling feature and a fourth modeling feature;

[0107] The second target determination unit is also used to pass the second motion information feature through the second one-dimensional convolutional neural network and the second bidirectional long short-term memory network layer in sequence to obtain the third modeling feature and the fourth modeling feature.

[0108] In some embodiments, the recognition result determination unit is further configured to:

[0109] Stepwise fusion of the first skeleton point feature, the first modeling feature, and the second modeling feature with the first motion information feature, the third modeling feature, and the fourth modeling feature to obtain a stepwise fusion feature;

[0110] Processing the step-by-step fusion features based on two randomly generated complementary third masks to obtain target fusion features from two different perspectives;

[0111] Determine a third target feature based on the third context module and the target fusion feature;

[0112] The target recognition result is determined based on the third target feature.

[0113] In some embodiments, the recognition result determination unit is further configured to:

[0114] Based on a first preset fusion method, the first skeleton point feature and the first motion information feature are fused to obtain a first fusion feature, wherein the first preset fusion method includes a series connection method;

[0115] Based on a second preset fusion method, the first modeling feature and the third modeling feature are fused to obtain a second fused feature, wherein the second preset fusion method includes an element addition method;

[0116] Based on the first preset fusion mode, the first fusion feature and the second fusion feature are fused to obtain a third fusion feature;

[0117] Based on the second preset fusion mode, the second modeling feature is fused with the fourth modeling feature to obtain a fourth fusion feature;

[0118] Based on the first preset fusion mode, the third fusion feature and the fourth fusion feature are fused to obtain the step-by-step fusion feature.

[0119] In some embodiments, the third context module includes a third one-dimensional convolutional neural network and a third bidirectional long short-term memory network layer, the output of the third one-dimensional convolutional neural network is used as the input of the third bidirectional long short-term memory network layer, and the third target feature includes a fifth modeling feature and a sixth modeling feature;

[0120] The recognition result determination unit is also used to pass the target fusion feature through the third one-dimensional convolutional neural network and the third bidirectional long short-term memory network layer in sequence to obtain the fifth modeling feature and the sixth modeling feature.

[0121] In some embodiments, the target gesture recognition model further includes a second main classifier and a second connectionist temporal classifier;

[0122] The recognition result determination unit is also used for:

[0123] determining a second prediction probability based on the third target feature and the second main classifier;

[0124] The target recognition result is determined based on the second prediction probability and the second connectionist temporal classifier.

[0125] In some embodiments, the apparatus further comprises a model training module;

[0126] The recognition model training module includes:

[0127] A skeleton point extraction unit, configured to extract sample skeleton points of a person in each frame of the sample video in response to acquiring the sample video, wherein the sample skeleton points include skeleton points of preset human body parts of the person, and the preset human body parts include at least a hand;

[0128] A skeleton point division unit, used for dividing the sample skeleton points into multiple groups, each group of sample skeleton points belongs to the same preset human body part;

[0129] A centralization processing unit, used for performing centralization processing on each group of sample skeleton points to obtain centralized sample skeleton points of each group;

[0130] The model training unit is used to train the pre-constructed model to be trained based on each group of centralized sample skeleton points to obtain the target posture recognition model.

[0131] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, the device comprising:

[0132] a processor and a memory for storing a computer program;

[0133] Wherein, the processor is configured to implement, when executing the computer program:

[0134] In response to receiving a target video to be identified, extracting target skeleton points of a person in each frame of the target video, wherein the target skeleton points include skeleton points of preset human body parts of the person, and the preset human body parts include at least a hand;

[0135] Dividing the target skeleton points into multiple groups, each group of skeleton points corresponds to the same preset human body part;

[0136] Performing centralization processing on each group of skeleton points respectively to obtain multiple groups of centralized skeleton points;

[0137] Based on the centralized multiple groups of skeleton points and the pre-trained target posture recognition model, a target recognition result corresponding to the target video is obtained.

[0138] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the following is implemented:

[0139] In response to receiving a target video to be identified, extracting target skeleton points of a person in each frame of the target video, wherein the target skeleton points include skeleton points of preset human body parts of the person, and the preset human body parts include at least a hand;

[0140] Dividing the target skeleton points into multiple groups, each group of skeleton points corresponds to the same preset human body part;

[0141] Performing centralization processing on each group of skeleton points respectively to obtain multiple groups of centralized skeleton points;

[0142] Based on the centralized multiple groups of skeleton points and the pre-trained target posture recognition model, a target recognition result corresponding to the target video is obtained.

[0143] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0144] The present disclosure extracts target skeleton points of a person in each frame image of the target video in response to receiving a target video to be identified, wherein the target skeleton points include skeleton points of preset human body parts of the person, and the preset human body parts include at least hands, and divides the target skeleton points into multiple groups, each group of skeleton points corresponds to the same preset human body part, and then centralizes each group of skeleton points to obtain centralized multiple groups of skeleton points, and then obtains a target recognition result corresponding to the target video based on the centralized multiple groups of skeleton points and a pre-trained target posture recognition model, thereby improving the robustness of posture recognition under different conditions.

[0145] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0146] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0147] Figure 1A is a flow chart of a gesture recognition method according to an exemplary embodiment of the present disclosure;

[0148] Figure 1Bis a schematic diagram of an application scenario of a gesture recognition method according to an exemplary embodiment of the present disclosure;

[0149] Figure 2A is a flowchart showing how to train the target gesture recognition model according to an exemplary embodiment of the present disclosure;

[0150] Figure 2B is a schematic diagram of a single-flow network structure according to an exemplary embodiment of the present disclosure;

[0151] Figure 2C is a schematic diagram of a dual-flow network structure according to an exemplary embodiment of the present disclosure;

[0152] Figure 3 is a block diagram of a gesture recognition device according to an exemplary embodiment of the present disclosure;

[0153] Figure 4 is a block diagram of another gesture recognition device according to an exemplary embodiment of the present disclosure;

[0154] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0155] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0156] Figure 1A is a flowchart of a gesture recognition method according to an exemplary embodiment; the method of this embodiment can be executed by a gesture recognition device, which can be configured in an electronic device, such as a server, a workstation, a personal computer, a mobile terminal (such as a mobile phone, a tablet computer, etc.), a wearable device (such as glasses, a watch, etc.), etc. Specifically, Figure 1A As shown, the method includes the following steps S101-S103:

[0157] In step S101 , in response to receiving a target video to be identified, target skeleton points of a person in each frame of the target video are extracted.

[0158] The target skeleton points include skeleton points of preset human body parts of the character, and the preset human body parts include at least hands.

[0159] In some embodiments, after acquiring the target video to be identified, the target skeleton points of the person in each frame of the target video can be extracted based on a preset skeleton point estimation algorithm. Exemplarily, the preset human body parts may include at least one of the body, mouth and face in addition to the hands.

[0160] In step S102, the target skeleton points are divided into multiple groups.

[0161] In this embodiment, after the target skeleton points of the person in each frame of the target video are extracted, the target skeleton points can be divided into multiple groups, wherein each group of skeleton points corresponds to the same preset human body part.

[0162] That is, after the target skeleton points of the person in each frame of the target video are extracted, the target skeleton points can be divided into multiple groups according to preset human body parts, so that the skeleton points of the same preset human body part are divided into one group.

[0163] In step S103, each group of skeleton points is centralized to obtain a plurality of centralized groups of skeleton points.

[0164] In this embodiment, after the target skeleton points are divided into multiple groups, each group of skeleton points may be centralized to obtain multiple groups of centralized skeleton points.

[0165] For example, when each group of skeleton points is centralized, the image of each group of skeleton points can be processed into the same specification (size) based on a preset image centering algorithm (including image segmentation and image scaling algorithm), thereby obtaining multiple groups of centralized skeleton points, thereby achieving better exploration of co-occurrence signals in the target video, while ensuring the accuracy of gesture recognition (such as continuous sign language recognition), reducing the computational overhead of the model, and improving the robustness of the gesture recognition scheme under different recognition conditions (such as for different people, lighting, background, etc.).

[0166] In step S104, based on the centralized multiple groups of skeleton points and the pre-trained target posture recognition model, a target recognition result corresponding to the target video is obtained.

[0167] In this embodiment, after the centralization process is performed on each group of skeleton points to obtain multiple centralized groups of skeleton points, the target recognition result corresponding to the target video can be obtained based on the multiple centralized groups of skeleton points and the pre-trained target posture recognition model. The training method of the above-mentioned target posture recognition model can refer to the following Figure 2A The illustrated embodiment will not be described in detail here.

[0168] For example, Figure 1BFIG. 1 is a schematic diagram of an application scenario of a gesture recognition method according to an exemplary embodiment of the present disclosure. Figure 1B As shown, the application scenario involves a deaf-mute 100 who signs, a healthy person 200, a fixed device 300 or a mobile device 400, and a target gesture recognition model 500. Specifically, in an actual scenario (such as a supermarket shopping scenario, etc.), in order to facilitate communication between the healthy person 200 and the deaf-mute 100, a target video of the deaf-mute 100 signing can be recorded by the fixed device 300 or the mobile device 400, and then the target skeleton points of the deaf-mute 100 can be determined from the video, and then the sign language of the deaf-mute 100 can be recognized based on the model 500, and then the recognition result can be output in text form. In this way, the healthy person 200 who has not undergone systematic sign language learning can easily understand the meaning expressed by the deaf-mute 100 by viewing the output of the fixed device 300 or the mobile device 400, meeting the application needs of the actual application scenario.

[0169] From the above description, it can be seen that the method of this embodiment extracts the target skeleton points of the person in each frame image of the target video in response to receiving the target video to be identified, the target skeleton points include the skeleton points of the preset human body parts of the person, and the preset human body parts include at least hands, and the target skeleton points are divided into multiple groups, each group of skeleton points corresponds to the same preset human body part, and then each group of skeleton points is centralized to obtain centralized groups of skeleton points, and then based on the centralized groups of skeleton points and a pre-trained target posture recognition model, the target recognition result corresponding to the target video is obtained, which can reduce the computational overhead of the model while ensuring the accuracy of posture recognition, and can improve the robustness of posture recognition under different conditions.

[0170] In some embodiments, the target posture recognition model may include a first single-stream network with skeleton points as input, wherein the first single-stream network includes a first group-specific graph convolution network (Group-specific Graph Convolution Network, Group-specific GCN for short) set, a first contextual module (ContextualModule) and a first complementary regularization module (Complementary Regularization);

[0171] On this basis, the above step S104 may include the following steps S1041-S1044:

[0172] S1041: Determine a first skeleton point feature based on the first specific graph convolutional network set and the centralized multiple groups of skeleton points, wherein the first skeleton point feature may include a feature in units of frames.

[0173] For example, when determining the first skeleton point feature, the centralized multiple groups of skeleton points can be mapped to the first linear space through a preset first fully connected layer to obtain each group of mapped skeleton points, and then based on each specific graph convolutional network set in the first specific graph convolutional network set, features are extracted from each group of mapped skeleton points to obtain each group of skeleton point features, and then each group of skeleton point features are spliced, and the spliced ​​skeleton point features are fused through a preset second fully connected layer to obtain the first skeleton point feature.

[0174] S1042: Determine second skeleton point features at two different viewing angles based on the two currently generated complementary first masks and the first skeleton point features.

[0175] In this embodiment, the first skeleton point features may be processed by the two currently generated complementary first masks to obtain the second skeleton point features of two different viewing angles. Exemplarily, the two complementary first masks may be randomly generated complementary masks.

[0176] S1043: Determine a first target feature based on the first context module and the second skeleton point features from the two different perspectives.

[0177] For example, the first context module may include a first one-dimensional convolutional neural network (CNN) and a first bidirectional long short-term memory network layer (Bi-directional Long Short-Term Memory, BiLSTM). The output of the first one-dimensional convolutional neural network is used as the input of the first bidirectional long short-term memory network layer, and the first target feature includes a first modeling feature and a second modeling feature;

[0178] On this basis, the above-mentioned step S1043 may include sequentially passing the second skeleton point feature through the first one-dimensional convolutional neural network and the first bidirectional long short-term memory network layer to obtain the first modeling feature and the second modeling feature.

[0179] S1044: Determine the target recognition result based on the first target feature.

[0180] In this embodiment, the target posture recognition model may further include a first main classifier and a first connectionist temporal classifier (Connectionist Temporal Classification, CTC for short). Furthermore, when determining the target recognition result based on the first target feature, a first prediction probability may be determined based on the first target feature and the first main classifier, and the target recognition result may be determined based on the first prediction probability and the first connectionist temporal classifier.

[0181] On the basis of the above embodiment, the target posture recognition model also includes a second single-stream network that uses the motion information of the skeleton points as input, wherein the second single-stream network includes a second set of specific graph convolutional network sets, a second context module, and a second complementary regularization module. In other words, the target posture recognition model can be a dual-stream network composed of a first single-stream network and a second single-stream network. It is worth noting that with the continuous increase of the features involved in the recognition, the recognition performance of the model is also continuously improved, but at the same time, the calculation amount of the model will also continue to increase. Therefore, in practical applications, it can be determined whether to use the first single-stream network or the dual-stream network to complete the posture recognition task according to the computing power of different hardware platforms, so as to achieve a balance between the recognition accuracy of the model and the computational overhead of the model.

[0182] On this basis, the method of this embodiment may further include the following steps S1045-S1048:

[0183] S1045: Extract target motion information of the target skeleton point.

[0184] In this embodiment, when extracting the target motion information of the target skeleton point, the position of the target skeleton point in each frame image in the target video can be determined, and then the position change information of the corresponding skeleton point can be determined based on the mapping relationship of the same skeleton point in each frame image, thereby obtaining the target motion information of the target skeleton point.

[0185] S1046: Determine a first motion information feature based on the second group of specific graph convolutional network sets and the target motion information.

[0186] For example, the target motion information can be divided into multiple groups, wherein the skeleton points corresponding to each group of motion information belong to the same preset human body part, and then each group of motion information is centrally processed to obtain each centralized group of motion information, and then the first motion information feature is determined based on the second group of specific graph convolutional network sets and each centralized group of motion information.

[0187] In some embodiments, when determining the first motion information feature, each group of the centralized motion information can be mapped to the second linear space through a preset third fully connected layer to obtain each group of mapped motion information, and then based on each group of specific graph convolutional networks in the second group of specific graph convolutional network sets, features are extracted from each group of mapped motion information to obtain each group of motion information features, and then each group of motion information features is spliced, and the spliced ​​motion information is fused through a preset fourth fully connected layer to obtain the first motion information feature.

[0188] S1047: Determine second motion information features of two different viewing angles based on the two currently generated complementary second masks and the first motion information features.

[0189] In this embodiment, the first motion information feature may be processed by two currently generated complementary second masks to obtain two second motion information features of different viewing angles. Exemplarily, the two complementary second masks may be randomly generated complementary masks.

[0190] S1048: Determine a second target feature based on the second context module and the second motion information features of the two different perspectives.

[0191] In this embodiment, the second motion information features of the two different perspectives may be input into a second context module to obtain a second target feature.

[0192] On this basis, the above S1044 may include determining the target recognition result based on the first target feature and the second target feature.

[0193] Further, the second context module may include a second one-dimensional convolutional neural network and a second bidirectional long short-term memory network layer, the output of the second one-dimensional convolutional neural network is used as the input of the second bidirectional long short-term memory network layer, and the second target feature includes a third modeling feature and a fourth modeling feature;

[0194] On this basis, the above S1048 may include passing the second motion information feature through the second one-dimensional convolutional neural network and the second bidirectional long short-term memory network layer in sequence to obtain the third modeling feature and the fourth modeling feature.

[0195] On this basis, the above S1044 may further include the following steps (1) to (4):

[0196] (1) gradually fusing the first skeleton point feature, the first modeling feature, and the second modeling feature with the first motion information feature, the third modeling feature, and the fourth modeling feature to obtain a gradually fused feature;

[0197] (2) processing the step-by-step fusion features based on two randomly generated complementary third masks to obtain target fusion features at two different perspectives;

[0198] (3) determining a third target feature based on the third context module and the target fusion feature;

[0199] (4) Determine the target recognition result based on the third target feature.

[0200] In some embodiments, the above step (1) may further include the following (11) to (15):

[0201] (11) Based on a first preset fusion method, the first skeleton point feature and the first motion information feature are fused to obtain a first fused feature, wherein the first preset fusion method includes a concatenation method;

[0202] (12) Based on a second preset fusion method, the first modeling feature and the third modeling feature are fused to obtain a second fused feature, wherein the second preset fusion method includes an element-wise addition method;

[0203] (13) Based on the first preset fusion mode, the first fusion feature and the second fusion feature are fused to obtain a third fusion feature;

[0204] (14) Based on the second preset fusion mode, the second modeling feature is fused with the fourth modeling feature to obtain a fourth fused feature;

[0205] (15) Based on the first preset fusion method, the third fusion feature and the fourth fusion feature are fused to obtain the step-by-step fusion feature.

[0206] In some embodiments, the third context module may include a third one-dimensional convolutional neural network and a third bidirectional long short-term memory network layer, the output of the third one-dimensional convolutional neural network is used as the input of the third bidirectional long short-term memory network layer, and the third target feature includes a fifth modeling feature and a sixth modeling feature;

[0207] Furthermore, the above step (3) may include passing the target fusion feature through the third one-dimensional convolutional neural network and the third bidirectional long short-term memory network layer in sequence to obtain the fifth modeling feature and the sixth modeling feature.

[0208] Based on the above embodiment, the target posture recognition model may further include a second main classifier and a second connectionist temporal classifier;

[0209] The above step (4) may also include determining a second prediction probability based on the third target feature and the second main classifier, and determining the target recognition result based on the second prediction probability and the second connectionist temporal classifier.

[0210] Figure 2AFIG. 1 is a flowchart showing how to train the target posture recognition model according to an exemplary embodiment of the present disclosure; based on the above embodiment, this embodiment takes how to train the target posture recognition model as an example for exemplary description. Figure 2A As shown, the gesture recognition method of this embodiment may further include training the target gesture recognition model based on the following steps S201-S204:

[0211] In step S201, in response to acquiring a sample video, sample skeleton points of a person in each frame of the sample video are extracted, wherein the sample skeleton points may include skeleton points of preset human body parts of the person, and the preset human body parts at least include hands.

[0212] In some embodiments, after the sample video is acquired, the sample skeleton points of the person in each frame of the sample video can be extracted based on a preset skeleton point estimation algorithm. Exemplarily, the preset human body parts may include at least one of the body, mouth and face in addition to the hands.

[0213] In step S202, the sample skeleton points are divided into multiple groups.

[0214] In this embodiment, after extracting sample skeleton points of a person in each frame of the sample video, the sample skeleton points may be divided into multiple groups, wherein each group of skeleton points corresponds to the same preset human body part.

[0215] That is, after extracting the sample skeleton points of the person in each frame of the sample video, the sample skeleton points can be divided into multiple groups according to the preset human body parts, so that the skeleton points of the same preset human body part are divided into one group.

[0216] In step S203, each group of sample skeleton points is centralized to obtain a centralized group of sample skeleton points.

[0217] In this embodiment, after the sample skeleton points are divided into multiple groups, each group of sample skeleton points may be centralized to obtain centralized sample skeleton points of each group.

[0218] For example, when each group of sample skeleton points is centralized, the image of each group of sample skeleton points can be processed into the same specification (size) based on a preset image centering algorithm (including image segmentation and image scaling algorithm), thereby obtaining multiple groups of centralized sample skeleton points, thereby achieving better exploration of co-occurrence signals in the target video, while ensuring the accuracy of gesture recognition (such as continuous sign language recognition), reducing the computational overhead of the model, and improving the robustness of the gesture recognition scheme under different recognition conditions (such as for different people, lighting, background, etc.).

[0219] In step S204, the pre-constructed model to be trained is trained based on each group of centralized sample skeleton points to obtain the target posture recognition model.

[0220] In this embodiment, after each group of centralized sample skeleton points is obtained, a pre-constructed model to be trained can be trained based on each group of centralized sample skeleton points to obtain the target posture recognition model.

[0221] For example, Figure 2B FIG. 1 is a schematic diagram of a single-flow network structure according to an exemplary embodiment of the present disclosure. Figure 2B As shown, the target posture recognition model in this embodiment may include a first single-stream network with skeleton points as input. Specifically, the first single-stream network includes a first set of specific graph convolutional network sets (i.e., Figure 2B A group-specific graph convolutional network set consisting of body GCN, hand GCN, mouth GCN, and face GCN), the first context module (i.e., Figure 2B The first one-dimensional convolutional neural network "1D CNN" and the first bidirectional long short-term memory network layer "BiLSTM" and the first complementary regularization module (Complementary Regularization);

[0222] Furthermore, the above step S204 may include the following steps S2041-S2045:

[0223] S2041, determining a first sample skeleton point feature based on the first specific graph convolutional network set and each group of centralized sample skeleton points;

[0224] In this embodiment, each group of centralized sample skeleton points (i.e., each group of sample skeleton points corresponding to the body, hand, mouth, and face) can be input into the body GCN, hand GCN, mouth GCN, and face GCN respectively to obtain the first sample skeleton point feature (i.e., Figure 2B It is worth noting that the skeleton points of the left and right hands can share the same hand GCN.

[0225] S2042: Determine second sample skeleton point features at two different viewing angles based on the two currently generated complementary first masks and the first sample skeleton point features.

[0226] In this embodiment, the first sample skeleton point features may be processed based on two complementary masks (named as first masks for easy distinction) currently generated randomly to obtain second sample skeleton point features of two different viewing angles.

[0227] S2043: Determine a first sample target feature based on the first context module and the second sample skeleton point features from the two different perspectives.

[0228] In this embodiment, based on the first context module (ie, Figure 2B The two 1D CNNs and BiLSTM in the upper and lower parts process the skeleton point features of the second samples from two different perspectives to obtain the target features of the first sample.

[0229] S2044. Construct a target loss function based on a first prediction probability corresponding to the first sample target feature, a second prediction probability corresponding to the second sample skeleton point feature, and a first regularized loss of the second sample skeleton point features from two different perspectives, wherein the first regularized loss is used to constrain the prediction consistency of the second sample skeleton point features from two different perspectives.

[0230] For example, this embodiment can express the first regularization loss based on the following formula (2-1):

[0231]

[0232] In the above formula, D KL represents KL divergence (Kullback-Leibler divergence), p1 and p2 represent the predicted probabilities generated by the second sample skeleton point features from two different perspectives after passing through the classifier.

[0233] On this basis, this embodiment can add a classifier after the features output by the 1D CNN layer and the final features (i.e., the features output by the BiLSTM). The two classifiers can be respectively called auxiliary classifiers and main classifiers, and the prediction probabilities of the two classifiers are constrained by CTC loss. Then, the overall loss function, i.e., the above-mentioned target loss function, can be constructed based on the following formula (2-2):

[0234]

[0235] Among them, v represents the predicted probability of the auxiliary classifier output, z represents the predicted probability of the main classifier output, subscripts 1 and 2 represent features from different perspectives, l represents the label, and α and β are hyperparameters used to control the impact of regularization loss.

[0236] S2045. Train the model to be trained based on the target loss function to obtain the target posture recognition model.

[0237] The above step S2041 may further include the following steps S411-S413:

[0238] S411, mapping each group of centralized sample skeleton points to a first linear space through a preset first fully connected layer to obtain each group of mapped sample skeleton points;

[0239] S412, based on each specific graph convolution network in the first specific graph convolution network set, extracting features from each group of sample skeleton points of the mapping to obtain features of each group of sample skeleton points;

[0240] S413 , concatenating each group of sample skeleton point features, and fusing the concatenated sample skeleton point features through a preset second fully connected layer to obtain the first sample skeleton point features.

[0241] Furthermore, the first context module may include a first one-dimensional convolutional neural network and a first bidirectional long short-term memory network layer. The output of the first one-dimensional convolutional neural network is used as the input of the first bidirectional long short-term memory network layer, and the first sample target feature includes a first sample modeling feature and a second sample modeling feature.

[0242] On this basis, the above S2043 may include passing the second sample skeleton point features through the first one-dimensional convolutional neural network and the first bidirectional long short-term memory network layer in sequence to obtain the first sample modeling features and the second sample modeling features.

[0243] For example, Figure 2C is a schematic diagram of a dual-stream network structure according to an exemplary embodiment of the present disclosure. In addition to the first single-stream network with skeleton points as input, the target posture recognition model of this embodiment may also include a second single-stream network with motion information of skeleton points as input. Similar to the structure of the first single-stream network, the second single-stream network includes a second specific graph convolutional network set, a second context module, and a second complementary regularization module. In other words, Figure 2C As shown, the target posture recognition model can be a dual-stream network consisting of a first single-stream network with skeleton points as input and a second single-stream network with skeleton point motion information as input. It is worth noting that with the continuous increase of features involved in recognition, the recognition performance of the model is also continuously improved, but at the same time, the amount of calculation of the model will continue to increase. Therefore, in practical applications, it can be determined whether to use the first single-stream network or the dual-stream network to complete the posture recognition task according to the computing power of different hardware platforms, so as to achieve a balance between the recognition accuracy of the model and the computational overhead of the model.

[0244] On this basis, the method of this embodiment may further include the following steps S2046-S2049:

[0245] S2046, extracting sample motion information of the sample skeleton points;

[0246] S2047, determining a first sample motion information feature based on the second specific graph convolutional network set and the sample motion information;

[0247] S2048, determining second sample motion information features of two different perspectives based on the two currently generated complementary second masks and the first sample motion information features;

[0248] S2049: Determine a second sample target feature based on the second context module and the second sample motion information features of the two different perspectives;

[0249] On this basis, constructing a target loss function based on the first prediction probability corresponding to the first sample target feature, the second prediction probability corresponding to the second sample skeleton point feature, and the first regularized loss of the second sample skeleton point features from two different perspectives described in S2044 may include:

[0250] The target loss function is constructed based on the first prediction probability, the second prediction probability, the first regularization loss, the third prediction probability corresponding to the second sample target feature, the fourth prediction probability corresponding to the second sample motion information feature, and the second regularization loss of the second sample motion information features of the two different perspectives, and the second regularization loss is used to constrain the prediction consistency of the second sample motion information features of the two different perspectives.

[0251] In some embodiments, the determining of the first sample motion information feature based on the second specific graph convolutional network set and the sample motion information described in S2047 above may include the following S471-S473:

[0252] S471, dividing the sample motion information into multiple groups, where the sample skeleton points corresponding to each group of sample motion information belong to the same preset human body part;

[0253] S472, performing centralization processing on each group of sample motion information respectively to obtain centralized sample motion information of each group;

[0254] S473. Determine a first sample motion information feature based on the second specific graph convolutional network set and each group of centralized sample motion information.

[0255] Furthermore, the above S473 may include the following steps S731-S733:

[0256] S731, mapping each group of the centralized sample motion information to a second linear space through a preset third fully connected layer to obtain mapped each group of sample motion information;

[0257] S732. Based on each specific graph convolution network in the second specific graph convolution network set, extract features from each group of sample motion information mapped to obtain features of each group of sample motion information;

[0258] S733: Concatenate each group of sample motion information features, and fuse the concatenated sample motion information through a preset fourth fully connected layer to obtain the first sample motion information features.

[0259] In some other embodiments, the second context module may include a second one-dimensional convolutional neural network and a second bidirectional long short-term memory network layer, the output of the second one-dimensional convolutional neural network is used as the input of the second bidirectional long short-term memory network layer, and the second sample target feature includes a third sample modeling feature and a fourth sample modeling feature;

[0260] Furthermore, the above-mentioned step S2049 may include sequentially passing the second sample motion information feature through the second one-dimensional convolutional neural network and the second bidirectional long short-term memory network layer to obtain the third sample modeling feature and the fourth sample modeling feature.

[0261] Based on the above embodiment, the method of this embodiment may further include determining a third sample target feature based on the following steps (1) to (3):

[0262] (1) gradually fusing the first sample skeleton point feature, the first sample modeling feature, and the second sample modeling feature with the first sample motion information feature, the third sample modeling feature, and the fourth sample modeling feature to obtain a sample gradually fused feature;

[0263] (2) processing the sample step-by-step fusion features based on two randomly generated complementary third masks to obtain sample target fusion features of two different perspectives;

[0264] (3) determining a third sample target feature based on the third context module and the sample target fusion feature;

[0265] The constructing of the target loss function based on the first prediction probability, the second prediction probability, the first regularization loss, the third prediction probability corresponding to the second sample target feature, the fourth prediction probability corresponding to the second sample motion information feature, and the second regularization loss of the second sample motion information features of the two different perspectives described in S2044 may further include:

[0266] Based on the first prediction probability, the second prediction probability, the first regularization loss, the third prediction probability, the fourth prediction probability, the second regularization loss, the target prediction probability corresponding to the third sample target feature, the sixth prediction probability corresponding to the sample target fusion feature, and the third regularization loss of the sample target fusion features of the two different perspectives, the target loss function is constructed, and the second regularization loss is used to constrain the prediction consistency of the sample target fusion features of the two different perspectives.

[0267] For example, the stepwise fusion of the first sample skeleton point feature, the first sample modeling feature, and the second sample modeling feature with the first sample motion information feature, the third sample modeling feature, and the fourth sample modeling feature in (1) above to obtain the sample stepwise fusion feature may include the following (11)-(15):

[0268] (11) Based on a first preset fusion method, the first sample skeleton point feature and the first sample motion information feature are fused to obtain a first sample fusion feature.

[0269] like Figure 2C As shown, when the first sample skeleton point feature is obtained and the first sample motion information feature After that, the two can be fused based on the first preset fusion method to obtain the first sample fusion feature The first preset fusion method includes a concatenation method.

[0270] (12) Based on a second preset fusion method, the first sample modeling feature and the third sample modeling feature are fused to obtain a second sample fusion feature.

[0271] like Figure 2C As shown, when the first sample modeling feature is obtained Modeling features with the third sample After that, the first sample modeling feature can be combined based on the second preset fusion method. Modeling features with the third sample A fusion process is performed to obtain a second sample fusion feature, wherein the second preset fusion method includes an element-wise addition method.

[0272] (13) Based on the first preset fusion mode, the first sample fusion feature and the second sample fusion feature are fused to obtain a third sample fusion feature;

[0273] like Figure 2CAs shown, when the first sample fusion feature is obtained After the features are fused with the second sample, the first sample fusion features can be fused with the second sample based on the first preset fusion method. The third sample fusion feature is obtained by fusing the second sample fusion feature

[0274] (14) Based on the second preset fusion mode, the second sample modeling feature is fused with the fourth sample modeling feature to obtain a fourth sample fusion feature;

[0275] like Figure 2C As shown, when the second sample modeling feature is obtained Modeling characteristics with the fourth sample Then, based on the second preset fusion mode, the second sample modeling feature Modeling characteristics with the fourth sample The fusion process is performed to obtain the fourth sample fusion feature.

[0276] (15) Based on the first preset fusion method, the third sample fusion feature and the fourth sample fusion feature are fused to obtain the sample step-by-step fusion feature.

[0277] like Figure 2C As shown, when the third sample fusion feature is obtained After the features are fused with the fourth sample, the third sample fusion features can be fused with the fourth sample based on the first preset fusion method. The sample is fused with the fourth sample fusion feature to obtain the sample step-by-step fusion feature.

[0278] On this basis, the samples can be gradually fused with features The features are sent to the contextual module for temporal modeling, and the obtained features are used for final recognition. Similar to the complementary regularization operation in the single-stream network, two complementary masks can also be generated here (for the sake of distinction, called the second mask) to act on the corresponding features to obtain features from two different perspectives (i.e., sample target fusion features from two different perspectives). In other words, Figure 2C The structure of the context module on the right side can be referred to Figure 2B The structure of the context module in the middle right half will not be described in detail here.

[0279] In some embodiments, during the model training process, in order to ensure the convergence of each branch, the present embodiment may first pre-train the skeleton point branch and the motion branch, and then start training the dual-stream network. At the same time, in the present embodiment, two additional auxiliary classifiers may be added after the skeleton point branch and the motion branch to ensure sufficient training of the two branches. Therefore, the overall loss function (i.e., the target loss function) may be as shown in the following formula (2-3):

[0280]

[0281] Among them, v s and v m Represent the predicted probabilities of the auxiliary classifier outputs after skeleton points and motion information, respectively. v represents the output probability of the auxiliary classifier after 1D CNN. z is the output of the main classifier. The meanings of subscripts 1 and 2 are the same as those of the single-stream network, that is, they represent features from different perspectives. α and β are also hyperparameters used to control the influence of regularization loss. λ is a hyperparameter that controls the influence of the auxiliary loss of the sub-skeleton points and motion information branches.

[0282] In some embodiments, the third context module may include a third one-dimensional convolutional neural network and a third bidirectional long short-term memory network layer, the output of the third one-dimensional convolutional neural network is used as the input of the third bidirectional long short-term memory network layer, and the third sample target feature includes a fifth sample modeling feature and a sixth sample modeling feature;

[0283] Furthermore, the step (3) of determining the third sample target feature based on the third context module and the sample target fusion feature may include: sequentially passing the sample target fusion feature through the third one-dimensional convolutional neural network and the third bidirectional long short-term memory network layer to obtain the fifth sample modeling feature and the sixth sample modeling feature.

[0284] Figure 3 is a block diagram of a gesture recognition device according to an exemplary embodiment of the present disclosure; the device of this embodiment can be configured in an electronic device, such as a server, a workstation, a personal computer, a mobile terminal (such as a mobile phone, a tablet computer, etc.), a wearable device (such as glasses, a watch, etc.), etc. Specifically, Figure 3 As shown, the device may include: a skeleton point extraction module 110, a skeleton point division module 120, a centralization processing module 130 and a recognition result acquisition module 140, wherein:

[0285] The skeleton point extraction module 110 is used for extracting target skeleton points of a person in each frame of the target video in response to receiving the target video to be identified, wherein the target skeleton points include skeleton points of preset human body parts of the person, and the preset human body parts include at least hands;

[0286] A skeleton point division module 120, for dividing the target skeleton points into multiple groups, each group of skeleton points corresponding to the same preset human body part;

[0287] A centralization processing module 130, configured to perform centralization processing on each group of skeleton points to obtain a plurality of centralized groups of skeleton points;

[0288] The recognition result acquisition module 140 is used to obtain the target recognition result corresponding to the target video based on the centralized multiple groups of skeleton points and the pre-trained target posture recognition model.

[0289] It can be seen from the above description that the device of this embodiment extracts the target skeleton points of the person in each frame image of the target video in response to receiving the target video to be identified, the target skeleton points include the skeleton points of preset human body parts of the person, and the preset human body parts include at least hands, and the target skeleton points are divided into multiple groups, each group of skeleton points corresponds to the same preset human body part, and then each group of skeleton points is centralized to obtain centralized groups of skeleton points, and then based on the centralized groups of skeleton points and a pre-trained target posture recognition model, the target recognition result corresponding to the target video is obtained, which can reduce the computational overhead of the model while ensuring the accuracy of posture recognition, and can improve the robustness of posture recognition under different conditions.

[0290] Figure 4 2 is a block diagram of another gesture recognition device according to an exemplary embodiment of the present disclosure; the device of this embodiment can be configured in an electronic device, such as a server, a workstation, a personal computer, a mobile terminal (such as a mobile phone, a tablet computer, etc.), a wearable device (such as glasses, watches, etc.), etc. Among them, the skeleton point extraction module 210, the skeleton point division module 220, the centralization processing module 230 and the recognition result acquisition module 240 are similar to the aforementioned Figure 3 The functions of the skeleton point extraction module 110 , the skeleton point division module 120 , the centralization processing module 130 and the recognition result acquisition module 140 in the illustrated embodiment are the same and will not be described in detail herein.

[0291] In this embodiment, the target posture recognition model may include a first single-stream network with skeleton points as input, wherein the first single-stream network includes a first set of specific graph convolutional networks, a first context module, and a first complementary regularization module;

[0292] Furthermore, the recognition result acquisition module 240 includes:

[0293] A first feature determining unit 241, configured to determine a first skeleton point feature based on the first specific graph convolutional network set and the centralized multiple groups of skeleton points;

[0294] A second feature determining unit 242, configured to determine second skeleton point features of two different viewing angles based on the two currently generated complementary first masks and the first skeleton point features;

[0295] A first target determining unit 243, configured to determine a first target feature based on the first context module and the second skeleton point features from the two different perspectives;

[0296] The recognition result determining unit 244 is configured to determine the target recognition result based on the first target feature.

[0297] In some embodiments, the first feature determination unit 241 may also be used to:

[0298] Mapping the centralized multiple groups of skeleton points to a first linear space through a preset first fully connected layer to obtain each group of mapped skeleton points;

[0299] Based on each specific graph convolutional network set in the first specific graph convolutional network set, extract features from each group of skeleton points of the mapping to obtain features of each group of skeleton points;

[0300] Each group of skeleton point features is spliced, and the spliced ​​skeleton point features are fused through a preset second fully connected layer to obtain the first skeleton point features.

[0301] In some embodiments, the first context module includes a first one-dimensional convolutional neural network and a first bidirectional long short-term memory network layer, the output of the first one-dimensional convolutional neural network is used as the input of the first bidirectional long short-term memory network layer, and the first target feature includes a first modeling feature and a second modeling feature;

[0302] Furthermore, the first target determination unit 243 is also used to pass the second skeleton point feature through the first one-dimensional convolutional neural network and the first bidirectional long short-term memory network layer in sequence to obtain the first modeling feature and the second modeling feature.

[0303] In some embodiments, the target gesture recognition model may further include a first main classifier and a first connectionist temporal classifier;

[0304] Furthermore, the recognition result determination unit 244 may also be used to:

[0305] determining a first prediction probability based on the first target feature and the first main classifier;

[0306] The target recognition result is determined based on the first prediction probability and the first connectionist temporal classifier.

[0307] In some embodiments, the target gesture recognition model further includes a second single-stream network that takes the motion information of the skeleton points as input, wherein the second single-stream network includes a second set of specific graph convolutional networks, a second context module, and a second complementary regularization module;

[0308] Furthermore, the recognition result acquisition module 240 may further include a second target determination unit 245;

[0309] The second target determination unit 245 may also be used to:

[0310] Extracting target motion information of the target skeleton points;

[0311] Determine a first motion information feature based on the second set of specific graph convolutional networks and the target motion information;

[0312] Determine two second motion information features of different viewing angles based on the two currently generated complementary second masks and the first motion information features;

[0313] Determine a second target feature based on the second context module and the second motion information features of the two different perspectives;

[0314] The recognition result determination unit 244 may also be configured to determine the target recognition result based on the first target feature and the second target feature.

[0315] In some embodiments, the second target determination unit 245 may also be used to:

[0316] Dividing the target motion information into multiple groups, the skeleton points corresponding to each group of motion information belong to the same preset human body part;

[0317] Centralizing each group of motion information to obtain centralized motion information;

[0318] The first motion information feature is determined based on the second set of specific graph convolutional network sets and each set of centralized motion information.

[0319] In some embodiments, the second target determination unit 245 may also be used to:

[0320] Mapping each group of the centralized motion information to the second linear space through a preset third fully connected layer to obtain each group of mapped motion information;

[0321] Based on each group of specific graph convolutional networks in the second group of specific graph convolutional network sets, extract features from each group of motion information of the mapping to obtain each group of motion information features;

[0322] Each group of motion information features is spliced, and the spliced ​​motion information is fused through a preset fourth fully connected layer to obtain the first motion information feature.

[0323] In some embodiments, the second context module may include a second one-dimensional convolutional neural network and a second bidirectional long short-term memory network layer, the output of the second one-dimensional convolutional neural network is used as the input of the second bidirectional long short-term memory network layer, and the second target feature includes a third modeling feature and a fourth modeling feature;

[0324] Furthermore, the second target determination unit 245 can also be used to pass the second motion information feature through the second one-dimensional convolutional neural network and the second bidirectional long short-term memory network layer in sequence to obtain the third modeling feature and the fourth modeling feature.

[0325] In some embodiments, the recognition result determination unit 244 may also be used to:

[0326] Stepwise fusion of the first skeleton point feature, the first modeling feature, and the second modeling feature with the first motion information feature, the third modeling feature, and the fourth modeling feature to obtain a stepwise fusion feature;

[0327] Processing the step-by-step fusion features based on two randomly generated complementary third masks to obtain target fusion features from two different perspectives;

[0328] Determine a third target feature based on the third context module and the target fusion feature;

[0329] The target recognition result is determined based on the third target feature.

[0330] In some embodiments, the recognition result determination unit 244 may also be used to:

[0331] Based on a first preset fusion method, the first skeleton point feature and the first motion information feature are fused to obtain a first fusion feature, wherein the first preset fusion method includes a series connection method;

[0332] Based on a second preset fusion method, the first modeling feature and the third modeling feature are fused to obtain a second fused feature, wherein the second preset fusion method includes an element addition method;

[0333] Based on the first preset fusion mode, the first fusion feature and the second fusion feature are fused to obtain a third fusion feature;

[0334] Based on the second preset fusion mode, the second modeling feature is fused with the fourth modeling feature to obtain a fourth fusion feature;

[0335] Based on the first preset fusion mode, the third fusion feature and the fourth fusion feature are fused to obtain the step-by-step fusion feature.

[0336] In some embodiments, the third context module may include a third one-dimensional convolutional neural network and a third bidirectional long short-term memory network layer, the output of the third one-dimensional convolutional neural network is used as the input of the third bidirectional long short-term memory network layer, and the third target feature includes a fifth modeling feature and a sixth modeling feature;

[0337] Furthermore, the recognition result determination unit 244 can also be used to pass the target fusion feature through the third one-dimensional convolutional neural network and the third bidirectional long short-term memory network layer in sequence to obtain the fifth modeling feature and the sixth modeling feature.

[0338] In some embodiments, the target gesture recognition model further includes a second main classifier and a second connectionist temporal classifier;

[0339] Furthermore, the recognition result determination unit 244 may also be used to:

[0340] determining a second prediction probability based on the third target feature and the second main classifier;

[0341] The target recognition result is determined based on the second prediction probability and the second connectionist temporal classifier.

[0342] In some embodiments, the above apparatus may further include a model training module 250;

[0343] The recognition model training module 250 may include:

[0344] A skeleton point extraction unit 251 is used for extracting sample skeleton points of a person in each frame of the sample video in response to acquiring the sample video, wherein the sample skeleton points include skeleton points of preset human body parts of the person, and the preset human body parts include at least hands;

[0345] A skeleton point division unit 252, used for dividing the sample skeleton points into multiple groups, each group of sample skeleton points belongs to the same preset human body part;

[0346] A centralization processing unit 253, used to perform centralization processing on each group of sample skeleton points to obtain centralized sample skeleton points of each group;

[0347] The model training unit 254 is used to train the pre-built model to be trained based on each group of centralized sample skeleton points to obtain the target posture recognition model.

[0348] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0349] Figure 5 900 is a block diagram of an electronic device according to an exemplary embodiment. For example, the device 900 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0350] Reference Figure 5 , device 900 may include one or more of the following components: a processing component 902 , a memory 904 , a power component 906 , a multimedia component 908 , an audio component 910 , an input / output (I / O) interface 912 , a sensor component 914 , and a communication component 916 .

[0351] The processing component 902 generally controls the overall operation of the device 900, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the gesture recognition method described above. In addition, the processing component 902 may include one or more modules to facilitate the interaction between the processing component 902 and other components. For example, the processing component 902 may include a multimedia module to facilitate the interaction between the multimedia component 908 and the processing component 902.

[0352] The memory 904 is configured to store various types of data to support operations on the device 900. Examples of such data include instructions for any application or method operating on the device 900, contact data, phone book data, messages, pictures, videos, etc. The memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0353] The power supply component 906 provides power to the various components of the device 900. The power supply component 906 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 900.

[0354] The multimedia component 908 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display panel and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 908 includes a front camera and / or a rear camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0355] The audio component 910 is configured to output and / or input audio signals. For example, the audio component 910 includes a microphone (MIC), and when the device 900 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 904 or sent via the communication component 916. In some embodiments, the audio component 910 also includes a speaker for outputting audio signals.

[0356] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0357] The sensor assembly 914 includes one or more sensors for providing various aspects of status assessment for the device 900. For example, the sensor assembly 914 can detect the open / closed state of the device 900, the relative positioning of components, such as the display panel and keypad of the device 900, and the sensor assembly 914 can also detect the position change of the device 900 or a component of the device 900, the presence or absence of user contact with the device 900, the orientation or acceleration / deceleration of the device 900, and the temperature change of the device 900. The sensor assembly 914 can also include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 914 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 914 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0358] The communication component 916 is configured to facilitate wired or wireless communication between the device 900 and other devices. The device 900 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G or 5G or a combination thereof. In an exemplary embodiment, the communication component 916 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 916 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0359] In an exemplary embodiment, the device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described gesture recognition method.

[0360] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, and the instructions can be executed by the processor 920 of the device 900 to perform the gesture recognition method described above. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0361] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0362] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A gesture recognition method, characterized in that: The method comprises: In response to receiving a target video to be identified, extracting target skeleton points of a person in each frame of the target video, wherein the target skeleton points include skeleton points of preset human body parts of the person, and the preset human body parts include at least a hand; Dividing the target skeleton points into multiple groups, each group of skeleton points corresponds to the same preset human body part; Performing centralization processing on each group of skeleton points respectively to obtain multiple groups of centralized skeleton points; Based on the centralized multiple groups of skeleton points and the pre-trained target posture recognition model, a target recognition result corresponding to the target video is obtained.

2. The method according to claim 1, characterized in that: The target posture recognition model includes a first single-stream network with skeleton points as input, the first single-stream network including a first set of specific graph convolutional networks, a first context module, and a first complementary regularization module; The obtaining of the target recognition result corresponding to the target video based on the centralized multiple groups of skeleton points and the pre-trained target posture recognition model includes: Determining a first skeleton point feature based on the first specific graph convolutional network set and the centralized multiple groups of skeleton points; Determine two second skeleton point features at different viewing angles based on the two currently generated complementary first masks and the first skeleton point features; Determine a first target feature based on the first context module and the second skeleton point features from the two different perspectives; The target recognition result is determined based on the first target feature.

3. The method according to claim 2, characterized in that The determining of the first skeleton point feature based on the first specific graph convolutional network set and the centralized multiple groups of skeleton points includes: Mapping the centralized multiple groups of skeleton points to a first linear space through a preset first fully connected layer to obtain each group of mapped skeleton points; Based on each specific graph convolutional network set in the first specific graph convolutional network set, extract features from each group of skeleton points of the mapping to obtain features of each group of skeleton points; Each group of skeleton point features is spliced, and the spliced ​​skeleton point features are fused through a preset second fully connected layer to obtain the first skeleton point features.

4. The method according to claim 2, characterized in that: The first context module includes a first one-dimensional convolutional neural network and a first bidirectional long short-term memory network layer, the output of the first one-dimensional convolutional neural network is used as the input of the first bidirectional long short-term memory network layer, and the first target feature includes a first modeling feature and a second modeling feature; The determining of the first target feature based on the first context module and the second skeleton point features from two different perspectives includes: The second skeleton point feature is sequentially passed through the first one-dimensional convolutional neural network and the first bidirectional long short-term memory network layer to obtain the first modeling feature and the second modeling feature.

5. The method according to claim 2, characterized in that: The target gesture recognition model also includes a first main classifier and a first connectionist temporal classifier; The determining the target recognition result based on the first target feature includes: determining a first prediction probability based on the first target feature and the first main classifier; The target recognition result is determined based on the first prediction probability and the first connectionist temporal classifier.

6. The method according to claim 2, characterized in that The target posture recognition model also includes a second single-stream network taking the motion information of the skeleton points as input, wherein the second single-stream network includes a second set of specific graph convolutional networks, a second context module, and a second complementary regularization module; The method further comprises: Extracting target motion information of the target skeleton points; Determine a first motion information feature based on the second set of specific graph convolutional networks and the target motion information; Determine two second motion information features of different viewing angles based on the two currently generated complementary second masks and the first motion information features; Determine a second target feature based on the second context module and the second motion information features of the two different perspectives; The determining the target recognition result based on the first target feature includes: The target recognition result is determined based on the first target feature and the second target feature.

7. The method according to claim 6, characterized in that The determining the first motion information feature based on the second group of specific graph convolutional network sets and the target motion information includes: Dividing the target motion information into multiple groups, the skeleton points corresponding to each group of motion information belong to the same preset human body part; Centralizing each group of motion information to obtain centralized motion information; The first motion information feature is determined based on the second set of specific graph convolutional network sets and each set of centralized motion information.

8. The method according to claim 7, characterized in that The determining the first motion information feature based on the second set of specific graph convolutional network sets and each set of centralized motion information includes: Mapping each group of the centralized motion information to the second linear space through a preset third fully connected layer to obtain each group of mapped motion information; Based on each group of specific graph convolutional networks in the second group of specific graph convolutional network sets, extract features from each group of motion information of the mapping to obtain each group of motion information features; Each group of motion information features is spliced, and the spliced ​​motion information is fused through a preset fourth fully connected layer to obtain the first motion information feature.

9. The method according to claim 6, characterized in that The second context module includes a second one-dimensional convolutional neural network and a second bidirectional long short-term memory network layer, the output of the second one-dimensional convolutional neural network is used as the input of the second bidirectional long short-term memory network layer, and the second target feature includes a third modeling feature and a fourth modeling feature; The determining the second target feature based on the second context module and the second motion information features of the two different perspectives includes: The second motion information feature is sequentially passed through the second one-dimensional convolutional neural network and the second bidirectional long short-term memory network layer to obtain the third modeling feature and the fourth modeling feature.

10. The method according to claim 9, characterized in that The determining the target recognition result based on the first target feature and the second target feature includes: Stepwise fusion of the first skeleton point feature, the first modeling feature, and the second modeling feature with the first motion information feature, the third modeling feature, and the fourth modeling feature to obtain a stepwise fusion feature; Processing the step-by-step fusion features based on two randomly generated complementary third masks to obtain target fusion features from two different perspectives; Determine a third target feature based on the third context module and the target fusion feature; The target recognition result is determined based on the third target feature.

11. The method according to claim 10, characterized in that The step of gradually fusing the first skeleton point feature, the first modeling feature, and the second modeling feature with the first motion information feature, the third modeling feature, and the fourth modeling feature to obtain a gradually fused feature includes: Based on a first preset fusion method, the first skeleton point feature and the first motion information feature are fused to obtain a first fusion feature, wherein the first preset fusion method includes a series connection method; Based on a second preset fusion method, the first modeling feature and the third modeling feature are fused to obtain a second fused feature, wherein the second preset fusion method includes an element addition method; Based on the first preset fusion mode, the first fusion feature and the second fusion feature are fused to obtain a third fusion feature; Based on the second preset fusion mode, the second modeling feature is fused with the fourth modeling feature to obtain a fourth fusion feature; Based on the first preset fusion mode, the third fusion feature and the fourth fusion feature are fused to obtain the step-by-step fusion feature.

12. The method according to claim 10, characterized in that The third context module includes a third one-dimensional convolutional neural network and a third bidirectional long short-term memory network layer, the output of the third one-dimensional convolutional neural network is used as the input of the third bidirectional long short-term memory network layer, and the third target feature includes a fifth modeling feature and a sixth modeling feature; The determining a third target feature based on the third context module and the target fusion feature includes: The target fusion feature is sequentially passed through the third one-dimensional convolutional neural network and the third bidirectional long short-term memory network layer to obtain the fifth modeling feature and the sixth modeling feature.

13. The method according to claim 10, characterized in that The target gesture recognition model also includes a second main classifier and a second connectionist temporal classifier; The determining the target recognition result based on the third target feature includes: determining a second prediction probability based on the third target feature and the second main classifier; The target recognition result is determined based on the second prediction probability and the second connectionist temporal classifier.

14. The method according to claim 1, characterized in that The method further includes training the target gesture recognition model based on: In response to acquiring a sample video, extracting sample skeleton points of a person in each frame of the sample video, wherein the sample skeleton points include skeleton points of a preset human body part of the person, and the preset human body part includes at least a hand; Dividing the sample skeleton points into multiple groups, each group of sample skeleton points belongs to the same preset human body part; Performing centralization processing on each group of sample skeleton points respectively to obtain centralized groups of sample skeleton points; The pre-constructed model to be trained is trained based on each group of centralized sample skeleton points to obtain the target posture recognition model.

15. A gesture recognition device, characterized in that: The device comprises: A skeleton point extraction module, configured to extract target skeleton points of a person in each frame of the target video in response to receiving the target video to be identified, wherein the target skeleton points include skeleton points of preset human body parts of the person, and the preset human body parts include at least hands; A skeleton point division module, used for dividing the target skeleton points into multiple groups, each group of skeleton points corresponding to the same preset human body part; A centralization processing module, used to perform centralization processing on each group of skeleton points to obtain multiple groups of centralized skeleton points; The recognition result acquisition module is used to obtain the target recognition result corresponding to the target video based on the centralized multiple groups of skeleton points and the pre-trained target posture recognition model.

16. An electronic device, characterized in that: The device comprises: a processor and a memory for storing a computer program; Wherein, the processor is configured to implement, when executing the computer program: In response to receiving a target video to be identified, extracting target skeleton points of a person in each frame of the target video, wherein the target skeleton points include skeleton points of preset human body parts of the person, and the preset human body parts include at least a hand; Dividing the target skeleton points into multiple groups, each group of skeleton points corresponds to the same preset human body part; Performing centralization processing on each group of skeleton points respectively to obtain multiple groups of centralized skeleton points; Based on the centralized multiple groups of skeleton points and the pre-trained target posture recognition model, a target recognition result corresponding to the target video is obtained.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, it realizes: In response to receiving a target video to be identified, extracting target skeleton points of a person in each frame of the target video, wherein the target skeleton points include skeleton points of preset human body parts of the person, and the preset human body parts include at least a hand; Dividing the target skeleton points into multiple groups, each group of skeleton points corresponds to the same preset human body part; Performing centralization processing on each group of skeleton points respectively to obtain multiple groups of centralized skeleton points; Based on the centralized multiple groups of skeleton points and the pre-trained target posture recognition model, a target recognition result corresponding to the target video is obtained.

Images