Posture recognition method and device, electronic equipment and storage medium

Abstract

The invention provides a posture recognition method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-recognized video clip, wherein each image frame of the video clip comprises a hand of a target object; performing feature recognition processing on each image frame to obtain a hand feature map of each image frame; the hand feature map comprises motion space features and motion time sequence features; the motion space feature comprises a hand shape information feature of a hand in the image frame; the motion time sequence features comprise hand motion features of the hand in multiple image frames; and performing classification prediction processing according to the hand feature map of each image frame to obtain sign language text information corresponding to the information transmitted by the hand action in the video clip. In this way, according to the embodiment, the hand shape information and the hand motion information of the target object are concerned, and the recognition ability of the hand posture is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a gesture recognition method and device, an electronic device, and a storage medium. Background Art

[0002] Sign language recognition is the use of computer vision technology to sequentially identify the sign language words that appear in sign language videos, providing a convenient communication tool for deaf and mute people and healthy people.

[0003] In actual applications, the solutions of related technologies may not be robust enough in paying attention to the background. When a non-specific user uses sign language to communicate in the video, the model's ability to recognize the sign language of the non-specific user is reduced, reducing the user experience. Summary of the invention

[0004] The present disclosure provides a gesture recognition method and device, an electronic device, and a storage medium to solve the above technical problems.

[0005] According to a first aspect of the present disclosure, there is provided a gesture recognition method, comprising:

[0006] Acquire a video clip to be identified, wherein each image frame of the video clip includes hand information of a target object;

[0007] Performing feature recognition processing on each of the image frames to obtain a hand feature map of each of the image frames; the hand feature map includes enhanced motion space features and motion time sequence features; the motion space features include hand shape information features of the hand in the image frame; the motion time sequence features include hand motion features of the hand in multiple image frames;

[0008] Classification prediction processing is performed based on the hand feature maps of each image frame to obtain sign language text information corresponding to the information conveyed by the hand movements in the video clip.

[0009] Optionally, performing feature recognition processing on each image frame to obtain a hand feature map of each image frame includes:

[0010] Inputting each of the image frames into a preset image feature recognition model in sequence to obtain an image feature map of each of the image frames;

[0011] The image feature map of each image frame is input into a preset gesture feature recognition model to obtain a hand feature map of each image frame.

[0012] Optionally, the gesture feature recognition model obtains a hand feature map of each image frame through the following steps, including:

[0013] Extracting motion space features of the image feature map in the first sliding window corresponding to each image frame;

[0014] Extracting motion timing features in the image feature map within the second sliding window corresponding to each image frame;

[0015] Temporal modeling is performed according to the image feature map of each image frame, the motion space feature and the motion temporal feature to obtain a hand feature map of each image frame.

[0016] Optionally, extracting motion space features of an image feature map within a first sliding window corresponding to each image frame includes:

[0017] Performing dimensionality reduction processing on the image feature map of each image frame to obtain a dimensionality reduction feature map;

[0018] Superimposing a first preset number of image frames in a first sliding window corresponding to each image frame and calculating the average value of each pixel to obtain a first average feature map of each image frame; the first preset number is less than or equal to a first number threshold;

[0019] The difference value of the pixel at the same position in the dimension reduction feature map of each image frame and the first average feature map is obtained to obtain the motion space feature of each image frame.

[0020] Optionally, extracting the motion timing features in the image feature map within the second sliding window corresponding to each image frame includes:

[0021] Performing dimensionality reduction processing on the image feature map of each image frame to obtain a dimensionality reduction feature map;

[0022] Superimposing a second preset number of image frames in a second sliding window corresponding to each image frame and calculating the average value of each pixel to obtain a second average feature map of each image frame; the second preset number is greater than or equal to a second number threshold; the second number threshold is greater than the first number threshold;

[0023] The difference value of the pixel at the same position in the dimension reduction feature map of each image frame and the second average feature map is obtained to obtain the motion timing characteristics of each image frame.

[0024] Optionally, performing time series modeling according to the image feature graph of each image frame, the motion space feature and the motion time series feature to obtain the hand feature graph of each image frame includes:

[0025] Obtaining a product of the motion space feature and a first weight value to obtain a weighted motion space feature;

[0026] Obtaining the product of the motion time series feature and the second weight value to obtain a weighted motion time series feature;

[0027] Obtaining the sum of the pixels at the same position of the motion spatial feature and the motion temporal feature to obtain a motion-static fusion feature;

[0028] Performing dimension-upgrading processing on the dynamic and static fusion features to obtain dimension-upgrading features;

[0029] Obtaining the product of the dimension-raising feature and the third weight value to obtain a weighted dimension-raising feature;

[0030] Obtaining a product of the image feature map and a fourth weight value to obtain a weighted image feature map;

[0031] The sum of the weighted dimensionality-raising feature and the pixel at the same position in the weighted image feature map is obtained to obtain a hand feature map.

[0032] Optionally, the dimension-raising unit for obtaining dimension-raising features in the gesture feature recognition model includes at least one time domain convolution layer; the time domain convolution layer includes a one-dimensional CNN convolution layer and a pooling layer; the at least one time domain convolution layer is used to perform time series modeling on the dynamic and static fusion features to obtain dimension-raising features including hand movement features.

[0033] Optionally, the image feature recognition model includes at least one of the following: a ResNet model, a VGGNet model, and a GoogLeNet model.

[0034] Optionally, the training step of the gesture feature recognition model includes:

[0035] Obtaining each image frame of each video clip in a preset training set; the preset training set includes a plurality of video clips, each video clip includes sign language content and a label sequence;

[0036] Inputting each of the image frames into a preset image feature recognition model in sequence to obtain an image feature map of each of the image frames;

[0037] In response to the loss value being less than a preset loss threshold, inputting the image feature map of each image frame into a gesture feature recognition model to obtain a hand feature map of each image frame;

[0038] Performing classification prediction processing according to the hand feature graphs of each image frame to obtain sign language text information corresponding to the information conveyed by the hand movements in the video clip;

[0039] Calculating a loss value of a loss function according to the sign language text information and the label sequence of each video clip;

[0040] In response to the loss value being greater than or equal to the preset loss threshold, training the gesture feature recognition model is stopped to obtain the preset gesture feature recognition model.

[0041] Optionally, the loss function is used to calculate the probability that the label sequence corresponding to the gesture text information appears in the label sequence corresponding to the video clip when the model parameters are determined.

[0042] Optionally, performing classification prediction processing according to the hand feature map of each image frame to obtain sign language text information corresponding to the information conveyed by the hand movement in the video clip includes:

[0043] Inputting the hand feature images of each image frame into a preset context information processing model in sequence to obtain context information of each image frame;

[0044] Inputting the context information of each image frame into a preset information classification model in sequence to obtain a classification prediction probability of the context information;

[0045] Decoding is performed according to the classification prediction probability of the context information of each image frame to obtain sign language text information corresponding to the information conveyed by the hand movement in the video clip.

[0046] Optionally, the context information processing model includes at least one of the following models: a CNN model, a Transformer model, and a BiLSTM model.

[0047] Optionally, the training process of the context information processing model and the information classification model is synchronized with the training process of the gesture feature recognition model, and the training of the context information processing model is terminated when the training of the gesture feature recognition model is completed.

[0048] According to a second aspect of the present disclosure, there is provided a gesture recognition device, comprising:

[0049] A video segment acquisition module, used to acquire a video segment to be identified, each image frame of the video segment includes hand information of the target object;

[0050] A hand feature map acquisition module is used to perform feature recognition processing on each image frame to obtain a hand feature map of each image frame; the hand feature map includes enhanced motion space features and motion time sequence features; the motion space features include hand shape information features of the hand in the image frame; the motion time sequence features include hand motion features of the hand in multiple image frames;

[0051] The sign language text information acquisition module is used to perform classification prediction processing based on the hand feature maps of each image frame to obtain the sign language text information corresponding to the information conveyed by the hand movements in the video clip.

[0052] Optionally, the hand feature map acquisition module includes:

[0053] An image feature map acquisition submodule, used to input each image frame into a preset image feature recognition model in sequence to obtain an image feature map of each image frame;

[0054] The hand feature map acquisition submodule is used to input the image feature map of each image frame into a preset gesture feature recognition model to obtain the hand feature map of each image frame.

[0055] Optionally, the gesture feature recognition model includes a motion space feature extraction module, a motion time sequence feature extraction module and a feature enhancement module;

[0056] The motion space feature extraction module is used to extract the motion space features of the image feature map in the first sliding window corresponding to each image frame;

[0057] The motion timing feature extraction module is used to extract the motion timing features in the image feature map within the second sliding window corresponding to each image frame;

[0058] The feature enhancement module is used to perform time series modeling based on the image feature map of each image frame, the motion space feature and the motion time series feature to obtain the hand feature map of each image frame.

[0059] Optionally, the motion space feature extraction module includes a dimension reduction unit, a first superposition unit and a first difference unit;

[0060] The dimension reduction unit is used to perform dimension reduction processing on the image feature map of each image frame to obtain a reduced dimension feature map;

[0061] The first superposition unit is used to superimpose a first preset number of image frames in the first sliding window corresponding to each image frame and calculate the average value of each pixel to obtain a first average feature map of each image frame; the first preset number is less than or equal to a first number threshold;

[0062] The first difference unit is used to obtain the difference value of the pixel at the same position in the reduced dimension feature map of each image frame and the first average feature map, so as to obtain the motion space feature of each image frame.

[0063] Optionally, the motion time series feature extraction module includes a dimension reduction unit, a second superposition unit and a second difference unit;

[0064] The dimension reduction unit is used to perform dimension reduction processing on the image feature map of each image frame to obtain a reduced dimension feature map;

[0065] The second superposition unit is used to superimpose a second preset number of image frames in the second sliding window corresponding to each image frame and calculate the average value of each pixel to obtain a second average feature map of each image frame; the second preset number is greater than or equal to a second number threshold; the second number threshold is greater than the first number threshold;

[0066] The second difference unit is used to obtain the difference value of the pixel at the same position in the reduced dimension feature map of each image frame and the second average feature map, so as to obtain the motion timing characteristics of each image frame.

[0067] Optionally, the feature enhancement module includes a first weighting unit, a second weighting unit, a first fusion unit, a dimension-increasing unit, a third weighting unit, a fourth weighting unit, and a second fusion unit;

[0068] The first weighting unit is used to obtain the product of the motion space feature and the first weight value to obtain the weighted motion space feature;

[0069] The second weighting unit is used to obtain the product of the motion time series feature and the second weight value to obtain the weighted motion time series feature;

[0070] The first fusion unit is used to obtain the sum of the pixels at the same position of the motion spatial feature and the motion temporal feature to obtain a motion-static fusion feature;

[0071] The dimension-increasing unit is used to perform dimension-increasing processing on the dynamic and static fusion features to obtain dimension-increasing features;

[0072] The third weighting unit is used to obtain the product of the dimension-raising feature and the third weight value to obtain the weighted dimension-raising feature;

[0073] The fourth weighting unit is used to obtain the product of the image feature map and a fourth weight value to obtain a weighted image feature map;

[0074] The second fusion unit is used to obtain the sum of the weighted dimensionality-upgraded feature and the pixel at the same position in the weighted image feature map to obtain a hand feature map.

[0075] Optionally, the dimension-raising unit for obtaining dimension-raising features in the gesture feature recognition model includes at least one time domain convolution layer; the time domain convolution layer includes a one-dimensional CNN convolution layer and a pooling layer; the at least one time domain convolution layer is used to perform time series modeling on the dynamic and static fusion features to obtain dimension-raising features including hand movement features.

[0076] Optionally, the image feature recognition model includes at least one of the following: a ResNet model, a VGGNet model, and a GoogLeNet model.

[0077] Optionally, a model training module is further included, and the model training module is used to train the gesture feature recognition model, including:

[0078] A video clip acquisition submodule, used to acquire each image frame of each video clip in a preset training set; the preset training set includes a plurality of video clips, each video clip includes sign language content and a label sequence;

[0079] An image feature map acquisition submodule, used to input each image frame into a preset image feature recognition model in sequence to obtain an image feature map of each image frame;

[0080] A hand feature map acquisition submodule, configured to input the image feature map of each image frame into a preset gesture feature recognition model to obtain the hand feature map of each image frame in response to the loss value being less than a preset loss threshold;

[0081] A sign language text information acquisition submodule, used to perform classification prediction processing according to the hand feature map of each image frame to obtain the sign language text information corresponding to the information conveyed by the hand movement in the video clip;

[0082] A loss value acquisition submodule, used to calculate the loss value of the loss function according to the sign language text information and the label sequence of each video clip;

[0083] The recognition model acquisition submodule is used to stop training the gesture feature recognition model in response to the loss value being greater than or equal to the preset loss threshold, and obtain the preset gesture feature recognition model.

[0084] Optionally, the loss function is used to calculate the probability that the label sequence corresponding to the gesture text information appears in the label sequence corresponding to the video clip when the model parameters are determined.

[0085] Optionally, the sign language text information acquisition module includes:

[0086] A context information acquisition submodule, used to sequentially input the hand feature images of each image frame into a preset context information processing model to obtain the context information of each image frame;

[0087] A prediction probability acquisition submodule, used to input the context information of each image frame into a preset information classification model in sequence to obtain the classification prediction probability of the context information;

[0088] The sign language text information acquisition submodule is used to decode according to the classification prediction probability of the context information of each image frame to obtain the sign language text information corresponding to the information conveyed by the hand movement in the video clip.

[0089] Optionally, the context information processing model includes at least one of the following models: a CNN model, a Transformer model, and a BiLSTM model.

[0090] Optionally, the training process of the context information processing model and the information classification model is synchronized with the training process of the gesture feature recognition model, and the training of the context information processing model is terminated when the training of the gesture feature recognition model is completed.

[0091] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0092] Processor and memory;

[0093] The memory is used to store a computer program executable by the processor;

[0094] The processor is configured to execute the computer program in the memory to implement the method as described in any one of the first aspects.

[0095] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, and when an executable computer program in the storage medium is executed by a processor, the method described in any one of the first aspects can be implemented.

[0096] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0097] The solution of this embodiment can obtain a video clip to be identified, and each image frame of the video clip includes the hand information of the target object; then, feature recognition processing is performed on each image frame to obtain a hand feature map of each image frame; the hand feature map includes enhanced motion space features and motion timing features; the motion space features include hand shape information features of the hand in the image frame; the motion timing features include hand motion features of the hand in multiple image frames; then, classification prediction processing is performed based on the hand feature maps of each image frame to obtain the sign language text information corresponding to the information conveyed by the hand action in the video clip. In this way, this embodiment focuses on the hand shape information and hand motion information of the target object in the video clip, and improves the recognition ability of hand postures.

[0098] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] Figure 1 The present invention is a flowchart of a gesture recognition method according to an embodiment of the present invention.

[0100] Figure 2The present invention is a flowchart of obtaining a hand feature map according to an embodiment of the present invention.

[0101] Figure 3 A block diagram of a gesture recognition model according to an embodiment of the present disclosure.

[0102] Figure 4 A block diagram of another gesture recognition model according to an embodiment of the present disclosure.

[0103] Figure 5 The figure is a schematic diagram of the effect of a motion space feature according to an embodiment of the present disclosure.

[0104] Figure 6 The figure is a schematic diagram of the effect of a motion timing feature according to an embodiment of the present disclosure.

[0105] Figure 7 The present invention is a flowchart of a training gesture feature recognition model according to an embodiment of the present invention.

[0106] Figure 8 A block diagram of a classification prediction module according to an embodiment of the present disclosure.

[0107] Fig. 9 The present invention is a block diagram of a gesture recognition device according to an embodiment of the present invention.

[0108] Fig.10 The present invention is a block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0109] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices consistent with some aspects of the present disclosure as detailed in the appended claims.

[0110] The embodiments of the present disclosure provide a gesture recognition method and apparatus, an electronic device, and a storage medium. The gesture recognition method can be applied to electronic devices. The electronic devices can include but are not limited to smart phones, tablet computers, servers, and other devices. Figure 1 , a posture recognition method includes steps 11 to 13.

[0111] In step 11, a video segment to be identified is obtained, wherein each image frame of the video segment includes hand information of the target object.

[0112] In this step, the processor of the electronic device can obtain the video clip to be identified. The video clip can be a part of the video captured in real time by the camera of the electronic device, and the camera sends the captured video to the processor; the processor segments the above video to obtain the video clip. Alternatively, the video is stored in a designated location (such as a cache or a local memory) of the electronic device, and the processor reads the video from the designated location and segments it to obtain the video clip.

[0113] It is understandable that each image frame of the above video clip includes hand information of the target object, such as the position of the hand in the image frame, the size of the hand, the posture of the hand, etc.

[0114] In step 12, feature recognition processing is performed on each of the image frames to obtain a hand feature map of each of the image frames; the hand feature map includes enhanced motion space features and motion time series features.

[0115] In this step, the processor of the electronic device can perform feature recognition processing on each image frame to obtain a hand feature map of each image frame, see Figure 2 , including step 21 and step 22.

[0116] In step 21, each image frame is sequentially input into a preset image feature recognition model to obtain an image feature map of each image frame.

[0117] In this step, the electronic device may store a preset image feature recognition model. The image feature recognition model includes at least one of the following: a ResNet model, a VGGNet model, and a GoogLeNet model. The technician may select a suitable image feature recognition model according to the specific scenario. If the features of each image frame can be determined, the corresponding solution falls within the protection scope of the present disclosure.

[0118] In this step, the electronic device may input each image frame into a preset image feature recognition model in sequence, and obtain a feature map output by the image feature recognition model as an image feature map of each image frame.

[0119] In step 22, the image feature map of each image frame is input into a preset gesture feature recognition model to obtain a hand feature map of each image frame.

[0120] In this step, the electronic device can store a preset gesture feature recognition model. The gesture feature recognition model can obtain the hand feature map of each image frame through the following steps, including: extracting the motion space features of the image feature map in the first sliding window corresponding to each image frame; the motion space features are used to strengthen the motion space hand shape information in each image frame; extracting the motion timing features in the image feature map in the second sliding window corresponding to each image frame; the motion timing features are used to strengthen the hand movement features of the hand within the corresponding time length of the second sliding window; performing timing modeling based on the image feature map of each image frame, the motion space features and the motion timing features to obtain the hand feature map of each image frame.

[0121] It should be noted that the first sliding window refers to the time length occupied by a first preset number of image frames obtained by selecting image frames before and after with each image frame as the intermediate frame; the second sliding window refers to the time length occupied by a second preset number of image frames obtained by selecting image frames before and after with each image frame as the intermediate frame; the second number is greater than the first number, that is, the length of the first sliding window is less than the length of the second sliding window.

[0122] See also Figure 3 The gesture recognition model includes a motion space feature extraction module 31, a motion time sequence feature extraction module 32 and a feature enhancement module 33.

[0123] A motion space feature extraction module 31 is used to extract the motion space features of the image feature map in the first sliding window corresponding to each image frame;

[0124] A motion timing feature extraction module 32, used to extract the motion timing features in the image feature map within the second sliding window corresponding to each image frame;

[0125] The feature enhancement module 33 is used to perform time series modeling according to the image feature map of each image frame, the motion space feature and the motion time series feature to obtain the hand feature map of each image frame.

[0126] In this way, this embodiment can guide subsequent models to focus on the motion space hand shape information and hand movement features in the image feature map by strengthening the motion space hand shape information and hand movement features, thereby capturing the effect of non-specified people in the image frame.

[0127] Based on the above analysis, this step can design a gesture feature recognition model with the following ideas, including:

[0128] First, assume that the image feature map output by the image feature extraction module 11 is t represents the acquisition time t of the image frame, which can be understood as the serial number of each image frame. The image feature maps of the w-1 frames before it form a set And do temporal difference to capture short-term motion salient information, that is, subsequent motion spatial features Wherein, w represents the number of image frames in the first sliding window, that is, the first preset number.

[0129] In order to reduce the number of parameters, the image feature map of each image frame can be reduced, such as 1*1 convolution dimension reduction, to obtain a reduced dimension feature map, as shown in formula (1).

[0130]

[0131] In formula (1), Conv down () indicates that the object in the brackets is subjected to dimensionality reduction convolution operation. represents the image feature map, Represents the dimension-reduced feature map.

[0132] Based on formula (1), each image frame in the image feature map set is superimposed and the difference is calculated to obtain the motion space feature of each image frame As shown in formula (2).

[0133]

[0134] In formula (2), i represents the feature map of the i-th frame, It means calculating the sum of pixels at the same position in the feature map of w frames.

[0135] Considering that the movement around the hand is relatively slow, the short-term first sliding window will lose a lot of spatial information. Therefore, this embodiment introduces a second sliding window to capture the long-term motion salient information, that is, the motion time series features. As shown in formula (3).

[0136]

[0137] In formula (3), T represents the second preset number. In one example, T is the length of the video segment.

[0138] Among them, the motion timing characteristics It can be understood as the motion space feature Stretch from length w to the length T of the entire video segment.

[0139] Then, the motion space features and motion timing characteristics Weighted and summed values ​​are obtained to obtain the dynamic and static fusion features of the video clips As shown in formula (4).

[0140]

[0141] In formula (4), β represents the second weight value, which is used to represent the proportion of motion time series features in the dynamic and static fusion features, and its value range is [0, 1]. In one example, β is 0.5, which means that both motion spatial features and motion time series features (i.e., time series features) are captured.

[0142] After obtaining the dynamic and static fusion features Afterwards, you can Perform dimensionality increase processing to obtain the dimensionality increase feature map M t , as shown in formula (5).

[0143]

[0144] In formula (5), Conv up () indicates that the object in the brackets is upgraded in dimension, such as 1*1 convolution.

[0145] Afterwards, the dynamic and static fusion features M t Fusion with the image feature map to obtain the hand feature map As shown in formula (6).

[0146]

[0147] In formula (6), α represents the relative weight of controlling the temporal characteristics and spatial information of the motion, and its value range is [0, 1]. In one example, α is 0.7, and the fourth weight value is used in the following.

[0148] Based on the above analysis, a motion space feature extraction module 31, a motion time series feature extraction module 32 and a feature enhancement module 33 are obtained respectively.

[0149] In this embodiment, the motion space feature extraction module 31 extracts the motion space features of the image feature map within the first sliding window corresponding to each image frame, including: performing dimensionality reduction processing on the image feature map of each image frame to obtain a reduced dimensionality feature map; superimposing a first preset number of image frames within the first sliding window corresponding to each image frame and calculating the average value of each pixel to obtain a first average feature map of each image frame; the first preset number is less than or equal to a first quantity threshold; obtaining the difference value of the pixel at the same position in the reduced dimensionality feature map of each image frame and the first average feature map to obtain the motion space features of each image frame.

[0150] See also Figure 4 The motion space feature extraction module 31 includes a dimension reduction unit 41, a first superposition unit 42 and a first difference unit 43; wherein,

[0151] A dimension reduction unit 41 is used to perform dimension reduction processing on the image feature map of each image frame to obtain a reduced dimension feature map, which can reduce the amount of subsequent data processing;

[0152] The first superposition unit 42 is used to superimpose the first preset number of image frames in the first sliding window corresponding to each image frame and calculate the average value of each pixel to obtain the first average feature map of each image frame; the first preset number is less than or equal to the first number threshold; wherein the first number threshold ranges from 2 to 10 and can be selected according to the specific scene. Since the motion timing characteristics will change within the first preset number of frames, their average value will become smaller and the average value of the motion space characteristics in the first preset number of image frames will become relatively larger, so as to achieve the effect of strengthening the motion space characteristics and weakening the motion timing characteristics;

[0153] The first difference unit 43 is used to obtain the difference value of the pixel at the same position in the dimension reduction feature map of each image frame and the first average feature map to obtain the motion space feature of each image frame. In this way, the first difference unit 43 obtains the motion timing characteristics of each image frame by difference, so as to strengthen the motion timing characteristics and weaken the motion space characteristics, which is conducive to guiding the subsequent model to focus on the motion timing characteristics. Figure 5 As shown, Figure 5 The information of the moving hand within the first sliding window is illustrated.

[0154] In one embodiment, the motion timing feature extraction module 32 extracts the motion timing features in the image feature map within the second sliding window corresponding to each image frame, including: performing dimensionality reduction processing on the image feature map of each image frame to obtain a reduced dimensionality feature map; superimposing a second preset number of image frames within the second sliding window corresponding to each image frame and calculating the average value of each pixel to obtain a second average feature map of each image frame; the second preset number is greater than or equal to a second quantity threshold; the second quantity threshold is greater than the first quantity threshold; obtaining the difference value of the pixel at the same position in the reduced dimensionality feature map of each image frame and the second average feature map to obtain the motion timing features of each image frame.

[0155] Continue to see Figure 4 , the motion time series feature extraction module 32 includes a dimension reduction unit 41, a second superposition unit 44 and a second difference unit 45;

[0156] A dimension reduction unit 41 is used to perform dimension reduction processing on the image feature map of each image frame to obtain a reduced dimension feature map, which can reduce the amount of subsequent data processing;

[0157] The second superposition unit 44 is used to superimpose a second preset number of image frames in the second sliding window corresponding to each image frame and calculate the average value of each pixel to obtain a second average feature map of each image frame; the second preset number is greater than or equal to the second number threshold; the second number threshold is greater than the first number threshold. In one example, the value of the second preset number is the number of image frames in the video clip. By lengthening the length of the sliding window, the second superposition unit 44 can relatively strengthen the motion spatial feature by calculating the average value of the slowly changing motion spatial feature, and relatively weaken the motion timing feature by calculating the average value of the rapidly changing motion timing feature.

[0158] The second difference unit 45 is used to obtain the difference value of the pixel at the same position in the dimension reduction feature map of each image frame and the second average feature map to obtain the motion time series feature of each image frame. Effects such as Figure 6 See Figure 6 , Figure 6 The target object body and hands in the motion space state are illustrated. In this way, after obtaining the motion time sequence feature, the motion space information in the second preset number of image frames can be obtained, thereby strengthening the motion space information of the hand in the video clip.

[0159] In one embodiment, the feature enhancement module 33 performs temporal modeling according to the image feature map of each image frame, the motion space feature and the motion timing feature to obtain a hand feature map of each image frame, including: obtaining the product of the motion space feature and the first weight value to obtain a weighted motion space feature; obtaining the product of the motion timing feature and the second weight value to obtain a weighted motion timing feature; obtaining the sum of the pixels at the same position of the motion space feature and the motion timing feature to obtain a dynamic-static fusion feature; performing dimensionality upscaling processing on the dynamic-static fusion feature to obtain an upscaling feature; obtaining the product of the upscaling feature and the third weight value to obtain a weighted upscaling feature; obtaining the product of the image feature map and the fourth weight value to obtain a weighted image feature map; obtaining the sum of the weighted upscaling feature and the pixels at the same position in the weighted image feature map to obtain a hand feature map.

[0160] Continue to see Figure 4 , the feature enhancement module includes a first weighting unit 46, a second weighting unit 47, a first fusion unit 48, a dimension-increasing unit 49, a third weighting unit 50, a fourth weighting unit 51 and a second fusion unit 52;

[0161] A first weighting unit 46, configured to obtain a product of the motion space feature and a first weight value to obtain a weighted motion space feature;

[0162] The second weighting unit 47 is used to obtain the product of the motion timing feature and the second weight value to obtain the weighted motion timing feature; the sum of the first weight value 1-β and the second weight value β is equal to 1, so that the proportion of the motion space feature and the motion timing feature can be adjusted.

[0163] A first fusion unit 48 is used to obtain the sum of the pixels at the same position of the motion spatial feature and the motion temporal feature to obtain a motion-static fusion feature;

[0164] A dimension-increasing unit 49 is used to perform dimension-increasing processing on the dynamic-static fusion feature to obtain the dimension-increasing feature, so that the dimension-increasing feature matches the size of the image frame; the dimension-increasing unit may include at least one time-domain convolution layer; the time-domain convolution layer includes a one-dimensional CNN convolution layer and a pooling layer; the at least one time-domain convolution layer is used to perform time-series modeling on the dynamic-static fusion feature to obtain the dimension-increasing feature including the hand motion feature;

[0165] A third weighting unit 50 is used to obtain the product of the dimension-raising feature and a third weight value to obtain a weighted dimension-raising feature;

[0166] The fourth weighting unit 51 is used to obtain the product of the image feature map and the fourth weight value to obtain a weighted image feature map; the sum of the third weight value 1-α and the fourth weight value α is equal to 1, so as to adjust the proportion of the weighted up-dimensional features in the subsequent hand feature map and the features in the weighted image feature map, and further adjust the proportion of the motion space features and the motion timing features, so that the subsequent model can pay attention to the motion space information and the motion timing information at the same time;

[0167] The second fusion unit 52 is used to obtain the sum of the weighted up-dimensional feature and the pixel at the same position in the weighted image feature map to obtain a hand feature map.

[0168] In one embodiment, the training steps of the gesture feature recognition model are as follows: Figure 7 , including steps 71 to 76.

[0169] In step 71, each image frame of each video segment in a preset training set is obtained; the preset training set includes a plurality of video segments, and each video segment includes sign language content and a label sequence.

[0170] In this step, the electronic device can obtain a preset training set, which may include an open source training set, such as the CSL-Daily data set, which includes 18,000 training video clips, or can record video clips of the target object using sign language to communicate according to specific scenes to form a preset training set; or add the self-recorded video clips to the above open source training set to form the above preset training set. It is understandable that each video clip includes the sign language content of the target object's sign language action and a label sequence for the sign language, and the label sequence is the correct sign language text information. In this way, the electronic device can obtain each image frame of each video clip.

[0171] In step 72, each of the image frames is sequentially input into a preset image feature recognition model to obtain an image feature map of each of the image frames.

[0172] In this step, a preset image feature recognition model, such as a CNN model, is stored in the electronic device. At this time, the electronic device can sequentially input each image frame into the above-mentioned image feature recognition model to obtain an image feature map of each image frame. The content of obtaining the image feature map can refer to the content of step 21, which will not be repeated here.

[0173] In step 73, in response to the loss value being less than a preset loss threshold, the image feature map of each image frame is input into a gesture feature recognition model to obtain a hand feature map of each image frame.

[0174] In this step, the electronic device can compare the loss value of the loss function during this training with the preset loss threshold. The loss value calculation of the loss function can refer to the content of step 75, which is not repeated here. Among them, the range of the preset loss threshold can be 90% to 99%, which can be set according to the specific scenario and is not limited here.

[0175] In this step, when the loss value is less than a preset loss threshold, the electronic device may input the image feature map of each image frame into the gesture feature recognition model to obtain the hand feature map of each image frame output by the gesture feature recognition model.

[0176] In step 74, classification prediction processing is performed based on the hand feature maps of each image frame to obtain sign language text information corresponding to the information conveyed by the hand movements in the video clip.

[0177] In this step, the electronic device can perform classification prediction processing based on the hand feature maps of each image frame to obtain sign language text information corresponding to the information conveyed by the hand movements in the video clip. For details, please refer to the content of step 13, which will not be elaborated here.

[0178] In step 75, a loss value of a loss function is calculated based on the sign language text information and the label sequence of each video clip.

[0179] In this step, the electronic device stores a loss function, which is used to calculate the probability of the label sequence corresponding to the gesture text information appearing in the label sequence corresponding to the video clip when the model parameters are determined. In one example, the loss function can be implemented using the function shown in formula (1).

[0180] L CTC (w,X)=-lnp(wX;θ); (7)

[0181] In formula (7), X represents the label sequence corresponding to the input image frame, w represents the sign language text information corresponding to the image frame, and θ represents the model parameters. The model parameters can be considered fixed in each training.

[0182] In this step, the electronic device can obtain the label sequence corresponding to each image frame and the sign language text information of the preset result, and obtain the loss value in combination with formula (1).

[0183] In step 76, in response to the loss value being greater than or equal to the preset loss threshold, the training of the gesture feature recognition model is stopped to obtain the preset gesture feature recognition model.

[0184] In this step, the electronic device may compare the loss value of the loss function with a preset loss threshold. When the loss value is greater than or equal to the preset loss threshold, the electronic device may stop training the gesture feature recognition model to obtain the preset gesture feature recognition model.

[0185] In step 22, the electronic device may input the image feature map of each image frame into a gesture feature recognition model to obtain the hand feature map of each image frame.

[0186] In one embodiment, the electronic device uses the Phoenix14 data set to verify the solution of the present disclosure, and obtains the experimental results shown in Table 1.

[0187] Referring to Table 1, the experiment was conducted according to Phoenix14-SI5 (non-specific person setting), and Phoenix14Sign01 was used to verify the performance in the specific person setting. Compared with the baseline model, the motion space feature extraction module 31 of this solution brings 0.1% performance improvement in the specific person setting, and the motion time series feature extraction module 32 brings 1.4% performance improvement. When both are used at the same time, the effect is general; in the non-specific person setting, the motion space feature extraction module 31 brings 2.5% performance improvement, and the motion time series feature extraction module 32 brings 3.9% performance improvement. In other words, the scheme disclosed in this disclosure can reduce the influence of factors irrelevant to gesture recognition such as the background and the characteristics of the sign language user himself, and enhance the generalization of the model. In addition, the technical solution has a more obvious improvement on the 1D-CNN model in the non-specific person setting. The motion space feature extraction module 31 brings a 3.0% performance improvement, and the motion time series feature extraction module 32 brings a 3.7% performance improvement. In addition, the CNN model has better real-time performance, which makes the scheme disclosed in this disclosure have better practical applications.

[0188] Table 1 Experimental results

[0189]

[0190] In step 13, classification prediction processing is performed based on the hand feature maps of each image frame to obtain sign language text information corresponding to the information conveyed by the hand movements in the video clip.

[0191] In this step, the electronic device can input the hand feature images of each image frame into a preset context information processing model in sequence to obtain the context information of each image frame; then, input the context information of each image frame into a preset information classification model in sequence to obtain the classification prediction probability of the context information; thereafter, decode according to the classification prediction probability of the context information of each image frame to obtain the sign language text information corresponding to the information conveyed by the hand movement in the video clip.

[0192] In one example, the electronic device stores a classification prediction module. Figure 8 The classification prediction module includes a context information processing model 131, an information classification model 132 and a decoder 133.

[0193] In one example, the context information processing model 131 includes at least one of the following models: a CNN model, a Transformer model, and a BiLSTM model. In this example, the CNN model may include multiple time-domain convolutional layers for modeling short-term temporal information, which is suitable for recognition scenarios with low requirements for gesture recognition delay, such as offline sign language translation scenarios, sign language learning scenarios, etc. The Transformer model can model all the information contained in the preamble, is suitable for scenarios where sentences are used as translation units, and can improve recognition accuracy. The BiLSTM model can model bidirectional long-term temporal information, is suitable for scenarios where sentences are used as translation units, and can improve recognition accuracy. It is understandable that as the receptive field in each model increases, its output delay gradually increases, and the recognition accuracy will increase accordingly. Technicians can select a suitable model according to the specific scenario, and the corresponding scheme falls within the scope of protection of this disclosure.

[0194] It should be noted that the training process of the above-mentioned context information processing model 131 is similar to Figure 4 The training process of the gesture feature recognition model shown can be performed synchronously, and the training of the context information processing model and the information classification model is completed when the gesture feature recognition model is trained.

[0195] In one example, the information classification model 132 may include a softmax classification algorithm implementation. In one example, the decoder 133 may include a greedy decoding algorithm implementation. The technician may select the corresponding information classification model 132 and decoder 133 according to the specific scenario, and the corresponding solution falls within the protection scope of the present disclosure.

[0196] In this step, the context information processing model 131 can perform context information processing on the hand feature images of each input image frame to obtain the context information of each image frame. The information classification model 132 can classify the context information of each image frame to obtain the classification prediction probability of the context information of each image frame. The decoder 133 can decode according to the preset classification probability of the context information to obtain the sign language text information corresponding to the information conveyed by the hand movement in the video clip.

[0197] At this point, the scheme of this embodiment can obtain the video clip to be identified, and each image frame of the video clip includes the hand information of the target object; then, feature recognition processing is performed on each image frame to obtain the hand feature map of each image frame; the hand feature map includes enhanced motion space features and motion timing features; then, classification prediction processing is performed based on the hand feature maps of each image frame to obtain the sign language text information corresponding to the information conveyed by the hand movement in the video clip. In this way, this embodiment performs enhanced processing on the hand motion space features and short-term motion features, so as to pay attention to the hand motion information and hand shape information, and improve the recognition ability of hand posture.

[0198] The present disclosure provides a gesture recognition method, and the present disclosure also provides a gesture recognition device. Fig. 9 , the device comprises:

[0199] A video segment acquisition module 91 is used to acquire a video segment to be identified, wherein each image frame of the video segment includes hand information of a target object;

[0200] The hand feature map acquisition module 92 is used to perform feature recognition processing on each image frame to obtain a hand feature map of each image frame; the hand feature map includes enhanced motion space features and motion time series features; the motion space features include hand shape information features of the hand in the image frame; the motion time series features include hand motion features of the hand in multiple image frames;

[0201] The sign language text information acquisition module 93 is used to perform classification prediction processing based on the hand feature map of each image frame to obtain the sign language text information corresponding to the information conveyed by the hand movement in the video clip.

[0202] Optionally, the hand feature map acquisition module includes:

[0203] An image feature map acquisition submodule, used to input each image frame into a preset image feature recognition model in sequence to obtain an image feature map of each image frame;

[0204] The hand feature map acquisition submodule is used to input the image feature map of each image frame into a preset gesture feature recognition model to obtain the hand feature map of each image frame.

[0205] Optionally, the gesture feature recognition model includes a motion space feature extraction module, a motion time sequence feature extraction module and a feature enhancement module;

[0206] The motion space feature extraction module is used to extract the motion space features of the image feature map in the first sliding window corresponding to each image frame;

[0207] The motion timing feature extraction module is used to extract the motion timing features in the image feature map within the second sliding window corresponding to each image frame;

[0208] The feature enhancement module is used to perform time series modeling based on the image feature map of each image frame, the motion space feature and the motion time series feature to obtain the hand feature map of each image frame.

[0209] Optionally, the motion space feature extraction module includes a dimension reduction unit, a first superposition unit and a first difference unit;

[0210] The dimension reduction unit is used to perform dimension reduction processing on the image feature map of each image frame to obtain a reduced dimension feature map;

[0211] The first superposition unit is used to superimpose a first preset number of image frames in the first sliding window corresponding to each image frame and calculate the average value of each pixel to obtain a first average feature map of each image frame; the first preset number is less than or equal to a first number threshold;

[0212] The first difference unit is used to obtain the difference value of the pixel at the same position in the reduced dimension feature map of each image frame and the first average feature map, so as to obtain the motion space feature of each image frame.

[0213] Optionally, the motion time series feature extraction module includes a dimension reduction unit, a second superposition unit and a second difference unit;

[0214] The dimension reduction unit is used to perform dimension reduction processing on the image feature map of each image frame to obtain a reduced dimension feature map;

[0215] The second superposition unit is used to superimpose a second preset number of image frames in the second sliding window corresponding to each image frame and calculate the average value of each pixel to obtain a second average feature map of each image frame; the second preset number is greater than or equal to a second number threshold; the second number threshold is greater than the first number threshold;

[0216] The second difference unit is used to obtain the difference value of the pixel at the same position in the reduced dimension feature map of each image frame and the second average feature map, so as to obtain the motion timing characteristics of each image frame.

[0217] Optionally, the feature enhancement module includes a first weighting unit, a second weighting unit, a first fusion unit, a dimension-increasing unit, a third weighting unit, a fourth weighting unit, and a second fusion unit;

[0218] The first weighting unit is used to obtain the product of the motion space feature and the first weight value to obtain the weighted motion space feature;

[0219] The second weighting unit is used to obtain the product of the motion time series feature and the second weight value to obtain the weighted motion time series feature;

[0220] The first fusion unit is used to obtain the sum of the pixels at the same position of the motion spatial feature and the motion temporal feature to obtain a motion-static fusion feature;

[0221] The dimension-increasing unit is used to perform dimension-increasing processing on the dynamic and static fusion features to obtain dimension-increasing features;

[0222] The third weighting unit is used to obtain the product of the dimension-raising feature and the third weight value to obtain the weighted dimension-raising feature;

[0223] The fourth weighting unit is used to obtain the product of the image feature map and a fourth weight value to obtain a weighted image feature map;

[0224] The second fusion unit is used to obtain the sum of the weighted dimensionality-upgraded feature and the pixel at the same position in the weighted image feature map to obtain a hand feature map.

[0225] Optionally, the dimension-raising unit for obtaining dimension-raising features in the gesture feature recognition model includes at least one time domain convolution layer; the time domain convolution layer includes a one-dimensional CNN convolution layer and a pooling layer; the at least one time domain convolution layer is used to perform time series modeling on the dynamic and static fusion features to obtain dimension-raising features including hand movement features.

[0226] Optionally, the image feature recognition model includes at least one of the following: a ResNet model, a VGGNet model, and a GoogLeNet model.

[0227] Optionally, a model training module is further included, and the model training module is used to train the gesture feature recognition model, including:

[0228] A video clip acquisition submodule, used to acquire each image frame of each video clip in a preset training set; the preset training set includes a plurality of video clips, each video clip includes sign language content and a label sequence;

[0229] An image feature map acquisition submodule, used to input each image frame into a preset image feature recognition model in sequence to obtain an image feature map of each image frame;

[0230] A hand feature map acquisition submodule, configured to input the image feature map of each image frame into a preset gesture feature recognition model to obtain the hand feature map of each image frame in response to the loss value being less than a preset loss threshold;

[0231] A sign language text information acquisition submodule, used to perform classification prediction processing according to the hand feature map of each image frame to obtain the sign language text information corresponding to the information conveyed by the hand movement in the video clip;

[0232] A loss value acquisition submodule, used to calculate the loss value of the loss function according to the sign language text information and the label sequence of each video clip;

[0233] The recognition model acquisition submodule is used to stop training the gesture feature recognition model in response to the loss value being greater than or equal to the preset loss threshold, and obtain the preset gesture feature recognition model.

[0234] Optionally, the loss function is used to calculate the probability that the label sequence corresponding to the gesture text information appears in the label sequence corresponding to the video clip when the model parameters are determined.

[0235] Optionally, the sign language text information acquisition module includes:

[0236] A context information acquisition submodule, used to sequentially input the hand feature images of each image frame into a preset context information processing model to obtain the context information of each image frame;

[0237] A prediction probability acquisition submodule, used to input the context information of each image frame into a preset information classification model in sequence to obtain the classification prediction probability of the context information;

[0238] The sign language text information acquisition submodule is used to decode according to the classification prediction probability of the context information of each image frame to obtain the sign language text information corresponding to the information conveyed by the hand movement in the video clip.

[0239] Optionally, the context information processing model includes at least one of the following models: a CNN model, a Transformer model, and a BiLSTM model.

[0240] Optionally, the training process of the context information processing model and the information classification model is synchronized with the training process of the gesture feature recognition model, and the training of the context information processing model is terminated when the training of the gesture feature recognition model is completed.

[0241] It should be noted that the device embodiment shown in this embodiment matches the content of the above method embodiment, and the content of the above method embodiment can be referred to, which will not be repeated here.

[0242] Fig.10 1 is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 1000 may be a smart phone, a computer, a digital broadcast terminal, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0243] Reference Fig.10 The electronic device 1000 may include one or more of the following components: a processing component 1002 , a memory 1004 , a power component 1006 , a multimedia component 1008 , an audio component 1010 , an input / output (I / O) interface 1012 , a sensor component 1014 , a communication component 1016 , and an image acquisition component 1018 .

[0244] The processing component 1002 generally controls the overall operation of the electronic device 1000, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1002 may include one or more processors 1020 to execute computer programs. In addition, the processing component 1002 may include one or more modules to facilitate interaction between the processing component 1002 and other components. For example, the processing component 1002 may include a multimedia module to facilitate interaction between the multimedia component 1008 and the processing component 1002. In an example, the processor 1020 may implement the scheme of the above-mentioned power supply control method.

[0245] The memory 1004 is configured to store various types of data to support operations on the electronic device 1000. Examples of such data include computer programs for any application or method operating on the electronic device 1000, contact data, phone book data, messages, pictures, videos, etc. The memory 1004 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0246] The power supply component 1006 provides power to various components of the electronic device 1000. The power supply component 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 1000. The power supply component 1006 may include a power chip, and the controller may communicate with the power chip to control the power chip to turn on or off the first switch device, so that the battery supplies power to the mainboard circuit or not. In one example, the power supply component 1006 includes Figure 3 The first power supply circuit and the second power supply circuit shown, and / or Figure 6 The third power supply circuit is shown.

[0247] The multimedia component 1008 includes a screen that provides an output interface between the electronic device 1000 and the target object. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input information from the target object. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0248] The audio component 1010 is configured to output and / or input audio file information. For example, the audio component 1010 includes a microphone (MIC), and when the electronic device 1000 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive external audio file information. The received audio file information can be further stored in the memory 1004 or sent via the communication component 1016. In some embodiments, the audio component 1010 also includes a speaker for outputting audio file information.

[0249] The I / O interface 1012 provides an interface between the processing component 1002 and a peripheral interface module, which may be a keyboard, a click wheel, a button, etc.

[0250] The sensor component 1014 includes one or more sensors for providing various aspects of status assessment for the electronic device 1000. For example, the sensor component 1014 can detect the open / closed state of the electronic device 1000, the relative positioning of components, such as the display screen and keypad of the electronic device 1000, and the sensor component 1014 can also detect the position change of the electronic device 1000 or a component, the presence or absence of contact between the target object and the electronic device 1000, the orientation or acceleration / deceleration of the electronic device 1000, and the temperature change of the electronic device 1000. In this example, the sensor component 1014 can include a magnetic sensor, a gyroscope, and a magnetic field sensor, and can also include an inertial sensor, an image sensor, etc., wherein the magnetic field sensor includes at least one of the following: a Hall sensor, a thin film magnetoresistive sensor, and a magnetic liquid acceleration sensor.

[0251] The communication component 1016 is configured to facilitate wired or wireless communication between the electronic device 1000 and other devices. The electronic device 1000 can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, or a combination thereof. In an exemplary embodiment, the communication component 1016 receives broadcast information or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1016 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0252] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0253] In an exemplary embodiment, the present disclosure also provides a non-transitory computer-readable storage medium, and when an executable computer program in the storage medium is executed by a processor, the above-mentioned method can be implemented.

[0254] In an exemplary embodiment, a chip is also provided, the chip including a processor and an interface for reading a computer program through the interface to implement the above method. The chip may be a conventional CPU (central processing unit) chip, a GPU (graphics processing unit) chip, etc., or an acceleration chip dedicated to artificial intelligence technology, such as an AI (Artificial Intelligence) accelerator, etc.

[0255] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0256] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A gesture recognition method, characterized in that: include: Acquire a video clip to be identified, wherein each image frame of the video clip includes hand information of a target object; Performing feature recognition processing on each of the image frames to obtain a hand feature map of each of the image frames; the hand feature map includes motion space features and motion time series features; the motion space features include hand shape information features of the hand in the image frame; the motion time series features include hand movement features of the hand in multiple image frames; Classification prediction processing is performed based on the hand feature maps of each image frame to obtain sign language text information corresponding to the information conveyed by the hand movements in the video clip.

2. The method according to claim 1, characterized in that Performing feature recognition processing on each of the image frames to obtain a hand feature map of each of the image frames includes: Inputting each of the image frames into a preset image feature recognition model in sequence to obtain an image feature map of each of the image frames; The image feature map of each image frame is input into a preset gesture feature recognition model to obtain a hand feature map of each image frame.

3. The method according to claim 2, characterized in that The gesture feature recognition model obtains a hand feature map of each image frame through the following steps, including: Extracting motion space features of the image feature map in the first sliding window corresponding to each image frame; Extracting motion timing features in the image feature map within the second sliding window corresponding to each image frame; Temporal modeling is performed according to the image feature map of each image frame, the motion space feature and the motion temporal feature to obtain a hand feature map of each image frame.

4. The method according to claim 3, characterized in that Extracting motion space features of the image feature map in the first sliding window corresponding to each image frame includes: Performing dimensionality reduction processing on the image feature map of each image frame to obtain a dimensionality reduction feature map; Superimposing a first preset number of image frames in a first sliding window corresponding to each image frame and calculating the average value of each pixel to obtain a first average feature map of each image frame; the first preset number is less than or equal to a first number threshold; The difference value of the pixel at the same position in the dimension reduction feature map of each image frame and the first average feature map is obtained to obtain the motion space feature of each image frame.

5. The method according to claim 3, characterized in that: Extracting the motion timing features in the image feature map within the second sliding window corresponding to each image frame includes: Performing dimensionality reduction processing on the image feature map of each image frame to obtain a dimensionality reduction feature map; Superimposing a second preset number of image frames in a second sliding window corresponding to each image frame and calculating the average value of each pixel to obtain a second average feature map of each image frame; the second preset number is greater than or equal to a second number threshold; the second number threshold is greater than the first number threshold; The difference value of the pixel at the same position in the dimension reduction feature map of each image frame and the second average feature map is obtained to obtain the motion timing characteristics of each image frame.

6. The method according to claim 3, characterized in that Performing time series modeling according to the image feature graph of each image frame, the motion space feature and the motion time series feature to obtain the hand feature graph of each image frame includes: Obtaining a product of the motion space feature and a first weight value to obtain a weighted motion space feature; Obtaining the product of the motion time series feature and the second weight value to obtain a weighted motion time series feature; Obtaining the sum of the pixels at the same position of the motion spatial feature and the motion temporal feature to obtain a motion-static fusion feature; Performing dimension-upgrading processing on the dynamic and static fusion features to obtain dimension-upgrading features; Obtaining the product of the dimension-raising feature and the third weight value to obtain a weighted dimension-raising feature; Obtaining a product of the image feature map and a fourth weight value to obtain a weighted image feature map; The sum of the weighted dimensionality-raising feature and the pixel at the same position in the weighted image feature map is obtained to obtain a hand feature map.

7. The method according to claim 6, characterized in that The dimension-raising unit for obtaining dimension-raising features in the gesture feature recognition model includes at least one time-domain convolution layer; the time-domain convolution layer includes a one-dimensional CNN convolution layer and a pooling layer; the at least one time-domain convolution layer is used to perform time series modeling on the dynamic and static fusion features to obtain dimension-raising features including hand movement features.

8. The method according to claim 2, characterized in that: The image feature recognition model includes at least one of the following: a ResNet model, a VGGNet model, and a GoogLeNet model.

9. The method according to claim 2, characterized in that: The training steps of the gesture feature recognition model include: Acquire each image frame of each video clip in a preset training set; the preset training set includes a plurality of video clips, each video clip includes sign language content and a label sequence; Inputting each of the image frames into a preset image feature recognition model in sequence to obtain an image feature map of each of the image frames; In response to the loss value being less than a preset loss threshold, inputting the image feature map of each image frame into a gesture feature recognition model to obtain a hand feature map of each image frame; Performing classification prediction processing according to the hand feature graphs of each image frame to obtain sign language text information corresponding to the information conveyed by the hand movements in the video clip; Calculating a loss value of a loss function according to the sign language text information and the label sequence of each video clip; In response to the loss value being greater than or equal to the preset loss threshold, training the gesture feature recognition model is stopped to obtain the preset gesture feature recognition model.

10. The method according to claim 9, characterized in that The loss function is used to calculate the probability that the label sequence corresponding to the gesture text information appears in the label sequence corresponding to the video clip when the model parameters are determined.

11. The method according to claim 1, characterized in that: Classification prediction processing is performed according to the hand feature map of each image frame to obtain sign language text information corresponding to the information conveyed by the hand movement in the video clip, including: Inputting the hand feature images of each image frame into a preset context information processing model in sequence to obtain context information of each image frame; Inputting the context information of each image frame into a preset information classification model in sequence to obtain a classification prediction probability of the context information; Decoding is performed according to the classification prediction probability of the context information of each image frame to obtain sign language text information corresponding to the information conveyed by the hand movement in the video clip.

12. The method according to claim 11, characterized in that The context information processing model includes at least one of the following models: a CNN model, a Transformer model, and a BiLSTM model.

13. The method according to claim 11, characterized in that The training process of the context information processing model and the information classification model is synchronized with the training process of the gesture feature recognition model, and the training of the context information processing model is ended when the training of the gesture feature recognition model is completed.

14. A gesture recognition device, characterized in that: include: A video segment acquisition module, used to acquire a video segment to be identified, each image frame of the video segment includes hand information of the target object; A hand feature map acquisition module is used to perform feature recognition processing on each image frame to obtain a hand feature map of each image frame; the motion space feature includes the hand shape information feature of the hand in the image frame; the motion time sequence feature includes the hand movement feature of the hand in multiple image frames; The sign language text information acquisition module is used to perform classification prediction processing based on the hand feature maps of each image frame to obtain the sign language text information corresponding to the information conveyed by the hand movements in the video clip.

15. An electronic device, characterized in that: include: Processor and memory; The memory is used to store a computer program executable by the processor; The processor is configured to execute the computer program in the memory to implement the method according to any one of claims 1 to 13.

16. A non-transitory computer-readable storage medium, characterized in that: When the executable computer program in the storage medium is executed by a processor, the method according to any one of claims 1 to 13 can be implemented.

Images