Gesture motion translation model training method and gesture motion translation method

Abstract

The invention discloses a training method of a gesture motion translation model, a gesture motion translation method and device, setting and a storage medium. The method comprises the following steps: when a target person performs a gesture action, acquiring video data corresponding to the target person and target information corresponding to the gesture action; performing at least one action type identification on the video data to obtain first feature information; performing gesture motion prediction on the gesture motion based on the target information to obtain second feature information; inputting the first feature information and the second feature information into the gesture motion translation model to obtain a loss function corresponding to the gesture motion translation model; and updating parameters corresponding to the gesture motion translation model based on the loss function.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a training method for a gesture translation model and a gesture translation method. Background Art

[0002] In related technologies, feature extraction requires strong theoretical knowledge and multiple experimental verifications in the two stages of classification and recognition. The classification results are overly dependent on empirical knowledge, and the model ignores the optimization processing and features of the data itself. For example, it ignores auxiliary information in pictures or videos, such as background information, body information, torso information, etc.; it ignores gesture features, such as the connection between the key points of the gesture skeleton and the human torso, the influence of the gesture palm angle on the same type of gestures, etc., which increases the difficulty of recognition. There is currently no effective solution to this problem. Summary of the invention

[0003] In order to solve the existing technical problems, the embodiments of the present invention provide a training method for a gesture translation model, a gesture translation method, an apparatus, a device and a storage medium.

[0004] To achieve the above object, the technical solution of the embodiment of the present invention is implemented as follows:

[0005] In a first aspect, an embodiment of the present invention provides a method for training a gesture translation model, the method comprising:

[0006] When the target person performs a gesture, obtaining video data corresponding to the target person and target information corresponding to the gesture;

[0007] Performing at least one action type recognition on the video data to obtain first feature information;

[0008] Performing gesture action prediction on the gesture action based on the target information to obtain second feature information;

[0009] Inputting the first feature information and the second feature information into the gesture translation model to obtain a loss function corresponding to the gesture translation model;

[0010] The parameters corresponding to the gesture translation model are updated based on the loss function.

[0011] In the above solution, the method further includes:

[0012] When there is an abnormality in the video data, obtaining the label data and target video data corresponding to the target person;

[0013] The labeled data, the target video data, the first feature information, and the second feature information are input into the gesture translation model to obtain the loss function.

[0014] In the above scheme, the annotation data includes first annotation data corresponding to the emotion category of the target person, second annotation data corresponding to the gesture action, and third annotation data corresponding to the translation of the gesture action into text; the step of obtaining the annotation data corresponding to the target person includes:

[0015] The first labeling data, the second labeling data and the third labeling data are determined according to the video data.

[0016] In the above solution, the step of obtaining the target video data corresponding to the target person includes:

[0017] Expanding the gesture action to obtain style image information corresponding to the gesture action;

[0018] Determining style migration data corresponding to the gesture action according to the video data and the style picture information;

[0019] The video data and the style migration data are fused to obtain the target video data.

[0020] In the above scheme, the at least one action type includes a first type, a second type and a third type; the first type represents the facial shape of the target person; the second type represents the torso shape of the target person; the third type represents the lip shape of the target person; the first feature information includes first sub-feature information, second sub-feature information and third sub-feature information; the at least one action type recognition is performed on the video data to obtain the first feature information, including:

[0021] Performing the first type identification on the video data to obtain the first sub-feature information;

[0022] Performing the second type identification on the video data to obtain the second sub-feature information;

[0023] The third type identification is performed on the video data to obtain the third sub-feature information.

[0024] In the above solution, performing the first type identification on the video data to obtain the first sub-feature information includes:

[0025] Acquire first target frame information corresponding to the video data;

[0026] Extracting facial information from the first target frame information to obtain facial feature information corresponding to the video data;

[0027] Performing convolutional neural network (CNN) processing on the first target frame information to obtain third feature information; the third feature information represents pixel feature information corresponding to the target frame information;

[0028] Performing graph convolutional neural network (GCN) processing on the facial feature information to obtain fourth feature information; the fourth feature information represents spatial information feature information corresponding to the facial feature information;

[0029] The third feature information and the fourth feature information are concatenated to obtain the first sub-feature information.

[0030] In the above solution, performing the second type identification on the video data to obtain the second sub-feature information includes:

[0031] Acquire second target frame information corresponding to the video data;

[0032] The torso information is extracted from the second target frame information to obtain the second sub-feature information.

[0033] In the above solution, performing the third type identification on the video data to obtain the third sub-feature information includes:

[0034] Acquire third target frame information corresponding to the video data;

[0035] Extracting lip information from the third target frame information to obtain lip feature information corresponding to the video data;

[0036] Performing dense convolutional network (Dense Convolutional Network, DenseNet) processing on the third target frame information to obtain fifth feature information; the fifth feature information represents pixel feature information corresponding to the third target frame information;

[0037] Performing GCN processing on the lip feature information to obtain sixth feature information; the sixth feature information represents spatial information feature information corresponding to the lip feature information;

[0038] The fifth characteristic information and the sixth characteristic information are concatenated to obtain the third sub-characteristic information.

[0039] In the above solution, the step of performing gesture action prediction on the gesture action based on the target information to obtain the second feature information includes:

[0040] Acquire a first parameter corresponding to a hand region in the video data and a second parameter corresponding to a region adjacent to the hand region;

[0041] The second feature information is determined based on the first parameter and the second parameter.

[0042] In the above solution, the second feature information includes fourth sub-feature information and fifth sub-feature information; the fourth sub-feature information represents feature information of a gesture type corresponding to the gesture action; the fifth sub-feature information represents predicted feature information corresponding to the gesture action; and determining the second feature information based on the first parameter and the second parameter includes:

[0043] Processing the first parameter and the second parameter using GCN and a temporal convolutional network (TCN) to obtain the fourth sub-feature information;

[0044] The gesture category feature information is processed by a temporal graph convolutional network (T-GCN) to obtain the fifth sub-feature information.

[0045] In the above solution, the step of inputting the first feature information and the second feature information into the gesture translation model to obtain a loss function corresponding to the gesture translation model includes:

[0046] Concatenate the first characteristic information and the second characteristic information to obtain seventh characteristic information;

[0047] Inputting the seventh feature information into the gesture translation model to obtain a prediction result corresponding to the target person;

[0048] The loss function is determined based on the prediction result and the labeled data.

[0049] In the above scheme, the seventh feature information includes the sixth sub-feature information, the seventh sub-feature information and the eighth sub-feature information; the sixth sub-feature information represents the emotion information corresponding to the target person; the seventh sub-feature information represents the information of the gesture category corresponding to the gesture action; the eighth sub-feature information represents the information for predicting the gesture action; the prediction result includes the emotion classification result, the text translation result and the gesture prediction result; the seventh feature information is input into the gesture action translation model to obtain the prediction result corresponding to the target person, including:

[0050] Decoding the sixth sub-feature information to obtain the emotion classification result;

[0051] Decoding the seventh sub-feature information to obtain the text translation result;

[0052] The eighth sub-feature information is decoded to obtain the gesture prediction result.

[0053] In the above solution, the loss function includes an emotion classification loss function, a gesture action prediction loss function and a text translation loss function; and determining the loss function based on the prediction result and the labeled data includes:

[0054] Determine the emotion classification loss function according to the emotion classification result and the emotion category labeling data;

[0055] Determine the text translation loss function based on the text translation result and the text annotation data;

[0056] The gesture action prediction loss function is determined according to the gesture prediction result and the gesture annotation data.

[0057] In a second aspect, an embodiment of the present invention provides a gesture translation method, the method comprising:

[0058] When the target person performs a gesture, obtain video data corresponding to the target person;

[0059] determining whether target information can be determined according to the video data, and if the target information can be determined, inputting the target information into a gesture translation model to obtain a gesture translation result;

[0060] The gesture translation model is trained based on the gesture translation model training method provided in an embodiment of the present invention.

[0061] In the above solution, after obtaining the video data corresponding to the target person when the target person performs a gesture, the method further includes:

[0062] The coordinate data corresponding to the target person is determined according to the video data; the coordinate data is used to determine the target information.

[0063] In the above solution, the method further includes:

[0064] determining style transfer data based on the video data without determining the target information;

[0065] Determine target video data according to the video data and the style migration data;

[0066] The coordinate data corresponding to the target person is re-determined based on the target video data.

[0067] In a third aspect, an embodiment of the present invention further provides a training device for a gesture translation model, the device comprising:

[0068] A first acquisition unit, configured to acquire video data corresponding to the target person and target information corresponding to the gesture action when the target person performs a gesture action;

[0069] an identification unit, configured to identify at least one action type of the video data to obtain first feature information;

[0070] A prediction unit, configured to perform gesture action prediction on the gesture action based on the target information to obtain second feature information;

[0071] An input unit, used for inputting the first feature information and the second feature information into the gesture translation model to obtain a loss function corresponding to the gesture translation model;

[0072] An updating unit is used to update parameters corresponding to the gesture translation model based on the loss function.

[0073] In a fourth aspect, an embodiment of the present invention further provides a gesture translation device, characterized in that the device comprises:

[0074] A second acquisition unit is used to acquire video data corresponding to the target person when the target person performs a gesture action;

[0075] a judgment unit, configured to judge whether target information can be determined according to the video data, and if the target information is determined, input the target information into a gesture translation model to obtain a gesture translation result;

[0076] The gesture translation model is trained based on the gesture translation model training method provided in an embodiment of the present invention.

[0077] In a fifth aspect, an embodiment of the present invention further provides an electronic device, comprising a memory for storing executable instructions;

[0078] The processor is used to implement the training method of the gesture action translation model provided by the embodiment of the present invention, or implement the gesture action translation method provided by the embodiment of the present invention when executing the executable instructions stored in the storage.

[0079] In a sixth aspect, an embodiment of the present invention further provides a storage medium, which stores executable instructions. When the executable instructions are executed by at least one processor, the training method of the gesture action translation model provided by the embodiment of the present invention is implemented, or the gesture action translation method provided by the embodiment of the present invention is implemented.

[0080] The present application provides a training method, a gesture translation method, a device, an equipment and a storage medium for a gesture translation model. Among them, the training method of the gesture translation model includes: when a target person performs a gesture, obtaining the video data corresponding to the target person and the target information corresponding to the gesture; performing at least one action type recognition on the video data to obtain the first feature information; performing gesture prediction on the gesture based on the target information to obtain the second feature information; inputting the first feature information and the second feature information into the gesture translation model to obtain the loss function corresponding to the gesture translation model; updating the parameters corresponding to the gesture translation model based on the loss function. By adopting the technical solution of the embodiment of the present invention, by inputting the first feature information and the second feature information into the sign language translation model to obtain the loss function corresponding to the gesture translation model, and updating the parameters corresponding to the gesture translation model based on the loss function, intelligent sign language learning can be realized without relying on previous experience, network parameter optimization can be achieved, and the accuracy of sign language translation can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 A schematic diagram of gesture recognition provided by an embodiment of the present invention;

[0082] Figure 2 A schematic diagram of style transfer provided by an embodiment of the present invention;

[0083] Figure 3 A schematic diagram of a graph neural network provided by an embodiment of the present invention;

[0084] Figure 4 A schematic diagram of key points of a human body provided by an embodiment of the present invention;

[0085] Figure 5 A flowchart of a method for training a gesture translation model provided by an embodiment of the present invention;

[0086] Figure 6 A schematic diagram of the implementation flow of the gesture translation method provided in an embodiment of the present invention;

[0087] Figure 7 A schematic diagram of a training process of a gesture translation model provided by an embodiment of the present invention;

[0088] Figure 8 A schematic diagram of sign language collection and annotation provided by an embodiment of the present invention;

[0089] Fig. 9 A flowchart of a sign language translation training model provided by an embodiment of the present invention;

[0090] Fig.10 A schematic diagram of a gesture key point provided by an embodiment of the present invention;

[0091] Fig.11 A schematic diagram of the relationship between a gesture and a torso position provided by an embodiment of the present invention;

[0092] Fig.12 A flowchart of another sign language translation training model provided by an embodiment of the present invention;

[0093] Fig.13 A schematic diagram of a flow chart of sign language translation reasoning provided by an embodiment of the present invention;

[0094] Fig.14 A schematic diagram of the structure of a training device for a gesture translation model provided by an embodiment of the present invention;

[0095] Fig.15 A schematic diagram of the structure of a gesture translation device provided by an embodiment of the present invention;

[0096] Fig.16 A schematic diagram of a hardware entity structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0097] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0098] Gesture recognition: The purpose of gesture recognition is to recognize human gestures through mathematical algorithms, which can come from the movement of various parts of the human body, often referring to the movement of the face and hands. Users can use simple gestures to control or interact with devices, allowing computers to understand human behavior. Figure 1 A schematic diagram of a gesture recognition provided by an embodiment of the present invention, such as Figure 1 As shown, Figure 1 Allows users to interact with the vehicle through gestures.

[0099] Style transfer: Style transfer refers to the use of an algorithm to fuse the style information of a given content image and style image while keeping the image content contained in the content image unchanged, thereby generating a new image with the same content as the input content image but the style of the input image. Figure 2 A schematic diagram of a style transfer provided by an embodiment of the present invention, such as Figure 2 As shown, Figure 2 The content generated by migration is the same as the original content, and the style generated by migration is the original style.

[0100] Graph Neural Network: Graph Neural Network (GNN) is a general term for algorithms that use neural networks to learn graph structured data, extract and discover features and patterns in graph structured data, and meet the needs of graph learning tasks such as clustering, classification, prediction, segmentation, and generation. Figure 3 A schematic diagram of a graph neural network provided for an embodiment of the present invention.

[0101] Key point detection: In image processing, a key point is essentially a feature, an abstract description of a fixed area or spatial physical relationship, describing a combination or contextual relationship within a certain neighborhood. It is not just a point information, or represents a position, but also represents the combination relationship between the context and the surrounding neighborhood. Human skeleton key point detection is the basis of many computer vision tasks, such as posture estimation, behavior recognition, human-computer interaction, virtual reality, smart home, and unmanned driving. Figure 4 A schematic diagram of key points of a human body provided by an embodiment of the present invention.

[0102] In the related art, sign language recognition is mainly divided into static sign language recognition and dynamic sign language recognition. Based on the fact that sign language in real scenes is generally continuous sign language recognition, the sign language recognition in the embodiments of the present invention refers to dynamic sign language recognition, and dynamic gesture recognition can eventually be converted into static gesture recognition.

[0103] The relevant solutions are mainly divided into two technical routes: 1) sign language recognition based on contact sensors; 2) sign language recognition based on non-contact sensors.

[0104] 1) Sign language recognition based on contact sensors: Contact sensor gesture recognition refers to a recognition method that obtains hand information through sensors. This method generally refers to sign language recognition based on data gloves. Data gloves contain multiple sensors, through which various data of hand changes can be obtained in real time and without omission, and then the data are feature extracted and classified to finally identify gesture and sign language information. However, sign language recognition based on contact sensors such as data gloves has a high accuracy rate and seems to be close to practical use, but data gloves are expensive and inconvenient to carry. At the same time, they have certain limitations on the flexibility and naturalness of the hands, making them difficult to popularize and promote.

[0105] 2) Sign language recognition based on non-contact sensors: Non-contact sensor gesture recognition generally refers to visual-based gesture recognition. This type of recognition method generally uses hand image information collected by a camera to analyze and process the image, including image preprocessing, gesture tracking and positioning, hand segmentation, feature extraction, and classification recognition to recognize sign language information. This type of method can be roughly divided into two methods based on different image algorithms: based on traditional image processing algorithms and based on deep learning image algorithms.

[0106] Based on traditional image processing algorithms: The feature extraction process generally models the image space appearance features, such as the contour, color, texture, and movement direction of the limbs, and generally uses traditional algorithms such as principal component analysis (PCA) and scale invariant feature transform (SIFT). The sign language recognition stage often uses machine learning classification methods such as K-nearest neighbor (KNN) and hidden Markov model (HMM). However, the disadvantage is that it relies too much on empirical knowledge: in traditional methods, feature extraction requires strong theoretical knowledge to conduct multiple experimental verifications of the two stages of feature extraction and classification recognition, and the classification results rely too much on empirical knowledge; poor universality: there are few traditional sign language recognition methods, and manual feature extraction takes a lot of time, and most of them are used for the recognition of isolated sign language words, which is not universal. At the same time, dynamic sign language recognition requires the establishment of reliable long-term temporal correlation, which is difficult for traditional algorithms to achieve.

[0107] Based on deep learning image algorithm: neural network is used to replace traditional manual feature model in feature extraction stage, such as three dimensional convolutional neural network (3D-CNN), GCN and other networks to obtain spatial information; recurrent neural network (RNN), long short-term memory network (LSTM), gated recurrent unit (GRU) and other networks are used to build language model for the extracted features and translate the sign language video sequence into complete sentences. However, there are the following defects, such as insufficient data feature learning: existing models often only focus on network structure innovation, ignoring the optimization processing and features of the data itself, resulting in the loss of a large amount of useful information in the model input stage, which increases the difficulty of recognition. For example, it ignores auxiliary information in images or videos, such as background information, body information, and trunk information; it ignores gesture features, such as the connection between the key points of gesture skeletons and the human trunk, the influence of the angle of the hand on gestures of the same category, etc.; it ignores the impact of detection in scenes with drastic changes in lighting: existing models often do not consider the impact of lighting on sign language detection in scenes with low visibility, low lighting, indoors and outdoors, and different environments during the day and night; it ignores gesture translation in scenes with oral assistance: existing models often implement sign language translation based on sign language movements, while the communication of deaf-mute people in real scenes may contain oral auxiliary information; it ignores the help of sign language prediction for gesture classification: existing models are based on dynamic sign language recognition, and often only consider the temporal and spatial information of sign language videos, and do not consider the assistance of gesture learning based on the prediction of the current frame movement in the sign language video to gesture learning; it ignores the expression of emotions in sign language: it does not consider the inconsistency between the word order and the normal word order when the deaf-mute communicate. In order to ensure the simplicity of sign language, deaf-mute people often use simple and straightforward ways to describe text. When directly translated into text, it is difficult to learn the expression of emotions in sign language. At the same time, deaf-mute people use different gestures when expressing different emotions. Emotion-based learning can help better learn gesture features.

[0108] Based on this, the embodiments of the present invention provide a training method for a gesture translation model, a gesture translation method, an apparatus, a device and a storage medium, which are independent of prior experience, do not require manual setting of short hair, and can effectively improve the accuracy of sign language translation.

[0109] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0110] The embodiment of the present invention provides a method for training a gesture translation model. Figure 5A flowchart of a method for training a gesture translation model provided by an embodiment of the present invention, the method comprising:

[0111] S501: When a target person performs a gesture, video data corresponding to the target person and target information corresponding to the gesture are obtained.

[0112] It should be noted that the target person may be a person who needs to translate the gestures, and as an example, the target person may be a deaf-mute. The gestures may be understood as sign language.

[0113] In this embodiment, the video data may be video RGB data; the acquisition of the video data corresponding to the target person may be understood as performing image acquisition on the target person to obtain the video data corresponding to the target person; as an example, the target person may be acquired in real time by a monocular camera to obtain video RGB data, and the video RGB data may be recorded as x d .

[0114] The target information corresponding to the gesture action can be understood as the key point coordinate information corresponding to the gesture action. The acquisition of the target information corresponding to the gesture action can be understood as acquiring the key point coordinate information corresponding to the gesture action; as an example, the key point coordinate information corresponding to the gesture action can be extracted based on the human posture estimation algorithm (openpose). It should be noted that while extracting the key point coordinate information, the confidence (x, y, p) corresponding to the key point coordinate information can also be extracted, where x represents the horizontal coordinate, y represents the vertical coordinate, and p represents the confidence, and each key point corresponds to a confidence value.

[0115] S502: Identify at least one action type on the video data to obtain first feature information.

[0116] It should be noted that the at least one action type recognition includes a first type, a second type and a third type; the first type represents the facial shape of the target person; the second type represents the torso shape of the target person; the third type represents the lip shape of the target person. The first feature information includes first sub-feature information, second sub-feature information and third sub-feature information.

[0117] In this embodiment, the performing of at least one action type recognition on the video data to obtain the first feature information can be understood as performing the first type recognition on the video data to obtain the first sub-feature information; performing the second type recognition on the video data to obtain the second sub-feature information; and performing the third type recognition on the video data to obtain the third sub-feature information.

[0118] S503: Predicting the gesture action based on the target information to obtain second feature information.

[0119] In this embodiment, the gesture action is predicted based on the target information to obtain the second feature information, which can be understood as obtaining a first parameter corresponding to the hand area in the video data and a second parameter corresponding to the area adjacent to the hand area; and determining the second feature information based on the first parameter and the second parameter.

[0120] S504: Input the first feature information and the second feature information into the gesture translation model to obtain a loss function corresponding to the gesture translation model.

[0121] In this embodiment, the inputting of the first feature information and the second feature information into the gesture translation model to obtain the loss function corresponding to the gesture translation model can be understood as concatenating the first feature information and the second feature information to obtain the seventh feature information;

[0122] Inputting the seventh feature information into the gesture translation model to obtain a prediction result corresponding to the target person;

[0123] The loss function is determined based on the prediction result and the labeled data.

[0124] S505: Update parameters corresponding to the gesture translation model based on the loss function.

[0125] In this embodiment, the loss function includes an emotion classification loss function, a gesture action prediction loss function and a text translation loss function. Updating the parameters corresponding to the gesture action translation model based on the loss function can be understood as tuning the weight of the loss function.

[0126] By adopting the technical solution of the embodiment of the present invention, it is possible to acquire gesture translation information based on data collected by common devices such as RGB cameras, mobile phones, and iPads. Data collection is easy, the equipment is simple and the cost is low, and intelligent sign language learning can be realized without relying on previous experience. Network parameters can be optimized to improve the accuracy of sign language translation.

[0127] In an optional embodiment of the present invention, the method further includes: when there is an abnormality in the video data, obtaining the labeled data and target video data corresponding to the target person; inputting the labeled data, the target video data, the first feature information and the second feature information into the gesture translation model to obtain the loss function.

[0128] It should be noted that, when the video data is normal, the label data corresponding to the target person is obtained. When the video data is normal, it can be understood as a situation where the lighting is normal and the video is clear.

[0129] In this embodiment, the video data has abnormal conditions, which can be understood as low visibility, low light, etc. The annotated data corresponding to the target person includes first annotated data corresponding to the target person's emotion category, second annotated data corresponding to the gesture action, and third annotated data corresponding to the translation of the gesture action into text, wherein the annotated data is real annotated data.

[0130] By adopting the technical solution of the embodiment of the present invention, various scenes such as low light and dimness are taken into consideration, and sign language is detected in real time without being disturbed by low light scenes, thereby helping sign language users to communicate normally.

[0131] In an optional embodiment of the present invention, the annotation data includes first annotation data corresponding to the emotion category of the target person, second annotation data corresponding to the gesture action, and third annotation data corresponding to the gesture action translated into text; obtaining the annotation data corresponding to the target person includes: determining the first annotation data, the second annotation data and the third annotation data based on the video data.

[0132] In this embodiment, the first annotation data corresponding to the emotion category of the target person can be understood as emotion annotation data; the second annotation data corresponding to the gesture action can be understood as gesture annotation data; and the third annotation data corresponding to the translation of the gesture action into text can be understood as text annotation data.

[0133] The determining of the first annotation data, the second annotation data and the third annotation data based on the video data can be illustrated by manually annotating the emotions of the target person in the video data to obtain emotion annotation data; manually annotating or calculating the gestures corresponding to the gestures of the target person in the video data using an algorithm model to obtain gesture annotation data; and translating the gestures of the target person in the video data into text to obtain text annotation data.

[0134] The solution of the embodiment of the present invention can be used to update the loss function parameters, optimize the parameters, and improve the accuracy of the sign language prediction results.

[0135] In an optional embodiment of the present invention, the obtaining of the target video data corresponding to the target person includes: expanding the gesture action to obtain style picture information corresponding to the gesture action; determining the style migration data corresponding to the gesture action based on the video data and the style picture information; and fusing the video data and the style migration data to obtain the target video data.

[0136] In this embodiment, the style transfer data corresponding to the gesture action is determined according to the video data and the style image information, which can be understood as converting the video data x d As input data, we combine the style image and use the style transfer model to get the style transfer data. The style image can be recorded as x style , the style migration data can be recorded as x sj .

[0137] It should be noted that the style transfer module network architecture adopts a wavelet transform style transfer (Whiten-Color Transform, WCT2) network based on the Visual Geometry Group (Visual Geometry Group, VGG) network structure. During training, the image reconstruction method based on the L2 loss function is directly used for training, that is, the image output by the model is made consistent with the input image as much as possible during the training process. The specific loss function is shown in formula (1):

[0138] Loss style =(y gt -y gt-pred ) 2 (1)

[0139] In the above formula (1), y gt is the correct data, that is, the input image data, y gt-pred It is the output data during model training.

[0140] The video data and the style transfer data are fused to obtain the target video data, which can be illustrated as: fusion x d 、x sj Generate full data, wherein the target video data can be understood as full data.

[0141] By adopting the technical solution of the embodiment of the present invention, a style transfer module is added to expand data diversity. Without changing the video content, the current low-visibility and low-light pictures are transformed into clear scenes to improve the detection results, which can help the deaf and mute people adapt to normal communication in various scenarios.

[0142] In an optional embodiment of the present invention, the at least one action type includes a first type, a second type and a third type; the first type characterizes the facial shape of the target person; the second type characterizes the torso shape of the target person; the third type characterizes the lip shape of the target person; the first feature information includes first sub-feature information, second sub-feature information and third sub-feature information; the identifying at least one action type of the video data to obtain the first feature information includes: performing the first type identification on the video data to obtain the first sub-feature information; performing the second type identification on the video data to obtain the second sub-feature information; performing the third type identification on the video data to obtain the third sub-feature information.

[0143] It should be noted that the first type of representation of the target person's facial morphology can be understood as the target person's facial expression; the second type of representation of the target person's torso morphology can be understood as the target person's torso; the third type of representation of the target person's lip morphology can be understood as the target person's lip movement.

[0144] In this embodiment, the first type identification is performed on the video data to obtain the first sub-feature information, which can be understood as obtaining the first target frame information corresponding to the video data; extracting facial information from the first target frame information to obtain facial feature information corresponding to the video data; performing convolutional neural network (CNN) processing on the first target frame information to obtain third feature information; the third feature information represents pixel feature information corresponding to the target frame information; performing graph convolutional neural network (GCN) processing on the facial feature information to obtain fourth feature information; the fourth feature information represents spatial information feature information corresponding to the facial feature information; and splicing the third feature information and the fourth feature information to obtain the first sub-feature information.

[0145] The performing the second type recognition on the video data to obtain the second sub-feature information can be understood as obtaining the second target frame information corresponding to the video data; and performing torso information extraction on the second target frame information to obtain the second sub-feature information.

[0146] The performing the third type identification on the video data to obtain the third sub-feature information can be understood as obtaining the third target frame information corresponding to the video data; performing lip information extraction on the third target frame information to obtain lip feature information corresponding to the video data; performing DenseNet processing on the third target frame information to obtain fifth feature information; the fifth feature information represents pixel feature information corresponding to the third target frame information; performing GCN processing on the lip feature information to obtain sixth feature information; the sixth feature information represents spatial information feature information corresponding to the lip feature information; and splicing the fifth feature information and the sixth feature information to obtain the third sub-feature information.

[0147] It should be noted that the first target frame information, the second target frame information and the third target frame information can be understood as the same frame information, and the frame is a key frame extracted from the video data.

[0148] By adopting the technical solution of the embodiment of the present invention, feature information of the face, lips and torso is extracted, thereby realizing sign language translation, gesture prediction and emotion recognition.

[0149] In an optional embodiment of the present invention, performing the first type identification on the video data to obtain the first sub-feature information includes: acquiring the first target frame information corresponding to the video data; performing facial information extraction on the first target frame information to obtain facial feature information corresponding to the video data; performing CNN processing on the first target frame information to obtain third feature information; the third feature information represents pixel feature information corresponding to the target frame information; performing GCN processing on the facial feature information to obtain fourth feature information; the fourth feature information represents spatial information feature information corresponding to the facial feature information; and splicing the third feature information and the fourth feature information to obtain the first sub-feature information.

[0150] In this embodiment, the first sub-feature information may be facial expression recognition feature information, which may be denoted as f 面部 The extracting of facial information from the first target frame information to obtain facial feature information corresponding to the video data can be understood as extracting facial information from the first frame target information to obtain 68 key point information corresponding to the facial area.

[0151] The first target frame information is processed by CNN to obtain third feature information; the third feature information represents pixel feature information corresponding to the target frame information, which can be illustrated by obtaining a facial area image corresponding to the first target frame information, processing the facial area image using a 3-layer CNN, and extracting pixel feature information.

[0152] The facial feature information is processed by GCN to obtain fourth feature information; the fourth feature information represents the spatial information feature information corresponding to the facial feature information, which can be illustrated by using GCN to learn the facial key point graph structure for the 68 facial key points to obtain the spatial information feature information.

[0153] The splicing of the third feature information and the fourth feature information to obtain the first sub-feature information can be illustrated as splicing the pixel feature information and the spatial feature information as the facial expression recognition feature f 面部 .

[0154] By adopting the technical solution of the embodiment of the present invention, the facial expression recognition features are determined through the facial key point information and facial area image of the target person, and the emotion recognition of the target person can be performed, thereby improving the accuracy of sign language recognition.

[0155] In an optional embodiment of the present invention, performing the second type identification on the video data to obtain the second sub-feature information includes: acquiring second target frame information corresponding to the video data; and extracting torso information from the second target frame information to obtain the second sub-feature information.

[0156] In this embodiment, the second sub-feature information can be understood as the trunk feature f 躯干 The obtaining of the second target frame information corresponding to the video data, extracting the torso information from the second target frame information, and obtaining the second sub-feature information can be illustrated as: obtaining the human torso key point information according to the second target frame information corresponding to the video data, and inputting the human torso key point information into the GCN network to extract the torso feature f 躯干 .

[0157] By adopting the technical solution of the embodiment of the present invention, the trunk features are extracted through the key point information of the human trunk, so that gesture work prediction can be realized and the accuracy of sign language detection can be improved.

[0158] In an optional embodiment of the present invention, performing the third type identification on the video data to obtain the third sub-feature information includes: acquiring the third target frame information corresponding to the video data; performing lip information extraction on the third target frame information to obtain lip feature information corresponding to the video data; performing DenseNet processing on the third target frame information to obtain fifth feature information; the fifth feature information represents pixel feature information corresponding to the third target frame information; performing GCN processing on the lip feature information to obtain sixth feature information; the sixth feature information represents spatial information feature information corresponding to the lip feature information; and splicing the fifth feature information and the sixth feature information to obtain the third sub-feature information.

[0159] In this embodiment, the third sub-feature information may be lip recognition feature information, which may be recorded as f 唇语 The extracting lip information from the third target frame information to obtain the lip feature information corresponding to the video data can be understood as extracting lip information from the third frame target information to obtain lip key point information corresponding to the lip area.

[0160] The third target frame information is subjected to DenseNet processing to obtain fifth feature information; the fifth feature information represents pixel feature information corresponding to the third target frame information, and can be illustrated by obtaining a lip area image based on the third target frame information, and extracting pixel features of the lip area using DenseNet.

[0161] The lip feature information is subjected to GCN processing to obtain sixth feature information; the sixth feature information represents the spatial information feature information corresponding to the lip feature information, which can be illustrated by using a GCN graph network to learn a graph structure for the lip key point information to obtain the spatial information feature information.

[0162] The splicing of the fifth feature information and the sixth feature information to obtain the third sub-feature information can be illustrated as: splicing pixel feature and spatial information feature information as lip recognition feature f 唇语 .

[0163] By adopting the technical solution of the embodiment of the present invention, the sign language translation task can be realized through lip recognition features, thereby improving the accuracy of sign language detection.

[0164] In an optional embodiment of the present invention, the gesture action is predicted based on the target information to obtain the second feature information, including: obtaining a first parameter corresponding to the hand area in the video data and a second parameter corresponding to the area adjacent to the hand area; and determining the second feature information based on the first parameter and the second parameter.

[0165] In this embodiment, the second feature information includes fourth sub-feature information and fifth sub-feature information. The first parameter corresponding to the hand area may be a gesture point adjacency matrix A1; the second parameter corresponding to the area adjacent to the hand area may be a hand and torso key point adjacency matrix A2.

[0166] It should be noted that the coordinate information of the key points of the hand in the video data and features such as confidence, angle, length, normal vector, and palm plane inclination are obtained.

[0167] The determining of the second feature information based on the first parameter and the second parameter can be understood as: using GCN and TCN to process the first parameter and the second parameter to obtain the fourth sub-feature information; and performing T-GCN processing on the gesture category feature information to obtain the fifth sub-feature information.

[0168] By adopting the technical solution of the embodiment of the present invention, multi-order feature information of gestures and the position information between hands and torso are extracted to learn the sign language submodule, thereby improving the accuracy of sign language learning.

[0169] In an optional embodiment of the present invention, the second feature information includes fourth sub-feature information and fifth sub-feature information; the fourth sub-feature information represents feature information of a gesture type corresponding to the gesture action; the fifth sub-feature information represents predicted feature information corresponding to the gesture action; determining the second feature information based on the first parameter and the second parameter includes: processing the first parameter and the second parameter using GCN and TCN to obtain the fourth sub-feature information; performing T-GCN processing on the gesture category feature information to obtain the fifth sub-feature information.

[0170] In this embodiment, the fourth sub-feature information may be a gesture category feature f 手部-c The fifth sub-feature information may be a predicted gesture action feature f 手部-act .

[0171] The use of GCN and the temporal convolutional network TCN to process the first parameter and the second parameter to obtain the fourth sub-feature information can be illustrated as follows: in terms of temporal information, the first parameter and the second parameter are input into the TCN network, and its backbone network is changed to a neural (Transformer) network to better realize the aggregation of time dimension information; based on the fusion of GCN and the improved TCN module, a T-GCN backbone network is obtained, and the features learned based on the T-GCN network and the f learned based on the picture information are 手势图片 Feature concatenation to obtain the gesture category feature f 手部-c .

[0172] The processing of the gesture category feature information by the time graph convolution network T-GCN to obtain the fifth sub-feature information can be illustrated as follows: based on the features learned by the T-GCN network, a layer of T-GCN module is added to predict the predicted gesture action feature f 手部-act , used to assist in gesture type recognition.

[0173] By adopting the technical solution of the embodiment of the present invention, the gesture movement information, the position change between gesture frames, the speed change and other frame difference information are used to learn the sign language submodule to improve the accuracy of sign language learning.

[0174] In an optional embodiment of the present invention, the first feature information and the second feature information are input into the gesture translation model to obtain a loss function corresponding to the gesture translation model, including: concatenating the first feature information and the second feature information to obtain seventh feature information; inputting the seventh feature information into the gesture translation model to obtain a prediction result corresponding to the target person; and determining the loss function based on the prediction result and the labeled data.

[0175] In this embodiment, the seventh feature information includes sixth sub-feature information, seventh sub-feature information and eighth sub-feature information; the sixth sub-feature information represents the emotion information corresponding to the target person; the seventh sub-feature information represents the information of the gesture category corresponding to the gesture action; the eighth sub-feature information represents the information for predicting the gesture action.

[0176] The concatenation of the first characteristic information and the second characteristic information to obtain the seventh characteristic information can be illustrated as: 面部 、f 躯干 and f 手部-c Splicing f 面部 、f 手部-c and f 手部-act Splicing According to f 手部-act get

[0177] The prediction results include emotion classification results, text translation results and gesture prediction results. The seventh feature information is input into the gesture translation model to obtain the prediction result corresponding to the target person, which can be understood as decoding the sixth sub-feature information to obtain the emotion classification result; decoding the seventh sub-feature information to obtain the text translation result; and decoding the eighth sub-feature information to obtain the gesture prediction result.

[0178] Determining the loss function based on the prediction result and the annotation data can be understood as determining the emotion classification loss function based on the emotion classification result and the emotion category annotation data; determining the text translation loss function based on the text translation result and the text annotation data; and determining the gesture action prediction loss function based on the gesture prediction result and the gesture annotation data.

[0179] By adopting the technical solution of the embodiment of the present invention, a consistency loss function is proposed. The network is based on multi-task learning, and the weights of the loss functions of sign language translation, sign language action prediction, and emotion recognition are optimized to achieve network parameter optimization.

[0180] In an optional embodiment of the present invention, the seventh feature information includes sixth sub-feature information, seventh sub-feature information and eighth sub-feature information; the sixth sub-feature information represents the emotion information corresponding to the target person; the seventh sub-feature information represents the information of the gesture category corresponding to the gesture action; the eighth sub-feature information represents the information for predicting the gesture action; the prediction result includes an emotion classification result, a text translation result and a gesture prediction result; inputting the seventh feature information into the gesture action translation model to obtain the prediction result corresponding to the target person includes: decoding the sixth sub-feature information to obtain the emotion classification result; decoding the seventh sub-feature information to obtain the text translation result; decoding the eighth sub-feature information to obtain the gesture prediction result.

[0181] In this embodiment, the decoding of the sixth sub-feature information to obtain the emotion classification result can be understood as passing the sixth sub-feature information through a decoding module to obtain the emotion classification result; it can be illustrated as, for example, the emotion feature vector of the current frame learned by the feature extraction module The final output feature vector obtained by the GRU module based on time series learning is combined with the normalized exponential function (soft version of max, softmax) to obtain the emotion classification result.

[0182] The decoding process of the seventh sub-feature information to obtain the text translation result can be understood as passing the seventh sub-feature information through a decoding module to obtain a text translation result; it can be illustrated as, for example, decoding the gesture category feature vector of the current frame learned by the feature extraction module The Transformer module is used to predict the text sequence based on the self-attention mechanism, and the text translation result is obtained by combining softmax.

[0183] The decoding process of the eighth sub-feature information to obtain the gesture prediction result can be understood as passing the eighth sub-feature information through a decoding module to obtain a gesture prediction result; it can be illustrated as, based on the current frame gesture prediction feature vector learned by the feature extraction module Based on the GRU and multi-layer perceptron (MLP) modules, the predicted gesture key point sequence for each frame is obtained, that is, the gesture prediction result.

[0184] By adopting the technical solution of the embodiment of the present invention, the network can predict the next frame of sign language action in real time based on the feature information of sign language, help users to predict gestures, and can be expanded to other subtasks, such as predicting the subsequent actions of competitors in boxing matches.

[0185] In an optional embodiment of the present invention, the loss function includes an emotion classification loss function, a gesture action prediction loss function and a text translation loss function; determining the loss function based on the prediction result and the annotation data includes: determining the emotion classification loss function according to the emotion classification result and the emotion category annotation data; determining the text translation loss function based on the text translation result and the text annotation data; determining the gesture action prediction loss function according to the gesture prediction result and the gesture annotation data.

[0186] It should be noted that the loss function corresponding to the gesture translation model is determined according to the emotion classification loss function, the text translation loss function and the gesture prediction loss function, which can be expressed by formula (2):

[0187] Loss = αLoss emo +βLoss act +(1-α-β)Loss T (2)

[0188] In the above formula (2), α and β represent the loss weights of the gesture translation model, and Loss represents the loss function corresponding to the gesture translation model; Loss emo Represents the emotion classification loss function; Loss act Represents the gesture action prediction loss function; Loss T Represents the text translation loss function.

[0189] In this embodiment, the emotion classification loss function is determined according to the emotion classification result and the emotion category labeling data, which can be expressed by formula (3) as follows:

[0190]

[0191] In the above formula (3), ye is the real emotion label data (real emotion label), y e-p is the emotion classification result (predicted emotion label), and n is the amount of image data.

[0192] The text translation loss function is determined based on the text translation result and the text annotation data, which can be expressed as follows using formula (4):

[0193]

[0194] In the above formula (4), Indicates that at the u-th step decoding, based on the learned feature h u The next prediction is word The probability of (text translation result), is the text annotation data (true prediction probability), D is the target language vocabulary size, and U is the number of learning steps.

[0195] The gesture action prediction loss function is determined according to the gesture prediction result and the gesture annotation data, which can be expressed by formula (5) as follows:

[0196]

[0197] In the above formula (5), X pred Annotate data for gestures (real gesture keypoint label sequences), is the gesture prediction result (predicted gesture key point label sequence), and n is the amount of image data.

[0198] By adopting the technical solution of the embodiment of the present invention, the network is based on multi-task learning, and the weights of the loss functions of sign language translation, sign language action prediction, and emotion recognition are optimized to achieve network parameter optimization.

[0199] The embodiment of the present invention also provides a gesture translation method. Figure 6 A schematic diagram of the implementation flow of the gesture translation method provided by the embodiment of the present invention is shown in FIG. Figure 6 As shown, the method comprises the following steps:

[0200] S601: When the target person performs a gesture, obtain video data corresponding to the target person;

[0201] S602: Determine whether target information can be determined according to the video data, and if the target information can be determined, input the target information into a gesture translation model to obtain a gesture translation result;

[0202] The gesture translation model is trained based on the gesture translation model training method provided in an embodiment of the present invention.

[0203] Here, the acquisition method of S601 is the same as that of the aforementioned S501, and will not be repeated here.

[0204] In S602, the target information can be understood as the gesture key point information corresponding to the target person. When the gesture key point information is detected, the video data, the detected gesture key point information and the corresponding image information are input into the gesture translation model to obtain the gesture translation result. It should be noted that the gesture translation result includes the sign language translation result, the sign language emotion recognition result and the sign language action prediction result.

[0205] By adopting the technical solution of the embodiment of the present invention, the sign language translation result obtained obtains the sign language semantic information corresponding to the video sequence (the filmed sign language video), helping the deaf-mute to translate sign language into text, thereby achieving normal communication. Based on the sign language emotion recognition result, the current emotion of the signer is obtained to better determine the signer's emotion and assist communication. Based on the sign language action prediction result, the signer's next gesture can be predicted in advance, thereby predicting the signer's intention. Of course, based on some non-sign language recognition scenarios, such as boxing matches or wrestling matches, the gesture prediction part can be combined with gesture recognition to predict the user's next action, and the success rate of the game can be increased based on action prediction.

[0206] In an optional embodiment of the present invention, after obtaining the video data corresponding to the target person when the target person performs a gesture, the method further includes: determining the coordinate data corresponding to the target person based on the video data; the coordinate data is used to determine the target information.

[0207] In this embodiment, determining the coordinate data corresponding to the target person based on the video data can be understood as obtaining the 2D coordinate data of the human skeleton nodes in each frame of the RGB video through a posture estimation algorithm (such as openpose), and determining the gesture key point information based on the coordinate data.

[0208] In an optional embodiment of the present invention, the method further includes: determining style transfer data based on the video data without determining the target information; determining target video data based on the video data and the style transfer data; and re-determining coordinate data corresponding to the target person based on the target video data.

[0209] In this embodiment, when the target information is not determined, style transfer data is determined based on the video data; target video data is determined based on the video data and the style transfer data; coordinate data corresponding to the target person is re-determined based on the target video data. It can be understood that when the gesture key point information is not detected, the video data is style transferred to obtain style transfer data, the video data and the style transfer data are merged into full data, and the full data (migrated video data) is used through Openpose to obtain the 2D coordinate data of each frame of the human skeleton node. It should be noted that the method of obtaining full data through style transfer of the video data is the same as the method in the gesture action translation model provided in the embodiment of the present invention, and will not be repeated here.

[0210] By adopting the technical solution of the embodiment of the present invention, sign language translation can be realized in scenes with drastic changes such as low visibility, low light, indoors and outdoors, and different environments and light conditions during the day and night.

[0211] For ease of understanding, here is an example to illustrate the above gesture translation model. Figure 7 A schematic diagram of a training process of a gesture translation model provided by an embodiment of the present invention, such as Figure 7 As shown, it includes a data collection and annotation module, a data expansion module and a sign language text translation module, which will be described in detail below.

[0212] Step 1: Sign language data collection, annotation and expansion.

[0213] In order to ensure the routine of sign language recognition, guarantee the accuracy of shooting results and high-quality image and video data for training or testing, an embodiment of the present invention proposes a data collection, annotation and expansion system, the purpose of which is to provide sufficient data for model training and reasoning stages. Figure 8 A schematic diagram of sign language collection and annotation provided by an embodiment of the present invention, such as Figure 8 shown.

[0214] Step 1.1 Data collection. When collecting data, use a monocular camera to collect image and video RGB data in real time. d The collected data is diverse and is a combination of the following features: full-body portrait, non-full-body portrait (upper body only), with or without oral assistance, with different facial expressions, etc. During collection, the angle between the optical axis direction of the video gesture capture camera and the signer's frontal orientation is controlled within 45 degrees.

[0215] Step 1.1.1 Collect video data.

[0216] Step 1.2 Training data annotation. Annotation method: When annotating gesture emotion categories, only the emotion categories need to be annotated. Text annotation requires the annotator to translate the gesture language normally.

[0217] Step 1.2.1 obtains emotion annotation information from video stream data, which is mainly used for sign language emotion recognition.

[0218] Step 1.2.2 obtains normal text annotation information for the video stream data. Based on the translation of sign language video into normal human text sentences, it is mainly used for sign language text translation.

[0219] Step 1.3 Data expansion. In actual sign language interaction scenarios, there may be low visibility, low light and other phenomena. At the same time, there may be drastic changes in different environments and lighting, such as indoors and outdoors, day and night, which may affect gesture recognition. Based on this, the model adds a style transfer module to expand data diversity. Without changing the current content, the current low visibility and low light pictures are turned into clear scenes to improve the detection effect.

[0220] Step 1.3.1: Expand the data. Get the input data x d , combined with style picture x style Generate style transfer data x sj , fusion x d 、x sj Generate full data.

[0221] Step 2: Sign language interpreter training.

[0222] Sign language is a visual language with specific grammar and word order. Gestures, facial expressions, and body movements can be freely arranged and combined to express different meanings. Considering that lighting conditions, facial expressions, body language, and lip shape are all important components of sign language learning, in order to fully learn the relevant information in the video and solve the problem that gesture information in image detection is difficult to distinguish subtle movements, resulting in low detection accuracy, a multimodal, multi-task learning sign language translation network that can recognize emotions is proposed.

[0223] The network is mainly divided into feature extraction module and decoding module. The feature extraction module is divided into four submodules, including facial expression recognition submodule, human body submodule, lip reading recognition submodule, gesture recognition and detection submodule, among which the gesture recognition and detection submodule is the main detection module, and the other modules are mainly used to assist sign language recognition and emotion recognition. The decoding module mainly performs feature decoding on the three subtasks completed by the network, thereby realizing the sign language translation task, gesture prediction task, and emotion recognition task. Fig. 9 A flow chart of a sign language translation training model provided by an embodiment of the present invention is shown in FIG. Fig. 9 As shown:

[0224] The details of each sub-module in the feature extraction module are as follows:

[0225] Gesture recognition and prediction module (main module): In order to fully learn the characteristics of gestures, the network architecture is improved and learning is performed based on gesture image information and gesture key point information. Fig.10 A schematic diagram of a gesture key point provided by an embodiment of the present invention, such as Fig.10 shown.

[0226] The gesture image network uses a 3-layer CNN to learn gesture features based on the gesture area. 手势图片 The gesture key point network is improved based on the ST-GCN network. A spatiotemporal graph with joint coordinates as graph nodes, hand structure connections and time frame connections as edges is constructed as the input of the graph network. Gesture categories are learned based on gesture spatial characteristics, gesture motion state characteristics and gesture temporal information. Fig.10 The left side represents the gesture space features. Fig.10 The right side represents the gesture motion state characteristics.

[0227] Main module feature extraction:

[0228] 1) In order to learn the spatial characteristics of gestures, the following features are extracted as network input based on information multi-order difference learning.

[0229] First-order information - coordinate information and confidence of key points extracted based on openpose (x, y, p). This feature is used to learn gesture joint information.

[0230] Second-order information - extract edge direction information based on key points to obtain hand adjacency matrix A1, angle, length, normal vector and other features. This feature is used to learn gesture direction information;

[0231] Third-order information - palm plane inclination. This feature is used to learn the differences in different angles of the same gesture, such as the different inclination angles of the thumbs-up gesture.

[0232] 2) In order to learn the characteristics of gesture motion state, based on the characteristics between the torso and the gesture, the following features are extracted as network input. Fig.11 A schematic diagram of the relationship between a gesture and a torso position provided by an embodiment of the present invention, such as Fig.11 As shown, Figure 7 Where I0, I1, I2 and I3 represent the distance between the torso and the gesture, θ1 represents the angle between I0 and I1, and θ2 represents the angle between I2 and I3. Adjacency matrix weight partitioning obtains adjacency matrix A2: Select the neck as the center of the torso, and define the eccentricity or approximateness of the gesture based on the position information of the center point and the hand to obtain a new adjacency matrix. This feature is used to learn the spatial features between the hand and the torso. Angle information and direction information of the gesture and the center point of the torso.

[0233] 3) To learn the timing information of gestures, extract the relevant feature information of bones in different frames based on the timing change information of skeleton key points, and obtain the displacement change, velocity, and acceleration information of key points between frames.

[0234] Main module network architecture: Based on the gesture spatial characteristics and gesture motion state characteristics, an improved graph network learning is adopted. In terms of spatial characteristics, a learnable spatial weight matrix (of the same size as the adjacency matrix) is introduced and bitwise multiplied with the adjacency matrix A1 to obtain A act , which is used to give a larger weight to important edges (nodes) in the adjacency matrix and suppress the weight of unimportant edges (nodes); in terms of motion characteristics, a learnable motion weight matrix (of the same size as the adjacency matrix) is introduced and multiplied bitwise with the adjacency matrix A2 to obtain A struct , which is used to give a larger weight to the torso information in the adjacency matrix and suppress the weight of the non-important edges; based on A act With A struct The information and the above-mentioned skeleton sequence features are input into the GCN network to realize the aggregation of spatial dimension information;

[0235] In terms of temporal information, it is input into the TCN network, and its backbone network is changed to the Transformer network to better realize the aggregation of temporal dimension information; the T-GCN backbone network is obtained by fusing the GCN and improved TCN modules, and the features learned from the T-GCN network and the f learned from the image information are combined. 手势图片 Feature concatenation to obtain the gesture category feature f 手部-c .

[0236] In terms of gesture prediction, based on the features learned by the T-GCN network, an additional T-GCN module is added to predict the gesture action features f 手部-act , used to assist in gesture type recognition.

[0237] Facial expression recognition submodule (auxiliary module): As a visual language, the sound of sign language scenes cannot be materialized. However, in real scenes, stiff expressions often cause misunderstandings during communication, and emotional auxiliary expression is particularly important. Considering that emotional expression is often based on facial expressions and torso expressions. For example, emotions such as questions in sign language are usually expressed by gestures and expressions. For example, gestures expressing questions are generally expressed by raising eyebrows, opening eyes wide, and slightly opening mouth. Based on this, the model adds a facial expression recognition module to extract facial expression features for assistance.

[0238] Network architecture: Recognize facial expressions based on 68 key points and facial region images. The network module uses a 3-layer CNN convolutional network to extract pixel features for the facial region; uses a GCN graph network to learn the structural spatial information features of the facial key point graph based on the 68 key point skeleton graph; splices the two features as facial expression recognition features f 面部 .

[0239] Human torso submodule (auxiliary module): Gestures and emotions are often expressed with body movements. For example, gestures that express affirmation or negation are also accompanied by nodding, shaking the head, and other movements; there are also some correlations between some gestures and overall torso movements. Based on this, the model adds a human torso module to extract human torso information to replicate gesture recognition and emotion recognition.

[0240] Network architecture: Based on the human torso key points, the GCN network is input to extract the torso features f 躯干 .

[0241] Lip reading recognition submodule (auxiliary module): In fact, the gestures of many words in spoken language are relatively small, so it is difficult to recognize the correct meaning. Lip shape can effectively reduce language ambiguity. Based on this, the model adds spoken language recognition as network auxiliary information to assist gesture recognition.

[0242] Network architecture: Lip language recognition based on lip key points and lip area images. The network module uses the DenseNet network to extract pixel features for the lip area; based on the lip key point skeleton image, the GCN graph network is used to learn the graph structure spatial information features; the above two features are spliced ​​as the lip language recognition feature f 唇语 .

[0243] Fig.12 A flow chart of another sign language translation training model provided by an embodiment of the present invention is as follows: Fig.12 As shown in the figure, based on the features extracted by the facial expression recognition submodule, the human body submodule, the lip reading recognition submodule, and the gesture recognition and prediction submodule, the splicing features are used to obtain the corresponding emotion feature vector in the current frame. Gesture category feature vector and gesture prediction feature vector This completes the feature extraction of the current frame data.

[0244] The decoding details of each subtask in the decoding module are as follows:

[0245] Emotion classification task: based on the current frame emotion feature vector learned by the feature extraction module The final output feature vector obtained by GRU module based on time series learning is combined with softmax to obtain the emotion classification result.

[0246] Sign language text translation task: based on the feature vector of the gesture category in the current frame learned by the feature extraction module The Transformer module is used to predict the text sequence based on the self-attention mechanism to obtain the text translation result.

[0247] Gesture prediction task: The gesture prediction feature vector of the current frame learned by the feature extraction module Based on the GRU+MLP module, the predicted gesture key point sequence for each frame is obtained.

[0248] The specific steps of sign language interpreter training are as follows:

[0249] Step 2.1 Video frame extraction: Based on the full video stream data obtained in step 1.3.1, extract frames for each video and extract valid key frames with the local average difference between frames as the local maximum value.

[0250] Step 2.2 Key point acquisition: Use Openpose to obtain key point information corresponding to the key frame. Note: Due to the richness of the data (including full-body portraits, full-body portraits (only upper limbs and torso), with oral assistance, without oral assistance, and different facial expressions), some key points may not be obtained. Based on this situation, when extracting key points of the data, all key points that cannot be detected are filled with 0.

[0251] Step 2.3: Based on the key points of the face, lips, and torso and the image information, each is input into the corresponding network module to extract the features of each module;

[0252] Step 2.4: Based on the gesture area image and key point information, extract the hand key point coordinate information and confidence (x, y, p), angle, length, normal vector, palm plane inclination and other features; obtain the hand key point adjacency matrix A1 and the hand and torso key point adjacency matrix A2; input them into the gesture recognition and prediction module to extract the sign language prediction features and gesture category features;

[0253] Step 2.5, based on the feature information extracted in steps 2.3 and 2.4, concatenates the features separately and inputs them into the corresponding decoding network to obtain the sign language translation results, gesture action prediction results and emotion recognition results.

[0254] Step 2.6 updates the network parameters based on the results obtained in step 2.5 and the actual annotations in steps 1.2.1 and 1.2.2.

[0255] Step 2.7 The training process ends.

[0256] For ease of understanding, here is an example of the gesture translation method. Fig.13A flowchart of a sign language translation reasoning method provided by an embodiment of the present invention is shown in FIG. Fig.13 As shown, the specific steps are as follows:

[0257] Step 3.1: Obtain the 2D coordinate data of the human skeleton nodes in each frame through the posture estimation algorithm (such as OpenPose) from the RGB video collected in the actual scene. Determine whether the gesture key point information can be detected. If not, jump to step 3.2; otherwise, jump to step 3.3;

[0258] Step 3.2 implements video style transfer based on the method in step 1.3.1 using the RGB video collected in the actual scene, and inputs the migrated video into OpenPose to obtain the 2D coordinate data information of the human skeleton nodes in each frame.

[0259] Step 3.3 inputs the video data, detected key point information and image information into the sign language translation model trained in step 2 to obtain the sign language translation results, sign language emotion recognition results and sign language action prediction results. The sign language action prediction results can help improve the accuracy of gesture recognition in model training. When used in actual scenarios, they can also predict the next action of the signer in advance and help predict the signer's intention.

[0260] Step 3.4 The process ends.

[0261] In real-life scenarios, when communicating with signers, we can obtain the sign language semantic information corresponding to the video sequence (shot sign language video) based on the sign language translation results obtained in the above process, and help the deaf-mute translate sign language into text, so as to achieve normal communication. The current emotion of the signer can be obtained based on the sign language emotion recognition results to better determine the signer's emotions and assist communication. Based on the sign language action prediction results, the signer's next gesture can be predicted in advance, thereby predicting the signer's intention. Of course, based on some non-sign language recognition scenarios, such as boxing or wrestling matches, the gesture prediction part can be combined with gesture recognition to predict the user's next action, and the success rate of the game can be increased based on action prediction.

[0262] In order to implement the method of the embodiment of the present invention, the embodiment of the present invention also provides a training device for a gesture translation model. Fig.14 A schematic diagram of the structure of a training device for a gesture translation model provided by an embodiment of the present invention is shown in FIG. Fig.14 As shown, the device 1400 includes:

[0263] The first acquisition unit 1401 is used to acquire video data corresponding to the target person and target information corresponding to the gesture action when the target person performs a gesture action;

[0264] The recognition unit 1402 is used to recognize at least one action type of the video data to obtain first feature information;

[0265] A prediction unit 1403, configured to perform gesture action prediction on the gesture action based on the target information to obtain second feature information;

[0266] An input unit 1404, configured to input the first feature information and the second feature information into the gesture translation model to obtain a loss function corresponding to the gesture translation model;

[0267] The updating unit 1405 is used to update the parameters corresponding to the gesture translation model based on the loss function.

[0268] In an optional embodiment, the first acquisition unit 1401 is further used to obtain the annotation data and target video data corresponding to the target person when there is an abnormality in the video data; input the annotation data, the target video data, the first feature information and the second feature information into the gesture translation model to obtain the loss function.

[0269] In an optional embodiment, the device 1400 further includes a first determining unit, configured to determine the first labeled data, the second labeled data, and the third labeled data according to the video data.

[0270] In an optional embodiment, the device 1400 also includes a fusion unit, which is used to expand the gesture action to obtain style image information corresponding to the gesture action; determine the style migration data corresponding to the gesture action based on the video data and the style image information; and fuse the video data and the style migration data to obtain the target video data.

[0271] In an optional embodiment, the identification unit 1402 is further used to perform the first type identification on the video data to obtain the first sub-feature information; perform the second type identification on the video data to obtain the second sub-feature information; and perform the third type identification on the video data to obtain the third sub-feature information.

[0272] In an optional embodiment, the recognition unit 1402 is also used to obtain the first target frame information corresponding to the video data; perform facial information extraction on the first target frame information to obtain facial feature information corresponding to the video data; perform convolutional neural network CNN processing on the first target frame information to obtain third feature information; the third feature information represents pixel feature information corresponding to the target frame information; perform graph convolutional neural network GCN processing on the facial feature information to obtain fourth feature information; the fourth feature information represents spatial information feature information corresponding to the facial feature information; and splice the third feature information and the fourth feature information to obtain the first sub-feature information.

[0273] In an optional embodiment, the recognition unit 1402 is further used to obtain second target frame information corresponding to the video data; and extract torso information from the second target frame information to obtain the second sub-feature information.

[0274] In an optional embodiment, the recognition unit 1402 is also used to obtain third target frame information corresponding to the video data; perform lip information extraction on the third target frame information to obtain lip feature information corresponding to the video data; perform dense convolution network DenseNet processing on the third target frame information to obtain fifth feature information; the fifth feature information represents pixel feature information corresponding to the third target frame information; perform GCN processing on the lip feature information to obtain sixth feature information; the sixth feature information represents spatial information feature information corresponding to the lip feature information; and concatenate the fifth feature information and the sixth feature information to obtain the third sub-feature information.

[0275] In an optional embodiment, the prediction unit 1403 is further used to obtain a first parameter corresponding to the hand area in the video data and a second parameter corresponding to an area adjacent to the hand area; and determine the second feature information based on the first parameter and the second parameter.

[0276] In an optional embodiment, the prediction unit 1403 is further used to process the first parameter and the second parameter using GCN and a temporal convolutional network TCN to obtain the fourth sub-feature information; and to process the gesture category feature information using a temporal graph convolutional network T-GCN to obtain the fifth sub-feature information.

[0277] In an optional embodiment, the input unit 1404 is also used to concatenate the first feature information and the second feature information to obtain seventh feature information; input the seventh feature information into the gesture translation model to obtain a prediction result corresponding to the target person; and determine the loss function based on the prediction result and the labeled data.

[0278] In an optional embodiment, the input unit 1404 is further used to decode the sixth sub-feature information to obtain the emotion classification result; decode the seventh sub-feature information to obtain the text translation result; and decode the eighth sub-feature information to obtain the gesture prediction result.

[0279] In an optional embodiment, the input unit 1404 is also used to determine the emotion classification loss function based on the emotion classification result and the emotion category annotation data; determine the text translation loss function based on the text translation result and the text annotation data; determine the gesture action prediction loss function based on the gesture prediction result and the gesture annotation data.

[0280] It should be noted that the training device for the gesture translation model and the training method for the gesture translation model provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0281] In order to implement the method of the embodiment of the present invention, the embodiment of the present invention also provides a gesture translation device. Fig.15 A schematic diagram of the structure of a gesture translation device provided in an embodiment of the present invention is shown in FIG. Fig.15 As shown, the device 1500 includes:

[0282] The second acquisition unit 1501 is used to acquire video data corresponding to the target person when the target person performs a gesture action;

[0283] A determination unit 1502 is used to determine whether target information can be determined according to the video data, and if the target information is determined, input the target information into a gesture translation model to obtain a gesture translation result;

[0284] The gesture translation model is trained based on the gesture translation model training method provided in an embodiment of the present invention.

[0285] In an optional embodiment, the device 1500 further includes a second determination unit, configured to determine coordinate data corresponding to the target person according to the video data; the coordinate data is used to determine the target information.

[0286] In an optional embodiment, the device also includes a third determination unit, which is used to determine style migration data based on the video data when the target information is not determined; determine target video data based on the video data and the style migration data; and re-determine coordinate data corresponding to the target person based on the target video data.

[0287] It should be noted that the gesture translation device and the gesture translation method provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0288] Based on the hardware implementation of the above-mentioned program modules, and in order to implement the method of the embodiments of the present application, the embodiments of the present application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, it implements the training method of the gesture action translation model provided in the above-mentioned embodiments, or implements the gesture action translation method provided in the embodiments of the present invention.

[0289] Correspondingly, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the training method of the gesture action translation model provided in the above embodiment, or implements the gesture action translation method provided in an embodiment of the present invention.

[0290] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0291] It should be noted that Fig.16 A schematic diagram of a hardware entity structure of an electronic device provided in an embodiment of the present application, such as Fig.16 As shown, the hardware entity of the electronic device 1600 includes: a processor 1601 and a memory 1603 . Optionally, the electronic device 1600 may also include a communication interface 1602 .

[0292] It can be understood that the memory 1603 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 1603 described in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memories.

[0293] The method disclosed in the above embodiment of the present application can be applied to the processor 1601, or implemented by the processor 1601. The processor 1601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1601. The above processor 1601 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 1601 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 1603, and the processor 1601 reads the information in the memory 1603 and completes the steps of the above method in combination with its hardware.

[0294] In an exemplary embodiment, the device may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), field programmable gate array (FPGA), general processor, controller, microcontroller (MCU), microprocessor, or other electronic components to execute the aforementioned method.

[0295] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0296] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0297] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0298] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0299] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0300] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A training method for a gesture translation model, characterized in that: The method comprises: When the target person performs a gesture, obtaining video data corresponding to the target person and target information corresponding to the gesture; Performing at least one action type recognition on the video data to obtain first feature information; Performing gesture action prediction on the gesture action based on the target information to obtain second feature information; Inputting the first feature information and the second feature information into the gesture translation model to obtain a loss function corresponding to the gesture translation model; The parameters corresponding to the gesture translation model are updated based on the loss function.

2. The method according to claim 1, characterized in that The method further comprises: When there is an abnormality in the video data, obtaining the label data and target video data corresponding to the target person; The labeled data, the target video data, the first feature information, and the second feature information are input into the gesture translation model to obtain the loss function.

3. The method according to claim 2, characterized in that The annotation data includes first annotation data corresponding to the emotion category of the target person, second annotation data corresponding to the gesture action, and third annotation data corresponding to the translation of the gesture action into text; The obtaining of the labeling data corresponding to the target person includes: The first labeling data, the second labeling data and the third labeling data are determined according to the video data.

4. The method according to claim 2, characterized in that: The obtaining of target video data corresponding to the target person includes: Expanding the gesture action to obtain style image information corresponding to the gesture action; Determining style migration data corresponding to the gesture action according to the video data and the style picture information; The video data and the style migration data are fused to obtain the target video data.

5. The method according to claim 3, characterized in that: The at least one action type includes a first type, a second type and a third type; the first type represents the facial shape of the target person; the second type represents the torso shape of the target person; the third type represents the lip shape of the target person; the first feature information includes first sub-feature information, second sub-feature information and third sub-feature information; the at least one action type recognition is performed on the video data to obtain the first feature information, including: Performing the first type identification on the video data to obtain the first sub-feature information; Performing the second type identification on the video data to obtain the second sub-feature information; The third type identification is performed on the video data to obtain the third sub-feature information.

6. The method according to claim 5, characterized in that The performing the first type identification on the video data to obtain the first sub-feature information includes: Acquire first target frame information corresponding to the video data; Extracting facial information from the first target frame information to obtain facial feature information corresponding to the video data; Performing convolutional neural network (CNN) processing on the first target frame information to obtain third feature information; the third feature information represents pixel feature information corresponding to the target frame information; Performing graph convolutional neural network (GCN) processing on the facial feature information to obtain fourth feature information; the fourth feature information represents spatial information feature information corresponding to the facial feature information; The third feature information and the fourth feature information are concatenated to obtain the first sub-feature information.

7. The method according to claim 5, characterized in that The performing the second type identification on the video data to obtain the second sub-feature information includes: Acquire second target frame information corresponding to the video data; The torso information is extracted from the second target frame information to obtain the second sub-feature information.

8. The method according to claim 5, characterized in that The performing the third type identification on the video data to obtain the third sub-feature information includes: Acquire third target frame information corresponding to the video data; Extracting lip information from the third target frame information to obtain lip feature information corresponding to the video data; Performing dense convolution network DenseNet processing on the third target frame information to obtain fifth feature information; the fifth feature information represents pixel feature information corresponding to the third target frame information; Performing GCN processing on the lip feature information to obtain sixth feature information; the sixth feature information represents spatial information feature information corresponding to the lip feature information; The fifth characteristic information and the sixth characteristic information are concatenated to obtain the third sub-characteristic information.

9. The method according to claim 1, characterized in that: The performing gesture action prediction on the gesture action based on the target information to obtain second feature information includes: Acquire a first parameter corresponding to a hand region in the video data and a second parameter corresponding to a region adjacent to the hand region; The second feature information is determined based on the first parameter and the second parameter.

10. The method according to claim 9, characterized in that The second feature information includes fourth sub-feature information and fifth sub-feature information; the fourth sub-feature information represents feature information of a gesture type corresponding to the gesture action; The fifth sub-feature information represents the predicted feature information corresponding to the gesture action; Determining the second feature information based on the first parameter and the second parameter includes: Processing the first parameter and the second parameter using GCN and a temporal convolutional network TCN to obtain the fourth sub-feature information; The gesture category feature information is processed by a temporal graph convolutional network T-GCN to obtain the fifth sub-feature information.

11. The method according to claim 3, characterized in that The step of inputting the first feature information and the second feature information into the gesture translation model to obtain a loss function corresponding to the gesture translation model includes: Concatenate the first characteristic information and the second characteristic information to obtain seventh characteristic information; Inputting the seventh feature information into the gesture translation model to obtain a prediction result corresponding to the target person; The loss function is determined based on the prediction result and the labeled data.

12. The method according to claim 11, characterized in that The seventh feature information includes sixth sub-feature information, seventh sub-feature information and eighth sub-feature information; the sixth sub-feature information represents the emotion information corresponding to the target person; the seventh sub-feature information represents the information of the gesture category corresponding to the gesture action; The eighth sub-feature information represents information for predicting the gesture action; the prediction result includes an emotion classification result, a text translation result and a gesture prediction result; The step of inputting the seventh feature information into the gesture translation model to obtain a prediction result corresponding to the target person includes: Decoding the sixth sub-feature information to obtain the emotion classification result; Decoding the seventh sub-feature information to obtain the text translation result; The eighth sub-feature information is decoded to obtain the gesture prediction result.

13. The method according to claim 12, characterized in that The loss function includes an emotion classification loss function, a gesture action prediction loss function and a text translation loss function; and determining the loss function based on the prediction result and the labeled data includes: Determine the emotion classification loss function according to the emotion classification result and the emotion category labeling data; Determine the text translation loss function based on the text translation result and the text annotation data; The gesture action prediction loss function is determined according to the gesture prediction result and the gesture annotation data.

14. A gesture translation method, characterized in that: The method comprises: When the target person performs a gesture, obtain video data corresponding to the target person; determining whether target information can be determined according to the video data, and if the target information can be determined, inputting the target information into a gesture translation model to obtain a gesture translation result; The gesture translation model is trained based on the method described in any one of claims 1 to 13 above.

15. The method according to claim 14, characterized in that After obtaining the video data corresponding to the target person when the target person performs a gesture, the method further includes: The coordinate data corresponding to the target person is determined according to the video data; the coordinate data is used to determine the target information.

16. The method according to claim 15, characterized in that The method further comprises: determining style transfer data based on the video data without determining the target information; Determine target video data according to the video data and the style migration data; The coordinate data corresponding to the target person is re-determined based on the target video data.

17. A training device for a gesture translation model, characterized in that: The device comprises: A first acquisition unit, configured to acquire video data corresponding to the target person and target information corresponding to the gesture action when the target person performs a gesture action; an identification unit, configured to identify at least one action type of the video data to obtain first feature information; A prediction unit, configured to perform gesture action prediction on the gesture action based on the target information to obtain second feature information; An input unit, used for inputting the first feature information and the second feature information into the gesture translation model to obtain a loss function corresponding to the gesture translation model; An updating unit is used to update parameters corresponding to the gesture translation model based on the loss function.

18. A gesture translation device, characterized in that: The device comprises: A second acquisition unit is used to acquire video data corresponding to the target person when the target person performs a gesture action; a judgment unit, configured to judge whether target information can be determined according to the video data, and if the target information is determined, input the target information into a gesture translation model to obtain a gesture translation result; The gesture translation model is trained based on the method described in any one of claims 1 to 13 above.

19. An electronic device, characterized in that: include: A memory for storing executable instructions; A processor, for implementing the training method of the gesture action translation model described in any one of claims 1 to 13, or implementing the gesture action translation method described in any one of claims 14 to 16, when executing the executable instructions stored in the storage.

20. A storage medium, characterized in that: The storage medium stores executable instructions. When the executable instructions are executed by at least one processor, the training method of the gesture translation model described in any one of claims 1 to 13 or the gesture translation method described in any one of claims 14 to 16 is implemented.

Images