Sign language recognition method, system, device and storage medium

Abstract

The application discloses a sign language recognition method, system, device and storage medium, and training is divided into two training stages of self-supervised pre-training and downstream fine-tuning. In the self-supervised pre-training, existing sign language data (unlabeled data) can be fully utilized. Specifically, pseudo labels are generated by using the sign language data, hidden semantic representations of pose triplets are covered by using a covering tuple modeling manner, hidden semantic representations corresponding to the covered tuples are reconstructed by capturing context clues, and conversion relationships between poses are learned. Then, in the downstream fine-tuning stage, model parameters are fine-tuned to improve the recognition accuracy of the sign language recognition model. Through the above training, the sign language data corresponding vocabulary can be accurately recognized in the recognition stage (downstream sign language isolated word classification task).

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sign language recognition, and in particular to a sign language recognition method, system, device and storage medium. BACKGROUND

[0002] According to the statistical data of the World Health Organization (WHO) in 2021, there are about 1.5 billion people with hearing impairment in the world, accounting for about one fifth of the total global population. Among the hearing-impaired population, sign language is the most commonly used communication medium. As a visual language, sign language has its own grammar. It mainly expresses semantic information through hand control features (hand shape, hand movement and position, etc.), supplemented by fine-grained non-hand control features (expression, lip shape, etc.).

[0003] In order to solve the communication barrier between hearing people and deaf people, sign language recognition has emerged and been widely studied. Through a sign language recognition scheme, an input sign language video is converted into corresponding text. Isolated word sign language recognition is a basic task, which recognizes the input sign language video as the corresponding word of the video. The usual recognition process is to first extract the representation of the input sign language video, and then convert the representation into a probability vector, and take the class corresponding to the maximum probability as the recognition result. Early recognition systems mostly use hand-designed features. With the development of deep learning and hardware computing power in recent years, sign language recognition systems based on deep learning have gradually dominated. It extracts the representation through a convolutional neural network (CNN), and then converts the representation into a probability vector through a fully connected and Softmax layer, and takes the class corresponding to the maximum probability as the recognition result.

[0004] Since the annotation of sign language datasets requires a large amount of human cost, the current labeled sign language data is of a small order of magnitude. At the same time, the current deep learning-based methods are all data-driven models, so they are prone to overfitting on a single sign language recognition dataset, thereby reducing their generalization ability. Recently, some sign language recognition methods have tried to solve this problem. Some works annotate more sign language data to supervise the training of deep learning networks, and some works introduce hand model priors as intermediate constraints, so as to better represent the hand posture information. However, most of the methods are directly optimized on the target dataset, without fully utilizing the existing large amount of unlabeled sign language data.

[0005] Notably, self-supervised pre-training techniques, such as BERT, have achieved great success in the field of natural language processing (NLP). BERT pre-trains the model by reconstructing the masked words from the randomly masked input text sequence, and then supervises the generated words through a cross-entropy loss function. However, due to the different characteristics of the input signal, there are great obstacles in directly migrating this pre-training mode to the field of sign language recognition. In the field of NLP, the input words are discrete, pre-defined, and have high semantics. Unlike this, the video signal in sign language data is continuous, with continuous spatial and temporal dimensions. This signal is quite low-level, making the original BERT loss function inapplicable. In addition, since the main feature of sign language data is the movement of hands and body, directly adopting the BERT framework may not be the best choice. SUMMARY

[0006] The purpose of the present application is to provide a sign language recognition method, system, device and storage medium, which can make the most of the existing sign language data for pre-training, train a more robust sign language model, improve the accuracy of sign language recognition, and have a wider range of application scenarios.

[0007] The purpose of the present application is achieved by the following technical solutions:

[0008] A sign language recognition method, comprising:

[0009] constructing an initial sign language recognition model;

[0010] The initial sign language recognition model is trained in two stages using a gesture motion sequence. The first stage is self-supervised pre-training. The initial sign language recognition model includes a gesture quantizer, a gesture embedding layer, a time sequence encoder, and a code word decoder. Each gesture triple in the gesture motion sequence is input into the gesture quantizer and the gesture embedding layer. The gesture quantizer is used to generate pseudo labels corresponding to each gesture triple. The gesture embedding layer is used to map each gesture triple to a corresponding hidden semantic representation. The hidden semantic representations corresponding to several tuples in each gesture triple are covered and then input into the time sequence encoder using a tuple covering modeling method. The time sequence encoder reconstructs the hidden semantic representations corresponding to the covered tuples by capturing context clues and outputs an encoding sequence corresponding to the gesture motion sequence. The code word decoder predicts the labels of each covered tuple using the encoding sequence and the positions of the covered tuples in each gesture triple. A loss function for the first stage is constructed by combining the pseudo labels, and the initial sign language recognition model is optimized. The covered tuples refer to the tuples whose hidden semantic representations are covered. The second stage is a fine-tuning stage. After the first stage is completed, the gesture quantizer is removed and replaced with a prediction head. A final sign language recognition model including a gesture embedding layer, a time sequence encoder, and a prediction head is obtained. The gesture motion sequence is input, each gesture triple is mapped to a corresponding hidden semantic representation by the gesture embedding layer, and an encoding sequence corresponding to the gesture motion sequence is obtained by the time sequence encoder. The prediction head is then used to predict a word sequence. The loss function for the second stage is constructed using the real labels corresponding to the gesture motion sequence and the predicted word sequence to fine-tune the parameters, and a trained sign language recognition model is obtained. The gesture motion sequence used in the first stage is unlabeled data, and the gesture motion sequence used in the second stage is labeled data.

[0011] In the sign language recognition stage, the gesture motion sequence to be recognized is input into the trained sign language recognition model to obtain a predicted word sequence.

[0012] A sign language recognition system includes:

[0013] An initial sign language recognition model construction unit is configured to construct an initial sign language recognition model.

[0014] The training unit is configured to perform two-stage training on the initial sign language recognition model by using the gesture motion sequence. The first stage is self-supervised pre-training. The initial sign language recognition model comprises a gesture quantizer, a gesture embedding layer, a time sequence encoder, and a code word decoder. Each gesture triple in the gesture motion sequence is input into the gesture quantizer and the gesture embedding layer. The gesture quantizer is configured to generate pseudo labels corresponding to each gesture triple by quantization. The gesture embedding layer is configured to map each gesture triple into corresponding hidden semantic representations, and the hidden semantic representations corresponding to several tuples in each gesture triple are input into the time sequence encoder in a manner of tuple masking. The time sequence encoder is configured to reconstruct the hidden semantic representations corresponding to the masked tuples by capturing context clues, and output an encoding sequence corresponding to the gesture motion sequence. The code word decoder is configured to predict the labels of each masked tuple by using the encoding sequence and the positions of the masked tuples in each gesture triple, and construct a loss function of the first stage by combining the pseudo labels, so as to optimize the initial sign language recognition model. The masked tuple refers to a tuple whose hidden semantic representation is masked. The second stage is a fine-tuning stage. After the training of the first stage is completed, the gesture quantizer is removed and the code word decoder is replaced with a prediction head, so as to obtain a final sign language recognition model comprising the gesture embedding layer, the time sequence encoder, and the prediction head. The gesture motion sequence is input into the final sign language recognition model. The gesture embedding layer is configured to map each gesture triple into corresponding hidden semantic representations, and the time sequence encoder is configured to obtain an encoding sequence corresponding to the gesture motion sequence. The prediction head is configured to predict a word sequence. The gesture motion sequence corresponding to the real labels and the predicted word sequence are used to construct a loss function of the second stage, so as to fine-tune the parameters and obtain a trained sign language recognition model. The gesture motion sequence used in the first stage is unlabeled data, and the gesture motion sequence used in the second stage is labeled data.

[0015] The sign language recognition unit is configured to input a gesture motion sequence to be recognized into the trained sign language recognition model, and obtain a predicted word sequence.

[0016] A processing device, comprising: one or more processors; a memory configured to store one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the foregoing method.

[0018] A readable storage medium storing a computer program, when the computer program is executed by a processor, the foregoing method is implemented.

[0019] From the technical solutions provided by the application, the training is divided into two training stages of self-supervised pre-training and downstream fine-tuning, the existing sign language data (unlabeled data) can be fully utilized in the self-supervised pre-training, specifically: pseudo labels are generated by using the sign language data, the hidden semantic representation of the pose triplets is covered by using the covering tuple modeling manner, the hidden semantic representation corresponding to the covered tuple is reconstructed by capturing context clues to learn the conversion relationship between the poses; then, the model parameters are fine-tuned through the downstream fine-tuning stage to improve the recognition accuracy of the sign language recognition model; through the above training, the sign language data corresponding vocabulary can be accurately recognized in the recognition stage (isolated word sign language recognition). BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] Figure 1 A flowchart of a sign language recognition method provided by the embodiment of the present application;

[0022] Figure 2 A framework diagram of a sign language recognition method provided by the embodiment of the present application;

[0023] Figure 3 A schematic diagram of a sign language recognition system provided by the embodiment of the present application;

[0024] Figure 4 A schematic diagram of a processing device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0026] First, the terms that can be used in this paper are explained as follows:

[0027] The term "and / or" means either one or both, for example, X and / or Y means three cases including "X", "Y" or "X and Y".

[0028] The terms "comprising", "containing", "including", "having" or other similar semantic descriptions should be interpreted to be non-exclusive inclusion. For example: including a technical feature element (such as raw materials, components, ingredients, carriers, dosage forms, materials, sizes, parts, components, mechanisms, devices, steps, processes, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or articles, etc.) should be interpreted as not only including the explicitly listed technical feature element, but also including other technical feature elements not explicitly listed but known in the art.

[0029] A sign language recognition method, system, device and storage medium provided by the present application are described in detail below. The content not described in detail in the embodiments of the present application belongs to the prior art known to those skilled in the art. If no specific conditions are specified in the embodiments of the present application, the conventional conditions or the conditions recommended by the manufacturer are followed.

[0030] Embodiment one

[0031] The embodiments of the present application provide a sign language recognition method, which designs a pre-training scheme for sign language recognition. The pre-training scheme includes two stages. Given a sequence of pose motions, in the self-supervised pre-training, the pose triplets of each frame are first converted into hidden semantic representations. Then, the input triplet sequence is polluted by using the occlusion triplet modeling method, and the hidden semantic representations corresponding to the occluded triplets are reconstructed by capturing context clues. Finally, the code word decoder uses the hidden layer features to predict the labels of the occluded triplets. In order to better learn the conversion relationship between gestures, a quantizer is trained in advance to discretize the pose triplets as pseudo labels to supervise the network training. The entire framework uses a video-level cross-entropy loss function for training. After the pre-training is completed, a prediction head is introduced for fine-tuning, so that it can be used to perform a downstream sign language isolated word classification task.

[0032] Specifically, as shown in Figure 1 The main flow of the sign language recognition method provided by the embodiments of the present application can be described as:

[0033] Step 1, constructing an initial sign language recognition model.

[0034] This step constructs an initial sign language recognition model. The model structure will be adjusted in the subsequent two-stage training.

[0035] Step 2, using a sequence of pose motions to perform two-stage training on the initial sign language recognition model.

[0036] 1) The first stage is self-supervised pre-training: the initial sign language recognition model includes a pose quantizer, a pose embedding layer, a temporal encoder, and a code word decoder; each pose triplet in the pose motion sequence (unlabeled sign language data) is input into the pose quantizer and the pose embedding layer, the pose quantizer is used for quantization to generate the corresponding pseudo label of each pose triplet; the pose embedding layer is used to map each pose triplet into the corresponding hidden semantic representation, and the hidden semantic representation corresponding to several tuples in each pose triplet is input into the temporal encoder in a way of covering tuples; the temporal encoder reconstructs the hidden semantic representation corresponding to the covered tuples by capturing context clues, and outputs the encoding sequence corresponding to the pose motion sequence; the code word decoder predicts the label of each covered tuple using the encoding sequence and the position of the covered tuple in each pose triplet, and combines the pseudo label to construct the loss function of the first stage, optimizes the initial sign language recognition model, and the covered tuple refers to the tuple whose hidden semantic representation is covered.

[0037] 2) The second stage is the fine-tuning stage. After the training of the first stage, the pose quantizer is removed and the prediction head is used to replace the code word decoder, obtaining the final sign language recognition model including the pose embedding layer, the temporal encoder, and the prediction head. The pose motion sequence (labeled sign language data) is input, each pose triplet is mapped into the corresponding hidden semantic representation through the pose embedding layer, and the encoding sequence corresponding to the pose motion sequence is obtained through the temporal encoder, and then the word sequence is predicted through the prediction head. The real label corresponding to the pose motion sequence and the predicted word sequence are used to construct the loss function of the second stage to fine-tune the parameters, and the trained sign language recognition model is obtained.

[0038] Step 3, sign language recognition stage, input the pose motion sequence to be recognized into the trained sign language recognition model to obtain the predicted word sequence.

[0039] Figure 2The diagram illustrates the framework of the sign language recognition method. The left-hand section, Input PoseTriplet Unit, represents the pose triplet input unit. Each column represents a pose triplet: Body represents the body pose, L-Hand represents the left-hand pose, and R-Hand represents the right-hand pose. Above it, Pre-Trained Tokenizer represents the pre-trained pose quantizer, and its output, Pose Triplet Unit Label, serves as a pseudo-label for the pose triplet, acting as a supervisory signal for the codeword decoder. The right-hand section, Pose Embedding Layer, represents the pose embedding layer. Transformer Encoder is the Transformer encoder, which functions as the temporal encoder in this invention, and Decoder is the codeword decoder. This framework is primarily used in the self-supervised pre-training stage. After entering the fine-tuning stage, the pose quantizer is removed, and the codeword decoder is replaced with a prediction head.

[0040] To more clearly demonstrate the technical solution and its effects provided by the present invention, a sign language recognition method provided by the present invention will be described in detail below with reference to specific embodiments.

[0041] 1. Pre-train the attitude quantizer.

[0042] During self-supervised pre-training, the pose quantizer provides pseudo-labels for the self-supervised pre-training task. In this embodiment of the invention, the pose quantizer can be implemented by a discrete autoencoder (d-VAE), which is responsible for converting each pose triplet (pose skeleton point triplet) in the pose motion sequence into a codeword triplet as a pseudo-label.

[0043] The pose quantizer needs to be pre-trained before self-supervised pre-training; the pose quantizer includes an encoder, a quantizer, and a decoder.

[0044] During pre-training, the input pose triple J sign The intermediate hidden layer vector z corresponding to the pose triple is obtained through the encoder and is represented as:

[0045] z = Concat(z) l , z r , z b )

[0046] Where Concat represents the concatenation operation, z l , z r , z b This represents the intermediate hidden vector (local representation) corresponding to each tuple in the pose triplet. Each pose triplet contains the left-hand pose J. left Right hand posture J right With body posture Jbody Each pose is a tuple, l, r, b are the labels of left hand, right hand and body respectively.

[0047] The intermediate hidden layer vector is discretized by a quantizer combined with a codebook, obtaining a quantized vector z q , which is expressed as:

[0048]

[0049] Wherein:

[0050]

[0051]

[0052]

[0053] In the above formula, Q_hand and Q_body are the hand quantizer and the body quantizer respectively, which together constitute the quantizer, and they can map the corresponding intermediate hidden layer vector of the tuple to the index in the codebook; h k is the kth codeword in the hand codebook, z k′ is the k'th codeword in the body codebook, k l and k r are the codeword indexes selected from the hand codebook corresponding to the left hand pose and the right hand pose, and are the corresponding codewords of the codeword indexes k l and k r , k b is the codeword index selected from the body codebook (the codebook of the upper body) corresponding to the body pose, is the corresponding codeword of the codeword index k b ; the hand codebook is expressed as The body codebook is expressed as M1 and M2 are the number of codewords in the corresponding codebook. It can be seen from the above expression that the quantized vector z q is composed of three codewords in two codebooks.

[0054] The pose triplet is reconstructed by the decoder combined with the quantized vector, which is expressed as:

[0055]

[0056] Wherein, are the reconstructed left hand pose, right hand pose and body pose respectively.

[0057] A training objective function is constructed by combining the input pose triplets and the reconstructed pose triplets, and the pose quantizer is trained. Considering that the quantization process is not differentiable, a pass-through estimator is used to copy the gradient from the decoder to the encoder. The training objective function is defined as:

[0058]

[0059]

[0060]

[0061] wherein, is the training objective function, represents a hand pose reconstruction loss term, represents a body pose reconstruction loss term, and is a quantization loss term, wherein represents a gradient backpropagation truncation operation, β1, β2 and β3 are proportional coefficients of each loss term.

[0062] After training, the decoder is removed, and in the first stage, the pseudo label corresponding to each pose triplet is generated by the trained encoder and quantizer.

[0063] II. Self-supervised pre-training.

[0064] In self-supervised pre-training, the training object is an initial sign language recognition model, which includes a pose quantizer, a pose embedding layer (pose latent vector mapping layer), a time sequence encoder and a code word decoder. The data used in self-supervised pre-training is a pose motion sequence represented as wherein, T represents the number of frames of the pose motion sequence, t is the frame number, and each frame is a pose triplet.

[0065] 1. Pose quantizer.

[0066] The pose motion sequence is input to the pose quantizer, and the pose quantizer generates a pseudo label sequence As a supervision signal for self-supervised pre-training, k t is the pseudo label corresponding to the t-th frame.

[0067] 2. Pose embedding layer.

[0068] The pose motion sequence is input to the pose embedding layer, and the pose embedding layer maps each pose triplet to a corresponding latent semantic representation, and uses the way of covering tuples to cover the latent semantic representations corresponding to several tuples in each pose triplet and then input to the time sequence encoder. Specifically:

[0069] 1) For frame t, the pose embedding layer maps the pose to the corresponding latent semantic representation, denoted as f. sign，t , where f sign，t It is composed of three parts: latent semantic representation (local representation) of the left hand, right hand, and body posture. The latent semantic representation sequence corresponding to the final posture motion sequence is denoted as: F sign =(f sign ,1,…,f sign，T ).

[0070] 2) The latent semantic representations corresponding to several tuples in each pose triplet are masked using the masking tuple modeling method, as follows:

[0071] F m =Mask(F sign )=[f mask，1 , ..., f mask，T ]

[0072] Where Mask(·) represents the mask operator for the latent semantic representation sequence, F m Denotes the mask sequence, f mask，t f represents the features in frame t after the masking operation. mask，t It also includes three parts: left hand, right hand, and body pose. The latent semantic representation of one or more poses is randomly masked; of course, it's also possible that none of the three parts are masked. This is a random process. The masking operation here refers to replacing the latent semantic representation of the tuple with a learnable feature. The set of mask locations is represented as... α represents the shielding efficiency.

[0073] 3) The final latent semantic representation of each pose triplet includes the feature sequence processed by the masking operator and the temporal feature sequence. The final latent semantic representation of the pose triplet corresponding to frame t is as follows:

[0074] f in，t =f mask，t +f temp，t

[0075] Among them, f temp，t The temporal feature sequence corresponding to frame t is represented by the structure of the Transformer model itself, which consists of a series of cosine and sine functions.

[0076] The sequence F0 = [f] is formed by the final latent semantic representations of all corresponding pose triples. in，1 , ..., f in，T Input to the timing encoder.

[0077] 3. Timing encoder.

[0078] In the embodiment of the present application, the time sequence encoder is a time sequence encoder based on a Transformer structure, which reconstructs the hidden semantic representation corresponding to the covered tuple by capturing context clues, and outputs an encoding sequence corresponding to the posture motion sequence. Specifically, the time sequence encoder includes N layers of Transformer modules, denoted as Block(·), the input of the first layer is the output of the posture embedding layer, and the input of each subsequent layer is the output of the previous layer. The output of the jth layer is denoted as: j Block(F j-1 ); where j = 1, …, N, F l-1 represents the output of the (j-1)th layer, and when j = 1, F0 is the output of the posture embedding layer; the output F N of the Nth layer is taken as the encoding sequence corresponding to the posture motion sequence output by the time sequence encoder.

[0079] 4. Code word decoder.

[0080] Given the encoding sequence F N , the position set of the mask and the pseudo label sequence , the code word decoder is used to reconstruct the posture triple. Since the encoding vector corresponding to each frame includes three component parts, for example, the encoding vector f out，t corresponding to the tth frame in the encoding sequence includes (i.e., components corresponding to the left hand, right hand, and body), when the hidden semantic representation corresponding to the posture is randomly covered as described above, whether different tuples are covered is mutually independent, so the mask position sets of the right hand, left hand, and body are three mutually independent sets and do not completely overlap. The mask position sets of the right hand, left hand, and body are denoted as and satisfy where each mask position in the mask position set corresponds to a frame number, so the above three mask position sets can also be referred to as covered frame sets.

[0081] For the final hidden semantic representation f out，t corresponding to each frame, the Softmax operation is used to predict the label corresponding to the covered tuple:

[0082]

[0083]

[0084]

[0085] where, Let W1 and W2 represent the sets of frames whose latent semantic representations corresponding to left-hand pose, right-hand pose, and body pose are occluded, respectively; W1 and W2 are weight parameters in the codeword decoder, and b1 and b2 are bias terms in the codeword decoder; when When the latent semantic representation corresponding to the left-hand pose in frame t is masked, the encoding vector f of frame t is used. out，t middle left hand component Predict the label corresponding to the left-hand pose in frame t. when When the right hand gesture corresponding to the t-th frame is masked, the latent semantic representation is masked using the encoding vector f of the t-th frame. out，t Right-hand component Predict the label corresponding to the right hand pose in frame t. when When the latent semantic representation corresponding to the body pose in frame t is masked, the encoding vector f of frame t is used. out，t medium body weight Predict the label corresponding to the body pose in frame t.

[0086] 5. Loss function for the first stage.

[0087] In this embodiment of the invention, the ultimate goal is to achieve the desired result for a given mask sequence F. m Under the condition of maximizing the maximum likelihood of the correct label, the loss function of the first stage (video-level cross-entropy loss function) is expressed as:

[0088]

[0089] in, This represents the loss function for the first stage. This represents a sequence of multiple posture motions V sign The training corpus constituted k represents the expectation. t For pseudo-labels, p(.|.) represents the predicted probability distribution, and the right-hand side (V sign The ) represents the condition, the left side represents the output variable, and p represents the probability of the output variable.

[0090] III. Fine-tuning phase (downstream task fine-tuning).

[0091] After self-supervised pre-training, the parameters are directly fine-tuned in a downstream sign language word classification task. A prediction head is used to replace the pre-trained code word decoder, which can be implemented by a fully connected layer, mainly used for label classification to obtain the corresponding word of the posture triple, and this stage uses a posture motion sequence with labels, so the posture quantizer is removed. During fine-tuning, the input posture motion sequence is no longer masked, and supervised training is performed using the corresponding true label. That is: input the posture motion sequence, map each posture triple to the corresponding hidden semantic representation through the posture embedding layer, obtain the encoding sequence corresponding to the posture motion sequence through the time sequence encoder, and then predict the word sequence through the prediction head. The true label corresponding to the posture motion sequence and the predicted word sequence are used to construct the loss function of the second stage to fine-tune the parameters, and obtain the trained sign language recognition model. The loss function of this stage also uses a cross-entropy loss function.

[0092] It should be noted that the three parts introduced above all need to be trained through the corresponding loss function (objective function) to optimize the model parameters of the relevant part, but considering that the related processing flow can be implemented according to conventional technology, therefore, no further description is made.

[0093] Four, sign language recognition stage.

[0094] After the above fine-tuning stage, the final sign language recognition model can be obtained, and isolated word sign language recognition is applied. The recognition process is similar to that of the fine-tuning stage, but there is no need to update the model parameters.

[0095] From the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application.

[0096] Embodiment two

[0097] The present application also provides a sign language recognition system, which is mainly based on the method provided in the foregoing embodiments, as shown in the figure, the system mainly includes: Figure 3

[0098] An initial sign language recognition model construction unit is used to construct an initial sign language recognition model.

[0099] ​The training unit is used to perform two-stage training on the initial sign language recognition model using a pose motion sequence. The first stage is self-supervised pre-training: the initial sign language recognition model includes a pose quantizer, a pose embedding layer, a temporal encoder, and a codeword decoder. Each pose triplet in the pose motion sequence is input into the pose quantizer and the pose embedding layer, respectively. The pose quantizer quantizes the triplet to generate a pseudo-label corresponding to each pose triplet. The pose embedding layer maps each pose triplet to a corresponding latent semantic representation, and uses a masking tuple modeling method to mask the latent semantic representations corresponding to several tuples in each pose triplet before inputting them into the temporal encoder. The temporal encoder reconstructs the latent semantic representations corresponding to the masked tuples by capturing contextual cues and outputs the encoded sequence corresponding to the pose motion sequence. The codeword decoder uses the encoded sequence and the position of the masked tuples in each pose triplet to predict each... The first stage involves using labels for masked tuples and combining them with pseudo-labels to construct a loss function for the initial sign language recognition model. The masked tuples refer to tuples whose latent semantic representations are masked. The second stage is a fine-tuning stage. After completing the first stage of training, the pose quantizer is removed, and the codeword decoder is replaced with a prediction head to obtain the final sign language recognition model, which includes a pose embedding layer, a temporal encoder, and a prediction head. A pose motion sequence is input, and each pose triple is mapped to its corresponding latent semantic representation through the pose embedding layer. The temporal encoder then obtains the encoded sequence corresponding to the pose motion sequence, and the prediction head predicts the word sequence. The second stage loss function is constructed using the real labels corresponding to the pose motion sequence and the predicted word sequence for parameter fine-tuning, resulting in the trained sign language recognition model. The pose motion sequence used in the first stage is unlabeled data, while the pose motion sequence used in the second stage is labeled data.

[0100] The sign language recognition unit is used to input the posture motion sequence to be recognized into the trained sign language recognition model to obtain the predicted word sequence.

[0101] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0102] Example 3

[0103] The present invention also provides a processing device, such as Figure 4 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0104] Further, the processing device further comprises at least one input device and at least one output device; in the processing device, the processor, the memory, the input device and the output device are connected through a bus.

[0105] In the embodiment of the present application, the specific types of the memory, the input device and the output device are not limited; for example:

[0106] The input device can be a touch screen, an image acquisition device, a physical button or a mouse, etc.

[0107] The output device can be a display terminal.

[0108] The memory can be a random access memory (RAM), and can also be a non-volatile memory, for example, a disk memory.

[0109] Embodiment four

[0110] The present application also provides a readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the method provided by the foregoing embodiments is realized.

[0111] In the embodiment of the present application, the readable storage medium as the computer readable storage medium can be arranged in the foregoing processing device, for example, as the memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk and various media which can store program codes.

[0112] The above is only the preferred specific implementation of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for sign language recognition, characterized in that, include: Construct an initial sign language recognition model; The initial sign language recognition model is trained in two stages using a sequence of gesture movements. The first stage is self-supervised pre-training: the initial sign language recognition model includes a pose quantizer, a pose embedding layer, a temporal encoder, and a codeword decoder. Each pose triple in the pose motion sequence is input into the pose quantizer and the pose embedding layer, respectively. The pose quantizer quantizes the triple to generate a pseudo-label for each pose triple. The pose embedding layer maps each pose triple to a corresponding latent semantic representation, and uses a masking tuple modeling method to mask the latent semantic representations corresponding to several tuples in each pose triple before inputting them into the temporal encoder. The temporal encoder reconstructs the latent semantic representations corresponding to the masked tuples by capturing contextual cues and outputs the encoded sequence corresponding to the pose motion sequence. The codeword decoder uses the encoded sequence and the position of the masked tuples in each pose triple to predict the label of each masked tuple, and combines it with the pseudo-labels to construct... The first stage loss function optimizes the initial sign language recognition model, where the occlusion tuples refer to the tuples whose latent semantic representations are occluded. The second stage is the fine-tuning stage. After completing the first stage training, the pose quantizer is removed and the codeword decoder is replaced with a prediction head to obtain the final sign language recognition model, which includes a pose embedding layer, a temporal encoder, and a prediction head. A pose motion sequence is input, and each pose triple is mapped to its corresponding latent semantic representation through the pose embedding layer. The temporal encoder obtains the encoded sequence corresponding to the pose motion sequence, and the prediction head predicts the word sequence. The second stage loss function is constructed using the real labels corresponding to the pose motion sequence and the predicted word sequence to fine-tune the parameters, resulting in the trained sign language recognition model. The pose motion sequence used in the first stage is unlabeled data, while the pose motion sequence used in the second stage is labeled data. In the sign language recognition stage, the sequence of posture movements to be recognized is input into the trained sign language recognition model to obtain the predicted word sequence.

2. The method of claim 1, wherein, The pose quantizer needs to be pre-trained before self-supervised pre-training; the pose quantizer includes: an encoder, a quantizer, and a decoder; During pre-training, the input pose triplet J sign The intermediate hidden layer vector z corresponding to the pose triplet is obtained by the encoder, denoted as: z = Concat(z l , z r , z b ) where Concat denotes a concatenation operation, z l , z r , z b denote the intermediate hidden layer vectors corresponding to each tuple in the pose triplet, each pose triplet contains left hand pose J left , right hand pose J right and body pose J body , each pose is a tuple, l, r, b are the markers of left hand, right hand and body respectively; The intermediate hidden layer vector is discretized by a quantizer combined with a codebook to obtain a quantized vector z q is expressed as: in: In the above equation, Q_hand and Q_body are the hand quantizer and the body quantizer, respectively, which together form the quantizer; h k is the kth codeword in the hand codebook, z k′ is the k'th codeword in the body codebook, k l and k r are the codeword indices selected from the hand codebook corresponding to the left hand pose and the right hand pose, respectively, and are the corresponding codeword indices k l and k r are the corresponding codewords, k b is the codeword index selected from the body codebook corresponding to the body pose, is the corresponding codeword index k b is the corresponding codeword. The pose triplet is reconstructed by the decoder and the quantization vector, and is represented as follows: wherein, are reconstructed left-hand pose, right-hand pose and body pose, respectively; A training objective function is constructed by combining the input pose triplet and the reconstructed pose triplet, and the pose quantizer is trained. After training is completed, the decoder is removed. In the first stage, pseudo-labels corresponding to each pose triplet are generated by the trained encoder and quantizer.

3. The method of claim 2, wherein, The training objective function constructed by combining the input pose triples and the reconstructed pose triples is expressed as follows: wherein, is a training objective function, denotes a hand pose reconstruction loss term, denotes a body pose reconstruction loss term, and is a quantization loss term, denotes a gradient backpropagation truncation operation, and β1, β2, and β3 are proportionality coefficients for each loss term.

4. The method of claim 1, wherein, The step of mapping each pose triple to a corresponding latent semantic representation through the pose embedding layer, and masking the latent semantic representations corresponding to several tuples in each pose triple using a masking tuple modeling method before inputting them into the temporal encoder includes: The pose motion sequence is expressed as Wherein, T represents the frame number of the pose motion sequence, t is the frame number sequence, each frame is a pose triplet, each pose triplet contains left hand pose, right hand pose and body pose, and each pose is a tuple. For the t-th frame, the pose embedding layer is mapped to the corresponding hidden semantic representation, denoted as f sign，t , the hidden semantic representation f sign，t is spliced by the three parts of the left hand, the right hand and the body pose, and the hidden semantic representation sequence corresponding to the pose motion sequence is denoted as F sign =(f sign，1 , …, f sign，T ); The latent semantic representations corresponding to several tuples in each pose triplet are masked using a masking tuple modeling approach, as follows: F m = Mask(F sign ) = [f mask，1 ,..., f mask，T ] wherein Mask(·) represents a mask operator of the latent semantic representation sequence, f mask，t represents the feature after being covered in the t-th frame; The final hidden semantic representation of each pose triplet includes a feature sequence and a time sequence feature sequence processed by the mask operator, and the final hidden semantic representation corresponding to the t-th frame of the pose triplet is: f in，t = f mask，t + f temp，t wherein f temp，t denotes the time sequence feature sequence corresponding to the t-th frame; The sequence F0= [f in，1 , …, f in，T ] consisting of the final latent semantic representations of all corresponding pose triplets is input into the temporal encoder.

5. The method of claim 1 or 4, wherein, The temporal encoder reconstructs the hidden semantic representation of the covered tuple by capturing context clues, and outputs an encoding sequence corresponding to the pose motion sequence, which includes: The temporal encoder is a temporal encoder based on a Transformer structure, including N layers of Transformer modules, denoted as Block(·), the input of the first layer is the output of the pose embedding layer, and the input of each subsequent layer is the output of the previous layer, and the output of the j-th layer is denoted as: F j = Block(F j-1 ) where j = 1,..., N, F j-1 denotes the output of the j-1 layer, when j = 1, F0is the output of the pose embedding layer; The output F of the Nth layer is taken as the input of the (N+1)th layer N The pose motion sequence output by the temporal encoder corresponds to the encoding sequence.

6. The method of claim 1, wherein, The code word decoder predicts the label of each covered tuple using the encoding sequence and the position of the covered tuple in each pose triplet, denoted as: In the posture motion sequence, each frame is a posture triplet, t is the frame number, each posture triplet contains the left and right hand postures and the body posture, each posture is a tuple, and l, r, and b are the label symbols for the left hand, right hand, and body, respectively. These represent the sets of frames whose latent semantic representations corresponding to left-hand pose, right-hand pose, and body pose are occluded, respectively; f out，t This represents the encoded vector corresponding to the t-th frame in the encoded sequence. They represent f respectively out，t The components corresponding to the left hand, right hand, and body; W1 and W2 are weight parameters in the codeword decoder, and b1 and b2 are bias terms in the codeword decoder; when When the latent semantic representation corresponding to the left-hand pose in frame t is masked, the encoding vector f of frame t is used. out，t middle left hand component Predict the label corresponding to the left-hand pose in frame t. when When the right hand gesture corresponding to the t-th frame is masked, the latent semantic representation is masked using the encoding vector f of the t-th frame. out，t Right-hand component Predict the label corresponding to the right hand pose in frame t. when When the latent semantic representation corresponding to the body pose in frame t is masked, the encoding vector f of frame t is used. out，t medium body weight Predict the label corresponding to the body pose in frame t.

7. The method of claim 6, wherein, The loss function of the first stage is constructed by combining the pseudo label, denoted as: wherein, represents the loss function of the first stage, represents the loss function of the second stage, sign consisting of a plurality of pose motion sequences V represents the expectation, k t is a pseudo label, p(.|.) represents the predicted probability distribution, 8. A sign language recognition system characterized by, The system is implemented based on the method of any one of claims 1-7, and the system comprises: An initial sign language recognition model construction unit configured to construct an initial sign language recognition model; A training unit configured to perform two-stage training on the initial sign language recognition model using the pose motion sequence; the first stage is a self-supervised pre-training: the initial sign language recognition model includes a pose quantizer, a pose embedding layer, a temporal encoder, and a code word decoder; each pose triplet in the pose motion sequence is input into the pose quantizer and the pose embedding layer, the pose quantizer is used to quantize to generate a corresponding pseudo label for each pose triplet; the pose embedding layer is used to map each pose triplet into a corresponding hidden semantic representation, and the hidden semantic representations of several tuples in each pose triplet are covered in a manner of covering tuple modeling and then input into the temporal encoder; the temporal encoder reconstructs the hidden semantic representation of the covered tuple by capturing context clues, and outputs an encoding sequence corresponding to the pose motion sequence; the code word decoder predicts the label of each covered tuple using the encoding sequence and the position of the covered tuple in each pose triplet, and constructs a loss function of the first stage by combining the pseudo label, and optimizes the initial sign language recognition model; the covered tuple refers to a tuple whose hidden semantic representation is covered; the second stage is a fine-tuning stage, after the training of the first stage, the pose quantizer is removed and a prediction head is used to replace the code word decoder, to obtain a final sign language recognition model including a pose embedding layer, a temporal encoder, and a prediction head; the pose motion sequence is input, each pose triplet is mapped into a corresponding hidden semantic representation by the pose embedding layer, and an encoding sequence corresponding to the pose motion sequence is obtained by the temporal encoder, and then a word sequence is predicted by the prediction head; the real label corresponding to the pose motion sequence and the predicted word sequence are used to construct a loss function of the second stage for fine-tuning of parameters, to obtain a trained sign language recognition model; wherein the pose motion sequence used in the first stage is unlabeled data, and the pose motion sequence used in the second stage is labeled data; A sign language recognition unit configured to input a pose motion sequence to be recognized into the trained sign language recognition model to obtain a predicted word sequence.

9. A processing device, characterized by It comprises: one or more processors; a memory for storing one or more programs; wherein the one or more programs, when executed by the one or more processors, enable the one or more processors to implement the method recited in any one of claims 1-7.

10. A readable storage medium, storing a computer program, characterized in that, A computer program, which when executed by a processor, implements the method recited in any one of claims 1-7.

Images