Sign language recognition method and device, readable storage medium and product

By fusing weighted value vectors of different modal data in the Transformer model, and using the first encoder stack and the second encoder stack in the sign language recognition technology for model learning, the problem of low sign language recognition accuracy in the prior art is solved. Achieve higher recognition accuracy and robustness.

CN118982874AActive Publication Date: 2024-11-19GEER TECH CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202411203889.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-11-19
Estimated Expiration
2044-08-29

AI Technical Summary

Technical Problem

Existing sign language recognition technology cannot fully utilize the complementarity between data of different modalities, resulting in low accuracy of sign language recognition results.

Method used

Using a sign language recognition method based on the Transformer model, by fusing weighted value vectors in the first encoder stack and the second encoder stack, model learning is used to use the correlation and complementarity between two different modal data to generate more accurate Sign language recognition results.

Benefits of technology

It improves the accuracy and robustness of sign language recognition, can better handle the situation of interference of single modal data, and realizes cross-modal semantic understanding and high-level feature representation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982874A_ABST
    Figure CN118982874A_ABST
Patent Text Reader

Abstract

The invention discloses a sign language recognition method and device, a readable storage medium and a product, and relates to the technical field of sign language recognition, a Transform model comprises a first encoder stack, a second encoder stack and a generator, and the method comprises the steps of obtaining first input data and second input data, the first input data and the second input data represent data of two different modes; inputting the first input data into a first encoder stack, and inputting the second input data into a second encoder stack; in the first encoder stack and the second encoder stack, for a first encoder and a second encoder which have a corresponding relationship, inputting the fusion weighted value vectors to the first encoder and the second encoder respectively; and inputting the first encoding matrix output by the first encoder stack and the second encoding matrix output by the second encoder stack into a generator to obtain a sign language recognition result. The accuracy of the sign language recognition result is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of network communication technology, and in particular to a sign language recognition method, device, readable storage medium and product. Background Art

[0002] Sign language is an important way for the deaf-mute to obtain and express information. The deaf-mute usually communicate with others through sign language, which requires both parties to understand sign language. However, it takes a lot of time and energy for normal people to learn sign language, so most normal people have low sign language skills and find it difficult to communicate with the deaf-mute. Therefore, intelligent sign language recognition is of great significance to the communication of the deaf-mute.

[0003] After collecting the sign language video data of the signer, the sign language video data can be input into the Transformer model (a deep learning model based on the self-attention mechanism) to output the sign language recognition result. The encoder in the Transformer model mainly converts the input data into q (query) matrix, k (key) matrix, and v (value) matrix, and then calculates the weighted value vector based on the converted q matrix, k matrix and v matrix, and then performs model learning based on the calculated weighted value vector. However, in this learning method, the encoder in the Transformer model is learned independently and cannot fully utilize the complementarity between different modal data, resulting in low accuracy of sign language recognition results.

[0004] Therefore, how to improve the accuracy of sign language recognition results is a technical problem that needs to be solved urgently in this technical field.

[0005] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the invention

[0006] The main purpose of this application is to provide a sign language recognition method, device, readable storage medium and product, aiming to solve the technical problem of how to improve the accuracy of sign language recognition results.

[0007] To achieve the above-mentioned object, the present application provides a sign language recognition method, which performs sign language recognition based on a Transformer model, wherein the Transformer model includes a first encoder stack, a second encoder stack and a generator, wherein the first encoder stack includes at least one first encoder connected in sequence, and the second encoder stack includes at least one second encoder connected in sequence, and the sign language recognition method includes the following steps:

[0008] Acquire first input data and second input data, wherein the first input data and the second input data represent data of two different modes;

[0009] Inputting the first input data into the first encoder stack, and inputting the second input data into the second encoder stack;

[0010] In the first encoder stack and the second encoder stack, for the first encoder and the second encoder having a corresponding relationship, the fused weighted value vector is input to the first encoder and the second encoder respectively, wherein the fused weighted value vector is a weighted sum vector of the first weighted sum value vector and the second weighted value vector, the first weighted sum value vector is a weighted value vector calculated by the first encoder, and the second weighted sum value vector is a weighted value vector calculated by the second encoder;

[0011] The first encoding matrix output by the first encoder stack and the second encoding matrix output by the second encoder stack are input into the generator to obtain a sign language recognition result.

[0012] In one embodiment, before the step of obtaining the first input data and the second input data, the method further includes:

[0013] Obtain a first layer number of each first encoder in the first encoder stack, and obtain a second layer number of each second encoder in the second encoder stack;

[0014] Selecting a first encoder whose first number of layers is an even number among the first encoders as a first target encoder, and selecting a second encoder whose second number of layers is an even number among the second encoders as a second target encoder;

[0015] The first target encoder and the second target encoder are set to correspond one to one.

[0016] In one embodiment, before the step of obtaining the first input data and the second input data, the method further includes:

[0017] Obtain a first layer number of each first encoder in the first encoder stack, and obtain a second layer number of each second encoder in the second encoder stack;

[0018] Selecting a first encoder whose first number of layers is an odd number among the first encoders as a first target encoder, and selecting a second encoder whose second number of layers is an odd number among the second encoders as a second target encoder;

[0019] The first target encoder and the second target encoder are set to correspond one to one.

[0020] In one embodiment, for a first encoder and a second encoder that have a corresponding relationship, the number of layers of the first encoder is positively correlated with the number of layers of the second encoder.

[0021] In one embodiment, the generator includes a fusion module, a decoding module and a prediction head, and the step of inputting the first encoding matrix and the second encoding matrix into the generator to obtain a sign language recognition result includes:

[0022] Performing weighted summation of the first coding matrix and the second coding matrix by the fusion module to obtain a coding matrix;

[0023] Inputting the encoding matrix into the decoding module to obtain a decoding matrix, wherein the decoding module is used to perform decoding processing on the encoding matrix;

[0024] The decoding matrix is ​​input into the prediction head to obtain a sign language recognition result.

[0025] In one embodiment, the prediction head includes a pooling layer, a plurality of regularization layers and an output layer, and the step of inputting the decoding matrix into the prediction head to obtain the sign language recognition result includes:

[0026] Inputting the decoding matrix into the pooling layer to obtain a fusion vector;

[0027] Inputting the fused vector into each of the regularization layers respectively to obtain a regularization vector;

[0028] All the regularized vectors are input into the output layer to obtain the sign language recognition result.

[0029] In one embodiment, the Transformer model further includes an image embedding module, a key coordinate embedding module, and a position encoding module, and the step of obtaining the first input data and the second input data includes:

[0030] Acquire image data to be subjected to sign language recognition;

[0031] Performing embedding processing on the image data by the image embedding module to obtain an image vector;

[0032] Extracting key points from the image data by means of the key coordinate embedding module, and embedding the key points to obtain key point vectors;

[0033] Performing position coding on the image vector by the position coding module to obtain an image position coding vector;

[0034] Performing position encoding on the key point vector by the position encoding module to obtain a key point position encoding vector;

[0035] The image position encoding vector is determined as first input data, and the key point position encoding vector is determined as second input data.

[0036] In addition, to achieve the above-mentioned purpose, the present application also provides a sign language recognition device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the sign language recognition method as described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a readable storage medium, which is a computer-readable storage medium, and a program for implementing the sign language recognition method is stored on the computer-readable storage medium. The program for implementing the sign language recognition method is executed by a processor to implement the steps of the sign language recognition method as described above.

[0038] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned sign language recognition method when executed by a processor.

[0039] One or more technical solutions proposed in this application have at least the following technical effects:

[0040] Sign language recognition is performed based on a Transformer model. The Transformer model includes a first encoder stack, a second encoder stack and a generator. The first encoder stack includes at least one first encoder connected in sequence, and the second encoder stack includes at least one second encoder connected in sequence. First input data and second input data are obtained, wherein the first input data and the second input data represent data of two different modalities; the first input data is input into the first encoder stack, and the second input data is input into the second encoder stack; in the first encoder stack and the second encoder stack, for the first encoder and the second encoder that have a corresponding relationship, a fused weighted value vector is input into the first encoder and the second encoder respectively, wherein the fused weighted value vector is a weighted sum vector of a first weighted sum value vector and a second weighted value vector, the first weighted sum value vector is a weighted value vector calculated by the first encoder, and the second weighted sum value vector is a weighted value vector calculated by the second encoder; a first encoding matrix output by the first encoder stack and a second encoding matrix output by the second encoder stack are input into the generator to obtain a sign language recognition result. Thus, in the embodiment of the present application, in the first encoder stack and the second encoder stack for processing two different modal data, for the first encoder and the second encoder with a corresponding relationship, the weighted fusion of the two is calculated to obtain the first weighted value vector and the second weighted value vector, so that the fused weighted value vector fuses the features of the two modal data, and the fused fused weighted value vector is respectively input into the first encoder and the second encoder with a corresponding relationship, so that the first encoder and the second encoder can learn the subsequent model based on the fused weighted value vector, instead of using their own query matrix, key matrix and value matrix to calculate the weighted value vector and then learn independently. By fusing the encoders for processing the two different modal data to calculate the weighted value vector, the encoders for processing the two different modal data can learn each other's information in the intermediate learning process, fully explore and learn the association and complementarity between the two different modal data, thereby improving the overall recognition performance of the Transformer model, and then improving the accuracy of the recognition results. At the same time, it can better handle the situation where the single modal data is interfered, realize cross-modal semantic understanding and high-level feature representation, and improve the robustness of sign language recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 This is a flow chart of the first embodiment of the sign language recognition method of the present application;

[0044] Figure 2 This is a schematic diagram of the data processing flow of the encoder in the Transformer model;

[0045] Figure 3 A schematic diagram of encoder data exchange for the sign language recognition method of the present application;

[0046] Figure 4 A schematic diagram of an encoder for screening exchange data involved in an embodiment of a sign language recognition method of the present application;

[0047] Figure 5 Schematic diagram of the overall structure of the Transformer model proposed in this application;

[0048] Figure 6 Schematic diagram of the device structure of the hardware operating environment involved in the sign language recognition device in the embodiment of the present application.

[0049] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0051] At present, the sign language recognition for the deaf and mute mainly focuses on learning with single-modal image data. Single-modal recognition technology is relatively mature: single-modal sign language recognition methods such as vision-based gesture recognition and sensor-based gesture capture have a long history of research and have accumulated a lot of experience and technical foundation. However, this sign language recognition method cannot fully utilize the complementarity between different modalities: learning alone cannot explore the association and synergy between multiple modal data such as vision and motion, which limits the further improvement of the accuracy of recognition results.

[0052] Based on this, the main solution of the present application is: to perform sign language recognition based on the Transformer model, the Transformer model includes a first encoder stack, a second encoder stack and a generator, the first encoder stack includes at least one first encoder connected in sequence, the second encoder stack includes at least one second encoder connected in sequence, obtain first input data and second input data, wherein the first input data and the second input data represent data of two different modalities; input the first input data into the first encoder stack, and input the second input data into the second encoder stack; in the first encoder stack and the second encoder stack, for the first encoder and the second encoder that have a corresponding relationship, input the fused weighted value vector into the first encoder and the second encoder respectively, wherein the fused weighted value vector is the weighted sum vector of the first weighted sum value vector and the second weighted value vector, the first weighted sum value vector is the weighted value vector calculated by the first encoder, and the second weighted sum value vector is the weighted value vector calculated by the second encoder; the first encoding matrix output by the first encoder stack and the second encoding matrix output by the second encoder stack are input into the generator to obtain the sign language recognition result.

[0053] The present application calculates the first weighted value vector and the second weighted value vector by weighted fusion in the first encoder stack and the second encoder stack for the first encoder and the second encoder with corresponding relationship, and inputs the fused fusion weighted value vector into the first encoder and the second encoder with corresponding relationship respectively, so that the fused weighted value vector fuses the features of the two modal data, so that the first encoder and the second encoder can learn the subsequent model based on the fused weighted value vector, instead of using their own query matrix, key matrix and value matrix to calculate the weighted value vector and then learn independently. By fusion processing the encoders of the two different modal data to calculate the weighted value vector, the encoders processing the two different modal data can learn each other's information in the intermediate learning process, fully explore and learn the association and complementarity between the two different modal data, thereby improving the overall recognition performance of the Transformer model, and then improving the accuracy of the recognition result. At the same time, it can better handle the situation where the single modal data is interfered, realize cross-modal semantic understanding and high-level feature representation, and improve the robustness of sign language recognition.

[0054] It should be noted that the execution subject of each embodiment of the sign language recognition method of the present application may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a sign language recognition device capable of realizing the above functions, such as AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, smart glasses, robots, etc. The embodiments of the sign language recognition method of the present application do not impose specific restrictions on this.

[0055] Based on this, the present application proposes a sign language recognition method of the first embodiment, which performs sign language recognition based on a Transformer model. The Transformer model includes a first encoder stack, a second encoder stack and a generator. The first encoder stack includes at least one first encoder connected in sequence, and the second encoder stack includes at least one second encoder connected in sequence. Please refer to Figure 1 , the sign language recognition method comprises steps S10 to S40:

[0056] Step S10, obtaining first input data and second input data, wherein the first input data and the second input data represent data of two different modes;

[0057] The total number of layers of the first encoder in the first encoder stack (hereinafter referred to as the first total number of layers) and the total number of layers of the second encoder in the second encoder stack (hereinafter referred to as the second total number of layers) may be the same or different. This embodiment does not impose any specific restrictions on this. Preferably, the first total number of layers is the same as the second total number of layers.

[0058] It should be noted that the first input data and the second input data represent data of two different modalities, such as the first input data represents data of image modality (data of text modality), and the second input data represents data of text modality (data of image modality). This embodiment does not impose any specific restrictions on this.

[0059] Step S20, inputting the first input data into the first encoder stack, and inputting the second input data into the second encoder stack;

[0060] The first input data is input into the first encoder stack. Specifically, the first input data is input into the first encoder in the first encoder stack, that is, the first input data is input into the first layer encoder in the first encoder stack.

[0061] Similarly, the second input data is input into the second encoder stack, specifically, the second input data is input into the first encoder in the second encoder stack, that is, the second input data is input into the first layer encoder in the second encoder stack.

[0062] Step S30: In the first encoder stack and the second encoder stack, for the first encoder and the second encoder that have a corresponding relationship, input the fused weighted value vector to the first encoder and the second encoder respectively, wherein the fused weighted value vector is a weighted sum vector of the first weighted sum value vector and the second weighted value vector, the first weighted sum value vector is a weighted value vector calculated by the first encoder, and the second weighted sum value vector is a weighted value vector calculated by the second encoder;

[0063] Understandably, referring to Figure 2 As shown in Figure 1, the basic data processing flow of the encoder in the Transformer model is as follows: in each self-attention layer of the multi-head self-attention module, an attention weight is calculated for each word in the input sequence. These weights reflect the importance of each word in the sequence to the current word. Each value matrix is ​​multiplied by its corresponding attention weight to obtain a weighted value vector. Each self-attention layer is followed by a simple feedforward neural network layer. The structure of this network layer is usually two linear layers and a nonlinear activation function. After the self-attention layer and the feedforward neural network, a residual connection is used, that is, the input is added to the output of the feedforward neural network layer, and then the layer is normalized and output. Finally, the output of the current layer is used as the input of the next layer. Among them, the specific calculation process of the weighted value vector is as follows: first calculate the query (q) matrix, the key matrix (k) and the value (v) matrix, then calculate the attention score based on the query matrix and the key matrix, and finally use the softmax function to convert the score into a probability form to obtain the attention weight, and multiply each value matrix by its corresponding attention weight to obtain the weighted value vector.

[0064] Based on this data processing flow, in this embodiment, refer to Figure 3 As shown, for the first encoder and the second encoder that have a corresponding relationship in the first encoder stack and the second encoder stack, the weighted value vector calculated inside the first encoder (that is, the first weighted value vector) and the weighted value vector calculated inside the second encoder (that is, the second weighted value vector) are weighted summed to obtain a fused weighted value vector, and the fused weighted value vector is input into the first encoder and the second encoder that have a corresponding relationship. Specifically, the information can be input into the feedforward neural network layer of the first encoder and the feedforward neural network layer of the second encoder, and the information will be transmitted to the residual connection layer in the form of residual. In this way, information exchange and learning between the first encoder and the second encoder are realized. It should be noted that, Figure 3 In the figure, q_VD, k_VD, and v_VD respectively represent the query matrix, key matrix, and value matrix calculated by the first encoder, and q_KP, k_KP, and v_KP respectively represent the query matrix, key matrix, and value matrix calculated by the second encoder.

[0065] If there is no second encoder corresponding to the first encoding, the first encoder is based on Figure 2 The data processing flow completes the data processing. Similarly, if there is no first encoder corresponding to the second encoder, the second encoder is based on Figure 2 The data processing flow shown completes the data processing, that is, the first encoder and the second encoder complete the data processing with the conventional data processing flow of the encoder, without using the fused weighted sum value vector.

[0066] It should be noted that the correspondence between the first encoder and the second encoder can be set in advance, such as the first encoder of the i-th layer corresponds one-to-one to the second encoder of the j-th layer, i ranges from 1 to the first total number of layers, j ranges from 1 to the second total number of layers, and when the value of i is different, the corresponding value of j is also different. This embodiment does not impose any specific restrictions on this.

[0067] Furthermore, i can take a preset number of values ​​from 1 to the first total number of layers, and the corresponding j also takes this preset number of values ​​from 1 to the second total number of layers, and the preset number is less than min{first total number of layers, second total number of layers}, that is, there is at least one pair of first encoders and second encoders that have no corresponding relationship. In this way, a preset number of first encoders and second encoders use a fused weighted value vector, and the remaining first encoders and second encoders use their own calculations to obtain weighted value vectors, which can reduce the phenomenon that the encoder branches that process the two modal data pay too much attention to the characteristics of the other branch, that is, by learning the characteristics of the other modal data by fusing the weighted value vector, and retaining the encoder that learns the characteristics of its own modal data, so as to mine, fuse and utilize the characteristics of the two modal data, thereby further improving the accuracy of the sign language recognition results.

[0068] Exemplarily, assuming that the first encoder stack includes four first encoders connected in sequence, namely [encoder 1_1, encoder 1_2, encoder 1_3, encoder 1_4], and the second encoder stack includes four second encoders connected in sequence, namely [encoder 2_1, encoder 2_2, encoder 2_3, encoder 2_4], and encoder 1_1 is set to correspond to encoder 2_1, encoder 1_3 corresponds to encoder 2_3, and encoder 1_4 corresponds to encoder 2_4, then the weighted value vector calculated by encoder 1_1 and the weighted value vector calculated by encoder 2_1 are fused, and the fusion is The weighted value vector (i.e., the fused weighted value vector) is respectively input to encoder 1_1 and encoder 2_1; the weighted value vector calculated by encoder 1_3 and the weighted value vector calculated by encoder 2_3 are fused, and the fused weighted value vectors are respectively input to encoder 1_3 and encoder 2_3; the weighted value vector calculated by encoder 1_4 and the weighted value vector calculated by encoder 2_4 are fused, and the fused weighted value vectors are respectively input to encoder 1_4 and encoder 2_4, while encoder 1_2 and encoder 2_2 complete data processing based on the weighted value vectors calculated by themselves.

[0069] Furthermore, the first weighted sum value vector and the second weighted sum value vector may be weighted summed based on the fusion weights corresponding to the first weighted sum value vector and the second weighted sum value vector to obtain a fused weighted value vector. Regarding the setting of the fusion weights, the fusion weights corresponding to the first weighted sum value vector and the second weighted sum value vector may be set by relevant personnel respectively, or the fusion weights corresponding to the matrices of the first weighted sum value vector and the second weighted sum value vector may be set as model parameters, and the fusion weights corresponding to the first weighted sum value vector and the second weighted sum value vector may be optimized during the model training process, and this embodiment does not impose any specific restrictions on this.

[0070] The fusion weight corresponding to the first weighted sum value vector is α, the first weighted sum value vector is V1, the fusion weight corresponding to the second weighted sum value vector is β, the second weighted sum value vector is V2, and the fusion weight value vector is V. Then V can be expressed by the formula V=α*V1+β*V2. In a specific implementation, refer to Figure 3 As shown, the sum of α and β is one, and V can be expressed by the formula V=α*V1+(1-α)*V2.

[0071] Step S40: input the first encoding matrix output by the first encoder stack and the second encoding matrix output by the second encoder stack into the generator to obtain a sign language recognition result.

[0072] It should be noted that the first encoding matrix output by the first encoder stack is specifically the output of the tail encoder in the first encoder stack, that is, the output of the last layer of encoder in the first encoder stack. Similarly, the second encoding matrix output by the second encoder stack is specifically the output of the tail encoder in the second encoder stack, that is, the output of the last layer of encoder in the second encoder stack.

[0073] It should be noted that the function of the generator is to predict the output based on the processed input sequence, such as the first encoding matrix and the second encoding matrix in this embodiment.

[0074] In the Transformer model (a deep learning model based on the self-attention mechanism), the prediction head (also called a decoder head or output head) is a part of the model. After the encoder or decoder is stacked, the prediction head maps these high-dimensional vectors to the dimensions of the target output space and is responsible for generating the final output. Therefore, in a specific embodiment, the generator includes a prediction head.

[0075] In this embodiment, sign language recognition is performed based on the Transformer model. The Transformer model includes a first encoder stack, a second encoder stack and a generator. The first encoder stack includes at least one first encoder connected in sequence, and the second encoder stack includes at least one second encoder connected in sequence. First input data and second input data are obtained, wherein the first input data and the second input data represent data of two different modalities; the first input data is input into the first encoder stack, and the second input data is input into the second encoder stack; in the first encoder stack and the second encoder stack, for the first encoder and the second encoder that have a corresponding relationship, a fused weighted value vector is input into the first encoder and the second encoder respectively, wherein the fused weighted value vector is a weighted sum vector of the first weighted sum value vector and the second weighted value vector, the first weighted sum value vector is a weighted value vector calculated by the first encoder, and the second weighted sum value vector is a weighted value vector calculated by the second encoder; a first encoding matrix output by the first encoder stack and a second encoding matrix output by the second encoder stack are input into the generator to obtain a sign language recognition result. Thus, in the first encoder stack and the second encoder stack for processing two different modal data, the present embodiment weights and fuses the first encoder and the second encoder with a corresponding relationship to calculate the first weighted value vector and the second weighted value vector, and inputs the fused weighted value vector into the first encoder and the second encoder with a corresponding relationship, so that the fused weighted value vector fuses the features of the two modal data, so that the first encoder and the second encoder can learn the subsequent model based on the fused weighted value vector, instead of using their own query matrix, key matrix and value matrix to calculate the weighted value vector and then learn independently. By fusing the encoders for processing the two different modal data to calculate the weighted value vector, the encoders for processing the two different modal data can learn each other's information in the intermediate learning process, fully explore and learn the association and complementarity between the two different modal data, thereby improving the overall recognition performance of the Transformer model, and further improving the accuracy of the recognition results. At the same time, it can better handle the situation where the single modal data is interfered, realize cross-modal semantic understanding and high-level feature representation, and improve the robustness of sign language recognition.

[0076] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated later. On this basis, before the step of obtaining the first input data and the second input data, the method further includes:

[0077] Step A10, obtaining a first layer number of each first encoder in the first encoder stack, and obtaining a second layer number of each second encoder in the second encoder stack;

[0078] It should be noted that the first number of layers refers to the number of layers of the first encoder in the first encoder stack, and the second number of layers refers to the number of layers of the second encoder in the second encoder stack. The number of layers of the encoders (including the first encoder and the second encoder) is arranged in sequence according to the connection order of the encoders.

[0079] Step A20, selecting a first encoder whose first number of layers is an even number among the first encoders as a first target encoder, and selecting a second encoder whose second number of layers is an even number among the second encoders as a second target encoder;

[0080] It should be noted that the total number of first target encoders is the same as the total number of second target encoders, that is, N first encoders with an even number of first layers are selected as the first target encoders and N second encoders with an even number of second layers are selected as the second target encoders, where N is less than or equal to min{first number of layers / 2, second number of layers / 2}.

[0081] Exemplarily, assuming that the first encoder stack includes four first encoders connected in sequence, namely [encoder 1_1, encoder 1_2, encoder 1_3, encoder 1_4], and the second encoder stack includes four second encoders connected in sequence, namely [encoder 2_1, encoder 2_2, encoder 2_3, encoder 2_4], then N is less than or equal to 2. When N is 2, encoder 1_2 and encoder 1_4 can be selected as the first target encoders, and encoder 2_2 and encoder 2_4 can be selected as the second target encoders.

[0082] Step A30, setting the first target encoder and the second target encoder in one-to-one correspondence.

[0083] The first target encoder and the second target encoder are set to correspond one to one. The first target encoder and the second target encoder can be set to correspond one to one randomly, or the first target encoder and the second target encoder can be set to correspond one to one based on a preset rule, such as making the first target encoder with the same number of layers as the second number of layers correspond to the second target encoder. For example, referring to Figure 4As shown, the first encoder stack includes n layers of first encoders connected in sequence, namely [Encoder VD1, EncoderVD2...Encoder VDn-1, Encoder VDn], and the second encoder stack includes n layers of first encoders connected in sequence, namely [Encoder KP1, Encoder KP2...Encoder KPn-1, Encoder KPn]. The first encoder with an even number of first layers is selected as the first target encoder, and the second encoder with an even number of second layers is selected as the second target encoder. The first target encoder Encoder VDi with the first layer number i is set to correspond to the second target encoder Encoder KPi with the second layer number i, where the value of i is

[0084] In another possible implementation, before the step of obtaining the first input data and the second input data, the method further includes:

[0085] Step B10, obtaining a first layer number of each first encoder in the first encoder stack, and obtaining a second layer number of each second encoder in the second encoder stack;

[0086] Step B20, selecting a first encoder whose first number of layers is an odd number among the first encoders as a first target encoder, and selecting a second encoder whose second number of layers is an odd number among the second encoders as a second target encoder;

[0087] Step B30, setting the first target encoder and the second target encoder in one-to-one correspondence.

[0088] It should be noted that the specific implementation method of selecting the first encoder whose first layer number is an odd number as the first target encoder, selecting the second encoder whose second layer number is an odd number as the second target encoder, and setting the one-to-one correspondence between the first target encoder and the second target encoder is similar to the specific implementation method of selecting the even-numbered layer encoder as the target encoder. Please refer to the above-mentioned description, and this embodiment will not be repeated.

[0089] In this embodiment, the first encoder whose first layer number is an even number (odd number) is selected as the first target encoder among each first encoder, and the second encoder whose second layer number is an even number (odd number) is selected as the second target encoder among each second encoder, and the first target encoder and the second target encoder are set to correspond one to one. In this way, the encoders of the even-numbered layers (odd-numbered layers) (including the first encoder and the second encoder) use the fused weighted value vector that fuses the weighted value vectors of the two to learn the features of the other party's modal data, and the encoders of the odd-numbered layers (even-numbered layers) fuse the weighted value vectors of the two to learn the features of their own modal data. In this way, the features of the other party's modal data and the features of their own modal data are learned alternately, achieving balanced learning of the features of the other party's modal data and balanced learning of the features of their own modal data, thereby balancedly fusing the features of the other party's modal data and the features of their own modal data, so that the encoder can fully explore the features of the other party's modal data and the features of their own modal data and utilize them, so as to further improve the accuracy of the sign language recognition results.

[0090] In a possible implementation manner, for a first encoder and a second encoder that have a corresponding relationship, the number of layers of the first encoder is positively correlated with the number of layers of the second encoder.

[0091] In this embodiment, the first encoder of the i-th layer is set to correspond to the second encoder of the j-th layer, and different values ​​of i correspond to different values ​​of j, and the smaller the value of i, the smaller the value of j. In this way, the first encoder of the lower layer corresponds to the second encoder of the lower layer, and the number of layers of the encoders corresponding to the fused weighted value vector is relatively consistent, which ensures that the abstract level of the features learned by the fused weighted value vector is relatively consistent, and then after the weighted value vector is fused, the encoder can learn subsequent features more effectively based on the fused weighted value vector.

[0092] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above-mentioned embodiments 1 and 2 can be referred to the above description, and will not be described in detail later. On this basis, the generator includes a fusion module, a decoding module and a prediction head, and the step of inputting the first encoding matrix output by the first encoder stack and the second encoding matrix output by the second encoder stack into the generator to obtain the sign language recognition result includes:

[0093] Step C10, performing weighted summation on a first coding matrix output by the first encoder stack and a second coding matrix output by the second encoder stack through the fusion module to obtain a coding matrix;

[0094] The fusion module is connected to the first encoder stack and the second encoder stack respectively to fuse the output of the first encoder stack with the output of the second encoder stack.

[0095] Further, the first coding matrix and the second coding matrix may be weightedly fused based on the fusion weights corresponding to the first coding matrix and the second coding matrix to obtain the coding matrix. For setting the fusion weights, the fusion weights corresponding to the first coding matrix and the second coding matrix may be set by relevant personnel respectively, or the fusion weights corresponding to the first coding matrix and the second coding matrix may be set as model parameters, and the fusion weights corresponding to the first coding matrix and the second coding matrix may be optimized during the model training process, and this embodiment does not impose specific restrictions on this.

[0096] The fusion weight corresponding to the first encoding matrix is ​​denoted as p, the first encoding matrix is ​​IA, the fusion weight corresponding to the second encoding matrix is ​​q, the second encoding matrix is ​​KA, and the encoding matrix is ​​EA, then EA can be expressed by the formula EA=p*IA+q*KA. In a specific implementation, the sum of p and q is one.

[0097] Step C20, inputting the encoding matrix into the decoding module to obtain a decoding matrix, wherein the decoding module is used to perform decoding processing on the encoding matrix;

[0098] The decoding module receives the encoding matrix input by the encoding module, and decodes the encoding matrix to obtain an output sequence, namely, a decoding matrix, and inputs the decoding matrix into the prediction head.

[0099] Relevant personnel can design a decoding module according to actual needs, and the decoding matrix is ​​the output of the decoding module. In a specific implementation, the decoding module can be a decoder stack composed of one or more decoders connected in sequence, and the decoding matrix is ​​the output of the decoder stack, that is, the output of the last layer of decoders in the decoder stack.

[0100] Step C30, inputting the decoding matrix into the prediction head to obtain a sign language recognition result.

[0101] It should be noted that the sign language recognition result is the output of the prediction head.

[0102] The prediction head in the Transformer model is used to convert the internal representation of the model into the final prediction output. In this embodiment, the prediction head is used to convert the decoding matrix into the final prediction output, that is, the sign language recognition result.

[0103] Relevant personnel can design a prediction head according to actual needs. In a specific implementation, the prediction head includes a pooling layer, a regularization layer (dropout), a fully connected layer and a softmax function connected in sequence, and finally predicts the output through the softmax function.

[0104] In a possible implementation, the prediction head includes a pooling layer, a plurality of regularization layers and an output layer, and the step of inputting the decoding matrix into the prediction head to obtain a sign language recognition result includes:

[0105] Step D10, inputting the decoding matrix into the pooling layer to obtain a fusion vector;

[0106] It can be understood that in the Transformer model (a deep learning model based on the self-attention mechanism), the prediction head (also called the decoder head or output head) is a part of the model that maps these high-dimensional vectors to the dimensions of the target output space after the encoder or decoder stack, and is responsible for generating the final output. In classification tasks, the prediction head usually consists of a pooling layer, a regularization layer (Dropout layer), and an output layer connected in sequence.

[0107] However, in the design of the prediction head of the Transformer model, a single-layer Dropout structure is used. Relevant personnel may use different Dropout ratios as hyperparameters to tune the model, but this method has limited effect on improving the generalization ability of the model. Therefore, in this embodiment, multiple layers of parallel regularization layers [Dropout1, Dropout 2...Dropout n] are designed to reduce the overfitting phenomenon of the model and improve the generalization ability of the model, where n represents the total number of regularization layers, and specifically, n is an integer greater than 1.

[0108] It should be noted that the fused vector is the output of the pooling layer.

[0109] The vector sequences in the feature matrix are fused through the pooling layer to aggregate the vectors into a fused vector of fixed dimension and fixed length.

[0110] The pooling layer may be an average pooling layer, a maximum pooling layer, a global pooling layer, etc., and this embodiment does not impose any specific restrictions on this. Considering that compared with other pooling operations, the average pooling operation performs an average operation on the sequence, so the overall characteristics of the sequence can be retained, not only the maximum and minimum characteristics in the sequence are considered, but also the overall trend and distribution are considered. Average pooling can also enhance robustness and reduce overfitting. Based on this, in a preferred embodiment, the pooling layer is specifically an average pooling layer.

[0111] Step D20, inputting the fused vector into each of the regularization layers to obtain a regularized vector;

[0112] It should be noted that the regularization vector is the output of the regularization layer.

[0113] Regularization (Dropout) is a technique that randomly "inactivates" (temporarily ignores) some neurons in a neural network during training. The inactivated neurons do not participate in the calculation during the current forward propagation and back propagation. By randomly "inactivating" some neurons in the network during training, the network's dependence on specific training samples is reduced, thereby improving the model's generalization ability on unseen data. Therefore, the Dropout layer is added after the average pooling layer.

[0114] The fusion vector is randomly inactivated through multiple regularization layers (Dropout) to obtain a regularized vector, that is, the fusion vector is input into each regularization layer respectively.

[0115] It should be noted that the discard ratios of the regularization layers may be the same or different, and this embodiment does not impose any specific limitation on this.

[0116] Step D30, inputting all the regularized vectors into the output layer to obtain the sign language recognition result.

[0117] It should be noted that the sign language recognition result is the output of the output layer.

[0118] Relevant personnel can design the output layer according to actual needs. In a specific implementation, the output layer includes a fully connected layer and a softmax function connected in sequence.

[0119] It should be noted that if the Transformer model for sign language recognition is currently being trained, that is, the Transformer model is a model that has not been pre-trained, then after obtaining the sign language recognition result, the function value of the preset loss function (hereinafter referred to as the loss value) can also be calculated based on the sign language recognition result, and the Transformer model can be optimized based on the loss value until the preset training end condition is met to obtain a trained Transformer model. Among them, the preset loss function can be any loss function set in advance, such as the cross entropy loss function, and this embodiment does not make specific restrictions on this.

[0120] In the Transformer model proposed in this embodiment, multiple regularization layers are set to perform random inactivation processing using multiple regularization layers, and multiple different sub-network models are generated using multiple regularization layers, so that the model algorithm will not only focus on a certain branch, thereby reducing the risk of overfitting of the model and improving the generalization ability of the model, so that the model can obtain relatively accurate recognition results even for data that has not been "seen" during training, thereby reducing errors and fluctuations in the recognition results and improving the stability of the sign language recognition results.

[0121] Furthermore, the weight parameters of all fully connected layers are consistent, wherein the weight parameters include a weight matrix and a bias term, that is, the parameters of all fully connected layers are shared. In this way, the fully connected layer after the Dropout layer uses parameter sharing to reduce the number of model parameters, thereby reducing the amount of model calculation and reducing the risk of overfitting.

[0122] Furthermore, the dropout ratio of each regularization layer is different. Different dropout ratios can provide different regularization strengths, thereby improving the generalization ability of the model. Applying different dropout ratios can increase the diversity of the model, so that the model can be trained under different configurations. This diversity helps the model to have better adaptability to unknown data when used, and different dropout ratios can be customized to more flexibly control the training process of the model, which helps to find the best training strategy to balance the accuracy and generalization ability of the model.

[0123] In a possible implementation, the Transformer model further includes an image embedding module, a key coordinate embedding module, and a position encoding module, and the step of obtaining the first input data and the second input data includes:

[0124] Step E10, acquiring image data to be subjected to sign language recognition, and embedding the image data through the image embedding module to obtain an image vector;

[0125] The image data includes one or more frames of images. Further, after the image data is acquired, key frame images in the image data can be extracted, such as extracting key frame images in the image data based on a key frame extraction algorithm of a shot boundary (Shot Boundary Detection, SBD), retaining key frame images in the image data and deleting non-key frame images to improve the image quality in the image data. The image data retaining the key frame images is subsequently processed.

[0126] The image data is embedded, specifically, the image in the image data is embedded.

[0127] Embedding refers to the process of converting input data (such as words, characters, sentences, or any serializable entity) into a fixed-length vector representation. Embedding processing is the basis for the model to understand the input data.

[0128] The image embedding module is used to embed the image data to obtain the image vector. Specifically, the embedding process is as follows:

[0129] (1) First, the image in the image data is scaled to a uniform size, such as 224×224×3, and then the image is divided into small patches, each patch size is also the same size, such as 16×16×3, and then the small patches are arranged;

[0130] (2) Perform an Embedding operation on the arranged patches to obtain an image vector.

[0131] Step E20, extracting key points from the image data by the key coordinate embedding module, and embedding the key points to obtain key point vectors;

[0132] The key point embedding module is used to extract key points in the image (actually extract coordinates). Specifically, the key points of the face and hands in the image can be extracted. The key points of the hand are recorded as hand key points, and the key points of the face are recorded as facial key points. Specifically, a preset number of hand key points and a preset number of facial key points can be extracted. The preset number of facial key points can specifically include eye key points (key points of the glasses part), nose key points (key points of the nose part), mouth key points (key points of the mouth part), contour key points (key points of the face contour part), etc. In a specific implementation, 21 hand key points are extracted from one hand based on the image, and a total of 42 hand key points are extracted from both hands. At the same time, 58 facial key points are extracted. The 58 facial key points specifically include 17 contour key points, 12 eye key points, 9 nose key points and 20 mouth key points.

[0133] It can be understood that a key point can be represented by coordinates, so that each key point is actually represented as a coordinate.

[0134] Step E30, performing position encoding on the image vector by the position encoding module to obtain an image position encoding vector;

[0135] Step E40, performing position encoding on the key point vector by the position encoding module to obtain a key point position encoding vector;

[0136] Position coding refers to adding a piece of information about the position of each word in the sentence, also called "information fragment". Specifically, absolute position coding, relative position coding, etc. can be used, and this embodiment does not impose specific restrictions on this.

[0137] Step E50, determining the image position encoding vector as the first input data, and determining the key point position encoding vector as the second input data.

[0138] It should be noted that, in this embodiment, the first input data represents data in image mode, and the second input data represents data in text mode (the key points are actually text data in coordinate form).

[0139] In this embodiment, in order to make the model pay more attention to the hand and facial information of the signer in the process of expressing sentences, a key point embedding module is set up to extract the key points of the hands and face in the image, thereby improving the performance of the model. At the same time, in order to avoid the influence of noise on the key points, an image embedding module is set up to embed the image, thereby fusing the two modal data of image and text for sign language recognition, utilizing the complementary advantages of the two modal data to jointly learn features, thereby further improving the accuracy of the sign language recognition results.

[0140] For example, to help understand the technical concept or technical principle of the sign language recognition method after the present embodiment is combined with the first embodiment and the second embodiment, please refer to Figure 5 As shown, the Transformer model specifically includes an input module, an image encoder stack (i.e., a first encoder stack), a key point encoder stack (i.e., a second encoder stack), and a generator. The generator includes a fusion module, a decoding module, and a prediction head connected in sequence. The image encoder stack includes n layers of encoders (Encoder VD) connected in sequence, the key point encoder stack includes n layers of encoders (EncoderKP) connected in sequence, the stack decoding module includes m layers of decoders (Decoder) connected in sequence, and the input module includes an image embedding (Embedding) module and a key point embedding module. The image embedding module and the key point embedding module are respectively connected to the position encoding module.

[0141] The encoders of the even-numbered layers in the image layer encoder stack and the key point encoder stack use the fused weight value vector, while the encoders of the odd-numbered layers have their own weight value vectors, and the number of encoder layers using the fused weight value vector is the same.

[0142] The fusion module performs weighted fusion on the outputs of the image encoder stack and the key point encoder stack, and inputs the fusion results into each decoder of the decoder module. Specifically, the fusion weight corresponding to the image encoding matrix output by the image encoder stack is α, and the fusion weight corresponding to the key point encoding matrix output by the key point encoder stack is 1-α. The image encoding matrix and the key point encoding matrix are weightedly fused based on their respective fusion weights.

[0143] It should be noted that the final output of the n-layer encoders connected in sequence is respectively input into each decoder of the m-layer decoders connected in sequence.

[0144] It should be noted that if the Transformer model is currently being trained, the preset expected result labels (such as Figure 5 The "business license" shown in the figure, which is the expected output of the model, is embedded and positionally encoded in sequence and then input into the decoding module. At the same time, the key frame image corresponding to the result label is input into the input module. The sign language recognition model is trained based on the input of the preset expected result label and the input of the image data to obtain a trained Transformer model for sign language recognition.

[0145] In addition, an embodiment of the present application also proposes a sign language recognition device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the sign language recognition method as described above.

[0146] refer to Figure 6 , which shows a schematic diagram of the structure of a sign language recognition device suitable for implementing the embodiment of the present application. The sign language recognition device in the embodiment of the present application may also include but is not limited to mobile terminals such as AR glasses, smart glasses, robots, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The sign language recognition device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0147] like Figure 6As shown, the sign language recognition device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the sign language recognition device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the sign language recognition device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a sign language recognition device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0148] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0149] The sign language recognition device provided in the embodiment of the present application adopts the sign language recognition method in the above embodiment, which can solve the technical problem of how to improve the accuracy of the sign language recognition result. Compared with the prior art, the beneficial effects of the sign language recognition device provided in the present application are the same as the beneficial effects of the sign language recognition method provided in the above embodiment, and the other technical features in the sign language recognition device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0150] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0151] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0152] In addition, to achieve the above-mentioned purpose, an embodiment of the present application also provides a readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, and the computer-readable program instructions are used to execute the sign language recognition method in the above-mentioned embodiment.

[0153] The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0154] The computer-readable storage medium may be included in the sign language recognition device; or may exist independently without being assembled into the sign language recognition device.

[0155] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the sign language recognition device, the sign language recognition device: obtains first input data and second input data, wherein the first input data and the second input data represent data of two different modalities; inputs the first input data into the first encoder stack, and inputs the second input data into the second encoder stack; in the first encoder stack and the second encoder stack, for the first encoder and the second encoder that have a corresponding relationship, inputs the fused weighted value vector into the first encoder and the second encoder respectively, wherein the fused weighted value vector is the weighted sum vector of the first weighted sum value vector and the second weighted value vector, the first weighted sum value vector is the weighted value vector calculated by the first encoder, and the second weighted sum value vector is the weighted value vector calculated by the second encoder; the first encoding matrix output by the first encoder stack and the second encoding matrix output by the second encoder stack are input into the generator to obtain the sign language recognition result.

[0156] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0157] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0158] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of a module does not, in some cases, constitute a limitation on the module itself.

[0159] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned sign language recognition method, and can solve the technical problem of how to improve the accuracy of the sign language recognition result. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the sign language recognition method provided in the above-mentioned embodiment, and will not be elaborated here.

[0160] In addition, an embodiment of the present application also proposes a computer program product, including a sign language recognition program, which implements the steps of the sign language recognition method described above when executed by a processor.

[0161] The specific implementation methods of the computer program product of the present application are basically the same as the above-mentioned embodiments of the sign language recognition method, and will not be repeated here.

[0162] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0163] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0164] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software sensor, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, including a number of instructions for a terminal device (which can be a mobile phone, computer, server or network device, etc.) to execute the methods described in each embodiment of the present application.

[0165] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A sign language recognition method, characterized in that: The sign language recognition method performs sign language recognition based on a Transformer model, wherein the Transformer model includes a first encoder stack, a second encoder stack, and a generator, wherein the first encoder stack includes at least one first encoder connected in sequence, and the second encoder stack includes at least one second encoder connected in sequence, and the sign language recognition method includes the following steps: Acquire first input data and second input data, wherein the first input data and the second input data represent data of two different modes; Inputting the first input data into the first encoder stack, and inputting the second input data into the second encoder stack; In the first encoder stack and the second encoder stack, for the first encoder and the second encoder having a corresponding relationship, the fused weighted value vector is input to the first encoder and the second encoder respectively, wherein the fused weighted value vector is a weighted sum vector of the first weighted sum value vector and the second weighted value vector, the first weighted sum value vector is a weighted value vector calculated by the first encoder, and the second weighted sum value vector is a weighted value vector calculated by the second encoder; The first encoding matrix output by the first encoder stack and the second encoding matrix output by the second encoder stack are input into the generator to obtain a sign language recognition result.

2. The sign language recognition method according to claim 1, characterized in that: Before the step of obtaining the first input data and the second input data, the method further includes: Obtain a first layer number of each first encoder in the first encoder stack, and obtain a second layer number of each second encoder in the second encoder stack; Selecting a first encoder whose first number of layers is an even number among the first encoders as a first target encoder, and selecting a second encoder whose second number of layers is an even number among the second encoders as a second target encoder; The first target encoder and the second target encoder are set to correspond one to one.

3. The sign language recognition method according to claim 1, wherein: Before the step of obtaining the first input data and the second input data, the method further includes: Obtain a first layer number of each first encoder in the first encoder stack, and obtain a second layer number of each second encoder in the second encoder stack; Selecting a first encoder whose first number of layers is an odd number among the first encoders as a first target encoder, and selecting a second encoder whose second number of layers is an odd number among the second encoders as a second target encoder; The first target encoder and the second target encoder are set to correspond one to one.

4. The sign language recognition method according to any one of claims 1 to 3, characterized in that: For the first encoder and the second encoder that have a corresponding relationship, the number of layers of the first encoder is positively correlated with the number of layers of the second encoder.

5. The sign language recognition method according to claim 1, wherein: The generator includes a fusion module, a decoding module and a prediction head, and the step of inputting the first encoding matrix and the second encoding matrix into the generator to obtain a sign language recognition result includes: Performing weighted summation of the first coding matrix and the second coding matrix by the fusion module to obtain a coding matrix; Inputting the encoding matrix into the decoding module to obtain a decoding matrix, wherein the decoding module is used to perform decoding processing on the encoding matrix; The decoding matrix is ​​input into the prediction head to obtain a sign language recognition result.

6. The sign language recognition method according to claim 5, characterized in that: The prediction head includes a pooling layer, a plurality of regularization layers and an output layer. The step of inputting the decoding matrix into the prediction head to obtain a sign language recognition result includes: Inputting the decoding matrix into the pooling layer to obtain a fusion vector; Inputting the fused vector into each of the regularization layers respectively to obtain a regularization vector; All the regularized vectors are input into the output layer to obtain the sign language recognition result.

7. The sign language recognition method according to claim 1, wherein: The Transformer model further includes an image embedding module, a key coordinate embedding module and a position encoding module. The step of obtaining the first input data and the second input data includes: Acquire image data to be subjected to sign language recognition; Performing embedding processing on the image data by the image embedding module to obtain an image vector; Extracting key points from the image data by means of the key coordinate embedding module, and embedding the key points to obtain key point vectors; Performing position coding on the image vector by the position coding module to obtain an image position coding vector; Performing position encoding on the key point vector by the position encoding module to obtain a key point position encoding vector; The image position encoding vector is determined as first input data, and the key point position encoding vector is determined as second input data.

8. A sign language recognition device, characterized in that: The sign language recognition device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the sign language recognition method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that: The readable storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the sign language recognition method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the sign language recognition method according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Method and device for generating user description text based on text generation network

    CN112949315A

  • Continuous sign language recognition method and device

    CN115393949A

  • Method for carrying out key content segmentation on video and electronic equipment

    CN116342880A

  • Image processing method and device

    CN116823991A

  • Sign language recognition method and device, interaction system and electronic equipment

    CN117095454A