Sign language recognition method, device, readable storage medium and product

Abstract

The application discloses a sign language recognition method and device, a readable storage medium and a product, relates to the technical field of sign language recognition, and a Transformer model comprises a first encoder stack, a second encoder stack and a generator. The method comprises the following steps: for each layer of the first encoder in the first encoder stack, if there is a second encoder corresponding to the first encoder, inputting a first query matrix and a first key matrix of the first encoder into the second encoder corresponding to the first encoder; for each layer of the second encoder in the second encoder stack, if there is a first encoder corresponding to the second encoder, inputting a second query matrix and a second key matrix of the second encoder into the first encoder corresponding to the second encoder; and inputting a first encoding matrix output by the first encoder stack and a second encoding matrix output by the second encoder stack into the generator to obtain a sign language recognition result. The application improves the accuracy of the sign language recognition result.

Description

Technical Field

[0001] This application relates to the field of sign language recognition technology, and in particular to a sign language recognition method, device, readable storage medium and product. Background Technology

[0002] Sign language is an important way for deaf and mute people to obtain and express information. They typically communicate through sign language, which requires both parties to understand it. However, learning sign language requires a significant investment of time and effort for most hearing people, resulting in low sign language proficiency and difficulty communicating with deaf and mute individuals. Therefore, intelligent sign language recognition is of great significance for communication among deaf and mute people.

[0003] After collecting sign language video data from sign language users, this video data can be input into the Transformer model (a deep learning model based on self-attention mechanism) to output sign language recognition results. In the Transformer model, the encoder mainly converts the input data into a q (query) matrix, a k (key) matrix, and a v (value) matrix, and then assigns attention weights based on the transformed q and k matrices, and then learns the model based on the assigned attention weights. However, in this learning method, the encoder in the Transformer model learns independently and cannot make full use of the complementarity between different modal data, resulting in low accuracy of sign language recognition results.

[0004] Therefore, how to improve the accuracy of sign language recognition results is a technical problem that urgently needs to be solved in this field.

[0005] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the Invention

[0006] The main objective of this application is to provide a sign language recognition method, device, readable storage medium, and product, aiming to solve the technical problem of how to improve the accuracy of sign language recognition results.

[0007] To achieve the above objectives, this application provides a sign language recognition method. The sign language recognition method is based on a Transformer model, which includes a first encoder stack, a second encoder stack, and a generator. The first encoder stack includes at least one first encoder connected in sequence, and the second encoder stack includes at least one second encoder connected in sequence. The sign language recognition method includes the following steps:

[0008] Acquire first input data and second input data, wherein the first input data and the second input data represent data of two different modalities;

[0009] The first input data is input to the first encoder stack. For each layer of the first encoder stack, if there is a second encoder corresponding to the first encoder, the first query matrix and the first key matrix of the first encoder are input to the second encoder corresponding to the first encoder.

[0010] The second input data is input to the second encoder stack. For each layer of the second encoder in the second encoder stack, if there is a first encoder corresponding to the second encoder, the second query matrix and the second key matrix of the second encoder are input to the first encoder corresponding to the second encoder.

[0011] The first encoding matrix output from the first encoder stack and the second encoding matrix output from the second encoder stack are input into the generator to obtain the sign language recognition result.

[0012] In one embodiment, before the step of acquiring the first input data and the second input data, the method further includes:

[0013] Obtain the first layer number of each first encoder in the first encoder stack, and obtain the second layer number of each second encoder in the second encoder stack;

[0014] The first encoder with an even number of first layers among the first encoders is selected as the first target encoder, and the second encoder with an even number of second layers among the second encoders is selected as the second target encoder;

[0015] The first target encoder and the second target encoder are configured to correspond one-to-one.

[0016] In one embodiment, before the step of acquiring the first input data and the second input data, the method further includes:

[0017] Obtain the first layer number of each first encoder in the first encoder stack, and obtain the second layer number of each second encoder in the second encoder stack;

[0018] The first encoder with an odd number of first layers is selected as the first target encoder, and the second encoder with an odd number of second layers is selected as the second target encoder.

[0019] The first target encoder and the second target encoder are configured to correspond one-to-one.

[0020] In one embodiment, for a first encoder and a second encoder that have a corresponding relationship, the number of layers of the first encoder is positively correlated with the number of layers of the second encoder.

[0021] In one embodiment, the generator includes a fusion module, a decoding module, and a prediction head. The step of inputting the first encoding matrix output from the first encoder stack and the second encoding matrix output from the second encoder stack into the generator to obtain the sign language recognition result includes:

[0022] The fusion module performs a weighted summation of the first encoding matrix output from the first encoder stack and the second encoding matrix output from the second encoder stack to obtain the encoding matrix.

[0023] The encoding matrix is ​​input into the decoding module to obtain the decoding matrix, wherein the decoding module is used to perform decoding processing on the encoding matrix;

[0024] The decoding matrix is ​​input into the prediction head to obtain the sign language recognition result.

[0025] In one embodiment, the prediction head includes a pooling layer, multiple regularization layers, and an output layer. The step of inputting the decoding matrix into the prediction head to obtain the sign language recognition result includes:

[0026] The decoding matrix is ​​input into the pooling layer to obtain the fusion vector;

[0027] The fusion vector is input into each of the regularization layers to obtain the regularization vector;

[0028] All the regularized vectors are input into the output layer to obtain the sign language recognition result.

[0029] In one embodiment, the Transformer model further includes an image embedding module, a key coordinate embedding module, and a position encoding module. The step of obtaining the first input data and the second input data includes:

[0030] Image data to be recognized by sign language is acquired, and image vectors are obtained by embedding the image data through the image embedding module.

[0031] The key points in the image data are extracted by the key coordinate embedding module, and the key points are embedded to obtain a key point vector.

[0032] The image vector is obtained by performing position encoding on the image vector using the position encoding module;

[0033] The key point location encoding vector is obtained by performing location encoding on the key point vector through the location encoding module.

[0034] The image location encoding vector is determined as the first input data, and the key point location encoding vector is determined as the second input data.

[0035] In addition, to achieve the above objectives, this application also provides a sign language recognition device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the sign language recognition method as described above.

[0036] In addition, to achieve the above objectives, this application also provides a readable storage medium, which is a computer-readable storage medium, on which a program implementing a sign language recognition method is stored, and the program implementing the sign language recognition method is executed by a processor to implement the steps of the sign language recognition method as described above.

[0037] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the sign language recognition method described above.

[0038] One or more technical solutions proposed in this application have at least the following technical effects:

[0039] Sign language recognition is performed based on the Transformer model. The Transformer model includes a first encoder stack, a second encoder stack, and a generator. The first encoder stack includes at least one first encoder connected in sequence, and the second encoder stack includes at least one second encoder connected in sequence. First input data and second input data are acquired, where the first and second input data represent data from two different modalities. The first input data is input into the first encoder stack. For each layer of the first encoder stack, if a second encoder corresponding to the first encoder exists, the first query matrix and first key matrix of the first encoder are input into the corresponding second encoder. The second input data is input into the second encoder stack. For each layer of the second encoder stack, if a first encoder corresponding to the second encoder exists, the second query matrix and second key matrix of the second encoder are input into the corresponding second encoder. The first encoding matrix output from the first encoder stack and the second encoding matrix output from the second encoder stack are input into the generator to obtain the sign language recognition result. Thus, in the embodiments of this application, in the first encoder stack and the second encoder stack that process two different modalities of data, there are corresponding first encoders and second encoders that input their own query matrices and key matrices to each other. This allows the first encoder and second encoder to exchange query matrices and key matrices, enabling them to perform subsequent processing based on each other's query matrices and key matrices, as well as their own value matrices, instead of learning independently using their own query matrices, key matrices, and value matrices. By exchanging query matrices and key matrices, the encoders processing the two different modalities of data can learn from each other during the intermediate learning process, fully exploring and learning the correlation and complementarity between the two different modalities of data, thereby improving the overall recognition performance of the Transformer model and thus improving the accuracy of the recognition results. At the same time, it can better handle the situation where single modal data is disturbed, realize cross-modal semantic understanding and high-level feature representation, and improve the robustness of sign language recognition. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1This is a flowchart illustrating the first embodiment of the sign language recognition method of this application;

[0043] Figure 2 This is a schematic diagram of the data processing flow of the encoder in the Transformer model;

[0044] Figure 3 This is a schematic diagram of encoder data exchange in the sign language recognition method of this application;

[0045] Figure 4 This is a schematic diagram of an encoder for filtering and exchanging data, as described in an embodiment of the sign language recognition method of this application.

[0046] Figure 5 This is a schematic diagram of the overall structure of the Transformer model proposed in this application;

[0047] Figure 6 This is a schematic diagram of the hardware operating environment of the sign language recognition device in the embodiments of this application.

[0048] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0049] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Currently, sign language recognition for deaf and mute individuals primarily focuses on learning from single-modal image data. Single-modal recognition technology is relatively mature: vision-based gesture recognition and sensor-based gesture capture, among other single-modal sign language recognition methods, have a long research history and have accumulated a wealth of experience and technical foundation. However, this approach cannot fully utilize the complementarity between different modalities: independent learning cannot uncover the correlations and synergies between visual, motor, and other modal data, thus limiting further improvements in the accuracy of recognition results.

[0051] Based on this, the main solution of this application is as follows: Sign language recognition is performed based on a Transformer model. The Transformer model includes a first encoder stack, a second encoder stack, and a generator. The first encoder stack includes at least one first encoder connected in sequence, and the second encoder stack includes at least one second encoder connected in sequence. First input data and second input data are acquired, wherein the first input data and second input data represent data from two different modalities. The first input data is input to the first encoder stack. For each layer of the first encoder in the first encoder stack, if a second encoder corresponding to the first encoder exists, the first query matrix and the first key matrix of the first encoder are input to the second encoder corresponding to the first encoder. The second input data is input to the second encoder stack. For each layer of the second encoder in the second encoder stack, if a first encoder corresponding to the second encoder exists, the second query matrix and the second key matrix of the second encoder are input to the second encoder corresponding to the second encoder. The first encoding matrix output from the first encoder stack and the second encoding matrix output from the second encoder stack are input to the generator to obtain the sign language recognition result.

[0052] This application proposes a method where, in the first and second encoder stacks processing two different modalities of data, corresponding first and second encoders input their query and key matrices to each other. This allows the first and second encoders to exchange their query and key matrices, enabling them to perform subsequent processing based on each other's query and key matrices and their own value matrices, instead of learning independently using their own query, key, and value matrices. By exchanging query and key matrices, the encoders processing the two different modalities can learn from each other during the intermediate learning process, fully exploring and learning the correlation and complementarity between the two modalities, thereby improving the overall recognition performance of the Transformer model and thus increasing the accuracy of the recognition results. Simultaneously, it better handles situations where single-modal data is disturbed, achieving cross-modal semantic understanding and high-level feature representation, and improving the robustness of sign language recognition.

[0053] It should be noted that the executing entity of the various embodiments of the sign language recognition method of this application can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or a sign language recognition device capable of realizing the above functions, such as AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, smart glasses, or a robot. The various embodiments of the sign language recognition method of this application do not impose specific limitations in this regard.

[0054] Based on this, this application proposes a sign language recognition method according to a first embodiment. The sign language recognition method is based on a Transformer model for sign language recognition. The Transformer model includes a first encoder stack, a second encoder stack, and a generator. The first encoder stack includes at least one first encoder connected in sequence, and the second encoder stack includes at least one second encoder connected in sequence. Please refer to... Figure 1 The sign language recognition method includes steps S10 to S40:

[0055] Step S10: Obtain first input data and second input data, wherein the first input data and the second input data represent data of two different modalities;

[0056] The total number of layers of the first encoder in the first encoder stack (hereinafter referred to as the first total number of layers) and the total number of layers of the second encoder in the second encoder stack (hereinafter referred to as the second total number of layers) may be the same or different. This embodiment does not impose specific restrictions on this. Preferably, the first total number of layers and the second total number of layers are the same.

[0057] It should be noted that the first input data and the second input data represent two different modalities of data. For example, the first input data represents image modal data (text modal data), and the second input data represents text modal data (image modal data). This embodiment does not impose any specific limitations on this.

[0058] Step S20: Input the first input data into the first encoder stack. For each layer of the first encoder in the first encoder stack, if there is a second encoder corresponding to the first encoder, input the first query matrix and the first key matrix of the first encoder into the second encoder corresponding to the first encoder.

[0059] The first input data is input to the first encoder stack. Specifically, the first input data is input to the first encoder in the first encoder stack, that is, the first input data is input to the first layer encoder in the first encoder stack.

[0060] Understandably, referring to Figure 2As shown, the basic data processing flow of the encoder in the Transformer model is as follows: In each self-attention layer of the multi-head self-attention module, an attention weight is calculated for each word in the input sequence. These weights reflect the importance of each word in the sequence to the current word. Each value matrix is ​​multiplied by its corresponding attention weight to obtain a weighted vector. Each self-attention layer is followed by a simple feedforward neural network layer. The structure of this network layer is usually two linear layers and a non-linear activation function. After the self-attention layer and the feedforward neural network, a residual connection is used, that is, the input is added to the output of the feedforward neural network layer, and then the output is normalized. Finally, the output of the current layer is used as the input of the next layer. The specific calculation process of the weighted vector is as follows: First, the query (q) matrix, key matrix (k), and value (v) matrix are calculated. Then, the attention score is calculated based on the query matrix and key matrix. Finally, the softmax function is used to convert the score into a probabilistic form to obtain the attention weight. Each value matrix is ​​multiplied by its corresponding attention weight to obtain the weighted vector.

[0061] Based on this data processing flow, in this embodiment, for each layer of the first encoder in the first encoder stack, after the query matrix (i.e., the first query matrix) and key matrix (the first key matrix) calculated internally by the first encoder, if there is a second encoder corresponding to the first encoder, then the first query matrix and the first key matrix are input to the second encoder corresponding to the first encoder. Specifically, they can be input into the self-attention layer of the second encoder, so that the second encoder can perform a cross-attention mechanism based on the first query matrix, the first key matrix and the second value matrix calculated by itself, and perform subsequent processing.

[0062] If there is no second encoder corresponding to the first encoder, then this first encoder is based on Figure 2 The data processing flow shown completes the data processing. That is, this first encoder completes the data processing using the encoder's regular data processing flow, without interactively querying the matrix and key matrix.

[0063] Step S30: Input the second input data into the second encoder stack. For each layer of the second encoder in the second encoder stack, if there is a first encoder corresponding to the second encoder, input the second query matrix and the second key matrix of the second encoder into the first encoder corresponding to the second encoder.

[0064] The second input data is input to the second encoder stack. Specifically, the second input data is input to the first encoder in the second encoder stack, that is, the second input data is input to the first layer encoder in the second encoder stack.

[0065] The data processing flow of each layer of the second encoder in the second encoder stack is similar to that of the first encoder, and will not be described again in this embodiment.

[0066] It is understandable that, for the first encoder and the second encoder that have a corresponding relationship, refer to Figure 3 As shown, the first encoder calculates the first query matrix by itself. Figure 3 The q_VD shown in the figure is related to the first bond matrix ( Figure 3 The k_VD shown in the figure is input to the second encoder, which calculates the second query matrix from itself. Figure 3 The q_KP shown in the figure is related to the second bond matrix ( Figure 3 The k_KP shown is input to the second encoder, thus the first encoder and the second encoder exchange the query matrix and the key matrix. It should be noted that... Figure 3 In this context, v_VD represents the first value matrix calculated by the first encoder, and v_KP represents the second value matrix calculated by the second encoder. The first encoder and the second encoder do not exchange their value matrices.

[0067] It should be noted that the correspondence between the first encoder and the second encoder can be set in advance. For example, the first encoder of the i-th layer and the second encoder of the j-th layer correspond one-to-one. The value of i is from 1 to the first total layer number, and the value of j is from 1 to the second total layer number. When the value of i is different, the value of j is also different. This embodiment does not impose specific restrictions on this.

[0068] Furthermore, i can take a preset number of values ​​from 1 to the first total number of layers, and j can also take the same preset number of values ​​from 1 to the second total number of layers. The preset number is less than min{first total number of layers, second total number of layers}, meaning there is at least one pair of first encoders and second encoders that do not have a corresponding relationship. In this way, the preset number of first encoders and second encoders exchange query matrices and key matrices, while the remaining first encoders and second encoders do not exchange query matrices and key matrices, but still use their own query matrices and key matrices. This can reduce the phenomenon that encoder branches that process two modalities of data overemphasize the features of the other branch. That is, by exchanging query matrices and key matrices, the encoders learn the features of the other modal of data, while retaining the encoders that learn the features of their own modal of data, so as to mine, fuse and utilize the features of the two modalities of data, thereby further improving the accuracy of sign language recognition results.

[0069] For example, suppose the first encoder stack includes four sequentially connected first encoders, namely [Encoder 1_1, Encoder 1_2, Encoder 1_3, Encoder 1_4], and the second encoder stack includes four sequentially connected second encoders, namely [Encoder 2_1, Encoder 2_2, Encoder 2_3, Encoder 2_4]. Assume encoder 1_1 corresponds to encoder 2_1, encoder 1_3 corresponds to encoder 2_3, and encoder 1_4 corresponds to encoder 2_4. Then, encoder 1_1 inputs its calculated first query matrix and first key matrix to encoder 2_1. Correspondingly, encoder 2_1 also inputs its calculated second query matrix and second key matrix to encoder 1_1; encoder 1_3 inputs its calculated second query matrix and second key matrix to encoder 1_1. The first query matrix and the first key matrix are input to encoder 2_3. Correspondingly, encoder 2_3 also calculates and inputs the second query matrix and the second key matrix to encoder 1_3. Encoder 1_4 calculates and inputs the first query matrix and the first key matrix to encoder 2_4. Correspondingly, encoder 2_4 calculates and inputs the second query matrix and the second key matrix to encoder 1_4. That is, encoder 1_1 and encoder 2_1 exchange query matrices and key matrices, encoder 1_3 and encoder 2_3 exchange query matrices and key matrices, encoder 1_4 and encoder 2_4 exchange query matrices and key matrices, while encoder 1_2 and encoder 2_2 complete data processing based on their own calculated query matrices and key matrices.

[0070] Step S40: Input the first encoding matrix output by the first encoder stack and the second encoding matrix output by the second encoder stack into the generator to obtain the sign language recognition result.

[0071] It should be noted that the first encoding matrix output by the first encoder stack is specifically the output of the tail encoder in the first encoder stack, that is, the output of the last encoder layer in the first encoder stack. Similarly, the second encoding matrix output by the second encoder stack is specifically the output of the tail encoder in the second encoder stack, that is, the output of the last encoder layer in the second encoder stack.

[0072] It should be noted that the generator's function is to predict the output based on the processed input sequence, such as the first encoding matrix and the second encoding matrix in this embodiment.

[0073] In the Transformer model (a deep learning model based on self-attention mechanism), the prediction head (also known as the decoder head or output head) is part of the model. After being stacked by the encoder or decoder, the prediction head maps these high-dimensional vectors to the dimensions of the target output space and is responsible for generating the final output. Thus, in one specific implementation, the generator includes the prediction head.

[0074] In this embodiment, sign language recognition is performed based on the Transformer model. The Transformer model includes a first encoder stack, a second encoder stack, and a generator. The first encoder stack includes at least one first encoder connected in sequence, and the second encoder stack includes at least one second encoder connected in sequence. First input data and second input data are acquired, where the first and second input data represent data from two different modalities. The first input data is input to the first encoder stack. For each layer of the first encoder stack, if a second encoder corresponding to the first encoder exists, the first query matrix and first key matrix of the first encoder are input to the corresponding second encoder. The second input data is input to the second encoder stack. For each layer of the second encoder stack, if a first encoder corresponding to the second encoder exists, the second query matrix and second key matrix of the second encoder are input to the corresponding second encoder. The first encoding matrix output from the first encoder stack and the second encoding matrix output from the second encoder stack are input to the generator to obtain the sign language recognition result. Thus, in this embodiment, within the first encoder stack and second encoder stack that process two different modalities of data, corresponding first and second encoders input their query matrices and key matrices to each other. This allows the first and second encoders to exchange query and key matrices, enabling them to perform subsequent processing based on each other's query and key matrices and their own value matrices, rather than learning independently using their own query, key, and value matrices. By exchanging query and key matrices, the encoders processing the two different modalities of data can learn from each other during the intermediate learning process, fully exploring and learning the correlation and complementarity between the two modalities, thereby improving the overall recognition performance of the Transformer model and thus improving the accuracy of the recognition results. Simultaneously, it can better handle situations where single-modal data is interfered with, achieving cross-modal semantic understanding and high-level feature representation, and improving the robustness of sign language recognition.

[0075] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Furthermore, before the step of obtaining the first input data and the second input data, the method further includes:

[0076] Step A10: Obtain the first layer number of each first encoder in the first encoder stack, and obtain the second layer number of each second encoder in the second encoder stack;

[0077] It should be noted that the first layer number refers to the layer number of the first encoder in the first encoder stack, and the second layer number refers to the layer number of the second encoder in the second encoder stack. The layers of the encoders (including the first encoder and the second encoder) are arranged sequentially according to the connection order of the encoders.

[0078] Step A20: Select the first encoder in each of the first encoders whose first layer number is even as the first target encoder, and select the second encoder in each of the second encoders whose second layer number is even as the second target encoder;

[0079] It should be noted that the total number of the first target encoders is the same as the total number of the second target encoders. That is, N first encoders with an even number of first layers are selected as the first target encoders and N second encoders with an even number of second layers are selected as the second target encoders, where N is less than or equal to min{number of first layers / 2, number of second layers / 2}.

[0080] For example, suppose the first encoder stack includes four first encoders connected in sequence, namely [encoder1_1, encoder1_2, encoder1_3, encoder1_4], and the second encoder stack includes four second encoders connected in sequence, namely [encoder2_1, encoder2_2, encoder2_3, encoder2_4]. If N is less than or equal to 2, then when N is 2, encoder1_2 and encoder1_4 can be selected as the first target encoders, and encoder2_2 and encoder2_4 can be selected as the second target encoders.

[0081] Step A30: Set the first target encoder and the second target encoder to correspond one-to-one.

[0082] The first target encoder and the second target encoder are assigned a one-to-one correspondence. This correspondence can be randomized or based on preset rules, such as assigning the first target encoder with the same number of layers as the second target encoder. For example, refer to... Figure 4As shown, the first encoder stack includes n layers of first encoders connected in sequence, namely [Encoder VD1, Encoder VD2, ..., Encoder VDn-1, Encoder VDn]. The second encoder stack includes n layers of first encoders connected in sequence, namely [Encoder KP1, Encoder KP2, ..., Encoder KPn-1, Encoder KPn]. The first encoder with an even first layer number is selected as the first target encoder, and the second encoder with an even second layer number is selected as the second target encoder. The first target encoder Encoder VDi with an i-th first layer number is set to correspond with the second target encoder Encoder KPi with an i-th second layer number, where i takes the value of...

[0083] As another implementation, before the step of acquiring the first input data and the second input data, the method further includes:

[0084] Step B10: Obtain the first layer number of each first encoder in the first encoder stack, and obtain the second layer number of each second encoder in the second encoder stack;

[0085] Step B20: Select the first encoder with an odd number of first layers among the first encoders as the first target encoder, and select the second encoder with an odd number of second layers among the second encoders as the second target encoder;

[0086] Step B30: Set the first target encoder and the second target encoder to correspond one-to-one.

[0087] It should be noted that the specific implementation method of selecting the first encoder with an odd number of layers as the first target encoder and selecting the second encoder with an odd number of layers as the second target encoder, and setting the first target encoder and the second target encoder to correspond one-to-one, is similar to the specific implementation method of selecting the encoder with an even number of layers as the target encoder described above. It can be referred to the above description, and will not be repeated in this embodiment.

[0088] In this embodiment, the first encoder with an even (odd) number of first layers among the first encoders is selected as the first target encoder, and the second encoder with an even (odd) number of second layers among the second encoders is selected as the second target encoder. A one-to-one correspondence is established between the first and second target encoders. Thus, the encoders of even (odd) layers (including the first and second encoders) exchange query matrices and key matrices to learn the features of the other modality data, while the encoders of odd (even) layers do not exchange query matrices and key matrices and learn the features of their own modality data. This alternating learning of the features of the other modality data and the features of the own modality data achieves balanced learning of both the other modality data and the features of the own modality data. This balanced fusion of the features of the other modality data and the features of the own modality data allows the encoder to fully extract and utilize the features of both the other modality data, thereby further improving the accuracy of sign language recognition results.

[0089] In one possible implementation, for a first encoder and a second encoder that have a corresponding relationship, the number of layers of the first encoder is positively correlated with the number of layers of the second encoder.

[0090] In this embodiment, the first encoder of layer i corresponds to the second encoder of layer j. Different values ​​of i correspond to different values ​​of j, and the smaller the value of i, the smaller the value of j. Thus, the lower the layer of the first encoder corresponds to the lower the layer of the second encoder. The number of layers of the encoders exchanging the query matrix and the key matrix is ​​relatively consistent, ensuring that the abstraction level of the features learned by the exchanged query matrix and the key matrix is ​​relatively consistent. This allows the encoder to more effectively learn subsequent features based on the exchanged query matrix and key matrix.

[0091] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the first and second embodiments described above can be referred to the above description and will not be repeated hereafter. Based on this, the generator includes a fusion module, a decoding module, and a prediction head. The step of inputting the first encoding matrix output from the first encoder stack and the second encoding matrix output from the second encoder stack into the generator to obtain the sign language recognition result includes:

[0092] Step C10: The first encoding matrix output by the first encoder stack and the second encoding matrix output by the second encoder stack are weighted and summed by the fusion module to obtain the encoding matrix;

[0093] The fusion module is connected to the first encoder stack and the second encoder stack respectively to fuse the output of the first encoder stack and the output of the second encoder stack.

[0094] Furthermore, the first and second encoding matrices can be weighted and fused based on their respective fusion weights to obtain the encoding matrix. Regarding the setting of the fusion weights, relevant personnel can set the fusion weights corresponding to the first and second encoding matrices separately, or the fusion weights can be set as model parameters and optimized during model training. This embodiment does not impose specific limitations on this approach.

[0095] Let p be the fusion weight corresponding to the first encoding matrix, IA be the first encoding matrix, q be the fusion weight corresponding to the second encoding matrix, KA be the second encoding matrix, and EA be the encoding matrix. Then EA can be expressed by the formula EA = p * IA + q * KA. In one specific implementation, the sum of p and q is one.

[0096] Step C20: Input the encoding matrix into the decoding module to obtain the decoding matrix, wherein the decoding module is used to perform decoding processing on the encoding matrix;

[0097] The decoding module receives the encoding matrix input from the encoding module, decodes the encoding matrix to obtain the output sequence, i.e., the decoding matrix, and inputs the decoding matrix into the prediction head.

[0098] Personnel can design decoding modules according to actual needs, and the decoding matrix is ​​the output of the decoding module. In one specific implementation, the decoding module can be a decoder stack consisting of one or more decoders connected in sequence. In this case, the decoding matrix is ​​the output of the decoder stack, that is, the output of the last layer of the decoder in the decoder stack.

[0099] Step C30: Input the decoding matrix into the prediction head to obtain the sign language recognition result.

[0100] It should be noted that the sign language recognition result is the output of the prediction head.

[0101] In the Transformer model, the prediction head is used to convert the model's internal representation into the final prediction output. In this embodiment, the prediction head is used to convert the decoding matrix into the final prediction output, which is the sign language recognition result.

[0102] Personnel can design a prediction head according to actual needs. In one specific implementation, the prediction head includes a pooling layer, a regularization layer (dropout), a fully connected layer and a softmax function connected in sequence, and finally predicts the output through the softmax function.

[0103] In one possible implementation, the prediction head includes a pooling layer, multiple regularization layers, and an output layer. The step of inputting the decoding matrix into the prediction head to obtain the sign language recognition result includes:

[0104] Step D10: Input the decoding matrix into the pooling layer to obtain the fusion vector;

[0105] Understandably, in the Transformer model (a deep learning model based on self-attention), the prediction head (also known as the decoder head or output head) is part of the model. After being stacked by the encoder or decoder, the prediction head maps these high-dimensional vectors to the dimensions of the target output space, responsible for generating the final output. In classification tasks, the prediction head typically consists of a pooling layer, a regularization layer (Dropout layer), and an output layer connected in sequence.

[0106] However, this Transformer model's prediction head design uses a single-layer Dropout structure. While researchers might use different Dropout ratios as hyperparameters to fine-tune the model, this approach has limited impact on improving generalization ability. Therefore, this embodiment designs multiple parallel regularization layers [Dropout1, Dropout2, ..., Dropout n] to reduce overfitting and improve generalization ability. Here, n represents the total number of regularization layers, specifically an integer greater than 1.

[0107] It should be noted that this fusion vector is the output of the pooling layer.

[0108] This pooling layer fuses the vector sequences in the feature matrix to aggregate them into a fused vector with fixed dimensions and a fixed length.

[0109] The pooling layer can specifically be an average pooling layer, a max pooling layer, a global pooling layer, etc., and this embodiment does not impose any specific limitations on it. Considering that compared with other pooling operations, the average pooling operation can preserve the overall characteristics of the sequence because it performs an averaging operation on the sequence, it not only considers the maximum and minimum features in the sequence, but also the overall trend and distribution. Average pooling can also enhance robustness and reduce overfitting. Based on this, in a preferred embodiment, the pooling layer is specifically an average pooling layer.

[0110] Step D20: Input the fusion vector into each of the regularization layers to obtain the regularization vector;

[0111] It should be noted that the regularization vector is the output of the regularization layer.

[0112] Dropout is a technique that randomly "deactivates" (i.e., temporarily ignores) a subset of neurons in a neural network during training. These deactivated neurons do not participate in computation during the current forward and backward propagation. By randomly "deactivating" a portion of neurons in the network during training, the network's dependence on specific training samples is reduced, thereby improving the model's generalization ability on unseen data. Therefore, a Dropout layer is added after the average pooling layer.

[0113] The fusion vector is obtained by randomly deactivating the fusion vector through multiple regularization layers (Dropout), that is, the fusion vector is input into each regularization layer.

[0114] It should be noted that the discard ratio of each regularization layer can be the same or different, and this embodiment does not impose specific restrictions on this.

[0115] Step D30: Input all the regularized vectors into the output layer to obtain the sign language recognition result.

[0116] It should be noted that the sign language recognition result is the output of the output layer.

[0117] Relevant personnel can design the output layer according to actual needs. In one specific implementation, the output layer includes a fully connected layer and a softmax function connected in sequence.

[0118] It should be noted that if the current process involves training a Transformer model for sign language recognition, meaning this Transformer model is not pre-trained, then after obtaining the sign language recognition result, the function value of a preset loss function (hereinafter referred to as the loss value) can be calculated based on the sign language recognition result. This loss value is then used to optimize the Transformer model until a preset training termination condition is met, resulting in a fully trained Transformer model. The preset loss function can be any pre-set loss function, such as the cross-entropy loss function; this embodiment does not impose specific restrictions on it.

[0119] The Transformer model proposed in this embodiment sets up multiple regularization layers to perform random deactivation processing using multiple regularization layers. Multiple regularization layers generate multiple different sub-network models, so that the model algorithm does not focus on only a certain branch, thereby reducing the risk of overfitting and improving the model's generalization ability. This allows the model to obtain relatively accurate recognition results even for data that was not "seen" during training, thereby reducing the occurrence of errors and fluctuations in recognition results and improving the stability of sign language recognition results.

[0120] Furthermore, the weight parameters of all fully connected layers are consistent, including the weight matrix and the bias term. That is, all fully connected layers share the same parameters. In this way, the use of parameter sharing in the fully connected layers after the Dropout layer reduces the number of model parameters, thereby reducing the computational cost of the model and also reducing the risk of overfitting.

[0121] Furthermore, different dropout ratios in each regularization layer can provide different regularization strengths, thereby improving the model's generalization ability. Applying different dropout ratios can also increase the model's diversity, allowing the model to be trained under different configurations. This diversity helps the model adapt better to unknown data when in use, and different dropout ratios can be customized, thus allowing for more flexible control of the model's training process. This helps in finding the optimal training strategy to balance the model's accuracy and generalization ability.

[0122] In one possible implementation, the Transformer model further includes an image embedding module, a key coordinate embedding module, and a position encoding module. The step of obtaining the first input data and the second input data includes:

[0123] Step E10: Obtain image data to be recognized by sign language, and perform embedding processing on the image data through the image embedding module to obtain an image vector;

[0124] The image data includes one or more frames. Furthermore, after acquiring the image data, keyframe images can be extracted. For example, a Shot Boundary Detection (SBD) algorithm can be used to extract keyframe images from the image data. Keyframe images are retained while non-keyframe images are deleted to improve the image quality. The image data with retained keyframe images is then used for subsequent processing.

[0125] Embedding processing of image data can specifically involve embedding images within the image data.

[0126] Embedding is the process of converting input data (such as words, characters, sentences, or any serializable entity) into a fixed-length vector representation. Embedding is the foundation for a model to understand input data.

[0127] The image embedding module is used to embed image data to obtain image vectors. Specifically, the embedding process is as follows:

[0128] (1) First, scale the image in the image data to a uniform size, such as 224×224×3. Then, divide the image into small patches, each patch having the same size, such as 16×16×3. Then, arrange the small patches.

[0129] (2) Perform an Embedding operation on the arranged patches to obtain the image vector.

[0130] Step E20: Extract key points from the image data through the key coordinate embedding module, and embed the key points to obtain key point vectors;

[0131] The keypoint embedding module is used to extract keypoints (actually, coordinates) from an image. Specifically, it can extract keypoints of the face and hands. Keypoints of the hands are denoted as hand keypoints, and keypoints of the face as facial keypoints. Specifically, a preset number of hand keypoints and a preset number of facial keypoints can be extracted. These preset number of facial keypoints may include eye keypoints (keypoints for the glasses), nose keypoints (keypoints for the nose), mouth keypoints (keypoints for the mouth), and contour keypoints (keypoints for the facial contour). In one specific embodiment, 21 hand keypoints are extracted from a single hand, for a total of 42 hand keypoints from both hands. Simultaneously, 58 facial keypoints are extracted, specifically including 17 contour keypoints, 12 eye keypoints, 9 nose keypoints, and 20 mouth keypoints.

[0132] Understandably, a key point can be represented by coordinates, thereby extracting the actual coordinates of each key point.

[0133] Step E30: The image vector is position-encoded by the position encoding module to obtain the image position-encoded vector;

[0134] Step E40: The key point vector is position-encoded by the position encoding module to obtain the key point position encoding vector;

[0135] Positional encoding refers to adding information about the position of each word in a sentence, also known as "information fragment". Specifically, absolute positional encoding, relative positional encoding, etc. can be used, but this embodiment does not impose any specific restrictions on this.

[0136] Step E50: Determine the image position encoding vector as the first input data, and determine the key point position encoding vector as the second input data.

[0137] It should be noted that in this embodiment, the first input data represents the data of the image modality, and the second input data represents the data of the text modality (the key points are actually text data in coordinate form).

[0138] In this embodiment, to allow the model to focus more on the hand and facial information of the sign language interpreter during the expression of sentences, a key point embedding module is set up to extract key points of the hands and face in the image, thereby improving the performance of the model. At the same time, to avoid the influence of noise on key points, an image embedding module is set up to embed the image, thereby fusing image and text data for sign language recognition. By utilizing the complementary advantages of the two modalities of data, features are learned together, further improving the accuracy of sign language recognition results.

[0139] As an example, to aid in understanding the technical concept or principle of the sign language recognition method combined with the first and second embodiments, please refer to... Figure 5 As shown, the Transformer model specifically includes an input module, an image encoder stack (i.e., the first encoder stack), a keypoint encoder stack (i.e., the second encoder stack), and a generator. The generator includes a fusion module, a decoding module, and a prediction head connected in sequence. The image encoder stack includes n layers of encoders (Encoder VD) connected in sequence, and the keypoint encoder stack includes n layers of encoders (Encoder KP) connected in sequence. The decoding module includes m layers of decoders connected in sequence. The input module includes an image embedding module and a keypoint embedding module, which are respectively connected to the location encoding module.

[0140] In the image layer encoder stack and the keypoint encoder stack, encoders in even-numbered layers exchange query matrices and key matrices, while encoders in odd-numbered layers do not exchange query matrices. Furthermore, encoders that exchange query and key matrices have the same number of layers. Specifically, the encoders that exchange query and key matrices... VD The q_VD and k_VD of the branch will be passed to the Encoder. KP In the branch, with Encoder KP The v_VD of the branch performs a cross-attention mechanism. Similarly, this comes from the Encoder. KP The q_KP and k_KP of the branch will be passed to the Encoder. VD In the branch, with Encoder VD The v_VD of the branch performs a cross-attention mechanism. This implements the Encoder. KP and Encoder VD Information exchange and learning between them.

[0141] The fusion module performs weighted fusion of the outputs of the image encoder stack and the keypoint encoder stack, and inputs the fusion result into each decoder of the decoder module. Specifically, the fusion weight corresponding to the image encoding matrix output by the image encoder stack is α, and the fusion weight corresponding to the keypoint encoding matrix output by the keypoint encoder stack is 1-α. The image encoding matrix and the keypoint encoding matrix are weighted and fused based on their respective fusion weights.

[0142] It should be noted that the final output of the n-layer encoder connected in sequence is input to each of the m-layer decoders connected in sequence.

[0143] It should be noted that if the current process is training a Transformer model, the preset expected result labels (such as...) will also be applied. Figure 5 The "business license" shown in the image, which is the expected output of the model, is embedded and encoded sequentially before being input into the decoding module. At the same time, the keyframe image corresponding to the result label is input into the input module. Based on the input of the preset expected result label and the input of the image data, the sign language recognition model is trained to obtain the trained Transformer model for sign language recognition.

[0144] Furthermore, this application also proposes a sign language recognition device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the sign language recognition method described above.

[0145] refer to Figure 6 The diagram illustrates a structural schematic of a sign language recognition device suitable for implementing embodiments of this application. The sign language recognition device in the embodiments of this application may also include, but is not limited to, mobile terminals such as AR glasses, smart glasses, robots, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The sign language recognition device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0146] like Figure 6As shown, the sign language recognition device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the sign language recognition device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the sign language recognition device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show sign language recognition devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0147] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0148] The sign language recognition device provided in this application, employing the sign language recognition method described in the above embodiments, can solve the technical problem of how to improve the accuracy of sign language recognition results. Compared with the prior art, the beneficial effects of the sign language recognition device provided in this application are the same as those of the sign language recognition method provided in the above embodiments, and other technical features of this sign language recognition device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0149] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0150] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0151] In addition, to achieve the above objectives, embodiments of this application also provide a readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the sign language recognition method in the above embodiments.

[0152] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0153] The aforementioned computer-readable storage medium may be included in the sign language recognition device; or it may exist independently and not be assembled into the sign language recognition device.

[0154] The aforementioned computer-readable storage medium carries one or more programs. When the one or more programs are executed by the sign language recognition device, the sign language recognition device: acquires first input data and second input data, wherein the first input data and the second input data represent data of two different modalities; inputs the first input data to a first encoder stack, and for each layer of the first encoder stack, if there exists a second encoder corresponding to the first encoder, inputs the first query matrix and the first key matrix of the first encoder to the second encoder corresponding to the first encoder; inputs the second input data to a second encoder stack, and for each layer of the second encoder stack, if there exists a first encoder corresponding to the second encoder, inputs the second query matrix and the second key matrix of the second encoder to the first encoder corresponding to the second encoder; and inputs the first encoding matrix output by the first encoder stack and the second encoding matrix output by the second encoder stack to the generator to obtain a sign language recognition result.

[0155] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0157] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the modules themselves.

[0158] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described sign language recognition method, thereby solving the technical problem of how to improve the accuracy of sign language recognition results. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the sign language recognition method provided in the above embodiments, and will not be repeated here.

[0159] Furthermore, embodiments of this application also propose a computer program product, including a sign language recognition program, which, when executed by a processor, implements the steps of the sign language recognition method described above.

[0160] The specific implementation of the computer program product in this application is basically the same as the various embodiments of the sign language recognition method described above, and will not be repeated here.

[0161] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0162] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0163] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software sensor. This computer software sensor is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0164] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A sign language recognition method, characterized in that, The sign language recognition method is based on a Transformer model for sign language recognition. The Transformer model includes a first encoder stack, a second encoder stack, and a generator. The first encoder stack includes at least one first encoder connected in sequence, and the second encoder stack includes at least one second encoder connected in sequence. The sign language recognition method includes the following steps: Acquire first input data and second input data, wherein the first input data and the second input data represent data of two different modalities; The first input data is input to the first encoder stack. For each layer of the first encoder stack, if there is a second encoder corresponding to the first encoder, the first query matrix and the first key matrix of the first encoder are input to the second encoder corresponding to the first encoder. The second input data is input to the second encoder stack. For each layer of the second encoder in the second encoder stack, if there is a first encoder corresponding to the second encoder, the second query matrix and the second key matrix of the second encoder are input to the first encoder corresponding to the second encoder. The first encoding matrix output from the first encoder stack and the second encoding matrix output from the second encoder stack are input into the generator to obtain the sign language recognition result.

2. The sign language recognition method as described in claim 1, characterized in that, Before the step of acquiring the first input data and the second input data, the method further includes: Obtain the first layer number of each first encoder in the first encoder stack, and obtain the second layer number of each second encoder in the second encoder stack; The first encoder with an even number of first layers among the first encoders is selected as the first target encoder, and the second encoder with an even number of second layers among the second encoders is selected as the second target encoder; The first target encoder and the second target encoder are configured to correspond one-to-one.

3. The sign language recognition method as described in claim 1, characterized in that, Before the step of acquiring the first input data and the second input data, the method further includes: Obtain the first layer number of each first encoder in the first encoder stack, and obtain the second layer number of each second encoder in the second encoder stack; The first encoder with an odd number of first layers is selected as the first target encoder, and the second encoder with an odd number of second layers is selected as the second target encoder. The first target encoder and the second target encoder are configured to correspond one-to-one.

4. The sign language recognition method as described in any one of claims 1 to 3, characterized in that, For a first encoder and a second encoder that have a corresponding relationship, the number of layers of the first encoder is positively correlated with the number of layers of the second encoder.

5. The sign language recognition method as described in claim 1, characterized in that, The generator includes a fusion module, a decoding module, and a prediction head. The step of inputting the first encoding matrix output from the first encoder stack and the second encoding matrix output from the second encoder stack into the generator to obtain the sign language recognition result includes: The fusion module performs a weighted summation of the first encoding matrix output from the first encoder stack and the second encoding matrix output from the second encoder stack to obtain the encoding matrix. The encoding matrix is ​​input into the decoding module to obtain the decoding matrix, wherein the decoding module is used to perform decoding processing on the encoding matrix; The decoding matrix is ​​input into the prediction head to obtain the sign language recognition result.

6. The sign language recognition method as described in claim 5, characterized in that, The prediction head includes a pooling layer, multiple regularization layers, and an output layer. The step of inputting the decoding matrix into the prediction head to obtain the sign language recognition result includes: The decoding matrix is ​​input into the pooling layer to obtain the fusion vector; The fusion vector is input into each of the regularization layers to obtain the regularization vector; All the regularized vectors are input into the output layer to obtain the sign language recognition result.

7. The sign language recognition method as described in claim 1, characterized in that, The Transformer model further includes an image embedding module, a key coordinate embedding module, and a position encoding module. The steps of obtaining the first input data and the second input data include: Image data to be recognized by sign language is acquired, and image vectors are obtained by embedding the image data through the image embedding module. The key points in the image data are extracted by the key coordinate embedding module, and the key points are embedded to obtain a key point vector. The image vector is obtained by performing position encoding on the image vector using the position encoding module; The key point location encoding vector is obtained by performing location encoding on the key point vector through the location encoding module. The image location encoding vector is determined as the first input data, and the key point location encoding vector is determined as the second input data.

8. A sign language recognition device, characterized in that, The sign language recognition device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the sign language recognition method as described in any one of claims 1 to 7.

9. A readable storage medium, characterized in that, The readable storage medium is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the sign language recognition method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the sign language recognition method as described in any one of claims 1 to 7.

Images