A sign language recognition method, device, electronic device and computer storage medium

Abstract

The embodiments of the present application provide a sign language recognition method, device, electronic device and computer-readable storage medium, wherein the method comprises: obtaining multiple video frames of a sign language video; obtaining image features of the multiple video frames according to the multiple video frames; encoding the image features of the multiple video frames to obtain a first encoding feature; obtaining text corresponding to the multiple video frames; encoding the text corresponding to the multiple video frames to obtain a second encoding feature; inputting the first encoding feature and the second encoding feature into an encoder to obtain an encoding feature vector; inputting the encoding feature vector into a decoder to obtain synchronized speech. The implementation of the above embodiments can solve the mapping problem of video stream, standard sign language and synchronized speech, thereby improving the accuracy of the synchronized speech obtained.

Description

Technical Field

[0001] The present application relates to the field of intelligent recognition technology, and in particular to a sign language recognition method, device, electronic device and computer-readable storage medium. Background Art

[0002] Sign language is the mother tongue of the hearing-impaired and their main medium of communication. As a visual language, sign language naturally uses the shape, movement and posture of the palm to express specific semantics, and also uses facial expressions and body language to supplement the meaning and improve the expression. Due to the low popularity of sign language, in real life, communication barriers with normal people are still a difficult problem that the hearing-impaired face and need to solve urgently. At present, the sign language recognition tools or systems in the industry are generally based on image processing recognition algorithms and text-to-speech technology. The principle of image processing algorithm recognition is to split the sign language expression into a sequence of pictures, identify the standard sign language corresponding to the picture sequence, and then translate the standard sign language into speech. Since sign language is a visual language, the carriers of its information transmission include not only gestures but also expressions and body movements, so it is difficult to simply map the picture sequence and the standard sign language. At the same time, because the expression order of sign language and speech is not exactly the same, it is difficult to handle the direct mapping relationship between sign language and speech by relying solely on the current image processing algorithm and text-to-speech technology. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a sign language recognition method that can generate synchronized speech based on various information in the acquired sign language video screen and the text corresponding to the sign language video screen.

[0004] In a first aspect, an embodiment of the present application provides a sign language recognition method, comprising:

[0005] Obtain multiple frames of sign language video;

[0006] Acquire image features of the multiple video frames according to the multiple video frames;

[0007] Encoding the image features of the multiple video frames to obtain first encoded features;

[0008] Obtain text corresponding to the multiple video frames;

[0009] Encoding the texts corresponding to the multiple frames of video to obtain a second encoding feature;

[0010] Inputting the first encoding feature and the second encoding feature into an encoder to obtain an encoding feature vector;

[0011] The encoded feature vector is input into a decoder to obtain synchronized speech.

[0012] In the above implementation process, multiple video frames of the sign language video are obtained, image features in the multiple video frames are extracted, and the image features are encoded to obtain a first encoding feature. Different from the prior art, the image features of the video screen are directly obtained instead of just obtaining the sign language semantics based on video screen recognition. The obtained first encoding feature preserves a variety of information in the video screen, which is conducive to generating more accurate speech; the text corresponding to the multiple video frames is obtained, the text corresponding to the multiple video frames is encoded, and the second encoding feature is obtained. The first encoding feature and the second encoding feature are input into the encoder to obtain an encoding feature vector, and the encoding feature vector is input into the decoder to obtain synchronized speech. In the process of generating synchronized speech, image features and text are used, and the text and image features are integrated through encoding. Through the intermediate medium of text, the sign language-language conversion problem can be converted into a language translation problem, solving the mapping problem between sign language video stream-standard sign language-synchronized speech, and thus generating more accurate and real-time synchronized speech.

[0013] Furthermore, the step of acquiring image features of the multiple video frames according to the multiple video frames includes:

[0014] Generate a pixel matrix of the multiple video frames according to the multiple video frames;

[0015] The pixel matrix of the multiple frames of video images is input into a convolutional neural network to obtain image features of the multiple frames of video images.

[0016] In the above implementation process, the convolutional neural network has the advantage of accurately extracting features. It generates a corresponding pixel matrix based on multiple frames of video, and inputs the pixel matrix into the convolutional neural network as input information. The obtained image features are useful features in the multiple frames of video.

[0017] Furthermore, the step of encoding the image features of the multiple frames of video to obtain a first encoding feature includes:

[0018] generating a spatial embedding tensor according to the image features;

[0019] generating a first position encoding according to the spatial embedding tensor;

[0020] The spatial embedding tensor and the first position encoding are processed by using a preset algorithm to obtain the first encoding feature.

[0021] In the above implementation process, the image features are processed to generate a spatial embedding tensor so that the image features meet the calculation requirements. The embedding tensor contains information corresponding to gestures and expressions. The order of this information can determine whether the generated synchronized speech is fluent. Therefore, a first position code is generated according to the spatial embedding tensor, and the spatial embedding tensor and the first position code are processed using a preset algorithm to obtain a first coding feature. Based on the above implementation, the generated synchronized speech can be made more fluent.

[0022] Furthermore, the step of using a preset algorithm to process the spatial embedding tensor and the first position code to obtain the first coding feature includes:

[0023] Generate a first tensor according to the spatial embedding tensor and the first position encoding;

[0024] Generate a plurality of first sub-tensors according to the first tensor;

[0025] Obtaining weight matrices corresponding to the multiple first sub-tensors;

[0026] Generate a plurality of computation tensors corresponding to the plurality of first sub-tensors according to the plurality of first sub-tensors and weight matrices corresponding to the plurality of first sub-tensors;

[0027] Encoding the multiple calculation tensors to obtain sign language representation tensors of the multiple video frames;

[0028] The sign language representation tensor is the first encoding feature.

[0029] In the above implementation process, a method for obtaining the first coding feature is proposed. This method constructs multiple first sub-tensors. Different first sub-tensors correspond to different vector spaces. Based on this, the encoding ability of the algorithm can be improved, and the final generated sign language representation tensor has more accurate information.

[0030] Furthermore, the step of encoding the multiple calculation tensors corresponding to the multiple first sub-tensors to obtain the sign language representation tensors of the multiple frames of video images includes:

[0031] Inputting a plurality of calculation tensors corresponding to the first tensor into a feedforward neural network to obtain a plurality of output tensors;

[0032] Normalizing the multiple output tensors;

[0033] A sign language representation tensor of the multiple frames of video images is generated according to the multiple normalized output tensors.

[0034] In the above implementation process, a method for generating a sign language representation tensor is proposed. Multiple calculation tensors corresponding to multiple first tensors are input into a feedforward neural network and normalized. This can further extract features from the video screen, thereby further improving the matching degree of synchronized speech and sign language.

[0035] Furthermore, the step of encoding the text corresponding to the multiple frames of video to obtain a second encoding feature includes:

[0036] Generate a word embedding tensor corresponding to the text according to the text;

[0037] Generating a second positional encoding according to the word embedding tensor;

[0038] The word embedding tensor and the second position code are input into a time series model to obtain the second coding feature.

[0039] In the above implementation process, the text is processed to generate a word embedding tensor so that the text meets the computing requirements. The embedding tensor contains vocabulary information. The order of the vocabulary information can determine whether the generated synchronized speech is fluent. Therefore, a second position code is generated according to the spatial embedding tensor, and the word embedding tensor and the second position code are input into the time series model to obtain a second coding feature. Based on the above implementation, the generated synchronized speech can be made more fluent.

[0040] Furthermore, the encoder and the decoder include: a Self-Attention layer.

[0041] In the above implementation process, the Self-Attention layer is based on the sub-attention mechanism, which can focus on the local details of the input encoding features, thereby improving the accuracy of sign language recognition.

[0042] In a second aspect, an embodiment of the present application provides a sign language recognition device, comprising:

[0043] A video image acquisition module is used to acquire multiple frames of video images of a sign language video;

[0044] An image feature acquisition module, used for acquiring image features of the multiple video frames according to the multiple video frames;

[0045] A first encoding module, used for encoding image features of the multiple video frames to obtain first encoding features;

[0046] A text acquisition module, used to acquire texts corresponding to the multiple video frames;

[0047] A second encoding module is used to encode the text corresponding to the multiple frames of video images to obtain a second encoding feature;

[0048] A third encoding module, used for inputting the first encoding feature and the second encoding feature into an encoder to obtain an encoding feature vector;

[0049] The decoding module is used to input the encoded feature vector into a decoder to obtain synchronized speech.

[0050] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in any one of the first aspects when executing the computer program.

[0051] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which instructions are stored. When the instructions are executed on a computer, the computer executes the method as described in any one of the first aspects.

[0052] Other features and advantages disclosed in the present application will be described in the following description, or some features and advantages can be inferred or determined without doubt from the description, or can be learned by implementing the above-mentioned technology disclosed in the present application.

[0053] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 A flowchart of a sign language recognition method provided in an embodiment of the present application;

[0056] Figure 2 A schematic diagram of a process for obtaining image features provided in an embodiment of the present application;

[0057] Figure 3 A schematic diagram of a process for obtaining a first coding feature provided in an embodiment of the present application;

[0058] Figure 4 A schematic diagram of a process for obtaining a first coding feature using a preset algorithm provided in an embodiment of the present application;

[0059] Figure 5 A schematic diagram of a process for obtaining a sign language representation tensor provided in an embodiment of the present application;

[0060] Figure 6 A schematic diagram of a process for obtaining a second coding feature provided in an embodiment of the present application;

[0061] Figure 7 A schematic diagram of the structure of a sign language recognition device provided in an embodiment of the present application;

[0062] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0064] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0065] Example 1

[0066] See also Figure 1 , the embodiment of the present application provides a sign language recognition method, comprising:

[0067] S1: Acquire multiple frames of sign language video;

[0068] S2: Obtaining image features of multiple video frames according to the multiple video frames;

[0069] S3: Encode the image features of multiple video frames to obtain first encoded features;

[0070] S4: Obtain text corresponding to multiple video frames;

[0071] S5: Encode the texts corresponding to the multiple frames of video to obtain a second encoding feature;

[0072] S6: Input the first encoding feature and the second encoding feature into the encoder to obtain an encoding feature vector;

[0073] S7: Input the encoded feature vector into the decoder to obtain synchronized speech.

[0074] In the above embodiment, a video of a user expressing in sign language can be captured by a camera. At the same time, in order to improve the recognition accuracy, the sign language video screen should include the user's expression, the movements of other parts of the body, etc. as much as possible. The image feature can be in the form of a pixel matrix. The text corresponding to the multi-frame video screen is the text corresponding to the sign language in the multi-frame video screen that is recognized. Each frame of the video screen has its corresponding pixel matrix and text. Exemplarily, the vocabulary represented by the sign language in the multi-frame video screen can be obtained by recognizing and annotating the video screen in advance, thereby obtaining the text corresponding to the multi-frame video screen.

[0075] In the above embodiment, multiple video frames of the sign language video are obtained, image features in the multiple video frames are extracted, and the image features are encoded to obtain a first encoding feature. Different from the prior art, the image features of the video screen are directly obtained instead of just obtaining the sign language semantics based on video screen recognition. The obtained first encoding feature preserves a variety of information in the video screen, which is conducive to generating more accurate speech; the text corresponding to the multiple video frames is obtained, the text corresponding to the multiple video frames is encoded, and the second encoding feature is obtained. The first encoding feature and the second encoding feature are input into the encoder to obtain an encoding feature vector, and the encoding feature vector is input into the decoder to obtain synchronized speech. In the process of generating synchronized speech, image features and text are used, and the text and image features are fused through encoding, so that the mapping relationship between speech and text is clearer, thereby generating more accurate and real-time synchronized speech.

[0076] Further, see Figure 2 , the embodiment of the present application provides a method for implementing S2, including:

[0077] S21: Generate a pixel matrix of multiple video frames according to the multiple video frames;

[0078] S22: Inputting the pixel matrix of the multiple video frames into the convolutional neural network to obtain image features of the multiple video frames.

[0079] In the above implementation process, the convolutional neural network has the advantage of accurately extracting features. It generates a corresponding pixel matrix based on multiple frames of video, and inputs the pixel matrix into the convolutional neural network as input information. The obtained image features are useful features in the multiple frames of video.

[0080] Exemplarily, the convolutional neural network formula processes the pixel matrix of multiple video frames as follows:

[0081]

[0082] Among them, t is the serial number of the video screen, u is the number of rows in the weight matrix, v is the number of columns in the weight matrix, k is the dimension of the weight matrix corresponding to the convolution kernel, i is the row number of the pixel matrix, and j is the column number of the pixel matrix. is the image feature corresponding to the pixel image of the tth frame; is the element in the uth row and vth column of the weight matrix corresponding to the convolution kernel corresponding to the pixel image of the tth frame, It is the element in the i+uth row and j+vth column in the pixel matrix corresponding to the tth frame of the video.

[0083] See also Figure 3 , the embodiment of the present application provides an implementation method of S3, including:

[0084] S31: Generate spatial embedding tensor based on image features;

[0085] The spatial embedding tensor is a tensor corresponding to a preset tensor space. For example, the matrix corresponding to the image features is transformed to obtain a spatial embedding tensor corresponding to the dimension of the tensor space.

[0086] Tensor space is an important concept in multilinear algebra, and is defined as a vector space with tensor mappings.

[0087] S32: Generate a first position code according to the spatial embedding tensor;

[0088] Positional encoding is the process of encoding the position of a word in a sequence.

[0089] S33: Using a preset algorithm to process the spatial embedding tensor and the first position encoding to obtain a first encoding feature.

[0090] In the above embodiment, the image features are processed to generate a spatial embedding tensor so that the image features meet the computing requirements. The embedding tensor contains information corresponding to gestures and expressions. The order of this information can determine whether the generated synchronized speech is fluent. Therefore, a first position code is generated according to the spatial embedding tensor, and the spatial embedding tensor and the first position code are processed using a preset algorithm to obtain a first coding feature. Based on the above implementation, the generated synchronized speech can be made more fluent.

[0091] See also Figure 4 , the embodiment of the present application provides a method for implementing S33, including:

[0092] S331: Generate a first tensor according to the spatial embedding tensor and the first position code;

[0093] S332: Generate multiple first sub-tensors according to the first tensor;

[0094] S333: Obtain weight matrices corresponding to multiple first sub-tensors;

[0095] S334: Generate multiple calculation tensors corresponding to the multiple first sub-tensors according to the multiple first sub-tensors and the weight matrices corresponding to the multiple first sub-tensors;

[0096] Exemplarily, multiple first sub-tensors are multiplied by their corresponding weight matrices respectively to obtain multiple calculation tensors; it should be noted that the multiple weight matrices are obtained by pre-training based on training data.

[0097] S335: Encode the multiple calculation tensors to obtain sign language representation tensors of multiple video frames;

[0098] The sign language representation tensor is the first encoded feature.

[0099] In the above implementation process, a method for obtaining the first coding feature is proposed. This method constructs multiple first sub-tensors. Different first sub-tensors correspond to different vector spaces. Based on this, the encoding ability of the algorithm can be improved, and the final generated sign language representation tensor has more accurate information.

[0100] See also Figure 5 In a possible implementation, S355 includes the following sub-steps:

[0101] S3351: Input multiple calculation tensors corresponding to the first tensor into a feedforward neural network to obtain multiple output tensors;

[0102] S3352: normalize multiple output tensors;

[0103] Exemplarily, the normalization algorithm may be softmax;

[0104] S3353: Generate a sign language representation tensor of multiple video frames based on multiple normalized output tensors.

[0105] In the above implementation process, a method for generating a sign language representation tensor is proposed. Multiple calculation tensors corresponding to multiple first tensors are input into a feedforward neural network and normalized. This can further extract features from the video screen, thereby further improving the matching degree of synchronized speech and gestures.

[0106] See also Figure 6 , the embodiment of the present application provides a method for implementing S4, including:

[0107] S41: Generate a word embedding tensor corresponding to the text based on the text;

[0108] S42: Generate a second position code according to the word embedding tensor;

[0109] S43: Input the word embedding tensor and the second position code into the time series model to obtain a second coding feature.

[0110] Exemplarily, the timing model can be a neural network based on an attention mechanism.

[0111] The implementation method of the specific steps in obtaining the second coding feature is the same as the method of obtaining the first coding feature, and will not be repeated here.

[0112] In the above implementation process, the text is processed to generate a word embedding tensor so that the text meets the computing requirements. The embedding tensor contains vocabulary information. The order of the vocabulary information can determine whether the generated synchronized speech is fluent. Therefore, a second position code is generated according to the spatial embedding tensor, and the word embedding tensor and the second position code are input into the time series model to obtain a second coding feature. Based on the above implementation, the generated synchronized speech can be made more fluent.

[0113] In summary, the proposed recognition method using sign language video and text solves the mapping problem in the process of sign language video-standard sign language-synchronized speech. Through the intermediate medium of text, combined with sign language video, the sign language-language conversion problem can be transformed into a language translation problem. In the above method, the sign language video clip is directly processed without the need to capture the most suitable gesture image in the sign language gesture, nor to discard the non-standardized sign language intermediate process gesture, so there is no need to mark the spatial position of the image. It only needs to mark the text meaning of the sign language video stream and the corresponding voice clip, so it can greatly reduce the difficulty of collecting and marking sign language samples.

[0114] Example 2

[0115] See also Figure 7 , the embodiment of the present application provides a sign language recognition device, comprising:

[0116] The video image acquisition module 1 is used to acquire multiple frames of video images of the sign language video;

[0117] An image feature acquisition module 2 is used to acquire image features of multiple video frames according to the multiple video frames;

[0118] A first encoding module 3, used to encode image features of multiple video frames to obtain first encoding features;

[0119] The text acquisition module 4 is used to acquire the text corresponding to the multiple frames of video images;

[0120] A second encoding module 5 is used to encode the text corresponding to the multiple frames of video images to obtain a second encoding feature;

[0121] A third encoding module 6, used for inputting the first encoding feature and the second encoding feature into an encoder to obtain an encoding feature vector;

[0122] The decoding module 7 is used to input the encoded feature vector into a decoder to obtain synchronized speech.

[0123] In the above implementation process, multiple video frames of the sign language video are obtained, image features in the multiple video frames are extracted, and the image features are encoded to obtain a first encoding feature. Different from the prior art, the image features of the video screen are directly obtained instead of just obtaining the gesture semantics based on video screen recognition. The obtained first encoding feature preserves a variety of information in the video screen, which is conducive to generating more accurate speech; the text corresponding to the multiple video frames is obtained, the text corresponding to the multiple video frames is encoded, and the second encoding feature is obtained. The first encoding feature and the second encoding feature are input into the encoder to obtain an encoding feature vector, and the encoding feature vector is input into the decoder to obtain synchronized speech. In the process of generating synchronized speech, image features and text are used, and the text and image features are fused through encoding, so that the mapping relationship between speech and text is clearer, thereby generating more accurate and real-time synchronized speech.

[0124] In a possible implementation manner, the image feature acquisition module 2 is further used to generate a pixel matrix of multiple video frames according to the multiple video frames;

[0125] The pixel matrix of multiple video frames is input into the convolutional neural network to obtain the image features of the multiple video frames.

[0126] In a possible implementation, the first encoding module 3 is also used to generate a spatial embedding tensor based on image features; generate a first position code based on the spatial embedding tensor; and process the spatial embedding tensor and the first position code using a preset algorithm to obtain a first coding feature.

[0127] In a possible implementation, the first encoding module 3 is also used to generate a first tensor based on the spatial embedding tensor and the first position code; generate multiple first sub-tensors based on the first tensor; obtain weight matrices corresponding to the multiple first sub-tensors; generate multiple calculation tensors corresponding to the multiple first sub-tensors based on the multiple first sub-tensors and the weight matrices corresponding to the multiple first sub-tensors; encode the multiple calculation tensors to obtain sign language representation tensors of multiple frames of video images; the sign language representation tensor is the first encoding feature.

[0128] In a possible implementation, the first encoding module 3 is also used to input multiple calculation tensors corresponding to multiple first tensors into a feedforward neural network to obtain multiple output tensors; normalize the multiple output tensors; and generate sign language representation tensors of multiple frames of video images based on the multiple normalized output tensors.

[0129] In a possible implementation, the second encoding module 5 is also used to generate a word embedding tensor corresponding to the text based on the text; generate a second position code based on the word embedding tensor; input the word embedding tensor and the second position code into the time series model to obtain a second encoding feature.

[0130] In one possible implementation, the encoder and the decoder include: a Self-Attention layer.

[0131] This application also provides an electronic device, see Figure 8 , Figure 8 A block diagram of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 81, a communication interface 82, a memory 83, and at least one communication bus 84. The communication bus 84 is used to realize direct connection and communication between these components. The communication interface 82 of the electronic device in the embodiment of the present application is used to communicate signaling or data with other node devices. The processor 81 may be an integrated circuit chip with signal processing capabilities.

[0132] The processor 81 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor 81 can also be any conventional processor, etc.

[0133] The memory 83 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electric erasable programmable read-only memory (EEPROM), etc. The memory 83 stores computer readable instructions. When the computer readable instructions are executed by the processor 81, the electronic device can execute the above-mentioned Figures 1 to 6 The method embodiment involves various steps.

[0134] Optionally, the electronic device may further include a storage controller and an input / output unit.

[0135] The memory 83, storage controller, processor 81, peripheral interface, input and output unit components are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses 84. The processor 81 is used to execute executable modules stored in the memory 83, such as software function modules or computer programs included in the electronic device.

[0136] The input and output unit is used to provide users with the task creation and to create an optional time period or preset execution time for the task to enable interaction between the user and the server. The input and output unit can be, but is not limited to, a mouse and a keyboard.

[0137] Understandably, Figure 8 The structure shown is for illustration only. The electronic device may also include Figure 8 More or fewer components as shown, or with Figure 8 Different configurations shown. Figure 8 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0138] The embodiment of the present application also provides a computer-readable storage medium, on which instructions are stored. When the instructions are run on a computer, the method of the method embodiment is implemented when the computer program is executed by the processor. To avoid repetition, it will not be described here.

[0139] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0140] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0141] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0142] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0143] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0144] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A sign language recognition method, characterized in that: The method comprises: Obtain multiple frames of sign language video; Acquire image features of the multiple video frames according to the multiple video frames; Encoding the image features of the multiple video frames to obtain first encoded features; Obtain text corresponding to the multiple video frames; Encoding the texts corresponding to the multiple frames of video to obtain a second encoding feature; Inputting the first encoding feature and the second encoding feature into an encoder to obtain an encoding feature vector; Inputting the encoded feature vector into a decoder to obtain synchronized speech; The step of encoding the image features of the multiple frames of video to obtain a first encoding feature comprises: generating a spatial embedding tensor according to the image features; generating a first position encoding according to the spatial embedding tensor; Processing the spatial embedding tensor and the first position code using a preset algorithm to obtain the first coding feature; Generating a spatial embedding tensor according to the image feature comprises: Transforming the matrix corresponding to the image features to obtain a spatial embedding tensor corresponding to the tensor space dimension; The step of using a preset algorithm to process the spatial embedding tensor and the first position code to obtain the first coding feature includes: Generate a first tensor according to the spatial embedding tensor and the first position encoding; Generate a plurality of first sub-tensors according to the first tensor; Obtaining weight matrices corresponding to the multiple first sub-tensors; Generate a plurality of computation tensors corresponding to the plurality of first sub-tensors according to the plurality of first sub-tensors and weight matrices corresponding to the plurality of first sub-tensors; Encoding the multiple calculation tensors to obtain sign language representation tensors of the multiple video frames; The sign language representation tensor is the first encoding feature.

2. The sign language recognition method according to claim 1, characterized in that: The step of acquiring image features of the multiple video frames according to the multiple video frames comprises: Generate a pixel matrix of the multiple video frames according to the multiple video frames; The pixel matrix of the multiple frames of video images is input into a convolutional neural network to obtain image features of the multiple frames of video images.

3. The sign language recognition method according to claim 1, characterized in that: The step of encoding the multiple calculation tensors to obtain the sign language representation tensors of the multiple video frames includes: Inputting the plurality of computation tensors into a feedforward neural network to obtain a plurality of output tensors; Normalizing the multiple output tensors; A sign language representation tensor of the multiple frames of video images is generated according to the multiple normalized output tensors.

4. The sign language recognition method according to claim 3, characterized in that: The step of encoding the text corresponding to the multiple frames of video to obtain a second encoding feature includes: Generate a word embedding tensor corresponding to the text according to the text; Generating a second positional encoding according to the word embedding tensor; The word embedding tensor and the second position code are input into a time series model to obtain the second coding feature.

5. The sign language recognition method according to claim 4, characterized in that: The encoder and the decoder include: a Self-Attention layer.

6. A sign language recognition device, characterized in that: The device comprises: A video image acquisition module is used to acquire multiple frames of video images of a sign language video; An image feature acquisition module, used for acquiring image features of the multiple video frames according to the multiple video frames; A first encoding module, used for encoding image features of the multiple video frames to obtain first encoding features; A text acquisition module, used to acquire texts corresponding to the multiple frames of video images; A second encoding module is used to encode the text corresponding to the multiple frames of video images to obtain a second encoding feature; A third encoding module, used for inputting the first encoding feature and the second encoding feature into an encoder to obtain an encoding feature vector; A decoding module, used for inputting the encoded feature vector into a decoder to obtain synchronized speech; The first encoding module is used to generate a spatial embedding tensor according to the image feature; generate a first position code according to the spatial embedding tensor; and process the spatial embedding tensor and the first position code using a preset algorithm to obtain a first encoding feature; Generating a spatial embedding tensor according to the image feature comprises: Transforming the matrix corresponding to the image features to obtain a spatial embedding tensor corresponding to the tensor space dimension; The first encoding module is also used to generate a first tensor based on the spatial embedding tensor and the first position code; generate multiple first sub-tensors based on the first tensor; obtain weight matrices corresponding to the multiple first sub-tensors; generate multiple calculation tensors corresponding to the multiple first sub-tensors based on the multiple first sub-tensors and the weight matrices corresponding to the multiple first sub-tensors; encode the multiple calculation tensors to obtain the sign language representation tensor of the multiple frames of video; the sign language representation tensor is the first encoding feature.

7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the sign language recognition method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the steps of the sign language recognition method according to any one of claims 1 to 5.

Images