Sign language recognition methods, devices, computer equipment and storage media

By using a multimodal fusion method, a probability matrix is ​​generated using sign language video data, key point data, and optical flow data. This solves the problem of information integration in traditional sign language recognition, improves the accuracy and robustness of sign language recognition, and simplifies model training.

CN118609214BActive Publication Date: 2026-11-17SHENZHEN INST OF ADVANCED TECH CHINESE ACAD OF SCI
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410751173.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-12
Publication Date
2026-11-17
Estimated Expiration
2044-06-12

AI Technical Summary

Technical Problem

Traditional sign language recognition methods struggle to accurately segment and identify details in sign language videos under weakly supervised learning environments, and multimodal information is difficult to integrate effectively, resulting in poor recognition performance.

Method used

A multimodal fusion method is adopted to acquire sign language video data, key point data and optical flow data, and use the first branch network, second branch network, third branch network and CTC decoder to generate a probability matrix for sign language analysis, capturing subtle hand movements and body dynamic information.

Benefits of technology

It significantly improves the accuracy and robustness of sign language recognition, reduces reliance on large-scale, finely labeled datasets, simplifies the model training process, and provides a more comprehensive understanding of sign language.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118609214B_ABST
    Figure CN118609214B_ABST
Patent Text Reader

Abstract

This application relates to the field of sign language recognition technology, and discloses a sign language recognition method, device, computer equipment, and storage medium. The method includes: acquiring sign language video data, key point data, and optical flow data, wherein the key point data is generated based on key point generation from the sign language video data, and the optical flow data is extracted from the sign language video data; generating a probability matrix based on the sign language video data, the key point data, the optical flow data, and a trained sign language recognition model; and performing sign language analysis based on the probability matrix. The sign language recognition model includes a first branch network, a second branch network, a third branch network, a head network, and a CTC decoder, which can capture subtle hand movements and dynamic information of various body parts, significantly improving the sign language recognition effect.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of sign language recognition technology, and in particular to a sign language recognition method, apparatus, computer device, and storage medium. Background Technology

[0002] In traditional sign language recognition research, video data is typically labeled only with the entire sequence of sign language words, without labeling each specific word within a segment of the video. This labeling method presents a significant challenge because accurately segmenting and identifying video segments corresponding to specific words from long videos becomes extremely difficult. First, sign language movements and expressions are highly temporal and spatial, and the differences between different gestures can be very subtle, requiring the recognition system to extract highly accurate visual features.

[0003] However, in weakly supervised learning environments, the lack of precise time annotations for each action or word makes it difficult for the trained visual feature extractor to accurately interpret the details in sign language videos. Secondly, the multimodal nature of sign language recognition means that it is necessary to simultaneously process and understand both visual information and corresponding textual information (such as sign language words or sentences) in the video. Under a weakly supervised learning framework, effectively aligning and integrating this information from different modalities is another technical challenge for achieving high-precision sign language recognition. Summary of the Invention

[0004] Based on this, it is necessary to address the technical problem of poor sign language recognition performance in existing technologies by proposing a sign language recognition method, device, computer equipment, and storage medium.

[0005] Firstly, a sign language recognition method is provided, the method comprising:

[0006] Acquire sign language video data, key point data, and optical flow data, wherein the key point data is obtained by generating key points from the sign language video data, and the optical flow data is obtained by extracting from the sign language video data;

[0007] Based on the sign language video data, the key point data, the optical flow data, and the trained sign language recognition model, a probability matrix is ​​generated, and sign language analysis is performed based on the probability matrix. The sign language recognition model includes a first branch network, a second branch network, a third branch network, a head network, and a CTC decoder. The outputs of the first branch network, the second branch network, and the third branch network are respectively connected to the input of the head network, and the output of the head network is connected to the input of the CTC decoder.

[0008] Secondly, a sign language recognition device is provided, the device comprising:

[0009] The acquisition module is used to acquire sign language video data, key point data, and optical flow data, wherein the key point data is obtained by generating key points from the sign language video data, and the optical flow data is obtained by extracting from the sign language video data;

[0010] The analysis module is used to generate a probability matrix based on the sign language video data, the key point data, the optical flow data, and the trained sign language recognition model, and to perform sign language analysis based on the probability matrix. The sign language recognition model includes a first branch network, a second branch network, a third branch network, a head network, and a CTC decoder. The outputs of the first branch network, the second branch network, and the third branch network are respectively connected to the input of the head network, and the output of the head network is connected to the input of the CTC decoder.

[0011] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described sign language recognition method.

[0012] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described sign language recognition method.

[0013] The sign language recognition method proposed in this invention acquires sign language video data, key point data, and optical flow data. The key point data is generated from the sign language video data, and the optical flow data is extracted from the sign language video data. Finally, a probability matrix is ​​generated based on the sign language video data, the key point data, the optical flow data, and a trained sign language recognition model. Sign language analysis is then performed based on this probability matrix. The sign language recognition model includes a first branch network, a second branch network, a third branch network, a head network, and a CTC decoder. The outputs of the first, second, and third branch networks are connected to the input of the head network, and the output of the head network is connected to the input of the CTC decoder. This method can capture subtle hand movements and dynamic information from various parts of the body, significantly improving the sign language recognition effect. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] in:

[0016] Figure 1 This is a diagram illustrating the application environment of a sign language recognition method in one embodiment;

[0017] Figure 2 Here is a flowchart of a sign language recognition method in one embodiment;

[0018] Figure 3 This is a schematic diagram of the branch network structure of a sign language recognition method in one embodiment;

[0019] Figure 4 This is a structural block diagram of a sign language recognition device in one embodiment;

[0020] Figure 5 This is a structural block diagram of a computer device in one embodiment;

[0021] Figure 6 This is a structural block diagram of a computer device in another embodiment. Detailed Implementation

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0023] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] The sign language recognition method provided in this invention can be applied to, for example... Figure 1 In this application environment, client 110 communicates with server 120 via a network. Server 120 can receive and acquire sign language video data, key point data, and optical flow data from client 110. The key point data is generated based on key point generation from the sign language video data, and the optical flow data is extracted from the sign language video data. Finally, server 120 generates a probability matrix based on the sign language video data, the key point data, the optical flow data, and a trained sign language recognition model. Sign language analysis is then performed based on this probability matrix. The sign language recognition model includes a first branch network, a second branch network, a third branch network, a head network, and a CTC decoder. The outputs of the first, second, and third branch networks are connected to the input of the head network, and the output of the head network is connected to the input of the CTC decoder. This model can capture subtle hand movements and dynamic information from various body parts, significantly improving the sign language recognition effect. Client 110 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server 120 can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0026] Please see Figure 2 As shown, Figure 2 A flowchart illustrating a sign language recognition method according to an embodiment of the present invention includes the following steps:

[0027] Step S101: Obtain sign language video data, key point data, and optical flow data, wherein the key point data is obtained by generating key points from the sign language video data, and the optical flow data is obtained by extracting from the sign language video data;

[0028] Keypoint data can be extracted by a network specifically designed to generate keypoints, while optical flow data can be extracted using a deep network architecture.

[0029] Step S102: Based on the sign language video data, the key point data, the optical flow data, and the trained sign language recognition model, generate a probability matrix, and perform sign language analysis based on the probability matrix. The sign language recognition model includes a first branch network, a second branch network, a third branch network, a head network, and a CTC decoder. The outputs of the first branch network, the second branch network, and the third branch network are respectively connected to the input of the head network, and the output of the head network is connected to the input of the CTC decoder.

[0030] It should be noted that each branch network contains four components. Each branch independently processes the input data and generates frame-by-frame features, which are then fed into their respective temporal head networks. The head network comprises separate temporal heads for the first, second, and third network branches, and combines the outputs of all branches to form another joint temporal head. Each temporal head consists of a temporal linear layer, two temporal convolutional layers, and a linear layer, ultimately outputting frame-by-frame word probabilities.

[0031] In this embodiment, the sign language video data, the key point data, and the optical flow data are input into the trained sign language recognition model to generate a frame-by-frame probability matrix. The columns corresponding to the labeled words are extracted from the matrix, and the path with the highest probability product is found to determine the frame-by-frame visual features corresponding to each word.

[0032] As an example, a schematic diagram of the structure of the first branch network, the second branch network, or the third branch network can be referenced. Figure 3 .

[0033] Further, in one embodiment, the step of generating a probability matrix based on the sign language video data, the key point data, the optical flow data, and the trained sign language recognition model includes:

[0034] Step S201: Input the sign language video data into the first S3D model in the first branch network for processing to obtain the first feature output by the first S3D model;

[0035] Step S202: Input the key point data into the second S3D model in the second branch network for processing to obtain the second feature output by the second S3D model;

[0036] Step S203: Input the optical flow data into the third S3D model in the third branch network for processing to obtain the third feature output by the third S3D model;

[0037] Step S203: Perform feature fusion on the first feature, the second feature, and the third feature to obtain the first fused feature;

[0038] For example, the first feature, the second feature, and the third feature are fused by a multimodal fusion module, which is implemented by a fully connected multilayer perceptron consisting of two hidden layers.

[0039] Step S204: The first fused feature is added to the first feature to obtain the fourth feature; the first fused feature is added to the second feature to obtain the fifth feature; the first fused feature is added to the third feature to obtain the sixth feature.

[0040] Step S205: Generate a probability matrix based on the fourth feature, the fifth feature, the sixth feature, and the trained sign language recognition model.

[0041] In the second branch network, the first convolutional layer of the second S3D model needs to be adjusted to adapt to the input format of the keypoint data.

[0042] Further, in one embodiment, the step of generating a probability matrix based on the fourth feature, the fifth feature, the sixth feature, and the trained sign language recognition model includes:

[0043] Step S301: Input the fourth feature into the fourth S3D model in the first branch network for processing to obtain the seventh feature output by the fourth S3D model;

[0044] Step S302: Input the fifth feature into the fifth S3D model in the second branch network for processing to obtain the eighth feature output by the fifth S3D model;

[0045] Step S303: Input the sixth feature into the sixth S3D model in the third branch network for processing to obtain the ninth feature output by the sixth S3D model;

[0046] Step S304: Perform feature fusion on the seventh feature, the eighth feature, and the ninth feature to obtain the fused second feature;

[0047] Step S305: The second fused feature is added to the seventh feature to obtain the tenth feature; the second fused feature is added to the eighth feature to obtain the eleventh feature; the second fused feature is added to the ninth feature to obtain the twelfth feature.

[0048] Step S306: Generate a probability matrix based on the tenth feature, the eleventh feature, the twelfth feature, and the trained sign language recognition model.

[0049] Further, in one embodiment, the step of generating a probability matrix based on the tenth feature, the eleventh feature, the twelfth feature, and the trained sign language recognition model includes:

[0050] Step S401: Input the tenth feature into the seventh S3D model in the first branch network for processing to obtain the thirteenth feature output by the seventh S3D model;

[0051] Step S402: Input the eleventh feature into the eighth S3D model in the second branch network for processing to obtain the fourteenth feature output by the eighth S3D model;

[0052] Step S403: Input the twelfth feature into the ninth S3D model in the third branch network for processing to obtain the fifteenth feature output by the ninth S3D model;

[0053] Step S404: Perform feature fusion on the thirteenth feature, the fourteenth feature, and the fifteenth feature to obtain the third fused feature;

[0054] Step S405: The third fusion feature is added to the thirteenth feature to obtain the sixteenth feature; the third fusion feature is added to the fourteenth feature to obtain the seventeenth feature; and the third fusion feature is added to the fifteenth feature to obtain the eighteenth feature.

[0055] Step S406: Input the sixteenth feature into the tenth S3D model in the first branch network for processing to obtain the nineteenth feature output by the tenth S3D model;

[0056] Step S407: Input the seventeenth feature into the eleventh S3D model in the second branch network for processing to obtain the twentieth feature output by the eleventh S3D model;

[0057] Step S408: Input the eighteenth feature into the twelfth S3D model in the third branch network for processing to obtain the twenty-first feature output by the twelfth S3D model;

[0058] Step S409: Generate a probability matrix based on the nineteenth feature, the twentieth feature, the twenty-first feature, and the trained sign language recognition model.

[0059] Further, in one embodiment, the step of generating a probability matrix based on the nineteenth feature, the twentieth feature, and the twenty-first feature, as well as the trained sign language recognition model, includes:

[0060] Step S501: Perform a deconvolution operation on the nineteenth feature, and add the deconvolutioned nineteenth feature to the thirteenth feature output by the seventh S3D model to obtain the twenty-second feature; perform a deconvolution operation on the twenty-second feature, and add the deconvolutioned twenty-second feature to the seventh feature output by the fourth S3D model to obtain the twenty-third feature; merge the nineteenth feature, the twenty-second feature, and the twenty-third feature to obtain the twenty-fourth feature, and input the twenty-fourth feature into the time header corresponding to the sign language video data in the head network to obtain the first probability matrix output by the head network;

[0061] Step S502: Perform a deconvolution operation on the twentieth feature, and add the deconvolutioned twentieth feature to the fourteenth feature output by the eighth S3D model to obtain the twenty-sixth feature; perform a deconvolution operation on the twenty-sixth feature, and add the deconvolutioned twenty-sixth feature to the eighth feature output by the fifth S3D model to obtain the twenty-seventh feature; merge the channels of the twentieth, twenty-sixth, and twenty-seventh features to obtain the twenty-eighth feature, and input the twenty-eighth feature into the time header corresponding to the key point data in the head network to obtain the second probability matrix output by the head network;

[0062] Step S503: Perform a deconvolution operation on the 21st feature, and add the 21st feature after the deconvolution operation to the 15th feature output by the 9th S3D model to obtain the 30th feature; perform a deconvolution operation on the 30th feature, and add the 30th feature after the deconvolution operation to the 9th feature output by the 6th S3D model to obtain the 31st feature; merge the 21st feature, the 30th feature, and the 31st feature into channels to obtain the 32nd feature, and input the 32nd feature into the time header corresponding to the optical flow data in the head network to obtain the third probability matrix output by the head network;

[0063] Step S504: Generate a probability matrix based on the first probability matrix, the second probability matrix, the third probability matrix, and the trained sign language recognition model.

[0064] Further, in one embodiment, the step of generating a probability matrix based on the first probability matrix, the second probability matrix, the third probability matrix, and the trained sign language recognition model includes:

[0065] Step S601: Input the first probability matrix, the second probability matrix, and the third probability matrix into the joint time header of the head network to obtain the fourth probability matrix output by the head network;

[0066] Step S602: Calculate the average value based on the first probability matrix, the second probability matrix, the third probability matrix, and the fourth probability matrix to obtain the calculation result;

[0067] Step S603: Input the calculation result into the CTC decoder to obtain the probability matrix output by the CTC decoder.

[0068] As an example, during training, the multimodal fusion feature map, obtained by merging the feature maps output from the first, second, and third branches, is input into the Visual-to-Text (V2T) adapter to obtain a set of visual feature values. Then, the labeled sign language vocabulary is input into the text encoder, and the output data of the text encoder is input into the Text-to-Visual (T2V) adapter to obtain a set of text feature values. These two sets of feature values ​​are used to calculate the losses (sentence alignment loss and word alignment loss) to train the network. Simultaneously, the first, second, third, and fourth probability matrices output from the four heads of the head network are used with the labeled sign language vocabulary to calculate the CTC loss and train the network. The visual-text alignment part consists of a text encoder and two feature adapters (visual-to-text adapter and text-to-visual adapter). The text encoder: A pre-trained text encoder is selected for efficient encoding of text data. The input sentence is converted into numbers by a tokenizer and then passed through the text encoder to generate high-dimensional semantic features. Feature Adapters: Visual-to-text and text-to-visual adapters are used to map visual and text features to a shared multimodal semantic space, respectively, thus maintaining the connection between visual and pre-trained text knowledge. Lexical Alignment: Frame-by-frame visual and lexical features are aligned in the latent space using a dynamic temporal warping algorithm. Sentence Alignment: Global visual and text features are obtained through a global pooling layer, and these two features are aligned to achieve sentence-level semantic and contextual information alignment. During the inference phase, a parameter-free CTC decoder can be used, employing a beam search algorithm to derive the final sign language prediction with a beamwidth of 5. Specifically, the input to the decoder is the average of the frame-by-frame word probabilities from the four temporal headers.

[0069] The objectives of this invention include the following aspects: (1) Improving the robustness of feature representation: By simultaneously training a sign language recognition model that includes video, key points, and optical flow, this invention can extract more robust visual feature representations. This multimodal fusion method can capture information from different modalities in sign language and improve the accuracy of sign language recognition. (2) Achieving accurate visual-text alignment: This invention introduces a visual-text alignment method that combines word-level and sentence-level alignment to ensure accurate correspondence between visual features and sign language annotations at the individual word and sentence levels. This alignment method can effectively utilize the structure of sign language videos and text annotations to improve the model's recognition performance. (3) Reducing dependence on large-scale fine-annotated datasets: By utilizing self-supervised learning and contrastive learning techniques, this invention can still achieve good feature representation and recognition performance even in the absence of large-scale fine-annotated datasets. This method not only reduces data requirements but also improves the model's generalization ability. (4) Simplifying the model training process: By introducing a multimodal fusion module and a label pyramid network, this invention achieves efficient multimodal information fusion and feature extraction without the need for an additional pre-training stage. This method not only simplifies the training process of the model, but also improves the training efficiency. (5) Provides more comprehensive sign language understanding: By integrating multimodal information from video, key points and optical flow, this invention can more comprehensively understand the complex dynamic information in sign language, and improve the accuracy and robustness of sign language recognition.

[0070] It should also be noted that, firstly, optical flow provides a dense representation of motion, capturing the inherent spatiotemporal dynamics of sign language. By calculating the displacement of pixels between consecutive frames, optical flow encodes motion information crucial for understanding sign language gestures. This enables the model to capture subtle details of sign language movements and facilitates accurate recognition. Secondly, using optical flow as an intermediate encoder allows the network to focus on capturing motion patterns and gestures, effectively ignoring irrelevant visual information such as static backgrounds or lighting changes. This enhances the network's discriminative ability by emphasizing the most relevant features in sign language recognition. Furthermore, by introducing optical flow data, the complementarity of appearance-based features in the video encoder and motion-based features in the optical flow encoder and keypoint encoder is utilized. This modal fusion allows the model to simultaneously capture both static and dynamic aspects of sign language, resulting in a more comprehensive understanding of sign language gestures. Through this multimodal fusion, this invention can capture subtle dynamic information and hand movements in sign language, significantly improving the accuracy and robustness of sign language recognition. This invention validates its effectiveness on three datasets, which are described below: Phoenix-2014 and Phoenix-2014T are two widely used German Sign Language datasets. The Phoenix-2014 dataset contains 5672 training samples, 540 development samples, and 629 test samples, with a vocabulary of 1295 annotations. On the other hand, Phoenix-2014T contains 7096 training samples, 519 development samples, and 642 test samples, with a vocabulary of 1085 annotations. CSL-Daily is a large-scale Chinese Sign Language dataset. It includes 18401 training videos, 1077 development videos, and 1176 test videos. This dataset has a vocabulary of 2000 annotations. Word Error Rate (WER) is a primary metric for evaluating sign language recognition performance. It quantifies the number of incorrect insertions, substitutions, and deletions required to match the predicted sentence with its corresponding reference sentence. A lower WER indicates higher accuracy. Table 1 shows a comparative analysis of the proposed method and existing state-of-the-art methods. The present invention achieves state-of-the-art performance on three datasets. The word recognition error rates achieved by this method on the development sets of the Phoenix-2014 and Phoenix-2014T datasets are 17.1% and 16.7%, respectively. On the test set, the state-of-the-art word recognition error rates achieved by the present invention are 1.4% and 1.6% lower than the previous best methods, respectively. A comparative analysis of the proposed method and previous state-of-the-art methods was also conducted on the CSL-Daily dataset. Compared with previous best methods, the sign language recognition model of the present invention significantly reduces the word recognition error rate by 1.2% on the development set and by 1.3% on the test set, demonstrating its superior performance.

[0071] Table 1: Experimental results. The indicator is the word recognition error rate. The lower the error rate, the better the method.

[0072]

[0073]

[0074] Please see Figure 4 As shown, in one embodiment, a sign language recognition device is provided, the device comprising:

[0075] The acquisition module 10 is used to acquire sign language video data, key point data, and optical flow data, wherein the key point data is obtained by generating key points from the sign language video data, and the optical flow data is obtained by extracting from the sign language video data;

[0076] Analysis module 20 is used to generate a probability matrix based on the sign language video data, the key point data, the optical flow data, and the trained sign language recognition model, and to perform sign language analysis based on the probability matrix. The sign language recognition model includes a first branch network, a second branch network, a third branch network, a head network, and a CTC decoder. The outputs of the first branch network, the second branch network, and the third branch network are respectively connected to the input of the head network, and the output of the head network is connected to the input of the CTC decoder.

[0077] In one embodiment, the analysis module 20 is further configured to input the sign language video data into the first S3D model in the first branch network for processing, and obtain the first feature output by the first S3D model;

[0078] The key point data is input into the second S3D model in the second branch network for processing to obtain the second feature output by the second S3D model.

[0079] The optical flow data is input into the third S3D model in the third branch network for processing to obtain the third feature output by the third S3D model;

[0080] The first feature, the second feature, and the third feature are fused to obtain the first fused feature;

[0081] The first fusion feature is added to the first feature to obtain the fourth feature; the first fusion feature is added to the second feature to obtain the fifth feature; and the first fusion feature is added to the third feature to obtain the sixth feature.

[0082] A probability matrix is ​​generated based on the fourth feature, the fifth feature, the sixth feature, and the trained sign language recognition model.

[0083] In one embodiment, the analysis module 20 is further configured to input the fourth feature into the fourth S3D model in the first branch network for processing, to obtain the seventh feature output by the fourth S3D model;

[0084] The fifth feature is input into the fifth S3D model in the second branch network for processing to obtain the eighth feature output by the fifth S3D model;

[0085] The sixth feature is input into the sixth S3D model in the third branch network for processing to obtain the ninth feature output by the sixth S3D model;

[0086] The seventh feature, the eighth feature, and the ninth feature are fused to obtain the fused second feature;

[0087] The second fusion feature is added to the seventh feature to obtain the tenth feature; the second fusion feature is added to the eighth feature to obtain the eleventh feature; and the second fusion feature is added to the ninth feature to obtain the twelfth feature.

[0088] A probability matrix is ​​generated based on the tenth feature, the eleventh feature, the twelfth feature, and the trained sign language recognition model.

[0089] In one embodiment, the analysis module 20 is further configured to input the tenth feature into the seventh S3D model in the first branch network for processing, to obtain the thirteenth feature output by the seventh S3D model;

[0090] The eleventh feature is input into the eighth S3D model in the second branch network for processing to obtain the fourteenth feature output by the eighth S3D model.

[0091] The twelfth feature is input into the ninth S3D model in the third branch network for processing, to obtain the fifteenth feature output by the ninth S3D model.

[0092] The thirteenth, fourteenth, and fifteenth features are fused to obtain the third fused feature;

[0093] The third fusion feature is added to the thirteenth feature to obtain the sixteenth feature; the third fusion feature is added to the fourteenth feature to obtain the seventeenth feature; and the third fusion feature is added to the fifteenth feature to obtain the eighteenth feature.

[0094] The sixteenth feature is input into the tenth S3D model in the first branch network for processing to obtain the nineteenth feature output by the tenth S3D model.

[0095] The seventeenth feature is input into the eleventh S3D model in the second branch network for processing, to obtain the twentieth feature output by the eleventh S3D model.

[0096] The eighteenth feature is input into the twelfth S3D model in the third branch network for processing, to obtain the twenty-first feature output by the twelfth S3D model;

[0097] Based on the nineteenth feature, the twentieth feature, and the twenty-first feature, as well as the trained sign language recognition model, a probability matrix is ​​generated.

[0098] In one embodiment, the analysis module 20 is further configured to perform a deconvolution operation on the nineteenth feature, add the nineteenth feature after the deconvolution operation to the thirteenth feature output by the seventh S3D model to obtain the twenty-second feature; perform a deconvolution operation on the twenty-second feature, add the twenty-second feature after the deconvolution operation to the seventh feature output by the fourth S3D model to obtain the twenty-third feature; merge the nineteenth feature, the twenty-second feature, and the twenty-third feature to obtain the twenty-fourth feature, and input the twenty-fourth feature into the time header corresponding to the sign language video data in the head network to obtain the first probability matrix output by the head network;

[0099] The twentieth feature is deconvolved, and the resulting twentieth feature is added to the fourteenth feature output by the eighth S3D model to obtain the twenty-sixth feature. The twenty-sixth feature is then deconvolved, and the resulting twenty-sixth feature is added to the eighth feature output by the fifth S3D model to obtain the twenty-seventh feature. The twentieth, twenty-sixth, and twenty-seventh features are then channel-merged to obtain the twenty-eighth feature. The twenty-eighth feature is then input into the time header corresponding to the keypoint data in the head network to obtain the second probability matrix output by the head network.

[0100] The 21st feature is deconvolved, and the resulting 21st feature is added to the 15th feature output by the 9th S3D model to obtain the 30th feature. The 30th feature is then deconvolved, and the resulting 30th feature is added to the 9th feature output by the 6th S3D model to obtain the 31st feature. The 21st feature, the 30th feature, and the 31st feature are then channel-merged to obtain the 32nd feature. The 32nd feature is then input into the time header corresponding to the optical flow data in the head network to obtain the third probability matrix output by the head network.

[0101] A probability matrix is ​​generated based on the first probability matrix, the second probability matrix, the third probability matrix, and the trained sign language recognition model.

[0102] In one embodiment, the analysis module 20 is further configured to input the first probability matrix, the second probability matrix, and the third probability matrix into the joint time header of the head network to obtain the fourth probability matrix output by the head network;

[0103] The average value is calculated based on the first probability matrix, the second probability matrix, the third probability matrix, and the fourth probability matrix to obtain the calculation result;

[0104] The calculation result is input into the CTC decoder to obtain the probability matrix output by the CTC decoder.

[0105] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a sign language recognition method server-side.

[0106] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of a sign language recognition method on the client side.

[0107] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, performs the following steps:

[0108] Acquire sign language video data, key point data, and optical flow data, wherein the key point data is obtained by generating key points from the sign language video data, and the optical flow data is obtained by extracting from the sign language video data;

[0109] Based on the sign language video data, the key point data, the optical flow data, and the trained sign language recognition model, a probability matrix is ​​generated, and sign language analysis is performed based on the probability matrix. The sign language recognition model includes a first branch network, a second branch network, a third branch network, a head network, and a CTC decoder. The outputs of the first branch network, the second branch network, and the third branch network are respectively connected to the input of the head network, and the output of the head network is connected to the input of the CTC decoder.

[0110] In one embodiment, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, performs the following steps:

[0111] Acquire sign language video data, key point data, and optical flow data, wherein the key point data is obtained by generating key points from the sign language video data, and the optical flow data is obtained by extracting from the sign language video data;

[0112] Based on the sign language video data, the key point data, the optical flow data, and the trained sign language recognition model, a probability matrix is ​​generated, and sign language analysis is performed based on the probability matrix. The sign language recognition model includes a first branch network, a second branch network, a third branch network, a head network, and a CTC decoder. The outputs of the first branch network, the second branch network, and the third branch network are respectively connected to the input of the head network, and the output of the head network is connected to the input of the CTC decoder.

[0113] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0114] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAM bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0115] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0116] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A sign language recognition method, characterized in that, The sign language recognition method includes: Acquire sign language video data, key point data, and optical flow data, wherein the key point data is obtained by generating key points from the sign language video data, and the optical flow data is obtained by extracting from the sign language video data; Based on the sign language video data, the key point data, the optical flow data, and the trained sign language recognition model, a probability matrix is ​​generated, and sign language analysis is performed based on the probability matrix. The sign language recognition model includes a first branch network, a second branch network, a third branch network, a head network, and a CTC decoder. The outputs of the first branch network, the second branch network, and the third branch network are respectively connected to the input of the head network, and the output of the head network is connected to the input of the CTC decoder. The step of generating a probability matrix based on the sign language video data, the key point data, the optical flow data, and the trained sign language recognition model includes: The sign language video data is input into the first S3D model in the first branch network for processing to obtain the first feature output by the first S3D model. The key point data is input into the second S3D model in the second branch network for processing to obtain the second feature output by the second S3D model. The optical flow data is input into the third S3D model in the third branch network for processing to obtain the third feature output by the third S3D model; The first feature, the second feature, and the third feature are fused to obtain the first fused feature; The first fusion feature is added to the first feature to obtain the fourth feature; the first fusion feature is added to the second feature to obtain the fifth feature; and the first fusion feature is added to the third feature to obtain the sixth feature. Based on the fourth feature, the fifth feature, and the sixth feature, as well as the trained sign language recognition model, a probability matrix is ​​generated; The step of generating a probability matrix based on the fourth feature, the fifth feature, the sixth feature, and the trained sign language recognition model includes: The fourth feature is input into the fourth S3D model in the first branch network for processing to obtain the seventh feature output by the fourth S3D model; The fifth feature is input into the fifth S3D model in the second branch network for processing to obtain the eighth feature output by the fifth S3D model; The sixth feature is input into the sixth S3D model in the third branch network for processing to obtain the ninth feature output by the sixth S3D model; The seventh feature, the eighth feature, and the ninth feature are fused to obtain the fused second feature; The second fusion feature is added to the seventh feature to obtain the tenth feature; the second fusion feature is added to the eighth feature to obtain the eleventh feature; and the second fusion feature is added to the ninth feature to obtain the twelfth feature. A probability matrix is ​​generated based on the tenth feature, the eleventh feature, the twelfth feature, and the trained sign language recognition model.

2. The sign language recognition method according to claim 1, characterized in that, The step of generating a probability matrix based on the tenth feature, the eleventh feature, the twelfth feature, and the trained sign language recognition model includes: The tenth feature is input into the seventh S3D model in the first branch network for processing to obtain the thirteenth feature output by the seventh S3D model. The eleventh feature is input into the eighth S3D model in the second branch network for processing to obtain the fourteenth feature output by the eighth S3D model. The twelfth feature is input into the ninth S3D model in the third branch network for processing, to obtain the fifteenth feature output by the ninth S3D model. The thirteenth, fourteenth, and fifteenth features are fused to obtain the third fused feature; The third fusion feature is added to the thirteenth feature to obtain the sixteenth feature; the third fusion feature is added to the fourteenth feature to obtain the seventeenth feature; and the third fusion feature is added to the fifteenth feature to obtain the eighteenth feature. The sixteenth feature is input into the tenth S3D model in the first branch network for processing to obtain the nineteenth feature output by the tenth S3D model. The seventeenth feature is input into the eleventh S3D model in the second branch network for processing, to obtain the twentieth feature output by the eleventh S3D model. The eighteenth feature is input into the twelfth S3D model in the third branch network for processing, to obtain the twenty-first feature output by the twelfth S3D model; Based on the nineteenth feature, the twentieth feature, and the twenty-first feature, as well as the trained sign language recognition model, a probability matrix is ​​generated.

3. The sign language recognition method according to claim 2, characterized in that, The step of generating a probability matrix based on the nineteenth feature, the twentieth feature, the twenty-first feature, and the trained sign language recognition model includes: The nineteenth feature is deconvolved, and the deconvolved nineteenth feature is added to the thirteenth feature output by the seventh S3D model to obtain the twenty-second feature; the twenty-second feature is deconvolved, and the deconvolved twenty-second feature is added to the seventh feature output by the fourth S3D model to obtain the twenty-third feature; the nineteenth feature, the twenty-second feature, and the twenty-third feature are channel-merged to obtain the twenty-fourth feature, and the twenty-fourth feature is input into the time header corresponding to the sign language video data in the head network to obtain the first probability matrix output by the head network; The twentieth feature is deconvolved, and the resulting twentieth feature is added to the fourteenth feature output by the eighth S3D model to obtain the twenty-sixth feature. The twenty-sixth feature is then deconvolved, and the resulting twenty-sixth feature is added to the eighth feature output by the fifth S3D model to obtain the twenty-seventh feature. The twentieth, twenty-sixth, and twenty-seventh features are then channel-merged to obtain the twenty-eighth feature. The twenty-eighth feature is then input into the time header corresponding to the keypoint data in the head network to obtain the second probability matrix output by the head network. The 21st feature is deconvolved, and the resulting 21st feature is added to the 15th feature output by the 9th S3D model to obtain the 30th feature. The 30th feature is then deconvolved, and the resulting 30th feature is added to the 9th feature output by the 6th S3D model to obtain the 31st feature. The 21st feature, the 30th feature, and the 31st feature are then channel-merged to obtain the 32nd feature. The 32nd feature is then input into the time header corresponding to the optical flow data in the head network to obtain the third probability matrix output by the head network. A probability matrix is ​​generated based on the first probability matrix, the second probability matrix, the third probability matrix, and the trained sign language recognition model.

4. The sign language recognition method according to claim 3, characterized in that, The step of generating a probability matrix based on the first probability matrix, the second probability matrix, the third probability matrix, and the trained sign language recognition model includes: The first probability matrix, the second probability matrix, and the third probability matrix are input into the joint time header of the head network to obtain the fourth probability matrix output by the head network. The average value is calculated based on the first probability matrix, the second probability matrix, the third probability matrix, and the fourth probability matrix to obtain the calculation result; The calculation result is input into the CTC decoder to obtain the probability matrix output by the CTC decoder.

5. A sign language recognition device, characterized in that, For performing the sign language recognition method according to any one of claims 1 to 4, the sign language recognition device comprises: The acquisition module is used to acquire sign language video data, key point data, and optical flow data. The key point data is generated based on key points generated from the sign language video data, and the optical flow data is extracted from the sign language video data.

6. The sign language recognition device according to claim 5, characterized in that, The sign language recognition device also includes: The analysis module is used to generate a probability matrix based on the sign language video data, the key point data, the optical flow data, and the trained sign language recognition model, and to perform sign language analysis based on the probability matrix. The sign language recognition model includes a first branch network, a second branch network, a third branch network, a head network, and a CTC decoder. The outputs of the first branch network, the second branch network, and the third branch network are respectively connected to the input of the head network, and the output of the head network is connected to the input of the CTC decoder.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the sign language recognition method as described in any one of claims 1 to 4.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the sign language recognition method as described in any one of claims 1 to 4.

Citation Information

Patent Citations

  • Sign language recognition method and system

    CN111340005A

  • Continuous sign language statement recognition method based on modal matching

    CN113609922A