A multi-modal real-time sign language recognition method and system in a medical scenario
By collecting sign language images in medical settings and annotating hand skeletal joints and extracting spatiotemporal features, combined with a language reasoning module, medical semantic reconstruction of gestures was achieved. This solves the problem of insufficient data resources in existing technologies and improves the accuracy and robustness of sign language recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-30
- Publication Date
- 2026-07-03
AI Technical Summary
Existing sign language recognition methods lack high-quality data resources covering medical terminology in medical scenarios. The sample size is limited and the categories are imbalanced, making it difficult to achieve stable and robust real-time recognition.
By acquiring a series of consecutive sign language image sequences, marking the joints of the hand bones, and combining the spatiotemporal feature extraction module and the language reasoning module, the medical semantics of the gestures are reconstructed, and standardized and readable medical terms are output.
It improves the accuracy and robustness of sign language recognition in medical scenarios, can output standardized and readable medical terminology, meets the communication needs in the medical process, and has stronger reproducibility and cross-scenario adaptability.
Smart Images
Figure CN122336848A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of sign language recognition technology, and in particular to a multimodal real-time sign language recognition method and system in medical settings. Background Technology
[0002] In recent years, with the advancement of smart healthcare construction and the informatization upgrade of hospital offline service processes, sign language recognition technology has gradually attracted attention and is being attempted to be introduced into medical service scenarios. It is hoped that it can provide stable communication support for hearing-impaired patients in registration, consultation and other processes, alleviate the problems of limited information access, low efficiency of doctor-patient communication and high risk of misunderstanding in the process of seeking medical treatment, and improve the accessibility and fairness of medical treatment for patients.
[0003] Existing sign language recognition methods typically focus on end-to-end recognition and classification of everyday vocabulary. While these methods have achieved some recognition results, they still have significant shortcomings in medical scenarios. On the one hand, existing publicly available sign language datasets mainly consist of everyday expressions and lack high-quality data resources covering medical terminology. Furthermore, medical data collection and annotation are limited by privacy and scenario constraints, resulting in limited sample size, severe class imbalance, and difficulty in forming stable supervisory signals. On the other hand, existing methods generally treat the task as visual action classification, lacking the ability to model and utilize prior medical semantics. This makes it difficult to achieve robust recognition and standardized output in complex environments. Therefore, there is an urgent need for a real-time sign language recognition method for medical scenarios to improve the recognition accuracy and system robustness of professional terminology. Summary of the Invention
[0004] In view of the above-mentioned defects or deficiencies in the existing technology, it is desirable to provide a multimodal real-time sign language recognition method and system in medical scenarios. By integrating visual spatiotemporal features with medical semantic reasoning at the language level, it can achieve high accuracy and low latency recognition of medical terminology sign language, realize the effective connection from low-level visual perception to high-level medical semantic understanding, and provide reliable technical support for efficient communication for hearing-impaired patients in real medical scenarios.
[0005] This invention provides a multimodal real-time sign language recognition method for medical scenarios, comprising the following steps:
[0006] 1) Acquire a continuous sequence of multiple frames of sign language images;
[0007] 2) Annotate the hand skeletal joints of the sign language image sequence;
[0008] 3) The annotated sign language image sequence is identified using a spatiotemporal feature extraction module to obtain a gesture recognition sequence;
[0009] 4) Input the gesture recognition sequence into the language reasoning module for medical semantic reconstruction, and output structured and highly readable Chinese medical terms or sentences.
[0010] Furthermore, step 2) includes the following steps:
[0011] 21) Use HandDetector to annotate the hand bone joints of each frame of the sign language image sequence, including but not limited to wrist joints, finger root joints, finger joints, and fingertip joints;
[0012] 22) Connect the joints to construct a hand skeletal structure diagram to represent hand posture and its spatial topological relationship.
[0013] Furthermore, step 3) includes the following steps:
[0014] 31) Represent the gestures in each frame of the sign language image sequence as a vector or matrix containing the coordinates of the joints;
[0015] 32) Analyze the relative position, angle and length changes of the connecting lines in the hand skeletal structure diagram to extract discriminative gesture spatial features.
[0016] Furthermore, in step 3), the spatiotemporal feature extraction module includes a convolutional neural network and a bidirectional long short-term memory network; for the input single-frame sign language image, convolution operation is used to extract local spatial features, and the convolution process is defined as follows: ; Where f is the input sign language image, g is the convolution kernel, and (i,j) are the position coordinates in the output feature map.
[0017] Furthermore, the spatiotemporal feature extraction module introduces a max pooling operation after the convolutional layer, where max pooling is defined as: ;
[0018] Where s is the pooling step size, and x and y are the feature maps before and after pooling, respectively.
[0019] Furthermore, the spatial dimensions of the feature maps after convolution and pooling in the spatiotemporal feature extraction module are: ; ;
[0020] Among them, H out W out The size of the output feature map;
[0021] Indicates rounding down;
[0022] H in W in Input feature map size;
[0023] K is the kernel size, P is the padding size, and S is the stride.
[0024] Furthermore, the input gate i of the bidirectional long short-term memory network t Forgotten Gate t and output gate o t They are defined as follows: ; ; ;
[0025] Where σ is the activation function;
[0026] W i The weight parameters for the input gate; W f W represents the weight parameters for the forget gate. o These are the weight parameters for the output gate;
[0027] h t As the current hidden state, x t This is the current input;
[0028] b i For the input gate deviation; b f For the deviation of the forget gate; b o This represents the deviation of the output gate;
[0029] The cell state update process is as follows: ; ;
[0030] Among them, C t This represents the current cell state; ⊙ indicates element-wise multiplication;
[0031] Candidate cell state; W C b represents the weighting parameter of the memory cell. C This refers to the deviation of the memory unit;
[0032] The output in the hidden state is: ;
[0033] Forward LSTM and backward LSTM model the forward and backward directions of the time series respectively, and their dependency relationship is as follows: ; ;
[0034] in, It is in a forward-hidden state; It is in a backward hidden state;
[0035] The final output is obtained by concatenating or summing the two-way hidden states: or .
[0036] Furthermore, in step 4), the language reasoning module, based on a large language model and combined with prior knowledge in the medical field, performs semantic-level reasoning and reconstruction on the gesture recognition sequence, completing medical terminology-level sequence completion, resolving ambiguities caused by similar or identical gestures, context-based error correction, medical semantic consistency verification, and semantic aggregation of multi-frame recognition results.
[0037] Furthermore, this invention also provides a sign language recognition system employing the above-mentioned multimodal real-time sign language recognition method in medical scenarios, comprising:
[0038] The data acquisition module is used to acquire a sequence of sign language images across multiple consecutive frames;
[0039] The hand joint detection module is used to mark the position of each hand bone joint in a single frame of sign language image, connect the hand bone joints according to a preset topology, construct a hand bone structure map, and represent the spatial shape and posture changes of the gesture.
[0040] The spatiotemporal feature extraction module is used to recognize the annotated sign language image sequence to obtain the gesture recognition sequence;
[0041] The language reasoning module is used to perform semantic reasoning and reconstruction on the gesture recognition sequence output by the spatiotemporal feature extraction module.
[0042] The output and interaction module is used to display the output results of the language reasoning module in text form on the terminal screen, or output them in voice broadcast form, or output them in the form of medical information system interface data, or as input to the electronic medical record system or consultation system.
[0043] Furthermore, the spatiotemporal feature extraction module includes:
[0044] The spatial feature submodule is used to extract the spatial features of the hand skeletal structure in a single frame of sign language image;
[0045] The time series submodule is used to perform time-dimensional reasoning on the gesture spatial features output by the spatial feature submodule across multiple frames.
[0046] Compared with the prior art, the beneficial effects of the present invention are:
[0047] (1) This invention constructs a real-time sign language recognition mechanism for medical professional terms, which expands the recognition target from general everyday words to strong semantically constrained term categories such as drug names, examination items, and disease names, reducing the risk of recognition loss and category confusion. It can output standardized and readable medical terminology results, meet the communication needs of accurate, real-time and usable communication in offline consultation, medication, examination guidance and other links, and provide practical technical support for barrier-free medical interaction.
[0048] (2) Based on HandDetector, this invention marks the joint points of the hand bones, realizes automatic data interception and normalization processing, and forms training data resources covering medical scene changes with multiple lighting, multiple perspectives and multiple poses. Thus, stable supervision signals can still be obtained under the conditions of scarce samples and class imbalance. It makes up for the deficiency that existing public datasets cannot cover medical terminology expressions, reduces the dependence on external data resources, and makes the model more reproducible, scalable and adaptable across scenes, laying the foundation for the engineering deployment of medical sign language recognition.
[0049] (3) This invention enhances the ability to express complex dynamics such as transitional movements, short pauses and superimposed gestures by performing spatiotemporal joint recognition of continuous gesture sequences; then, semantic reasoning is used to perform semantic completion, ambiguity resolution and error correction on the preliminary recognition results, thereby reducing misjudgments caused by similar movements, occlusion interference and individual differences. While ensuring real-time performance, it improves the semantic accuracy, output standardization and system robustness of the recognition results, significantly enhancing the usability and practical value of sign language recognition in medical scenarios.
[0050] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of the present invention, nor is it intended to restrict the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description
[0051] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0052] Figure 1 A flowchart of a sign language recognition method;
[0053] Figure 2 This is a framework diagram for sign language recognition methods;
[0054] Figure 3 A schematic diagram illustrating the skeletal joints of the hand in sign language images;
[0055] Figure 4 A visualization diagram of AUC-ROC;
[0056] Figure 5This is a schematic diagram of a one-to-many t-SNE comparison. Detailed Implementation
[0057] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.
[0058] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0059] Please refer to Figures 1-5 The present invention provides a multimodal real-time sign language recognition method in a medical setting, comprising the following steps:
[0060] 1) Acquire a continuous sequence of multiple frames of sign language images;
[0061] 2) Annotate the hand skeletal joints in the sign language image sequence;
[0062] 21) Use HandDetector to annotate the joint points of the hand bones in each frame of the sign language image sequence, including but not limited to wrist joint points, finger root joint points, finger joint joint points, and fingertip joint points;
[0063] 22) Connect the joints to construct a hand skeleton structure map to represent the hand posture and its spatial topological relationship; this not only preserves the geometric information of the gesture, but also effectively reduces the impact of background interference on the recognition results, providing a stable input for subsequent feature extraction;
[0064] 3) The annotated sign language image sequence is identified using the Spatiotemporal Feature Extraction (STFE) module to obtain the gesture recognition sequence;
[0065] 31) Represent the gestures in each frame of the sign language image sequence as a vector or matrix containing the coordinates of the joints; compared with the original image, this representation is more compact, which helps to reduce the amount of computation and improve real-time performance;
[0066] 32) Analyze the relative positions, angles, and length changes between the connecting lines in the hand skeletal structure diagram, extract discriminative gesture spatial features, and provide basic feature inputs for sequence recognition;
[0067] Among them, the spatiotemporal feature extraction module is responsible for the joint recognition of spatial structural features and temporal dynamic features in sign language images, including convolutional neural network (CNN) and bidirectional long short-term memory network (BiLSTM). In order to characterize the dynamic changes of sign language in the time dimension, BiLSTM is introduced to recognize the feature sequences extracted by CNN.
[0068] For a single frame of sign language image as input, convolution operation is used to extract local spatial features. The convolution process is defined as follows: ; Where f is the input sign language image, g is the convolution kernel, and (i,j) is the position coordinate in the output feature map;
[0069] To enhance the translation invariance of features and reduce computational complexity, the spatiotemporal feature extraction module introduces a max-pooling operation after the convolutional layer. Max pooling is defined as: ;
[0070] Where s is the pooling stride, and x and y are the feature maps before and after pooling, respectively;
[0071] The spatial dimensions of the feature maps after convolution and pooling in the spatiotemporal feature extraction module are: ; ;
[0072] Among them, H out W out The size of the output feature map;
[0073] Indicates rounding down;
[0074] H in W in Input feature map size;
[0075] K is the kernel size, P is the padding size, and S is the stride;
[0076] Input gate i of a bidirectional long short-term memory network t Forgotten Gate t and output gate o t They are defined as follows: ; ; ;
[0077] Where σ is the activation function;
[0078] W i The weight parameters for the input gate; Wf W represents the weight parameters for the forget gate. o These are the weight parameters for the output gate;
[0079] h t As the current hidden state, x t This is the current input;
[0080] b i For the input gate deviation; b f For the deviation of the forget gate; b o This represents the deviation of the output gate;
[0081] The cell state update process is as follows: ; ;
[0082] Among them, C t This represents the current cell state; ⊙ indicates element-wise multiplication;
[0083] Candidate cell state; W C b represents the weighting parameter of the memory cell. C This refers to the deviation of the memory unit;
[0084] The output in the hidden state is:
[0085] ;
[0086] Forward LSTM and backward LSTM model the forward and backward directions of the time series respectively, and their dependency relationship is as follows: ; ;
[0087] in, It is in a forward-hidden state; It is in a backward hidden state;
[0088] The final output is obtained by concatenating or summing the two-way hidden states: or ;
[0089] This structure can effectively capture the evolution of complex gestures over time, enhancing the ability to express gesture switching, transitions, and combinations.
[0090] 4) Input the gesture recognition sequence into the Language Reasoning Module (LRM) for medical semantic reconstruction, and output structured and highly readable Chinese medical terms or sentences;
[0091] The language reasoning module is based on a large language model and combines prior knowledge in the medical field to perform semantic-level reasoning and reconstruction on gesture recognition sequences. It completes medical terminology-level sequence completion, dissolves ambiguities caused by similar or identical gestures, corrects errors based on context, verifies medical semantic consistency, and aggregates semantics of multi-frame recognition results.
[0092] In this embodiment, to comprehensively evaluate the discriminative ability and generalization performance of the proposed algorithm in practical applications, the Area Under the Curve (AUC) is used as the main evaluation metric to compare and analyze the model's performance on the training and test sets. AUC can comprehensively reflect the model's ability to distinguish between positive and negative samples at different discrimination thresholds and is a commonly used metric for measuring the stability and robustness of classification models.
[0093] like Figure 4 As shown in the experimental results, the algorithm achieves an AUC of 0.92 on the training set and 0.89 on the test set. The small performance difference between the training and test sets indicates that the model does not exhibit significant overfitting while learning the features of the training data, demonstrating good generalization ability. The high AUC on the test set further validates the model's stable discrimination performance on unknown samples.
[0094] To provide an intuitive analysis of the model's learning performance at the feature space level, t-SNE (t-distributed Stochastic Neighbor Embedding) is used to reduce the dimensionality of the high-dimensional features extracted by the model and visualize them. t-SNE effectively preserves local structural information, thereby revealing the distribution of samples of different categories in the feature space.
[0095] like Figure 5 As shown in the t-SNE visualization results, samples of the same sign language category exhibit a relatively tight clustering distribution in the low-dimensional space, while clear intervals and boundaries are observed between different sign language categories. Especially between sign language categories with similar pronunciations or those that are easily confused, the model is still able to learn discriminative feature representations, enabling good separation of each sign language category in the feature space. These results demonstrate that the proposed algorithm possesses strong discriminative ability in the feature recognition stage, effectively capturing fine-grained differences between different sign language categories, and providing a reliable feature foundation for subsequent classification or recognition tasks.
[0096] Based on the combined quantitative AUC evaluation results and t-SNE feature visualization analysis, we can conclude that the proposed algorithm not only demonstrates good classification performance in terms of numerical metrics, but also possesses strong class discrimination ability at the feature space level. The model exhibits stable performance on both the training and test sets, indicating its good generalization ability and practical application potential, and its ability to meet the accuracy and reliability requirements of relevant tasks.
[0097] This application presents a medical multimodal real-time sign language recognition algorithm based on LRM. Through a multi-level processing flow of "skeletal joint annotation - spatiotemporal feature extraction - language semantic reasoning", it achieves an effective connection from low-level visual perception to high-level medical semantic understanding, providing reliable technical support for efficient communication for hearing-impaired patients in real medical scenarios.
[0098] Furthermore, embodiments of the present invention also provide a sign language recognition system employing the above-described multimodal real-time sign language recognition method in medical scenarios, comprising:
[0099] The data acquisition module is used to acquire a continuous multi-frame sequence of sign language images. It can be acquired through an RGB camera, an RGB-D depth camera, a binocular or multi-view vision acquisition device, a mobile terminal, a tablet terminal, or a fixed medical terminal. The acquired data is preprocessed by frame extraction, resolution adjustment, brightness normalization, etc., and then input to the subsequent modules.
[0100] The hand joint detection module is used to mark the position of each hand bone joint in a single frame of sign language image, connect the hand bone joints according to a preset topology, construct a hand bone structure map, and represent the spatial shape and posture changes of the gesture.
[0101] The spatiotemporal feature extraction module is used to recognize the annotated sign language image sequence to obtain a gesture recognition sequence; it includes a spatial feature submodule and a temporal sequence submodule. The spatial feature submodule is used to extract the gesture spatial features of the hand skeletal structure in a single frame of sign language image; the temporal sequence submodule is used to perform temporal dimension inference on the gesture spatial features output by the spatial feature submodule in multiple frames.
[0102] The language reasoning module is used to perform semantic reasoning and reconstruction on the gesture recognition sequence output by the spatiotemporal feature extraction module; through language reasoning processing, it outputs Chinese medical terms or medical expressions that conform to medical semantic norms.
[0103] The output and interaction module is used to display the output results of the language reasoning module in text form on the terminal screen, or output them in voice broadcast form, or output them in the form of medical information system interface data, or as input to the electronic medical record system or consultation system.
[0104] In the description of this specification, the terms "connection," "installation," and "fixing," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.
[0105] In the description of this specification, the terms "one embodiment," "some embodiments," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0106] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A multimodal real-time sign language recognition method in a medical setting, characterized in that, Includes the following steps: 1) Acquire a continuous sequence of multiple frames of sign language images; 2) Annotate the hand skeletal joints of the sign language image sequence; 3) The annotated sign language image sequence is identified using a spatiotemporal feature extraction module to obtain a gesture recognition sequence; 4) Input the gesture recognition sequence into the language reasoning module for medical semantic reconstruction, and output structured and highly readable Chinese medical terms or sentences.
2. The multimodal real-time sign language recognition method in a medical setting according to claim 1, characterized in that, Step 2) includes the following steps: 21) Use HandDetector to annotate the hand bone joints of each frame of the sign language image sequence, including but not limited to wrist joints, finger root joints, finger joints, and fingertip joints; 22) Connect the joints to construct a hand skeletal structure diagram to represent hand posture and its spatial topological relationship.
3. The multimodal real-time sign language recognition method in a medical setting according to claim 1, characterized in that, Step 3) includes the following steps: 31) Represent the gestures in each frame of the sign language image sequence as a vector or matrix containing the coordinates of the joints; 32) Analyze the relative position, angle and length changes of the connecting lines in the hand skeletal structure diagram to extract discriminative gesture spatial features.
4. The multimodal real-time sign language recognition method in a medical setting according to claim 3, characterized in that, In step 3), the spatiotemporal feature extraction module includes a convolutional neural network and a bidirectional long short-term memory network; for the input single-frame sign language image, convolution operation is used to extract local spatial features, and the convolution process is defined as follows: ； Where f is the input sign language image, g is the convolution kernel, and (i,j) are the position coordinates in the output feature map.
5. The multimodal real-time sign language recognition method in a medical setting according to claim 4, characterized in that, The spatiotemporal feature extraction module introduces a max pooling operation after the convolutional layer. The max pooling is defined as follows: ； Where s is the pooling step size, and x and y are the feature maps before and after pooling, respectively.
6. The multimodal real-time sign language recognition method in a medical setting according to claim 5, characterized in that, The spatial dimensions of the feature maps after convolution and pooling in the spatiotemporal feature extraction module are: ； ； Among them, H out W out The size of the output feature map; Indicates rounding down; H in W in Input feature map size; K is the kernel size, P is the padding size, and S is the stride.
7. The multimodal real-time sign language recognition method in a medical setting according to claim 4, characterized in that, The input gate i of the bidirectional long short-term memory network t Forgotten Gate t and output gate o t They are defined as follows: ； ； ； Where σ is the activation function; W i The weight parameters for the input gate; W f W represents the weight parameters for the forget gate. o These are the weight parameters for the output gate; h t As the current hidden state, x t This is the current input; b i For the input gate deviation; b f For the deviation of the forget gate; b o This represents the deviation of the output gate; The cell state update process is as follows: ； ； Among them, C t This represents the current cell state; ⊙ indicates element-wise multiplication; Candidate cell state; W C b represents the weighting parameter of the memory cell. C This refers to the deviation of the memory unit; The output in the hidden state is: ； Forward LSTM and backward LSTM model the forward and backward directions of the time series respectively, and their dependency relationship is as follows: ； ； in, It is in a forward-hidden state; It is in a backward hidden state; The final output is obtained by concatenating or summing the two-way hidden states: or .
8. The multimodal real-time sign language recognition method in a medical setting according to claim 1, characterized in that, In step 4), the language reasoning module, based on a large language model and combined with prior knowledge in the medical field, performs semantic-level reasoning and reconstruction on the gesture recognition sequence, completing medical terminology-level sequence completion, resolving ambiguities caused by similar or identical gestures, context-based error correction, medical semantic consistency verification, and semantic aggregation of multi-frame recognition results.
9. A sign language recognition system employing the multimodal real-time sign language recognition method for medical scenarios according to any one of claims 1-8, characterized in that, include: The data acquisition module is used to acquire a sequence of sign language images across multiple consecutive frames; The hand joint detection module is used to mark the position of each hand bone joint in a single frame of sign language image, connect the hand bone joints according to a preset topology, construct a hand bone structure map, and represent the spatial shape and posture changes of the gesture. The spatiotemporal feature extraction module is used to recognize the annotated sign language image sequence to obtain the gesture recognition sequence; The language reasoning module is used to perform semantic reasoning and reconstruction on the gesture recognition sequence output by the spatiotemporal feature extraction module. The output and interaction module is used to display the output results of the language reasoning module in text form on the terminal screen, or output them in voice broadcast form, or output them in the form of medical information system interface data, or as input to the electronic medical record system or consultation system.
10. The sign language recognition system according to claim 9, characterized in that, The spatiotemporal feature extraction module includes: The spatial feature submodule is used to extract the spatial features of the hand skeletal structure in a single frame of sign language image; The time series submodule is used to perform time-dimensional reasoning on the gesture spatial features output by the spatial feature submodule across multiple frames.