Sign language recognition system based on cross-modal graph alignment
By combining medical sign language vocabulary and visual features with cross-modal graph alignment technology, the individual differences and lighting angle effects of sign language recognition in medical emergency scenarios have been resolved, enabling rapid and accurate recognition of key information and improving the application effect of the model in medical emergency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 天津市急救中心
- Filing Date
- 2026-01-07
- Publication Date
- 2026-04-17
AI Technical Summary
Existing sign language recognition technologies suffer from limitations in generalization capabilities in medical emergency scenarios due to individual differences and complex lighting angles. They fail to fully capture medical semantic features, and traditional alignment mechanisms lack sufficient accuracy, thus failing to meet the requirements for speed and accuracy in extracting key information during emergency care.
By employing a medical sign language vocabulary feature extraction module and a visual feature extraction module, combined with a cross-modal graph alignment module, a cross-modal joint graph is generated, node connection weights are calculated, and a CTC loss function and visual-medical vocabulary alignment constraints are constructed to achieve fine-grained semantic association and improve the model's generalization ability and alignment accuracy.
Quickly and accurately identify key information expressed in sign language, improve emergency response efficiency and patient survival rate, reduce redundant frames and interfering actions, and meet the speed and accuracy requirements for extracting key information in emergency situations.
Smart Images
Figure CN121884392A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the technical field of artificial intelligence sign language recognition, and in particular to a sign language recognition system based on cross-modal graph alignment. Background Technology
[0002] In emergency medical settings, continuous sign language recognition technology is crucial for enabling barrier-free communication between deaf and hearing individuals. This technology automatically converts sign language video sequences into text word sequences, helping medical professionals quickly and accurately understand critical information from hearing-impaired patients, such as symptom descriptions, allergy history, and medical history. This information plays an irreplaceable role in improving emergency response efficiency and ensuring patient safety.
[0003] Despite significant progress made in sign language recognition using deep learning methods, these technologies still face some prominent limitations when applied to high-stress, high-risk emergency medical environments. First, different sign language users exhibit significant individual differences in gesture shape, range of motion, and execution speed when expressing the same medical terms. These differences, coupled with complex factors such as lighting and angles in emergency environments, severely limit the model's generalization ability, thus affecting the reliability of information acquisition.
[0004] Secondly, most existing methods employ static word embeddings or pre-trained language models from general domains to extract lexical features. These methods fail to fully capture the specific co-occurrence relationships and domain-specific grammatical structures between words in medical emergency scenarios. For example, in a medical context, the word "pain" is often associated with a specific "location," while the word "drug" may be related to an "allergic" reaction. This lack of specialized representation of medical semantics limits the effectiveness of sign language recognition systems in the field of medical emergency care.
[0005] Finally, traditional alignment mechanisms, such as cross-attention, often focus on global feature interactions while neglecting the fine-grained semantic relationships between visual features and corresponding medical terms. In emergency videos, the presence of redundant frames and interfering actions limits alignment accuracy, failing to meet the requirements for speed and accuracy in extracting critical information during emergency care.
[0006] Therefore, to address these challenges, there is an urgent need to develop a novel network architecture capable of deeply integrating prior knowledge in the medical field and achieving structured cross-modal alignment. Such an architecture would help improve the communication efficiency and reliability of the system in the special and critical scenario of emergency medical care, thereby better serving deaf and mute patients and medical professionals. Summary of the Invention
[0007] In view of this, the purpose of the present invention is to provide a sign language recognition system based on cross-modal graph alignment in order to solve the above problems.
[0008] In a first aspect, embodiments of the present invention provide a sign language recognition method for medical emergency care, applied to a terminal, characterized in that the terminal is equipped with a medical emergency sign language recognition model, the medical emergency sign language recognition model comprising: a medical sign language vocabulary feature extraction module, a medical sign language visual feature extraction module, and a cross-modal graph alignment module, the method comprising: Obtain a vocabulary set of medical emergency sign language videos, and input the vocabulary set of medical emergency sign language videos into the medical sign language vocabulary feature extraction module and the medical sign language visual feature extraction module for training; The medical sign language vocabulary feature extraction module is configured to: output the medical sign language vocabulary modal map corresponding to each medical emergency sign language video in the medical emergency sign language video-vocabulary set; and the medical sign language visual feature extraction module: extracts the visual features corresponding to each medical emergency sign language video to output the medical sign language video modal map corresponding to each medical emergency sign language video. The medical sign language visual feature extraction module is trained using the medical sign language vocabulary modal diagram and the medical sign language video modal diagram corresponding to each medical emergency sign language video. The step of training the medical sign language visual feature extraction module using the medical sign language vocabulary modal map corresponding to each medical emergency sign language video and the medical sign language video modal map corresponding to each medical emergency sign language video includes: The medical sign language vocabulary modal graph corresponding to each medical first aid sign language video and the medical sign language video modal graph corresponding to each medical first aid sign language video are fused to generate a cross-modal joint graph, which includes video nodes and vocabulary nodes. Calculate the connection weights between the video nodes and the vocabulary nodes in the cross-modal joint graph, and determine the category probability of each frame in each medical first aid sign language video based on the connection weights. A CTC loss function and a visual-medical vocabulary alignment constraint are constructed. An objective function is constructed based on the visual-medical vocabulary alignment constraint and the CTC loss function. The hyperparameters in the medical sign language visual feature extraction module are then modified based on the objective function.
[0009] The medical emergency sign language recognition method provided in this application combines lexical and visual features of medical sign language and achieves cross-modal graph alignment. This enables rapid and accurate identification of key information expressed in sign language, such as symptom descriptions, allergy history, and medical history, thereby improving emergency response efficiency and patient survival rates. Furthermore, by training the medical sign language visual feature extraction module, individual differences among sign language users in terms of gesture shape, range of motion, and execution speed, as well as complex conditions in the emergency environment such as lighting and angles, are considered, enhancing the model's generalization ability. Finally, fine-grained semantic association between video nodes and lexical nodes is established. This alignment mechanism reduces the impact of redundant frames and interfering actions, improves alignment accuracy, and meets the requirements for speed and accuracy in extracting key information during emergency care.
[0010] As a preferred approach, the nodes in the medical sign language vocabulary modal graph are determined based on medical emergency sign language vocabulary, and the edges in the medical sign language video modal graph are determined based on the co-occurrence frequency of medical emergency sign language vocabulary in a single sentence. The nodes in the medical sign language video modal graph are determined based on the visual features of the medical emergency sign language video, and the edges in the medical sign language video modal graph are determined based on the similarity of adjacent frames in the medical emergency sign language video.
[0011] As a preferred approach, the visual features include one or more of frame-level visual features, long-term temporal features, and local temporal features.
[0012] As a preferred approach, the medical sign language vocabulary feature extraction module includes a BERT submodule and a dual self-supervised reinforcement learning submodule. The steps of acquiring a vocabulary set of medical emergency sign language videos and inputting the vocabulary set into the medical sign language vocabulary feature extraction module and the medical sign language visual feature extraction module for training include: The BERT submodule is used to extract the vocabulary features of medical emergency sign language from the video-vocabulary set, and a vocabulary structure graph is constructed based on the vocabulary features. The nodes in the vocabulary structure graph are determined based on the medical emergency sign language vocabulary, and the edges in the vocabulary structure graph are determined based on the co-occurrence frequency of the medical emergency sign language vocabulary in a single sentence. A portion of the nodes in the lexical structure graph are used as mask nodes, and at least part of the other node in the lexical structure graph and the edges in the lexical structure graph are input into the dual self-supervised reinforcement learning submodule for learning, and the masking result of the mask nodes is output. Construct a loss function, and adjust the parameters in the dual self-supervised reinforcement learning submodule based on the loss function; The loss function is as follows: ; —Loss function; —Represents the set of mask nodes; —mask node The corresponding blocking results; —Mask node.
[0013] As a preferred approach, in the step of calculating the connection weights between the video nodes and the vocabulary nodes in the cross-modal joint graph, and determining the category probability corresponding to each frame in each medical first aid sign language video based on the connection weights, the connection weights of each node in the cross-modal joint graph are calculated in the following manner: ; —Video Node With vocabulary nodes Connection weights between them; —Activation function; —Characteristics of video node V; —Features of vocabulary node T; The class probability for each frame is determined as follows: ; -frame The corresponding category probability; -frame With vocabulary nodes Connection weights between them; — The number of vocabulary nodes.
[0014] As a preferred approach, the steps of constructing the CTC loss function and visual-medical vocabulary alignment constraints, constructing an objective function based on the visual-medical vocabulary alignment constraints and the CTC loss function, and correcting the hyperparameters in the medical sign language visual feature extraction module based on the objective function are as follows: The objective function is as follows: Min ; —Objective function; —CTC loss function; —Visual-medical vocabulary alignment constraints; , —Weighting coefficient; The constructed CTC loss function is as follows: ; —CTC loss function; —A sequence of medical sign language vocabulary corresponding to the medical emergency sign language video; —Medical First Aid Sign Language Video The category probability corresponding to each frame; The constructed visual-medical vocabulary alignment constraints are as follows: ; —Visual-medical vocabulary alignment constraints; T—Length of the medical first aid sign language video; N—Length of the medical sign language vocabulary sequence; —Medical sign language words aligned with video clip i in the medical first aid sign language video; S—cosine similarity matrix, the cosine similarity of adjacent frames is determined by the cosine similarity matrix; —Medical sign language vocabulary in medical first aid sign language videos.
[0015] As a preferred embodiment, the method further includes optimizing the medical emergency sign language recognition model using the Adam optimizer.
[0016] Secondly, this application provides a medical emergency sign language recognition device applied to a terminal. The terminal is equipped with a medical emergency sign language recognition model, which includes: a medical sign language vocabulary feature extraction module, a medical sign language visual feature extraction module, and a cross-modal graph alignment module. The method includes: The first training module is used to acquire a vocabulary set of medical emergency sign language videos. The vocabulary set of medical emergency sign language videos is then input into the medical sign language vocabulary feature extraction module and the medical sign language visual feature extraction module for training. The medical sign language vocabulary feature extraction module is configured to: output the medical sign language vocabulary modal map corresponding to each medical emergency sign language video in the medical emergency sign language video-vocabulary set; and the medical sign language visual feature extraction module: extracts the visual features corresponding to each medical emergency sign language video to output the medical sign language video modal map corresponding to each medical emergency sign language video. The second training module is used to train the medical sign language visual feature extraction module using the medical sign language vocabulary modal diagram corresponding to each medical emergency sign language video and the medical sign language video modal diagram corresponding to each medical emergency sign language video. The second training module is configured as follows: The medical sign language vocabulary modal graph corresponding to each medical first aid sign language video and the medical sign language video modal graph corresponding to each medical first aid sign language video are fused to generate a cross-modal joint graph, which includes video nodes and vocabulary nodes. Calculate the connection weights between the video nodes and the vocabulary nodes in the cross-modal joint graph, and determine the category probability of each frame in each medical first aid sign language video based on the connection weights. A CTC loss function and a visual-medical vocabulary alignment constraint are constructed. An objective function is constructed based on the visual-medical vocabulary alignment constraint and the CTC loss function. The hyperparameters in the medical sign language visual feature extraction module are then modified based on the objective function.
[0017] Thirdly, this application provides an electronic device, comprising: At least one processor; and at least one memory communicatively connected to said processor, wherein: The memory stores program instructions that can be executed by the processor, which can invoke the program instructions to perform the method as described in the first aspect.
[0018] Fourthly, this application provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions that cause the computer to perform the method as described in the first aspect.
[0019] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.
[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0021] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0022] Figure 1 A flowchart of a sign language recognition method for medical emergency care provided in an embodiment of the present invention; Figure 2 A structural diagram of the sign language recognition device for medical emergency care provided in an embodiment of the present invention; Figure 3 This is a structural diagram of an electronic device. Detailed Implementation
[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0024] In emergency medical settings, continuous sign language recognition technology is crucial for enabling barrier-free communication between deaf and hearing individuals. This technology automatically converts sign language video sequences into text word sequences, helping medical professionals quickly and accurately understand critical information from hearing-impaired patients, such as symptom descriptions, allergy history, and medical history. This information plays an irreplaceable role in improving emergency response efficiency and ensuring patient safety.
[0025] Despite significant progress made in sign language recognition using deep learning methods, these technologies still face some prominent limitations when applied to high-stress, high-risk emergency medical environments. First, different sign language users exhibit significant individual differences in gesture shape, range of motion, and execution speed when expressing the same medical terms. These differences, coupled with complex factors such as lighting and angles in emergency environments, severely limit the model's generalization ability, thus affecting the reliability of information acquisition.
[0026] Secondly, most existing methods employ static word embeddings or pre-trained language models from general domains to extract lexical features. These methods fail to fully capture the specific co-occurrence relationships and domain-specific grammatical structures between words in medical emergency scenarios. For example, in a medical context, the word "pain" is often associated with a specific "location," while the word "drug" may be related to an "allergic" reaction. This lack of specialized representation of medical semantics limits the effectiveness of sign language recognition systems in the field of medical emergency care.
[0027] Finally, traditional alignment mechanisms, such as cross-attention, often focus on global feature interactions while neglecting the fine-grained semantic relationships between visual features and corresponding medical terms. In emergency videos, the presence of redundant frames and interfering actions limits alignment accuracy, failing to meet the requirements for speed and accuracy in extracting critical information during emergency care.
[0028] Therefore, to address these challenges, there is an urgent need to develop a novel network architecture capable of deeply integrating prior knowledge in the medical field and achieving structured cross-modal alignment. Such an architecture would help improve the communication efficiency and reliability of the system in the special and critical scenario of emergency medical care, thereby better serving deaf and mute patients and medical professionals.
[0029] To address the aforementioned issues, this application provides a sign language recognition method for medical emergency care. This method is applied to a terminal equipped with a medical emergency sign language recognition model, which includes: a medical sign language vocabulary feature extraction module, a medical sign language visual feature extraction module, and a cross-modal graph alignment module.
[0030] The sign language recognition method for medical emergency provided in this application specifically includes the following steps: S1: Obtain the vocabulary set of medical emergency sign language videos, and input the vocabulary set of medical emergency sign language videos into the medical sign language vocabulary feature extraction module and the medical sign language visual feature extraction module for training.
[0031] It should be noted that, in the embodiments provided in this application, the medical emergency sign language video-vocabulary set specifically includes: medical emergency sign language videos and medical sign language vocabulary.
[0032] Before being input into the medical sign language vocabulary feature extraction module and the medical sign language visual feature extraction module, the data in the medical emergency sign language video-vocabulary set is preprocessed. Specifically, this includes flipping the medical emergency sign language video, randomly cropping it, and normalizing the data. For each medical emergency sign language video, the co-occurrence frequency of different words in a sentence is calculated.
[0033] Here, the medical sign language vocabulary feature extraction module is configured to: output the medical sign language vocabulary modal map corresponding to each medical emergency sign language video in the medical emergency sign language video-vocabulary set; and the medical sign language visual feature extraction module: extracts the visual features corresponding to each medical emergency sign language video to output the medical sign language video modal map corresponding to each medical emergency sign language video.
[0034] The medical sign language visual feature extraction module extracts the visual features corresponding to each medical emergency sign language video to output a medical sign language video modal map corresponding to each medical emergency sign language video.
[0035] Here, the nodes in the medical sign language vocabulary modal graph are determined based on medical first aid sign language vocabulary; the edges in the medical sign language video modal graph are determined based on the co-occurrence frequency of medical first aid sign language vocabulary in a single sentence; the nodes in the medical sign language video modal graph are determined based on the visual features of the medical first aid sign language video; and the edges in the medical sign language video modal graph are determined based on the similarity between adjacent frames in the medical first aid sign language video. It should be noted that the above-mentioned construction methods for medical sign language vocabulary modal diagrams and medical sign language video modal diagrams are only one feasible method. Those skilled in the art can use other methods to construct medical sign language vocabulary modal diagrams and medical sign language video modal diagrams, which are not limited here.
[0036] S2: The medical sign language visual feature extraction module is trained using the medical sign language vocabulary modal diagram and the medical sign language video modal diagram corresponding to each medical emergency sign language video.
[0037] Specifically, in this step, firstly, the medical sign language vocabulary modal diagram corresponding to each medical emergency sign language video and the medical sign language video modal diagram corresponding to each medical emergency sign language video are fused to generate a cross-modal joint graph.
[0038] It should be noted that the cross-modal joint graph includes video nodes and vocabulary nodes. The vocabulary nodes are generated based on the nodes in the aforementioned medical sign language vocabulary modal graph, while the video nodes correspond to the nodes in the aforementioned medical sign language video modal graph.
[0039] Then, the connection weights between the video nodes and the vocabulary nodes in the cross-modal joint graph are calculated, and the category probability corresponding to each frame in each medical first aid sign language video is determined based on the connection weights.
[0040] Specifically, the connection weights of each node in the cross-modal joint graph are obtained in the following manner: ; —Video Node With vocabulary nodes Connection weights between them; —Normalized activation function; —Characteristics of video node V; —Features of vocabulary node T; The class probability for each frame is determined as follows: ; -frame The corresponding category probability; -frame With vocabulary nodes Connection weights between them; — The number of vocabulary nodes.
[0041] Once the class probability for each frame is calculated, a class probability sequence can be constructed.
[0042] In some embodiments, a gated graph neural network is applied to iteratively update the vocabulary node and video node representations of the cross-modal joint graph to suppress noise associations.
[0043] Finally, a CTC loss function and a visual-medical vocabulary alignment constraint are constructed. Based on the visual-medical vocabulary alignment constraint and the CTC loss function, an objective function is constructed. Based on the objective function, the hyperparameters in the medical sign language visual feature extraction module are modified.
[0044] Specifically, in the embodiments provided in this application... The constructed CTC loss function is as follows: ; —CTC loss function; —A sequence of medical sign language vocabulary corresponding to the medical emergency sign language video; —Medical First Aid Sign Language Video The category probability corresponding to each frame; The constructed visual-medical vocabulary alignment constraints are as follows: ; —Visual-medical vocabulary alignment constraints; T—Length of the medical first aid sign language video; N—Length of the medical sign language vocabulary sequence; —Medical sign language words aligned with video clip i in the medical first aid sign language video; S—cosine similarity matrix, the cosine similarity of adjacent frames is determined by the cosine similarity matrix;
[0045] —Medical sign language vocabulary.
[0046] The objective function is as follows: Min ; —Objective function; —CTC loss function; —Visual-medical vocabulary alignment constraints; , —Weighting coefficient; The minimum value of the objective function indicates that the medical emergency sign language recognition model has the best recognition effect.
[0047] The medical emergency sign language recognition method provided in this application combines lexical and visual features of medical sign language and achieves cross-modal graph alignment. This enables rapid and accurate identification of key information expressed in sign language, such as symptom descriptions, allergy history, and medical history, thereby improving emergency response efficiency and patient survival rates. Furthermore, by training the medical sign language visual feature extraction module, individual differences among sign language users in terms of gesture shape, range of motion, and execution speed, as well as complex conditions in the emergency environment such as lighting and angles, are considered, enhancing the model's generalization ability. Finally, fine-grained semantic association between video nodes and lexical nodes is established. This alignment mechanism reduces the impact of redundant frames and interfering actions, improves alignment accuracy, and meets the requirements for speed and accuracy in extracting key information during emergency care.
[0048] The following section will elaborate on the medical sign language vocabulary feature extraction module in this application: In the embodiments provided in this application, the medical sign language vocabulary feature extraction module includes a BERT submodule and a dual self-supervised reinforcement learning submodule. In order to train the medical sign language vocabulary feature extraction module, the BERT submodule is first used to extract medical emergency sign language vocabulary features from the medical emergency sign language video-vocabulary set.
[0049] Specifically, the structure of a lexical structure diagram is represented as follows: , Corresponding medical first aid sign language vocabulary, i.e., nodes in the vocabulary structure diagram. Based on the determination of medical emergency sign language vocabulary, and the edges in the vocabulary structure graph Determining the co-occurrence frequency of medical emergency sign language vocabulary in a single sentence.
[0050] Then, a portion of the nodes in the lexical structure graph are used as mask nodes, and at least a portion of the other nodes and edges in the lexical structure graph are input into the dual self-supervised reinforcement learning submodule for learning, and the masking result of the mask nodes is output.
[0051] Finally, a loss function is constructed, and the parameters in the dual self-supervised reinforcement learning submodule are adjusted based on the loss function.
[0052] Specifically, in the embodiments provided in this application, a portion of the nodes in the vocabulary structure graph are replaced with learnable vectors, thereby turning the replaced nodes in the vocabulary structure graph into mask nodes. LSTM is then used as a decoder to reconstruct the masked features, i.e., to reconstruct the masking result.
[0053] In the embodiments provided in this application, the loss function is specifically constructed using the distance between the masking result and the original node of the mask node. Specifically, the loss function is as follows: ; —Loss function; —Represents the set of mask nodes; —mask node The corresponding blocking results; —Mask node.
[0054] In the embodiments provided in this application, the BERT submodule is used to extract the vocabulary features of medical emergency sign language from the medical emergency sign language video-vocabulary set, and a vocabulary structure graph is constructed based on these features. This helps to better understand the relationships between sign language words, thereby improving the accuracy of recognition. Simultaneously, by using a portion of the nodes in the vocabulary structure graph as mask nodes and employing a dual self-supervised reinforcement learning submodule for learning, this invention further improves the model's ability to understand and generalize sign language vocabulary, thus enabling it to better handle complex situations that may be encountered in practical applications.
[0055] The following section will describe the method for extracting visual features in this application.
[0056] As described above, in the embodiments provided in this application, the visual features specifically include one or more of frame-level visual features, long-term temporal features, and local temporal features. Frame-level visual features are the spatial features of the medical emergency sign language video, while long-term temporal features and local temporal features are the temporal features of the medical emergency sign language video.
[0057] Specifically, ResNet18 can be used to extract frame-level visual features from medical emergency sign language videos, a bidirectional long short-term memory network (BLSTM) can be used to extract long-term temporal features from medical emergency sign language videos, and a 1D CNN can be used to extract local temporal features from medical emergency sign language videos.
[0058] The above methods can be used to extract the visual features of medical emergency sign language videos, laying the foundation for constructing a modal diagram of medical sign language videos.
[0059] In some embodiments, the medical emergency sign language recognition model is also equipped with an Adam optimizer, which is specifically used to optimize the medical emergency sign language recognition model.
[0060] Based on the foregoing, this application also provides a medical emergency sign language recognition device applied to a terminal. The terminal is equipped with a medical emergency sign language recognition model, which includes: a medical sign language vocabulary feature extraction module, a medical sign language visual feature extraction module, and a cross-modal graph alignment module. The method includes: The first training module is used to acquire a vocabulary set of medical emergency sign language videos. The vocabulary set of medical emergency sign language videos is then input into the medical sign language vocabulary feature extraction module and the medical sign language visual feature extraction module for training. The medical sign language vocabulary feature extraction module is configured to: output the medical sign language vocabulary modal map corresponding to each medical emergency sign language video in the medical emergency sign language video-vocabulary set; and the medical sign language visual feature extraction module: extracts the visual features corresponding to each medical emergency sign language video to output the medical sign language video modal map corresponding to each medical emergency sign language video. The second training module is used to train the medical sign language visual feature extraction module using the medical sign language vocabulary modal diagram corresponding to each medical emergency sign language video and the medical sign language video modal diagram corresponding to each medical emergency sign language video. The second training module is configured as follows: The medical sign language vocabulary modal graph corresponding to each medical first aid sign language video and the medical sign language video modal graph corresponding to each medical first aid sign language video are fused to generate a cross-modal joint graph, which includes video nodes and vocabulary nodes. Calculate the connection weights between the video nodes and the vocabulary nodes in the cross-modal joint graph, and determine the category probability of each frame in each medical first aid sign language video based on the connection weights. A CTC loss function and a visual-medical vocabulary alignment constraint are constructed. An objective function is constructed based on the visual-medical vocabulary alignment constraint and the CTC loss function. The hyperparameters in the medical sign language visual feature extraction module are then modified based on the objective function.
[0061] Unless otherwise specifically stated, the relative steps, numerical expressions, and values of the components and steps described in these embodiments do not limit the scope of the invention.
[0062] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.
[0063] In all examples shown and described herein, any specific values should be interpreted as merely exemplary and not as limitations; therefore, other examples of exemplary embodiments may have different values.
[0064] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0065] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0066] Figure 3 A block diagram is shown that is suitable for implementing embodiments of the present invention. Figure 3 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.
[0067] like Figure 3As shown, the electronic device is represented in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: one or more processors 410, memory 430, and communication bus 440 connecting different system components (including memory 430 and processing unit 410).
[0068] Communication bus 440 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MAC) buses, Enhanced ISA buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses.
[0069] Electronic devices typically include a variety of computer-readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, and removable and non-removable media.
[0070] Memory 430 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. Memory 430 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.
[0071] A program / utility having a set (at least one) of program modules can be stored in memory 430. Such program modules include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules typically perform the functions and / or methods described in the embodiments of this invention.
[0072] Processor 410 executes various functional applications and data processing by running programs stored in memory 430, such as implementing embodiments of the present invention. Figure 1 The illustrated embodiment provides a sign language recognition method for medical emergency care.
[0073] This invention provides a non-transitory computer-readable storage medium that stores computer instructions, which cause the computer to execute embodiments of this invention. Figure 1 The illustrated embodiment provides a sign language recognition method for basic medical emergency care.
[0074] The aforementioned computer-readable storage medium may be any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by or in connection with an instruction execution system, apparatus, or device.
[0075] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.
[0076] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.
[0077] Computer program code for performing the operations of embodiments of the present invention can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0078] The foregoing has described specific embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0079] In the description of the embodiments of the present invention, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In the embodiments of the present invention, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in the embodiments of the present invention, as well as the features of the different embodiments or examples.
[0080] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of embodiments of the present invention, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0081] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.
[0082] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."
[0083] It should be noted that the terminals involved in the embodiments of the present invention may include, but are not limited to, personal computers (PCs), personal digital assistants (PDAs), wireless handheld devices, tablet computers, mobile phones, MP3 players, MP4 players, etc.
[0084] In the several embodiments provided in this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.
[0085] Furthermore, in the various embodiments of the present invention, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.
[0086] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0087] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A sign language recognition method for medical emergency care, applied to a terminal, characterized in that, The terminal is equipped with a medical emergency sign language recognition model, which includes: a medical sign language vocabulary feature extraction module, a medical sign language visual feature extraction module, and a cross-modal graph alignment module. The method includes: Obtain a vocabulary set of medical emergency sign language videos, and input the vocabulary set of medical emergency sign language videos into the medical sign language vocabulary feature extraction module and the medical sign language visual feature extraction module for training; The medical sign language vocabulary feature extraction module is configured to output a medical sign language vocabulary modal diagram corresponding to each medical emergency sign language video in the medical emergency sign language video-vocabulary set; The medical sign language visual feature extraction module extracts the visual features corresponding to each medical emergency sign language video to output a medical sign language video modal map corresponding to each medical emergency sign language video. The medical sign language visual feature extraction module is trained using the medical sign language vocabulary modal diagram and the medical sign language video modal diagram corresponding to each medical emergency sign language video. The step of training the medical sign language visual feature extraction module using the medical sign language vocabulary modal map corresponding to each medical emergency sign language video and the medical sign language video modal map corresponding to each medical emergency sign language video includes: The medical sign language vocabulary modal graph corresponding to each medical emergency sign language video and the medical sign language video modal graph corresponding to each medical emergency sign language video are fused to generate a cross-modal joint graph, which includes video nodes and vocabulary nodes; Calculate the connection weights between the video nodes and the vocabulary nodes in the cross-modal joint graph, and determine the category probability of each frame in each medical first aid sign language video based on the connection weights. A CTC loss function and a visual-medical vocabulary alignment constraint are constructed. An objective function is constructed based on the visual-medical vocabulary alignment constraint and the CTC loss function. The hyperparameters in the medical sign language visual feature extraction module are then modified based on the objective function.
2. The method according to claim 1, characterized in that, The nodes in the medical sign language vocabulary modal graph are determined based on medical emergency sign language vocabulary, and the edges in the medical sign language video modal graph are determined based on the co-occurrence frequency of medical emergency sign language vocabulary in a single sentence. The nodes in the medical sign language video modal graph are determined based on the visual features of the medical emergency sign language video, and the edges in the medical sign language video modal graph are determined based on the similarity of adjacent frames in the medical emergency sign language video.
3. The method according to claim 1, characterized in that, The visual features include one or more of frame-level visual features, long-term temporal features, and local temporal features.
4. The method according to claim 3, characterized in that, The medical sign language vocabulary feature extraction module includes a BERT submodule and a dual self-supervised reinforcement learning submodule; The steps of acquiring a vocabulary set of medical emergency sign language videos and inputting the vocabulary set into the medical sign language vocabulary feature extraction module and the medical sign language visual feature extraction module for training include: The BERT submodule is used to extract the vocabulary features of medical emergency sign language from the video-vocabulary set, and a vocabulary structure graph is constructed based on the vocabulary features. The nodes in the vocabulary structure graph are determined based on the medical emergency sign language vocabulary, and the edges in the vocabulary structure graph are determined based on the co-occurrence frequency of the medical emergency sign language vocabulary in a single sentence. A portion of the nodes in the lexical structure graph are used as mask nodes, and at least part of the other node in the lexical structure graph and the edges in the lexical structure graph are input into the dual self-supervised reinforcement learning submodule for learning, and the masking result of the mask nodes is output. Construct a loss function, and adjust the parameters in the dual self-supervised reinforcement learning submodule based on the loss function; The loss function is as follows: ; —Loss function; —Represents the set of mask nodes; —mask node The corresponding blocking results; —Mask node.
5. The method according to claim 4, characterized in that, In the step of calculating the connection weights between the video nodes and the vocabulary nodes in the cross-modal joint graph, and determining the category probability corresponding to each frame in each medical first aid sign language video based on the connection weights, the connection weights of each node in the cross-modal joint graph are calculated in the following manner: ; —Video Node With vocabulary nodes Connection weights between them; —Activation function; —Characteristics of video node V; —Features of vocabulary node T; The class probability for each frame is determined as follows: ; -frame The corresponding category probability; -frame With vocabulary nodes Connection weights between them; — The number of vocabulary nodes.
6. The method according to any one of claims 2 to 5, characterized in that, The steps of constructing the CTC loss function and visual-medical vocabulary alignment constraints, constructing an objective function based on the visual-medical vocabulary alignment constraints and the CTC loss function, and correcting the hyperparameters in the medical sign language visual feature extraction module based on the objective function are as follows: The objective function is as follows: Min ; —Objective function; —CTC loss function; —Visual-medical vocabulary alignment constraints; , —Weighting coefficient; The constructed CTC loss function is as follows: ; —CTC loss function; —A sequence of medical sign language vocabulary corresponding to the medical emergency sign language video; —Medical First Aid Sign Language Video The category probability corresponding to each frame; The constructed visual-medical vocabulary alignment constraints are as follows: ； —Visual-medical vocabulary alignment constraints; T—Length of the medical first aid sign language video; N—Length of the medical sign language vocabulary sequence; —Medical sign language words aligned with video clip i in the medical first aid sign language video; S—cosine similarity matrix, the cosine similarity of adjacent frames is determined by the cosine similarity matrix; —Medical sign language vocabulary in medical first aid sign language videos.
7. The method according to claim 6, characterized in that, The method further includes optimizing the medical emergency sign language recognition model using the Adam optimizer.
8. A sign language recognition device for medical emergency care, applied to a terminal, characterized in that, The terminal is equipped with a medical emergency sign language recognition model, which includes: a medical sign language vocabulary feature extraction module, a medical sign language visual feature extraction module, and a cross-modal graph alignment module. The method includes: The first training module is used to acquire a vocabulary set of medical emergency sign language videos. The vocabulary set of medical emergency sign language videos is then input into the medical sign language vocabulary feature extraction module and the medical sign language visual feature extraction module for training. The medical sign language vocabulary feature extraction module is configured to: output the medical sign language vocabulary modal map corresponding to each medical emergency sign language video in the medical emergency sign language video-vocabulary set; and the medical sign language visual feature extraction module: extracts the visual features corresponding to each medical emergency sign language video to output the medical sign language video modal map corresponding to each medical emergency sign language video. The second training module is used to train the medical sign language visual feature extraction module using the medical sign language vocabulary modal diagram corresponding to each medical emergency sign language video and the medical sign language video modal diagram corresponding to each medical emergency sign language video. The second training module is configured as follows: The medical sign language vocabulary modal graph corresponding to each medical emergency sign language video and the medical sign language video modal graph corresponding to each medical emergency sign language video are fused to generate a cross-modal joint graph, which includes video nodes and vocabulary nodes. Calculate the connection weights between the video nodes and the vocabulary nodes in the cross-modal joint graph, and determine the category probability of each frame in each medical first aid sign language video based on the connection weights. A CTC loss function and a visual-medical vocabulary alignment constraint are constructed. An objective function is constructed based on the visual-medical vocabulary alignment constraint and the CTC loss function. The hyperparameters in the medical sign language visual feature extraction module are then modified based on the objective function.
9. An electronic device, characterized in that, include: At least one processor; as well as At least one memory communicatively connected to the processor, wherein: The memory stores program instructions that can be executed by the processor, which can invoke the program instructions to perform the method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause the computer to perform the method as described in any one of claims 1 to 7.