An end-side personalized and cloud pre-training fused multi-modal sign language recognition system

Abstract

The application discloses a multi-modal sign language recognition system combining end-side individualization with cloud-side pre-training, relates to the field of artificial intelligence barrier-free interaction technology, and comprises a user end and a server end.The user end comprises a camera acquisition module, a hand key point extraction module, an upper limb, trunk and facial feature key point extraction module, a feature processing module, a motion start and end detection module, a training module, a template quality evaluation module, a scene vocabulary management module, a candidate recall and precision matching module, an end-cloud fusion decision module, a voice broadcast and bidirectional communication module, a data backup and privacy authorization module; the server end comprises a user account management module, a data backup module, a public sign language library, a dialect sign language library, a cloud-side pre-training model training module, a quality analysis module, a background management module, a version management module, a personal template training mechanism, and the adaptation capability of individual sign language differences is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence-based barrier-free interaction technology, and in particular to a multimodal sign language recognition system that integrates edge-side personalization and cloud-based pre-training. Background Technology

[0002] There are over 27 million hearing-impaired people in my country, and sign language is the most frequently used natural language form in their daily communication. The expression of sign language is greatly influenced by factors such as region, age, and personal habits, resulting in distinct regional dialect sign language systems across the country, and localized sign language expressions in ethnic minority areas. At the same time, some hearing-impaired users choose to use single-handed sign language as their primary means of communication due to physical limitations or long-term usage habits. The highly personalized and non-standardized nature of actual sign language use presents significant challenges to the research and development and implementation of sign language recognition technology.

[0003] Current sign language recognition technologies are mostly based on the standard vocabulary of the "National Common Sign Language," requiring users to strictly adhere to standard gestures. The system matches user gestures using image recognition or deep learning models. However, these solutions have encountered significant adaptation issues in practical application: the vast majority of hearing-impaired users do not use standard common sign language in their daily lives, but rather personalized sign language incorporating regional characteristics and personal habits. For example, in my country, the word "thank you" alone has multiple variations in different regional dialects, and current recognition systems are insufficient to recognize these variations.

[0004] Meanwhile, some existing solutions rely on cloud-deployed deep learning models for recognition calculations, requiring users to upload complete sign language video data to the server. This model is difficult to operate in environments with no or weak network coverage, such as when users visit hospitals, conduct business in government service halls, or travel outdoors, where network conditions are often unstable, potentially causing interruptions or significant delays in the recognition service. Furthermore, the raw video data contains information about the user's face, limbs, and environmental background, posing privacy risks during the upload and storage processes.

[0005] A few solutions attempt to achieve local recognition on the device side, but due to the limitations of computing power and power consumption of mobile devices, lightweight solutions struggle to meet practical requirements in terms of recognition accuracy. They also lack effective methods to handle issues such as blurred action boundaries and lack of pauses between words in long, continuous sign language sentences. Regarding personalized training with small samples, existing solutions present a contradiction: requiring users to record a large number of samples is cumbersome and has too high a barrier to entry, while collecting only a small number of samples results in insufficient model learning, and there is a lack of sample quality screening and optimization mechanisms. Overall, current technologies have not yet formed a complete sign language recognition solution that can balance personalized adaptation, dialect coverage, offline usability, privacy protection, and recognition accuracy. Summary of the Invention

[0006] The purpose of this invention is to at least solve one of the technical problems existing in the prior art, and to provide a multimodal sign language recognition system that integrates edge-side personalization and cloud-based pre-training, which can solve the problem of poor adaptability to individual sign language differences.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a multimodal sign language recognition system that integrates edge-side personalization and cloud-based pre-training, comprising a user terminal and a server terminal. The user terminal includes a camera acquisition module, a hand key point extraction module, an upper limb, torso, and facial key point extraction module, a feature processing module, an action start and stop detection module, a training module, a template quality evaluation module, a scene lexicon management module, a candidate recall and fine matching module, an edge-cloud fusion decision module, a voice broadcast and two-way communication module, and a data backup and privacy authorization module. The server side includes a user account management module, a data backup module, a public sign language library, a dialect sign language library, a cloud-based pre-trained model training module, a quality analysis module, a backend management module, and a version management module. The client and server exchange data via an encrypted communication protocol.

[0008] Preferably, the output end of the camera acquisition module is connected to the input end of the hand key point extraction module and the upper limb, torso and facial key point extraction module; The outputs of the two key point extraction modules are respectively connected to the input of the feature processing module; The output of the feature processing module is divided into two paths: one path is connected to the input of the training module, and the other path is connected to the input of the action start and stop detection module.

[0009] Preferably, the output of the training module is connected to the input of the template quality evaluation module, and the output of the template quality evaluation module is connected to the local personal template library.

[0010] Preferably, the output of the action start and stop detection module is connected to the input of the candidate recall and fine matching module. The candidate recall and fine matching module reads data from the local personal template library and the general offline language library package respectively for matching calculation, and its output is connected to the input of the end-to-cloud fusion decision module. The output of the edge-cloud converged decision module is connected to the input of the voice broadcast and two-way communication module.

[0011] Preferably, the data backup and privacy authorization module interacts with the training module and the edge-cloud fusion decision module respectively; The scenario-based vocabulary management module interacts with the local personal template library and the general offline vocabulary package to organize and manage scenario-based vocabulary.

[0012] Preferably, the user account management module interacts with the data backup module, and the data backup module interacts with the user's data backup and privacy authorization module and the quality analysis module, respectively.

[0013] Preferably, the output terminals of the public sign language library and the dialect sign language library are connected to the input terminal of the cloud-based pre-trained model training module.

[0014] Preferably, the output of the cloud-based pre-trained model training module is connected to the input of the version management module, the output of the version management module is connected to the lexicon generation module (which is located inside the background management module), and the output of the lexicon generation module sends a general offline lexicon package to the user terminal.

[0015] Preferably, the output of the quality analysis module is connected to the input of the public sign language library and the dialect sign language library.

[0016] Preferably, the backend management module interacts with the public sign language library, the dialect sign language library, the quality analysis module, and the version management module.

[0017] Compared with the prior art, the beneficial effects of the present invention are: 1. This multimodal sign language recognition system, which integrates edge-side personalization and cloud-based pre-training, enhances the adaptability to individual sign language differences. Traditional sign language recognition systems require users to strictly follow standard sign language movements and lack adaptation methods for personalized differences caused by different users' age, region, hand shape, body shape, and long-term usage habits. This invention, through an edge-side personal template training mechanism, allows users to independently train commonly used words and phrases. The system adaptively adjusts the feature weights of different sign language semantic types, enabling the system to adapt to individual unique sign language expressions without requiring users to change their own habits to adapt to the system standard.

[0018] 2. This multimodal sign language recognition system, which integrates edge-side personalization and cloud-based pre-training, broadens the coverage of dialect sign language and single-handed sign language. Existing technologies use the national standard sign language as the core training sample, which is insufficient in recognizing dialect sign language, ethnic minority sign language, and single-handed sign language. This invention incorporates non-standard sign language samples such as local dialect sign language, ethnic minority sign language, single-handed sign language, and elderly user sign language during the sample construction stage of the cloud-based pre-training model, and distributes them to the user terminal through a general offline language library package, enabling the system to recognize various sign language expressions that cannot be adapted by existing technologies in actual use.

[0019] 3. This multimodal sign language recognition system, which integrates edge-side personalization and cloud-based pre-training, solves the problems of network dependence and privacy protection in cloud-based recognition. Existing cloud-based recognition solutions require uploading raw video data, which can lead to recognition delays or failures when the network is unstable. At the same time, uploading raw video data poses a risk of privacy leakage. This invention adopts an edge-side priority recognition strategy, which allows a large number of sign language actions to be recognized and output locally without relying on a network connection. For scenarios that require cloud-based enhanced recognition, the transmitted content is only the de-identified key points and time-series compressed features. The raw video data never leaves the user's device, thus controlling privacy risks from the source of data collection.

[0020] 4. This multimodal sign language recognition system, which integrates edge-side personalization and cloud-based pre-training, reduces the computational requirements of edge-side recognition and improves recognition stability. Directly deploying large-scale recognition models on mobile terminals can lead to problems such as overheating, lag, and rapid power consumption. Meanwhile, lightweight and simple template solutions lack sufficient recognition accuracy and stability. This invention reduces edge-side computational overhead by narrowing the candidate matching range through scene partitioning management, filtering a limited number of candidate templates for fine matching through a candidate recall mechanism, and filtering low-confidence results through a fixed baseline mechanism. At the same time, it employs multiple preprocessing methods such as normalization, filtering, outlier removal, and occlusion compensation to ensure the stability of feature data, enabling edge-side recognition to maintain usable accuracy under low power consumption conditions.

[0021] 5. This multimodal sign language recognition system, which integrates edge-side personalization and cloud-based pre-training, improves the accuracy of continuous sign language recognition. Daily sign language communication often consists of long, continuous sentences with no obvious pauses between words. Existing technologies struggle to accurately determine action boundaries. This invention employs a sliding window and attention integral fusion mechanism to divide continuous sign language sequences into overlapping windows and perform multi-window integral fusion. By determining semantic switching points through attention weights, it improves the accuracy of word boundary determination in continuous expressions and reduces the probability of missed segmentation and misidentification.

[0022] 6. This multimodal sign language recognition system, which integrates edge-side personalization and cloud-based pre-training, optimizes the effect and ease of use of small-sample training. Existing personalization solutions require users to record a large number of samples, which is cumbersome, while collecting only a small number of samples results in insufficient learning. This invention introduces a data augmentation strategy into the training process to generate augmented templates from limited original samples to expand the template set. At the same time, a template quality evaluation module is set to score the quality of the generated templates. If the quality is insufficient, the user is prompted to supplement the recording or retrain. Furthermore, the local template library is continuously optimized through an outlier template elimination mechanism, which improves the training effect and the stability of the template library under small-sample conditions.

[0023] 7. This multimodal sign language recognition system, which integrates edge-side personalization and cloud-based pre-training, constructs a complementary mechanism for edge-cloud collaboration. Existing technologies are singular and do not construct a fusion architecture of "local personalization and cloud-based public language database supplementation". This invention adopts a strategy of edge-side priority, cloud-based supplementation, and dynamic fusion. The edge focuses on personalized adaptation and offline availability, while the cloud focuses on general semantic coverage and large-scale sample support. The two are comprehensively judged by the edge-cloud fusion decision module, which takes into account both personalized recognition accuracy and general sign language coverage. At the same time, the public language database is continuously iterated and optimized through a user-authorized feature contribution mechanism. Attached Figure Description

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1 This is a schematic diagram of the traditional cloud-based sign language recognition process of the present invention; Figure 2 This is a schematic diagram of the traditional standard sign language recognition process of the present invention; Figure 3 This is a diagram of the overall system architecture of the present invention; Figure 4 This is a flowchart of the personalized training process of the present invention; Figure 5 This is a flowchart illustrating the terminal-side identification process of the present invention; Figure 6 This is a flowchart of the edge-cloud fusion identification process of the present invention; Figure 7 This is a flowchart of the edge-cloud fusion identification process of the present invention; Figure 8 This is a diagram illustrating the collaboration between the personal template library and the offline language library package of this invention. Figure 9 This is a flowchart illustrating the privacy protection and feature contribution process of this invention. Detailed Implementation

[0025] This section will describe in detail specific embodiments of the present invention. Preferred embodiments of the present invention are shown in the accompanying drawings. The purpose of the drawings is to supplement the textual description with graphics, so that people can intuitively and vividly understand each technical feature and overall technical solution of the present invention, but they should not be construed as limiting the scope of protection of the present invention.

[0026] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0027] In the description of this invention, terms such as greater than, less than, and exceeding are understood to exclude the stated number, while terms such as above, below, and within are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0028] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0029] Please see Figures 1-9 This invention provides a technical solution: a multimodal sign language recognition system that integrates edge-side personalization and cloud-based pre-training. The system adopts an edge-cloud collaborative architecture and is divided into two main modules: the user end and the server end.

[0030] The user terminal is deployed on devices with cameras and computing capabilities, such as mobile phones, tablets, portable terminals, and public service window equipment. It is responsible for functions such as sign language action collection, key point extraction, local recognition, training and learning, and two-way communication and display. The server is deployed on a cloud server and is responsible for functions such as public sign language library management, dialect sign language library management, cloud pre-trained model training and inference, user account management, and generation and distribution of offline sign language library packages.

[0031] The user and the server exchange data through an encrypted communication protocol. The content of the exchange is de-identified motion feature data, and the user's original video data does not leave the user's device. The user interface includes the following functional modules: camera acquisition module, hand key point extraction module, upper limb, torso, and facial key point extraction module, feature processing module, action start and stop detection module, training module, template quality evaluation module, scene lexicon management module, candidate recall and fine matching module, edge-cloud fusion decision module, voice broadcast and two-way communication module, and data backup and privacy authorization module. The output of the camera acquisition module is connected to the input of the hand key point extraction module and the upper limb, torso, and facial key point extraction module.

[0032] The outputs of the two key point extraction modules are connected to the inputs of the feature processing module, respectively.

[0033] The output of the feature processing module is divided into two paths: one path is connected to the input of the training module, and the other path is connected to the input of the action start and stop detection module.

[0034] The output of the training module is connected to the input of the template quality evaluation module, and the output of the template quality evaluation module is connected to the local personal template library.

[0035] The output of the action start and stop detection module is connected to the input of the candidate recall and fine matching module. The candidate recall and fine matching module reads data from the local personal template library and the general offline language library package respectively for matching calculation, and its output is connected to the input of the end-to-cloud fusion decision module.

[0036] The output of the edge-cloud converged decision module is connected to the input of the voice broadcast and two-way communication module.

[0037] The data backup and privacy authorization module interacts with the training module and the edge-cloud fusion decision-making module, respectively.

[0038] The scenario-based vocabulary management module interacts with the local personal template library and the general offline vocabulary package to organize and manage scenario-based vocabulary. Furthermore, the server-side includes the following functional modules: user account management module, data backup module, public sign language library, dialect sign language library, cloud pre-trained model training module, quality analysis module, backend management module, and version management module; The user account management module interacts with the data backup module, which in turn interacts with the user's data backup and privacy authorization module and the quality analysis module.

[0039] The output of the quality analysis module is connected to the input of the public sign language library and the dialect sign language library.

[0040] The outputs of the public sign language library and the dialect sign language library are connected to the input of the cloud-based pre-trained model training module.

[0041] The output of the cloud-based pre-trained model training module is connected to the input of the version management module. The output of the version management module is connected to the lexicon generation module (which is located inside the backend management module). The output of the lexicon generation module sends a general offline lexicon package to the user.

[0042] The backend management module interacts with the public sign language library, the dialect sign language library, the quality analysis module, and the version management module. During system operation, the camera acquisition module continuously monitors the screen. When the action start and stop detection module detects a hand entering the screen, the system initiates the sign language action acquisition process. The camera acquisition module begins to capture video frame sequences of sign language actions. The hand key point extraction module and the upper limb, torso, and facial key point extraction module extract multi-dimensional desensitized action features from the video frames, including hand key point coordinates, upper limb joint angles, torso posture, and facial key point positions. The feature processing module performs preprocessing operations such as normalization, filtering, outlier removal, and occlusion compensation on the above features and outputs the processed stable feature data to subsequent modules. In sign language training mode, preprocessed feature data enters the training module. The system adaptively adjusts feature weights according to the semantic type of sign language, generates original templates, and generates enhanced templates through data augmentation strategies to form a complete template set. The template quality evaluation module scores the quality of the generated templates. Templates that pass the quality test are stored in the local personal template library, while templates that fail the quality test trigger a prompt for supplementation or retraining. In sign language recognition mode, the action start and stop detection module stops feature acquisition and determines the end of a single sign language action after the hand moves out of the screen. The candidate recall and fine matching module recalls a set of candidate templates from the local personal template library and the general offline language library package according to the current scene, performs temporal similarity matching calculation, and outputs the candidate results and their matching degree. The edge-cloud fusion decision module integrates the edge matching results and the cloud-enhanced recognition results to make an intelligent judgment, and outputs the final recognition result to the voice broadcast and two-way communication module, which displays the result in text and provides voice broadcast. The core principle of the system lies in the recognition strategy of "device-side priority, cloud-based supplementation, and dynamic fusion". The user terminal has a built-in local personal template library and a general offline language library package. The local personal template library stores the user's self-trained personal sign language templates, which include the user's commonly used personalized sign language vocabulary and short phrases, and are saved in the form of desensitized action features. The general offline language library package is a compressed package of general recognition capabilities generated by a cloud-based pre-trained model. It covers desensitized feature data of general standard sign language, sign language of various dialects, single-hand sign language, sign language of ethnic minorities, sign language of elderly users, and short phrase sign language exclusive to multiple industry scenarios. It is regularly updated and distributed to the user terminal with the iteration of the cloud language library. When the system performs sign language recognition, it first matches the user's actions on the device using a local personal template library. If there are candidate results in the local template library with a matching degree that reaches the baseline, the system directly outputs the recognition result on the device without calling cloud resources. If the local template library does not cover the current sign language action or the matching degree is lower than the baseline, the system automatically calls a general offline language library package for supplementary recognition. For sign language actions that the general offline language library package cannot cover or have insufficient confidence, the system calls the cloud-based enhanced recognition path with the user's authorization. The cloud-based pre-trained model performs inference based on a massive sign language sample library of millions of samples and returns candidate results from the cloud. The device-cloud fusion decision module obtains device-side data (including candidate labels, local matching degree, template quality score, scene relevance, action completeness, and key point stability) and cloud data (including candidate labels, confidence, semantic similarity, sign language type, and scene classification results), performs a comprehensive judgment, and outputs the final recognition result. The system adopts a fixed matching baseline mechanism. The recognition results that exceed the baseline are displayed and announced by voice, while the results that are below the baseline are not output, thus avoiding the problem of misrecognition. In the continuous sign language recognition scenario, the system adopts a sliding window and attention integral fusion mechanism. The camera acquisition module continuously acquires continuous sign language sequences, the feature processing module performs frame processing on the sequence, the candidate recall and fine matching module generates candidate results for each sliding window and calculates the window attention weight, and the edge-cloud fusion decision module integrates the multi-window integral results to determine the semantic switching point or judge it as a continuous expression, and finally outputs the word or short sentence sequence. The operational logic of the user, scenario, and thesaurus management system: The system creates an independent account for each user, and the training data and usage data of different users are completely isolated. The system pre-sets multiple high-frequency usage scenarios (such as medical treatment, government affairs, travel, education, etc.), and each scenario is independently configured with exclusive sign language vocabulary and short phrases to form a scenario-specific exclusive library. After the user selects a specific scenario, the system prioritizes candidate recall and matching within the vocabulary range corresponding to that scenario, narrowing the matching range and improving matching efficiency. Privacy protection and the operating mechanism of the data ecosystem: The system does not store or upload the user's original sign language video by default. The original video frames acquired by the camera acquisition module are only used for real-time key point extraction and calculation. After the calculation is completed, they are not saved to disk. The user only retains the desensitized action feature data, which is stored in the local personal template library as a personal template. The system has three independent and configurable privacy permissions: The first is cloud backup permission, which, when enabled, allows users' anonymized feature data to be backed up to the cloud server for feature recovery when users change devices. Backup data from different users are isolated from each other. The second is feature contribution permission, which, when enabled, allows users' anonymized feature data to be scored by the quality analysis module. High-quality samples selected are used to enrich the public sign language feature library and the dialect sign language feature library. The third is cloud enhancement permission, which, when enabled, allows the system to call cloud-pre-trained models for enhanced recognition when local recognition is insufficient. These three permissions are independent of each other, and users can enable or disable them individually without interference. The cloud-based quality analysis module performs quality scoring, regional classification, scene clustering, and semantic analysis on the de-identified features of user-authorized contributions. It selects high-quality samples to iteratively optimize the offline lexicon package and the cloud-based pre-trained model. After multiple rounds of iteration, the optimized offline lexicon package is packaged and distributed to the user end through the version management module and the backend management module, realizing a lexicon ecosystem with user participation and continuous iteration. Furthermore, it enhances the adaptability to individual sign language differences. Traditional sign language recognition systems require users to strictly follow standard sign language movements, and lack adaptation methods for personalized differences brought about by different users' age, region, hand shape, body shape, and long-term usage habits. This invention, through the on-device personal template training mechanism, allows users to train commonly used words and phrases independently. The system adaptively adjusts the feature weights of different sign language semantic types, enabling the system to adapt to the unique sign language expression of individuals, without requiring users to change their own habits to adapt to the system standard. Furthermore, the coverage of dialect sign language and single-hand sign language has been broadened. Existing technologies use the national standard sign language as the core training sample, which is insufficient in recognizing dialect sign language, ethnic minority sign language, and single-hand sign language. This invention incorporates non-standard sign language samples such as local dialect sign language, ethnic minority sign language, single-hand sign language, and elderly user sign language during the sample construction stage of the cloud pre-trained model, and distributes them to the user terminal through a general offline language library package, so that the system can recognize a variety of sign language expressions that cannot be adapted by existing technologies in actual use. Furthermore, this invention addresses the issues of cloud-based recognition's dependence on the network and privacy protection. Existing cloud-based recognition solutions require uploading raw video data, which can lead to recognition delays or failures when the network is unstable. Uploading raw video data also poses a risk of privacy leakage. This invention adopts a device-side priority recognition strategy, allowing a large number of sign language actions to be recognized and output locally without relying on a network connection. For scenarios that require cloud-based enhanced recognition, the transmitted content consists only of de-identified key points and time-compressed features. The raw video data never leaves the user's device, thus controlling privacy risks from the data collection source. Furthermore, this invention reduces the computational requirements of edge recognition and improves recognition stability. Deploying large recognition models directly on mobile terminals can lead to problems such as overheating, lag, and rapid power consumption. Meanwhile, lightweight and simple template solutions lack recognition accuracy and stability. This invention reduces edge computing overhead by narrowing the candidate matching range through scene partitioning management, screening a limited number of candidate templates for fine matching through a candidate recall mechanism, and filtering low-confidence results through a fixed baseline mechanism. At the same time, it employs multiple preprocessing methods such as normalization, filtering, outlier removal, and occlusion compensation to ensure the stability of feature data, enabling edge recognition to maintain usable accuracy under low power consumption conditions. Furthermore, the accuracy of continuous sign language recognition has been improved. Daily sign language communication often consists of long, continuous sentences with no obvious pauses between words. Existing technologies struggle to accurately determine action boundaries. This invention employs a sliding window and attention integral fusion mechanism to divide continuous sign language sequences into overlapping windows and perform multi-window integral fusion. By determining semantic switching points through attention weights, the accuracy of word boundary determination in continuous expressions has been improved, reducing the probability of missed segmentation and misidentification. Furthermore, the effectiveness and ease of use of small sample training have been optimized. In existing personalized solutions, users are required to record a large number of samples, which is cumbersome, while collecting only a small number of samples results in insufficient learning. This invention introduces a data augmentation strategy into the training process to generate augmented templates from limited original samples to expand the template set. At the same time, a template quality evaluation module is set to score the quality of the generated templates. If the quality is insufficient, the user is prompted to supplement the recording or retrain. The local template library is continuously optimized through an outlier template elimination mechanism, which improves the training effect and the stability of the template library under small sample conditions. Furthermore, a complementary mechanism for end-to-end cloud collaboration has been established. Existing technology systems are singular and have not built a fusion architecture of "local personalization and cloud-based public language database supplementation". This invention adopts a strategy of prioritizing the end side, supplementing the cloud side, and dynamically integrating. The end side focuses on personalized adaptation and offline availability, while the cloud side focuses on general semantic coverage and large-scale sample support. The two are comprehensively judged through the end-to-cloud fusion decision module, which takes into account both personalized recognition accuracy and general sign language coverage. At the same time, the continuous iterative optimization of the public language database is achieved through a user-authorized feature contribution mechanism.

[0043] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A multimodal sign language recognition system that integrates edge-side personalization and cloud-based pre-training, comprising a user terminal and a server terminal, characterized in that: The user terminal includes a camera acquisition module, a hand key point extraction module, an upper limb, torso, and facial key point extraction module, a feature processing module, an action start and stop detection module, a training module, a template quality evaluation module, a scene lexicon management module, a candidate recall and fine matching module, an end-to-cloud fusion decision module, a voice broadcast and two-way communication module, and a data backup and privacy authorization module. The server side includes a user account management module, a data backup module, a public sign language library, a dialect sign language library, a cloud-based pre-trained model training module, a quality analysis module, a backend management module, and a version management module. The client and server exchange data via an encrypted communication protocol.

2. The multimodal sign language recognition system that integrates edge-side personalization and cloud-based pre-training according to claim 1, characterized in that: The output of the camera acquisition module is connected to the input of the hand key point extraction module and the upper limb, torso and facial key point extraction module. The outputs of the two key point extraction modules are respectively connected to the input of the feature processing module; The output of the feature processing module is divided into two paths: one path is connected to the input of the training module, and the other path is connected to the input of the action start and stop detection module.

3. The multimodal sign language recognition system that integrates edge-side personalization and cloud-based pre-training according to claim 1, characterized in that: The output of the training module is connected to the input of the template quality evaluation module, and the output of the template quality evaluation module is connected to the local personal template library.

4. The multimodal sign language recognition system that integrates edge-side personalization and cloud-based pre-training according to claim 1, characterized in that: The output of the action start and stop detection module is connected to the input of the candidate recall and fine matching module. The candidate recall and fine matching module reads data from the local personal template library and the general offline language library package respectively for matching calculation, and its output is connected to the input of the end-to-cloud fusion decision module. The output of the edge-cloud converged decision module is connected to the input of the voice broadcast and two-way communication module.

5. The multimodal sign language recognition system that integrates edge-side personalization and cloud-based pre-training according to claim 1, characterized in that: The data backup and privacy authorization module interacts with the training module and the edge-cloud fusion decision module, respectively. The scenario-based vocabulary management module interacts with the local personal template library and the general offline vocabulary package to organize and manage scenario-based vocabulary.

6. The multimodal sign language recognition system that integrates edge-side personalization and cloud-based pre-training according to claim 1, characterized in that: The user account management module interacts with the data backup module, and the data backup module interacts with the user's data backup and privacy authorization module and the quality analysis module, respectively.

7. The multimodal sign language recognition system that integrates edge-side personalization and cloud-based pre-training according to claim 1, characterized in that: The outputs of the public sign language library and the dialect sign language library are connected to the input of the cloud-based pre-trained model training module.

8. The multimodal sign language recognition system that integrates edge-side personalization and cloud-based pre-training according to claim 1, characterized in that: The output of the cloud-based pre-trained model training module is connected to the input of the version management module, and the output of the version management module is connected to the lexicon generation module (which is located inside the backend management module). The output of the lexicon generation module sends a general offline lexicon package to the user.

9. A multimodal sign language recognition system that integrates edge-side personalization and cloud-based pre-training according to claim 1, characterized in that: The output of the quality analysis module is connected to the input of the public sign language library and the dialect sign language library.

10. A multimodal sign language recognition system that integrates edge-side personalization and cloud-based pre-training according to claim 1, characterized in that: The backend management module interacts with the public sign language library, the dialect sign language library, the quality analysis module, and the version management module.

Images