A real-time streaming translation method for continuous sign language vocabulary for sign language translation machines

Abstract

The application discloses a continuous sign language vocabulary real-time streaming translation method for a sign language translation machine, relates to the technical field of sign language translation, and adopts a streaming acquisition and piece-by-piece inference output architecture, so that the translation output delay can be shortened, thereby meeting the needs of daily real-time communication between hearing-impaired people and hearing people. Through multi-modal data joint analysis combined with a sliding window streaming segmentation mechanism, the continuity and simultaneity of sign language expression can be effectively adapted, more accurate vocabulary-level boundary segmentation can be realized, semantic crosstalk between vocabularies is greatly reduced, and the occurrence of translation errors is reduced. Through the sliding window, the context information of continuous sign language is retained, meanwhile, sign language grammar-related features such as gesture space actions and facial expressions are synchronously extracted, a pre-trained translation language model is combined to complete semantic reasoning and word order adjustment, the problems of semantic ambiguity and word order confusion caused by isolated vocabulary translation are effectively solved, and the accuracy and readability of the translation result are improved.

Description

Technical Field

[0001] This invention relates to the field of sign language translation technology, specifically a method for real-time streaming translation of continuous sign language vocabulary using a sign language translator. Background Technology

[0002] Sign language interpretation refers to the professional service of two-way language interpretation between hearing-impaired and hearing people, and also refers to the professionals who perform this work.

[0003] Traditional methods in existing technologies employ a batch processing approach of recognizing and translating entire sentences, resulting in a significant delay (typically 2-5 seconds) between sign language input and text / speech output, failing to meet the demands of real-time communication. Sign language expressions are characterized by continuity and simultaneity, with no clear boundaries between words, making precise word-level segmentation difficult with existing technologies, leading to semantic crosstalk and translation errors. Isolated word translation ignores contextual information such as gesture space utilization and facial expression linkage in sign language grammar, causing semantic ambiguity and word order confusion. Therefore, this invention provides a real-time streaming translation method for continuous sign language vocabulary in a sign language translator. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method for real-time streaming translation of continuous sign language vocabulary for sign language translators, thereby solving the aforementioned problems.

[0005] To achieve the above objectives, the present invention provides a method for real-time streaming translation of continuous sign language vocabulary using a sign language translator, comprising the following steps: Step S1: Stream sign language data, including video, inertial measurement and electromyography signals, using multimodal sensors, and perform time-stamp synchronization preprocessing. Step S2: Extract multi-dimensional differential features from the synchronized preprocessed multimodal sign language data; Step S3: Use a lightweight multimodal fusion network to identify the segmented word fragments; Step S4: Input the fused encoded features into the pre-trained translation language model, and output the text translation results of the corresponding continuous sign language words to achieve real-time streaming translation output of continuous sign language; Step S5: Optimize model performance and power consumption through edge-cloud collaborative architecture.

[0006] Preferably, the multimodal sensor in step S1 includes: A dual-camera array is used to capture visual appearance features of the hand contour, movement trajectory, and facial expression, respectively. An inertial measurement unit, worn on the wrist, is used to collect inertial motion parameters such as acceleration, angular velocity, and spatial attitude angle during hand movements. A multi-channel electromyography (EMG) sensor is attached to the corresponding muscle group on the forearm to collect electrical activity signals of the arm muscles during sign language movements.

[0007] Preferably, step S1 specifically includes the following steps: S11. Perform preliminary cleaning processing on the raw data of each modality: For the video stream acquired by the dual-camera array, screen and remove blurry and abnormal frames caused by hand occlusion and rapid movement frame by frame, and at the same time complete image normalization and hand area localization and segmentation to unify the size and subject position of different frames; For the inertial data acquired by the inertial measurement unit, use Kalman filtering to filter out random noise caused by wearing offset and environmental vibration, and correct the zero-point offset error of the sensor; For the electrical signals acquired by the multi-channel electromyography sensor, use 50Hz notch filtering to remove power frequency interference, and then use bandpass filtering to retain the effective signal components related to muscle activity and remove baseline drift noise; S12. Using the output clock of the inertial measurement unit as a unified reference, timestamps are matched to the cleaned three types of modal data. The three types of data within the same time window are aligned and integrated to form a continuous multimodal sign language data stream with a unified time stamp, thus completing the overall preprocessing process.

[0008] Preferably, step S2 specifically includes the following steps: S21. For the preprocessed visual video data, static appearance features of hand contour and gesture shape are extracted based on a lightweight convolutional neural network. The hand displacement between adjacent frames is calculated by combining optical flow method to extract dynamic trajectory features of sign language actions. At the same time, semantic features of facial expression region are extracted separately. The three types of features are spliced ​​and fused to obtain a complete visual modality feature vector. S22. For the preprocessed inertial measurement data, a fixed-length sliding window is used to segment the continuous inertial sequence. In each window, the time-domain statistical features such as signal mean, variance, and peak value are calculated. Then, the time-domain signal is converted to the frequency domain by fast Fourier transform, and the energy proportion features of different frequency bands are extracted. The two types of features are spliced ​​together to obtain the inertial mode feature vector. S23. For the preprocessed electromyographic signals, segment them with sliding windows of the same length, calculate the average absolute value, root mean square, zero-crossing rate and other characteristic parameters reflecting muscle activation intensity for each channel signal, and integrate the characteristic parameters of all channels to obtain the physiological feature vector of the electromyographic mode, thus completing the entire extraction process of multi-dimensional differential features.

[0009] Preferably, the lightweight multimodal fusion network in step S3 includes: The video coding branch uses MobileNetV3-SSD to extract visual features; The IMU encoding branch uses 1D-CNN+BiGRU to extract motion features; The sEMG encoding branch uses an attention-enhanced 1D-CNN to extract physiological features; An adaptive gating fusion mechanism dynamically adjusts the contribution weights of each modality feature.

[0010] Preferably, step S3 specifically includes the following steps: S31. Input the three types of modal features extracted in step S2 into the corresponding encoding branches to complete feature encoding: Input the visual modal feature vector into the video encoding branch, perform deep semantic extraction and dimensional compression of the visual features through MobileNetV3-SSD, and output the simplified visual high-level features; Input the inertial modal feature vector into the IMU encoding branch, extract the local correlation information of the inertial features through one-dimensional convolution, and then input it into the bidirectional gated recurrent unit to capture the temporal dependency relationship of the sign language action sequence, and output the motion high-level features containing complete temporal information; Input the electromyography modal feature vector into the sEMG encoding branch, extract the local electromyography features through one-dimensional convolution, and then introduce a channel attention mechanism to assign differentiated weights to the features of different channels, highlight the feature contribution of channels that are highly related to the current sign language action, and output the enhanced physiological high-level features; S32. The three types of high-level features output from the three coding branches are concatenated and input into the adaptive gating fusion mechanism. Dynamic weight coefficients corresponding to the three modalities are generated through the fully connected layer. The weight coefficients are multiplied with the corresponding modal features to complete the weighted fusion. Finally, a unified fusion feature integrating visual appearance, motion trajectory and physiological activity is obtained to complete the recognition output of the current sign language vocabulary segment.

[0011] Preferably, step S4 specifically includes the following steps: S41. The fusion features output segment by segment in step S3 are spliced ​​into a complete continuous sign language feature sequence according to the timestamp order. A fixed-length sliding window is used to prune the feature sequence. Each time a new word fusion feature is input, the window is slid forward once to remove historical features that exceed the window length. This controls the computational amount of the model in a single inference and ensures the real-time requirements of streaming translation. S42. Input the complete feature sequence updated by the sliding window into the end-to-end translation language model that has been pre-trained on a large-scale parallel sign language-text corpus. The model outputs the probability distribution of the candidate words for each sign language segment. The word sequence with the highest overall probability is selected by beam search. Then, the word order of the output sequence is adjusted by combining general sign language grammar rules to eliminate the semantic deviation caused by the word order of sign language expression and obtain a continuous text result that conforms to the target natural language norm. S43. The generated translation text results are pushed word by word to the output terminal of the sign language translator according to the streaming output rules to complete the translation output of this continuous sign language segment. At the same time, the data collection and reasoning process is kept uninterrupted, waiting for the continuous input and processing of subsequent sign language action data.

[0012] Preferably, the edge-cloud collaborative architecture described in step S5 includes: Lightweight models are run at the edge to process high-frequency words; Large models run in the cloud to process uncommon words and identify low-confidence words; A model update mechanism based on federated learning.

[0013] A sign language translator includes: The multimodal data acquisition module is used to acquire sign language multimodal data in real time, including gesture visual data, hand movement inertial data and arm muscle electromyography signals. After performing preliminary cleaning and timestamp alignment and synchronization preprocessing on the raw data of each modality, it outputs a continuous multimodal sign language data stream with a unified time stamp. A multi-dimensional differential feature extraction module, connected to the multi-modal data acquisition module, is used to extract visual modal features, inertial modal features and electromyographic modal features from the preprocessed multi-modal sign language data, respectively, to obtain an integrated multi-dimensional differential feature set; The multimodal fusion recognition module, connected to the multidimensional differential feature extraction module, is used to input the three types of modal features into the corresponding coding branches to complete deep feature encoding, and then dynamically weight them through an adaptive gating fusion mechanism to obtain a unified fusion feature that integrates multi-source information, thereby completing the recognition output of sign language vocabulary fragments. The streaming translation inference module, connected to the multimodal fusion recognition module, is used to organize the fusion features output segment by segment into a continuous feature sequence according to the timestamp order. The inference computation is controlled by a sliding window. After inputting into the pre-trained translation language model, it outputs a continuous text translation result that conforms to the target natural language standard. The edge-cloud collaborative scheduling module is connected to the multimodal data acquisition module, the multidimensional differential feature extraction module, the multimodal fusion recognition module, and the streaming translation inference module, respectively. It is used to dynamically schedule the computing resources of the edge and the cloud. The edge handles the inference of high-frequency and commonly used sign language words, while the cloud handles the recognition of rare words and low confidence. Combined with the federated learning mechanism, it completes the model iteration update, balances translation performance and device power consumption, and realizes real-time streaming translation output of continuous sign language. The output module, connected to the streaming translation inference module, is used to display the translated text or play audio.

[0014] Beneficial effects Compared with the prior art, the present invention has the following advantages: This invention adopts a streaming acquisition and segment-by-segment reasoning output architecture, which can effectively solve the inherent delay of the traditional whole sentence batch processing mode, shorten the translation output delay, and meet the daily real-time communication needs of hearing-impaired people and hearing people. By combining multimodal data joint analysis with a sliding window streaming segmentation mechanism, it is possible to effectively adapt to the characteristics of continuity and simultaneity in sign language expression, achieve more accurate lexical-level boundary segmentation, significantly reduce semantic crosstalk between words, and reduce the occurrence of translation errors. By retaining the contextual information of continuous sign language through a sliding window, and simultaneously extracting features related to sign language grammar such as gesture spatial movement and facial expressions, and combining them with a pre-trained translation language model to complete semantic reasoning and word order adjustment, the problem of semantic ambiguity and word order disorder caused by isolated word translation is effectively solved, and the accuracy and readability of the translation results are improved. It integrates three types of differentiated features: visual appearance, inertial motion, and electromyography. Compared with single visual recognition solutions, it can maintain recognition accuracy in complex scenarios such as rapid hand movements and partial occlusion, and is more robust. By scheduling computing resources through an edge-cloud collaborative architecture, low latency for translating common words is ensured through lightweight processing at the edge, while the translation accuracy for uncommon words and low-confidence scenarios is improved by leveraging large models in the cloud. At the same time, device power consumption is balanced, adapting to the usage needs of portable sign language translators. The model update mechanism based on federated learning can also continuously iterate and optimize model performance while protecting user data privacy. Attached Figure Description

[0015] Figure 1 This is a flowchart of the process of this invention; Figure 2 This is a flowchart illustrating the workflow of multimodal sensor streaming acquisition of sign language data in this invention. Figure 3 This is a flowchart illustrating the workflow for extracting multi-dimensional differential features from synchronously preprocessed multimodal sign language data in this invention. Figure 4 This is a flowchart illustrating the workflow of using a lightweight multimodal fusion network to identify segmented word fragments in this invention. Figure 5 This is a flowchart illustrating the workflow of the present invention, which inputs the fused and encoded features into a pre-trained translation language model and outputs the text translation results corresponding to continuous sign language words, thereby realizing the real-time streaming translation output of continuous sign language. Figure 6 This is a module connection diagram of the sign language translator in this invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Please see Figure 1-5 This invention provides a method for real-time streaming translation of continuous sign language vocabulary using a sign language translator, comprising the following steps: Step S1: Stream sign language data, including video, inertial measurement, and electromyography signals, using multimodal sensors, and perform time-stamped synchronization preprocessing. The multimodal sensors include: A dual-camera array is used to capture visual appearance features of the hand contour, movement trajectory, and facial expression, respectively. An inertial measurement unit, worn on the wrist, is used to collect inertial motion parameters such as acceleration, angular velocity, and spatial attitude angle during hand movements. A multi-channel electromyography (EMG) sensor is attached to the corresponding muscle group on the forearm to collect electrical activity signals of the arm muscles during sign language movements.

[0018] Specifically, the steps include the following: S11. Perform preliminary cleaning processing on the raw data of each modality: For the video stream acquired by the dual-camera array, screen and remove blurry and abnormal frames caused by hand occlusion and rapid movement frame by frame, and at the same time complete image normalization and hand area localization and segmentation to unify the size and subject position of different frames; For the inertial data acquired by the inertial measurement unit, use Kalman filtering to filter out random noise caused by wearing offset and environmental vibration, and correct the zero-point offset error of the sensor; For the electrical signals acquired by the multi-channel electromyography sensor, use 50Hz notch filtering to remove power frequency interference, and then use bandpass filtering to retain the effective signal components related to muscle activity and remove baseline drift noise; S12. Using the output clock of the inertial measurement unit as a unified reference, timestamps are matched to the cleaned three types of modal data. The three types of data within the same time window are aligned and integrated to form a continuous multimodal sign language data stream with a unified time stamp, thus completing the overall preprocessing process.

[0019] The above steps enable the cleaning, correction, and temporal alignment of multimodal sign language raw data. This eliminates the impact of noise interference and temporal misalignment on data quality, fully preserving the three types of information contained in sign language movements: visual appearance, motion state, and physiological activation, thus ensuring the input quality of subsequent data processing.

[0020] Step S2: Extract multi-dimensional differential features from the synchronized preprocessed multimodal sign language data; Specifically, the steps include the following: S21. For the preprocessed visual video data, static appearance features of hand contour and gesture shape are extracted based on a lightweight convolutional neural network. The hand displacement between adjacent frames is calculated by combining optical flow method to extract dynamic trajectory features of sign language actions. At the same time, semantic features of facial expression region are extracted separately. The three types of features are spliced ​​and fused to obtain a complete visual modality feature vector. S22. For the preprocessed inertial measurement data, a fixed-length sliding window is used to segment the continuous inertial sequence. In each window, the time-domain statistical features such as signal mean, variance, and peak value are calculated. Then, the time-domain signal is converted to the frequency domain by fast Fourier transform, and the energy proportion features of different frequency bands are extracted. The two types of features are spliced ​​together to obtain the inertial mode feature vector. S23. For the preprocessed electromyographic signals, segment them with sliding windows of the same length, calculate the average absolute value, root mean square, zero-crossing rate and other characteristic parameters reflecting muscle activation intensity for each channel signal, and integrate the characteristic parameters of all channels to obtain the physiological feature vector of the electromyographic mode, thus completing the entire extraction process of multi-dimensional differential features.

[0021] The above steps enable targeted processing of visual video data, inertial measurement data, and electromyographic signals, ensuring both the comprehensiveness and accuracy of data acquisition.

[0022] Step S3: Use a lightweight multimodal fusion network to identify the segmented word fragments; Lightweight multimodal fusion networks include: The video coding branch uses MobileNetV3-SSD to extract visual features; The IMU encoding branch uses 1D-CNN+BiGRU to extract motion features; The sEMG encoding branch uses an attention-enhanced 1D-CNN to extract physiological features; An adaptive gating fusion mechanism dynamically adjusts the contribution weights of each modality feature.

[0023] Specifically, the steps include the following: S31. Input the three types of modal features extracted in step S2 into the corresponding encoding branches to complete feature encoding: Input the visual modal feature vector into the video encoding branch, perform deep semantic extraction and dimensional compression of the visual features through MobileNetV3-SSD, and output the simplified visual high-level features; Input the inertial modal feature vector into the IMU encoding branch, extract the local correlation information of the inertial features through one-dimensional convolution, and then input it into the bidirectional gated recurrent unit to capture the temporal dependency relationship of the sign language action sequence, and output the motion high-level features containing complete temporal information; Input the electromyography modal feature vector into the sEMG encoding branch, extract the local electromyography features through one-dimensional convolution, and then introduce a channel attention mechanism to assign differentiated weights to the features of different channels, highlight the feature contribution of channels that are highly related to the current sign language action, and output the enhanced physiological high-level features; S32. The three types of high-level features output from the three coding branches are concatenated and input into the adaptive gating fusion mechanism. Dynamic weight coefficients corresponding to the three modalities are generated through the fully connected layer. The weight coefficients are multiplied with the corresponding modal features to complete the weighted fusion. Finally, a unified fusion feature integrating visual appearance, motion trajectory and physiological activity is obtained to complete the recognition output of the current sign language vocabulary segment.

[0024] The above steps enable the fusion of multimodal features, thereby ensuring the accuracy of recognition and, consequently, the precision of translation.

[0025] Step S4: Input the fused encoded features into the pre-trained translation language model, and output the text translation results of the corresponding continuous sign language words to achieve real-time streaming translation output of continuous sign language; Specifically, the steps include the following: S41. The fusion features output segment by segment in step S3 are spliced ​​into a complete continuous sign language feature sequence according to the timestamp order. A fixed-length sliding window is used to prune the feature sequence. Each time a new word fusion feature is input, the window is slid forward once to remove historical features that exceed the window length. This controls the computational amount of the model in a single inference and ensures the real-time requirements of streaming translation. S42. Input the complete feature sequence updated by the sliding window into the end-to-end translation language model that has been pre-trained on a large-scale parallel sign language-text corpus. The model outputs the probability distribution of the candidate words for each sign language segment. The word sequence with the highest overall probability is selected by beam search. Then, the word order of the output sequence is adjusted by combining general sign language grammar rules to eliminate the semantic deviation caused by the word order of sign language expression and obtain a continuous text result that conforms to the target natural language norm. S43. The generated translation text results are pushed word by word to the output terminal of the sign language translator according to the streaming output rules to complete the translation output of this continuous sign language segment. At the same time, the data collection and reasoning process is kept uninterrupted, waiting for the continuous input and processing of subsequent sign language action data.

[0026] The above steps enable streaming incremental processing and word-by-word output of continuous sign language vocabulary, eliminating the need to wait for the entire sign language action to be completed before starting translation inference. The sliding window mechanism controls the computational load of a single inference by the model, adapting to the embedded hardware computing power limitations of portable sign language translators and significantly reducing translation output latency, thus meeting the daily real-time communication needs of hearing-impaired individuals. At the same time, by combining general sign language grammar rules to correct the output word order, the differences in word order between sign language expression habits and natural language are effectively eliminated, significantly improving the semantic accuracy and fluency of the translation results.

[0027] Step S5: Optimize model performance and power consumption through an edge-cloud collaborative architecture. This architecture specifically includes: running a lightweight model at the edge to process high-frequency words; running a large model in the cloud to process uncommon words and low-confidence recognition; and a model update mechanism based on federated learning. In other words, the edge-cloud collaborative architecture fully utilizes the local computing power of edge devices to achieve low-power, low-latency translation, while relying on the capabilities of the large model in the cloud to ensure translation accuracy for uncommon words and low-confidence scenarios, thus achieving a balance between performance, power consumption, and accuracy.

[0028] Please see Figure 6 A sign language translator includes: The multimodal data acquisition module is used to acquire sign language multimodal data in real time, including gesture visual data, hand movement inertial data and arm muscle electromyography signals. After performing preliminary cleaning and timestamp alignment and synchronization preprocessing on the raw data of each modality, it outputs a continuous multimodal sign language data stream with a unified time stamp. The multi-dimensional differential feature extraction module is connected to the multi-modal data acquisition module. It is used to extract visual modal features, inertial modal features and electromyographic modal features from the preprocessed multi-modal sign language data to obtain an integrated multi-dimensional differential feature set. The multimodal fusion recognition module, connected to the multidimensional differential feature extraction module, is used to input the three types of modal features into the corresponding coding branches to complete deep feature encoding. Then, through an adaptive gating fusion mechanism, the features are dynamically weighted to obtain a unified fusion feature that integrates multi-source information, thereby completing the recognition output of sign language vocabulary fragments. The streaming translation inference module, connected to the multimodal fusion recognition module, is used to organize the fusion features output segment by segment into a continuous feature sequence according to the timestamp order. The inference computation is controlled by a sliding window. After inputting into the pre-trained translation language model, it outputs a continuous text translation result that conforms to the target natural language standard. The edge-cloud collaborative scheduling module is connected to the multimodal data acquisition module, the multidimensional differential feature extraction module, the multimodal fusion recognition module, and the streaming translation inference module, respectively. It is used to dynamically schedule the computing resources of the edge and the cloud. The edge handles the inference of high-frequency and commonly used sign language words, while the cloud handles the rare words and low confidence recognition tasks. Combined with the federated learning mechanism, it completes the model iteration and update, balances translation performance and device power consumption, and realizes real-time streaming translation output of continuous sign language. The output module, connected to the streaming translation inference module, is used to display the translated text or play audio.

[0029] Working principle: The multimodal data acquisition module acquires input signals: it captures visual images of hand gestures through a camera, acquires inertial data by collecting the acceleration and angular velocity of the hand and arm through a built-in inertial measurement unit, and acquires electromyographic activity signals of the arm muscle groups through a surface electromyography sensor. After the three types of raw data are cleaned by noise reduction and format normalization, they are aligned with timestamps according to a unified time base and output a synchronous continuous multimodal sign language data stream, which is then transmitted to the multidimensional differential feature extraction module. The multi-dimensional differentiated feature extraction module designs appropriate feature extraction schemes for the different data characteristics of the three modalities: for the visual modality, it extracts appearance features such as gesture contours, finger shape changes, and spatial position; for the inertial modality, it extracts action temporal motion trajectory features; and for the electromyographic modality, it extracts action force pattern features. Finally, it obtains a multi-dimensional differentiated feature set that is non-redundant and retains the unique information of each modality, which is then input into the multi-modal fusion recognition module. The multimodal fusion recognition module assigns independent deep coding branches to the three types of features to complete the deep abstract coding of each modality feature. Then, through an adaptive gating fusion mechanism, the fusion weights are dynamically adjusted according to the signal-to-noise ratio and confidence of each modality data. If the quality of a certain modality data is reduced due to interference, its weight is automatically reduced and the contribution ratio of the other modalities is increased. Finally, the robust unified fusion feature integrating multi-source information is obtained through dynamic weighting, which completes the recognition output of single-segment sign language vocabulary and is passed to the streaming translation inference module. The streaming translation inference module does not need to wait for all sign language actions to be completed before making overall inference. Instead, it splices the fused features output segment by segment into a continuous feature sequence according to the timestamp order. It uses a fixed-size sliding window to frame the feature segment to be inferred, controls the amount of computation for each inference to ensure the inference speed, and then inputs the features within the window into a pre-trained translation language model. It combines the context semantics to correct the single-segment recognition error and finally outputs a continuous text translation result that conforms to the grammatical norms of the target natural language. The edge-cloud collaborative scheduling module monitors the operating status and recognition results of each module throughout the process: for high-frequency daily sign language words, it directly schedules the local edge computing resources of the translator to complete the inference, shortening the response latency and reducing the power consumption of the device; for recognition results with low confidence, rare words or professional sign language, it schedules a large model with stronger computing power in the cloud to complete the recognition inference. At the same time, relying on the federated learning mechanism, it only transmits the model gradient without leaking the user's original privacy data, and works with multiple devices to complete the iterative update of the model, continuously improving the recognition accuracy.

[0030] The final translation result is transmitted to the output module, which can display the translated text as needed, or play it as natural speech through speech synthesis, completing the entire process of real-time streaming translation of continuous sign language.

[0031] Furthermore, any content not described in detail in this specification is existing technology known to those skilled in the art.

[0032] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0033] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for real-time streaming translation of continuous sign language vocabulary using a sign language translator, characterized in that, Includes the following steps: Step S1: Stream sign language data, including video, inertial measurement and electromyography signals, using multimodal sensors, and perform time-stamp synchronization preprocessing. Step S2: Extract multi-dimensional differential features from the synchronized preprocessed multimodal sign language data; Step S3: Use a lightweight multimodal fusion network to identify the segmented word fragments; Step S4: Input the fused encoded features into the pre-trained translation language model, and output the text translation results of the corresponding continuous sign language words to achieve real-time streaming translation output of continuous sign language; Step S5: Optimize model performance and power consumption through edge-cloud collaborative architecture.

2. The method for real-time streaming translation of continuous sign language vocabulary using a sign language translator according to claim 1, characterized in that, The multimodal sensor mentioned in step S1 includes: A dual-camera array is used to capture visual appearance features of the hand contour, movement trajectory, and facial expression, respectively. An inertial measurement unit, worn on the wrist, is used to collect inertial motion parameters such as acceleration, angular velocity, and spatial attitude angle during hand movements. A multi-channel electromyography (EMG) sensor is attached to the corresponding muscle group on the forearm to collect electrical activity signals of the arm muscles during sign language movements.

3. The method for real-time streaming translation of continuous sign language vocabulary using a sign language translator according to claim 2, characterized in that, Step S1 specifically includes the following steps: S11. Perform preliminary cleaning processing on the raw data of each modality: For the video stream acquired by the dual-camera array, screen and remove blurry and abnormal frames caused by hand occlusion and rapid movement frame by frame, and at the same time complete image normalization and hand area localization and segmentation to unify the size and subject position of different frames; For the inertial data acquired by the inertial measurement unit, use Kalman filtering to filter out random noise caused by wearing offset and environmental vibration, and correct the zero-point offset error of the sensor; For the electrical signals acquired by the multi-channel electromyography sensor, remove power frequency interference by 50Hz notch filtering, and then retain the effective signal components related to muscle activity by bandpass filtering, and remove baseline drift noise; S12. Using the output clock of the inertial measurement unit as a unified reference, timestamps are matched to the cleaned three types of modal data. The three types of data within the same time window are aligned and integrated to form a continuous multimodal sign language data stream with a unified time stamp, thus completing the overall preprocessing process.

4. The method for real-time streaming translation of continuous sign language vocabulary using a sign language translator according to claim 1, characterized in that, Step S2 specifically includes the following steps: S21. For the preprocessed visual video data, static appearance features of hand contour and gesture shape are extracted based on a lightweight convolutional neural network. The hand displacement between adjacent frames is calculated by combining optical flow method to extract dynamic trajectory features of sign language actions. At the same time, semantic features of facial expression region are extracted separately. The three types of features are spliced ​​and fused to obtain a complete visual modality feature vector. S22. For the preprocessed inertial measurement data, a fixed-length sliding window is used to segment the continuous inertial sequence. In each window, the time-domain statistical features such as signal mean, variance, and peak value are calculated. Then, the time-domain signal is converted to the frequency domain by fast Fourier transform, and the energy proportion features of different frequency bands are extracted. The two types of features are spliced ​​together to obtain the inertial mode feature vector. S23. For the preprocessed electromyographic signals, segment them with sliding windows of the same length, calculate the average absolute value, root mean square, zero-crossing rate and other characteristic parameters reflecting muscle activation intensity for each channel signal, and integrate the characteristic parameters of all channels to obtain the physiological feature vector of the electromyographic mode, thus completing the entire extraction process of multi-dimensional differential features.

5. A method for real-time streaming translation of continuous sign language vocabulary using a sign language translator according to claim 1, characterized in that, The lightweight multimodal fusion network mentioned in step S3 includes: The video coding branch uses MobileNetV3-SSD to extract visual features; The IMU encoding branch uses 1D-CNN+BiGRU to extract motion features; The sEMG encoding branch uses an attention-enhanced 1D-CNN to extract physiological features; An adaptive gating fusion mechanism dynamically adjusts the contribution weights of each modality feature.

6. A method for real-time streaming translation of continuous sign language vocabulary using a sign language translator according to claim 1, characterized in that, Step S3 specifically includes the following steps: S31. Input the three types of modal features extracted in step S2 into the corresponding encoding branches to complete feature encoding: Input the visual modal feature vector into the video encoding branch, perform deep semantic extraction and dimensional compression of the visual features through MobileNetV3-SSD, and output the simplified visual high-level features; Input the inertial modal feature vector into the IMU encoding branch, extract the local correlation information of the inertial features through one-dimensional convolution, and then input it into the bidirectional gated recurrent unit to capture the temporal dependency relationship of the sign language action sequence, and output the motion high-level features containing complete temporal information; Input the electromyography modal feature vector into the sEMG encoding branch, extract the local electromyography features through one-dimensional convolution, and then introduce a channel attention mechanism to assign differentiated weights to the features of different channels, highlight the feature contribution of channels that are highly related to the current sign language action, and output the enhanced physiological high-level features; S32. The three types of high-level features output from the three coding branches are concatenated and input into the adaptive gating fusion mechanism. Dynamic weight coefficients corresponding to the three modalities are generated through the fully connected layer. The weight coefficients are multiplied with the corresponding modal features to complete the weighted fusion. Finally, a unified fusion feature integrating visual appearance, motion trajectory and physiological activity is obtained to complete the recognition output of the current sign language vocabulary segment.

7. A method for real-time streaming translation of continuous sign language vocabulary using a sign language translator according to claim 1, characterized in that, Step S4 specifically includes the following steps: S41. The fusion features output segment by segment in step S3 are spliced ​​into a complete continuous sign language feature sequence according to the timestamp order. A fixed-length sliding window is used to prune the feature sequence. Each time a new word fusion feature is input, the window is slid forward once to remove historical features that exceed the window length. This controls the computational amount of the model in a single inference and ensures the real-time requirements of streaming translation. S42. Input the complete feature sequence updated by the sliding window into the end-to-end translation language model that has been pre-trained on a large-scale parallel sign language-text corpus. The model outputs the probability distribution of the candidate words for each sign language segment. The word sequence with the highest overall probability is selected by beam search. Then, the word order of the output sequence is adjusted by combining general sign language grammar rules to eliminate the semantic deviation caused by the word order of sign language expression and obtain a continuous text result that conforms to the target natural language norm. S43. The generated translation text results are pushed word by word to the output terminal of the sign language translator according to the streaming output rules to complete the translation output of this continuous sign language segment. At the same time, the data collection and reasoning process is kept uninterrupted, waiting for the continuous input and processing of subsequent sign language action data.

8. A method for real-time streaming translation of continuous sign language vocabulary using a sign language translator according to claim 1, characterized in that, The edge-cloud collaborative architecture described in step S5 includes: Lightweight models are run at the edge to process high-frequency words; Large models run in the cloud to process uncommon words and identify low-confidence words; A model update mechanism based on federated learning.

9. A sign language translator, characterized in that, include: The multimodal data acquisition module is used to acquire sign language multimodal data in real time, including gesture visual data, hand movement inertial data and arm muscle electromyography signals. After performing preliminary cleaning and timestamp alignment and synchronization preprocessing on the raw data of each modality, it outputs a continuous multimodal sign language data stream with a unified time stamp. A multi-dimensional differential feature extraction module, connected to the multi-modal data acquisition module, is used to extract visual modal features, inertial modal features and electromyographic modal features from the preprocessed multi-modal sign language data, respectively, to obtain an integrated multi-dimensional differential feature set; The multimodal fusion recognition module, connected to the multidimensional differential feature extraction module, is used to input the three types of modal features into the corresponding coding branches to complete deep feature encoding, and then dynamically weight them through an adaptive gating fusion mechanism to obtain a unified fusion feature that integrates multi-source information, thereby completing the recognition output of sign language vocabulary fragments. The streaming translation inference module, connected to the multimodal fusion recognition module, is used to organize the fusion features output segment by segment into a continuous feature sequence according to the timestamp order. The inference computation is controlled by a sliding window. After inputting into the pre-trained translation language model, it outputs a continuous text translation result that conforms to the target natural language standard. The edge-cloud collaborative scheduling module is connected to the multimodal data acquisition module, the multidimensional differential feature extraction module, the multimodal fusion recognition module, and the streaming translation inference module, respectively. It is used to dynamically schedule the computing resources of the edge and the cloud. The edge handles the inference of high-frequency and commonly used sign language words, while the cloud handles the recognition of rare words and low confidence. Combined with the federated learning mechanism, it completes the model iteration update, balances translation performance and device power consumption, and realizes real-time streaming translation output of continuous sign language. The output module, connected to the streaming translation inference module, is used to display the translated text or play audio.

Images