Sign language translation method and device, computer device and storage medium

Abstract

The application relates to a sign language translation method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring multi-modal sign language data to be processed; inputting the multi-modal sign language data into a preset pre-training neural network; extracting features of the multi-modal sign language data by using the pre-training neural network to obtain visual features and motion features; performing dynamic fusion processing on the visual features and the motion features based on a preset attention mechanism to obtain multi-modal fusion features; performing dynamic sequence modeling on the multi-modal fusion features to obtain a sign language feature sequence; and performing encoding and decoding processing on the sign language feature sequence to obtain target language text. The application can more comprehensively and intelligently understand the complex semantics of sign language and improve the accuracy of sign language translation.

Description

Technical Field

[0001] This application relates to the field of sign language translation technology, and in particular to a sign language translation method, apparatus, computer device and storage medium. Background Technology

[0002] Most existing sign language translation technologies rely on a single modality for recognition, with visual-based methods being the most common. These methods primarily use cameras to capture video frames of the signer, and then utilize computer vision technology to analyze and recognize gestures, body movements, and facial expressions to achieve translation output.

[0003] However, this single-modal recognition paradigm faces many challenges in practical applications. First, visual information is highly susceptible to interference from the external environment. For example, changes in lighting conditions (such as strong light, shadow, and backlight), deviations in shooting angles, or partial limb occlusion (such as a hand being blocked by the body or other objects) can all lead to the loss or distortion of key visual features, affecting the accuracy of recognition and consequently resulting in inaccurate translation results. Summary of the Invention

[0004] This application provides a sign language translation method, apparatus, computer device, and storage medium, aiming to solve the problem that traditional sign language translation algorithms rely on only a single modality of data for recognition, resulting in low recognition accuracy and inaccurate translation results.

[0005] In a first aspect, embodiments of this application provide a sign language translation method, the sign language translation method comprising:

[0006] Acquire multimodal sign language data to be processed, wherein the multimodal sign language data includes image data and sensor data;

[0007] The multimodal sign language data is input into a preset pre-trained neural network;

[0008] The pre-trained neural network is used to extract features from the multimodal sign language data to obtain visual and motion features;

[0009] The visual features and motion features are dynamically fused based on a preset attention mechanism to obtain multimodal fused features;

[0010] Dynamic sequence modeling is performed on the multimodal fusion features to obtain the sign language feature sequence;

[0011] The sign language feature sequence is encoded and decoded to obtain the target language text.

[0012] A further technical solution is that the multimodal sign language data includes image data and sensor data, and the pre-trained neural network is used to extract features from the multimodal sign language data to be processed to obtain visual features and motion features, including:

[0013] Extract key points from the image data;

[0014] The key points are normalized to obtain normalized key points;

[0015] The normalized key points are then subjected to feature transformation to obtain visual features.

[0016] A further technical solution is that the key points include hand and facial key points as well as body posture key points; the normalization process for the key points to obtain normalized key points includes:

[0017] Local normalization processing was performed on the key points of the hands and face;

[0018] And the key points of the body posture are globally normalized to obtain normalized key points.

[0019] A further technical solution is that, in addition to using the pre-trained neural network to extract features from the multimodal sign language data to be processed, to obtain visual and motion features, the method further includes:

[0020] The sensor data is denoised to obtain denoised sensor data.

[0021] The denoised sensor data and the image data are time-aligned and language-aligned to obtain aligned sensor data;

[0022] Feature extraction is performed on the aligned sensor data to obtain motion features.

[0023] A further technical solution is that the encoding and decoding of the sign language feature sequence to obtain the target language text includes:

[0024] The sign language feature sequence is input into a trained sequence-to-sequence model, wherein the trained sequence-to-sequence model has an encoder and a decoder;

[0025] The encoder is used to encode and compress the input sign language feature sequence to obtain a fixed-length context vector;

[0026] The context vector is input into the decoder, which uses an attention mechanism to decode the context vector to obtain the target language text.

[0027] A further technical solution is that the visual features and motion features are dynamically fused based on a preset attention mechanism to obtain multimodal fusion features, including:

[0028] The visual features and motion features are input into a preset multi-stream fusion module;

[0029] The multi-stream fusion module uses a preset gating mechanism to calculate the contribution values ​​corresponding to the visual features and the motion features, respectively.

[0030] The contribution values ​​corresponding to the visual features and the motion features are weighted and summed to obtain the multimodal fusion features.

[0031] A further technical solution is that the image data includes skeletal data, and the method further includes:

[0032] The skeletal data is input into a preset spatiotemporal joint attention module;

[0033] The spatiotemporal joint attention module is used to perform temporal global max pooling and spatial global max pooling on the skeletal data to obtain the corresponding first feature vector and second feature vector, respectively.

[0034] The first feature vector and the second feature vector are concatenated to obtain the concatenated feature vector;

[0035] The concatenated feature vector is then subjected to dimensionality reduction processing to obtain the dimensionality-reduced concatenated feature vector;

[0036] Based on the dimensionality-reduced concatenated feature vectors, a temporal attention map and a spatial attention map are generated.

[0037] The temporal attention map and the spatial attention map are multiplied by their channels to obtain a joint attention map.

[0038] The target bone data is obtained by performing element-wise multiplication based on the joint attention map and the bone data.

[0039] Secondly, embodiments of this application also provide a sign language translation device, which includes a unit for performing the above-described method.

[0040] Thirdly, embodiments of this application also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0041] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0042] This application provides a sign language translation method, apparatus, computer device, and storage medium. The method includes: acquiring multimodal sign language data to be processed; inputting the multimodal sign language data to be processed into a preset pre-trained neural network, using the pre-trained neural network to extract features from the multimodal sign language data to obtain visual features and motion features; dynamically fusing the visual features and motion features based on a preset attention mechanism to obtain multimodal fused features; dynamically modeling the multimodal fused features to obtain a sign language feature sequence; and encoding and decoding the sign language feature sequence to obtain target language text.

[0043] This application embodiment acquires multimodal sign language data to be processed, including image data and sensor data, thereby enabling multi-dimensional perception of sign language movements. Furthermore, visual and motion features can be extracted from different modal data. Visual features reflect gesture shape, texture, and facial expressions, while motion features depict the three-dimensional spatial trajectory, speed, and acceleration of the hand. This allows for the complete capture of key motion information even under adverse conditions such as changes in lighting, partial occlusion, or background interference.

[0044] In addition, a dynamic fusion strategy is adopted, which adaptively adjusts the fusion weights of visual features and motion features based on the current semantic content and context of the sign language segment. This dynamic weighting method of semantic perception can improve the flexibility of fusion, thereby enabling a more comprehensive and intelligent understanding of the complex semantics of sign language and improving the accuracy of sign language translation.

[0045] Furthermore, traditional translation systems often only recognize sign language frame by frame / word by word without performing contextual semantic modeling, lacking an understanding of the overall sentence structure and context, which easily leads to problems such as semantic loss and word order disorder. This application, however, uses dynamic sequence modeling of multimodal fusion features to capture the temporal dependencies between sign language actions, understand complex grammatical phenomena such as subject-verb-object structures, time adverbs, and interrogative moods, thereby improving the ability to understand complex sign language grammar and context, resulting in more accurate, fluent, and natural language expression in the final translation output. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0049] Figure 1 A flowchart illustrating a first embodiment of a sign language translation method provided in this application;

[0050] Figure 2 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0053] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0054] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0055] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0056] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0057] To address the aforementioned issues, this application provides a sign language translation method that can more comprehensively and intelligently understand the complex semantics of sign language and improve the accuracy of sign language translation.

[0058] See Figure 1 , Figure 1 This application provides a flowchart illustrating a first embodiment of a sign language translation method, which includes the following steps:

[0059] Step 110: Obtain the multimodal sign language data to be processed.

[0060] The multimodal sign language data includes image data and sensor data.

[0061] Step 120: Input the multimodal sign language data into a preset pre-trained neural network.

[0062] Step 130: Use the pre-trained neural network to extract features from the multimodal sign language data to obtain visual features and motion features.

[0063] Step 140: Dynamically fuse the visual features and motion features based on a preset attention mechanism to obtain multimodal fusion features.

[0064] Step 150: Perform dynamic sequence modeling on the multimodal fusion features to obtain the sign language feature sequence.

[0065] Step 160: Encode and decode the sign language feature sequence to obtain the target language text.

[0066] This embodiment acquires multimodal sign language data to be processed, including image data and sensor data, thus enabling multi-dimensional perception of sign language movements. Furthermore, visual and motion features can be extracted from different modalities. Visual features reflect gesture shapes, textures, and facial expressions, providing macroscopic posture and expression information, while motion features depict the three-dimensional spatial trajectory, speed, and acceleration of the hand, supplementing microscopic and high-frequency motion details. Based on this, this application can maintain complete capture of key motion information even under adverse conditions such as changes in lighting, partial occlusion, or background interference, thereby constructing a richer and more accurate internal sign language representation.

[0067] In addition, a dynamic fusion strategy is adopted, which adaptively adjusts the fusion weights of visual features and motion features based on the current semantic content and context of the sign language segment. This dynamic weighting method of semantic perception can improve the flexibility of fusion, thereby enabling a more comprehensive and intelligent understanding of the complex semantics of sign language and improving the accuracy of sign language translation.

[0068] Furthermore, traditional translation systems often only recognize sign language frame by frame / word by word without performing contextual semantic modeling, lacking an understanding of the overall sentence structure and context, which easily leads to problems such as semantic loss and word order disorder. This application, however, uses dynamic sequence modeling of multimodal fusion features to capture the temporal dependencies between sign language actions, understand complex grammatical phenomena such as subject-verb-object structures, time adverbs, and interrogative moods, thereby improving the ability to understand complex sign language grammar and context, resulting in more accurate, fluent, and natural language expression in the final translation output.

[0069] In some possible implementations, visual feature extraction can employ the following process:

[0070] 1) Extract key points from the image data;

[0071] In some possible implementations, a high-resolution video stream of gestures, facial expressions, and body movements can be captured using a wide-angle 1080p camera. The video stream can then be divided into frames to obtain single-frame image data.

[0072] In this way, high-resolution video stream input can provide sufficient visual information for subsequent detailed analysis.

[0073] In addition, direct use of raw pixel data from video streams for sign language recognition is susceptible to interference from various factors, such as complex backgrounds, lighting changes, differences in shooting angles, and individual user characteristics (such as skin color, clothing, and body shape), which seriously affect the stability and generalization ability of the system.

[0074] Based on this, this application employs advanced pose estimation algorithms (such as OpenPose or MediaPipeHolistic) to extract high-precision human key point information from video frames.

[0075] The key points extracted include, but are not limited to, body posture, hands, and face.

[0076] Specifically, body posture can include major joints such as the head, shoulders, elbows, wrists, and hips, used to capture overall limb movements and spatial positions;

[0077] The hands can be extracted in 21 places per hand to accurately describe the hand posture, orientation, and the bending and extension of each finger;

[0078] The face can cover key points in areas such as eyebrows, eyes, and lips, and is used to identify non-manual grammatical components such as facial expressions and lip movements.

[0079] Thus, by first extracting key points, this application effectively shields the interference caused by cluttered backgrounds, lighting fluctuations, skin color differences, and changes in camera distance, significantly improving the system's adaptability and recognition stability in diverse real-world scenarios. Simultaneously, the key point data has lower dimensionality and a more compact structure, which is beneficial for subsequent real-time processing and multimodal fusion, providing a reliable data foundation for achieving high-precision, low-latency sign language translation.

[0080] 2) Normalize the key points to obtain normalized key points;

[0081] In this way, by standardizing the extracted key points, the influence of individual differences (such as height and arm length) and collection conditions (such as shooting distance and angle) can be eliminated.

[0082] No, normalization can effectively enhance the robustness of the model to translation, rotation and scaling changes, ensuring that key point features are comparable for different users and in different scenarios.

[0083] In some possible implementations, the key points include hand and facial key points as well as body posture key points; the normalization process of the key points to obtain normalized key points includes:

[0084] 2-1) Perform local normalization processing on the key points of the hands and face;

[0085] Specifically, the key point coordinates of each hand and face area can be extracted separately and scaled to a uniform standardized range, such as (-1, 1), while maintaining their original aspect ratio.

[0086] In this way, by using local normalization, the focus can be placed on the geometric structure of local regions. The feature extraction process no longer depends on the absolute position of the hand or face in the image, but can focus on the geometric structure of local regions. This effectively eliminates scale and displacement differences caused by changes in shooting distance, user position, or posture, thereby more accurately capturing subtle gestures (such as finger flexion and extension) and dynamic changes in facial expressions (such as eyebrow and eye movements, and mouth shape changes), significantly improving the model's sensitivity to local semantic information and recognition accuracy.

[0087] 2-2) Perform global normalization on the key points of the body posture to obtain normalized key points.

[0088] Specifically, the relative relationships between body parts can be maintained by creating bounding boxes between shoulders (such as the left and right shoulders) and scaling key points.

[0089] In this way, by global normalization, while keeping the relative spatial relationship of different parts of the body unchanged, the scale and displacement changes caused by differences in user height, shooting distance and position in the picture are eliminated. This ensures that the same sign language action has a stable spatial representation in different frames or under different recording conditions, effectively reduces intra-frame offset, improves the continuity and consistency of cross-frame action sequences, and provides more robust input features for subsequent temporal modeling and action recognition.

[0090] In addition, by normalizing the key points, the robustness of the model to the diversity of sign language styles and subtle differences in movements is enhanced, enabling it to achieve good generalization ability among different individuals. This improves the universality and adaptability of the algorithm and reduces the reliance on personalized training. In other words, there is no need to conduct a lot of customized training for each individual. The system can serve a wider range of users more efficiently, including hearing-impaired people of different ages, body types and expression habits.

[0091] 3) Perform feature transformation on the normalized key points to obtain visual features.

[0092] In some possible implementations, physical sensors such as inertial measurement units or data gloves can be used to capture fine motion data of hand, arm, and limb movements, such as acceleration and angular velocity. This sensor data provides high-frequency, accurate motion information, complementing image data and achieving more comprehensive motion capture.

[0093] In some possible implementations, motion feature extraction can be performed using the following process:

[0094] 1) The sensor data is denoised to obtain denoised sensor data;

[0095] In some possible implementations, noise reduction and signal quality can be improved by methods such as filtering (e.g., low-pass filtering or Kalman filtering).

[0096] 2) Perform time alignment and language alignment processing on the denoised sensor data and the image data to obtain aligned sensor data;

[0097] Specifically, precise time synchronization between the denoised sensor data and image data ensures that multimodal information is strictly aligned in the time dimension.

[0098] After time alignment, further feature extraction is performed, including trend analysis, frequency domain transformation, and motion pattern recognition. In this way, semantically meaningful motion features can be extracted from the original time series data, providing high-quality input for subsequent fusion and recognition.

[0099] Furthermore, to enable Large Language Models (LLMs) to understand and reason about sensor data, this embodiment also introduces a semantic alignment mechanism between sensor data and natural language. Specifically, a pre-trained time-series encoder (such as Transformer or TCN) is used to extract deep temporal features from the sensor data, and these features are mapped into the semantic embedding space of the LLM through a learnable projection module.

[0100] This effectively alleviates the limitations of traditional text tokenization methods in processing continuous, non-linguistic modal data, enabling sensor signals to be parsed by LLM in a "language-like" form. Consequently, the model can achieve joint understanding of multimodal information (such as actions and descriptions) within a unified semantic framework, supporting the interpretation and generation of sign language actions through natural language interaction, significantly improving the system's interpretability and interactivity.

[0101] 3) Extract features from the aligned sensor data to obtain motion features.

[0102] Typically, sequence-to-sequence models consist of an encoder and a decoder, and are widely used in sequence conversion tasks such as machine translation. The encoder is responsible for compressing the input sequence (such as a sign language feature sequence) into a fixed-length context vector, capturing its semantic information; the decoder then generates the target sequence (natural language text) based on this context vector.

[0103] However, traditional sequence-to-sequence models face inherent limitations when processing long input sequences. Because the encoder compresses the entire source sequence into a fixed-length context vector, for longer sequences, this vector struggles to capture all the crucial information, easily leading to "information bottlenecks" and "information loss." This information loss directly impacts the accuracy and fluency of translation, especially in languages ​​like sign language that heavily rely on time sequences and context, where semantic understanding and expression in complex contexts are particularly challenging.

[0104] Based on this, to overcome the problems of information bottleneck and insufficient context modeling in traditional sequence-to-sequence models when processing long sequences, this application adopts an optimized sequence-to-sequence architecture, using a bidirectional long short-term memory network as the encoder, combined with an attention mechanism and a gated recurrent unit, to achieve efficient modeling of dynamic sequence features and contextual dependencies in sign language. Specifically, refer to the second embodiment of a sign language translation method provided in this application, which includes the following steps:

[0105] Step 210: Obtain the multimodal sign language data to be processed.

[0106] The multimodal sign language data includes image data and sensor data.

[0107] Step 220: Input the multimodal sign language data to be processed into a preset pre-trained neural network.

[0108] Step 230: Use the pre-trained neural network to extract features from the multimodal sign language data to be processed, and obtain visual features and motion features.

[0109] Step 240: Dynamically fuse the visual features and motion features based on a preset attention mechanism to obtain multimodal fusion features.

[0110] Step 250: Perform dynamic sequence modeling on the multimodal fusion features to obtain the sign language feature sequence.

[0111] Step 260: Input the sign language feature sequence into a trained sequence-to-sequence model, wherein the trained sequence-to-sequence model has an encoder and a decoder.

[0112] Step 270: Use the encoder to encode and compress the input sign language feature sequence to obtain a fixed-length context vector.

[0113] Step 270: Input the context vector into the decoder, and use the decoder to decode the context vector based on the attention mechanism to obtain the target language text.

[0114] In the process of generating the target language sequence, the decoder can employ an attention mechanism to dynamically focus on different parts of the encoder's output and assign differentiated weights to each part. This avoids the information bottleneck and semantic loss caused by using a single, fixed context vector in traditional encoder-decoder frameworks, allowing the model to flexibly focus on the most relevant semantic segments in the source sequence based on the currently generated words. For example, when translating a sign language action description, the model can focus on the starting frame of the gesture when generating the word "raise hand," and focus on the region of change in motion speed when generating "rapidly fall," thereby achieving fine-grained semantic alignment.

[0115] Through this dynamic focusing mechanism, the model significantly enhances its ability to capture key information from the source input, improving not only the accuracy and contextual coherence of the translation results but also effectively modeling the complex mapping relationship between the source and target sequences. Furthermore, this mechanism maintains good performance even with a limited number of model parameters, contributing to the realization of an efficient and lightweight sign language translation system.

[0116] In some possible implementations, this application employs a bidirectional long short-term memory network as the encoder, capable of simultaneously processing the forward and backward information flows of the input sequence. This structure models the time series from the forward direction (from the beginning to the present) and the backward direction (from the end to the present) using two independent hidden states, thereby achieving bidirectional awareness of the context.

[0117] This mechanism enables the model not only to capture the dependency between the current sign language action and historical actions, but also to perceive its semantic association with subsequent actions. For example, when a preceding facial expression or gesture in sign language is used to express question or negation, the reverse information flow can capture this contextual clue in advance, enhancing the accuracy of semantic understanding.

[0118] By using bidirectional long short-term memory networks for bidirectional temporal information processing, the model can obtain complete contextual information of long sign language sentences, effectively overcoming the terminal information decay problem of traditional unidirectional RNNs when processing long sequences. It achieves more comprehensive and in-depth semantic encoding of the source language, fully explores the global dependencies of the input sequence in the time dimension, and significantly improves the modeling ability of complex sign language structures and contextual dynamics.

[0119] Building upon the bidirectional long short-term memory network, this application further introduces a gated recurrent unit (GRU). The GRU is an efficient variant of a recurrent neural network that significantly reduces model parameters and computational complexity by simplifying the internal gate control mechanism (containing only update and reset gates). This maintains the bidirectional long short-term memory network's ability to handle long sequences and avoid gradient vanishing / exploding problems.

[0120] Gated recurrent units (GRUs) enable more efficient transmission and updating of information in long sequences. Through their built-in update and reset gate mechanisms, the model can adaptively retain and transmit key sign language features and contextual information (such as grammatical structure, facial expression changes, and action sequence) while filtering out redundant or irrelevant input. This selective information regulation mechanism effectively alleviates the long-term dependency loss and semantic bottleneck problems caused by traditional sequence-to-sequence models that rely on a single, fixed-length context vector.

[0121] Based on this, since the structure of the gated recurrent unit is relatively simple and involves fewer parameters, the training efficiency and convergence stability of the model in complex contexts can be further improved, providing a reliable sequence modeling foundation for achieving accurate and robust sign language translation.

[0122] This embodiment combines a bidirectional long short-term memory network (LSTM) and a gated recurrent unit (GRU) to achieve dual optimization for long sequence context understanding and information flow. The LSM provides global context awareness, capturing complete sequence dependencies through bidirectional processing; while the GRU optimizes information transfer efficiency and model training stability in long sequences, ensuring the effective transmission of important information. The synergistic effect of both enables the model to more accurately capture the dynamic changes and internal logic of sign language sequences, thereby improving the accuracy of sign language translation results.

[0123] Furthermore, when learning the mapping from sign language to natural language, the model in this application internalizes the grammatical rules and expression habits of the target language and performs in-depth optimization at the language generation level, rather than simply matching words. This allows the output translation results to better conform to the grammatical logic of natural language, resulting in more natural and fluent expression. This solves the problem of stiffness and unnaturalness in traditional machine translation, making communication between deaf and hearing people smoother and improving the user experience.

[0124] Referring to a third embodiment of a sign language translation method provided in this application, the sign language translation method includes the following steps:

[0125] Step 310: Obtain the multimodal sign language data to be processed.

[0126] Step 320: Input the multimodal sign language data to be processed into a preset pre-trained neural network.

[0127] Step 330: Use the pre-trained neural network to extract features from the multimodal sign language data to be processed, and obtain visual features and motion features.

[0128] Step 340: Input the visual features and motion features into a preset multi-stream fusion module.

[0129] Step 350: The multi-stream fusion module uses a preset gating mechanism to calculate the contribution values ​​corresponding to the visual features and the motion features, respectively.

[0130] Step 360: The contribution values ​​corresponding to the visual features and the motion features are weighted and summed to obtain the multimodal fusion features.

[0131] For steps 340-360, the Multi-Stream Fusion Module (MFM) uses a pre-defined gating mechanism to calculate the contribution value of each modal stream and performs a weighted sum, thereby dynamically adjusting the weights. Specifically, the Multi-Stream Fusion Module (MFM) compresses the channel dimension of each modal stream and then inputs the result into the gating unit. The gating unit calculates the gating value for each modal stream, which determines the relative importance or contribution of each modal stream to the final prediction. The final prediction output is a weighted sum of the output of each stream and its corresponding gating value.

[0132] In other words, in the complex context of sign language translation, the model can intelligently adjust the contribution weights of image data (such as gestures, faces, and limbs) and sensor data based on the semantic content and context of the sign language. For example, when sign language expresses emotions, subtle changes in facial expressions are crucial, and the model can give higher weight to facial expression data; while when expressing numbers, letters, or specific hand movements, the accuracy of hand movements and the fine motion data captured by inertial sensors are more critical, so the model can focus more on information from these modalities.

[0133] In this way, the dynamic fusion approach enables the model to understand the complex semantics of sign language more comprehensively and intelligently, significantly improving the accuracy and robustness of translation.

[0134] Step 370: Perform dynamic sequence modeling on the multimodal fusion features to obtain the sign language feature sequence.

[0135] Step 380: Encode and decode the sign language feature sequence to obtain the target language text.

[0136] In some possible implementations, the image data includes skeletal data, and the method further includes:

[0137] Step 11: Input the skeletal data into the preset spatiotemporal joint attention module.

[0138] Step 12: Use the spatiotemporal joint attention module to perform temporal global max pooling and spatial global max pooling on the skeletal data to obtain the corresponding first feature vector and second feature vector, respectively.

[0139] Step 13: Concatenate the first feature vector and the second feature vector to obtain the concatenated feature vector.

[0140] For example, you can refer to the following formula 1:

[0141] X'=TGMP(Xin)⊕SGMP(Xin), Formula 1.

[0142] Where X' is the concatenated feature vector, Xin is the input skeletal data, TGMP(Xin) is the first feature vector obtained by performing temporal global max pooling on the skeletal data, and SGMP(Xin) is the second feature vector obtained by performing spatial global max pooling on the skeletal data.

[0143] Step 14: Perform dimensionality reduction processing on the spliced ​​feature vector to obtain the dimensionality-reduced spliced ​​feature vector.

[0144] In some embodiments, channel compression of the concatenated features can be performed using a 1x1 convolutional layer and a HardSwish activation function to reduce computational complexity and enhance nonlinear expressive power. Specifically, the following formula 2 can be used:

[0145] X''=HS(Conv_k=1(X')), Formula 2.

[0146] Where X'' is the concatenated feature vector after dimensionality reduction.

[0147] Step 15: Based on the dimensionality-reduced concatenated feature vector, generate a temporal attention map and a spatial attention map.

[0148] The concatenated feature vectors after dimensionality reduction can be processed by the Sigmoid function to generate temporal attention maps and spatial attention maps. For details, please refer to Formulas 3 and 4 below.

[0149] Mt=Sigmoid(Conv_k=1(X'')), Formula 3.

[0150] Mv=Sigmoid(MLP(X'')), Formula 4.

[0151] Where Mt is the temporal attention map and Mv is the spatial attention map.

[0152] Step 16: Perform an outer product operation on the channels of the temporal attention map and the spatial attention map to obtain a joint attention map.

[0153] Step 17: Perform element-wise multiplication based on the joint attention map and the bone data to obtain the target bone data.

[0154] For steps 16-17, please refer to Formula 5 below.

[0155] Xout=(Mt*Mv)⊙Xin, Formula 5.

[0156] Where Xout represents the target skeleton data, and Mt*Mv represents the joint attention map obtained by the outer product operation of the temporal attention map and the spatial attention map channels.

[0157] Thus, by introducing a spatiotemporal joint attention module, this module first performs global max pooling in the temporal and spatial dimensions respectively to extract the temporal evolution features and spatial structure features of the action, and then concatenates the two to form a joint representation. Subsequently, a temporal attention map and a spatial attention map are generated through two independent attention branches respectively. Finally, a two-dimensional joint attention mechanism is constructed through outer product operation, and then fused with the original features element by element to generate a weighted and enhanced output.

[0158] Furthermore, the model in this application can capture the intrinsic connection between spatial posture and temporal dynamics in sign language, as well as better understand the subtle differences and contextual meanings of sign language. In turn, it can intelligently adjust the focus on different modal information according to the context of the sign language content, thereby achieving more "intelligent" translation and improving the accuracy and fluency of sign language translation.

[0159] In some possible implementations, the model adopted in this application can be integrated into an embedded system, possessing offline operating capabilities. This ensures stable system operation even in environments without network connectivity and effectively protects user privacy. This feature is particularly suitable for public service scenarios with high requirements for data security and service continuity, such as hospitals, government service windows, and banks. By localizing processing, sensitive sign language interaction data is avoided from being uploaded to the cloud, which not only enhances privacy protection but also enables the continued provision of reliable services in the event of network interruptions or poor signal, significantly improving system availability and user trust.

[0160] In some possible implementations, user feature modeling and adaptive training mechanisms can be introduced to achieve customized recognition and translation based on individual sign language habits.

[0161] The sign language translation method provided in this application can be used in communication devices such as sign language translators and dual-screen AI portable communication terminals. The human-computer interaction is closer to natural dialogue, enabling two-way barrier-free communication between deaf and hearing people, and improving the convenience of communication and user experience.

[0162] Furthermore, the sign language translation method provided in this application can be widely applied in various practical scenarios, including hospital guidance, government service windows, banking services, school teaching assistance, and daily family communication, while also greatly enhancing the social participation of the deaf and mute community.

[0163] In some possible implementations, the sign language translation method provided in this application can also be integrated into AR / VR devices to realize real-time sign language communication and multimodal communication in virtual space.

[0164] In some possible implementations, the sign language translation system provided in this application can add sign language learning functions and use virtual sign language generation technology to provide learners with interactive teaching content.

[0165] In some possible implementations, this application can provide dedicated optimized models and interactive interfaces for industries with high privacy and high accuracy requirements, such as healthcare, government affairs, and banking.

[0166] In addition, the sign language translation method provided in this application can also support interoperability with remote sign language translation platforms, realizing a combination of human assistance and automatic translation in complex scenarios.

[0167] Furthermore, this application can support sign language translation in multiple languages, enabling bidirectional translation between sign language and spoken language in languages ​​such as Chinese, English, and Japanese, thus meeting the needs of international communication across languages ​​and regions.

[0168] Corresponding to the above sign language translation method, this application also provides a sign language translation device. The sign language translation device includes a unit for performing the above sign language translation method, and the device can be configured in a desktop computer, tablet computer, laptop computer, or other terminal.

[0169] like Figure 2 As shown in the figure, this application provides a computer device including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0170] Memory 113 is used to store computer programs;

[0171] In one embodiment of this application, when the processor 111 executes the program stored in the memory 113, it implements the sign language translation method provided in any of the foregoing method embodiments, including:

[0172] Acquire multimodal sign language data to be processed, wherein the multimodal sign language data includes image data and sensor data;

[0173] The multimodal sign language data is input into a preset pre-trained neural network;

[0174] The pre-trained neural network is used to extract features from the multimodal sign language data to obtain visual and motion features;

[0175] The visual features and motion features are dynamically fused based on a preset attention mechanism to obtain multimodal fused features;

[0176] Dynamic sequence modeling is performed on the multimodal fusion features to obtain the sign language feature sequence;

[0177] The sign language feature sequence is encoded and decoded to obtain the target language text.

[0178] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0179] Therefore, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the sign language translation method provided in any of the foregoing method embodiments, including:

[0180] Acquire multimodal sign language data to be processed, wherein the multimodal sign language data includes image data and sensor data;

[0181] The multimodal sign language data is input into a preset pre-trained neural network;

[0182] The pre-trained neural network is used to extract features from the multimodal sign language data to obtain visual and motion features;

[0183] The visual features and motion features are dynamically fused based on a preset attention mechanism to obtain multimodal fused features;

[0184] Dynamic sequence modeling is performed on the multimodal fusion features to obtain the sign language feature sequence;

[0185] The sign language feature sequence is encoded and decoded to obtain the target language text.

[0186] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0187] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0188] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0189] The steps in the methods of this application embodiment can be adjusted, merged, or deleted according to actual needs. The units in the apparatus of this application embodiment can be merged, divided, or deleted according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0190] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0191] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0192] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Since these modifications and variations fall within the scope of the claims and their equivalents, this application also intends to include these modifications and variations.

[0193] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A sign language translation method, characterized in that, The sign language translation method includes: Acquire multimodal sign language data to be processed, wherein the multimodal sign language data includes image data and sensor data; The multimodal sign language data is input into a preset pre-trained neural network; The pre-trained neural network is used to extract features from the multimodal sign language data to obtain visual and motion features; The visual features and motion features are dynamically fused based on a preset attention mechanism to obtain multimodal fused features; Dynamic sequence modeling is performed on the multimodal fusion features to obtain the sign language feature sequence; The sign language feature sequence is encoded and decoded to obtain the target language text; The process of encoding and decoding the sign language feature sequence to obtain the target language text includes: The sign language feature sequence is input into a trained sequence-to-sequence model, wherein the trained sequence-to-sequence model has an encoder and a decoder; The encoder is used to encode and compress the input sign language feature sequence to obtain a fixed-length context vector; The context vector is input into the decoder, and the decoder uses an attention mechanism to decode the context vector to obtain the target language text. The step of dynamically fusing the visual features and motion features based on a preset attention mechanism to obtain multimodal fusion features includes: The visual features and motion features are input into a preset multi-stream fusion module; The multi-stream fusion module uses a preset gating mechanism to calculate the contribution values ​​corresponding to the visual features and the motion features, respectively. The contribution values ​​corresponding to the visual features and the motion features are weighted and summed to obtain the multimodal fusion features; The multi-stream fusion module compresses the channel dimension of each modal stream and then inputs the result into the gating unit. The gating unit calculates the gating value of each modal stream, which determines the relative importance or contribution of each modal stream to the final prediction. The final prediction output is the weighted sum of the output of each stream and its corresponding gating value.

2. The method according to claim 1, characterized in that, When the multimodal sign language data is image data, the step of using the pre-trained neural network to extract features from the multimodal sign language data to be processed, obtaining visual features and motion features, includes: Extract key points from the image data; The key points are normalized to obtain normalized key points; The normalized key points are then subjected to feature transformation to obtain visual features.

3. The method according to claim 2, characterized in that, The key points include hand and facial key points as well as body posture key points; the normalization process for the key points to obtain normalized key points includes: Local normalization processing was performed on the key points of the hands and face; And the key points of the body posture are globally normalized to obtain normalized key points.

4. The method according to claim 2, characterized in that, When the multimodal sign language data is sensor data, the step of using the pre-trained neural network to extract features from the multimodal sign language data to be processed, obtaining visual features and motion features, further includes: The sensor data is denoised to obtain denoised sensor data. The denoised sensor data and the image data are time-aligned and language-aligned to obtain aligned sensor data; Feature extraction is performed on the aligned sensor data to obtain motion features.

5. The method according to claim 2, characterized in that, The image data includes skeletal data, and the method further includes: The skeletal data is input into a preset spatiotemporal joint attention module; The spatiotemporal joint attention module is used to perform temporal global max pooling and spatial global max pooling on the skeletal data to obtain the corresponding first feature vector and second feature vector, respectively. The first feature vector and the second feature vector are concatenated to obtain the concatenated feature vector; The concatenated feature vector is then subjected to dimensionality reduction processing to obtain the dimensionality-reduced concatenated feature vector; Based on the dimensionality-reduced concatenated feature vectors, a temporal attention map and a spatial attention map are generated. The temporal attention map and the spatial attention map are multiplied by their channels to obtain a joint attention map. The target bone data is obtained by performing element-wise multiplication based on the joint attention map and the bone data.

6. A sign language translation device, characterized in that, Includes a unit for performing the method as described in any one of claims 1-5.

7. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-5.

Images