A sign language translation method based on a smart wrist-worn device motion sensor

Abstract

This invention relates to a sign language translation method based on motion sensors in a smart wrist-worn device, belonging to the field of artificial intelligence and mobile computing application technology. This method establishes a mapping relationship between hand movements, using the user's dominant hand movement data as a benchmark to generate non-dominant hand movement data. By intelligently estimating the non-dominant hand movement data and combining neural networks and large language models, it can achieve accurate recognition and translation of hand-held sign language without increasing the burden on additional equipment. This method achieves seamless integration with users' daily lives, featuring a simple and easy-to-use design. Users do not need to purchase additional dedicated equipment to quickly integrate it into their daily use scenarios, lowering the threshold for technology application and increasing the accessibility of sign language translation technology. It not only improves translation accuracy and ensures real-time response, but also makes communication between sign language users and non-sign language users smoother and more barrier-free.

Description

Technical Field

[0001] This invention relates to a sign language translation method, specifically a sign language translation method based on motion sensors of a smart wrist-worn device, belonging to the field of artificial intelligence and mobile computing application technology. Background Technology

[0002] According to the latest survey by the World Health Organization, more than 1.5 billion people worldwide currently suffer from varying degrees of hearing impairment. Among them, tens of millions rely on sign language as their primary means of daily communication and are known as sign language users. Sign language expression involves a complex combination of hand shapes, hand gestures, and facial expressions; each subtle movement or expression can convey different meanings. However, for those unfamiliar with sign language, learning and understanding these complex expressions in a short period is extremely difficult. Therefore, a significant barrier exists in communication between sign language users and non-sign language users.

[0003] Currently, researchers are exploring various sign language translation methods to facilitate communication between sign language speakers and non-sign language speakers.

[0004] The first category is computer vision-based sign language translation methods. These methods capture video recordings of sign language interpreters using a camera and analyze their hand movements and facial expressions to convert sign language into spoken language. However, these methods are easily affected by lighting conditions and background interference, resulting in low translation accuracy.

[0005] The second category is sign language translation methods based on wireless signals, which mainly analyze the time and frequency domain changes of wireless signals (such as Wi-Fi, millimeter waves, infrared light, and sound waves) triggered by sign language gestures. However, the limited portability and high cost of sensing devices make it difficult to promote and apply this method on a large scale.

[0006] The third category is sign language translation methods based on wearable devices. These methods rely on wearable inertial measurement units (IMUs), electromyography (EMG) sensors, and photoplethysmography (PPG) sensors to capture and translate sign language gestures anytime, anywhere. However, these methods either can only recognize a few specific gestures, making it difficult to meet the needs of daily communication, or they depend on wearing the device with one hand, resulting in low accuracy when recognizing two-handed sign language gestures.

[0007] Despite some progress in sign language translation, a method that is efficient, reliable, and low-cost across various environments has yet to emerge. Therefore, there is an urgent need to find a new sign language translation method to better facilitate communication between non-sign language speakers and sign language speakers. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings and defects of existing technologies and to solve the current technical problem of lacking low-cost, high-precision sign language translation by creatively proposing a sign language translation method based on motion sensors of a smart wrist-worn device.

[0009] In conventional techniques for sign language translation using wearable devices, high-precision translation typically requires capturing motion signals from both hands simultaneously. However, for ordinary users, wearing multiple smart devices in daily life, such as two smartwatches, is not only cumbersome but can also lead to awkward experiences, hindering the widespread adoption and effectiveness of these devices. This invention, however, requires no training data from new users, offering significant advantages in terms of immediate usability and greatly enhancing the convenience and adaptability of sign language translation.

[0010] The innovations of this invention include: by establishing a mapping relationship between the movements of both hands, and using the movement data of the user's dominant hand (usually the right hand) as a benchmark, generating movement data of the non-dominant hand (usually the left hand), it is possible to achieve accurate recognition and translation of sign language by both hands without increasing the burden on additional equipment.

[0011] The present invention is achieved through the following technical solution.

[0012] A sign language translation method based on motion sensors of a smart wrist-worn device includes the following steps:

[0013] Step 1: Use the motion sensor in the smart wrist-worn device to collect the motion signal of the user's wrist when performing sign language gestures, remove random noise from the signal, and segment the signal according to the sentence.

[0014] Specifically, when a user makes sign language gestures, accelerometers and gyroscopes worn on the wrist of the dominant hand (usually the right hand) collect acceleration and angular velocity data along the X, Y, and Z axes, which are denoted as a. x ,a y ,a z and g x ,g y ,g z .

[0015] Step 1.1: Use a low-pass filter to remove random noise from the sensor device hardware in the motion signal.

[0016] Theoretically, the frequency of human wrist movement is generally below 10Hz, and the hardware itself introduces high-frequency random noise. Therefore, a low-pass fourth-order Butterworth filter can be used to filter out this noise. x ,a y ,a z and g x ,g y ,g zLow-pass filtering (specifically, 10Hz) is applied to remove random noise.

[0017] Step 1.2: Calculate the linear acceleration signal, use the threshold method to determine obvious pauses between hand sentences, thereby identifying the start and end points of the hand sentences and segmenting the motion signal corresponding to the hand sentences.

[0018] Step 2: Input the six-axis motion data of the dominant hand (usually right hand) into the motion data generation network of the non-dominant hand (usually left hand) and output the six-axis motion data of the non-dominant hand.

[0019] The non-dominant hand motion data generation network proposed in this invention includes:

[0020] Forward Generator G R2L It is used to take the motion data of the right hand as input and generate corresponding motion data for the left hand.

[0021] Backward generator G L2R It is used to perform inverse operations, inputting left-hand movement data and generating corresponding right-hand movement data.

[0022] Discriminator D R This is used to classify real and generated right-hand movement data. R Includes two sub-discriminators: and They are used to classify the time-domain and frequency-domain responses of signals, respectively.

[0023] Discriminator D L This is used to perform similar judgments on left-hand motion data. L Includes two sub-discriminators: and They are used to classify the time-domain and frequency-domain responses of signals, respectively.

[0024] Step 3: Using hand movement data as input to a lightweight sign language recognizer, identify the annotation sequence. Feed the annotation sequence and customized prompts as input to a large language model to generate spoken sentences.

[0025] Specifically, it includes the following steps:

[0026] Step 3.1: The sign language recognition network first converts motion data into feature vectors, and then passes these vectors to the encoder of the Transformer to generate a sequence of annotations.

[0027] Step 3.2: After obtaining the annotated sequence of each spoken sentence, pass them to the large language model and input custom prompts to translate them into meaningful spoken sentences.

[0028] Specifically, ChatGPT and LLaMA2-70b can be selected as online and offline large language models, respectively, to generate translation results for sign language sentences. The following is a hint:

[0029] Note: You are a sign language expert.

[0030] Question: Is it possible to generate a meaningful spoken sentence based on a sequence of comments input by the user?

[0031] Rules: Only responses containing the generated sentence will be provided. If a spoken sentence cannot be created, the response will be "No translation".

[0032] Beneficial effects

[0033] This method has the following advantages compared with the prior art:

[0034] 1. This method achieves seamless integration with users' daily lives. Its simple and easy-to-use design allows users to quickly integrate it into their daily routines without purchasing additional specialized equipment. This not only lowers the barrier to entry for the technology but also increases the accessibility of sign language translation technology.

[0035] 2. This invention provides efficient sign language translation capabilities by intelligently estimating the movement data of the non-dominant hand, combined with neural networks and large language models. While maintaining user convenience, the system can accurately and in real-time translate sign language using both hands. This not only improves translation accuracy but also ensures real-time response, making communication between sign language users and non-sign language users smoother and more barrier-free. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the method principle of the present invention;

[0037] Figure 2 This invention presents a non-dominant hand motion data generation network structure.

[0038] Figure 3 This invention presents the sign language recognition network structure.

[0039] Figure 4 This invention demonstrates the performance of sign language translation among 7 participants in an embodiment of the invention.

[0040] Figure 5 This invention demonstrates the performance of translating 50 handwritten sentences in an embodiment of the invention.

[0041] Figure 6 This demonstrates the performance of sign language translation under different training data conditions in embodiments of the present invention.

[0042] Figure 7 This invention demonstrates the performance of sign language translation under different sign language speed conditions in its embodiments. Detailed Implementation

[0043] The method of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0044] Example

[0045] Due to linguistic diversity and cultural specificity, there are over 300 sign languages ​​worldwide, such as American Sign Language, British Sign Language, and Australian Sign Language. Each sign language has unique grammar, vocabulary, and cultural background, making it a complete language system. In this embodiment, the research focus is on American Sign Language, which is also one of the most studied sign languages.

[0046] like Figure 1 As shown, a sign language translation method based on the motion sensor of a smart wrist-worn device includes the following steps:

[0047] Step 1: Use the motion sensor in the smart wrist-worn device to collect the motion signal of the user's wrist when performing sign language gestures, remove random noise from the signal, and segment the signal according to the sentence.

[0048] Specifically, when a user makes sign language gestures, an accelerometer and gyroscope worn on the wrist of the dominant hand (usually the right hand) collect acceleration and angular velocity data along the X, Y, and Z axes, denoted as a. x ,a y ,a z and g x ,g y ,g z .

[0049] Step 1.1: Use a low-pass filter to remove random noise from the sensor device hardware in the motion signal.

[0050] Theoretically, the frequency of human wrist movement is generally below 10Hz, and the hardware itself introduces high-frequency random noise. Therefore, a fourth-order Butterworth low-pass filter can be used to filter out this noise. x ,a y ,a z and g x ,g y ,g z Low-pass filtering (specifically, 10Hz) is applied to remove random noise.

[0051] Step 1.2: Calculate the linear acceleration signal and use a threshold method to determine obvious pauses between handwritten sentences. This identifies the start and end points of the handwritten sentences and segments the motion signals corresponding to them.

[0052] Specifically, a 200ms sliding window with 50% overlap can be used to calculate the linear acceleration (MLA) within the window:

[0053]

[0054] Among them, A x A y A z The average accelerations on the X, Y, and Z axes are respectively, where g is the acceleration due to gravity.

[0055] Based on practical experience, the threshold is set to twice the maximum value of the MLA calculated from the first 200ms period. If the MLA calculated in two consecutive windows is greater than / less than the threshold, the starting point of the first window is determined as the start / end point. Finally, for each segment of the handwritten sentence, the triaxial acceleration and triaxial angular velocity data are normalized to [-1, 1]. Furthermore, the motion signal data points corresponding to each sentence can be resampled to 1024 to ensure consistency in the length of all sample data.

[0056] Step 2: Input the six-axis motion data of the dominant hand (usually right hand) into the motion data generation network of the non-dominant hand (usually left hand) and output the six-axis motion data of the non-dominant hand.

[0057] Specifically, the framework of the non-dominant hand motion data generation network is as follows: Figure 2 As shown, it includes:

[0058] Forward Generator G R2L It is used to take the motion data of the right hand as input and generate corresponding motion data for the left hand.

[0059] Backward generator G L2R It is used to perform inverse operations, inputting left-hand movement data and generating corresponding right-hand movement data.

[0060] Discriminator D R It is used to classify real and generated right-hand movement data. It includes two sub-discriminators: and They are used to classify the time-domain and frequency-domain responses of signals, respectively.

[0061] Discriminator D L A similar judgment is performed on the left-hand motion data, including two sub-discriminators: and They are used to classify the time-domain and frequency-domain responses of signals, respectively.

[0062] Among them, generator G R2L and G L2RThe same encoder and decoder structure is used. The input is 6 × 1024 data points because each handwritten sentence has 6 channels of signal: X, Y, and Z axes for acceleration and angular velocity data. The encoder stage consists of two stacked convolutional neural network (CNN) layers, interleaved with max-pooling layers. The first CNN layer has 32 kernels, and the second has 64 kernels, both with a kernel size of 3 × 3, a stride of 1, and activation using scaling exponential linear units (SELU). Each CNN layer is followed by a 2 × 2 max-pooling layer with a stride of 2 to downsample the feature maps. The decoder stage consists of two deconvolutional neural network (DCNN) layers, each with 64 and 32 kernels of 3 × 3 size, a stride of 1, and SELU activation, interleaved with two unpooling layers that invert the max-pooling operation, upsampling the data back to its original size.

[0063] Considering that hand movements in sign language gestures are reflected in both the time and frequency domains, this invention uses a dual discriminator strategy, where D... t and D f This is used to classify the time-domain and frequency-domain responses of real and generated data. Specifically, the time-series signal is input into a time-domain discriminator. and A judgment is made. Simultaneously, a Short-Time Fourier Transform (STFT) is performed on the time series data to obtain the frequency domain response. Then, the six generated spectrograms are connected along a shared time axis to form a feature representation map. Finally, the feature representation map serves as a frequency domain discriminator. and The input is [data missing]. Both discriminators have four convolutional layers with 64, 128, 256, and 512 kernels respectively. The kernel size is 3×3 with a stride of 1. Each convolutional layer is followed by layer normalization and rectified linear unit function (ReLU) activation. Finally, there is a fully connected layer and a sigmoid activation function.

[0064] The training process involves two key loss functions: adversarial loss and cycle consistency loss. Adversarial loss is achieved by using a discriminator D. L and D R Ensure that the generated motion data is indistinguishable from real motion data. The forward adversarial loss is given by Equations 2 and 3:

[0065]

[0066] in, and These are the adversarial losses in the time and frequency domains of the forward process, respectively. E represents the expected value, l represents the input real left-hand motion signal, L represents the generated left-hand motion signal, r represents the input right-hand motion signal, R represents the generated right-hand motion signal, and f represents the short-time Fourier transform of the signal.

[0067] Similarly, the adversarial loss in the backward direction is:

[0068]

[0069]

[0070] in, and These are the adversarial losses in the time and frequency domains of the backward process, respectively.

[0071] Cyclic consistency loss enforces the condition that if motion data is transferred from the right hand to the left hand and then back to the right hand, the original right-hand motion data should be recovered (and vice versa). Cyclic consistency loss It is given by the following formula:

[0072]

[0073] Where |·|1 represents the L1 norm.

[0074] The overall objective function is a combination of these losses:

[0075]

[0076] Wherein, α and β are adversarial loss coefficients, which can take values ​​of 0.2 and 0.2 respectively. It should be noted that other settings belonging to [0, 1] are also within the scope of this invention; λ is the cycle consistency loss coefficient, which takes a value of 0.5. It should be noted that other settings belonging to [0, 1] are also within the scope of this invention.

[0077] Step 3: Using hand movement data as input to a lightweight sign language recognizer, identify the annotation sequence. Feed the annotation sequence and customized prompts as input to a large language model to generate spoken sentences.

[0078] Specifically, it includes the following steps:

[0079] Step 3.1: The sign language recognition network first converts motion data into feature vectors, and then passes these vectors to the encoder of the Transformer to generate a sequence of annotations.

[0080] In sign language, annotations are a written representation that uses spoken language words to mark and describe individual sign words while maintaining the original sign language order and grammar. It's important to note that spoken and sign language have different grammatical structures, and the order in which words are presented also differs.

[0081] First, a short-time Fourier transform is applied to the motion data of both hands, with the frame length and window step size set to 256 and 128, respectively. Then, the six-axis motion data of each hand are concatenated into a single time-frequency representation. Thus, each sign language is represented by two spectrograms, which serve as input to the sign language recognizer network.

[0082] Lightweight feature extraction networks use two-dimensional convolutional neural networks to process each spectrogram into feature vectors, such as... Figure 3 As shown, the network comprises two parallel branches based on the ResNet-34 architecture, each dedicated to processing spectrograms from both the dominant and non-dominant hands. The first layer in each branch is a convolutional layer applying 64 kernels of size 3×3, followed by a max-pooling layer of size 2×2 to reduce spatial dimensionality. Subsequently, each branch includes four residual blocks, each containing multiple convolutional layers with 3×3 kernels and ReLU activation functions. The features extracted from each branch are then fed into a global average pooling layer, which transforms the spatial feature maps into a single feature vector by averaging each feature map. This structure enables the network to capture local and global patterns in the time-frequency domain, leveraging complementary information from both hands to improve the accuracy of sign language annotation recognition.

[0083] The extracted feature sequence is first flattened into a one-dimensional vector and then processed by a four-layer Transformer encoder. The Transformer encoder is crucial for processing sequential data and is particularly effective in natural language processing tasks due to its self-attention mechanism. To embed this sequence into the latent space, this invention utilizes linear embedding layers and learnable one-dimensional positional encoding. This embedded sequence is projected onto multiple key, value, and query spaces. The self-attention mechanism is computed as follows:

[0084]

[0085] Where Q, K, V represent the query, key, and value matrix, respectively; C ′ It is the dimension of the bond space. T denotes the matrix transpose.

[0086] This mechanism enables the model to effectively capture temporal dependencies and relationships in sequential data, and leverages self-attention to enhance feature representations.

[0087] The self-attention output is passed through a non-linear pointwise feedforward layer, followed by residual connections and normalization to improve training stability. Then, the resulting spatiotemporal representation is used to obtain annotation probabilities through a linear projection layer and a softmax activation function. Finally, connectionist temporal classification (CTC) is employed as a sequence-to-sequence loss function to align the predicted sign language annotation sequences with their corresponding ground truth labels. CTC is particularly suitable for classification problems involving temporal data because it can compute the difference between the predicted sequence and the actual label without pre-splitting the training data.

[0088] Step 3.2: After obtaining the annotated sequence of each spoken sentence, pass them to the large language model and input custom prompts to translate them into meaningful spoken sentences.

[0089] ChatGPT and LLaMA2-70b were selected as online and offline large language models, respectively, to generate translation results for sign language sentences. The prompt is as follows:

[0090] Note: You are a sign language expert.

[0091] Question: Is it possible to generate a meaningful spoken sentence based on a sequence of comments input by the user?

[0092] Rules: Only responses containing the generated sentence will be provided. If you are unable to create a spoken sentence, please answer "No translation".

[0093] Experimental verification

[0094] To verify the performance of the invention, a wrist-worn prototype was developed. The prototype included an integrated inertial motion sensor containing a three-axis accelerometer and a three-axis gyroscope, equivalent to a smartwatch or smart bracelet. Inertial motion sensor readings were recorded at a sampling rate of 100Hz and processed by a laptop computer paired with the prototype via Bluetooth. The laptop was equipped with an Intel Core i7-12700H processor, 16GB of RAM, and an Nvidia GeForce RTX 4060 graphics card. A camera was placed in front of the participants to provide a live view.

[0095] A total of 27 participants (14 men and 13 women), aged 18 to 52, were recruited. All participants were hearing-normal, right-handed, and had varying levels of sign language knowledge. The sign language corpus consisted of 250 American Sign Language sentences. Participants were divided into two groups: the first group of 20 participants wore prototypes on both wrists and performed each sentence 5 times. The training data consisted of 200 sentences. The second group of 7 participants wore prototypes on both wrists and each performed all sentences 5 times. Data collected from the second group was primarily used for testing purposes.

[0096] Word Error Rate (WER) is used as an evaluation metric to measure the minimum number of insertion, deletion, and replacement operations required to change an identified text sentence to its true value. WER is calculated as follows:

[0097]

[0098] Where S is the number of replacements, D is the number of deletions, I is the number of insertions, and N is the total number of words in the actual value. A lower WER indicates better performance, and a WER of 0 indicates perfect translation.

[0099] First, the performance of the invention was tested with new users, and then a second set of data was used to conduct independent user tests. Figure 4 The WER (Way Estimation and Response Rate) for each participant is displayed. It can be seen that the WERs of all seven users are below 10%. The highest, lowest, and average WERs are 8.3%, 2.2%, and 4.8%, respectively. Furthermore, participants 3 and 4 had the lowest WERs; their native language is English, and they have relatively rich experience with sign language, allowing the sensor to capture more stable and reliable information. Overall, this invention is very effective for promotion to new users and meets design considerations for immediate usability.

[0100] Secondly, to understand whether the invention could be generalized to new handwritten sentences without retraining or adjustment, an invisible sentence test was conducted. Specifically, data from 50 handwritten sentences not in the training dataset were used to control for individual sentence variables in the first group. Figure 5 The WER (Warnings-Effectiveness) for each unseen sentence is shown. The WER results varied between 1.4% and 16.5%, with a mean error of 8.3%. By examining the sentence content, it was observed that sentences containing words that appeared frequently in the training data tended to have lower WERs. Overall, this performance demonstrates the generalization ability of the invention, as it can effectively identify new sentence structures and vocabulary without further tweaking or training.

[0101] The effectiveness of the non-dominant hand motion data generation network of this invention was then verified. The following model variants were all generated by the sign language recognizer model: Model 1 was constructed using real motion data from both the dominant and non-dominant hands. Model 2 was constructed using real motion data from the dominant hand and generated non-dominant hand motion data. Model 3 was constructed using only motion data from the dominant hand. The three models were evaluated using the WER metric of sign language annotations in user-independent and unseen sentence tests. Figure 6The results shown indicate that Model 2 performs comparably to Model 1, with only minor differences in WER (e.g., 0.4% and 0.2% in the two tests, respectively). This demonstrates that the generated non-dominant hand motion data can be effectively used for sign language translation without significantly impacting performance. Furthermore, Model 3 exhibits the worst performance, highlighting the importance of non-dominant hand data in maintaining sign language translation performance.

[0102] To evaluate the impact of different sign language speeds on the performance of the invention, an experiment was conducted in which participants in a second group were asked to perform sign language at three different speeds: approximately 1.0 words / second, 1.5 words / second, and 1.9 words / second, representing slow, normal, and fast sign language speeds, respectively. Figure 7 The WER (Warning Estimation) is displayed at different sign language speeds. It can be seen that the WER fluctuates slightly as the sign language speed increases. However, even at very fast speeds, the WER remains below 10%, because the resampling mechanism adds fewer data samples within a fixed sampling rate, thus compensating for the differences in sign language speed.

[0103] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is used to explain the present invention. It is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A sign language translation method based on motion sensors of a smart wrist-worn device, characterized in that, Includes the following steps: Step 1: Use the motion sensor in the smart wrist-worn device to collect the motion signal of the user's wrist when performing sign language gestures, remove random noise from the signal, and segment the signal according to the sentence; Step 2: Input the six-axis motion data of the dominant hand into the non-dominant hand motion data generation network, and output the six-axis motion data of the non-dominant hand; The non-dominant hand motion data generation network includes: Forward Generator G R2L It is used to take the motion data of the right hand as input and generate corresponding motion data for the left hand; Backward generator G L2R This is used to perform inverse operations, inputting left-hand motion data and generating corresponding right-hand motion data; Discriminator D R Used to classify real and generated right-hand movement data; D R Includes two sub-discriminators: and These are used to classify the time-domain and frequency-domain responses of signals, respectively. Discriminator D L Used to classify real and generated left-hand movement data; D L Includes two sub-discriminators: and These are used to classify the time-domain and frequency-domain responses of signals, respectively. Step 3: Use hand movement data as input to a lightweight sign language recognizer to identify the annotation sequence; provide the annotation sequence and customized prompts as input to a large language model to generate spoken sentences; Specifically, the following steps are taken: First, a short-time Fourier transform is applied to the motion data of both hands, with the frame length and window step size set to 256 and 128, respectively; then, the six-axis motion data of each hand are concatenated into a single time-frequency representation; thus, each sign language is represented by two spectrograms, which serve as the input to the sign language recognition network. The lightweight feature extraction network uses a two-dimensional convolutional neural network to process each spectrogram into a feature vector. The network comprises two parallel branches based on the ResNet-34 architecture, each dedicated to processing spectrograms from both the idiosyncratic and non-idiosyncratic sources. The first layer in each branch is a convolutional layer applying 64 kernels of size 3×3, followed by a max-pooling layer of size 2×2 to reduce spatial dimensionality. Subsequently, each branch includes four residual blocks, each containing multiple convolutional layers with 3×3 kernels and ReLU activation functions. The features extracted from each branch are then fed into a global average pooling layer, which transforms the spatial feature maps into a single feature vector by averaging each feature map. The extracted feature sequence is first flattened into a one-dimensional vector, and then processed by a four-layer Transformer encoder. This sequence is then embedded into the latent space using a linear embedding layer and a learnable one-dimensional positional encoding. This embedded sequence is projected into multiple key, value, and query spaces. The self-attention mechanism is calculated as follows: Where Q, K, V represent the query, key, and value matrix, respectively; C ' It is the dimension of the key space; T denotes the matrix transpose; The self-attention output is passed through a nonlinear pointwise feedforward layer, followed by residual connection and normalization. Then, the resulting spatiotemporal representation is used to obtain the annotation probability through a linear projection layer and a softmax activation function. Finally, connectionist temporal classification is used as the sequence-to-sequence loss function to align the predicted sign language annotation sequence with the corresponding ground truth labels.

2. The sign language translation method based on the motion sensor of a smart wrist-worn device as described in claim 1, characterized in that, In step 1, when the user makes sign language gestures, accelerometers and gyroscopes worn on the dominant wrist collect acceleration and angular velocity data along the X, Y, and Z axes, which are denoted as a. x ,a y ,a z and g x ,g y ,g z ; First, use a low-pass filter for a x ,a y ,a z and g x ,g y ,g z Low-pass filtering is performed separately to remove random noise from the sensor device hardware in the motion signal; Then, the linear acceleration signal is calculated, and the threshold method is used to determine obvious pauses between hand sentences, thereby identifying the start and end points of the hand sentences and segmenting the motion signals corresponding to the hand sentences.

3. The sign language translation method based on the motion sensor of a smart wrist-worn device as described in claim 1, characterized in that, In step 3, the sign language recognition network first converts motion data into feature vectors, and then passes these vectors to the encoder of the transformer to generate a sequence of annotations; After obtaining the annotated sequence of each spoken sentence, they are passed to a large language model and given custom hints to translate them into meaningful spoken sentences.

4. The sign language translation method based on the motion sensor of a smart wrist-worn device as described in claim 3, characterized in that, ChatGPT and LLaMA2-70b were selected as online and offline large language models, respectively, to generate translation results for handwritten sentences.

5. The sign language translation method based on the motion sensor of a smart wrist-worn device as described in claim 2, characterized in that, When calculating the linear acceleration signal, a 200ms sliding window with 50% overlap is used to calculate the linear acceleration (MLA) within the window: Among them, A x A y A z The average magnitudes of accelerations along the X, Y, and Z axes, respectively, where g is the acceleration due to gravity; The threshold is set to twice the maximum value of the MLA calculated from the first 200ms period; if the MLA calculated in two consecutive windows is greater than / less than the threshold, the starting point of the first window is determined as the start / end point; finally, for each segment of the hand sentence, the triaxial acceleration and triaxial angular velocity data are normalized to [-1,1].

6. The sign language translation method based on the motion sensor of a smart wrist-worn device as described in claim 5, characterized in that, The data points of the motion signal corresponding to each sentence are resampled to 1024 to ensure the consistency of the length of all sample data.

7. The sign language translation method based on the motion sensor of a smart wrist-worn device as described in claim 1, characterized in that, Generator G R2L and G L2R They use the same encoder and decoder structure; The encoder consists of two stacked convolutional neural network layers, interleaved with max pooling layers; the first convolutional layer has 32 kernels and the second convolutional layer has 64 kernels, both of which are 3×3 in size with a stride of 1 and are activated using scaled exponential linear units; each convolutional layer is followed by a 2×2 max pooling layer with a stride of 2 to downsample the feature maps. The decoder consists of two deconvolutional neural network layers, each with 64 and 32 kernels of size 3×3 with a stride of 1 and SELU activations, respectively, interleaved with two unpooling layers that invert the max pooling operation to upsample the data back to its original size.

8. The sign language translation method based on the motion sensor of a smart wrist-worn device as described in claim 1, characterized in that, A dual-discriminator strategy is used, where D t and D f Used to classify the time-domain and frequency-domain responses of real and generated data; The time series signal is input to the time domain discriminator. and The process involves making a judgment; simultaneously, performing a short-time Fourier transform on the time series data to obtain the frequency domain response; then, connecting the six generated spectrograms along a shared time axis to form a feature representation map; finally, the feature representation map serves as a frequency domain discriminator. and The input is ; both discriminators have 4 convolutional layers with 64, 128, 256 and 512 convolutional kernels respectively; the kernel size is 3×3 and the stride is 1; each convolutional layer is followed by layer normalization and rectified linear unit function activation; finally, there is a fully connected layer and a sigmoid activation function.

9. The sign language translation method based on the motion sensor of a smart wrist-worn device as described in claim 1, characterized in that, The training process involves two key loss functions: adversarial loss and cycle consistency loss; the adversarial loss is achieved by using a discriminator D. L and D R Ensure that the generated motion data is indistinguishable from the real motion data; the forward adversarial loss is given by Equations 2 and 3: in, and These are the adversarial losses in the time and frequency domains of the forward process, respectively. E represents the expected value, l represents the input real left-hand motion signal, L represents the generated left-hand motion signal, r represents the input right-hand motion signal, R represents the generated right-hand motion signal, and f represents the short-time Fourier transform of the signal. The backward adversarial loss is: in, and These are the adversarial losses in the time and frequency domains of the backward process, respectively. Cyclic consistency loss enforces the following condition: if motion data is converted from right hand to left hand and then back to right hand, the original right-hand motion data should be recovered. Cyclic consistency loss It is given by the following formula: Where |·|1 represents the L1 norm; The overall objective function is a combination of losses: Where α and β are adversarial loss coefficients, with values ​​ranging from [0, 1]; and λ is the cycle consistency loss coefficient, with values ​​ranging from [0, 1].

Images