Sign language translation method based on motion sensor of intelligent wrist-worn device

Abstract

The invention relates to a sign language translation method based on a motion sensor of an intelligent wrist-worn device, and belongs to the technical field of artificial intelligence and mobile computing application. According to the method, the motion data of the non-habitual hand is generated by establishing the mapping relation of the motion of the two hands and taking the motion data of the habitual hand of the user as the reference, and the motion data of the non-habitual hand is intelligently estimated and combined with the neural network and the large language model, so that the motion data of the non-habitual hand can be intelligently estimated on the premise of not increasing extra equipment burden. Precise recognition and translation of sign languages of both hands are realized. According to the method, seamless integration with the daily life of the user is realized, the design is simple, the method is easy to use, the user can quickly integrate the sign language into a daily use scene without purchasing extra special equipment, the threshold of technology application is reduced, the universality of the sign language translation technology is improved, the translation accuracy is improved, the response real-time performance is ensured, and the user experience is improved. And the communication between the sign language person and the non-sign language person is smoother and barrier-free.

Description

Technical Field

[0001] The present invention relates to a sign language translation method, in particular to a sign language translation method based on a motion sensor of an intelligent wrist-worn device, and belongs to the technical field of artificial intelligence and mobile computing applications. Background Art

[0002] According to the latest survey of the World Health Organization, currently, there are more than 1.5 billion people in the world suffering from hearing impairments of varying degrees. Among them, tens of millions of people rely on sign language as the main way of daily communication and are called signers. The expression of sign language includes the combination of multiple complex elements such as hand shape, hand movements and facial expressions. Each subtle movement or expression may convey different meanings. However, for people who do not understand sign language, it is very difficult to learn and understand these complex expressions in a short period of time. Therefore, there are huge barriers to communication between signers and non-signers.

[0003] Currently, researchers have explored various sign language translation methods to facilitate communication between signers and non-signers.

[0004] The first category is the sign language translation methods based on computer vision, which capture the video recording of the signer through a camera and analyze the signer's hand movements and facial expressions to convert the sign language into spoken language. However, these methods are easily affected by lighting conditions and background interference, resulting in low translation accuracy.

[0005] The second category is the sign language translation method based on wireless signals, which mainly analyzes the changes in the time domain and frequency domain of wireless signals (such as Wi-Fi, millimeter waves, infrared light, and sound waves) caused by sign language gestures. However, the limited portability and high cost of sensing devices make this method difficult to promote and apply on a large scale.

[0006] The third category is the sign language translation method based on wearable devices, which rely on wearable inertial measurement units (IMUs), electromyography (EMG) sensors, photoplethysmography (PPG) sensors and other devices to capture and translate sign language gestures anytime and anywhere. However, these methods can only recognize a few specific gestures, which is difficult to meet the needs of daily communication, or rely on single-handed wearable devices, which have low accuracy when recognizing two-handed sign language gestures.

[0007] Although the existing technology has made some progress in sign language translation, there is no sign language translation method that can be efficient, reliable and low-cost in a variety of environments. Therefore, it is urgent to find a new sign language translation method to better help non-sign language speakers and sign language speakers communicate. Summary of the invention

[0008] The purpose of the present invention is to address the deficiencies and defects of the prior art and to solve the current technical problem of lack of low-cost, high-precision sign language translation, and creatively propose a sign language translation method based on a motion sensor of an intelligent wrist-worn device.

[0009] In conventional technologies for sign language translation using wearable devices, in order to achieve high-precision translation, it is usually necessary to capture the motion signals of both hands at the same time. However, for ordinary users, wearing multiple smart devices in daily life, such as two smart watches, is not only cumbersome to operate, but may also cause embarrassing experiences, affecting the popularity and application effects of the devices. The present invention does not require new users to provide training data, has the significant feature of instant usability, and greatly improves the convenience and adaptability of sign language translation.

[0010] The innovative points of the present invention include: by establishing a mapping relationship between the movements of both hands, the movement data of the user's dominant hand (usually the right hand) is used as a benchmark to generate the movement data of the non-dominant hand (usually the left hand), thereby achieving accurate recognition and translation of two-handed sign language without adding additional equipment burden.

[0011] The present invention is achieved through the following technical solutions.

[0012] A sign language translation method based on a motion sensor of an intelligent wrist-worn device comprises the following steps:

[0013] Step 1: Use the motion sensor in the smart wrist-worn device to collect the motion signal of the user's wrist when performing sign language gestures, remove random noise in the signal, and segment the signal according to sentences.

[0014] Specifically, when the user makes a sign language gesture, the accelerometer and gyroscope worn on the wrist of the dominant hand (usually the right hand) collect the acceleration data and angular velocity data of the X, Y, and Z axes, which are recorded as a and b respectively. x ,a y ,a z and g x ,g y ,g z .

[0015] Step 1.1: Use a low-pass filter to remove random noise introduced by the sensor device hardware from the motion signal.

[0016] Theoretically, the movement frequency of a person's wrist is generally below 10Hz, and the hardware itself brings high-frequency random noise. Therefore, a low-pass fourth-order Butterworth filter can be used to filter a x ,a y ,a z and g x ,g y ,g zPerform low-pass filtering (specifically, it can be set to 10Hz) to remove random noise.

[0017] Step 1.2: Calculate the linear acceleration signal and use the threshold method to determine the obvious pauses between sign language sentences, thereby identifying the starting and ending points of the sign language sentences and segmenting the motion signals corresponding to the sign language sentences.

[0018] Step 2: Input the six-axis motion data of the dominant hand (usually the right hand) into the motion data generation network of the non-dominant hand (usually the left hand), and output the six-axis motion data of the non-dominant hand.

[0019] The non-dominant hand motion data generation network proposed by the present invention includes:

[0020] Forward generator G R2L , which takes the motion data of the right hand as input and generates corresponding motion data for the left hand.

[0021] Backward Generator G L2R , used to perform the inverse operation, input the left hand motion data and generate the corresponding right hand motion data.

[0022] Discriminator D R , used to classify real and generated right-hand motion data. D R It consists of two sub-discriminators: and They are used to classify the time domain and frequency domain responses of a signal, respectively.

[0023] Discriminator D L , used to perform similar judgment on the left hand motion data. L It consists of two sub-discriminators: and They are used to classify the time domain and frequency domain responses of a signal, respectively.

[0024] Step 3: Use the hand motion data as input to a lightweight sign language recognizer to identify the annotation sequence. Provide the annotation sequence and customized prompts as input to the large language model to generate spoken sentences.

[0025] Specifically, the steps include:

[0026] Step 3.1: The sign language recognizer network first converts the motion data into feature vectors, and then passes these vectors to the encoder of the Transformer to generate an annotation sequence.

[0027] Step 3.2: After obtaining the annotated sequences for each sign sentence, pass them to the big language model and input customized prompts to translate into meaningful spoken sentences.

[0028] Specifically, ChatGPT and LLaMA2-70b can be selected as online and offline large language models, respectively, to generate translation results for sign language sentences. The prompts are as follows:

[0029] Explanation: You are a sign language expert.

[0030] Question: Can a meaningful spoken sentence be generated from a sequence of comments entered by the user?

[0031] Rules: Only provide responses that contain generated sentences. If a spoken sentence cannot be created, respond with "No Translation".

[0032] Beneficial Effects

[0033] Compared with the prior art, this method has the following advantages:

[0034] 1. This method achieves seamless integration with the user's daily life. The design is simple and easy to use, and users can quickly integrate it into their daily use scenarios without purchasing additional dedicated equipment. It not only lowers the threshold for technology application, but also increases the popularity of sign language translation technology.

[0035] 2. The present invention provides efficient sign language translation capabilities by intelligently estimating the motion data of the non-dominant hand, combining neural networks and large language models. While maintaining user convenience, the system can accurately and in real time translate the sign language of both hands. It not only improves the accuracy of the translation, but also ensures the real-time response, making the communication between signers and non-signers smoother and barrier-free. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is the principle diagram of the method of the present invention;

[0037] Figure 2 Generate a network structure for the non-dominant hand motion data proposed by the present invention;

[0038] Figure 3 The sign language recognition network structure proposed by the present invention;

[0039] Figure 4 The performance of the embodiment of the present invention in sign language interpretation among 7 participants;

[0040] Figure 5 The performance of translating 50 sign sentences for the embodiment of the present invention;

[0041] Figure 6 The performance of the embodiment of the present invention in performing sign language translation under different training data conditions.

[0042] Figure 7 The performance of the embodiment of the present invention in performing sign language translation under different sign language speed conditions; DETAILED DESCRIPTION

[0043] The method of the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0044] Example

[0045] Due to linguistic diversity and cultural specificity, there are more than 300 sign languages ​​in the world, such as American Sign Language, British Sign Language and Australian Sign Language. Each sign language has a unique grammar, vocabulary and cultural background, making it a complete language system. In this embodiment, the research target is American Sign Language, which is also one of the most studied sign languages.

[0046] like Figure 1 As shown, a sign language translation method based on a motion sensor of a smart wrist-worn device comprises the following steps:

[0047] Step 1: Use the motion sensor in the smart wrist-worn device to collect the motion signal of the user's wrist when performing sign language gestures, remove random noise in the signal, and segment the signal according to sentences.

[0048] Specifically, when the user makes a sign language gesture, the accelerometer and gyroscope worn on the wrist of the dominant hand (usually the right hand) collect acceleration data and angular velocity data of the X, Y, and Z axes, recorded as a x ,a y ,a z and g x ,g y ,g z .

[0049] Step 1.1: Use a low-pass filter to remove random noise introduced by the sensor device hardware from the motion signal.

[0050] Theoretically, the movement frequency of a person's wrist is generally below 10Hz, and the hardware itself brings high-frequency random noise. Therefore, a fourth-order Butterworth low-pass filter can be used to filter a x ,a y ,a z and g x ,g y ,g z Perform low-pass filtering (specifically, it can be set to 10Hz) to remove random noise.

[0051] Step 1.2: Calculate the linear acceleration signal and use the threshold method to determine the obvious pauses between sign language sentences, thereby identifying the starting and ending points of the sign language sentences and segmenting the motion signals corresponding to the sign language sentences.

[0052] Specifically, a sliding window of 200ms with 50% overlap can be used to calculate the linear acceleration MLA within the window:

[0053]

[0054] Among them, A x ,A y ,A z The average acceleration on the X, Y, and Z axes respectively, where g is the acceleration due to gravity.

[0055] According to practical experience, the threshold is set to twice the maximum value of the MLA calculated from the first 200ms period. If the MLA calculated for two consecutive windows is greater than / less than the threshold, the starting point of the first window is determined as the starting point / end point. Finally, for each segmented sign sentence, the three-axis acceleration and three-axis angular velocity data are normalized to [-1,1]. In addition, the data points of the motion signal corresponding to each sentence can be resampled to 1024 to ensure the consistency of the length of all sample data.

[0056] Step 2: Input the six-axis motion data of the dominant hand (usually the right hand) into the motion data generation network of the non-dominant hand (usually the left hand), and output the six-axis motion data of the non-dominant hand.

[0057] Specifically, the framework of the non-dominant hand motion data generation network is as follows: Figure 2 As shown, including:

[0058] Forward generator G R2L , which takes the motion data of the right hand as input and generates corresponding motion data for the left hand.

[0059] Backward Generator G L2R , used to perform the inverse operation, input the left hand motion data and generate the corresponding right hand motion data.

[0060] Discriminator D R , used to classify real and generated right-hand motion data. It includes two sub-discriminators: and They are used to classify the time domain and frequency domain responses of a signal, respectively.

[0061] Discriminator D L A similar judgment is performed on the left hand motion data, including two sub-discriminators: and They are used to classify the time domain and frequency domain responses of a signal, respectively.

[0062] Among them, the generator G R2L and G L2RThe same encoder and decoder architecture is used. The input is 6×1024 data points, since each sign sentence has 6 channels of signals: X, Y, and Z axes of acceleration and angular velocity data. The encoder stage consists of two stacked convolutional neural network (CNN) layers interleaved with max pooling layers. The first convolutional layer has 32 kernels and the second has 64 kernels, both of size 3×3, stride 1, and use scaled exponential linear unit (SELU) activations. Each convolutional layer is followed by a max pooling layer of size 2×2 and stride 2 to downsample the feature maps. The decoder stage consists of two deconvolutional neural network (DCNN) layers, each with 64 and 32 kernels of size 3×3, stride 1, and SELU activations, interleaved with two unpooling layers that reverse the max pooling operation and upsample the data back to the original size.

[0063] Considering that the hand movement of sign language gestures is reflected in both time domain and frequency domain, the present invention uses a dual discriminator strategy, where D t and D f It is used to classify the time domain and frequency domain responses of real data and generated data. Specifically, the time series signal is input into the time domain discriminator and At the same time, the time series data is subjected to a short-time Fourier transform (STFT) to obtain the frequency domain response. Then, the six generated spectrograms are connected along the shared time axis to form a feature representation graph. Finally, the feature representation graph is used as the frequency domain discriminator. and The input of both discriminators is . Both discriminators have 4 convolutional layers with 64, 128, 256, and 512 kernels, respectively. The kernel size is 3×3 and the stride is 1. Each convolutional layer is followed by layer normalization and rectified linear unit (ReLU) activation. Finally, there is a fully connected layer and sigmoid activation function.

[0064] The training process involves two key loss functions: adversarial loss and cycle consistency loss. The adversarial loss is implemented by using the discriminator D L and D R Ensure that the generated motion data is indistinguishable from the real motion data. The forward adversarial loss is given by Equation 2 and Equation 3:

[0065]

[0066] in, and are the adversarial losses in the time domain and frequency domain of the forward process respectively, E represents the expected value, l represents the real input left-hand motion signal, L represents the generated left-hand motion signal, r represents the input right-hand motion signal, R represents the generated right-hand motion signal, and f represents the short-time Fourier transform performed on the signal.

[0067] Similarly, the backward adversarial loss is:

[0068]

[0069]

[0070] in, and They are the adversarial losses in the time domain and frequency domain of the backward process respectively.

[0071] The cycle consistency loss enforces the condition that if you convert motion data from the right hand to the left hand and then back to the right hand, the original right hand motion data should be recovered (and vice versa). Given by:

[0072]

[0073] Here, |·|1 represents the L1 norm.

[0074] The overall objective function is a combination of these losses:

[0075]

[0076] Among them, α and β are adversarial loss coefficients, respectively, and their values ​​can be 0.2 and 0.2. It should be noted that other settings belonging to [0, 1] are also within the scope of the present invention; λ is the cycle consistency loss coefficient, and its value is 0.5. It should be noted that other settings belonging to [0, 1] are also within the scope of the present invention.

[0077] Step 3: Use the hand motion data as input to a lightweight sign language recognizer to identify the annotation sequence. Provide the annotation sequence and customized prompts as input to the large language model to generate spoken sentences.

[0078] Specifically, the steps include:

[0079] Step 3.1: The sign language recognizer network first converts the motion data into feature vectors, and then passes these vectors to the encoder of the Transformer to generate an annotation sequence.

[0080] In sign language, a gloss is a written representation that uses words from the spoken language to label and describe individual signs while maintaining the original sign order and grammar. It is important to note that spoken and signed languages ​​have different grammatical structures, and the order in which words are presented is also different.

[0081] First, a short-time Fourier transform is applied to the motion data of both hands, with the frame length and window step size set to 256 and 128. Then, the six-axis motion data of each hand are concatenated into a single time-frequency representation. Therefore, each sign language sentence is represented by two spectrograms as the input of the sign language recognizer network.

[0082] The lightweight feature extraction network uses a two-dimensional convolutional neural network to process each spectrogram into a feature vector, such as Figure 3 As shown in the figure. The network contains two parallel branches based on the ResNet-34 architecture, each branch is dedicated to processing the spectrograms from the dominant hand and the non-dominant hand respectively. The first layer in each branch is a convolutional layer that applies 64 kernels of size 3×3, followed by a maximum pooling layer with size 2×2 to reduce the spatial dimension. Subsequently, each branch includes four residual blocks, each of which contains multiple convolutional layers with kernels of size 3×3 and ReLU activation function. The features extracted from each branch are then input into the global average pooling layer, which converts the spatial feature maps into a single feature vector by averaging each feature map. This structure enables the network to capture local and global patterns in the time-frequency domain, leveraging the complementary information of both hands to improve the accuracy of sign language annotation recognition.

[0083] The extracted feature sequence is first flattened into a one-dimensional vector and then processed by a four-layer Transformer encoder. The Transformer encoder is essential for processing sequential data and is particularly effective in natural language processing tasks due to its self-attention mechanism. To embed the sequence into the latent space, the present invention utilizes a linear embedding layer along with a learnable one-dimensional positional encoding. The embedded sequence is projected into multiple key, value, and query spaces. The self-attention mechanism Attention is calculated as follows:

[0084]

[0085] Where Q, K, V represent query, key, and value matrices; C ′ is the dimension of the key space. T represents the matrix transpose.

[0086] This mechanism enables the model to effectively capture temporal dependencies and relations in sequential data, leveraging the self-attention mechanism to enhance feature representation.

[0087] The self-attention output is passed through a nonlinear point-wise feed-forward layer, followed by residual connections and normalization to promote training stability. Afterwards, the resulting spatiotemporal representation is used to obtain annotation probabilities through a linear projection layer and a Softmax activation function. Finally, connectionist temporal classification (CTC) is adopted as the sequence-to-sequence loss function to align the predicted sign language annotation sequence with the corresponding true label. CTC is particularly suitable for classification problems involving temporal data because it can calculate the difference between the predicted sequence and the actual label without pre-segmenting the training data.

[0088] Step 3.2: After obtaining the annotated sequences for each sign sentence, pass them to the big language model and input customized prompts to translate into meaningful spoken sentences.

[0089] ChatGPT and LLaMA2-70b are selected as online and offline large language models respectively to generate translation results of sign sentences. The prompt is as follows:

[0090] Explanation: You are a sign language expert.

[0091] Question: Can a meaningful spoken sentence be generated from a sequence of comments entered by the user?

[0092] Rules: Only provide responses that contain generated sentences. If you cannot create a spoken sentence, answer "No Translation".

[0093] Experimental verification

[0094] To verify the performance of the present invention, the present invention was developed into a wrist-worn prototype, which includes an integrated inertial motion sensor, including a three-axis accelerometer and a three-axis gyroscope, which is equivalent to a smart watch or smart bracelet. The inertial motion sensor readings are recorded at a sampling rate of 100Hz and processed by a laptop, which is paired with the prototype via Bluetooth. The laptop is equipped with an Intel Core i7-12700H processor, 16GB RAM, and an Nvidia GeForceRTX 4060 graphics card. A camera is placed in front of the participant to provide real-life conditions.

[0095] A total of 27 participants (14 males and 13 females) were recruited, ranging in age from 18 to 52 years old. All participants were hearing, right-handed, and had varying levels of sign language knowledge. The sign language corpus consisted of 250 American Sign Language sentences. The participants were divided into two groups: The first group consisted of 20 participants who wore the prototype on both wrists and performed each sentence 5 times. The training data consisted of 200 sentences. The second group consisted of an additional 7 participants who wore the prototype on both wrists and performed all the sentences 5 times each. The data collected from the second group was mainly used for testing purposes.

[0096] The word error rate (WER) is used as an evaluation metric to measure the minimum number of insertion, deletion, and substitution operations required to change the recognized text sentence to the true value. WER is calculated as follows:

[0097]

[0098] Where S is the number of substitutions, D is the number of deletions, I is the number of insertions, and N is the total number of words in the ground truth. A lower WER indicates better performance, and a WER of 0 indicates a perfect translation.

[0099] First, the performance of the present invention for new users is tested, and user-independent testing is performed using the second group of data. Figure 4 The WER of each participant is shown. It can be seen that the WER of the seven users is less than 10%. The highest, lowest and average WER are 8.3%, 2.2% and 4.8% respectively. In addition, participants 3 and 4 have the lowest WER. They are both native English speakers and have relatively rich experience in sign language, so the sensor can capture more stable and reliable information. Overall, the present invention is very effective for promotion to new users and meets the design consideration of immediate usability.

[0100] Second, to understand whether the invention can be generalized to new sign sentences without retraining or tuning, an unseen sentence test was conducted. Specifically, the data of 50 sign sentences in the first group that were not in the training dataset were used to control for individual sentence variables. Figure 5 The WER for each unseen sentence is shown. The WER results vary between 1.4% and 16.5%, with an average error of 8.3%. By inspecting the content of the sentences, it is observed that sentences containing words that appear more frequently in the training data tend to have lower WER. Overall, this performance demonstrates the generalization ability of the invention as it can effectively recognize new sentence structures and vocabulary without further tuning or training.

[0101] Afterwards, the effectiveness of the non-dominant hand motion data generation network of the present invention was verified. The following model variants were all generated from the sign language recognizer model: Model 1 was built using real motion data from both the dominant and non-dominant hands. Model 2 was built using real motion data from the dominant hand and generated motion data from the non-dominant hand. Model 3 was built using motion data from the dominant hand only. The three models were evaluated using the WER metric of sign language annotations in user-independent and unseen sentence tests. Figure 6The results shown in show that Model 2 performs comparably to Model 1, with only minor differences in WER (e.g., 0.4% and 0.2% for the two tests, respectively). This suggests that the generated non-dominant hand motion data can be effectively used for sign language translation without significantly affecting performance. In addition, Model 3 exhibits the worst performance, highlighting the importance of non-dominant hand data in maintaining sign language translation performance.

[0102] In order to evaluate the effects of different signing speeds on the performance of the present invention, an experiment was conducted in which participants in the second group were asked to perform signing at three different speeds: approximately 1.0 word / second, 1.5 words / second, and 1.9 words / second, representing slow, normal, and fast signing speeds, respectively. Figure 7 The WER at different signing speeds is shown. It can be seen that the WER fluctuates slightly as the signing speed increases. However, even at high speeds, the WER is still below 10%, which is due to the resampling mechanism adding fewer data samples within a fixed sampling rate, thus compensating for the difference in signing speed.

[0103] The specific description above further illustrates in detail the purpose, technical solutions and beneficial effects of the invention. It should be understood that the above is only a specific embodiment of the present invention, which is used to explain the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A sign language translation method based on a motion sensor of an intelligent wrist-worn device, characterized in that: The following steps are involved: Step 1: Use the motion sensor in the smart wrist-worn device to collect the motion signal of the user's wrist when performing sign language gestures, remove random noise in the signal, and segment the signal according to sentences; Step 2: Input the six-axis motion data of the dominant hand into the non-dominant hand motion data generation network, and output the six-axis motion data of the non-dominant hand; Among them, the non-dominant hand motion data generation network includes: Forward generator G R2L , used to take the motion data of the right hand as input and generate corresponding motion data for the left hand; Backward Generator G L2R , used to perform inverse operation, input left hand motion data and generate corresponding right hand motion data; Discriminator D R , used to classify real and generated right-hand motion data; D R It consists of two sub-discriminators: and They are used to classify the time domain and frequency domain responses of the signal respectively; Discriminator D L , used to perform similar judgment on the left hand motion data; D L It consists of two sub-discriminators: and They are used to classify the time domain and frequency domain responses of the signal respectively; Step 3: Use the two-hand motion data as input to a lightweight sign language recognizer to recognize the annotation sequence; provide the annotation sequence and customized prompts as input to the large language model to generate spoken sentences.

2. A sign language translation method based on a motion sensor of a smart wrist-worn device as claimed in claim 1, characterized in that: In step 1, when the user makes a sign language gesture, the acceleration data and angular velocity data of the X, Y, and Z axes are collected by the accelerometer and gyroscope worn on the wrist of the dominant hand, which are recorded as a and x ,a y ,a z and g x ,g y ,g z ; First, use a low-pass filter to x ,a y ,a z and g x ,g y ,g z Perform low-pass filtering to remove random noise caused by sensor device hardware in motion signals; Then, the linear acceleration signal is calculated, and the threshold method is used to determine the obvious pauses between sign sentences, so as to identify the starting and ending points of the sign sentences and segment the motion signals corresponding to the sign sentences.

3. A sign language translation method based on a motion sensor of a smart wrist-worn device as claimed in claim 1, characterized in that: In step 3, the sign language recognizer network first converts the motion data into feature vectors, and then passes these vectors to the encoder of the transformer to generate an annotation sequence; After obtaining the annotated sequences for each sign sentence, they are passed to a large language model and input with customized prompts to translate into meaningful spoken sentences.

4. A sign language translation method based on a motion sensor of a smart wrist-worn device as claimed in claim 3, characterized in that: ChatGPT and LLaMA2-70b are selected as online and offline large language models, respectively, to generate translation results of sign sentences.

5. A sign language translation method based on a motion sensor of a smart wrist-worn device as claimed in claim 2, characterized in that: When calculating the linear acceleration signal, a 200ms sliding window with 50% overlap is used to calculate the linear acceleration MLA within the window: Among them, A x ,A y ,A z The average acceleration on the X, Y, and Z axes, respectively, g is the acceleration due to gravity; The threshold is set to twice the maximum value of the MLA calculated from the previous 200ms period; if the MLA calculated in two consecutive windows is greater than / less than the threshold, the starting point of the first window is determined as the starting point / end point; finally, for each segmented sign sentence, the three-axis acceleration and three-axis angular velocity data are normalized to [-1,1].

6. A sign language translation method based on a motion sensor of a smart wrist-worn device as claimed in claim 5, characterized in that: The data points of the motion signal corresponding to each sentence are resampled to 1024 to ensure the consistency of the length of all sample data.

7. A sign language translation method based on a motion sensor of a smart wrist-worn device as claimed in claim 1, characterized in that: Generator G R2L and G L2R Use the same encoder and decoder structure; The encoder consists of two stacked convolutional neural network layers interleaved with max pooling layers; the first convolutional layer has 32 kernels and the second has 64 kernels, both of which are 3×3 in size, stride 1, and activated with scaled exponential linear units; each convolutional layer is followed by a max pooling layer of 2×2 with stride 2 to downsample the feature maps; The decoder consists of two deconvolutional neural network layers, each with 64 and 32 kernels of size 3×3, stride 1 and SELU activation, interleaved with two unpooling layers that invert the max pooling operation and upsample the data back to its original size.

8. The sign language translation method based on the motion sensor of the smart wrist-worn device according to claim 1, characterized in that: Using a dual discriminator strategy, where D t and D f Used to classify time-domain and frequency-domain responses of real and generated data; The time series signal is input to the time domain discriminator and At the same time, the time series data is short-time Fourier transformed to obtain the frequency domain response; then, the six generated spectrograms are connected according to the shared time axis to form a feature representation graph; finally, the feature representation graph is used as the frequency domain discriminator and The input of both discriminators is 4 convolutional layers with 64, 128, 256 and 512 convolution kernels respectively; the kernel size is 3×3 and the stride is 1; each convolutional layer is followed by layer normalization and rectified linear unit function activation; and finally there is a fully connected layer and sigmoid activation function.

9. The sign language translation method based on the motion sensor of the smart wrist-worn device according to claim 1, characterized in that: The training process involves two key loss functions: adversarial loss and cycle consistency loss. The adversarial loss is achieved by using the discriminator D L and D R Ensure that the generated motion data is indistinguishable from the real motion data; the forward adversarial loss is given by Equation 2 and Equation 3: in, and are the adversarial losses in the time domain and frequency domain of the forward process, respectively. E represents the expected value, l represents the real input left-hand motion signal, L represents the generated left-hand motion signal, r represents the input right-hand motion signal, R represents the generated right-hand motion signal, and f represents the short-time Fourier transform performed on the signal. The backward adversarial loss is: in, and They are the adversarial losses in the time and frequency domains of the backward process, respectively; The cycle consistency loss enforces the following condition: if the motion data is converted from the right hand to the left hand and then converted back to the right hand, the original right hand motion data should be recovered; Cycle consistency loss Given by: Among them, |·|1 represents the L1 norm; The overall objective function is a combination of losses: Among them, α and β are adversarial loss coefficients, whose value range is [0, 1]; λ is the cycle consistency loss coefficient, whose value range is [0, 1].

10. A sign language translation method based on a motion sensor of a smart wrist-worn device as claimed in claim 1 or 3, characterized in that: First, short-time Fourier transform is applied to the motion data of both hands, with the frame length and window step size set to 256 and 128; then, the six-axis motion data of each hand are concatenated into a single time-frequency representation; therefore, each sign language sentence is represented by two spectrograms as the input of the sign language recognizer network; A lightweight feature extraction network processes each spectrogram into a feature vector using a two-dimensional convolutional neural network. The network contains two parallel branches based on the ResNet-34 architecture, each dedicated to processing spectrograms from the dominant and non-dominant hands respectively. The first layer in each branch is a convolutional layer that applies 64 kernels of size 3×3, followed by a maximum pooling layer with size 2×2 to reduce the spatial dimension. Subsequently, each branch includes four residual blocks, each of which contains multiple convolutional layers with 3×3 kernels and ReLU activation functions; Then, the features extracted from each branch are input into the global average pooling layer, which converts the spatial feature map into a single feature vector by averaging each feature map; The extracted feature sequence is first flattened into a one-dimensional vector and then processed by a four-layer Transformer encoder; a linear embedding layer is used along with a learnable one-dimensional positional encoding; the embedded sequence is projected into multiple key, value, and query spaces; the self-attention mechanism Attention is calculated as follows: Where Q, K, V represent query, key, and value matrices; C ′ is the dimension of the key space; T represents the matrix transpose; The self-attention output is passed through a nonlinear point-wise feed-forward layer, followed by a residual connection and normalization to promote training stability; the resulting spatiotemporal representation is then used to pass through a linear projection layer and a Softmax activation function. The number is used to obtain the annotation probability; finally, connectionist temporal classification is used as the sequence-to-sequence loss function. Align the predicted sign language annotation sequences with the corresponding true labels.

Images