Neck-hung visual recognition sign language device and implementation method thereof

Abstract

The application discloses a hanging-neck type visual recognition sign language device and an implementation method thereof. The hanging-neck type visual recognition sign language device comprises a hanging-neck belt, two ends of the hanging-neck belt are connected with a box body through connecting lines, at least one sign language collection camera capable of shooting hand actions is arranged on the hanging-neck belt and faces outward, and at least one expression camera capable of shooting faces is arranged above the box body. The application has the advantages of high sign language translation semantic precision and fast translation speed.

Description

Technical Field

[0001] This invention belongs to the field of computer vision recognition devices, specifically relating to a neck-mounted visual sign language recognition device and its implementation method that features high semantic accuracy and fast translation speed. Background Technology

[0002] Real-time sign language translation systems fall at the intersection of computer vision, natural language processing, and human-computer interaction. Their goal is to translate the gestures of hearing-impaired individuals into readable and audible natural language in real time. Currently, approximately 70 million people worldwide with severe hearing impairments rely on sign language for daily communication, while the hearing population generally lacks sign language knowledge, resulting in a gap in two-way "gesture-language" communication. Although non-contact solutions based on monocular cameras have become mainstream, in real-world scenarios, the three core pain points of "poor real-time performance, semantic drift, and low portability" still cannot simultaneously meet application requirements, forming the overall bottleneck restricting the practical application of this technology.

[0003] The aforementioned bottleneck stems from three contradictions: First, the size of sign language datasets is insufficient; the largest existing public dataset is only about 200 hours long, making it difficult to support high-capacity model training. Second, sign language movements have high-frequency, subtle changes and long temporal dependencies, and traditional CNN and RNN structures either have limited receptive fields or excessive inference latency in spatiotemporal modeling. Third, sign language word order differs significantly from written Chinese; sentences generated directly by end-to-end models often have word order inversions and missing components, causing semantic drift. The limited computing power of portable devices further amplifies the "accuracy-latency" contradiction.

[0004] To address these challenges, the academic community has proposed two improvement approaches: one focuses on the "data-model" side, enhancing spatiotemporal representation capabilities through 3DCNNs, Transformers, or graph convolutions, and attempting to achieve lightweighting through knowledge distillation and pruning compression; the other focuses on the "word order" side, employing an encoder-decoder architecture, first using a visual model to output sign language gloss, and then using a language model to rearrange it into natural word order. However, the former still relies on large-scale labeled data and its robustness drops sharply in occluded and dynamically blurred scenarios; the latter's staged training leads to error accumulation, and the high cost of gloss annotation makes it difficult to generalize to sign languages ​​of multiple dialects.

[0005] Recent research has merged two approaches into an end-to-end framework of "single-frame 3D keypoints + large language model," aiming to replace the original video with keypoint sequences and reduce reliance on large-scale annotation. However, this approach presents new dilemmas: on the one hand, keypoint errors are amplified temporally, especially during rapid finger spelling or hand crossing, where even minor perturbations can lead to word-level mistranslations; on the other hand, the pure neural network rearrangement module introduced to correct word order, in a flexible and analytical language like Chinese, suffers from unstable output due to the lack of explicit syntactic constraints, with the BLEU-1 index decreasing exponentially with sentence length. Many sign languages ​​require a combination of hand gestures and facial expressions to achieve accurate semantics, but existing technologies only focus on hand gestures, resulting in low sign language translation accuracy. Current technologies cannot simultaneously address the three inherent defects of "keypoint drift," "word order instability," and "low accuracy in sign language translation with facial expressions," leaving a gap in high-precision, low-latency, and portable real-time sign language translation systems, urgently requiring new system-level inventions. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes a neck-mounted visual sign language recognition device with high semantic accuracy and fast translation speed.

[0007] The present invention is achieved through the following technical measures: a neck strap visual recognition sign language device, comprising a neck strap, the two ends of which are connected to a box via connecting lines, at least one sign language acquisition camera facing outwards that can capture hand movements is provided on the neck strap, and at least one facial expression camera that can capture facial expressions is provided above the box.

[0008] As a preferred embodiment, the facial expression camera is rotatably mounted on the outer edge of the upper part of the box.

[0009] As a preferred embodiment, earphones are also provided on both sides of the neck strap.

[0010] This invention also discloses a method for implementing a neckband-style visual sign language recognition system, comprising the following steps:

[0011] S2: Acquire continuous sign language videos, reconstruct the hand mesh for each frame of the sign language video, and calculate the 2D key point data of the hand using the DoF parameters of the intermediate hand skeleton.

[0012] S3: Acquire facial expression video, compare the facial expression with the corresponding frame image in the sign language video, adjust the hand mesh, and finally output the hand mesh and 2D key point data of the hand;

[0013] S4: Use 2D key point data of the hand to complete the division of hand mesh in continuous sign language video;

[0014] S5: Based on the hand mesh segmentation in S4, obtain the corresponding isolated words in the "hand mesh--isolated words" dataset model;

[0015] S6: The obtained series of isolated words are processed by a large language model to output a clear and fluent sentence.

[0016] As a preferred method, step S1 is also included:

[0017] A "hand mesh-isolated word" dataset model was established. The word segmentation model was trained by sign language videos to complete the hand mesh modeling process and generate the "hand mesh-isolated word" dataset model.

[0018] As a preferred embodiment, step S1 specifically includes the following steps:

[0019] S11: Collect sign language videos, hand meshes, and 2D key points of the hand in the motion capture studio, while manually completing gloss-level word segmentation;

[0020] S12: Based on the dataset obtained in S11 and the sign language dataset CSL, complete the hand mesh reconstruction model and generate the "hand mesh-isolated word" dataset model.

[0021] As a preferred approach, the large language model in S6 is the Llama2-7B local large language model.

[0022] As a preferred embodiment, step S2 specifically includes the following steps:

[0023] S21: Data preprocessing: Capture sign language video frame images through a camera, normalize the pixels of each frame of the input sign language video, and divide it into non-overlapping image blocks of 16×16 pixels, and calculate the embedding vector for each block.

[0024] S22: ViT-H is used as the backbone network to extract global context features of image patches, and hand pose parameters are regressed through a Transformer decoder. Hand shape parameters Camera external parameters ;

[0025] S23: Input the regressed hand pose and hand shape parameters into the MANO parametric model to generate hand mesh vertex coordinates. Coordinates of 21 3D key points on the hand ;

[0026] S24: Project the 3D keypoint X onto the image plane using pre-calibrated camera extrinsic parameters and fixed intrinsic parameters to obtain the 2D keypoint coordinates. ;

[0027] S25: Synchronously cache the hand mesh vertex sequence for each frame of video. 2D keypoint sequence .

[0028] As a preferred embodiment, step S4 specifically includes the following steps:

[0029] S41: Retrieve the cached temporal 2D keypoint sequence of the hand. After translating with the wrist as the origin, the sequence is standardized according to the mean and variance of the whole sequence, and short segments are extracted using a sliding window of predetermined length and predetermined step size.

[0030] S42: Calculate inter-frame displacement frame by frame within each short segment. speed change and display area Three types of lightweight dynamic features are concatenated into a 63-dimensional feature vector F;

[0031] S43: Input the feature sequence into a two-layer 64-dimensional bidirectional LSTM followed by a BiLSTM-Segmenter with Sigmoid() output to obtain the probability of each frame being an isolated word boundary. ;

[0032] S44: Yes Threshold binarization, followed by nonmaximum suppression and minimum duration filtering, yields the final boundary, which segments the continuous sign language video into isolated word units and outputs start and end time codes.

[0033] As a preferred embodiment, step S5 specifically includes the following steps:

[0034] S51: Obtain the hand mesh vertex sequence corresponding to each isolated word unit output by S4. Render a continuous depth map sequence from a unified perspective. And normalize the depth map frame by frame;

[0035] S52: Include 21 2D key points for the hand Expanded to 21×3 dimensions, with added depth values. Obtain the 3D key point sequence of the hand. Then, the depth map sequences are aligned according to the time window to construct a time-consistent cross-modal input;

[0036] S53: Sequence of 3D key points for the hand Input a three-layer spatial GCN to generate hand pose features ,Will Lightweight 3D-CNN generates deep visual features Using the Prior-guided Fusion module in the Uni-Sign framework, the hand's 3D keypoint coordinates are used as priors to... and Frame-by-frame fusion yields unified features ;

[0037] S54: Will After being projected onto the word embedding dimension of a large language model, the text is fed into a pre-trained language model of Uni-Sign to generate isolated word semantic text in an autoregressive manner. Complete isolated word prediction.

[0038] As a preferred embodiment, step S6 specifically includes the following steps:

[0039] S61: Collect translated sentences that correspond one-to-one with isolated words from the CSL-Daily dataset and construct training samples. C is the sentence with the correct word order;

[0040] S62: Generate a randomized input sequence from the N isolated word semantic texts output by S5 using a random permutation strategy. And add [CLS] and [SEP] markers at the beginning and end of the sequence respectively;

[0041] S63: Will Input the Deep Seek-Coder-7B-Instruct model, use Uni-Sign pre-trained parameters as initialization, freeze all weights except the last three Transformer layers, and generate the target correct statement in an autoregressive manner. ;

[0042] S64: Calculate using BERT-Score The three metrics of R, P, and F1 are compared with real statements from CSL-Daily, with F1 used as the differentiable loss. Fine-tune the weights of the last three layers of Deep Seek;

[0043] S65: After training, the inference phase directly inputs a disordered sequence of isolated words, and the Deep Seek model outputs a semantically coherent and grammatically correct final translation, achieving real-time sign language sentence translation.

[0044] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described above.

[0045] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0046] This invention's device is equipped with a sign language capture camera that can capture hand gestures and a facial expression camera that can capture facial expressions, simultaneously capturing sign language gestures and expressions. Through the interaction and supplementation of these two cameras, the accuracy of sign language translation can be greatly improved. Using the sign language video and facial expressions captured by the device's cameras, the corresponding frame images are obtained, and the hand mesh is reconstructed and 2D key points of the hand are calculated. Then, the effective hand mesh is segmented using the 2D key points. The hand mesh is adjusted by comparing each frame image of the facial expressions with the corresponding images in the sign language video, and finally, the hand mesh and 2D key point data are output. Next, the corresponding isolated words are obtained from the "hand mesh-isolated word" dataset model, and a clear and fluent sentence is output using a large language model. Finally, the fluent sentence is displayed as text or played back as audio, thus possessing the advantages of high semantic accuracy and fast translation speed in sign language translation. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the structure of an embodiment of the neckband-style visual sign language recognition device of the present invention.

[0048] Figure 2 This is a flowchart illustrating an embodiment of the neckband-style visual sign language recognition method of the present invention. Detailed Implementation

[0049] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the present invention will be further described in detail below with reference to the embodiments and the accompanying drawings.

[0050] A neck-mounted visual sign language recognition device, reference Figure 1 The device includes a neck strap 1, with two ends of the neck strap 1 connected to a housing 5 via a connecting line 3. The neck strap 1 is equipped with at least one sign language capture camera 2 facing outwards to capture hand movements, and the housing 5 is equipped with at least one facial expression camera 4.

[0051] This device simultaneously captures sign language gestures and facial expressions using a sign language capture camera 2 that captures hand movements and a facial expression capture camera 4. Through the interaction and supplementation of these two cameras, the accuracy of sign language translation can be greatly improved.

[0052] In an embodiment of the neck-hook visual recognition sign language device, reference is made to... Figure 1 Based on the previous technical solution, the facial expression camera 4 is rotated and positioned on the outer edge of the upper part of the box body 5, so that the facial expression camera 4 can be rotated to adapt to different people.

[0053] In an embodiment of the neckband-style visual sign language recognition device, based on the preceding technical solution, earphones are also provided on both sides of the neckband 1 to play the translated semantics to the sign language interpreter, so that errors in the translation can be corrected in a timely manner.

[0054] The translation method implemented using the aforementioned neckband-style visual recognition sign language device is referenced. Figure 2 It includes the following steps:

[0055] S1: Initialization, establish a unique Chinese sign language dataset model of "hand mesh - isolated words", train the word segmentation model through sign language videos, complete the hand mesh modeling process, and generate the "hand mesh - isolated words" dataset model;

[0056] S2: Capture continuous sign language videos using cameras and other devices, reconstruct the hand mesh (3D hand mesh) for each frame, and calculate the 2D key points of the hand using the DoF parameters (degrees of freedom parameters) of the hand skeleton from the intermediate results. Finally, output the hand mesh and 2D key point data.

[0057] S3: Acquire facial expression video, compare the facial expression with the corresponding frame image in the sign language video, adjust the hand mesh, and finally output the hand mesh and 2D key point data of the hand;

[0058] S4: Use 2D key point data of the hand to complete the division of hand mesh in continuous sign language video;

[0059] S5: Based on the hand mesh segmentation in S4, obtain the corresponding isolated words in the "hand mesh--isolated words" dataset model;

[0060] S6: The predicted series of isolated words are processed by the Llama2-7B local large language model to output a fluent sentence with clear meaning.

[0061] Step S1 is not a necessary step to implement this method. It can be generated by first collecting sign language videos by professionals and then performing the acquisition and calculation through specialized equipment. Alternatively, it can be a functional module of the product, which can collect sign language videos of specific people according to the user's habits and preferences and then perform the acquisition and calculation through its own functional module.

[0062] This real-time sign language translation method uses sign language videos and facial expressions captured by the signer's own camera to reconstruct the hand mesh and calculate 2D key points. The hand mesh is adjusted by comparing each frame of the facial expression with the corresponding frame in the sign language video. Then, the effective hand mesh is segmented using the 2D key points. Next, the corresponding isolated words are obtained from the "hand mesh-isolated word" dataset model, and a large language model is used to output a clear and fluent sentence. Finally, the fluent sentence is displayed as text or played aloud, thus offering advantages such as high semantic accuracy and fast translation speed in sign language translation. To evaluate the semantic performance of this method in sign language translation, the Bert-score is used as the core evaluation metric for semantic consistency. This is achieved by extracting the generated sentence... The token-level cosine similarity between the model and the context embedding representation of the real sentence C is calculated and weighted to obtain the Precision, Recall and F1_Bert scores. The experiment was conducted on the CSL-Daily test set. The model achieved a Bert-score F1 mean of 0.849 under the condition of disordered isolated word input, which verifies the effectiveness of the three-layer fine-tuning strategy in maintaining semantic consistency and grammatical correctness.

[0063] In an embodiment of the neck-hanging visual sign language recognition method, step S1, based on the preceding technical solution, specifically includes the following sub-steps:

[0064] S11: Collect sign language videos, hand meshes, and 2D key points of the hand in the motion capture studio, while manually completing gloss-level word segmentation;

[0065] S12: Based on the dataset in S11 and the current largest Chinese sign language dataset CSL, complete the hand mesh reconstruction model and generate the "hand mesh-isolated word" dataset model.

[0066] In an embodiment of the neck-hook visual sign language recognition method, step S2, based on the preceding technical solution, specifically includes the following sub-steps:

[0067] S21: Data preprocessing, capturing sign language video frame images via camera, and processing each frame of the input sign language video. Using ImageNet's mean Standard deviation Normalize and divide into A non-overlapping 16×16 image block, image block Embedded by linear projection Add 1D learnable positional encoding ;

[0068] S22: Using ViT-Huge (ViT-H) as the backbone network, global contextual features of image patches are extracted, and after processing by a 12-layer Transformer encoder, the output feature sequence is obtained. The sequence is then subjected to average pooling to obtain image-level feature representations. The data is fed into a 3-layer 512-dimensional decoder to regress 61-dimensional parameters: hand posture. (16 joints × 3DoF), hand shape parameters Camera external parameters Translation ;

[0069] S23: Regressed hand posture parameters and hand shape parameters enter Model: 778 hand mesh vertices were obtained. , and coordinates of 21 3D key points ;

[0070] S24: By pre-calibrating camera extrinsic parameters , And a fixed internal parameter matrix to define the coordinates of 3D key points Projecting onto the image plane yields the 2D keypoint coordinate projection. ,in ;

[0071] S25: Synchronously cache the hand mesh vertex sequence for each frame of video. With 2D keypoint sequence This is for use by subsequent modules;

[0072] S26: Performance evaluation of 2D keypoint detection for the hand. To verify the accuracy of 2D keypoint localization in a real-world scenario, the 2D keypoint sequence output in S25 above is used in a self-built dataset. , As an evaluation metric, experimental results show that the proposed method achieves a PCK@0.05 score of 47.5 on the test set.

[0073] In an embodiment of the neck-hook visual sign language recognition method, step S4, based on the preceding technical solution, specifically includes the following sub-steps:

[0074] S41: Retrieve the cached 2D keypoint sequence ,in This represents the k-th joint in frame t, using the wrist point as an example. Translate the origin Then calculate the global mean by channel. and variance Standardization Then, a short segment is extracted using a sliding window with a length of 50 frames and a step size of 10 frames. ;

[0075] S42: In the fragment Calculate three types of dynamic features frame by frame – inter-frame displacement speed change , exhibition width After concatenation, a frame-level feature vector is formed. Fragment feature matrix ;

[0076] S43: Transform the segment feature matrix Inputting two layers of a bidirectional LSTM (Bidirectional Long Short-Term Memory) with a hidden dimension of 256, we obtain the forward and backward hidden layers, which are then concatenated. Then through a linear layer Output Frame The probability of the boundary ;

[0077] S44: For all With threshold Binarization yields the candidate boundary set, followed by ±5 frames of non-maximum suppression: if In Centered, half the width of the window If the value is not the largest in its neighborhood, it is discarded; finally, it is sorted by shortest duration. Filter and retain those that meet the requirements. The boundary is defined to segment continuous sign language videos into isolated word units and output start and end time codes. ;

[0078] S45: To verify the boundary detection accuracy of this invention in real-world scenarios, Dynamic Time Warping Accuracy (DTWA) was used as the core metric on a self-built dataset, allowing a maximum offset of 15 frames (0.6 seconds) between the predicted boundary and the ground truth boundary. Experimental results show that the DTW-15 error of this method on the test set is 8.8%, verifying the effectiveness of this invention in natural sign language videos.

[0079] In an embodiment of the neck-hook visual sign language recognition method, step S5, based on the preceding technical solution, specifically includes the following sub-steps:

[0080] S51: Obtain the hand mesh vertex sequence corresponding to the j-th isolated word unit output by S44. Depth map sequences are obtained by rendering according to unified virtual camera parameters. Normalize the depth map frame by frame to obtain ;

[0081] S52: Expand the coordinates of 21 2D keypoints into a sequence of 3D keypoints for the hand: Forming a time-consistent cross-modal input , ;

[0082] S53: Transmit Q through three layers of space Hand posture features were obtained. ; Lightweight 3D-CNN: Generating Deep Visual Features In the Prior-guidedFusion module of the Uni-Sign framework, with Coordinates are used to calculate attention weights based on spatial priors. Output unified features ;

[0083] S54: Will Linear projection to the word embedding dimension of a large language model: The data is fed into the Uni-Sign autoregressive language model to generate isolated word semantic text frame by frame. Complete isolated word prediction;

[0084] S55: To evaluate the performance of this invention in the isolated word prediction task, the evaluation metric consistent with Uni-Sign was adopted, using Top-1 accuracy as the primary metric, calculated under the per-instance (PI) setting. On a self-built isolated word test set, this method achieved a Top-1 accuracy of 91.8% under the PI setting, verifying the effectiveness and robustness of this invention in the sign language isolated word recognition task.

[0085] In an embodiment of the neck-hook visual sign language recognition method, step S6, based on the preceding technical solution, specifically includes the following sub-steps:

[0086] S61: Collect Chinese translations of isolated words from CSL-Daily to construct training samples. C is the Chinese sentence with the correct word order;

[0087] S62: Generate a randomized sequence from the N isolated words output by S5 using a random permutation strategy. And add to the beginning and end respectively. and mark;

[0088] S63: Will Input Deep Seek-Coder-7B-Instruct, initialize the parameters using Uni-Sign pre-training, freeze all weights except the last three Transformer layers, and generate the target correct statement C in an autoregressive manner;

[0089] S64: Using BERT-Score to Calculate Sentence Generation With true sentences The R, P, and F1 indices between them, and the negative log-likelihood. As a differentiable loss, gradient updates are performed only on the weights of the last three layers of Deep Seek;

[0090] S65: After training, the model directly inputs a randomized isolated word sequence during the inference phase, and outputs a semantically coherent and grammatically correct final translation, achieving real-time sign language sentence translation.

[0091] The above describes the device and implementation method of the neck-hanging visual sign language recognition of the present invention, which is used to help understand the present invention. However, the implementation of the present invention is not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the principle of the present invention should be considered as equivalent substitutions and are included within the protection scope of the present invention.

Claims

1. A method for implementing a neckband-style visual sign language recognition system, characterized in that, Includes the following steps: S2: Acquire continuous sign language videos, reconstruct the hand mesh for each frame of the sign language video, and calculate the 2D key point data of the hand using the DoF parameters of the intermediate hand skeleton. S3: Acquire facial expression video, compare the facial expression with the corresponding frame image in the sign language video, adjust the hand mesh, and finally output the hand mesh; S4: Use 2D key point data of the hand to complete the division of hand mesh in continuous sign language video; S5: Based on the hand mesh segmentation in S4, obtain the corresponding isolated words in the "hand mesh--isolated words" dataset model; S6: The obtained series of isolated words are processed by a large language model to output a clear and fluent sentence.

2. The method for implementing neckband-style visual sign language recognition according to claim 1, characterized in that: It also includes step S1: A "hand mesh-isolated word" dataset model was established. The word segmentation model was trained by sign language videos to complete the hand mesh modeling process and generate the "hand mesh-isolated word" dataset model.

3. The method for implementing neckband-style visual sign language recognition according to claim 2, characterized in that: S1 specifically includes the following steps: S11: Collect sign language videos, hand meshes, and 2D key points of the hand in the motion capture studio, while manually completing gloss-level word segmentation; S12: Based on the dataset obtained in S11 and the sign language dataset CSL, complete the hand mesh reconstruction model and generate the "hand mesh-isolated word" dataset model.

4. The method for implementing neckband-style visual sign language recognition according to claim 1, characterized in that: S2 specifically includes the following steps: S21: Data preprocessing: Capture sign language video frame images through a camera, normalize the pixels of each frame of the input sign language video, and divide it into non-overlapping image blocks of 16×16 pixels, and calculate the embedding vector for each block. S22: ViT-H is used as the backbone network to extract global context features of image patches, and hand pose parameters are regressed through a Transformer decoder. Hand shape parameters Camera external parameters ; S23: Input the regressed hand pose and hand shape parameters into the MANO parametric model to generate hand mesh vertex coordinates. Coordinates of 21 3D key points on the hand ; S24: Project the 3D keypoint X onto the image plane using pre-calibrated camera extrinsic parameters and fixed intrinsic parameters to obtain the 2D keypoint coordinates. ; S25: Synchronously cache the hand mesh vertex sequence for each frame of video. }, 2D keypoint sequence { } 5. The method for implementing neckband-style visual sign language recognition according to claim 1, characterized in that: S4 specifically includes the following steps: S41: Retrieve the cached temporal 2D keypoint sequence of the hand { After translating with the wrist as the origin, the sequence is standardized according to the mean and variance of the whole sequence, and short segments are extracted using a sliding window of predetermined length and predetermined step size. S42: Calculate inter-frame displacement frame by frame within each short segment. speed change and display area Three types of lightweight dynamic features are concatenated into a 63-dimensional feature vector F; S43: Input the feature sequence into a two-layer 64-dimensional bidirectional LSTM followed by a BiLSTM-Segmenter with Sigmoid() output to obtain the probability of each frame being an isolated word boundary. ; S44: Yes Threshold binarization, followed by nonmaximum suppression and minimum duration filtering, yields the final boundary, which segments the continuous sign language video into isolated word units and outputs start and end time codes.

6. The method for implementing neckband-style visual sign language recognition according to claim 1, characterized in that: S5 specifically includes the following steps: S51: Obtain the hand mesh vertex sequence corresponding to each isolated word unit output by S4. }, Render a continuous depth map sequence from a unified perspective { } and normalize the depth map frame by frame; S52: 21 hand 2D key points { Expanded to 21×3 dimensions, with added depth values. Obtain the 3D key point sequence of the hand { Then, the depth map sequences are aligned according to the time window to construct a temporally consistent cross-modal input; S53: Sequence of 3D key points for the hand { } Input a three-layer spatial GCN to generate hand pose features ,Will{ Input lightweight 3D-CNN to generate deep visual features Using the Prior-guided Fusion module in the Uni-Sign framework, the hand's 3D keypoint coordinates are used as priors to... and Frame-by-frame fusion yields unified features ; S54: Will After being projected onto the word embedding dimension of a large language model, the text is fed into a pre-trained language model of Uni-Sign to generate isolated word semantic text in an autoregressive manner. Complete isolated word prediction.

7. The method for implementing neckband-style visual sign language recognition according to claim 1, characterized in that: S6 specifically includes the following steps: S61: Collect translated sentences that correspond one-to-one with isolated words from the CSL-Daily dataset and construct training samples. C is the sentence with the correct word order; S62: Generate a randomized input sequence from the N isolated word semantic texts output by S5 using a random permutation strategy. And add [CLS] and [SEP] markers at the beginning and end of the sequence respectively; S63: Will Input the Deep Seek-Coder-7B-Instruct model, use Uni-Sign pre-trained parameters as initialization, freeze all weights except the last three Transformer layers, and generate the target correct statement in an autoregressive manner. ; S64: Calculate using BERT-Score The three metrics of R, P, and F1 are compared with real statements from CSL-Daily, with F1 used as the differentiable loss. Fine-tune the weights of the last three layers of Deep Seek; S65: After training, the inference phase directly inputs a disordered sequence of isolated words, and the Deep Seek model outputs a semantically coherent and grammatically correct final translation, achieving real-time sign language sentence translation.

8. A device for implementing the neckband-style visual sign language recognition method according to any one of claims 1 to 7, characterized in that: The device includes a neck strap, with both ends of which are connected to a housing via connecting lines. The neck strap is equipped with at least one sign language capture camera facing outwards to capture hand gestures, and the housing is equipped with at least one facial expression camera.

9. The neck-hanging visual sign language recognition device according to claim 8, characterized in that: The facial expression camera is rotatably mounted on the outer edge of the upper part of the box.

10. The neck-mounted visual sign language recognition device according to claim 8, characterized in that: Earphones are also provided on both sides of the neck strap.

Images