Neck hanging type sign language interpretation equipment and sign language semantic recognition interpretation method thereof

Abstract

The invention discloses a neck-hung sign language interpretation device and a sign language semantic recognition interpretation method thereof.The neck-hung sign language interpretation device comprises a wearing support which defines a neck surrounding space and comprises a middle connecting arm and two shells, and the two shells are connected to the two sides of the middle connecting arm respectively; the image acquisition unit is arranged on the front side of the shell, comprises a three-dimensional depth camera and an adjusting mechanism for driving the three-dimensional depth camera to rotate, and is used for dynamically tracking and acquiring continuous sign language videos; the processing module is integrated in the shell, comprises a main control processor and a memory, and is used for operating a sign language recognition algorithm; the voice output unit is electrically connected with the main control processor and is used for playing the translation result; and the power supply unit supplies power to the image acquisition unit, the processing module and the voice output unit. Gesture languages used by the deaf-mute for communication are converted into natural languages which can be easily understood by normal people, the deaf-mute group is helped to speak, and the problem of information output when the deaf-mute communicates with the normal people is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of wearable smart devices for deaf-mutes, and in particular to a neck-mounted sign language translation device and a sign language semantic recognition and translation method thereof. BACKGROUND

[0002] Sign language is a special language that helps deaf-mutes to communicate through visual recognition of hand gestures. Sign language simulates images or syllables according to changes in hand gestures to form a certain meaning or word, and it is a hand language used by hearing-impaired aphasia groups to communicate and exchange ideas with each other. However, normal people who have not undergone professional sign language learning and training cannot understand the sign language actions of deaf-mutes, not to mention communicating with deaf-mutes in sign language. Deaf-mutes write or type to express themselves, which is low in efficiency, and the use of existing real-time sign language translation software is not natural enough, and there is still a certain gap in communication.

[0003] At present, many studies have been carried out around the topic of translating gesture language into natural language. The more mainstream ways to translate gesture language are:

[0004] (1) The existing gesture action images are recognized through computer vision recognition technology, the gesture actions are matched with the current gesture language library, and the meaning represented by the gesture actions is translated.

[0005] (2) Wearable smart gloves or smart bracelets are used to collect gesture action data, and the user's gesture actions can be perceived without the aid of visual sensors. The gesture action data is analyzed through an algorithm, and the sign language is translated into digital text for display or into voice language for broadcast.

[0006] However, the current sign language translation based on visual recognition technology requires special equipment, most of which are not portable, and the fluency and naturalness of use are not high. The accuracy and convenience of sign language translation through smart gloves or smart bracelets need to be improved.

[0007] The lack of understanding and familiarity of normal people with sign language semantics limits effective communication between deaf-mutes and normal people. Therefore, smart devices for sign language recognition and translation are an important means to facilitate barrier-free communication between deaf-mutes and normal people. Based on the current research status of smart devices for sign language recognition at home and abroad, technical difficulties are analyzed. First, smart devices for sign language recognition must meet the accuracy requirements of accurately recognizing sign language semantics. Second, smart devices for sign language recognition must meet the portability requirements for deaf-mutes to use in daily life. Finally, smart devices for sign language recognition must meet the naturalness and fluency requirements for barrier-free communication between deaf-mutes and normal people.

[0008] The application comprehensively considers the technical requirements of sign language semantic recognition, the daily communication habits of deaf-mutes, and the actual communication situation between deaf-mutes and normal people, and investigates the existing intelligent devices for sign language recognition and translation, and designs and invents a neck-mounted sign language translation device and a sign language semantic recognition and translation method thereof to meet the accuracy, naturalness and fluency required by sign language semantic recognition and translation, and the portability required by the intelligent device for sign language recognition. SUMMARY

[0009] The application aims at the deficiencies and defects in the prior art, and mainly aims to provide a neck-mounted sign language translation device and a sign language semantic recognition and translation method thereof, which can convert the gesture language used by deaf-mutes in communication into natural language easy for normal people to understand, help the deaf-mute group to "speak", and solve the information output problem of deaf-mutes in communication with normal people, and break the barrier that hinders the communication between deaf-mutes and normal people.

[0010] In order to achieve the above-mentioned purpose, the application adopts the following technical solutions:

[0011] The neck-mounted sign language translation device comprises a wearing support, a middle connecting arm and two housings, and the two housings are connected to the two sides of the middle connecting arm.

[0012] An image acquisition unit is arranged on the front side of the housing and comprises a three-dimensional depth camera and an adjusting mechanism for driving the rotation of the three-dimensional depth camera, and is used for dynamically tracking and acquiring continuous sign language videos.

[0013] A processing module is integrated in the housing and comprises a main control processor and a memory, and is used for running a sign language recognition algorithm.

[0014] A voice output unit is electrically connected with the main control processor and is used for playing the translation result.

[0015] A power supply unit is used for supplying power to the image acquisition unit, the processing module and the voice output unit.

[0016] Beneficial effects: The application provides a neck-mounted sign language translation device, which significantly improves the unobtrusiveness, naturalness and fluency of deaf-mute users in actual communication with normal people. The deaf-mute user wears the neck-mounted sign language translation device in communication with normal people, the neck-mounted sign language translation device enters and acquires the video of the continuous sentence expressed by the deaf-mute user through the front three-dimensional depth camera, and the gesture language used by the deaf-mute in communication is converted into natural language easy for normal people to understand through the sign language recognition algorithm built in the main control processor and is played through the loudspeaker, helping the deaf-mute group to "speak" and solving the information output problem of deaf-mutes in communication with normal people. The whole information output process is natural and fluent, and gives the deaf-mute user an unobtrusive and comfortable experience in communication with normal people.

[0017] In an alternative embodiment, a speaker sound hole, a ventilation hole and a fixing member are arranged on the shell; the fixing member is used for fixing the adjusting mechanism, the voice output unit, the processing module and the power supply unit inside the shell.

[0018] In an alternative embodiment, an inertial measurement unit is arranged inside the shell, and the inertial measurement unit is electrically connected with the main control processor, and is used for monitoring three-axis acceleration and angular velocity data of the device in real time.

[0019] Beneficial effects: the hardware level anti-shake design, the inertial measurement unit arranged inside the shell, the real-time monitoring of three-axis acceleration and angular velocity data of the device, the time stamp synchronization of the main control processor and the video stream, and the compensation of the image displacement caused by the body shaking during the sign language action.

[0020] In an alternative embodiment, a USB Type-C interface for charging the power supply unit is arranged on the shell, and the USB Type-C interface supports the connection of a personal computer for program editing and user data annotation.

[0021] In an alternative embodiment, the adjusting mechanism comprises a servo motor, and a connecting shaft, one end of which is connected with a driving shaft of the servo motor, and the other end of which is connected with the three-dimensional depth camera.

[0022] The application further discloses a sign language semantic recognition and translation method applied to the neck-mounted sign language translation device, and comprises the following steps:

[0023] S1: acquiring continuous sign language videos through an image acquisition unit;

[0024] S2: performing dynamic motion compensation preprocessing based on an optical flow method on the continuous sign language videos acquired in step S1;

[0025] S3: detecting a hand position and locating a knuckle joint, and combining an LSTM model to predict a stable coordinate in a shaking scene;

[0026] S4: driving an adjusting mechanism to adjust a three-dimensional depth camera view angle through a PID control algorithm, and realizing smooth tracking through Kalman filtering;

[0027] S5: extracting continuous dynamic position information of the knuckle joint of the hand;

[0028] S6: matching and recognizing semantics by combining a motion track of the knuckle joint of the hand with a user personalized sign language semantic library, wherein the user personalized sign language semantic library is trained based on a basic sign language data set;

[0029] S7: calibrating a recognition result by combining context semantics through an adaptive model of a Transformer architecture.

[0030] S8: output the calibrated voice through the voice output unit.

[0031] Beneficial effect: algorithm-level image stabilization, using a dynamic motion compensation algorithm based on the optical flow method, eliminating inter-frame displacement caused by jitter through feature point matching between adjacent frames.

[0032] Introduce spatiotemporal consistency constraints, combine continuous multi-frame hand position information, and use LSTM (Long Short-Term Memory Network) model to predict the stable coordinates of the joint points in the jitter scene.

[0033] To avoid frequent jitter, the control algorithm introduces motion smoothing filtering (such as Kalman filtering), combines hand motion speed and acceleration to predict future position, and realizes smooth transition of camera view angle.

[0034] Context-aware calibration, adaptive model uses Transformer architecture, combines context information of current communication scene, such as conversation topic, historical sentences, dynamically adjusts semantic weight of recognition result. For example, in the "dining" scene, preferentially match the semantic library related to "food" and "tableware".

[0035] In an optional embodiment of the sign language semantic recognition and translation method, in step S4, the host processor calculates the motion trajectory of the hand in three-dimensional space based on the hand joint point coordinates in real time, and generates a rotation angle instruction of the adjusting mechanism through a PID control algorithm; the adjusting mechanism drives and controls the three-dimensional depth camera according to the rotation angle instruction, so that the three-dimensional depth camera adjusts the view angle along the preset limited rotation range, ensuring that the hand is always located in the center area of the camera view.

[0036] In an optional embodiment of the sign language semantic recognition and translation method, in step S6, the matching recognition uses a gated multi-stream spatiotemporal convolutional network, which includes:

[0037] Spatial feature stream: processing a single frame of RGB image to generate a hand joint point heat map;

[0038] Spatiotemporal feature stream: extracting spatiotemporal features of consecutive video frames;

[0039] Joint trajectory encoding stream: processing three-dimensional coordinate sequence of joint points;

[0040] Gated fusion module: dynamically weighted multi-stream feature output fusion feature, suppresses noise and enhances effective information;

[0041] Adaptive temporal segmentation pooling: dividing vocabulary boundaries and weightedly aggregating features;

[0042] Joint loss function: simultaneously optimizing CTC temporal alignment loss and vocabulary classification loss.

[0043] Beneficial Effects: This invention innovatively constructs a multi-stream convolutional sign language recognition model based on deep learning methods, significantly improving the accuracy of sign language semantic recognition and interpretation. The deep learning-based multi-stream convolutional sign language recognition model combines multi-scale spatial information, hand finger joint data from video images, and temporal and spatial information from deep video streams to process and recognize continuous sign language semantics. The deep learning-based multi-stream convolutional sign language recognition model combines convolutional networks with data augmentation techniques to avoid overfitting problems.

[0044] A gated multi-stream spatiotemporal convolutional network is proposed. Its joint trajectory encoding stream directly processes the motion trajectories of 3D joints, reducing image jitter interference. The gated fusion mechanism dynamically adjusts the multimodal feature weights, improving robustness in noisy environments. The ATSP-CTC adaptively delineates word boundaries, solving the "word entanglement" problem in continuous sign language. The joint loss function simultaneously optimizes word classification accuracy and temporal alignment accuracy. This scheme, through multimodal feature gated fusion and adaptive temporal modeling, meets the requirement of accurately recognizing the semantics of continuous sign language.

[0045] In an optional embodiment of the sign language semantic recognition and interpretation method, the formula for calculating the spatial feature flow is:

[0046] H t =σ(Conv2D 5×5 (ReLU(Conv2D 3×3 (I t )))),in,

[0047] First, input: the time step of a single frame RGB image.

[0048] Output: Hand joint heatmap H t ∈R h×W×K ;

[0049] σ is the Sigmoid activation function;

[0050] Conv2D 5×5 A two-dimensional convolutional layer with 5×5 convolutional kernels;

[0051] ReLU is the modified linear unit activation function;

[0052] Conv2D 3×3 A two-dimensional convolutional layer with 3×3 convolutional kernels;

[0053] I t The time step for a single frame of RGB image;

[0054] H is the image height;

[0055] W is the image width;

[0056] It has three RGB channels;

[0057] H t Heat map of hand joints;

[0058] h represents the height of the heat map;

[0059] w represents the width of the heatmap;

[0060] K represents the number of joints in the hand;

[0061] The calculation formula for the spatiotemporal feature flow is as follows:

[0062] F 3D =MaxPool3D(Conv3D) 3×3×3 (V)), where

[0063] First, input: a sequence of consecutive video frames.

[0064] Output: Temporal and spatial features F 3D ∈R T'×d ;

[0065] V [t-Δt,t+Δt] It is a continuous frame video sequence;

[0066] t is the center time step;

[0067] Δt is the time radius;

[0068] T is the time step;

[0069] H' is the frame height;

[0070] W' is the frame width;

[0071] F 3D Features of time and space;

[0072] T ' This is the compressed time step;

[0073] d represents the feature dimension;

[0074] Conv3D 3×3×3 A three-dimensional convolutional layer with 3×3×3 convolutional kernels;

[0075] The calculation formula for the joint trajectory encoding stream is as follows:

[0076] f traj =BiLSTM(Linear(J t )),in,

[0077] First, input: a sequence of three-dimensional coordinates of key points in consecutive frames.

[0078] Re-output: Trajectory feature vector F traj ∈R d ;

[0079] J t is a sequence of three-dimensional coordinates of hand joint nodes for continuous frames;

[0080] is the dimension of three-dimensional coordinates (X, Y, Z axis coordinates);

[0081] F traj is a trajectory feature vector output by the joint trajectory encoding stream;

[0082] BiLSTM is a bidirectional long short-term memory network that captures forward and backward dependencies;

[0083] The calculation formula of the gating fusion module is:

[0084] G=Sigmoid(Linear([F 2D ;F 3D ;F traj ]))

[0085] F fused =G 2D ⊙F 2D +G 3D ⊙F 3D +G traj ⊙F traj

[0086] wherein, is a gating weight vector;

[0087] ⊙ is an element-wise multiplication, and the weight G 流 needs to be broadcast to the same dimension as the corresponding feature F 流 ;

[0088] ;] is a feature concatenation operation;

[0089] F 2D is a spatial feature (2D-CNN extraction) output by a spatial feature stream;

[0090] F 3D is a spatio-temporal feature (3D-CNN extraction) output by a spatio-temporal feature stream;

[0091] Linear is a fully connected layer;

[0092] F fused is a fusion feature vector;

[0093] G=[G 2D ,G 3D ,Gtraj ]∈[0,1] 3 is the confidence weight of each feature stream;

[0094] The calculation formula of the adaptive temporal segment pooling is:

[0095]

[0096] wherein,

[0097] S i ∈R d is the feature representation of the ith sign language vocabulary;

[0098] w t is a learnable time importance weight;

[0099] α t is a learnable time normalization weight;

[0100] is the fusion feature vector of the time step;

[0101] a i is the starting time step of the vocabulary i;

[0102] b i is the ending time step of the vocabulary i;

[0103] y i is the label of the ith sign language vocabulary;

[0104] The calculation formula of the joint loss function is:

[0105]

[0106] wherein Y is the real sign language sentence label sequence;

[0107] S is the vocabulary feature sequence;

[0108] S i ∈R d is the feature representation of the ith sign language vocabulary;

[0109] λ is a balance hyperparameter, default 0.5;

[0110] is the joint loss function;

[0111] CTC-Loss is the time alignment loss;

[0112] CE-Loss is the cross-entropy loss;

[0113] Linear(S i) as the vocabulary classifier output;

[0114] N is the total number of words in the sentence.

[0115] In an optional embodiment of the sign language semantic recognition and translation method, in step S7, the user's personalized sign language semantic library is updated through incremental learning and knowledge distillation, and user rating feedback is received to optimize the calibration strategy.

[0116] Beneficial effects: the model periodically updates the user's personalized sign language semantic library through an incremental learning algorithm, avoiding catastrophic forgetting; the knowledge distillation technology is used to compress the complex model of the basic sign language semantic library into a lightweight sub-model, adapting to the user's personalized needs. BRIEF DESCRIPTION OF DRAWINGS

[0117] Figure 1 is a schematic diagram of the overall structure of the neck-mounted sign language translation device of the present application;

[0118] Among them, 1, three-dimensional depth camera; 2, RGB camera; 3, infrared depth sensor; 4, speaker sound hole; 5, ventilation hole; 6, connecting arm; 7, rear neck support; 8, soft silica gel pad; 9, volume adjustment roller; 10, power button; 11, power display light; 12, shell;

[0119] Figure 2 is a schematic diagram of the internal structure of the neck-mounted sign language translation device of the present application;

[0120] Among them, 13, control connecting shaft; 14, servo motor; 15, speaker; 16, mainboard; 17, communication module; 18, main control processor; 19, storage unit; 20, battery; 22, fixing piece.

[0121] Figure 3 is a schematic diagram of the bottom of the neck-mounted sign language translation device of the present application;

[0122] Among them, 21, USB Type-C interface;

[0123] Figure 4 is a schematic diagram of the three-dimensional depth camera and control connecting shaft structure of the neck-mounted sign language translation device of the present application;

[0124] Figure 5 is a schematic diagram of the whole process of the sign language semantic recognition and translation method of the present application;

[0125] Figure 6 is a schematic diagram of the technical route of the sign language semantic recognition and translation method of the present application;

[0126] Figure 7 is a framework diagram of the gated multi-stream spatio-temporal convolutional network structure of the present application;

[0127] Figure 8It is a hand gesture semantic recognition and translation process hand joint extraction positioning diagram. DETAILED DESCRIPTION

[0128] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all.

[0129] As shown in Figure 1 The neck-mounted sign language translation device includes a three-dimensional depth camera 1, a shell 12, a connecting arm 6, a neck support module, a speaker 15, a mainboard 16, and a battery 20.

[0130] As shown in Figure 5 and Figure 6 The sign language semantic recognition and translation method is applied to the neck-mounted sign language translation device.

[0131] As shown in Figures 1-4 The shell 12 includes a speaker sound hole 4, a ventilation hole 5, and a fixing piece 22; the shell 12 is equipped with a three-dimensional depth camera 1, a neck support module, a volume adjustment roller 9, a power button 10, a power display lamp 11, and a USB Type-C interface 21.

[0132] As shown in Figure 1 The shell 12 is connected by the connecting arm 6 and can be deformed within a limited range to facilitate the user to wear.

[0133] As shown in Figure 1 The three-dimensional depth camera 1 includes an RGB camera 2 and an infrared depth sensor 3, which can effectively collect the video of the continuous sentence expressed by the deaf-mute user in sign language, so as to facilitate the subsequent feature extraction and matching recognition of the sign language gesture action in the sign language video image.

[0134] As shown in Figure 1 The neck support module includes a rear neck support 7 and a soft silica gel pad 8, which can effectively improve the comfort of the deaf-mute user wearing the neck-mounted sign language translation device.

[0135] As shown in Figure 2 and Figure 4 The three-dimensional depth camera 1 can rotate within a limited range by following the control connecting shaft 13 driven by the servo motor 14 to track and capture the sign language gestures of the deaf-mute user, so as to realize the dynamic tracking of the sign language gestures by the three-dimensional depth camera.

[0136] Wherein, in order to realize the dynamic tracking of sign language gestures by the three-dimensional depth camera, the device adopts two technical solutions of gesture position detection and servo motor linkage control and dynamic smoothing tracking. Gesture position detection and servo motor linkage control, when the three-dimensional depth camera 1 collects sign language video in real time, the two-dimensional position and three-dimensional depth information of the hand are acquired through the built-in RGB camera 2 and infrared depth sensor 3; the main control processor 18 calculates the motion trajectory of the hand in the three-dimensional space based on the hand joint coordinates, such as wrist, fingertip and other key points, and generates the rotation angle instruction of the servo motor 14 through the PID control algorithm; the servo motor 14 drives the control connecting shaft 13 according to the instruction, so that the three-dimensional depth camera 1 adjusts the viewing angle along the preset limited position rotation range, and ensures that the hand is always located in the center area of the camera field of view. Dynamic smoothing tracking, in order to avoid frequent shaking, the control algorithm introduces motion smoothing filter (such as Kalman filter), combines hand motion speed and acceleration to predict future position, and realizes the smooth transition of camera viewing angle.

[0137] Please refer to Figure 2 As shown in the figure, the mainboard 16 includes a communication module 17, a main control processor 18 and a storage unit 19, and the powerful performance of the mainboard provides strong support for the use of personalized sign language semantic library based on user use data, multi-stream convolution sign language recognition model based on deep learning method and sign language semantic output model of adaptive user communication context.

[0138] Please refer to Figure 2 As shown in the figure, the fixing member 22 can fix the servo motor 14, the loudspeaker 15, the mainboard 16 and the battery 20 inside the shell 12.

[0139] Please refer to Figure 2 And Figure 3 As shown in the figure, the USB Type-C interface 21 can connect the charging line to charge the battery 20; also can be connected with personal computer through USB Type-C interface 21 and data line, and the user can edit and set the program of the neck hanging type sign language translation device on the personal computer.

[0140] Please refer to Figures 5-8 As shown in the figure, the neck hanging type sign language translation device of the application, the matching sign language semantic recognition and translation method, specifically, is a sign language recognition technology based on deep learning method, which only relies on hand features to perform dynamic sign language recognition, and translates the video of continuous sentences expressed by the hearing-impaired aphasia population in sign language into voice information that can be understood by normal people.

[0141] Please refer to Figure 5 As shown in the figure, the whole process of the sign language semantic recognition and translation method of the application includes the following steps:

[0142] S1, data collection: the input data in the sign language recognition process is collected by a three-dimensional depth camera 1 on the neck-mounted sign language interpretation device, and the three-dimensional depth camera 1 records the video of the continuous sentence expressed by the deaf-mute user in sign language;

[0143] S2, video preprocessing: the input video of the continuous sentence expressed by the deaf-mute user in sign language is preprocessed to facilitate the detection, recognition and extraction of sign language features in the subsequent process;

[0144] S3, detection and positioning: the hand information in the collected continuous sign language video image is detected to find the specific position of the hand in the sign language video image, and the knuckle joints of the fingers of the hand are recognized and positioned;

[0145] S4, gesture tracking: continuously tracking the sign language gesture action in the sign language video image;

[0146] S5, feature extraction: feature extraction is performed on the sign language video image to extract the continuous dynamic position information of the knuckle joints of the fingers of the hand;

[0147] S6, matching and recognition: the continuous dynamic position data of the knuckle joints of the fingers of the hand collected is converted into the three-dimensional motion trajectory of the knuckle joints of the fingers of the hand, and then the three-dimensional motion trajectory of the knuckle joints of the fingers of the hand is matched with the user's personalized sign language semantic library. Once the matching is successful, the sign language semantics can be recognized;

[0148] S7, calibration: the recognition result of the sign language semantics is calibrated and optimized according to the user's communication context and communication habit;

[0149] S8, output: the calibrated result of the sign language semantics is output through the loudspeaker 15 on the neck-mounted sign language interpretation device.

[0150] Among them, in view of the shaking problem caused by the neck-mounted wearing, the device adopts hardware level anti-shaking design, algorithm level image stabilization, robustness joint detection and multi-modal data fusion technology. Hardware level anti-shaking design, an inertial measurement unit (IMU) is added inside the shell 12 to monitor the three-axis acceleration and angular velocity data of the device in real time. The time stamp of the main control processor 18 is synchronized with the video stream to compensate for the image displacement caused by shaking. Algorithm level image stabilization, a dynamic motion compensation algorithm based on optical flow method is adopted to eliminate the inter-frame displacement caused by shaking through feature point matching between adjacent frames. Robustness joint detection, spatiotemporal consistency constraints are introduced, combined with continuous multi-frame hand position information, LSTM long short-term memory network model is used to predict the stable coordinates of the joint in the shaking scene. Multi-modal data fusion, the depth data of the three-dimensional depth camera 1 and the inertial data of the IMU are fused to optimize the estimation accuracy of the three-dimensional coordinates of the hand through extended Kalman filtering.

[0151] Please refer to Figure 5and Figure 6 As shown in the accompanying drawings, the present application provides a sign language semantic recognition and translation method for a lanyard sign language translation device, which innovatively constructs a personalized sign language semantic library based on user usage data, significantly improving the accuracy of sign language semantic recognition and translation. The personalized sign language semantic library based on user usage data is trained and optimized on the basis of a basic sign language semantic library. The basic sign language semantic library is composed of commonly used sign language data sets, such as Chinese sign language data sets. On the basis of the basic sign language semantic library, the user's historical usage data and the user's personal communication habits are used to continuously self-learn through artificial neural networks to construct a user personalized sign language semantic library. This process completes the "knowledge distillation" of the basic sign language semantic library, simplifies and optimizes the huge and complex, rigorous and orderly basic sign language semantic library, and forms a user personalized sign language semantic library that is lighter and more in line with the user's communication habits. In addition, the deaf user can also customize the coding of the complex gesture semantics commonly used by himself, simplify the complex sign language into a simplified sign language set by himself, and enter the user personalized sign language semantic library, forming a "shortcut gesture action" that is unique to the user. The user can express the meaning that can only be expressed by complex sign language using his own simplified sign language, reducing the burden of sign language gestures of the deaf user in communication, and making the user personalized sign language semantic library better serve the user.

[0152] Please refer to Figures 5-7As shown, the present application provides a neck-mounted sign language translation device and a sign language semantic recognition and translation method. The present application innovatively constructs a multi-stream convolution sign language recognition model based on a deep learning method, which significantly improves the accuracy of sign language semantic recognition and translation. The multi-stream convolution sign language recognition model based on a deep learning method is responsible for the "feature extraction" and "matching recognition" links in the whole process of sign language semantic recognition and translation. The multi-stream convolution sign language recognition model based on a deep learning method combines multi-scale spatial information, hand joint data in video image, and time and space information in depth video stream for continuous sign language semantic processing and recognition. The multi-stream convolution sign language recognition model based on a deep learning method combines convolution network and data enhancement technology to avoid overfitting. Specifically, the multi-stream convolution neural network includes a two-dimensional convolution neural network and a three-dimensional convolution neural network. The two-dimensional convolution neural network (2D-CNN) is responsible for feature extraction of single-frame images, and the sign language features are mainly hand joint information in video image. The three-dimensional convolution neural network (3D-CNN) is responsible for processing the context relationship of continuous sign language video and extracting the time and space features of continuous sign language video. The multi-stream convolution sign language recognition model based on a deep learning method considers the advantages and disadvantages of convolution network, introduces and combines the Connectionist Temporal Classification (CTC) method to complete the continuous sign language recognition task. CTC can make the word interval more clear and represent the dependency relationship within the word in the continuous sign language recognition task. The multi-stream convolution sign language recognition model based on a deep learning method is pre-trained and adjusted according to the basic sign language semantic library before use. During use, the user's operation and use data are continuously collected. After use, the historical use data of the user are analyzed and summarized through a deep learning algorithm, and the sign language matching recognition algorithm and feedback mechanism are automatically optimized through continuous self-learning and iteration.

[0153] As shown in Figure 6 and Figure 7 As shown, the present application constructs a multi-stream convolution sign language recognition model based on a deep learning method. Specifically, the multi-stream convolution sign language recognition model based on a deep learning method proposes a gated multi-stream spatio-temporal convolution network that fuses multi-modal data and dynamic time series modeling. The core innovative module and formula are as follows: 1. Multi-stream feature extraction layer

[0154] Stream 1: spatial feature stream (2D-CNN branch)

[0155] Input: single-frame RGB image

[0156] Output: Hand joint heat map H t ∈R h×w×K

[0157] Formula:

[0158] H t = σ(Conv2D 5×5 (ReLU(Conv2D 3×3 (I t ))))

[0159] σ is a Sigmoid activation function;

[0160] Conv2D 5×5 is a two-dimensional convolution layer with a 5x5 convolution kernel;

[0161] ReLU is a rectified linear unit activation function;

[0162] Conv2D 3×3 is a two-dimensional convolution layer with a 3x3 convolution kernel;

[0163] I t is the time step of a single-frame RGB image;

[0164] H is the image height;

[0165] W is the image width;

[0166] is the RGB three-channel;

[0167] H t is the hand joint heat map;

[0168] h is the heat map height;

[0169] w is the heat map width;

[0170] K is the number of hand joint points;

[0171] · Stream 2: Spatio-temporal feature stream (3D-CNN branch)

[0172] Input: Continuous frame sequence

[0173] Output: Temporal-spatial feature F 3D ∈R T'×d

[0174] Formula:

[0175] F 3D = MaxPool3D(Conv3D 3×3×3 (V))

[0176] V[t-Δt,t+Δt] for a continuous frame video sequence;

[0177] t is a center time step;

[0178] Δt is a time radius;

[0179] T is a time step;

[0180] H' is a frame height;

[0181] W' is a frame width;

[0182] F 3D is a time-space feature;

[0183] T ' is a compressed time step;

[0184] d is a feature dimension;

[0185] Conv3D 3×3×3 is a three-dimensional convolution layer with a 3x3x3 convolution kernel.

[0186] Stream 3: Joint trajectory encoding stream

[0187] Input: continuous frame joint three-dimensional coordinate

[0188] Output: trajectory feature F traj ∈R d

[0189] Formula:

[0190] F traj = BiLSTM(Linear(J t ))

[0191] J t is a continuous frame hand joint three-dimensional coordinate sequence;

[0192] T is a time step;

[0193] is a three-dimensional coordinate dimension (X, Y, Z axis coordinates);

[0194] F traj is a trajectory feature vector output by the joint trajectory encoding stream;

[0195] BiLSTM is a bidirectional long short-term memory network that captures forward and backward dependencies.

[0196] 2. Gating feature fusion module

[0197] Dynamically weighted multi-stream features, suppress noise and enhance effective information:

[0198] G = Sigmoid(Linear([F 2D ; F 3D ; F traj ]))

[0199] F fused = G 2D ⊙ F 2D + G 3D ⊙ F 3D + G traj ⊙ F traj

[0200] where, G is the gating weight vector;

[0201] ⊙ is element-wise multiplication, weight G 流 needs to be broadcast to the same dimension as corresponding feature F 流 ;

[0202] [; ] is the feature concatenation operation;

[0203] F 2D is the spatial feature output by the spatial feature stream (2D-CNN extraction);

[0204] F 3D is the spatio-temporal feature output by the spatio-temporal feature stream (3D-CNN extraction);

[0205] Linear is a fully connected layer;

[0206] F fused is the fused feature vector;

[0207] G = [G 2D , G 3D , G traj ] ∈ [0, 1] 3 is the confidence weight of each feature stream.

[0208] 3. Spatio-temporal context modeling

[0209] Adaptive Temporal Segment Pooling (ATSP) is introduced to solve the problem of ambiguous word boundaries:

[0210]

[0211] where,

[0212] S i ∈ R d is the feature representation of the i-th sign language word;

[0213] w ttime importance weight that is learnable;

[0214] a t time normalization weight that is learnable;

[0215] fusion feature vector of time step;

[0216] a i start time step of vocabulary i;

[0217] b i end time step of vocabulary i;

[0218] y i label of the i-th sign language vocabulary.

[0219] 4. Loss function design

[0220] Joint optimization of classification and boundary alignment:

[0221]

[0222] where Y is the real sign language sentence label sequence;

[0223] S is the vocabulary feature sequence;

[0224] S i ∈R d is the feature representation of the i-th sign language vocabulary;

[0225] λ is a balance hyper-parameter, default 0.5;

[0226] is the joint loss function;

[0227] CTC-Loss is the time alignment loss;

[0228] CE-Loss is the cross-entropy loss;

[0229] Linear(S i ) is the vocabulary classifier output;

[0230] N is the total number of vocabularies in the sentence.

[0231] where the joint trajectory encoding stream directly processes the three-dimensional joint motion trajectory, reducing image jitter interference; the gating fusion mechanism dynamically adjusts the multi-modal feature weight, improving the noise environment robustness; the ATSP-CTC adaptively divides the vocabulary boundary, solving the "vocabulary sticking" problem in continuous sign language; the joint loss function synchronously optimizes the vocabulary classification accuracy and the time alignment accuracy.

[0232] Please refer to Figure 6As shown, the application provides a sign language semantic recognition and translation method for a neck-mounted sign language translation device, which innovatively constructs a sign language semantic output model that is adaptive to user communication context, significantly improving the accuracy of sign language semantic recognition and translation. The sign language semantic output model is responsible for the "calibration" link in the whole process of sign language semantic recognition and translation, and optimizes the sign language semantic recognition results output by the sign language recognition model. The adaptive sign language semantic output model continuously analyzes and summarizes the user's historical use data through deep learning algorithm and continuously self-learns and self-iterates, so that it can adapt to the user's communication context and communication habits, intelligently match the sign language semantic recognition results output by the sign language recognition model with the communication context, calibrate and optimize the sign language semantic recognition results output by the sign language recognition model, and finally output.

[0233] Among them, the implementation details and effects of adaptive technology are embodied in the following aspects.

[0234] First, user data collection and annotation. The storage unit 19 records the user's historical sign language video, semantic recognition result and final calibrated output text, forming a user-specific data set. The user can manually annotate misrecognized cases through the USB Type-C interface 21, and the annotation data is used for model fine-tuning.

[0235] Second, context awareness calibration. The adaptive model uses the Transformer architecture, combines the context information of the current communication scene (such as the conversation topic and historical sentences), and dynamically adjusts the semantic weight of the recognition result. For example, in the "dining" scene, the semantic library related to "food" and "tableware" is preferentially matched.

[0236] Third, incremental learning and knowledge distillation. The model regularly updates the user's personalized sign language semantic library through incremental learning algorithm to avoid catastrophic forgetting; knowledge distillation technology is used to compress the complex model of the basic sign language semantic library into a lightweight sub-model to adapt to the user's personalized needs.

[0237] The deaf-mute group has long faced the predicament of high language barrier, low information transmission efficiency and limited communication scene in social interaction. As the core communication method, sign language has a complete language system, but due to the general lack of sign language cognitive ability in the normal population, a communication gap is formed that is difficult to overcome. In daily communication, the deaf-mute people need to rely on complex hand movements, expressions and body language to transmit information, while the receiving party often causes information misreading or complete failure due to the professional threshold of sign language understanding, forcing both parties into a parallel dilemma of "visual language" and "auditory language". The neck-mounted sign language translation device and the sign language semantic recognition and translation method thereof convert the gesture language used in the communication of the deaf-mute people into natural language that is easy for normal people to understand, helping the deaf-mute group to "speak", solving the problem of information output of the deaf-mute group in communication with normal people, breaking the predicament of the deaf-mute group communicating with normal people in sign language, and providing humanistic care for the deaf-mute group.

[0238] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art within the technical range disclosed by the present application, according to the technical scheme and the inventive concept of the present application, equivalent replacement or change, should be covered within the protection scope of the present application.

Claims

1. A neck-mounted sign language interpretation device, characterized in that, include: The support frame defines the space around the neck and includes a central connecting arm and two housings (12), which are respectively connected to the two sides of the central connecting arm; The image acquisition unit is located on the front side of the housing (12) and includes a three-dimensional depth camera (1) and an adjustment mechanism that drives its rotation, for dynamically tracking and acquiring continuous sign language video. The processing module, integrated within the housing (12), includes a main control processor (18) and a memory (19) for running sign language recognition algorithms; The voice output unit (15) is electrically connected to the main control processor (18) and is used to play the translation results; The power supply unit (20) supplies power to the image acquisition unit, the processing module and the voice output unit (15).

2. The neck-mounted sign language interpretation device according to claim 1, characterized in that, The housing (12) includes a speaker outlet (4), a ventilation and heat dissipation hole (5), and a fixing member (22); the fixing member (22) is used to fix the adjustment mechanism, the voice output unit (15), the processing module, and the power supply unit (20) inside the housing (12).

3. The neck-mounted sign language interpretation device according to claim 1, characterized in that, The housing (12) is equipped with an inertial measurement unit, which is electrically connected to the main control processor (18) for real-time monitoring of the device's three-axis acceleration and angular velocity data. The main control processor (18) synchronizes with the video stream timestamp to compensate for jitter displacement.

4. The neck-mounted sign language interpretation device according to claim 1, characterized in that, The housing is provided with a USB Type-C interface (21) for charging the power supply unit (20), and the USB Type-C interface (21) supports connecting to a personal computer for program editing and user data annotation.

5. The neck-mounted sign language interpretation device according to claim 1, characterized in that, The adjustment mechanism includes: Servo motor (14); The connecting shaft (13) is connected at one end to the drive shaft of the servo motor (14) and at the other end to the three-dimensional depth camera (1).

6. A sign language semantic recognition and interpretation method, applied to the neck-mounted sign language interpretation device according to any one of claims 1-5, characterized in that, Includes the following steps: S1: Acquire continuous sign language video through the image acquisition unit; S2: Perform dynamic motion compensation preprocessing on the video based on optical flow; S3: Detect hand position and locate finger joints, and combine LSTM model to predict stable coordinates in shaking scenarios; S4: The PID control algorithm drives the adjustment mechanism to adjust the viewing angle of the three-dimensional depth camera (1), and Kalman filtering is used to achieve smooth tracking; S5: Extract continuous dynamic position information of the finger joints of the hand; S6: Match the joint motion trajectory with the user's personalized sign language semantic library to identify the semantics, wherein the user's personalized sign language semantic library is trained based on a basic sign language dataset; S7: Match the recognition results by combining the adaptive model of the Transformer architecture with the contextual semantic calibration step S6; S8: Output the calibrated voice through the voice output unit (15).

7. The sign language semantic recognition and interpretation method according to claim 6, characterized in that, In step S4, the main control processor (18) calculates the motion trajectory of the hand in three-dimensional space in real time based on the coordinates of the hand joints, and generates the rotation angle command of the adjustment mechanism through the PID control algorithm; the adjustment mechanism drives the control of the three-dimensional depth camera according to the rotation angle command, so that the three-dimensional depth camera adjusts the viewing angle along the preset limited rotation range, ensuring that the hand is always located in the center area of ​​the camera's field of view.

8. The sign language semantic recognition and interpretation method according to claim 6, characterized in that, In step S6, the matching recognition employs a gated multi-stream spatiotemporal convolutional network, including: Spatial Feature Flow: Processing single-frame RGB images to generate heatmaps of hand joints; Spatiotemporal feature stream: Extracting spatiotemporal features from consecutive video frames; Joint trajectory encoding stream: processes the three-dimensional coordinate sequence of joint points; Gated fusion module: dynamically weights multi-stream features to output fused features, suppressing noise and enhancing effective information; Adaptive temporal segmented pooling: delineates word boundaries and aggregates features using weighted methods; Joint loss function: Simultaneously optimize CTC temporal alignment loss and vocabulary classification loss.

9. The sign language semantic recognition and interpretation method according to claim 8, characterized in that, The formula for calculating the spatial feature flow is: H t =σ(Conv2D 5×5 (ReLU(Conv2D 3×3 (I t )))),in, First, input: the time step of a single frame RGB image. Output: Hand joint heatmap H t ∈R h×w×K ; σ is the Sigmoid activation function; Conv2D 5×5 A two-dimensional convolutional layer with 5×5 convolutional kernels; ReLU is the modified linear unit activation function; Conv2D 3×3 A two-dimensional convolutional layer with 3×3 convolutional kernels; I t The time step for a single frame of RGB image; H is the image height; W is the image width; It has three RGB channels; H t Heat map of hand joints; h represents the height of the heat map; w represents the width of the heatmap; K represents the number of joints in the hand; The calculation formula for the spatiotemporal feature flow is as follows: F 3D =MaxPool3D(Conv3D) 3×3×3 (V)), where First, input: a sequence of consecutive video frames. Output: Temporal and spatial features F 3D ∈R T'×d ; V [t-Δt,t+Δt] It is a continuous frame video sequence; t is the center time step; Δt is the time radius; T is the time step; H' is the frame height; W' is the frame width; F 3D Features of time and space; T ' This is the compressed time step; d represents the feature dimension; Conv3D 3×3×3 A three-dimensional convolutional layer with 3×3×3 convolutional kernels; The calculation formula for the joint trajectory encoding stream is as follows: F traj =BiLSTM(Linear(J t )),in, First, input: a sequence of 3D coordinates of key points in consecutive frames. Output: Trajectory feature vector F traj ∈R d ; J t A sequence of three-dimensional coordinates of hand joints in consecutive frames; For three-dimensional coordinates (X, Y, Z axis coordinates); F traj The trajectory feature vector output by the joint trajectory encoding stream; BiLSTM is a bidirectional long short-term memory network that captures dependencies between preceding and subsequent operations. The calculation formula for the gated fusion module is: G=Sigmoid(Linear([F 2D ;F 3D ;F traj ])) F fused =G 2D ⊙F 2D +G 3D ⊙F 3D +G traj ⊙F traj in, This is the gating weight vector; ⊙ is element-wise multiplication, weight G 流 It needs to be broadcast to the corresponding feature F 流 Same dimension; ;] represents the feature concatenation operation; F 2D Spatial features output as spatial feature stream (2D-CNN extraction); F 3D Spatiotemporal features output as a spatiotemporal feature stream (extracted by 3D-CNN); Linear is a fully connected layer; F fused To fuse feature vectors; G = [G 2D G 3D G traj ]∈[0,1] 3 , which is the confidence weight of each feature stream; The calculation formula for the adaptive temporal segmented pooling is: in, S i ∈R d , is the feature representation of the i-th sign language word; w t Weights based on the time-related importance of learnable time. α t For learnable time-normalized weights; This represents the fused feature vector at each time step. a i The starting time step for word i; b i For the end time step of word i; y i The label for the i-th sign language word; The formula for calculating the joint loss function is: Where Y is the sequence of actual sign language phrase labels; S is the lexical feature sequence; S i ∈R d , is the feature representation of the i-th sign language word; λ is the balancing hyperparameter, with a default value of 0.5; For the joint loss function; CTC-Loss is the timing alignment loss; CE-Loss is the cross-entropy loss; Linear(S i () is the output of the vocabulary classifier; N represents the total number of words in the sentence.

10. The sign language semantic recognition and interpretation method according to claim 6, characterized in that, In step S7, the user's personalized sign language semantic database is updated through incremental learning and knowledge distillation, and the calibration strategy is optimized by receiving user rating feedback.

Images