Sign language translation method, wearable device and readable storage medium

Abstract

The invention discloses a sign language translation method, wearable equipment and a readable storage medium, and relates to the technical field of computer vision, and the method comprises the steps: obtaining an environment image in response to a sign language translation request, and determining a current sign language scene based on the environment image; if the current sign language scene is a first person expression scene, calling a first person sign language translation model to process the environment image to obtain a first sign language translation result; and if the current sign language scene is a third-person observation scene, calling a third-person sign language translation model to process the environment image to obtain a second sign language translation result. According to the method, the performance limitation of the sign language translation model during cross-view application is effectively improved, and the cross-view generalization ability of sign language translation is improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a sign language translation method, a wearable device, and a readable storage medium. Background Technology

[0002] In today's society, people with hearing impairments face numerous communication challenges. Sign language, as one of their important means of communication, plays a crucial role in many scenarios such as daily life, learning, and work. Therefore, developing automatic, real-time sign language translation technology is essential for promoting the social integration and information accessibility of the hearing-impaired community.

[0003] There is currently a considerable amount of research on computer vision-based sign language translation systems. The mainstream approach is to process sign language videos from a fixed perspective using a single model, and then recognize and convert them into text or speech. These systems are typically optimized for specific perspectives (such as facing directly at the user or the user themselves).

[0004] However, due to fundamental differences in image scale, spatial deformation, occlusion, and background complexity between first-person and third-person perspectives of hand gestures, sign language recognition models trained on single-viewpoint data experience a significant drop in accuracy when faced with untrained sign language input from different perspectives. This makes it difficult for existing sign language translation systems to simultaneously guarantee the recognition performance of both the user's own gestures and those of others, limiting their practicality and reliability in complete two-way dialogue scenarios.

[0005] Therefore, it is urgent to address the performance limitations of sign language translation models in cross-perspective applications, which result in insufficient cross-perspective generalization ability. Summary of the Invention

[0006] The main purpose of this application is to provide a sign language translation method, wearable device and readable storage medium, which aims to solve the technical problem that the sign language translation model has performance limitations in cross-perspective applications, resulting in insufficient cross-perspective generalization ability.

[0007] To achieve the above objectives, this application provides a sign language translation method, which includes the following steps: In response to a sign language translation request, an environmental image is acquired, and the current sign language scene is determined based on the environmental image; If the current sign language scene is a first-person expression scene, then the first-person sign language translation model is invoked to process the environmental image to obtain the first sign language translation result; If the current sign language scene is a third-person observation scene, then the third-person sign language translation model is invoked to process the environmental image to obtain the second sign language translation result.

[0008] In one embodiment, determining the current sign language scene based on the environmental image includes: Face detection and hand detection are performed on the environmental image to obtain face detection results and hand detection results; If the face detection result indicates that a face has been detected, and the hand detection result indicates that the hand area is located in front of the face area, then the current sign language scene is determined to be a third-person observation scene. If the face detection result indicates that no face was detected, or the hand detection result indicates that the hand area is not in front of the face area, then the current sign language scene is determined to be a first-person expression scene.

[0009] In one embodiment, before the step of determining that the current sign language scene is a first-person expression scene, the method further includes: Based on the environmental image, the initial position and movement trajectory of key hand points are detected; If the initial position is located within a preset bottom area of ​​the environmental image, and the movement trajectory points to the center area of ​​the environmental image, then the step of determining the current sign language scene as a first-person expression scene is executed; If the initial position is not located within the preset bottom area, or if the movement trajectory points to the side area of ​​the environmental image, then the current sign language scene is determined to be a third-person observation scene.

[0010] In one embodiment, before the step of determining that the current sign language scene is a first-person expression scene, the method further includes: Acquire depth data synchronized with the environmental image collected by the depth sensor, and obtain the coordinates of the elbow joint and the wrist joint based on the depth data; Calculate the forearm space vector pointing from the elbow joint coordinates to the wrist joint coordinates; Analyze the variation magnitude of the depth axis component of the forearm spatial vector in consecutive time frames; If the change is greater than a preset threshold, then the step of determining the current sign language scene as a first-person expression scene is executed. If the change is less than or equal to a preset threshold, the current sign language scene is determined to be a third-person observation scene.

[0011] In one embodiment, the step of calling a first-person sign language translation model to process the environmental image and obtain a first-person sign language translation result includes: Acquire motion data collected by the inertial measurement unit; Motion compensation is performed on the environmental image based on the motion data to obtain a stable image sequence; The stable image sequence is input into the first-person sign language translation model to obtain the first sign language translation result.

[0012] In one embodiment, the step of calling a third-person sign language translation model to process the environmental image and obtain a second sign language translation result includes: Acquire depth data synchronized with the environmental image collected by the depth sensor; Based on the environmental image and the depth data, a three-dimensional coordinate sequence of key hand points is determined; The three-dimensional coordinate sequence is input into the third-person sign language translation model to obtain the second sign language translation result.

[0013] In one embodiment, after obtaining the first sign language translation result or the second sign language translation result, the method further includes: The first sign language translation result or the second sign language translation result is combined into a speech signal; If the current sign language scenario is a first-person expression scenario, then control the speaker to play the voice signal at the first volume parameter; If the current sign language scenario is a third-person observation scenario, then control the headphones to play the voice signal at a second volume parameter lower than the first volume parameter, or display the second sign language translation result on the screen.

[0014] In one embodiment, the method further includes: Continuously acquire environmental images in consecutive frames; For each frame of the environment image, perform the step of determining the current sign language scene based on the environment image; When it is detected that the current sign language scene switches from a first-person expression scene to a third-person observation scene, or from a third-person observation scene to a first-person expression scene, the sign language translation model being called is switched synchronously.

[0015] In addition, to achieve the above objectives, this application also provides a wearable device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the sign language translation method as described above.

[0016] In addition, to achieve the above objectives, this application also provides a readable storage medium, which is a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the steps of the sign language translation method as described above.

[0017] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the sign language translation method described above.

[0018] One or more technical solutions proposed in this application have at least the following technical effects: This application's embodiments effectively improve the performance limitations and insufficient generalization ability of single-viewpoint models in cross-view applications by identifying sign language scenarios (first-person expression scenarios and third-person observation scenarios) and selectively calling corresponding sign language translation models. Specifically, firstly, based on the acquired environmental image, it is determined whether the current sign language scenario belongs to first-person expression or third-person observation. Based on this, specially optimized models are called for processing in each scenario: for first-person expression scenarios, a specially trained first-person sign language translation model is called; for third-person observation scenarios, a specially trained third-person sign language translation model is called. This design eliminates the need for each model to forcibly learn and adapt to the inherent differences brought about by another perspective (such as significant differences in image scale, deformation, occlusion, and background), thus enabling more focused and robust recognition within its respective viewpoint domain. Therefore, it fundamentally reduces the performance trade-offs and recognition rate decline problems that inevitably occur when using a single model to process cross-view data, thereby effectively improving the performance limitations of sign language translation models in cross-view applications and enhancing the cross-view generalization ability of sign language translation. This enables the sign language translation system to reliably process the user's own sign language expressions and understand the sign language input of others within a complete framework, greatly enhancing its practicality and overall reliability in real two-way dialogue scenarios. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the first embodiment of the sign language translation method of this application; Figure 2 This is a schematic diagram of the facial correlation detection process involved in an embodiment of the sign language translation method of this application; Figure 3 This is a schematic diagram of the hand-in-picture area detection process in one embodiment of the sign language translation method of this application; Figure 4 This is a schematic diagram of the forearm vector depth detection process involved in an embodiment of the sign language translation method of this application; Figure 5 This is a schematic diagram of the hardware operating environment of the sign language translation method device in the embodiments of this application.

[0022] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] It should be noted that the execution subject of the various embodiments of the sign language translation method of this application can be a computing service device with data processing, network communication and program running functions, such as a server, tablet computer, personal computer, mobile phone, etc., or a wearable device that can realize the above functions, such as smart glasses, VR (Virtual Reality) headset, AR (Augmented Reality) headset, etc. The various embodiments of the sign language translation method of this application do not impose specific limitations on this.

[0025] Based on this, this application proposes a sign language translation method according to a first embodiment. In this embodiment, refer to... Figure 1 As shown, the sign language translation method includes the following steps S10~S30: Step S10: In response to the sign language translation request, acquire an environmental image and determine the current sign language scene based on the environmental image; When a user initiates a sign language translation request through a specific sign language translation device (such as a dedicated translation terminal, smartphone app, etc.), the system responds immediately. At this point, the device's camera or a connected external camera begins working, capturing real-time image information of the user's environment. These images may include the user's own hand gestures (in a first-person perspective scenario) or sign language gestures performed by others at a distance (in a third-person perspective scenario). The camera continuously captures images at a certain frame rate (e.g., 30 frames per second) to ensure a complete recording of the dynamic process of the sign language gestures.

[0026] Furthermore, the acquired images can undergo preliminary preprocessing, such as adjusting the image resolution, contrast, and brightness, to improve image quality and provide clearer and more accurate input data for subsequent scene judgment and translation processing.

[0027] Based on the acquired environmental images, the current sign language scene is determined to be either a first-person expression scene or a third-person observation scene. The criterion originates from decoding the image's perspective features: one implementation uses a pre-trained lightweight convolutional neural network for end-to-end scene classification. This network extracts hierarchical features of the image and outputs confidence scores for the two scene types, making a judgment accordingly. Another implementation analyzes the spatial geometric features of the image, such as extracting key points like shoulders, elbows, and wrists through a human pose estimation model, and then calculating parameters such as the relative position of the hands, arm angles, and the visible proportion of the torso. If both hands are located in the lower middle part of the image and the forearms point downwards towards the center of the image, it tends to be judged as a first-person expression scene; if a complete frontal human pose is detected and the hand activity area is concentrated in the middle of the image, it tends to be judged as a third-person observation scene.

[0028] Step S20: If the current sign language scene is a first-person expression scene, then the first-person sign language translation model is invoked to process the environmental image to obtain the first sign language translation result; If the current scene is determined to be a first-person perspective scene, a first-person sign language translation model specifically trained and optimized for this viewpoint is invoked for processing. This model can be based on a deep learning architecture, such as a three-dimensional convolutional neural network (3D-CNN) or a spatial-temporal graph convolutional network (ST-GCN), and is specifically optimized for the visual characteristics of a first-person perspective, such as the hands being close to the camera, large in scale, and clear in detail, but certain parts may be occluded, and the background being relatively stable. Its processing flow integrates multiple technical modules: First, image preprocessing is performed, including denoising, enhancement, and deformation correction; then, the coordinates of 21 three-dimensional key points of both hands are extracted using a hand key point detection model (such as the MediaPipe Hands model); the spatiotemporal feature encoding module uses a model such as Long Short-Term Memory (LSTM) or Transformer Encoder to encode the key point sequence to capture the dynamic temporal pattern of the gesture; the sign language recognition module uses a model such as Connectionist Temporal Classification (CTC) or Sequence to Sequence (Seq2Seq) to parse the encoded features and identify sign language words or morphemes; the sequence to sequence conversion module adjusts the recognition results into an intermediate representation that conforms to the word order of natural language; finally, the natural language generation module uses a lightweight Transformer neural machine translation model to convert the intermediate representation into fluent text, outputting the first sign language translation result.

[0029] Step S30: If the current sign language scene is a third-person observation scene, then the third-person sign language translation model is invoked to process the environmental image to obtain the second sign language translation result.

[0030] If the current scene is determined to be a third-person observation scene, a third-person sign language translation model specifically trained and optimized for this perspective is invoked for processing. This model architecture can share basic components with the first-person model, but its training dataset is concentrated on third-person viewpoint sign language videos, making it more robust to challenges such as varying hand scales, complex backgrounds, significant perspective distortion, and potential occlusion brought about by this perspective. Its processing flow is specifically designed at each stage: the preprocessing stage focuses on background suppression and viewpoint invariance enhancement; hand keypoint detection needs to maintain high recall and accuracy in complex environments; spatiotemporal feature encoding focuses on learning richer context and global dependencies under third-person perspective; and the sign language recognition, sequence conversion, and natural language generation modules are all trained on large-scale third-person data to accurately understand gesture sequences that conform to the grammar and semantic habits of this perspective, ultimately outputting the second sign language translation result. The two dedicated models, through a shared architecture but independent optimization, ensure high accuracy and high reliability under their respective perspectives.

[0031] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. On this basis, refer to Figure 2 As shown, determining the current sign language scene based on the environmental image includes: Step A10: Perform face detection and hand detection on the environmental image to obtain face detection results and hand detection results; The acquired environmental images undergo face and hand detection. Face detection can be achieved using pre-trained face detection algorithms (such as deep learning-based detection models like MTCNN and YOLO), which can quickly locate faces in the image and output the coordinates of the face regions. Simultaneously, hand detection algorithms (such as OpenPose and MediaPipe Hands) are used to detect hand regions in the image and obtain the bounding box coordinates of the hands. For example, a face region might be detected as a rectangular area in the upper left corner of the image, and a hand region as a rectangular area in the center of the image. These detection results will be used in subsequent scene judgment logic.

[0032] Step A20: If the face detection result indicates that a face has been detected, and the hand detection result indicates that the hand area is located in front of the face area, then the current sign language scene is determined to be a third-person observation scene. After obtaining the face detection and hand detection results, these results are analyzed. If the face detection result indicates the presence of a face in the image, and the hand detection result indicates that the hand region is located in front of the face region (i.e., the hand is directly in front of or diagonally in front of the face, which can be determined by comparing the coordinate positions of the hand and the face), then the current sign language scene can be determined to be a third-person observation scene. This is because in this scenario, the observer (camera) is usually observing another person's sign language actions from a third-person perspective. The hand movements are relatively small in the image and are located in front of the face, while the background may be relatively complex. For example, when a user observes someone performing sign language in a meeting, the image captured by the camera will simultaneously include the observed person's face and hands, with the hands located in front of the face, which matches the characteristics of a third-person observation scene.

[0033] Step A30: If the face detection result indicates that no face was detected, or the hand detection result indicates that the hand area is not in front of the face area, then the current sign language scene is determined to be a first-person expression scene.

[0034] If the face detection result indicates that no face is detected in the image, or the hand detection result indicates that the hand area is not in front of the face area (i.e., the hand may be close to the face or the hand appears alone in the image without a clear face as a reference), then the current sign language scene is determined to be a first-person expression scene. In this scenario, the user is usually facing the camera and expressing themselves through sign language, with hand movements occupying a large proportion of the image and a relatively simple background. For example, when a user uses a sign language translation device to express themselves, the camera mainly captures the user's hand movements, while the face may not be in the image, or the hand may be close to the face, but the hand movements are the main content of expression, which matches the characteristics of a first-person expression scene.

[0035] In one possible implementation, refer to Figure 3 As shown, before the step of determining the current sign language scene as a first-person expression scene, the method further includes: Step B10: Based on the environmental image, detect the initial position and movement trajectory of key points on the hand; The initial features of hand movements are captured through continuous frame analysis. First, keypoint detection is performed: a lightweight hand keypoint detection model (such as the MediaPipe Hands model) is used to process the image sequence, detecting and outputting the coordinates of one or more keypoints of the hand in real time, including their three-dimensional spatial positions (X, Y, Z). In a preferred embodiment, special attention is paid to the wrist keypoint, as it is a stable anatomical reference connecting the hand and arm, and its spatial position is highly representative in determining whether the hand belongs to "itself" or "another person." Next, initial position determination is performed: the coordinates of the hand (represented by the wrist keypoint) are recorded when it is first stably detected in the image sequence; this position defines the initial boundary position of the hand movement entering the camera's field of view. Simultaneously, motion trajectory tracking is performed: by tracking the coordinate changes of this keypoint in subsequent consecutive frames, its two-dimensional motion path and direction vector on the image plane are fitted or calculated, thereby quantifying its motion trajectory.

[0036] Step B20: If the initial position is located within a preset bottom area of ​​the environmental image and the motion trajectory points to the center area of ​​the environmental image, then the step of determining the current sign language scene as a first-person expression scene is executed. The preset bottom region is a predefined lower area based on the image height (e.g., a rectangular area at the bottom 25%-35% of the image). First, it's determined whether the initial coordinates of the wrist keypoint are within this region. This condition aligns with the physical principle that, in a first-person perspective, the user's hand typically enters the head-mounted or chest-facing device's field of vision from the lower front of the body (waist or chest). Simultaneously, the calculated motion trajectory direction vector is analyzed. If the vector's extension trend points towards the center of the image (e.g., its angle with the image center is less than a preset threshold), the trajectory is determined to "point towards the center." This characteristic matches the user's behavioral pattern of moving their hand towards their field of vision / camera center for clear expression. When both conditions are met, the current sign language scene is determined to be a first-person expression scene.

[0037] Step B30: If the initial position is not located within the preset bottom area, or the movement trajectory points to the side area of ​​the environmental image, then the current sign language scene is determined to be a third-person observation scene.

[0038] If the initial position of the wrist keypoint is not within the preset bottom area (e.g., initially appearing on the side or upper-middle of the image), or if the main direction of its movement trajectory points towards the side area of ​​the image (e.g., making horizontal movements), then the current scene is determined to be a third-person observation scene. This initial position and movement pattern are consistent with the common positions (side or center) and movement characteristics (more horizontal displacement) of the hands in the image when the observer is communicating with someone directly opposite them using sign language.

[0039] This embodiment adds an independent judgment dimension based on kinematic initiation features. It is particularly suitable for scenarios where facial information is missing or static spatial relationships are ambiguous. By analyzing the two dynamic behavioral features of "where the hand appears" and "where the hand moves," it can effectively capture the essential difference between first-person expression and observing others' gestures. This effectively complements or verifies detection schemes based on static image features (such as face-hand spatial relationship judgment), jointly improving the accuracy and robustness of sign language scene classification.

[0040] In one possible implementation, refer to Figure 4 As shown, before the step of determining the current sign language scene as a first-person expression scene, the method further includes: Step C10: Obtain depth data synchronized with the environmental image collected by the depth sensor, and obtain the elbow joint coordinates and wrist joint coordinates based on the depth data. Before determining the sign language scene, depth data is acquired using a depth sensor, which is simultaneously acquired with the environmental images. The depth sensor provides depth information for each pixel in the image, enabling the location of key points in 3D space. Using the depth data, image processing and joint detection algorithms are employed to obtain the coordinates of the elbow and wrist joints. This coordinate information, including the position of the elbow (or virtual elbow) and wrist in 3D space, provides the foundation for subsequent vector calculations and motion analysis.

[0041] Step C20: Calculate the forearm space vector pointing from the elbow joint coordinates to the wrist joint coordinates; After obtaining the coordinates of the elbow and wrist joints, a spatial vector of the forearm is calculated, pointing from the elbow joint coordinates to the wrist joint coordinates. This vector represents the direction and length of the forearm in three-dimensional space. Specifically, the calculation method involves subtracting the elbow joint coordinates from the wrist joint coordinates to obtain a three-dimensional vector. This vector contains components of the forearm in the depth direction (Z-axis), horizontal direction (X-axis), and vertical direction (Y-axis).

[0042] Step C30: Analyze the variation amplitude of the depth axis component of the forearm spatial vector in consecutive time frames; Further analysis was conducted on the changes in the forearm spatial vector across consecutive time frames, particularly the magnitude of the changes in the depth axis component. The magnitude of the changes in the depth axis component reflects the arm's movement in the depth direction. The magnitude of these changes was calculated by comparing the depth axis components in consecutive frames. If the arm exhibits significant forward or backward swinging or extension movements in the depth direction, the magnitude of the changes in the depth axis component will be larger; conversely, if the arm is relatively stationary or its movement is minimal, the magnitude of the changes will be smaller.

[0043] Step C40: If the change amplitude is greater than a preset amplitude threshold, then the step of determining the current sign language scene as a first-person expression scene is executed. If, within consecutive time frames, the change in the depth axis component of the forearm spatial vector exceeds a preset amplitude threshold, this indicates significant arm movement in the depth direction. This movement characteristic typically conforms to the natural arm swing pattern in first-person expression scenarios. In this case, the step of determining the current sign language scene as a first-person expression scenario will be executed. The preset amplitude threshold is set based on extensive experimental data and statistical characteristics of arm movements in real-world application scenarios to ensure the accuracy of the judgment.

[0044] Step C50: If the change amplitude is less than or equal to a preset amplitude threshold, then the current sign language scene is determined to be a third-person observation scene.

[0045] Conversely, if the change in the depth axis component of the forearm spatial vector is less than or equal to a preset amplitude threshold, it indicates that the arm's movement in the depth direction is relatively small. This movement characteristic is more consistent with the pattern of relatively stationary or minimally moving arms in third-person observation scenarios. In this case, the system will determine the current sign language scene as a third-person observation scenario. This judgment logic is based on the characteristic that sign language actions are usually relatively stable and the arm movement amplitude is small in third-person observation scenarios.

[0046] This embodiment utilizes depth sensing data to provide a novel and robust criterion for scene determination from the perspective of three-dimensional motion dynamics. This approach effectively overcomes the misjudgment problems caused by complex backgrounds, lighting variations, or interference from similar colors when relying solely on two-dimensional image features (such as position, color, and texture). By capturing the essential differences in arm movements in the physical space dimension between "self-expression" and "observing others" (the intensity of depth motion), it strongly complements and validates other visual determination methods, thereby improving the accuracy and environmental adaptability of scene classification.

[0047] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the first and second embodiments described above can be referred to the above description and will not be repeated hereafter. Based on this, the step of calling the first-person sign language translation model to process the environmental image and obtain the first sign language translation result includes: Step D10: Acquire motion data collected by the inertial measurement unit; While acquiring environmental images, raw motion data is simultaneously acquired by the device's built-in Inertial Measurement Unit (IMU). This unit typically integrates a three-axis accelerometer, a three-axis gyroscope, and may include a three-axis magnetometer. Timestamp alignment ensures that the acquired motion data (including instantaneous angular velocity, linear acceleration, and orientation information) is synchronized with the environmental image frames. This data, with high frequency and low latency, reflects the six degrees of freedom motion of the device (typically worn, neck-mounted, or handheld by the user) in three-dimensional space—namely, translation and rotation along three axes—providing reference information for quantifying the physical motion during image acquisition.

[0048] Step D20: Perform motion compensation on the environmental image based on the motion data to obtain a stable image sequence; After acquiring motion data from the inertial measurement unit (IMU), motion compensation is performed on the environmental image to eliminate image blurring and instability caused by device movement. Motion compensation can be implemented as follows: First, sensor fusion (e.g., using Kalman filtering or complementary filtering algorithms) is performed on the raw IMU data to calculate the camera pose change (including rotation matrix and translation vector) at each frame acquisition time. Then, digital image stabilization algorithms are applied to geometrically correct the original environmental image sequence. Specifically: 1) Rotation compensation: Based on the calculated angular motion, the image is rotated in reverse to correct image tilt and jitter caused by natural head movements; 2) Translation compensation: Reverse translation is performed within the image plane to stabilize the position of the hand's region of interest in the frame and reduce inter-frame jumps; 3) Roll shutter compensation: For cameras using roll shutter sensors, image clipping distortion caused by rapid movement is corrected based on motion data. Finally, a stable image sequence with enhanced spatiotemporal continuity between the background and foreground (especially the hand area) is generated. This process effectively suppresses motion blur and inter-frame inconsistencies caused by unavoidable autonomous body micro-movements in first-person perspective, providing high-quality input for visual recognition.

[0049] Step D30: Input the stable image sequence into the first person sign language translation model to obtain the first sign language translation result.

[0050] The stabilized image sequence, optimized for motion compensation, is input into a first-person sign language translation model specifically optimized for first-person perspective. This model is trained on a large amount of first-person sign language data, and its network architecture (such as spatiotemporal convolutional networks or transformers) is specifically designed for close-range, large-scale hand movements, specific occlusion patterns, and relatively simple background features. Because the stability and consistency of the input sequence are guaranteed, the model's feature extraction module can focus more on the spatiotemporal patterns of the gestures themselves, rather than irrelevant motion artifacts. The model performs end-to-end processing on this stabilized sequence, including hand detection, keypoint tracking, gesture recognition, grammar conversion, and natural language generation, ultimately outputting a more accurate and robust first-person sign language translation (text or speech).

[0051] This embodiment, by integrating the aforementioned IMU-based active motion compensation mechanism, fundamentally alleviates the core pain point of first-person application scenarios—image quality degradation caused by user movement. By providing the translation model with "cleaned" visual input, it improves the accuracy of gesture recognition and the overall reliability of the system in dynamic usage environments, enhancing user experience and the practicality of the solution.

[0052] In one possible implementation, the step of calling a third-person sign language translation model to process the environmental image and obtain a second sign language translation result includes: Step E10: Acquire depth data synchronized with the environmental image collected by the depth sensor; Depth data is acquired synchronously with environmental images to ensure temporal consistency. The depth sensor provides depth information for each pixel in the image, helping the system more accurately locate and analyze hand movements in three-dimensional space. Depth data is typically stored in the form of a depth map, where the value of each pixel represents the distance from that pixel to the camera.

[0053] Step E20: Based on the environmental image and the depth data, determine the three-dimensional coordinate sequence of key hand points; After acquiring environmental images and depth data, these two are combined to determine the 3D coordinate sequence of key hand points. The specific process is as follows: First, two-dimensional detection of hand key points is performed using environmental images. A hand key point detection algorithm (such as MediaPipe Hands) can be used to detect the two-dimensional coordinates (x, y) of 21 key points on the hand in real time. These key points include important locations such as the wrist and finger joints. These two-dimensional coordinates reflect the position of the hand on the image plane.

[0054] Next, the two-dimensional coordinates are converted into three-dimensional coordinates using depth data. For each detected keypoint, depth information (z-axis coordinates) is obtained based on its corresponding pixel value in the depth map. By combining the two-dimensional coordinates and depth information, the precise position of the hand keypoints in three-dimensional space can be calculated, forming a three-dimensional coordinate sequence of the hand keypoints. This process considers not only the position of the hand on the image plane but also its position in the depth direction, providing richer spatial information for subsequent sign language translation.

[0055] Step E30: Input the three-dimensional coordinate sequence into the third-person sign language translation model to obtain the second sign language translation result.

[0056] After determining the 3D coordinate sequence of key hand points, this sequence is input into a third-person sign language translation model. This model is specifically designed for sign language translation from a third-person perspective and has been trained on a large amount of third-person sign language data. Its architecture (e.g., based on graph convolutional networks or 3D spatiotemporal transformers) is specifically designed for processing 3D skeletal sequence data, with training data derived from 3D sign language motion capture data from a third-person perspective. The model directly learns and infers from the 3D coordinate sequence, overcoming core challenges faced by purely 2D vision methods in third-person scenes, such as scale blur, perspective distortion, and variations in action appearance due to different shooting angles. Based on these robust 3D spatiotemporal features, the model performs gesture classification, temporal modeling, and language generation, ultimately outputting accurate and stable second-person sign language translation results.

[0057] This embodiment introduces 3D information extraction based on depth sensors, providing essential feature representations for third-person sign language translation that are insensitive to changes in viewpoint and distance. This approach effectively addresses the fundamental challenge of "variable image scale and significant spatial deformation" in third-person observation scenarios. By enhancing the spatial invariance of input features, it significantly improves the generalization ability and robustness of the translation model under complex and variable observation conditions, achieving highly reliable third-person sign language translation.

[0058] In one possible implementation, after obtaining the first sign language translation result or the second sign language translation result, the method further includes: Step F10: Combine the first sign language translation result or the second sign language translation result into a speech signal; The translated text is converted into a speech signal so that users or others can access the translated information through hearing. Mature speech synthesis technologies (such as deep learning-based speech synthesis engines) can be used to generate natural and fluent speech signals based on the translation results. The speech synthesis engine considers factors such as the semantics, grammatical structure, and intonation of the text to ensure that the generated speech signal conforms to the expression habits of natural language in terms of sound quality, speech rate, and intonation, thereby improving the intelligibility and naturalness of the speech signal.

[0059] Step F20: If the current sign language scenario is a first-person expression scenario, then control the speaker to play the voice signal at the first volume parameter; If the current sign language scenario is determined to be a first-person expression scenario, meaning the user is expressing themselves through sign language, a specific volume parameter will be used to output the speech signal, which will then be played out through a speaker. Specifically, a higher volume parameter (the first volume parameter) can be set to ensure the speech signal can be clearly heard from a greater distance. This higher volume setting is suitable for scenarios where the user is in public places or needs to communicate with people at a distance, allowing the translated speech to be effectively conveyed to those around them. By controlling the speaker output, the synthesized speech signal is played out at the set volume, thus achieving effective conversion and dissemination of sign language to speech.

[0060] Step F30: If the current sign language scenario is a third-person observation scenario, control the headphones to play the voice signal at a second volume parameter lower than the first volume parameter, or display the second sign language translation result on the screen.

[0061] If the current sign language scenario is determined to be a third-person observation scenario, meaning the user is observing another person's sign language expression, a different processing approach is adopted. In this scenario, the voice signal is output at a second volume parameter, lower than the first volume parameter. The lower volume parameter is suitable for more private or quiet environments, avoiding excessive interference with the surrounding environment. Furthermore, multiple output options can be provided to accommodate different user needs. Users can choose to play the voice signal through headphones, which is more private and will not disturb those around them.

[0062] In addition, the translation results can be displayed on the screen as text, providing users with another way to receive information. This is especially helpful for users who have difficulty hearing voice signals in noisy environments.

[0063] In one possible implementation, the method further includes: Step G10: Continuously acquire environmental images of consecutive frames; To ensure the real-time performance and accuracy of the sign language translation system, continuous frame images of the environment are acquired via a camera, ensuring that dynamic changes in sign language gestures can be captured in real time. The camera captures images at a certain frame rate (such as 30 frames per second or higher), and these continuous frame images enable real-time monitoring of changes in the sign language scene.

[0064] Step G20: For each frame of the environmental image, perform the step of determining the current sign language scene based on the environmental image; For each frame of the acquired environmental image, the step of determining the current sign language scene based on the environmental image is performed. Each frame is analyzed in this way to determine whether the current sign language scene is a first-person expression scene or a third-person observation scene. Specific analysis processes include, but are not limited to, detecting key hand points, analyzing hand position and movement trajectory, and performing 3D spatial analysis using depth data. Through these analyses, the type of sign language scene corresponding to each frame can be quickly and accurately determined.

[0065] Step G30: When it is detected that the current sign language scene switches from a first-person expression scene to a third-person observation scene or vice versa in consecutive preset frames, the called sign language translation model is switched synchronously.

[0066] During continuous monitoring of sign language scenarios, attention is paid to scenario transitions. If a transition is detected within a consecutive preset number of frames (e.g., 5 or 10 consecutive frames), such as switching from a first-person expression scenario to a third-person observation scenario, or vice versa, the invoked sign language translation model is switched accordingly. This switching process is automatic and real-time, ensuring immediate adaptation to new scenario types and the invocation of the corresponding optimized sign language translation model. For example, if switching from first-person to third-person, the first-person sign language translation model is discontinued, and the third-person sign language translation model is switched, and vice versa. This dynamic switching mechanism guarantees translation accuracy and efficiency across different scenarios, providing users with a seamless translation experience.

[0067] Furthermore, embodiments of this application also propose a wearable device, the wearable device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the method described above.

[0068] like Figure 5As shown, the wearable device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the wearable device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the wearable device to communicate wirelessly or wiredly with other devices to exchange data. While wearable devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0069] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0070] The wearable device provided in this application, employing the sign language translation method described in the above embodiments, can solve the technical problem of insufficient cross-perspective generalization ability caused by the performance limitations of the sign language translation model in cross-perspective applications. Compared with the prior art, the beneficial effects of the wearable device provided in this application are the same as those of the sign language translation method provided in the above embodiments, and other technical features of the wearable device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0071] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0072] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0073] In addition, to achieve the above objectives, this application also provides a readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the sign language translation method described in the above embodiments.

[0074] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0075] The aforementioned computer-readable storage medium may be included in the wearable device; or it may exist independently and not assembled into the wearable device.

[0076] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by a wearable device, cause the wearable device to perform the process steps of any embodiment of the aforementioned sign language translation method.

[0077] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0078] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0079] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the modules themselves.

[0080] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described sign language translation method. This addresses the technical problem of performance limitations in cross-perspective applications of sign language translation models, leading to insufficient cross-perspective generalization ability. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the sign language translation method provided in the above embodiments, and will not be elaborated upon here.

[0081] Furthermore, embodiments of this application also propose a computer program product, including a computer program that, when executed by a processor, implements the steps of the sign language translation method described above.

[0082] The specific implementation of the computer program product in this application is basically the same as the various embodiments of the sign language translation method described above, and will not be repeated here.

[0083] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0084] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0085] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software sensor. This computer software sensor is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a wearable device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0086] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A sign language translation method, characterized in that, The sign language translation method includes the following steps: In response to a sign language translation request, an environmental image is acquired, and the current sign language scene is determined based on the environmental image; If the current sign language scene is a first-person expression scene, then the first-person sign language translation model is invoked to process the environmental image to obtain the first sign language translation result; If the current sign language scene is a third-person observation scene, then the third-person sign language translation model is invoked to process the environmental image to obtain the second sign language translation result.

2. The sign language translation method as described in claim 1, characterized in that, Determining the current sign language scene based on the environmental image includes: Face detection and hand detection are performed on the environmental image to obtain face detection results and hand detection results; If the face detection result indicates that a face has been detected, and the hand detection result indicates that the hand area is located in front of the face area, then the current sign language scene is determined to be a third-person observation scene. If the face detection result indicates that no face was detected, or the hand detection result indicates that the hand area is not in front of the face area, then the current sign language scene is determined to be a first-person expression scene.

3. The sign language translation method as described in claim 2, characterized in that, Before the step of determining that the current sign language scene is a first-person expression scene, the method further includes: Based on the environmental image, the initial position and movement trajectory of key hand points are detected; If the initial position is located within a preset bottom area of ​​the environmental image, and the movement trajectory points to the center area of ​​the environmental image, then the step of determining the current sign language scene as a first-person expression scene is executed; If the initial position is not located within the preset bottom area, or if the movement trajectory points to the side area of ​​the environmental image, then the current sign language scene is determined to be a third-person observation scene.

4. The sign language translation method as described in claim 2, characterized in that, Before the step of determining that the current sign language scene is a first-person expression scene, the method further includes: Acquire depth data synchronized with the environmental image collected by the depth sensor, and obtain the coordinates of the elbow joint and the wrist joint based on the depth data; Calculate the forearm space vector pointing from the elbow joint coordinates to the wrist joint coordinates; Analyze the variation magnitude of the depth axis component of the forearm spatial vector in consecutive time frames; If the change is greater than a preset threshold, then the step of determining the current sign language scene as a first-person expression scene is executed. If the change is less than or equal to a preset threshold, the current sign language scene is determined to be a third-person observation scene.

5. The sign language translation method as described in any one of claims 1 to 4, characterized in that, The step of calling a first-person sign language translation model to process the environmental image and obtain a first-person sign language translation result includes: Acquire motion data collected by the inertial measurement unit; Motion compensation is performed on the environmental image based on the motion data to obtain a stable image sequence; The stable image sequence is input into the first-person sign language translation model to obtain the first sign language translation result.

6. The sign language translation method according to any one of claims 1 to 4, characterized in that, The step of calling a third-person sign language translation model to process the environmental image and obtain a second sign language translation result includes: Acquire depth data synchronized with the environmental image collected by the depth sensor; Based on the environmental image and the depth data, a three-dimensional coordinate sequence of key hand points is determined; The three-dimensional coordinate sequence is input into the third-person sign language translation model to obtain the second sign language translation result.

7. The sign language translation method according to any one of claims 1 to 4, characterized in that, After obtaining the first sign language translation result or the second sign language translation result, the method further includes: The first sign language translation result or the second sign language translation result is combined into a speech signal; If the current sign language scenario is a first-person expression scenario, then control the speaker to play the voice signal at the first volume parameter; If the current sign language scenario is a third-person observation scenario, then control the headphones to play the voice signal at a second volume parameter lower than the first volume parameter, or display the second sign language translation result on the screen.

8. The sign language translation method according to any one of claims 1 to 4, characterized in that, The method further includes: Continuously acquire environmental images in consecutive frames; For each frame of the environment image, perform the step of determining the current sign language scene based on the environment image; When it is detected that the current sign language scene switches from a first-person expression scene to a third-person observation scene, or from a third-person observation scene to a first-person expression scene, the sign language translation model being called is switched synchronously.

9. A wearable device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the sign language translation method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a device control program, which, when executed by a processor, implements the steps of the sign language translation method as described in any one of claims 1 to 8.

Images