Robot sign language communication method, related device and storage medium
By collecting and analyzing the user's gesture image characteristics, combining the multimodal language model to generate response text that conforms to sign language grammar, driving the robot to simulate sign language actions, solving the problem of inconvenient communication in the existing sign language translation system and achieving efficient communication between deaf and mute people.
Patent Information
- Application Number
- CN202510715809.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-29
- Publication Date
- 2025-09-02
AI Technical Summary
The existing sign language translation system cannot accurately express the meaning of the deaf and mute, and requires continuous alternating equipment for interaction, resulting in inconvenience and inefficiency of communication.
The user's gesture image features are collected, the gesture symbol sequence is recognized through the pre-trained gesture recognition model, and text correction is carried out in combination with the multimodal large language model and visual language model to generate response text that conforms to sign language grammar, and finally drive the robot to simulate sign language actions.
It realizes accurate expression of user meaning, improves communication efficiency between deaf and mute people and healthy listeners, and simplifies the handling of life matters.
Smart Images

Figure CN120578294A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of sign language communication, and in particular to a robot sign language communication method and related devices and storage media. Background Art
[0002] With the advancement of society, the needs of the deaf-mute community are gradually gaining attention. Sign language, as the primary means of communication for the deaf-mute, is widely used within certain groups. However, in daily life, most people do not understand sign language, resulting in significant communication barriers between the deaf-mute and hearing people. This prevents the deaf-mute from effectively conducting business, purchasing items, and other daily tasks.
[0003] Sign language translation systems have emerged to facilitate communication with the deaf and mute. Existing sign language translation systems primarily collect images of the deaf user's sign language movements. These systems then identify the user's hand movements by sequentially analyzing the images. Based on a mapping relationship between hand movements and vocabulary, these systems map the corresponding vocabulary words and provide feedback. In response, the deaf user hands the device to a hearing user, who then inputs a response after consulting the sign language recognition results. Based on this mapping relationship, these systems then directly map each vocabulary word to a hand movement and provide the deaf user with reference.
[0004] However, the existing methods only focus on the user's hand movements, while sign language is a visual language that includes not only gestures, but also expressions, postures, etc., and the meaning is different in different scenarios, so the existing analysis methods cannot achieve accurate communication. Moreover, the grammar of sign language is different from the grammatical expression of normal spoken language, so the existing direct mapping method will lead to semantic incomprehension and fail to accurately express the user's meaning. In addition, it is necessary to constantly switch devices for communication, which is very inconvenient and inefficient, so deaf and mute users still have a lot of trouble in handling business, purchasing items and other daily matters. Summary of the Invention
[0005] Based on the above-mentioned deficiencies of the prior art, the present application provides a robot sign language communication method and related devices and storage media to solve the problem that the prior art cannot easily and accurately communicate.
[0006] In order to achieve the above objectives, this application provides the following technical solutions:
[0007] The first aspect of the present application provides a robot sign language communication method, comprising:
[0008] Collecting each frame of gesture images of the current user and extracting features of each frame of the gesture image; wherein the features of the gesture image include features of multiple parts;
[0009] Inputting the features of the gesture image of each frame into a pre-trained gesture recognition model, and identifying a current gesture recognition result through the gesture recognition model; wherein the current gesture recognition result at least includes a sequence of each recognized gesture symbol;
[0010] Based on the mapping relationship between gesture symbols and vocabulary, mapping the current gesture recognition result to obtain a vocabulary sequence;
[0011] Composing the vocabulary sequence into the current complete recognition text according to the preset grammatical rules;
[0012] Correcting the current complete recognition text by combining the current complete recognition text, current scene understanding information, and current action understanding information with a multimodal large language model; wherein the current scene understanding information includes environmental information and emotional state information obtained by performing scene analysis on the current scene image of the user using a first visual language model; and the current action understanding information includes gesture information and semantic information obtained by performing action analysis on a sequence of gesture images of each frame using a second visual language model;
[0013] Inputting the corrected current complete recognition text, the current scene image, and the historical communication information into a third visual language model, and analyzing the current response text through the third visual language model;
[0014] Converting the current response text into a current sign language sequence; wherein the current sign language sequence is a trajectory sequence of each joint point of the sign language action;
[0015] The robot is driven to simulate sign language movements according to the current sign language word sequence.
[0016] Optionally, in the above-mentioned robot sign language communication method, collecting each frame of gesture image of the current user and extracting features of each frame of the gesture image includes:
[0017] Collecting a video stream of gestures of the current user;
[0018] Extracting basic frames from the gesture video stream at fixed intervals to obtain multiple frames of gesture images;
[0019] Locating the upper half of the human body in each frame of the gesture image using a detection algorithm;
[0020] Extracting information of key points of the upper half of the human body in each frame of the gesture image; wherein the key points include at least a plurality of key points of the hands, arms, face, and upper body;
[0021] Supplementing a gesture image between each two frames of the gesture image based on the distance between the key points of the hand in each two adjacent frames of the gesture image;
[0022] Extracting information of key points of the upper half of the human body in each frame of the supplemented gesture image;
[0023] The key point information of each part of the gesture image of each frame is normalized based on the reference of each part, and the normalized key point information of each part of the gesture image is used as the feature of the gesture image.
[0024] Optionally, in the above-mentioned robot sign language communication method, the training method of the gesture recognition model includes:
[0025] Constructing the initial gesture recognition model;
[0026] Acquire a sign language dataset, and extract key point information of each frame of sign language image in the sign language dataset;
[0027] By transforming the key points of the sign language image of each frame, data enhancement is performed on the information of the key points of the sign language image of each frame;
[0028] Normalizing the key point information of each frame of the sign language image after data enhancement to obtain training sample data;
[0029] The pre-trained gesture recognition model is fine-tuned and iteratively trained using the training sample data until the loss converges.
[0030] Optionally, in the above-mentioned robot sign language communication method, inputting the features of the gesture image of each frame into a pre-trained gesture recognition model, and identifying the current gesture recognition result through the gesture recognition model includes:
[0031] Performing image extraction on the sequence of the gesture images of each frame through a sliding window to obtain a plurality of window image data; wherein each of the window image data includes one or more consecutive frames of the gesture image;
[0032] Inputting the features of each of the window image data into a pre-trained gesture recognition model, and identifying the gesture category probability distribution corresponding to each of the window image data through the gesture recognition model;
[0033] Performing time-series decoding on the probability distribution of gesture categories corresponding to each of the window image data to obtain a gesture symbol sequence corresponding to each of the window image data;
[0034] Calculating the confidence of the gesture symbol sequence corresponding to each of the window image data;
[0035] taking the gesture symbol sequence corresponding to each window image data having a confidence level not less than a confidence threshold as the recognition result of each window image data, and smoothing the gesture symbol sequence corresponding to each window image data having a confidence level less than the confidence threshold with the gesture symbol sequence of the adjacent window image data;
[0036] Merging the recognition results of the overlapping parts of the window image data to obtain a current gesture recognition result;
[0037] The current gesture recognition result is corrected using an N-gram language model.
[0038] Optionally, in the above-mentioned robot sign language communication method, converting the current response text into a current sign language sequence includes:
[0039] Segmenting the current response text and performing part-of-speech tagging on the segmented words;
[0040] Using dependency syntax analysis to construct a grammatical structure tree of the current response text, and extracting keywords from the current response text;
[0041] Based on the grammatical rules of sign language, the current response text is adjusted in combination with the segmentation and its annotation, the grammatical structure tree, and keywords in the current response text to obtain a sign language response text that conforms to the grammatical rules of sign language;
[0042] Converting the sign language response text based on a mapping relationship between gesture symbols and vocabulary to obtain various response gesture symbols;
[0043] Assigning a timestamp to each of the response gesture symbols to obtain a response gesture symbol sequence;
[0044] Inputting the response gesture symbol sequence, the historical trajectory of the joint points, and the visual context into a pre-trained sign language sequence generation model to generate a sign language sequence corresponding to the response gesture symbol sequence;
[0045] Performing time sequence alignment and smoothing optimization on the sign language sequence corresponding to the response gesture symbol sequence to obtain the current sign language sequence.
[0046] Optionally, in the above-mentioned robot sign language communication method, driving the robot to simulate sign language movements according to the current sign language word order includes:
[0047] Based on the current sign language sequence, calculating the motion angles of each joint of the arm and fingers using an inverse kinematics algorithm to obtain sign language joint sequence data; wherein the motion angle of each joint is no greater than a corresponding maximum value;
[0048] Based on the sign language joint sequence data and the degrees of freedom of each joint of the robot's hand, a smooth joint trajectory is generated through multiple polynomial differences, and each joint of the robot's hand is controlled to move according to the smooth joint trajectory.
[0049] Optionally, in the above-mentioned robot sign language communication method, the method further includes:
[0050] Performing language conversion on the current response text to obtain a current response voice;
[0051] Aligning the current response speech with morphemes, and converting the aligned morphemes into lip data using a speech synthesis markup language;
[0052] Play the current response voice and the lip shape data.
[0053] Optionally, in the above-mentioned robot sign language communication method, the method further includes:
[0054] When a text containing a wake-up word is currently detected, the robot is switched from a sleep state to a listening state to monitor the generated current complete recognition text in real time;
[0055] When no input is received within a preset time, returning the robot to the sleep state;
[0056] When the current complete recognition text is generated within the preset time, the current complete recognition text is forwarded to the multimodal large language model and switched to a waiting state;
[0057] When the processing result of the multimodal large language model is received, the state is switched to the play state to start the text-to-sign language conversion action;
[0058] When controlling the robot to start playing the sign language action, switching to the waiting end state;
[0059] When the sign language action playback is finished, switch to the monitoring state.
[0060] A second aspect of the present application provides a robot sign language communication device, comprising:
[0061] An image acquisition unit, configured to acquire each frame of gesture images of the current user and extract features of each frame of the gesture image; wherein the features of the gesture image include features of multiple parts;
[0062] A gesture recognition unit, configured to input the features of the gesture image of each frame into a pre-trained gesture recognition model, and identify a current gesture recognition result through the gesture recognition model; wherein the current gesture recognition result at least includes a sequence of each recognized gesture symbol;
[0063] a first mapping unit, configured to map the current gesture recognition result based on a mapping relationship between gesture symbols and vocabulary to obtain a vocabulary sequence;
[0064] A word order adjustment unit, configured to compose the current complete recognition text according to preset grammatical rules according to the word sequence;
[0065] a text optimization unit, configured to modify the current complete recognized text by combining the current complete recognized text, current scene understanding information, and current action understanding information using a multimodal large language model; wherein the current scene understanding information includes environmental information and emotional state information obtained by performing scene analysis on the user's current scene image using a first visual language model; and the current action understanding information includes gesture information and semantic information obtained by performing action analysis on a sequence of gesture images in each frame using a second visual language model;
[0066] a reply analysis unit, configured to input the corrected current complete recognized text, the current scene image, and the historical communication information into a third visual language model, and analyze the current response text using the third visual language model;
[0067] A trajectory conversion unit, configured to convert the current response text into a current sign language sequence; wherein the current sign language sequence is a trajectory sequence of each joint point of the sign language action;
[0068] The simulation unit is used to drive the robot to simulate sign language movements according to the current sign language word order.
[0069] Optionally, in the above-mentioned robot sign language communication device, the image acquisition unit includes:
[0070] A video acquisition unit, configured to acquire a video stream of the user's gestures;
[0071] An image extraction unit, configured to extract basic frames from the gesture video stream at fixed intervals to obtain multiple frames of gesture images;
[0072] A region detection unit, configured to locate the upper half of the human body in each frame of the gesture image using a detection algorithm;
[0073] A first extraction unit is configured to extract information of key points of the upper half of the human body in each frame of the gesture image; wherein the key points include at least a plurality of key points of the hands, arms, face, and upper body;
[0074] an image supplementing unit, configured to supplement a gesture image between each two frames of the gesture image based on a distance between key points of the hand in each two adjacent frames of the gesture image;
[0075] A second extraction unit is used to extract information of each key point of the upper half of the human body in each frame of the gesture image supplemented;
[0076] The first normalization processing unit is used to normalize the information of the key points of each part of the gesture image in each frame based on the reference of each part, and use the normalized information of the key points of each part of the gesture image as the feature of the gesture image.
[0077] Optionally, the above-mentioned robot sign language communication device further includes:
[0078] A model building unit, configured to build an initial gesture recognition model;
[0079] A sample data extraction unit, configured to obtain a sign language dataset and extract key point information of each frame of sign language image in the sign language dataset;
[0080] a data enhancement unit, configured to enhance the information of the key points of the sign language image of each frame by transforming the key points of the sign language image of each frame;
[0081] a second normalization processing unit, configured to perform normalization processing on information of key points of each frame of the sign language image after data enhancement to obtain training sample data;
[0082] An iterative unit is used to perform fine-tuning iterative training on the pre-trained gesture recognition model using the training sample data until the loss converges.
[0083] Optionally, in the above-mentioned robot sign language communication device, the gesture recognition unit includes:
[0084] An image extraction unit, configured to extract images from a sequence of gesture images of each frame through a sliding window to obtain a plurality of window image data; wherein each window image data includes one or more consecutive frames of the gesture image;
[0085] a probability analysis unit, configured to input features of each of the window image data into a pre-trained gesture recognition model, and identify a probability distribution of gesture categories corresponding to each of the window image data through the gesture recognition model;
[0086] A decoding unit, configured to perform time-series decoding on the probability distribution of gesture categories corresponding to each of the window image data to obtain a gesture symbol sequence corresponding to each of the window image data;
[0087] A confidence calculation unit, configured to calculate the confidence of the gesture symbol sequence corresponding to each of the window image data;
[0088] a processing unit, configured to take the gesture symbol sequence corresponding to each window image data having a confidence level not less than a confidence threshold as a recognition result of each window image data, and to smooth the gesture symbol sequence corresponding to each window image data having a confidence level less than the confidence threshold with the gesture symbol sequence of adjacent window image data;
[0089] a merging unit, configured to merge the recognition results of the overlapping parts of the image data of each window to obtain a current gesture recognition result;
[0090] The result correction unit is used to correct the current gesture recognition result by using the N-gram language model.
[0091] Optionally, in the above-mentioned robot sign language communication device, the trajectory conversion unit includes:
[0092] A word segmentation processing unit, configured to segment the current response text and perform part-of-speech tagging on the segmented words;
[0093] A syntactic analysis unit, configured to construct a grammatical structure tree of the current response text using dependency syntactic analysis, and extract keywords from the current response text;
[0094] a text adjustment unit, configured to adjust the current response text based on the grammatical rules of sign language, in combination with the word segmentation and its annotation, the grammatical structure tree, and keywords in the current response text, to obtain a sign language response text that conforms to the grammatical rules of sign language;
[0095] a second mapping unit, configured to convert the sign language response text based on a mapping relationship between gesture symbols and vocabulary to obtain various response gesture symbols;
[0096] a time allocating unit, configured to allocate a time stamp to each of the response gesture symbols to obtain a response gesture symbol sequence;
[0097] a sign language sequence generation unit, configured to input the response gesture symbol sequence, the historical trajectory of the joint points, and the visual context into a pre-trained sign language sequence generation model to generate a sign language sequence corresponding to the response gesture symbol sequence;
[0098] The sequence smoothing unit is used to perform time sequence alignment and smoothing optimization on the sign language sequence corresponding to the response gesture symbol sequence to obtain the current sign language sequence.
[0099] Optionally, in the above-mentioned robot sign language communication device, the simulation unit includes:
[0100] An angle analysis unit is configured to calculate the motion angles of each joint of the arm and fingers based on the current sign language sequence using an inverse kinematics algorithm to obtain sign language joint sequence data; wherein the motion angle of each joint is not greater than a corresponding maximum value;
[0101] An estimation smoothing unit is used to generate a smooth joint trajectory through multiple polynomial differences based on the sign language joint sequence data and the degrees of freedom of each joint of the robot's hand, and control each joint of the robot's hand to move according to the smooth joint trajectory.
[0102] Optionally, the above-mentioned robot sign language communication device further includes:
[0103] A voice conversion unit, configured to convert the current response text into a language to obtain a current response voice;
[0104] a lip-shape conversion unit, configured to align the current response speech with morphemes and convert the aligned morphemes into lip-shape data using a speech synthesis markup language;
[0105] The language playing unit is used to play the current response voice and the lip shape data.
[0106] Optionally, the above-mentioned robot sign language communication device further includes:
[0107] A first switching unit is configured to switch the robot from a sleep state to a listening state when a text containing a wake-up word is currently detected, so as to monitor the generated current complete recognition text in real time;
[0108] a state returning unit, configured to return the robot to the sleep state when no input is received within a preset time;
[0109] A second switching unit is configured to forward the current complete recognition text to the multimodal large language model and switch to a waiting state when the current complete recognition text is generated within the preset time;
[0110] a third switching unit, configured to switch to a play state to start a text-to-sign language conversion action when waiting for a processing result of the multimodal large language model;
[0111] A fourth switching unit, configured to switch to a waiting end state when controlling the robot to start playing the sign language action;
[0112] The fifth switching unit is used to switch to the monitoring state when the sign language action playback is completed.
[0113] A third aspect of the present application provides a computer storage medium for storing a computer program, wherein when the computer program is executed by a processor, it is used to implement the robot sign language communication method as described in any one of the above.
[0114] This application provides a robot sign language communication method that collects gesture images of the current user and extracts features from each gesture image. The gesture image features encompass features from multiple locations, focusing not only on the hands, ensuring accurate recognition. The features of each gesture image are then input into a pre-trained gesture recognition model, which then generates a current gesture recognition result. The current gesture recognition result includes at least a sequence of recognized gesture symbols. Based on a mapping relationship between gesture symbols and vocabulary, the current gesture recognition result is mapped to a vocabulary sequence. This vocabulary sequence is then combined according to preset grammatical rules to form a complete recognized text, ensuring that the word order of the text conforms to spoken language grammar. Furthermore, the current complete recognized text is corrected using a multimodal large language model, combining the current complete recognized text, current scene understanding information, and current action understanding information. The current scene understanding information includes environmental information and emotional state information obtained by performing scene analysis on the user's current scene image using a first visual language model. The current action understanding information includes gesture information and semantic information obtained by performing action analysis on the sequence of gesture images using a second visual language model. Thus, the missing and repeated content caused by the differences between the grammatical expressions of sign language and spoken language are processed to correct and optimize the generated text, ensure the accuracy and naturalness of the text, and accurately express the user's meaning. Then, the corrected current complete recognition text, the current scene image, and the historical communication information are input into the third visual language model, and the current response text is analyzed by the third visual language model, and the current response text is converted into the current sign language sequence. Among them, the current sign language sequence is the trajectory sequence of each joint point of the sign language action. Finally, according to the current sign language word order, the robot is driven to simulate the sign language action, so that the content of the reply to the user can be automatically obtained, and the user can be replied through sign language, thereby realizing sign language communication with the user, making it easier to communicate with the user and thus easier to achieve various matters. BRIEF DESCRIPTION OF THE DRAWINGS
[0115] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.
[0116] Figure 1A flowchart of a robot sign language communication method provided in an embodiment of the present application;
[0117] Figure 2 A flowchart of a method for collecting gesture images and extracting features provided in an embodiment of the present application;
[0118] Figure 3 A flowchart of a method for training a gesture recognition model provided in an embodiment of the present application;
[0119] Figure 4 A flowchart of a method for identifying a current gesture recognition result provided by an embodiment of the present application;
[0120] Figure 5 A schematic diagram of the architecture of a multimodal large language model provided in an embodiment of the present application;
[0121] Figure 6 A schematic diagram of the architecture of a third visual language model provided in an embodiment of the present application;
[0122] Figure 7 A flowchart of converting a current response text into a current sign language sequence is provided in an embodiment of the present application;
[0123] Figure 8 A flowchart of a method for outputting response information in a language provided in an embodiment of the present application;
[0124] Figure 9 A flowchart of a method provided in this application for driving a robot to simulate sign language movements according to the current sign language word order;
[0125] Figure 10 A schematic diagram of a robot execution state machine provided in an embodiment of the present application;
[0126] Figure 11 A schematic diagram of the architecture of a robot sign language communication device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0127] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0128] In this application, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.
[0129] Another embodiment of the present application provides a robot sign language communication method, such as Figure 1 As shown, the following steps are included:
[0130] S101 : collecting gesture images of the current user and extracting features of the gesture images of the frames.
[0131] The features of the gesture image include features of multiple parts. Optionally, the features of the gesture image can be key position features of each part, or features of image dimensions, such as pixel features.
[0132] It should be noted that in the embodiment of the present application, not only the user's hand movements are focused on, but also their arms and face, so the collected gesture image needs to include at least the user's upper body.
[0133] Optionally, in another embodiment of the present application, a specific implementation of step S101 is as follows: Figure 2 As shown, the following steps are included:
[0134] S201: Collect the gesture video stream of the current user.
[0135] In order to effectively obtain continuous multi-frame gesture images, in an embodiment of the present application, when the user expresses himself in sign language, the user's video stream is directly captured to obtain the user's current gesture video stream.
[0136] S202 : Extracting basic frames from the gesture video stream at fixed intervals to obtain multiple frames of gesture images.
[0137] Specifically, basic frames are extracted at fixed time intervals to ensure that basic timing information is complete.
[0138] S203 , locating the upper half of the human body in each frame of the gesture image using a detection algorithm.
[0139] To accurately identify the key points of the hands, arms, and face, we first locate the upper body. Alternatively, we can use YOLO or OpenCV to locate the upper body to ensure complete coverage of key action points and remove redundant information.
[0140] S204: Extract information of key points of the upper half of the human body in each frame of the gesture image.
[0141] The key points include at least multiple key points of the hands, arms, face and upper body.
[0142] Optionally, the hand specifically refers to the palm part, and its key points may include the center of the palm, the five finger joints and the fingertips, which are the main information for recognizing sign language. The key points of the arm may include the wrist, elbow, shoulder, etc., which are convenient for recognizing large gestures. The key points of the face may include eyebrows, mouth, eyes, nose, etc., to assist in distinguishing different sign language expressions. The key points of the upper body may include key points of the neck, shoulders, torso, etc., to provide a spatial reference for the overall posture, ensuring that correct gesture recognition is maintained when the hand or arm is partially blocked.
[0143] Optionally, the key point extraction structure data of the key points is as follows. Where n, m, p, q represent the number of key points of the hands, arms, face, and upper body, respectively. And the key point feature vector of a video frame is the key point coordinates of different parts spliced into a vector in sequence: [
[0145] {
[0146] "frame": 1,
[0147] "hand_keypoints": ,
[0148] "arm_keypoints": ,
[0149] "face_keypoints": ,
[0150] "body_keypoints":
[0151] },
[0152] {
[0153] "frame": 2,
[0154] "hand_keypoints":[[, ], ..., [, ]],
[0155] "arm_keypoints":[...],
[0156] "face_keypoints":[...],
[0157] "body_keypoints":[...]
[0158] }
[0159] … ]
[0161] S205 : Supplement a gesture image between every two frames of gesture images based on the distance between the key points of the hand in every two adjacent frames of gesture images.
[0162] It's important to note that to ensure consistency between adjacent frames and accurately identify gestures, one or two additional frames are needed between relatively discontinuous gesture images. Since the hands are the primary expression of sign language, the distance between key points of the arms in the gesture image can be used to determine consistency. A large distance indicates poor consistency, and in this case, one or two additional frames can be linearly generated to supplement the two frames.
[0163] Specifically, the distance of the key points of the hand is calculated as follows:
[0164]
[0165] in, is the number of hand key points, is the coordinate of the i-th key point in the current frame t, is the corresponding coordinate of the previous frame t-1, and D represents the change in the movement of the entire hand. Correspondingly, if When θ (threshold), the specific formula for inserting the supplementary frame is as follows:
[0166]
[0167] so, Indicates that the change in motion between two frames is small, that is, small motion, and no frame insertion is required; It is a medium motion, and 1 frame can be inserted according to the linear interpolation formula. Indicates that a large amount of motion has occurred, so two frames are inserted. Two thresholds 、 This can be determined by statistically observing the motion amplitude data, that is, adjusting the frame rate (usually the motion amplitude is smaller when the frame rate is higher, and the motion amplitude is larger when the frame rate is lower) to find the baseline value.
[0168] S206: Extract information of key points of the upper half of the human body in each frame of the supplemented gesture image.
[0169] Since the supplementary gesture images also need to be subsequently processed, the same method is used to extract the information of the same key points.
[0170] S207 , normalizing the key point information of each part of each frame of the gesture image based on the reference of each part, and using the normalized key point information of each part of the gesture image as the feature of the gesture image.
[0171] In order to normalize the key point coordinates to a uniform size, eliminate individual differences, and facilitate model processing, the key points of different parts are normalized using corresponding benchmarks to ensure that the proportions of each part are consistent.
[0172] Optionally, the key points of the hand can be normalized based on the palm width or height. For example, assuming the palm width is , in wrist or palm coordinates As a reference point, each hand key point after normalization Converts to:
[0173]
[0174] Normalized This ensures that no matter how the palm size changes, the key point data are compared at a uniform scale.
[0175] Arm key points: Normalize based on shoulder width. Assume that the coordinates of the left and right shoulder key points are and , then the shoulder width is:
[0176]
[0177] Also define the shoulder center point is the mean of the two shoulders:
[0178]
[0179] So for any arm key point , normalized to:
[0180]
[0181] This treatment can ensure that the difference in upper limb size between different individuals has less impact on recognition.
[0182] Facial key points: Normalized based on the eye distance. Let the coordinates of the left and right eye key points be and , calculate the interocular distance:
[0183]
[0184] Define the center point of the face The midpoint between the two eyes:
[0185]
[0186] For any facial landmark Normalization:
[0187]
[0188] This normalization method can effectively eliminate the influence of face size and distance.
[0189] Other body (upper body or torso) key points: normalized based on torso length. Assume the neck is selected as the reference point and the torso base length is defined. , for the key points of the upper body :
[0190]
[0191] here, It can be calculated based on the key points of the torso (the distance from the neck to the waist).
[0192] S102: Input the features of each frame of gesture image into a pre-trained gesture recognition model, and use the gesture recognition model to identify a current gesture recognition result.
[0193] The current gesture recognition result includes at least a sequence of recognized gesture symbols. Each gesture symbol represents a corresponding gesture category, i.e., a gesture representing a specific meaning. Therefore, in this embodiment of the present application, a pre-trained gesture recognition model is used to analyze the features of each frame of gesture image to determine a sequence of symbols representing the gestures performed sequentially by the user.
[0194] Optionally, another embodiment of the present application provides a method for training a gesture recognition model, such as Figure 3 As shown, the following steps are included:
[0195] S301: Build an initial gesture recognition model.
[0196] Optionally, the gesture recognition model can be based on temporal modeling, such as the long short-term memory network (LSTM), which is used to capture the dynamic temporal information of gestures and is suitable for processing changes in key points in consecutive frames. Alternatively, it can be based on topological structure modeling, such as the convolutional neural network (GCN) shown in the figure, to model the topological structure between key points and improve the accuracy of sign language recognition.
[0197] S302: Acquire a sign language dataset, and extract key point information of each frame of sign language image in the sign language dataset.
[0198] Alternatively, you can use a sign language dataset from the current work situation or create your own dataset for model training. However, the sign language dataset must cover a variety of shooting angles, lighting conditions, and individual differences to improve the generalization of the model.
[0199] It should be noted that in this embodiment of the present application, the training model uses information about key points of various parts as features, so it is necessary to extract information about key points of each sign language image frame in the sign language dataset. The specific extraction method and information extracted can be found in steps S203 to S206 above and will not be repeated here.
[0200] S303: Transforming the key points of each frame of sign language image to enhance the data of the key points of each frame of sign language image.
[0201] In order to increase the robustness of the model in various scenarios, in an embodiment of the present application, the key points of each frame of sign language image are transformed to enhance the training data.
[0202] Optionally, key point information enhancement can be achieved by performing operations such as translation, rotation, and scaling on the data.
[0203] Specifically, for translation, you can randomly offset the key point position within a certain range. The formula is as follows:
[0204]
[0205] The original coordinates are , the coordinates after translation are , and It is a random value within a predetermined range. All key points of the hands, arms, face, and torso are translated simultaneously to maintain the relative position relationship between each part.
[0206] Rotation: Perform a small angle rotation around a specific center point to simulate a change in viewing angle. The formula is as follows:
[0207]
[0208] The center of rotation is , rotation angle Within the predetermined range value, is the maximum rotation angle, and the original key point coordinates are , the coordinates after rotation are Usually a key point of the body (such as the center of the torso) is selected as the rotation center, and all key points are rotated to maintain overall consistency.
[0209] Scaling: Maintain the proportional relationship between each part and scale it to simulate the change of distance. The formula is as follows:
[0210]
[0211] The scaling ratio is , randomly select values within a range close to 1, and the original key point coordinates are , the zoom center is , the scaled coordinates are Usually, the center of the torso or the center of the camera view is used as the scaling center, and all key points are scaled to keep the overall proportions unchanged.
[0212] Add noise: Add small random noise to the key point coordinates to simulate detection errors and improve model robustness. The formula is as follows:
[0213]
[0214] in are the original key point coordinates, is A random value in the range, is the noise amplitude, and the coordinates after adding noise are Noise is added to all key points to simulate the detection error in real scenes.
[0215] S304: Normalize the key point information of each frame of sign language image after data enhancement to obtain training sample data.
[0216] For the specific normalization process, reference may be made to the specific process of step S207 .
[0217] S305 , fine-tuning the pre-trained gesture recognition model using the training sample data and iteratively training until the loss converges.
[0218] It should be noted that in the embodiments of this application, the constructed model is first pre-trained on a large-scale video or image dataset to enable it to process images. Then, based on the pre-trained model, training for the sign language recognition task is performed through fine-tuning, thereby accelerating model convergence and improving model performance.
[0219] Optionally, in another embodiment of the present application, a specific implementation of step S102 is as follows: Figure 4 As shown, the following steps are included:
[0220] S401 : performing image extraction on a sequence of gesture images of each frame through a sliding window to obtain a plurality of window image data.
[0221] Each window image data includes one or more frames of continuous gesture images.
[0222] It should be noted that a single gesture may capture multiple frames of image data. Therefore, a sliding window approach can be used to select multiple consecutive frames of gesture images and process them as a whole to identify each gesture. Of course, since subsequent fusion will also be performed, each frame can also be processed separately, meaning each window of image data contains one frame of image data.
[0223] Specifically, the size of the timing window is set, that is, N frames of gesture images and the number of overlapping frames of adjacent serial ports are taken, and then the image selection is performed with the size of the timing window and the number of overlapping frames as the step number.
[0224] S402: Input the features of each window image data into a pre-trained gesture recognition model, and use the gesture recognition model to identify the probability distribution of gesture categories corresponding to each window image data.
[0225] It should be noted that the features of each frame image in the window image data are arranged into a sequence according to the order of the images and input into a pre-trained gesture recognition model, which analyzes the probability that the gestures of each frame image in the window image data belong to each gesture category, that is, the gesture category probability distribution corresponding to the window image data.
[0226] S403 : Perform time-series decoding on the probability distribution of gesture categories corresponding to the image data of each window to obtain a gesture symbol sequence corresponding to the image data of each window.
[0227] It should be noted that the gesture category with the highest probability is selected and formed into a sequence, and then the category is decoded into a corresponding gesture serial number to obtain a gesture symbol sequence corresponding to each window image data.
[0228] S404: Calculate the confidence level of the gesture symbol sequence corresponding to each window image data.
[0229] Specifically, the gesture symbol sequence corresponding to each window image data may be calculated by utilizing the gesture category probability distribution corresponding to each window image data.
[0230] S405. The gesture symbol sequence corresponding to each window image data having a confidence level not less than a confidence threshold is used as the recognition result of each window image data, and the gesture symbol sequence corresponding to each window image data having a confidence level less than the confidence threshold is smoothed with the gesture symbol sequence of the adjacent window image data.
[0231] It should be noted that for gesture symbol sequences corresponding to window image data with a confidence level not less than the confidence threshold, the recognition result is accurate and can be directly used as the recognition result. For window image data with a confidence level less than the confidence threshold, the recognition result is unreliable and is therefore fused into the results of other adjacent window image data.
[0232] S406: Merge the recognition results of the overlapping parts of the image data of each window to obtain a current gesture recognition result.
[0233] Optionally, the overlapping parts may be merged by majority voting or weighted averaging.
[0234] S407: Use the N-gram language model to correct the current gesture recognition result.
[0235] It should be noted that, considering that there may be some recognition errors of similar gestures in the recognition results, it is possible to perform semantic analysis and thus correct them. Therefore, in an embodiment of the present application, the current gesture recognition result is semantically analyzed using an N-gram language model, and the current gesture recognition result is corrected based on the analysis results. For example, it is detected that the semantics of the sequence "HELLO FOOD FINISH Q" has problems, and its probability in the sign language corpus is low. This is specifically caused by the frequency of the "FOOD FINISH" bigram in the corpus. The system finds that "HELLO EATFINISH Q" is a high-frequency sign language expression sequence, and the gestures of "EAT" and "FOOD" have similar movement characteristics, so "FOOD" is corrected to "EAT".
[0236] S103 : Based on the mapping relationship between gesture symbols and vocabulary, the current gesture recognition result is mapped to obtain a vocabulary sequence.
[0237] Specifically, a mapping table between gestures and vocabulary is pre-established based on the actual meaning of the gesture symbols. Each recognized gesture (independent gesture, static gesture, or dynamic gesture sequence) is mapped to a pre-defined vocabulary. Furthermore, for systems supporting multiple sign languages, a separate mapping table can be established for each sign language. The meaning of each gesture symbol sequence in the current gesture recognition result can be mapped using the corresponding mapping table to obtain the corresponding vocabulary sequence.
[0238] S104: Composing the vocabulary sequence into the current complete recognition text according to the preset grammatical rules.
[0239] It should be noted that the mapped vocabulary sequence is only the vocabulary corresponding to each hand sign, and is a sequence composed according to the order of the hand signs, so it conforms to the grammatical structure of sign language. However, the grammatical structure of sign language is significantly different from that of spoken language. For example, sign language may have different word order, pauses, repetitions, and other features. Therefore, it is necessary to design a set of grammatical rules based on the unique grammatical structure of sign language, combining individual words into complete sentences according to grammatical conventions, so that they can symbolize the results of spoken language.
[0240] S105. Correct the current complete recognition text by combining the current complete recognition text, the current scene understanding information, and the current action understanding information through a multimodal large language model.
[0241] It's important to note that while the previous step only adjusted the word order, some words may be omitted in sign language to facilitate faster expression, such as function words. Therefore, to accurately convey the intended meaning, these omitted words need to be supplemented. Furthermore, sign language may also repeat gestures to convey specific meanings, so words with repetitive meanings also need to be removed.
[0242] Therefore, in an embodiment of the present application, a multimodal large language model, i.e., an MMLM large model, is used to analyze semantics in combination with the current complete recognition text, the current scene understanding information, and the current action understanding information. The omitted words are supplemented based on the semantics and repeated information is removed, thereby achieving correction of the current complete recognition text.
[0243] The current scene understanding information includes environmental information and emotional state information obtained by analyzing the user's current scene image using the first visual language model. The current action understanding information includes gesture information and semantic information obtained by analyzing the sequence of gesture images using the second visual language model.
[0244] Specific as Figure 5 As shown in the figure, the MMLM model consists of multiple subsystems working together to improve the accuracy and naturalness of sign language translation text. Among them, a VLM module, namely the first visual language model, takes the current scene image as input to analyze information such as the environment and emotional state of the deaf-mute person. Since the scene does not change drastically, its operating frequency can be 2fps. Similarly, since the scene usually does not change drastically, this module does not process each frame of the image in real time. Instead, its parsing results are stored in a buffer and updated at intervals of 500ms to ensure the real-time and stability of the information, which is then input into the LLM model.
[0245] Another VLM module, the second visual language model, takes a sequence of gesture images (i.e., the current sign language gesture) as input and leverages the model's emergent capabilities to efficiently understand human motion. This module processes at a frequency of 10 frames per second and not only recognizes the gesture itself but also provides additional semantic information, such as the emotion conveyed by the gesture and emphasized words.
[0246] Finally, the LLM model, as the core processing unit, integrates the outputs of the two VLM modules and the current complete recognized text, combines the visual scene to infer the potential context, enhances the overall semantic consistency, and infers the tone, emotion, and emphasis of the sentence based on gesture action information to correct possible errors and ambiguities.
[0247] S106: Input the corrected current complete recognition text, the current scene image, and the historical communication information into a third visual language model, and analyze the current response text through the third visual language model.
[0248] In order to quickly reply to users and make communication more convenient, so that users can more conveniently handle business matters, etc., in the embodiment of the present application, a VLM multimodal large model is used, combined with the current complete recognition text, the current scene picture and the historical communication information, to analyze the current complete recognition text after the answer is corrected based on the prompt words.
[0249] So the specific Figure 6 As shown, the corrected current complete recognition text, the current scene picture and the historical communication information are input into the third visual language model, and the model is prompted to understand the current scene picture, analyze the historical communication information, and reply to the corrected current complete recognition text, thereby outputting the natural word order text in response to the user.
[0250] S107: Convert the current response text into the current sign language sequence.
[0251] The current sign language sequence is a trajectory sequence of each joint point of the sign language action.
[0252] It should be noted that in order to facilitate the user's response through sign language, the current response text is converted into a trajectory sequence of the corresponding sign language action joint points, so that the robot can be controlled to respond in sign language according to this trajectory sequence. Of course, the current response text can also be directly fed back to the user at the same time.
[0253] Specifically, the current response text is converted into a sequence of corresponding sign language symbols, that is, the gestures corresponding to each word are determined, and then the current sign language sequence is generated according to the trajectory of the nodes corresponding to each sign language sequence number.
[0254] Optionally, in another embodiment of the present application, a specific implementation of step S107 is as follows: Figure 7 As shown, the following steps are included:
[0255] S701: Segment the current response text and perform part-of-speech tagging on the segmented words.
[0256] It should be noted that since the grammatical expression of sign language is different from that of spoken language, accurate conversion requires a thorough analysis of relevant information such as the grammar of the current response text.
[0257] Optionally, the natural language processing tool Jieba may be used to segment the input text and assign part-of-speech tags.
[0258] S702: Use dependency syntax analysis to construct a grammatical structure tree of the current response text, and extract keywords from the current response text.
[0259] In order to understand the relationship between the various words in the current response text and facilitate subsequent conversion, a dependency parsing method is used to construct a grammatical structure of the current response text. During conversion, the focus is on converting key information, so the keywords in the current response text are extracted.
[0260] S703. Based on the grammatical rules of sign language, the current response text is adjusted in combination with the segmentation and its annotation, the grammatical structure tree, and the keywords in the current response text to obtain a sign language response text that conforms to the grammatical rules of sign language.
[0261] Specifically, according to the grammatical rules of sign language, based on the analyzed segmentation words and their annotations, the grammatical structure, and the keywords in the current response text, the vocabulary in the current response text is adjusted to obtain a sign language response text that conforms to the grammatical rules of sign language. For example, "I go to the store to buy apples" is adjusted to "I, store, buy, apples."
[0262] S704: Convert the sign language response text based on the mapping relationship between the gesture symbols and the vocabulary to obtain various response gesture symbols.
[0263] Specifically, each word in the sign language response text can be converted into a corresponding gesture symbol based on the mapping relationship of the region.
[0264] S705: Assign a timestamp to each response gesture symbol to obtain a response gesture symbol sequence.
[0265] In order to facilitate the production of coherent sign language movements, a time stamp is assigned to each response gesture symbol to establish a time sequence.
[0266] S706: Input the response gesture symbol sequence, the historical trajectory of the joint points, and the visual context into a pre-trained sign language sequence generation model to generate a sign language sequence corresponding to the response gesture symbol sequence.
[0267] Among them, visual context can be obtained by analyzing the collected scene depth map, which can provide spatial reference coordinates, enabling the model to perform accurate indicative gestures and make sign language movements naturally adapt to the actual environment, especially playing a key role in sign language expression that needs to express spatial relationships or interact with the environment.
[0268] Optionally, a deep learning model may be used to generate a sequence of natural sign language movements.
[0269] Specifically, the model uses a pre-trained text encoder to convert the response gesture symbol sequence into a time-series vector. The 3D joint point history trajectory is normalized. A convolutional network is used to extract visual context features. Finally, the resulting multimodal features are evaluated into a unified conditional vector, which serves as the conditional input for the diffusion model. The diffusion model uses multi-step iterative denoising to generate semantically consistent 3D key motion trajectories. Physical constraints (such as joint angle limits) are added to prevent unreasonable movements, resulting in a sign language sequence corresponding to the response gesture symbol sequence.
[0270] S707: Perform time sequence alignment and smoothing optimization on the sign language sequence corresponding to the response gesture symbol sequence to obtain a current sign language sequence.
[0271] In order to ensure that the start and end times of each gesture symbol in the response gesture symbol sequence are consistent with the motion capture device, alignment and smoothing processing are required.
[0272] Specifically, based on the coarse-grained trajectories generated by the diffusion model, a dynamic time warping (DTW) algorithm is used to align the temporal boundaries of gesture symbol units and action segments. Transitions between consecutive gesture symbol units can be smoothed using Bezier curve interpolation or LSTM prediction to ensure gesture continuity and avoid abrupt transitions.
[0273] Optionally, in another embodiment of the present application, it further includes voice output response information. Figure 8 As shown, an embodiment of the present application provides a method for outputting language response information, including:
[0274] S801: Perform language conversion on the current response text to obtain the current response voice.
[0275] Specifically, the current response text may be converted into voice data through a TTS (Text To Speech) algorithm.
[0276] S802: Align the current response speech with morphemes, and convert the aligned morphemes into lip data using a speech synthesis markup language.
[0277] To facilitate lip reading recognition for the deaf and mute, the voice data is aligned with morphemes during the conversion of the TTS algorithm into speech data. Furthermore, morphemes are converted into lip shape data using the Speech Synthesis Markup Language (SSML). This synchronizes the lip shape changes when the voice data is played through the speakers, achieving a lip reading effect.
[0278] S803: Play the current response voice and lip shape data.
[0279] S108: driving the robot to simulate sign language movements according to the current sign language word order.
[0280] Specifically, each joint in the robot is controlled to move according to the current sign language sequence, thereby simulating the corresponding sign language.
[0281] Optionally, in another embodiment of the present application, a specific implementation of step S108 is as follows: Figure 9 Shown, including:
[0282] S901. Based on the current sign language sequence, calculate the motion angles of each joint of the arm and fingers through an inverse kinematics algorithm to obtain sign language joint sequence data.
[0283] Among them, the movement angle of each joint is not greater than the corresponding maximum value.
[0284] Specifically, the motion angles of each joint of the arm, fingers, etc. are calculated through the Inverse Kinematics (IK) algorithm, so that the hand, arm, etc. can accurately reach the target position, while limiting the joint angles. For example, the wrist rotation angle should not exceed 180°.
[0285] S902: Based on the sign language joint sequence data and the degrees of freedom of each joint of the robot's hand, a smooth joint trajectory is generated through multiple polynomial differences, and each joint of the robot's hand is controlled to move according to the smooth joint trajectory.
[0286] Optionally, a 23-DOF (one-hand degree of freedom) simulated human hand can be used to more closely resemble the joint structure of the human body, enabling more natural and delicate finger movements and more convenient adaptation to dexterous hands from different manufacturers.
[0287] The design of the degree of freedom of each part can be specifically as follows:
[0288] Thumb: has 4 degrees of freedom.
[0289] CMC (Carpometacarpal Joint): Bidirectional motion.
[0290] MCP (Metacarpophalangeal joint): Unidirectional flexion and extension.
[0291] IP (interphalangeal joint): unidirectional flexion and extension.
[0292] Four fingers: Each finger has 4 degrees of freedom.
[0293] MCP (Metacarpophalangeal Joint): Bidirectional movement.
[0294] PIP (Proximal Interphalangeal Joint): Unidirectional flexion and extension.
[0295] DIP (distal interphalangeal joint): unidirectional flexion and extension.
[0296] Wrist: has 3 degrees of freedom (rotation, pitch, yaw).
[0297] Then, multiple polynomial interpolation is used to generate smooth joint trajectories to ensure the continuity of position, velocity and acceleration during the movement, thereby avoiding sudden changes in velocity and motion state.
[0298] Optionally, in order to facilitate the determination of data flow and the coordination of various modules so that the robot can perform sign language communication in real time and stably, another embodiment of the present application provides a robot control method, that is, an end-to-end robot execution state machine, such as Figure 10 Shown, including:
[0299] When a text containing a wake-up word is detected, the robot is switched from the sleep state to the listening state to monitor the generated complete recognition text in real time.
[0300] Optionally, the detected text containing the wake-up word may be sign language input, text input, or language input, thereby waking up the robot and switching the robot from a sleeping state (SLEEP state) to a listening state (LISTENING state).
[0301] Next, it is determined whether an input is received within a preset time in the monitoring state.
[0302] In the listening state, the robot monitors the sign language text generated by the model in real time. If no input is received within a preset time, such as 60 seconds, the robot returns to sleep (SLEEP). If the current complete recognition text is generated within the preset time, it is forwarded to the multimodal large language model and switches to the waiting state. That is, when sign language text is received, it is immediately forwarded to the LLM module and switches to the waiting state (WAIT_LLM).
[0303] In the WAIT_LLM state, the system waits for the LLM module to process the forwarded sign language text and return a response. Upon receiving the processing results from the multimodal large language model, the system switches to the Playing state (TTSLING state) to initiate the text-to-sign language conversion and begin playing the corresponding sign language gestures. When the robot starts playing the sign language gestures, the system switches to the Waiting for Completion state (WAIT_TTSLING_COMPLETE state). When the sign language gesture playback completes, the system switches to the Listening state (LISTENING state) to prepare for the next interaction.
[0304] Embodiments of the present application provide a robot sign language communication method that collects gesture images of a current user and extracts features from each gesture image. The gesture image features encompass features from multiple locations, focusing not only on the hands, to ensure accurate recognition. The features of each gesture image are then input into a pre-trained gesture recognition model, which then generates a current gesture recognition result. The current gesture recognition result includes at least a sequence of recognized gesture symbols. Based on a mapping relationship between gesture symbols and vocabulary, the current gesture recognition result is mapped to a vocabulary sequence. This vocabulary sequence is then combined according to preset grammatical rules to form a current complete recognition text, ensuring that the word order of the text conforms to spoken language grammar. Furthermore, the current complete recognition text is corrected using a multimodal large language model, combining the current complete recognition text, current scene understanding information, and current action understanding information. The current scene understanding information includes environmental information and emotional state information obtained by performing scene analysis on the user's current scene image using a first visual language model. The current action understanding information includes gesture information and semantic information obtained by performing action analysis on the sequence of gesture images using a second visual language model. Thus, the missing and repeated content caused by the differences between the grammatical expressions of sign language and spoken language are processed to correct and optimize the generated text, ensure the accuracy and naturalness of the text, and accurately express the user's meaning. Then, the corrected current complete recognition text, the current scene image, and the historical communication information are input into the third visual language model, and the current response text is analyzed by the third visual language model, and the current response text is converted into the current sign language sequence. Among them, the current sign language sequence is the trajectory sequence of each joint point of the sign language action. Finally, according to the current sign language word order, the robot is driven to simulate the sign language action, so that the content of the reply to the user can be automatically obtained, and the user can be replied through sign language, thereby realizing sign language communication with the user, making it easier to communicate with the user and thus easier to achieve various matters.
[0305] Another embodiment of the present application provides a robot sign language communication device, such as Figure 11 Shown, including:
[0306] The image acquisition unit 1101 is used to acquire each frame of gesture image of the current user and extract features of each frame of gesture image, wherein the features of the gesture image include features of multiple parts.
[0307] The gesture recognition unit 1102 is configured to input the features of each frame of gesture image into a pre-trained gesture recognition model, and use the gesture recognition model to identify a current gesture recognition result, wherein the current gesture recognition result at least includes a sequence of each recognized gesture symbol.
[0308] The first mapping unit 1103 is configured to map the current gesture recognition result based on the mapping relationship between gesture symbols and vocabulary to obtain a vocabulary sequence.
[0309] The word order adjustment unit 1104 is used to compose the current complete recognition text according to the preset grammatical rules of the word sequence.
[0310] The text optimization unit 1105 is configured to modify the current complete recognized text using a multimodal large language model in combination with the current complete recognized text, current scene understanding information, and current action understanding information. The current scene understanding information includes environmental information and emotional state information obtained by performing scene analysis on the user's current scene image using a first visual language model. The current action understanding information includes gesture information and semantic information obtained by performing action analysis on a sequence of gesture images using a second visual language model.
[0311] The reply analysis unit 1106 is configured to input the corrected current complete recognition text, the current scene image, and the historical communication information into the third visual language model, and analyze the current response text using the third visual language model.
[0312] The trajectory conversion unit 1107 is used to convert the current response text into a current sign language sequence, wherein the current sign language sequence is a trajectory sequence of each joint point of the sign language action.
[0313] The simulation unit 1108 is used to drive the robot to simulate sign language movements according to the current sign language word order.
[0314] Optionally, in the robot sign language communication device provided in another embodiment of the present application, the image acquisition unit includes:
[0315] The video acquisition unit is used to acquire the gesture video stream of the current user.
[0316] The image extraction unit is used to extract basic frames from the gesture video stream at fixed intervals to obtain multiple frames of gesture images.
[0317] The region detection unit is used to locate the upper half of the human body in each frame of the gesture image through a detection algorithm.
[0318] The first extraction unit is configured to extract information of key points of the upper half of the human body in each frame of the gesture image, wherein the key points include at least a plurality of key points of the hands, arms, face, and upper body.
[0319] The image supplement unit is used to supplement the gesture image between each two frames of gesture image based on the distance between the key points of the hand in each two adjacent frames of gesture image.
[0320] The second extraction unit is used to extract information of each key point in the upper half of the human body in each frame of the supplemented gesture image.
[0321] The first normalization processing unit is used to normalize the key point information of each part of each frame of the gesture image based on the reference of each part, and use the normalized key point information of each part of the gesture image as the feature of the gesture image.
[0322] Optionally, in another embodiment of the present application, the robot sign language communication device further includes:
[0323] The model building unit is used to build an initial gesture recognition model.
[0324] The sample data extraction unit is used to obtain a sign language data set and extract key point information of each frame of sign language image in the sign language data set.
[0325] The data enhancement unit is used to enhance the information of the key points of each frame of sign language image by transforming the key points of each frame of sign language image.
[0326] The second normalization processing unit is used to normalize the information of the key points of each frame of sign language image after data enhancement to obtain training sample data.
[0327] The iteration unit is used to fine-tune the pre-trained gesture recognition model using training sample data and perform iterative training until the loss converges.
[0328] Optionally, in the robot sign language communication device provided in another embodiment of the present application, the gesture recognition unit includes:
[0329] The image extraction unit is used to extract images from a sequence of gesture images using a sliding window to obtain a plurality of window image data, wherein each window image data includes one or more consecutive frames of gesture images.
[0330] The probability analysis unit is used to input the features of each window image data into a pre-trained gesture recognition model, and identify the gesture category probability distribution corresponding to each window image data through the gesture recognition model.
[0331] The decoding unit is used to perform time-series decoding on the probability distribution of gesture categories corresponding to the image data of each window to obtain a gesture symbol sequence corresponding to the image data of each window.
[0332] The confidence calculation unit is used to calculate the confidence of the gesture symbol sequence corresponding to each window image data.
[0333] The processing unit is used to use the gesture symbol sequence corresponding to each window image data with a confidence level not less than a confidence threshold as the recognition result of each window image data, and to smooth the gesture symbol sequence of each window image data with a confidence level less than the confidence threshold with the gesture symbol sequence of the adjacent window image data.
[0334] The merging unit is used to merge the recognition results of the overlapping parts of the image data of each window to obtain the current gesture recognition result.
[0335] The result correction unit is used to correct the current gesture recognition result using the N-gram language model.
[0336] Optionally, in the robot sign language communication device provided in another embodiment of the present application, the trajectory conversion unit includes:
[0337] The word segmentation processing unit is used to segment the current response text and perform part-of-speech tagging on the segmented words.
[0338] The syntactic analysis unit is used to construct a grammatical structure tree of the current response text using dependency syntactic analysis, and to extract keywords from the current response text.
[0339] The text adjustment unit is used to adjust the current response text based on the grammatical rules of sign language, combined with word segmentation and its annotation, grammatical structure tree and keywords in the current response text, to obtain a sign language response text that conforms to the grammatical rules of sign language.
[0340] The second mapping unit is used to convert the sign language response text based on the mapping relationship between the gesture symbols and the vocabulary to obtain various response gesture symbols.
[0341] The time allocation unit is used to allocate a time stamp to each response gesture symbol to obtain a response gesture symbol sequence.
[0342] The sign language sequence generation unit is used to input the response gesture symbol sequence, the historical trajectory of the joint points and the visual context into a pre-trained sign language sequence generation model to generate a sign language sequence corresponding to the response gesture symbol sequence.
[0343] The sequence smoothing unit is used to perform time sequence alignment and smoothing optimization on the sign language sequence corresponding to the response gesture symbol sequence to obtain the current sign language sequence.
[0344] Optionally, in the robot sign language communication device provided in another embodiment of the present application, the simulation unit includes:
[0345] The angle analysis unit is used to calculate the motion angle of each joint of the arm and fingers based on the current sign language sequence using an inverse kinematics algorithm to obtain sign language joint sequence data. The motion angle of each joint is not greater than the corresponding maximum value.
[0346] The estimation smoothing unit is used to generate a smooth joint trajectory through multiple polynomial differences based on the sign language joint sequence data and the degrees of freedom of each joint of the robot's hand, and control the various joints of the robot's hand to move according to the smooth joint trajectory.
[0347] Optionally, in another embodiment of the present application, the robot sign language communication device further includes:
[0348] The speech conversion unit is used to convert the language of the current response text to obtain the current response speech.
[0349] The lip-shape conversion unit is used to align the current response speech with morphemes and convert the aligned morphemes into lip-shape data through a speech synthesis markup language.
[0350] The language playing unit is used to play the current response voice and lip shape data.
[0351] Optionally, in another embodiment of the present application, the robot sign language communication device further includes:
[0352] The first switching unit is used to switch the robot from a sleeping state to a listening state when a text containing a wake-up word is currently detected, so as to monitor the current complete recognition text generated in real time.
[0353] The state return unit is used to return the robot to the sleep state when no input is received within a preset time.
[0354] The second switching unit is used to forward the current complete recognition text to the multimodal large language model and switch to a waiting state when the current complete recognition text is monitored to be generated within a preset time.
[0355] The third switching unit is used to switch to the play state when waiting for the processing result of the multimodal large language model to start the text-to-sign language conversion action.
[0356] The fourth switching unit is used to switch to the waiting end state when the robot starts to play the sign language action.
[0357] The fifth switching unit is used to switch to the monitoring state when the sign language action playback is completed.
[0358] It should be noted that the specific working process of each unit provided in the above embodiments of the present application can refer to the implementation process of the corresponding steps in the above method embodiments, and will not be repeated here.
[0359] Another embodiment of the present application provides a computer storage medium for storing a computer program. When the computer program is executed by a processor, it is used to implement the robot sign language communication method provided in any of the above embodiments.
[0360] Computer storage media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.
[0361] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0362] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A robot sign language communication method, characterized in that: include: Collecting each frame of gesture images of the current user and extracting features of each frame of the gesture image; wherein the features of the gesture image include features of multiple parts; Inputting the features of the gesture image of each frame into a pre-trained gesture recognition model, and identifying a current gesture recognition result through the gesture recognition model; wherein the current gesture recognition result at least includes a sequence of each recognized gesture symbol; Based on the mapping relationship between gesture symbols and vocabulary, mapping the current gesture recognition result to obtain a vocabulary sequence; Composing the vocabulary sequence into the current complete recognition text according to the preset grammatical rules; Correcting the current complete recognition text by combining the current complete recognition text, current scene understanding information, and current action understanding information with a multimodal large language model; wherein the current scene understanding information includes environmental information and emotional state information obtained by performing scene analysis on the current scene image of the user using a first visual language model; and the current action understanding information includes gesture information and semantic information obtained by performing action analysis on a sequence of gesture images of each frame using a second visual language model; Inputting the corrected current complete recognition text, the current scene image, and the historical communication information into a third visual language model, and analyzing the current response text through the third visual language model; Converting the current response text into a current sign language sequence; wherein the current sign language sequence is a trajectory sequence of each joint point of the sign language action; The robot is driven to simulate sign language movements according to the current sign language word sequence.
2. The method according to claim 1, characterized in that The collecting of gesture images of each frame of the current user and extracting features of the gesture images of each frame includes: Collecting a video stream of gestures of the current user; Extracting basic frames from the gesture video stream at fixed intervals to obtain multiple frames of gesture images; Locating the upper half of the human body in each frame of the gesture image using a detection algorithm; Extracting information of key points of the upper half of the human body in each frame of the gesture image; wherein the key points include at least a plurality of key points of the hands, arms, face, and upper body; Supplementing a gesture image between each two frames of the gesture image based on the distance between the key points of the hand in each two adjacent frames of the gesture image; Extracting information of key points of the upper half of the human body in each frame of the supplemented gesture image; The key point information of each part of the gesture image of each frame is normalized based on the reference of each part, and the normalized key point information of each part of the gesture image is used as the feature of the gesture image.
3. The method according to claim 1, characterized in that The training method of the gesture recognition model includes: Constructing the initial gesture recognition model; Acquire a sign language dataset, and extract key point information of each frame of sign language image in the sign language dataset; By transforming the key points of the sign language image of each frame, data enhancement is performed on the information of the key points of the sign language image of each frame; Normalizing the key point information of each frame of the sign language image after data enhancement to obtain training sample data; The pre-trained gesture recognition model is fine-tuned and iteratively trained using the training sample data until the loss converges.
4. The method according to claim 1, wherein Inputting the features of the gesture images of each frame into a pre-trained gesture recognition model, and identifying the current gesture recognition result through the gesture recognition model, includes: Performing image extraction on the sequence of the gesture images of each frame through a sliding window to obtain a plurality of window image data; wherein each of the window image data includes one or more consecutive frames of the gesture image; Inputting the features of each of the window image data into a pre-trained gesture recognition model, and identifying the gesture category probability distribution corresponding to each of the window image data through the gesture recognition model; Performing time-series decoding on the probability distribution of gesture categories corresponding to each of the window image data to obtain a gesture symbol sequence corresponding to each of the window image data; Calculating the confidence of the gesture symbol sequence corresponding to each of the window image data; taking the gesture symbol sequence corresponding to each window image data having a confidence level not less than a confidence threshold as the recognition result of each window image data, and smoothing the gesture symbol sequence corresponding to each window image data having a confidence level less than the confidence threshold with the gesture symbol sequence of the adjacent window image data; Merging the recognition results of the overlapping parts of the window image data to obtain a current gesture recognition result; The current gesture recognition result is corrected using an N-gram language model.
5. The method according to claim 1, characterized in that The converting the current response text into a current sign language sequence includes: Segmenting the current response text and performing part-of-speech tagging on the segmented words; Using dependency syntax analysis to construct a grammatical structure tree of the current response text, and extracting keywords from the current response text; Based on the grammatical rules of sign language, the current response text is adjusted in combination with the segmentation and its annotation, the grammatical structure tree, and keywords in the current response text to obtain a sign language response text that conforms to the grammatical rules of sign language; Converting the sign language response text based on a mapping relationship between gesture symbols and vocabulary to obtain various response gesture symbols; Assigning a timestamp to each of the response gesture symbols to obtain a response gesture symbol sequence; Inputting the response gesture symbol sequence, the historical trajectory of the joint points, and the visual context into a pre-trained sign language sequence generation model to generate a sign language sequence corresponding to the response gesture symbol sequence; Performing time sequence alignment and smoothing optimization on the sign language sequence corresponding to the response gesture symbol sequence to obtain the current sign language sequence.
6. The method according to claim 1, characterized in that Driving the robot to simulate sign language movements according to the current sign language sequence includes: Based on the current sign language sequence, calculating the motion angles of each joint of the arm and fingers using an inverse kinematics algorithm to obtain sign language joint sequence data; wherein the motion angle of each joint is no greater than a corresponding maximum value; Based on the sign language joint sequence data and the degrees of freedom of each joint of the robot's hand, a smooth joint trajectory is generated through multiple polynomial differences, and each joint of the robot's hand is controlled to move according to the smooth joint trajectory.
7. The method according to claim 1, characterized in that Also includes: Performing language conversion on the current response text to obtain a current response voice; Aligning the current response speech with morphemes, and converting the aligned morphemes into lip data using a speech synthesis markup language; Play the current response voice and the lip shape data.
8. The method according to claim 1, characterized in that Also includes: When a text containing a wake-up word is currently detected, the robot is switched from a sleep state to a listening state to monitor the generated current complete recognition text in real time; When no input is received within a preset time, returning the robot to the sleep state; When the current complete recognition text is generated within the preset time, the current complete recognition text is forwarded to the multimodal large language model and switched to a waiting state; When the processing result of the multimodal large language model is received, the state is switched to the play state to start the text-to-sign language conversion action; When controlling the robot to start playing the sign language action, switching to the waiting end state; When the sign language action playback is finished, switch to the monitoring state.
9. A robot sign language communication device, characterized in that: include: An image acquisition unit, configured to acquire each frame of gesture images of the current user and extract features of each frame of the gesture image; wherein the features of the gesture image include features of multiple parts; A gesture recognition unit, configured to input the features of the gesture image of each frame into a pre-trained gesture recognition model, and identify a current gesture recognition result through the gesture recognition model; wherein the current gesture recognition result at least includes a sequence of each recognized gesture symbol; a first mapping unit, configured to map the current gesture recognition result based on a mapping relationship between gesture symbols and vocabulary to obtain a vocabulary sequence; A word order adjustment unit, configured to compose the current complete recognition text according to preset grammatical rules according to the word sequence; a text optimization unit, configured to modify the current complete recognized text by combining the current complete recognized text, current scene understanding information, and current action understanding information using a multimodal large language model; wherein the current scene understanding information includes environmental information and emotional state information obtained by performing scene analysis on the user's current scene image using a first visual language model; and the current action understanding information includes gesture information and semantic information obtained by performing action analysis on a sequence of gesture images in each frame using a second visual language model; a reply analysis unit, configured to input the corrected current complete recognized text, the current scene image, and the historical communication information into a third visual language model, and analyze the current response text using the third visual language model; A trajectory conversion unit, configured to convert the current response text into a current sign language sequence; wherein the current sign language sequence is a trajectory sequence of each joint point of the sign language action; The simulation unit is used to drive the robot to simulate sign language movements according to the current sign language word order.
10. A computer storage medium, characterized in that Used to store a computer program, which, when executed by a processor, is used to implement the robot sign language communication method according to any one of claims 1 to 8.
Citation Information
Cited By
Sign language translation method and device, computer equipment and storage medium
CN120877390A
Sign language translation methods, devices, computer equipment and storage media
CN120877390B
Video sign language question and answer method, system and device and storage medium
CN121033944A
Sign language data processing method based on humanoid robot and related device
CN121893332A
Sign language data processing method based on humanoid robot and related device
CN121893332B
