Chinese sign language translation method and system based on large language model

Abstract

The invention provides a Chinese sign language translation method and system based on a large language model, and belongs to the technical field of sign language translation.The method comprises the steps that in response to the received first information, gesture recognition is conducted on the first information, and first text information is obtained; the first text information is translated, first translation information is generated and output, the first information comprises a gesture image sent by the second device, and the first translation information comprises text data corresponding to the first information; in response to the received second information, performing sign language translation on the second information to obtain second text information; the second text information is translated, second translation information is generated and sent to second equipment, the second information comprises to-be-recognized text data and / or voice data, and the second translation information comprises a gesture image corresponding to the second information. According to the Chinese sign language translation method and system based on the large language model, the accuracy and the real-time processing capability of gesture recognition and translation in user communication can be improved.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of sign language translation, and more specifically, to a Chinese sign language translation method and system based on a large language model. Background Art

[0002] In today's society, communication barriers between the hearing-impaired and hearing-normal people pose significant challenges to all aspects of life, including education, employment, and social interaction. Traditional sign language translation is not always available, and existing automated solutions often lack accuracy and real-time processing capabilities. Therefore, a powerful, efficient, and accurate real-time sign language translation method is needed that can accurately interpret Chinese sign language gestures and convert them into text and speech, as well as translate speech or written text into sign language gestures to enhance communication and inclusion. Summary of the invention

[0003] The purpose of the present disclosure is to provide a Chinese sign language translation method and system based on a large language model to improve the accuracy and real-time processing capability of gesture recognition and translation in user communication.

[0004] According to a first aspect of an embodiment of the present disclosure, a Chinese sign language translation method based on a large language model is provided, which is applied to a first device and includes: In response to receiving the first information, performing gesture recognition on the first information to obtain first text information; translating the first text information to generate and output first translation information, the first information includes the gesture image sent by the second device, the second device is a device connected to the first device, and the first translation information includes text data corresponding to the first information; In response to receiving the second information, the second information is translated into sign language to obtain a second text message; the second text message is translated to generate and send second translation information to the second device, the second information includes text data and / or voice data to be recognized, and the second translation information includes a gesture image corresponding to the second information.

[0005] A second aspect of the disclosed embodiment provides a Chinese sign language translation method based on a large language model, which is applied to a second device and includes: In response to capturing the gesture image, sending the gesture image to a first device; the first device is a device connected to the second device; In response to receiving the second translation information, the second translation information is displayed; the second translation information includes a gesture image corresponding to the second information; and the second information includes text data and / or voice data to be recognized.

[0006] According to a third aspect of the embodiments of the present disclosure, a Chinese sign language translation device based on a large language model is provided, which is applied to a first device and includes: A first translation module is used for, in response to receiving the first information, performing gesture recognition on the first information to obtain first text information; translating the first text information to generate and output first translation information, the first information includes a gesture image sent by a second device, the second device is a device connected to the first device, and the first translation information includes text data corresponding to the first information; The second translation module is used to, in response to receiving the second information, perform sign language translation on the second information to obtain a second text message; translate the second text message, generate and send second translation information to the second device, the second information includes text data and / or voice data to be recognized, and the second translation information includes a gesture image corresponding to the second information.

[0007] According to a fourth aspect of the embodiments of the present disclosure, a Chinese sign language translation system based on a large language model is provided, comprising an image capture module, a data processing module, an output module and a user interface.

[0008] The image capturing module is configured in the second device, and the data processing module, the output module and the user interface are configured in the first device.

[0009] The first device is connected to the second device.

[0010] According to a fifth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned Chinese sign language translation method based on a large language model applied to a first device or the Chinese sign language translation method based on a large language model applied to a second device are implemented.

[0011] The beneficial effect of a Chinese sign language translation method and system based on a large language model provided by the disclosed embodiment is that the disclosed embodiment realizes two-way translation between gestures and text and voice, greatly improving communication efficiency. Users can express through gestures and instantly obtain accurate text translation, which is convenient for non-sign language speakers to understand. At the same time, text or voice information can also be quickly converted into gesture images and displayed on the second device (wearable device), which is convenient for sign language speakers to receive in real time. This two-way translation capability is not only suitable for communication between sign language speakers and non-sign language speakers, but also promotes cross-language and cross-cultural communication, and has broad application prospects and social value. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 A flowchart of a Chinese sign language translation method based on a large language model provided in one embodiment of the present disclosure; Figure 2 A system architecture diagram of a Chinese sign language translation method based on a large language model provided in one embodiment of the present disclosure; Figure 3 A schematic diagram of a gesture recognition process provided by an embodiment of the present disclosure; Figure 4 A flowchart of another Chinese sign language translation method based on a large language model provided by an embodiment of the present disclosure; Figure 5 A flowchart of another Chinese sign language translation method based on a large language model provided in an embodiment of the present disclosure; Figure 6 A schematic diagram of a translation output process provided by an embodiment of the present disclosure; Figure 7 A flowchart of contextual semantic understanding and translation provided by an embodiment of the present disclosure; Figure 8 A schematic diagram of a signaling interaction process between devices provided in an embodiment of the present disclosure; Fig. 9 A structural block diagram of a Chinese sign language translation device based on a large language model provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0014] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0015] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0016] Please refer to Figure 1 , Figure 1 A flowchart of a Chinese sign language translation method based on a large language model is provided for an embodiment of the present disclosure. The method is applied to a first device and may include S101 to S102.

[0017] S101: In response to receiving a first message, performing gesture recognition on the first message to obtain a first text message. The first text message is translated to generate and output first translation information, the first message includes a gesture image sent by a second device, the second device is a device connected to the first device, and the first translation information includes text data corresponding to the first message.

[0018] S102: In response to receiving the second information, the second information is sign-translated to obtain a second text message. The second text message is translated to generate and send second translation information to the second device, the second information includes text data and / or voice data to be recognized, and the second translation information includes a gesture image corresponding to the second information.

[0019] In this embodiment, when the first device receives the gesture image sent by the second device, it can perform gesture recognition on the gesture image and convert it into the corresponding first text information. Subsequently, the first text information can be translated to obtain text data corresponding to the corresponding gesture image. Finally, the text data can be output externally so that the corresponding personnel can understand the meaning of the gesture expression through the display screen of the first device.

[0020] In this embodiment, when the first device receives the text data or voice data to be recognized, it can perform sign language translation on the text data or voice data and convert it into a corresponding second text message. Subsequently, the second text message can be further translated and converted into a corresponding gesture image. Finally, the gesture image can be sent to the second device so that the second device can display the gesture image.

[0021] In this embodiment, the first device may include a mobile terminal, such as a smart phone, a smart bracelet, or a dedicated auxiliary device. The second device may include a wearable device, such as smart glasses or a smart head display device.

[0022] A communication connection may be established between the first device and the second device, and information may be exchanged. For example, the first device and the second device may be connected via Bluetooth, and information may be exchanged between the two.

[0023] The target user may be a user who has both the first device and the second device. For example, the target user may be a user who has both a mobile phone and smart glasses, and the translation and conversion between sign language gestures and voice information can be achieved through information interaction between the two devices.

[0024] For example, when the target user is a hearing-impaired user, the target user can receive text data or voice data from others through his or her smartphone. The smartphone can convert the text data or voice data into corresponding gesture images and send the gesture images to the smart glasses worn by the target user. The corresponding gesture images can be displayed on the lenses of the smart glasses, so that the hearing-impaired user can understand the other party's intentions in a timely and accurate manner.

[0025] When the target user is a user with normal hearing, the target user can capture the gesture image of the hearing-impaired user through the smart glasses he wears and send the gesture image to his smartphone. The smartphone can translate the corresponding gesture image into voice or text so that the target user can understand the intention expressed by the hearing-impaired user.

[0026] Specifically, if Figure 2 As shown, in this embodiment, the first device and the second device can form a translation system for sign language, text, and voice. The translation system may include an image capture module, a data processing module, an output module, and a user interface. The first device may be a smart phone of the target user, and the second device may be smart glasses worn by the target user.

[0027] In this embodiment, the image capture module is configured in the smart glasses, and may include a camera and a sign language capture module. The smart glasses may monitor the user's surroundings through the camera to collect relevant video or image data. At the same time, the captured video or image may be recognized through the sign language capture module to capture the user's sign language gesture image. When a specific gesture image is captured, the smart glasses may send the gesture image to the data processing module for further processing.

[0028] In this embodiment, the data processing module is configured in the first device and may include: gesture recognition based on MediaPipe, a large language model, a large language model agent, a knowledge graph, contextual semantic understanding, translation generation, and retrieval-augmented generation (RAG). Among them, the large language model agent can call the large language model to perform related sign language gesture translation operations.

[0029] For example, when the first device receives the gesture image from the smart glasses, it can use the above technical means to process the data and translate the gesture image into text. In addition, the text can also be converted into speech according to actual needs.

[0030] In this embodiment, the output module is configured in the first device and may include: text and voice output, a speaker and a display.

[0031] For example, the output module may include two output modes, one is voice playback through a speaker, and the other is displaying the translated information (text or image). After the output module outputs the first translated information, the user can receive the required information.

[0032] In this embodiment, the user interface may include: touch control, voice instructions, and gesture interaction instructions.

[0033] For example, the user interface is a touch screen display, and the user can input information through touch control. The user can also perform voice input, gesture selection, text input and other operations in the user interface to input data and receive translation. After the above interactive operations, the user interface receives the relevant interactive instructions input by the user, processes the user instructions and sends them to the data processing module for data processing.

[0034] For example, smart glasses are equipped with an embedded camera that can capture sign language gestures. Smart glasses can transmit the captured gesture image information to a smartphone. Subsequently, the smartphone can use MediaPipe for gesture recognition and combine the large language model agent, RAG and knowledge graph for contextual translation to obtain translated text information. Finally, the text information is converted into voice information and played.

[0035] The Large Language Model agent is a model that is fine-tuned for sign language translation and context understanding based on the pre-trained Transformer model. MediaPipe is a framework for building perception pipelines for real-time gesture recognition. RAG can retrieve relevant information from external sources to enhance translation and improve translation accuracy. Knowledge graphs can provide structured gesture context data to improve the accuracy and relevance of gesture translation.

[0036] Specifically, Figure 3 As shown, the gesture recognition process may include: using smart glasses (i.e., the second device) and the capture module to capture the user's sign language gestures in real time, and sending the captured data to the data processing module for processing, using MediaPipe to perform gesture recognition on the processed data, and inputting the recognized gesture sequence (i.e., the first text information) into the large language model agent, calling the large language model to generate a translation result (i.e., the first translation information). After the output module receives the translation result, it can provide text and voice output for the translation content so that the target user (the hearing-impaired person using the first device and the second device) can receive the translation information corresponding to the gesture. For example, the target user makes a series of sign language gestures, and the smart glasses capture and send a series of gesture images to the smart phone, perform the above data processing to generate translation information and perform voice playback, so that the hearing-normal person communicating with the target user can understand the information the target user wants to express.

[0037] From the above, it can be concluded that in this embodiment, the user uses smart glasses to capture sign language gestures and transmit them to the mobile phone. The mobile phone can recognize the gestures and translate them into text in real time, and then convert them into voice playback. This greatly enhances the conversion efficiency between sign language and voice, making communication between the hearing-impaired and others smoother. At the same time, the first device combines the large language model agent, RAG and knowledge graph for contextual translation, ensuring the accuracy and relevance of the translation and improving the user experience. The multiple input methods of the user interface provide users with more operation options, further improving the flexibility and ease of use of translation.

[0038] In this embodiment, gesture recognition is performed on the first information to obtain the first text information, including: A first action and a second action of the target user are determined based on the first information. The first action is a hand action, and the second action is a body action. Both the first device and the second device are devices used by the target user.

[0039] Based on the first action and the second action, a plurality of first target gesture information is matched from the plurality of gesture information as the first text information.

[0040] In this embodiment, body movements may include head movements and limb movements.

[0041] For example, Figure 4 As shown in the figure, a high-resolution camera can be used to capture detailed hand and body movements, and synchronously capture video frames and depth data. Specifically, a high-resolution camera and a depth sensor are used to simultaneously capture sign language gesture image information and environmental depth information. These two types of data are strictly aligned in time, that is, at the same time point, each frame of the image recorded by the camera corresponds to the depth data captured by the depth sensor. This synchronization ensures that during subsequent gesture recognition, motion analysis, and 3D reconstruction, image data and depth data can complement each other, providing more accurate and rich analysis results.

[0042] Optionally, the captured video data is pre-processed, including noise reduction and image stabilization, to ensure smooth gesture capture. For example, the background sound or image content of the video is subjected to noise reduction, background noise is removed, and the video frame is stabilized.

[0043] Next, MediaPipe tracks the finger movements in the image data, identifies hand movement features (such as finger positions), and extracts body posture features (such as head movement features and limb movement features) by identifying the user's body movements to understand the gesture background. The extracted hand movement features and body posture features are input into the convolutional neural network and the recurrent neural network, and the tracked movement data can be classified into specific sign language gestures (i.e., the first target gesture information). The multiple sign language gesture classification results obtained will be used as the first text information. The classification results mainly include gesture sequences, and the gesture sequence includes multiple sign language gestures, each of which has its own text information description. For example, different identification information is set for each gesture (such as numbers 001-999). The text information description corresponding to gesture No. 001 can be "Semantics: 'Hello'", "Left and right: 'right hand'; gesture 1: 'clenched fist'; gesture 2 'extend thumb'; gesture 3: 'bend thumb'; number of actions: '2';". In addition, the description information corresponding to each sign language gesture can be vectorized to facilitate data processing and similarity matching.

[0044] Specifically, the recognition and classification of gestures by convolutional neural networks and recurrent neural networks can include: extracting key points, such as feature information of fingertips and joints. At the same time, analyzing the temporal features of gesture sequences. Based on the above feature information obtained, a recurrent neural network or a long short-term memory network is used to model the gesture sequence.

[0045] In this embodiment, MediaPipe can process the captured video frames to recognize specific gestures. A large pre-trained and fine-tuned language model can understand the contextual meaning of gestures and provide accurate, complete, and coherent translations.

[0046] It can be concluded from the above that this embodiment realizes the generation of text information based on gesture recognition. A high-resolution camera is used to capture detailed hand and body movements, and noise reduction technology and MediaPipe are combined to achieve accurate gesture recognition. The gestures are classified into sign language gestures through convolutional neural networks and recurrent neural networks, and coherent text information is generated. This method improves the accuracy of gesture recognition and the coherence of text generation, providing users with a more convenient and efficient interactive experience.

[0047] In this embodiment, translating the first text information to generate and output first translation information includes: The first text information includes a plurality of first target gesture information.

[0048] The plurality of first target gesture information and the time dependency relationship corresponding to each of the first target gesture information are input into the large language model, and the first translation information is generated and output.

[0049] In this embodiment, outputting the first translation information includes displaying text information through a display screen (ie, a user interface) of the first device, or playing translated voice information through a speaker of the first device.

[0050] For example, Figure 4 As shown in the figure, after modeling the gesture sequence, the feature information corresponding to each first target gesture and the temporal features (i.e., temporal dependencies) of the gesture sequence (i.e., the first text information) are input into the transformer-based large language model to generate context embeddings, and the context embeddings are translated into text. Based on the above translated text, the translation content is adjusted in combination with a wider range of contextual data (e.g., relevant background information of the conversation) to improve the accuracy and relevance of the translation. The adjusted translated text is grammatically corrected to ensure the language correctness of the generated text. The knowledge graph and RAG system are then used to improve and refine the translated text, and finally the first translated information is obtained. The first translated information is displayed on the screen or the first translated information is converted from text information to audio information using Text-To-Speech (TTS) technology and played. Finally, user feedback is provided to confirm the recognized gesture.

[0051] In this embodiment, the language model of the Transformer architecture is a deep learning model that can automatically learn how to capture the semantic and grammatical relationships of the context. Context embedding involves encoding the context information of the text into a low-dimensional vector. In this way, the large language model can better understand the context of each word or element in the text, thereby improving the large language model (LLM)'s ability to understand and process language.

[0052] For example, in a sentence "I went to the park today and the scenery there is beautiful", for the word "park", the context embedding will consider the words before and after it, "I went today" and "the scenery there is beautiful", and encode this information into a vector.

[0053] From the above, it can be concluded that this embodiment achieves accurate translation of complex gesture information by combining a large language model with gesture recognition technology, significantly improving translation efficiency and accuracy. At the same time, the use of recurrent neural networks and long short-term memory networks effectively captures the temporal relationship between gestures and enhances the coherence of translation. In addition, through the refinement of knowledge graphs and RAGs, the translation quality is further improved to ensure the language correctness of the generated text. Finally, the translation results are intuitively displayed through screen display or voice playback, and supplemented by a user feedback mechanism, a closed-loop optimization of the translation process is achieved.

[0054] In this embodiment, the voice data includes voice information in the surrounding environment of the target user who uses the first device and the second device, such as words spoken by a person who communicates with the target user.

[0055] For example, Figure 5 As shown, the collected speech data can be converted into text information as input text. First, the input text is preprocessed, including noise removal, grammar correction, and standardized format. The processed text is semantically analyzed, including part-of-speech tagging, named entity recognition, and dependency parsing, to understand its semantic meaning.

[0056] For example, we can extract lexical features from texts, including the meaning of the words themselves, parts of speech (nouns, verbs, adjectives, etc.), polysemy of words, and common collocations. We can extract grammatical features from texts, including sentence structure, such as the arrangement of subject, predicate, and object, the use of clauses, tense, and voice. We can extract semantic relationship features from texts, including logical relationships between words, such as cause-and-effect relationships, parallel relationships, and transitional relationships. We can extract cultural and idiomatic features from texts. By comprehensively analyzing these features, we can more accurately understand the semantic meaning of the text, thereby achieving more accurate sign language translation.

[0057] Next, a Transformer-based LLM is used to generate context embeddings. Phrases and sentences extracted from the text are mapped to corresponding sign language gestures in the gesture database based on a mapping algorithm. Gestures are adjusted based on broader contextual information. The mapped gestures are synthesized into a continuous sequence of gestures. MediaPipe is used to generate corresponding gestures and body postures, and the synthesized gestures are animated in the 3D model of the signer to generate real-time visual animated gestures that accurately present the movements of the hands and body. The grammatical correctness of the gesture sequence is checked based on the generated animated gestures. Gestures are optimized using the knowledge graph and contextual information from the RAG system. The final animated gestures are then displayed on a display screen (such as the lenses of smart glasses or the touch display of the user interface) or through augmented reality. Feedback is provided to the user to confirm the translated gestures and the corresponding text.

[0058] For example, Xiao Ming, who has hearing impairment, is communicating with others. His friend Xiao Mei says, "Let's go to the park today," and the voice data in the surrounding environment is acquired by Xiao Ming's smartphone (the first device).

[0059] The smartphone performs sign language translation on this voice message and obtains the second text message, including the gesture sequence information "You and I", "Today", "Go", "Park", "Play" and the feature description information corresponding to the gesture. Then, this second text message is further translated to generate the corresponding gesture image as the second translation information, and sent to the smart glasses (second device) used by Xiao Ming. These gesture images can be displayed on the lenses of the smart glasses, allowing Xiao Ming to accurately understand the information that Xiao Mei wants to express.

[0060] In this embodiment, the second information is translated into sign language to obtain the second text information, including: The second information is input into the large language model to generate feature information corresponding to the second information.

[0061] A plurality of second target gesture information matching the feature information is determined from the gesture database as the second text information. The gesture database includes a plurality of gesture information.

[0062] Exemplarily, when the second information is received, it is input into the large language model. The large language model uses the neural network structure and the learned knowledge to encode and transform the input second information to generate corresponding feature information. At the same time, a gesture database is established. Each gesture information in the gesture database has its corresponding feature description or label. After obtaining the feature information of the second information, the database query and matching algorithm are used to search for similar or matching features in the gesture database. For example, using similarity calculation, pattern matching and other technologies, multiple gesture information that meets the requirements are found and output as the second text information.

[0063] Exemplarily, a plurality of phrases or sentences are extracted based on the second information, and the phrases and sentences are mapped to corresponding sign language gestures. This step can be achieved by matching a sign language gesture database and a mapping algorithm. The mapped gestures are adjusted according to the user's conversation with others and the broader context to ensure coherence and relevance. The mapped gestures are synthesized into a continuous sequence to ensure a smooth transition between gestures. The MediaPipe algorithm is used to generate corresponding hand and body movements for each gesture. The synthesized gestures and body movements are animated in the 3D model. The grammatical correctness of the gesture sequence is checked according to the sign language grammar rules. The gesture sequence is adjusted according to the contextual information retrieved from the knowledge graph and RAG. The final perfected gesture sequence is displayed on the lenses of the smart glasses or through an augmented reality interface.

[0064] It can be concluded from the above that this embodiment converts voice or text information into sign language gestures, which not only improves the accuracy and efficiency of information transmission, but also enhances the naturalness and fluency of communication. At the same time, the method is based on a large language model and gesture database, which ensures the accuracy and diversity of translation and can meet the needs of different scenarios and contexts. In addition, the animated gestures are presented through the MediaPipe algorithm and 3D model, making the translation results more vivid and intuitive, improving the user experience.

[0065] Exemplarily, the Chinese sign language translation method based on a large language model provided in this embodiment can be applied to the following scenarios: Live Conversation. Facilitate real-time conversations between hearing-impaired and hearing people.

[0066] Educational settings. Assisting in classroom instruction by translating speech into sign language for teachers who are not proficient in sign language, and translating student signs into text and speech that the teacher can understand.

[0067] Customer Service. Enhance the customer service experience by enabling communication between hearing-impaired customers and hearing service providers.

[0068] Daily communication. It can capture the communication information of others in the environment, provide accurate and real-time translation, and assist in daily communication tasks such as shopping, travel, and socializing.

[0069] It can be concluded from the above that the disclosed embodiment realizes two-way translation between gestures and text and voice, which greatly improves communication efficiency. Users can express through gestures and instantly obtain accurate text translation, which is convenient for non-sign language speakers to understand. At the same time, text or voice information can also be quickly converted into gesture images and displayed on the second device (wearable device), which is convenient for sign language speakers to receive in real time. This two-way translation capability is not only suitable for communication between sign language speakers and non-sign language speakers, but also promotes cross-language and cross-cultural communication, and has broad application prospects and social value.

[0070] Specifically, the disclosed embodiments relate to a method for translating sign language into text and speech and text and speech into sign language using smart glasses, large language model agents, MediaPipe, retrieval-enhanced generation, and knowledge graphs. The method captures sign language gestures through a camera embedded in smart glasses, processes these gestures using MediaPipe for real-time gesture recognition, and uses a large language model agent combined with RAG and knowledge graphs to provide a method for translating text and speech into context and text and speech into sign language. The disclosed embodiments can provide accurate and efficient translation services to promote seamless communication between hearing-impaired people and people with normal hearing.

[0071] In one embodiment of the present disclosure, translating the first text information to generate and output the first translation information includes: The first text information is translated based on the third information to generate and output first translation information.

[0072] The third information is environmental information.

[0073] In this embodiment, environmental information can be collected based on environmental collection sensors. For example, a sound sensor is used to collect surrounding sounds, an image sensor is used to obtain surrounding images, and a positioning sensor is used to determine the current geographical location. The raw data collected by these sensors are pre-processed and feature extracted to be converted into environmental information that can be understood and processed by a computer.

[0074] During the translation process, the first text information is fused with the extracted environmental information. For example, using a machine learning algorithm or a deep learning model, the feature vectors of the text information and the environmental information are input into the model for training and translation. The model will learn how to adjust the translation results of the text information according to different environmental information. In terms of the specific selection of the translation model, a neural network model, such as a model with a Transformer architecture, can be used. During the training process, the model learns how to optimize the translation strategy according to different environmental characteristics.

[0075] For example, if the environment information indicates that the current scene is a medical one, then the translation of some medical terms will use specific professional vocabulary. If the scene is a tourist scene, the description of some places and activities will be more in line with tourist-related terminology.

[0076] This embodiment can provide richer context and background knowledge for the translation process by introducing environmental information, so that the generated translation information is more accurate and fits the actual situation.

[0077] For example, Figure 6 As shown, the translation output process includes: the user inputs gestures, voice or text, the data processing module recognizes and analyzes the input data, generates first text information, inputs the first text information into the large language model intelligent agent, generates text or voice, gesture translation results respectively, and the output module outputs the text and voice to the user. In this process, the user can also interact with information through the user interface.

[0078] For example, Figure 7 As shown in Figure 1, the contextual semantic understanding and translation process includes: MediaPipe is used to recognize gestures, and the gesture recognition results are input into the large language model agent, which is then used to translate gestures. At the same time, the retrieval enhancement generation system can be used to retrieve contextual semantic information, and the contextual semantic data can be queried through the knowledge graph and provided to the retrieval enhancement generation system. The retrieval enhancement generation system sends the final contextual information obtained to the large language model agent to assist in translation. The large language model agent generates text and voice translation content, which is then displayed or played by the output module.

[0079] In one embodiment of the present disclosure, a Chinese sign language translation method based on a large language model is applied to a second device, including: In response to capturing the gesture image, the gesture image is sent to the first device. The first device is a device connected to the second device.

[0080] In response to receiving the second translation information, the second translation information is displayed. The second translation information includes a gesture image corresponding to the second information. The second information includes text data and / or voice data to be recognized.

[0081] In this embodiment, the second device includes smart glasses. The high-resolution camera embedded in the second device can capture detailed images and videos of the surrounding environment. The second device is also equipped with a depth sensor (for measuring the distance from the device to objects, which helps spatial perception) and a microphone (for capturing ambient sound and user commands).

[0082] For example, smart glasses can continuously record the user's environment through a camera to provide real-time visual data, which can be used for object detection, obstacle recognition, and navigation assistance. Depth sensors can create a 3D map of the surrounding environment through the measured distance information to surrounding objects, and store the spatial arrangement and distance of various objects. Smart glasses can record and capture ambient sounds and specific audio cues related to the user's surroundings through a microphone, and provide contextual information and user commands after processing.

[0083] The Inertial Measurement Unit (IMU) can provide data about direction, motion, and acceleration to understand the user's physical state and movement. For example, the user's head movement and body movement can be detected and interpreted based on the IMU, and additional background or control information input can be provided to provide more data information for the large language model, making the translation result more accurate under the background constraints. The second device can communicate with the first device via wireless technologies such as Bluetooth and Wi-Fi.

[0084] For example, smart glasses use computer vision algorithms to identify and mark objects in the environment through information collected by depth sensors, and analyze wider scenes to understand the context, such as identifying crosswalks, bus stops, or specific rooms in a house. Combining data from multiple depth sensors can fully understand the user's environment and provide relevant help in a timely manner. The detection range of the depth sensor can cover 0.2 meters to 10 meters to achieve effective spatial mapping.

[0085] This embodiment uses smart glasses to capture gesture images and send them to the connected first device, achieving a more intuitive and efficient interaction method. At the same time, combined with multiple sensor data such as depth sensors, microphones and IMUs, the large language model can obtain more comprehensive background information, making the translation results more accurate and close to the actual scene. This method not only improves the accuracy of translation and user experience, but also broadens the application scenarios of smart glasses in daily life.

[0086] This embodiment also provides a translation system, including an image capturing module, a data processing module, an output module and a user interface.

[0087] The image capturing module is configured in the second device, and the data processing module, the output module and the user interface are configured in the first device.

[0088] The first device is connected to the second device.

[0089] refer to Figure 8 , Figure 8 A schematic diagram of a signaling interaction process between devices provided in an embodiment of the present disclosure. The signaling interaction process between the first device and the second device is as follows: A01: Capture gesture images.

[0090] A02: Send gesture image.

[0091] In response to capturing the gesture image, the second device sends the gesture image to the first device. The first device is a device connected to the second device.

[0092] A03: The first text message is generated.

[0093] In response to receiving the first information, the first device performs gesture recognition on the first information to obtain first text information. The first information includes the gesture image sent by the second device.

[0094] A04: Generate and output first translation information.

[0095] The first device translates the first text information, generates and outputs first translation information, the second device is a device connected to the first device, and the first translation information includes text data corresponding to the first information.

[0096] A05: The second text message is generated.

[0097] In response to receiving the second information, the first device performs sign language translation on the second information to obtain second text information. The second information includes text data and / or voice data to be recognized.

[0098] A06: Second translation information is generated.

[0099] A07: Send the second translation information.

[0100] The first device translates the second text information, generates and sends second translation information to the second device, and the second translation information includes a gesture image corresponding to the second information.

[0101] A08: Display the second translation information.

[0102] In response to receiving the second translation information, the second device displays the second translation information.

[0103] From the above, it can be concluded that this embodiment realizes the conversion and translation of gesture images into text information, and the conversion and translation of text / voice information into gesture images through signaling interaction between devices. It improves the efficiency of barrier-free communication, provides a more convenient communication method for people with hearing impairments and speech impairments, enriches the means of communication, and improves the user experience.

[0104] Corresponding to the Chinese sign language translation method based on a large language model applied to the first device in the above embodiment, Fig. 9 This is a structural block diagram of a Chinese sign language translation device based on a large language model provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Fig. 9 , a Chinese sign language translation device 20 based on a large language model, applied to a first device, comprising: The first translation module is used to perform gesture recognition on the first information in response to receiving the first information to obtain the first text information. The first text information is translated to generate and output the first translation information, the first information includes the gesture image sent by the second device, the second device is a device connected to the first device, and the first translation information includes the text data corresponding to the first information.

[0105] The second translation module is used to translate the second information into sign language in response to receiving the second information to obtain the second text information. The second text information is translated to generate and send the second translation information to the second device, wherein the second information includes the text data and / or voice data to be recognized, and the second translation information includes the gesture image corresponding to the second information.

[0106] In one embodiment of the present disclosure, the first translation module 21 is specifically used to determine the first action and the second action of the target user based on the first information. The first action is a hand action, and the second action is a body action. The first device and the second device are both devices used by the target user.

[0107] Based on the first action and the second action, a plurality of first target gesture information is matched from the plurality of gesture information as the first text information.

[0108] In an embodiment of the present disclosure, the first translation module 21 is further configured to: the first text information includes a plurality of first target gesture information.

[0109] The plurality of first target gesture information and the time dependency relationship corresponding to each of the first target gesture information are input into the large language model, and the first translation information is generated and output.

[0110] In one embodiment of the present disclosure, the second translation module is specifically configured to input the second information into the large language model to generate feature information corresponding to the second information.

[0111] A plurality of second target gesture information matching the feature information is determined from the gesture database as the second text information. The gesture database includes a plurality of gesture information.

[0112] In one embodiment of the present disclosure, the second translation module is further configured to determine a target hand motion and a target body motion based on the second text information.

[0113] The target hand motion and the target body motion are translated, and second translation information is generated and sent to the second device.

[0114] In an embodiment of the present disclosure, the first translation module 21 is further configured to translate the first text information based on the third information, and generate and output first translation information.

[0115] The third information is environmental information.

[0116] Corresponding to the Chinese sign language translation method based on a large language model applied to the second device in the above embodiment, an embodiment of the present disclosure further provides another Chinese sign language translation device based on a large language model, which is applied to the second device, including: The first response module is used to send the gesture image to the first device in response to capturing the gesture image. The first device is a device connected to the second device.

[0117] The second response module is used to display the second translation information in response to receiving the second translation information. The second translation information includes a gesture image corresponding to the second information. The second information includes text data and / or voice data to be recognized.

[0118] In another embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by the processor, all or part of the processes in the above-mentioned embodiment method are implemented, and the computer program can also be completed by instructing the relevant hardware through the computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0119] The computer-readable storage medium may be an internal storage unit of the translation system of any of the aforementioned embodiments, such as a hard disk or memory of the translation system. The computer-readable storage medium may also be an external storage device of the translation system, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the translation system. Further, the computer-readable storage medium may also include both an internal storage unit of the translation system and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the translation system. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0120] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0121] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the translation system and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0122] In the several embodiments provided in the present application, it should be understood that the disclosed translation system and method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or it can be an electrical, mechanical or other form of connection.

[0123] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.

[0124] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0125] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A Chinese sign language translation method based on a large language model, characterized in that: Applied to a first device, comprising: In response to receiving a first message, performing gesture recognition on the first message to obtain a first text message; translating the first text message to generate and output first translation information, wherein the first message includes a gesture image sent by a second device, the second device is a device connected to the first device, and the first translation information includes text data corresponding to the first message; In response to receiving a second message, the second message is translated into sign language to obtain a second text message; the second text message is translated to generate and send a second translation message to the second device, wherein the second message includes text data and / or voice data to be recognized, and the second translation message includes a gesture image corresponding to the second message.

2. A Chinese sign language translation method based on a large language model as claimed in claim 1, characterized in that: The performing gesture recognition on the first information to obtain first text information includes: Determine a first action and a second action of a target user based on the first information; the first action is a hand action, and the second action is a body action; the first device and the second device are both devices used by the target user; Based on the first action and the second action, a plurality of first target gesture information is matched from a plurality of gesture information as the first text information.

3. A Chinese sign language translation method based on a large language model as claimed in claim 1, characterized in that: The translating the first text information to generate and output first translation information includes: The first text information includes a plurality of first target gesture information; The multiple first target gesture information and the time dependency corresponding to each first target gesture information are input into a large language model to generate and output the first translation information.

4. The Chinese sign language translation method based on a large language model as claimed in claim 1, characterized in that: The performing sign language translation on the second information to obtain second text information includes: Inputting the second information into a large language model to generate feature information corresponding to the second information; A plurality of second target gesture information matching the characteristic information is determined from a gesture database as second text information; the gesture database includes a plurality of gesture information.

5. The Chinese sign language translation method based on a large language model as claimed in claim 1, characterized in that: The translating the second text information to generate and send second translation information to the second device includes: determining a target hand motion and a target body motion based on the second text information; The target hand motion and the target body motion are translated, and second translation information is generated and sent to the second device.

6. The Chinese sign language translation method based on a large language model as claimed in claim 1, characterized in that: The translating the first text information to generate and output first translation information includes: Translate the first text information based on the third information, generate and output first translation information; The third information is environmental information.

7. A Chinese sign language translation method based on a large language model, characterized in that: Applied to the second device, comprising: In response to capturing a gesture image, sending the gesture image to a first device; the first device is a device connected to the second device; In response to receiving the second translation information, the second translation information is displayed; the second translation information includes a gesture image corresponding to the second information; the second information includes text data and / or voice data to be recognized.

8. A Chinese sign language translation device based on a large language model, characterized in that: Applied to a first device, comprising: a first translation module, configured to, in response to receiving a first message, perform gesture recognition on the first message to obtain a first text message; translate the first text message to generate and output first translation information, wherein the first message includes a gesture image sent by a second device, the second device is a device connected to the first device, and the first translation information includes text data corresponding to the first message; The second translation module is used to, in response to receiving the second information, perform sign language translation on the second information to obtain the second text information; translate the second text information, generate and send the second translation information to the second device, the second information includes text data and / or voice data to be recognized, and the second translation information includes a gesture image corresponding to the second information.

9. A Chinese sign language translation system based on a large language model, characterized in that: It includes an image capturing module, a data processing module, an output module and a user interface; The image capturing module is configured in the second device, and the data processing module, the output module and the user interface are configured in the first device; The first device and the second device are connected.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 or the steps of the method according to claim 7 are implemented.

Images