Sign language translation method and device based on large language model

Abstract

The invention belongs to the crossing field of artificial intelligence and barrier-free assistance technology, and particularly discloses a sign language translation method and device based on a large language model.The method is applied to a terminal and comprises the steps that key points are extracted from sign language video frames; based on the coordinates, types and global indexes of the key points in the sign language video frames, obtaining fusion features through feature fusion; based on the fusion features corresponding to the sign language video frames in the frame window, time sequence modeling and classification are carried out, and a word prediction result of the frame window is obtained; a sliding window voting and deduplication mode is adopted, the candidate word sequence is preprocessed, a preprocessed word sequence is obtained, and the candidate word sequence is formed by connecting word prediction results of all frame windows in series according to a time sequence; based on the preprocessed word sequence, word error correction and statement generation are carried out by calling a large language model, and a sign language translation result is generated. According to the invention, sign language video translation can be realized for the terminal, and the accuracy and fluency of the translation result can be guaranteed.

Description

Technical Field

[0001] This application belongs to the interdisciplinary field of artificial intelligence and accessibility technology, and more specifically, relates to a sign language translation method and device based on a large language model. Background Technology

[0002] Traditional sign language video translation solutions typically use complex end-to-end models to recognize sign language videos and translate them into text. The high complexity of end-to-end models makes them difficult to deploy on mobile devices. Furthermore, traditional sign language video translation solutions suffer from problems such as misjudgment of temporal boundaries in transitional areas of continuous sign language, leading to reduced translation accuracy, and resulting in awkward or semantically disjointed sentences. How to achieve sign language video translation on mobile devices while ensuring the accuracy and fluency of the translation results is a pressing technical problem that needs to be solved in this field. Summary of the Invention

[0003] In view of the shortcomings of the existing technology, the purpose of this application is to realize sign language video translation for terminals and ensure the accuracy and fluency of the translation results.

[0004] To achieve the above objectives, firstly, this application provides a sign language translation method based on a large language model, applied to a terminal, comprising: Extract key points from sign language video frames and obtain the coordinates, type, and global index of key points in the sign language video frames; Based on the coordinates, type, and global index of key points in sign language video frames, the fused features corresponding to the sign language video frames are obtained through feature fusion. Based on the fusion features corresponding to each sign language video frame in the frame window, temporal modeling and classification are performed to obtain the word prediction results of the frame window. The frame window includes multiple sign language video frames and the multiple sign language video frames are temporally continuous. The candidate word sequence is preprocessed using a sliding window voting and deduplication method to obtain the preprocessed word sequence. The candidate word sequence is constructed by concatenating the word prediction results of each frame window in time sequence. Based on the preprocessed word sequence, word correction and sentence generation are performed by calling a large language model to generate sign language translation results. The large language model is configured on the server, and communication is established between the terminal and the server.

[0005] In one possible implementation, based on the coordinates, type, and global index of key points in the sign language video frame, feature fusion is used to obtain the fused features corresponding to the sign language video frame, including: Based on the coordinates of key points in sign language video frames, spatial feature vectors are obtained through linear projection; Based on the type of key points in the sign language video frame, the first semantic vector is obtained through embedding encoding; based on the global index of key points in the sign language video frame, the second semantic vector is obtained through embedding encoding. Based on spatial feature vectors, first semantic vectors, and second semantic vectors, fused features corresponding to sign language video frames are obtained through fusion and dimensionality reduction.

[0006] In one possible implementation, temporal modeling and classification are performed based on the fused features corresponding to each sign language video frame within the frame window to obtain the word prediction results for the frame window, including: Based on the fusion features corresponding to each sign language video frame in the frame window, a two-stage LSTM model is used for temporal modeling to obtain the temporal feature representation corresponding to the frame window. The two-stage LSTM model is composed of a cascaded first LSTM, a perceptron, and a second LSTM. Based on the temporal feature representation corresponding to the frame window, a classifier is used to classify the words and obtain the word prediction results of the frame window.

[0007] In one possible implementation, a sliding window voting and deduplication method is used to preprocess the candidate word sequence to obtain the preprocessed word sequence, including: The sliding window moves across the candidate word sequence according to a specified step size. The sliding window is used to select multiple consecutive words in the candidate word sequence based on a specified window size for prediction results. Each time the sliding window is moved, a voting operation is performed. The voting operation includes predicting the word selected in the sliding window, counting the number of times different sign language words appear, and selecting the sign language word with the most appearances as the representative. Based on the representatives selected by each sliding window movement, they are concatenated in chronological order to form a denoised word sequence; Perform deduplication detection on any two adjacent sign language words in the denoised word sequence to obtain the preprocessed word sequence. Deduplication detection includes deleting one of the first sign language word and the second sign language word if they are the same. The first sign language word and the second sign language word are any two adjacent sign language words in the denoised word sequence.

[0008] In one possible implementation, based on the preprocessed word sequence, word correction and sentence generation are performed by calling a large language model to generate sign language translation results, including: Based on the preprocessed word sequence, target prompt words are generated. The target prompt words are used to instruct the large language model to perform word correction and sentence generation on the preprocessed word sequence. Input the target prompt word into the large language model and obtain the sign language translation result output by the large language model.

[0009] In one possible implementation, target prompt words are generated based on the preprocessed word sequence, including: Based on the sign language vocabulary in the preprocessed word sequence, relevant sign language definitions, sign language grammar, and sign language example sentences are retrieved from the sign language knowledge base to generate search results; Based on the preprocessed word sequence and retrieval results, target prompt words are constructed. These target prompt words are used to indicate the reference retrieval results to the large language model, and word correction and sentence generation are performed on the preprocessed word sequence.

[0010] In one possible implementation, the tool used to extract key points from sign language video frames is MediaPipe.

[0011] Secondly, this application provides a sign language translation device based on a large language model, applied to a terminal, comprising: The key point extraction module is used to extract key points from sign language video frames and obtain the coordinates, type, and global index of key points in the sign language video frames. The feature fusion module is used to obtain the fused features corresponding to the sign language video frame by using feature fusion based on the coordinates, type and global index of key points in the sign language video frame; The temporal modeling and classification module is used to perform temporal modeling and classification based on the fusion features corresponding to each sign language video frame in the frame window, and to obtain the word prediction results of the frame window. The frame window includes multiple sign language video frames and the multiple sign language video frames are temporally continuous. The word sequence preprocessing module is used to preprocess the candidate word sequence using sliding window voting and deduplication methods to obtain the preprocessed word sequence. The candidate word sequence is constructed by concatenating the word prediction results of each frame window in time sequence. The large model calling module is used to perform word correction and sentence generation based on the preprocessed word sequence by calling the large language model, and generate sign language translation results. The large language model is configured on the server, and the terminal communicates with the server.

[0012] Thirdly, this application provides a terminal, including: a memory and one or more processors; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions; the one or more processors invoke the computer instructions to cause the terminal to perform the method described in the first aspect or any possible implementation of the first aspect.

[0013] Fourthly, this application provides a computer-readable storage medium including instructions that, when executed on a terminal, cause the terminal to perform the method described in the first aspect or any possible implementation thereof.

[0014] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art: (1) The sign language video recognition and word correction sentence generation processes in sign language video translation are decoupled. The terminal performs sign language video recognition, while the server performs word correction sentence generation. Specifically, the terminal performs sign language video recognition by acquiring key points in the sign language video frames, fusing features, performing temporal modeling and classification, and preprocessing candidate word sequences. The server uses a deployed Large Language Model (LLM) to perform word correction and sentence generation on the preprocessed word sequences to generate sign language translation results. This decoupling method enables the model deployed on the terminal to be lightweight, reducing the computational requirements on the terminal.

[0015] (2) By adopting sliding window voting and deduplication, instantaneous noise can be effectively filtered, the stability of word sequence output can be improved, the jitter and misidentification caused by action switching can be significantly reduced, and a clean and reliable input can be provided for the subsequent generation of high-quality sentences by LLM, thereby improving the accuracy of translation results and enhancing the overall robustness of the system.

[0016] (3) By using a large language model to perform word correction and sentence generation on the preprocessed word sequence, the natural language processing capabilities of the large language model can be utilized to make up for the shortcomings of the pure visual model at the language level, thereby effectively improving the fluency of the translation results. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the sign language translation method based on a large language model provided in an embodiment of this application; Figure 2 This is a schematic diagram of the feature fusion module provided in an embodiment of this application; Figure 3 This is a schematic diagram of the timing modeling module provided in an embodiment of this application; Figure 4 This is a schematic diagram illustrating the use of sliding window voting and deduplication in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of the sign language translation device based on a large language model provided in the embodiments of this application; Figure 6 This is a schematic diagram of the terminal structure provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] In this application, the terms "first" and "second," etc., are used to distinguish different objects, not to describe a specific order of objects. For example, "first semantic vector" and "second semantic vector," etc., are used to distinguish different semantic vectors, not to describe a specific order of semantic vectors.

[0020] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0021] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0022] The embodiments of this application are described below with reference to the accompanying drawings.

[0023] Figure 1 This is a flowchart illustrating the sign language translation method based on a large language model provided in an embodiment of this application, as shown below. Figure 1 As shown, the method is applied to a terminal and includes the following steps S101 to S105.

[0024] Step S101: Extract key points from the sign language video frame and obtain the coordinates, type, and global index of the key points in the sign language video frame; Step S102: Based on the coordinates, type, and global index of key points in the sign language video frame, obtain the fused features corresponding to the sign language video frame through feature fusion; Step S103: Based on the fusion features corresponding to each sign language video frame in the frame window, perform temporal modeling and classification to obtain the word prediction results of the frame window (the word prediction results are used to represent the sign language words extracted from the sign language video frames in the corresponding frame window, and the sign language words are the words corresponding to the sign language actions). The frame window includes multiple sign language video frames and the multiple sign language video frames are sequential in time. Step S104: The candidate word sequence is preprocessed using a sliding window voting and deduplication method to obtain the preprocessed word sequence. The candidate word sequence is constructed by concatenating the word prediction results of each frame window in time sequence. Step S105: Based on the preprocessed word sequence, word correction and sentence generation are performed by calling the large language model to generate sign language translation results. The large language model is configured on the server, and a communication connection is established between the terminal and the server.

[0025] Specifically, the method first extracts key points frame by frame from the sign language video on the terminal side, obtaining the coordinates, type, and global index of the key points, and then obtains the fused features of each frame through feature fusion. Next, based on a frame window composed of multiple consecutive frames, the fused features of each frame within the window are temporally modeled and classified, outputting the word prediction result corresponding to that window (i.e., the recognized sign language vocabulary). Then, the word prediction results of each window are concatenated temporally to form a candidate word sequence, and preprocessing is performed using sliding window voting and deduplication methods to filter instantaneous noise and improve sequence stability. Finally, the preprocessed word sequence is sent to the server, where a large language model deployed on the server is invoked for word correction and sentence generation, outputting the final sign language translation result. This method decouples sign language video recognition from language generation, achieving lightweight visual recognition on the terminal and utilizing the natural language processing capabilities of a large language model on the server, jointly ensuring the accuracy and fluency of the translation result.

[0026] Understandably, this application decouples sign language video recognition from word correction and sentence generation in the sign language video translation process. The terminal performs sign language video recognition, while the server performs word correction and sentence generation. Specifically, the terminal achieves sign language video recognition by acquiring key points in the sign language video frames, fusing features, performing temporal modeling and classification, and preprocessing candidate word sequences. The server uses a deployed Large Language Model (LLM) to perform word correction and sentence generation on the preprocessed word sequences to generate the sign language translation result. This decoupling method enables the model deployed on the terminal to be lightweight, reducing the computational requirements on the terminal.

[0027] Furthermore, by employing sliding window voting and deduplication methods, transient noise can be effectively filtered, improving the stability of word sequence output and significantly reducing jitter and misidentification caused by action switching. This provides clean and reliable input for subsequent LLM generation of high-quality sentences, thereby improving the accuracy of translation results and enhancing the overall robustness of the system. By using a large language model to perform word correction and sentence generation on the preprocessed word sequence, the natural language processing capabilities of the large language model can be leveraged to compensate for the shortcomings of pure visual models at the language level, effectively improving the fluency of the translation results.

[0028] Therefore, by combining the above decoupling methods, filtering instantaneous noise, and the natural language processing capabilities of the large language model, it is possible to achieve sign language video translation on the terminal and ensure the accuracy and fluency of the translation results.

[0029] The sign language translation method based on a large language model provided in this application can provide a technical foundation for VR / AR immersive barrier-free communication and promote barrier-free communication.

[0030] This section provides an example of how to acquire sign language video frames. The system captures the user's sign language video stream in real time using a camera, and then dynamically downsamples the video frames to obtain the sign language video frames. The sampling interval r can be adaptive according to the scene; for example, r=2 means taking 1 frame every 2 frames, and r=3 means taking 1 frame every 3 frames.

[0031] For example, a large language model can be implemented using DeepSeek-R1-Distill-Qwen-32B.

[0032] Optionally, the sign language translation result is text content, which can be converted into fluent speech using ChatTTS.

[0033] In one possible implementation, the tool used to extract key points from the aforementioned sign language video frames is MediaPipe.

[0034] MediaPipe is an open-source, cross-platform machine learning framework for building real-time multimodal (visual, audio, etc.) perception applications.

[0035] MediaPipe can extract 543 key points of human posture, hands, and face from videos to form spatiotemporal sequence data.

[0036] For example, MediaPipe is used to extract 543 keypoints for each frame of the image (468 for the face, 33 for the pose, 21 for the left hand, and 21 for the right hand). Each keypoint contains: 3D coordinates Confidence level (conf), type identifier (type_id) ), and a constant global index global_idx, ranging from 0 to 542, used to uniquely represent each keypoint.

[0037] The coordinates of key points are uniformly normalized; undetected key points are filled with fixed placeholder values: coordinates Confidence level .

[0038] Every 30 frames are packed into a sequence tensor of shape 30×543.

[0039] In one possible implementation, the above-mentioned feature fusion based on the coordinates, type, and global index of key points in the sign language video frame is used to obtain the fused features corresponding to the sign language video frame, including: Based on the coordinates of key points in sign language video frames, spatial feature vectors are obtained through linear projection; Based on the type of key points in the sign language video frame, the first semantic vector is obtained through embedding encoding; based on the global index of key points in the sign language video frame, the second semantic vector is obtained through embedding encoding. Based on spatial feature vectors, first semantic vectors, and second semantic vectors, fused features corresponding to sign language video frames are obtained through fusion and dimensionality reduction.

[0040] Specifically, the key points The spatial feature vector is obtained through linear projection; the type ID and global index of the key point are embedded into semantic vectors respectively; the three are concatenated and processed by a multilayer perceptron to obtain a 16-dimensional fusion feature for each point, which is then flattened into a frame-level feature vector and reduced to 768 dimensions.

[0041] For example, a feature fusion module (FeatureFusion) can be used to fuse features of key points in sign language video frames, such as... Figure 2 As shown, the linear projection unit can be constructed by cascading Linear layers, LayerNorm layers, GELU layers, and Dropout layers, and the perceptron can be constructed by cascading Linear layers, LayerNorm layers, GELU layers, and Dropout layers.

[0042] In one possible implementation, the above-mentioned temporal modeling and classification are performed based on the fused features corresponding to each sign language video frame in the frame window to obtain the word prediction results of the frame window, including: Based on the fusion features corresponding to each sign language video frame in the frame window, a two-stage LSTM (Long Short-Term Memory) network model is used for temporal modeling to obtain the temporal feature representation corresponding to the frame window. The two-stage LSTM model is composed of a cascaded first LSTM, a perceptron, and a second LSTM. Based on the temporal feature representation corresponding to the frame window, a classifier is used to classify the words and obtain the word prediction results of the frame window.

[0043] It is understandable that concatenating the word prediction results of each frame window in time sequence can form a candidate word sequence.

[0044] For example, a temporal modeling module (two-stage LSTM model) can be used to obtain the temporal feature representation corresponding to the frame window, such as... Figure 3 As shown, the time series modeling module is composed of cascaded LSTM, perceptron and LSTM, where the perceptron can be constructed by cascading Linear layer, LayerNorm layer, GELU layer and Dropout layer.

[0045] For example, the classifier can be constructed by cascading Linear layers, LayerNorm layers, GELU layers, Dropout layers, and Linear layers.

[0046] In one possible implementation, the above-mentioned method uses sliding window voting and deduplication to preprocess the candidate word sequence, obtaining the preprocessed word sequence, including: The sliding window moves across the candidate word sequence according to a specified step size. The sliding window is used to select multiple consecutive words in the candidate word sequence based on a specified window size for prediction results. Each time the sliding window is moved, a voting operation is performed. The voting operation includes predicting the word selected in the sliding window, counting the number of times different sign language words appear, and selecting the sign language word with the most appearances as the representative. Based on the representatives selected by each sliding window movement, they are concatenated in chronological order to form a denoised word sequence; Perform deduplication detection on any two adjacent sign language words in the denoised word sequence to obtain the preprocessed word sequence. Deduplication detection includes deleting one of the first sign language word and the second sign language word if they are the same. The first sign language word and the second sign language word are any two adjacent sign language words in the denoised word sequence.

[0047] Specifically, the obtained candidate word sequences are processed using a sliding window voting mechanism, with the window size being... The step size s can be adaptively set according to the actual situation: for steps starting from the first... A window for each word The number of times each word tag appears in the window is counted, and the mode is taken as the output tag of the window, thereby reducing the interference of instantaneous noise.

[0048] For example, such as Figure 4 As shown, the process of moving the sliding window and performing the voting operation is as follows: a~e: a. The system reads in the original 7 prediction sequences: [vocabulary 1, vocabulary 2, vocabulary 1, vocabulary 1, vocabulary 4, vocabulary 3, vocabulary 3]; b. The window size is 4. Window 1 contains 3 words 1 and 1 word 2. Voting determines word 1. c. Window 1 is moved one frame to the next frame and becomes Window 2; d. Repeat step b until all frames have been covered; e. The final sequence is: [vocabulary 1, vocabulary 1, vocabulary 1, vocabulary 3], which becomes [vocabulary 1, vocabulary 3] after deduplication.

[0049] like Figure 4 As shown, based on the representatives selected by each sliding window movement, they are concatenated in time to form a denoised word sequence. Then, the consecutive words of the same category in the denoised word sequence are deduplicated to obtain a stable word sequence, which is the preprocessed word sequence.

[0050] In one possible implementation, the above-mentioned preprocessed word sequence is used to generate sign language translation results by calling a large language model for word correction and sentence generation, including: Based on the preprocessed word sequence, target prompt words are generated. The target prompt words are used to instruct the large language model to perform word correction and sentence generation on the preprocessed word sequence. Input the target prompt word into the large language model and obtain the sign language translation result output by the large language model.

[0051] For example, the preprocessed word sequence is [you, good, very, happy, you, know, you, call, what, name]. Target prompt words can be generated based on this word sequence to instruct the large language model to perform word correction and sentence generation on the word sequence. Then, by inputting the target prompt words into the large language model, sign language translation results can be obtained, such as {Hello! It's nice to meet you! May I ask what your name is?}.

[0052] Optionally, a prompt word template can be pre-built, which includes specified placeholders, role descriptions, task descriptions, input requirements, reasoning process requirements, and output requirements. The specified placeholders in the prompt word template can be replaced with a pre-processed word sequence, resulting in the target prompt word.

[0053] The role description section is used to describe the sign language translation role represented by the large language model.

[0054] The task description section describes the task content of sign language translation for the large language model.

[0055] The task description includes: a description of the word correction task and a description of the sentence generation task.

[0056] The input requirements section describes the requirements for the content input to the large language model.

[0057] The reasoning process requirements are used to describe the requirements for the reasoning process of performing sign language translation on a large language model.

[0058] The output requirements section describes the requirements for the output content of the large language model.

[0059] In one possible implementation, the above-mentioned generation of target prompt words based on the preprocessed word sequence includes: Based on the sign language vocabulary in the preprocessed word sequence, relevant sign language definitions, sign language grammar, and sign language example sentences are retrieved from the sign language knowledge base to generate search results; Based on the preprocessed word sequence and retrieval results, target prompt words are constructed. These target prompt words are used to indicate the reference retrieval results to the large language model, and word correction and sentence generation are performed on the preprocessed word sequence.

[0060] Understandably, a retrieval enhancement mechanism (RAG) is introduced into the sign language translation process to construct an external terminology database (serving as a sign language knowledge base) that combines word and definition. When the word sequence input to the large language model contains proper nouns, the corresponding term card (standard name + brief definition) is retrieved from the database and input into the large language model along with the original word sequence. This constrains error correction and sentence formation, improving accuracy in professional scenarios, reducing mistranslations, and making the translation more fluent and in line with natural language habits.

[0061] The sign language translation device based on a large language model provided in this application is described below. The sign language translation device based on a large language model described below can be referred to in correspondence with the sign language translation method based on a large language model described above.

[0062] Figure 5 This is a schematic diagram of the structure of the sign language translation device based on a large language model provided in the embodiments of this application, such as... Figure 5 As shown, this device is applied to a terminal and includes: a key point extraction module 10, a feature fusion module 20, a temporal modeling and classification module 30, a word sequence preprocessing module 40, and a large model calling module 50. Among them: The key point extraction module 10 is used to extract key points from sign language video frames and obtain the coordinates, type and global index of key points in the sign language video frames. The feature fusion module 20 is used to obtain the fused features corresponding to the sign language video frame by fusion based on the coordinates, type and global index of key points in the sign language video frame. The temporal modeling and classification module 30 is used to perform temporal modeling and classification based on the fusion features corresponding to each sign language video frame in the frame window, and to obtain the word prediction results of the frame window. The frame window includes multiple sign language video frames and the multiple sign language video frames are temporally continuous. The word sequence preprocessing module 40 is used to preprocess the candidate word sequence using a sliding window voting and deduplication method to obtain the preprocessed word sequence. The candidate word sequence is constructed by concatenating the word prediction results of each frame window in time sequence. The large model calling module 50 is used to perform word correction and sentence generation based on the preprocessed word sequence by calling the large language model, and generate sign language translation results. The large language model is configured on the server, and the terminal communicates with the server.

[0063] It is understood that the detailed functional implementation of each of the above units / modules can be found in the description in the aforementioned method embodiments, and will not be repeated here.

[0064] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0065] Based on the methods in the above embodiments, this application provides a terminal, such as... Figure 6 As shown, the terminal may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the methods in the above embodiments.

[0066] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0067] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0068] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0069] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0070] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0071] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0072] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0073] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for sign language translation based on a large language model, characterized in that, Applied to a terminal, comprising: Key points are extracted from a sign language video frame, and coordinates, types, and global indexes of the key points in the sign language video frame are obtained; Based on the coordinates, types, and global indexes of the key points in the sign language video frame, a fusion feature corresponding to the sign language video frame is obtained through feature fusion; Based on the fusion features corresponding to each sign language video frame in a frame window, time sequence modeling and classification are performed to obtain a word prediction result of the frame window, and the frame window includes multiple sign language video frames and the multiple sign language video frames are continuous in time sequence; A sliding window voting and deduplication method is used to preprocess a candidate word sequence to obtain a preprocessed word sequence, and the candidate word sequence is formed by concatenating the word prediction results of each frame window in time sequence; Based on the preprocessed word sequence, a large language model configured in a server is called to perform word error correction and sentence generation to generate a sign language translation result, and the terminal is in communication connection with the server.

2. The sign language translation method based on a large language model according to claim 1, characterized in that, The method for obtaining the fusion feature corresponding to the sign language video frame based on the coordinates, types, and global indexes of the key points in the sign language video frame comprises: Based on the coordinates of the key points in the sign language video frame, a spatial feature vector is obtained through linear projection; Based on the types of the key points in the sign language video frame, a first semantic vector is obtained through embedding coding, and based on the global indexes of the key points in the sign language video frame, a second semantic vector is obtained through embedding coding; Based on the spatial feature vector, the first semantic vector, and the second semantic vector, the fusion feature corresponding to the sign language video frame is obtained through fusion and dimension reduction.

3. The sign language translation method based on a large language model according to claim 1, characterized in that, The method for obtaining the word prediction result of the frame window based on the fusion features corresponding to each sign language video frame in the frame window comprises: Based on the fusion features corresponding to each sign language video frame in the frame window, a two-stage LSTM model is used for time sequence modeling to obtain a time sequence feature representation corresponding to the frame window, and the two-stage LSTM model is formed by cascading a first LSTM, a perception machine, and a second LSTM; Based on the time sequence feature representation corresponding to the frame window, a classifier is used for classification to obtain the word prediction result of the frame window.

4. The sign language translation method based on a large language model according to claim 1, characterized in that, The method for preprocessing the candidate word sequence to obtain the preprocessed word sequence by using the sliding window voting and deduplication method comprises: A sliding window is controlled to move on the candidate word sequence according to a specified step size, and the sliding window is used to select multiple continuous word prediction results on the candidate word sequence based on a specified window size; Each time the sliding window moves, a voting operation is performed, and the voting operation includes counting the number of occurrences of different sign language words based on the selected word prediction results of the sliding window, and selecting a sign language word with the highest number of occurrences as a representative; The representative selected by each movement of the sliding window is concatenated in time sequence to form a denoised word sequence; Deduplication detection is performed on any two adjacent sign language words in the denoised word sequence to obtain the preprocessed word sequence, and the deduplication detection includes deleting one of the first sign language word and the second sign language word if the first sign language word and the second sign language word are the same, and the first sign language word and the second sign language word are any two adjacent sign language words in the denoised word sequence.

5. The sign language translation method based on a large language model according to claim 1, characterized in that, The hand sign translation result is generated by calling the large language model based on the preprocessed word sequence, including: Based on the preprocessed word sequence, a target prompt word is generated, which is used to instruct the large language model to perform word correction and sentence generation on the preprocessed word sequence; The target prompt word is input into the large language model to obtain the hand sign translation result output by the large language model.

6. The sign language translation method based on a large language model according to claim 5, characterized in that, The target prompt word is generated based on the preprocessed word sequence, including: Based on the hand sign vocabulary in the preprocessed word sequence, relevant hand sign definitions, hand sign grammar and hand sign example sentences are retrieved in the hand sign knowledge base to form a retrieval result; Based on the preprocessed word sequence and the retrieval result, a target prompt word is constructed, which is used to instruct the large language model to perform word correction and sentence generation on the preprocessed word sequence based on the retrieval result.

7. The sign language translation method based on a large language model according to claim 1, characterized in that, The tool used to extract key points from the hand sign video frame is MediaPipe.

8. A sign language translation apparatus based on a large language model, characterized by, Applied to a terminal, including: A key point extraction module is used to extract key points from the hand sign video frame to obtain the coordinates, types and global indexes of the key points in the hand sign video frame; A feature fusion module is used to obtain the fusion features corresponding to the hand sign video frame based on the coordinates, types and global indexes of the key points in the hand sign video frame through feature fusion; A time series modeling and classification module is used to perform time series modeling and classification based on the fusion features corresponding to each hand sign video frame in the frame window to obtain the word prediction result of the frame window, the frame window including multiple hand sign video frames that are continuous in time sequence; A word sequence preprocessing module is used to preprocess the candidate word sequence by using a sliding window voting and deduplication method to obtain a preprocessed word sequence, the candidate word sequence being formed by concatenating the word prediction results of each frame window in time sequence; A large model calling module is used to generate a hand sign translation result by calling a large language model based on the preprocessed word sequence, the large language model being configured in a server, and the terminal being in communication connection with the server.

9. A terminal, characterized by comprising: Including: A memory and one or more processors; The memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions; The one or more processors invoke the computer instructions to cause the terminal to perform the method of any one of claims 1-7.

10. A computer-readable storage medium comprising instructions, characterized in that: When the instructions run on the terminal, the terminal executes the method of any one of claims 1-7.

Images