Sign language recognition method, apparatus, and device

Abstract

The application discloses a sign language recognition method and device and equipment, and belongs to the technical field of artificial intelligence. The method comprises the following steps: in response to a first input of a user, a first sign language video is acquired; the first sign language video is input into a gesture understanding model, and gesture features of each video frame in the first sign language video are extracted; the gesture understanding model is trained based on a plurality of training samples, each training sample comprises a plurality of sign language video samples corresponding to a vocabulary sample; and a sign language recognition result is determined according to the gesture features.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a sign language recognition method, device, and equipment. Background Technology

[0002] Sign language is the primary language for communication among the hearing impaired. It is a visual-spatial language with its own independent grammar and vocabulary. With the development of artificial intelligence and computer vision technologies, the research and application of sign language learning are undergoing revolutionary changes. In recent years, the development of deep learning and multimodal large-scale models has made automatic sign language recognition and understanding technologies increasingly mature. However, current sign language recognition accuracy may decrease when considering different users' gesture styles, lighting conditions, background environments, shooting angles, and individual hand shape differences, resulting in poor generalization ability. Summary of the Invention

[0003] The purpose of this application is to provide a sign language recognition method, apparatus, and device that can solve the problem of poor generalization ability of sign language recognition.

[0004] In a first aspect, embodiments of this application provide a sign language recognition method, the method comprising: Responding to the user's first input, obtain the first sign language video; The first sign language video is input into the gesture understanding model, and the gesture understanding model extracts the gesture features of each video frame in the first sign language video; the gesture understanding model is trained based on multiple training samples, and each training sample includes multiple sign language video samples corresponding to a word sample; The sign language recognition result is determined based on the gesture characteristics.

[0005] Secondly, embodiments of this application provide a sign language recognition device, the device comprising: The acquisition module is used to acquire the first sign language video in response to the user's first input; The feature extraction module is used to input the first sign language video into the gesture understanding model and extract the gesture features of each video frame in the first sign language video through the gesture understanding model. The gesture understanding model is trained based on multiple training samples, and each training sample includes multiple sign language video samples corresponding to a word sample. The determination module is used to determine the sign language recognition result based on gesture features.

[0006] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0007] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0008] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0009] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0010] In this embodiment of the application, a first sign language video can be acquired in response to a user's first input; the first sign language video is input into a gesture understanding model, and gesture features of each video frame in the first sign language video are extracted by the gesture understanding model; wherein, the gesture understanding model is trained based on multiple training samples, each training sample including multiple sign language video samples corresponding to a word sample; and the sign language recognition result is determined based on the gesture features.

[0011] In this way, the gesture understanding model can process the first sign language video to obtain the gesture features corresponding to the first sign language video, thereby determining the sign language recognition results under sign language learning or sign language translation modes. Since this sign language understanding model is trained using multiple sign language video samples of the same word, it helps the sign language understanding model learn the common features of the same word under different expressions, thereby enhancing the adaptability of the sign language understanding model to individual user differences and changes in shooting conditions, and improving the generalization of sign language recognition. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating some embodiments of the sign language recognition method provided in this application; Figure 2 This is a schematic diagram of the model training of the gesture understanding model in the sign language recognition method provided in some embodiments of this application; Figure 3 This is a schematic diagram illustrating the functional principle of the gesture understanding module in the sign language recognition method provided in some embodiments of this application; Figure 4 This is a schematic diagram illustrating the overall functional principle of the gesture understanding model in the sign language recognition method provided in some embodiments of this application; Figure 5 This is a schematic flowchart of a scenario embodiment of the sign language recognition method provided in some embodiments of this application; Figure 6This is a schematic flowchart of another scenario embodiment of the sign language recognition method provided by some embodiments of this application; Figure 7 These are schematic diagrams of the structure of a sign language recognition device provided in some embodiments of this application; Figure 8 These are schematic diagrams of the structure of an electronic device provided in some embodiments of this application; Figure 9 These are schematic diagrams of the hardware structure of electronic devices provided in some embodiments of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0014] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0015] Sign language is the primary language for communication among the hearing impaired. It is a visual-spatial language with its own independent grammar and vocabulary. With the development of artificial intelligence and computer vision technologies, the research and application of sign language learning are undergoing revolutionary changes. In recent years, the development of deep learning and multimodal large-scale models has made automatic sign language recognition and understanding technologies increasingly mature.

[0016] Existing large-model-based sign language recognition algorithms all rely on key points of the person or raw video features to feed into existing large models, such as MT5's Encode-Decode architecture translation model or the Qwen model. These algorithms have the following drawbacks: The current multimodal vision component directly uses video information as input. Sign language videos vary depending on the shooting and acquisition background, and different background information has a significant impact on feature extraction. This results in good performance on open source data, but poor generalization. The performance is affected by different user scenarios, leading to poor results. Secondly, sign language translation videos are very scarce. Unlike video multimodal large models, which can obtain large amounts of data through crawling, sign language data, especially national standard data, is basically directly modeled using SFT when the amount of data is small. This method relies on a large amount of sign language data.

[0017] Furthermore, currently, word recognition and sentence recognition are separated into two tasks. Word recognition is usually achieved through classification, while sign language translation of the whole sentence is achieved through multimodal models or methods. However, in reality, the information of words and sentences is interconnected, and there is no mutual assistance in utilizing information technically.

[0018] Based on this, this application proposes a sign language recognition method, apparatus, and device. Combining the recognition and understanding capabilities of current multimodal large models, it constructs a general expression paradigm of sign language features on sign language data with limited data volume, extracts effective common features, and implements a unified architecture to realize a sign language learning and sign language translation system.

[0019] First, the technical terms that may be involved in the embodiments of this application will be explained: Sign language learning: Mastering the use of gestures, facial expressions, and body language to communicate with hearing-impaired people requires practicing finger spelling, vocabulary, and grammar.

[0020] Gesture translation: Using computer vision and artificial intelligence technology, it analyzes and recognizes sign language movements, facial expressions and body language, and translates sign language gestures into text or speech to help hearing-impaired people communicate without barriers.

[0021] Feature alignment: A technique that matches data features from different modalities or sources spatially or semantically to ensure consistency and improve model performance.

[0022] Sign language vocabulary: Independent units of meaning formed by gestures, facial expressions and body movements. In sign language, one sign language movement unit usually represents one word.

[0023] Sign language sentences: Complete expressions formed by combining sign language words according to grammatical rules.

[0024] The sign language recognition method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0025] Figure 1 This is a flowchart illustrating the sign language recognition method provided in an embodiment of this application. The sign language recognition method may include: Step 101: In response to the user's first input, obtain the first sign language video.

[0026] In step 101, the user's first input can be either an input that triggers sign language learning or an input that triggers sign language translation. For example, the application interface of a sign language app may include sign language learning function controls and sign language translation function controls. Users can enter the corresponding function page by clicking on the sign language learning function control or the sign language translation function control. Users can record sign language videos based on the relevant prompts on the function page, thereby obtaining the first sign language video.

[0027] The first sign language video can be the original sign language video recorded by the user, or it can be a sign language video after processing the original sign language video. For example, the first sign language video can be a video in which video frames unrelated to sign language have been removed from the original sign language video.

[0028] Step 102: Input the first sign language video into the gesture understanding model, and extract the gesture features of each video frame in the first sign language video through the gesture understanding model; wherein, the gesture understanding model is trained based on multiple training samples, and each training sample includes multiple sign language video samples corresponding to a word sample.

[0029] In step 102, a gesture understanding model can be pre-trained. This model may include modules such as a gesture understanding module, a temporal attention module, a multilayer perception module, and a large language module. It is understood that the large language module can stand alone as a complete large language model.

[0030] In some embodiments, the training method for the gesture understanding model may include the following steps: Obtain multiple training samples; Construct an initial model; the loss function of the initial model includes a consistency loss function and a cross-entropy loss function. The consistency loss function is used to characterize the consistency of gesture features of multiple sign language video samples corresponding to the same word sample, and the cross-entropy loss function is used to characterize the differences in gesture features of sign language video samples corresponding to different word samples. The initial model is trained using multiple training samples until the initial model converges based on the loss function, thus obtaining the gesture understanding model.

[0031] In this embodiment, multiple training samples can be acquired first. Each training sample can include multiple sign language video samples corresponding to a single word sample. For example, based on the vocabulary of the national standard sign language, there are more than 8,200 words in total. Each word can be collected by 50 different people, which means that each word sample has 50 sign language video samples and corresponding sign language annotation results, such as "you", "weather", "good", "yes", etc., and the sign language video samples of a single word sample are aligned.

[0032] Training of the gesture understanding model as follows Figure 2As shown, the number of samples in a single training run can be set to an even number, and they are fed into the initial model in the form of word pairs.

[0033] like Figure 2 As shown, taking a sample size of 2 as an example, the sign language video samples corresponding to two identical word samples can be fed frame by frame into the gesture understanding module 201 to obtain the gesture feature sample 2011 for each video frame, with a size of F×M, where F is the number of effective video frames. These are then input into the temporal attention module 202 and the multilayer perception module 203, respectively. The temporal attention module 202 is a commonly used module in video feature extraction, and it understands the temporal feature sample 2021 through a Transformer Encoder layer. The multilayer perception module 203 consists of two fully connected (Linear) layers that align visual features to text features, obtaining text feature samples 2031, which is an essential part of the multilayer perception module 203. After mapping to the text feature sample 2031, it is fed into the trained large language module 204. The predicted text 2041 output by the large language module 204 is compared with the text encoding 2042 corresponding to the word samples for loss calculation and constraint to obtain the weight parameters corresponding to the gesture understanding model. Based on these weight parameters, the initial model is adjusted to obtain the trained gesture understanding model.

[0034] The loss functions include consistency loss and cross-entropy loss. Consistency loss characterizes the consistency of gesture features among multiple sign language video samples corresponding to the same vocabulary sample, allowing the gesture understanding model to automatically capture common features of different people performing the same actions. For example... Figure 2 As shown, the expression for the consistency loss function is given by formula (1): in, The loss value is the consistency loss function. This refers to the gesture feature sample of the i-th video frame of a sign language video sample corresponding to two identical word samples.

[0035] The cross-entropy loss function can be used to characterize the differences in gesture features between sign language video samples corresponding to different word samples. For example... Figure 2 As shown, the expression for the cross-entropy loss function is given by formula (2): in, The loss value is the cross-entropy loss function. These are sign language video samples corresponding to two different vocabulary samples.

[0036] Thus, during the model training phase, a combination of consistency loss and cross-entropy loss is used to train the gesture understanding model. Consistency loss forces the gesture understanding model to extract similar features from different video samples of the same word, enhancing its inclusiveness towards different representations of the same concept. Cross-entropy loss, on the other hand, prompts the model to distinguish features from different words, fundamentally improving the generalization and recognition capabilities of the gesture understanding model.

[0037] The first sign language video can be input into a gesture understanding model, which can then extract gesture features from each frame of the video. In essence, the gesture understanding module within the gesture understanding model can extract gesture features from each frame of the first sign language video.

[0038] In some embodiments, such as Figure 3 As shown, the gesture understanding module 201 may include a Graph Convolutional Network (GCN) submodule 301, a Convolutional Neural Network (CNN) submodule 302, and a Feature Concatenation submodule 303; the gesture understanding module extracts gesture features from each video frame of the first sign language video, which may include: Image processing is performed on the first video frame to obtain a hand image, hand key point information, facial key point information, and body key point information; the first video frame is any video frame in the first sign language video; The hand image is input into a convolutional neural network submodule to obtain visual features; Input the key point information of the hands, the key point information of the face, and the key point information of the body into the graph convolutional network submodule to obtain the first key point feature corresponding to the key point information of the hands, the second key point feature corresponding to the key point information of the face, and the third key point feature corresponding to the key point information of the body. Visual features, first keypoint features, second keypoint features, and third keypoint features are input into the feature fusion submodule to obtain the gesture features of the first video frame.

[0039] In this embodiment, for any video frame in the first sign language video, i.e. the first video frame, image processing can be performed on the first video frame to extract hand images, including left hand images and right hand images, and the key point information can be split into hand key point information, facial key point information and body key point information, wherein the hand key point information includes left hand key point information and right hand key point information.

[0040] A hand image can be input into the convolutional neural network submodule 302 to obtain visual features. The convolutional neural network submodule 302 is the visual extraction part. After resizing the RGB hand image to 128×128, it is followed by convolutional layers, pooling layers, and fully connected layers to obtain a one-dimensional vector of visual features.

[0041] Hand, facial, and body keypoint information can be input into the graph convolutional network submodule 301 to obtain the first keypoint features corresponding to the hand keypoint information, the second keypoint features corresponding to the facial keypoint information, and the third keypoint features corresponding to the body keypoint information. The graph convolutional network submodule 301 can accept keypoint feature matrices from different parts as input, aggregate neighbor information through an adjacency matrix, and then connect a non-linear activation layer to obtain the corresponding keypoint features.

[0042] Subsequently, the visual features, first keypoint features, second keypoint features, and third keypoint features can be input into the feature fusion submodule 303 to obtain the gesture features of the first video frame. Specifically, the feature fusion submodule 303 can perform linear mapping on the visual features and keypoint features to 128 dimensions, then concatenate them to 256 dimensions, and then perform a linear mapping to 128 dimensions to finally obtain the fused gesture features.

[0043] In some examples, the gesture understanding module 201 may also include a concat encoding submodule 304. Before inputting the visual features and the first keypoint features into the feature fusion submodule 303, the visual features and the first keypoint features may be fed into the concat encoding submodule 304 together. After fusing the visual and keypoint features of the hand to obtain the hand fused features, these features are then input into the feature fusion submodule 303 with the second and third keypoint features to obtain the gesture features of the first video frame.

[0044] In this way, by integrating visual features with multiple key point features, a more comprehensive gesture representation is provided, which can capture the visual appearance information of the gesture and the motion information of the skeletal joints. This enables the model to understand the gesture from different dimensions, enhances the model's adaptability to changes in gesture details and different user body characteristics, and thus improves generalization.

[0045] Step 103: Determine the sign language recognition result based on the gesture features.

[0046] In step 103, if the first input is the input that triggers sign language learning, the sign language recognition result may include the consistency score between the first sign language video and the reference sign language video.

[0047] For example, in a sign language learning scenario, the gesture features of each video frame can be compared one by one with the reference gesture features of each reference video frame of a reference sign language video to determine whether the first sign language video input by the user is consistent with the reference sign language video, thereby obtaining a consistency score result. This consistency score result can be used to characterize whether the user's learning outcome is correct or incorrect.

[0048] If the first input is the input that triggers sign language translation, the sign language recognition result can include the translation result corresponding to the first sign language video.

[0049] For example, in a sign language translation scenario, the temporal information of the gesture features in each video frame can be combined and converted into a text feature vector. This allows the large language module to understand the contextual semantics and obtain an accurate sign language translation result corresponding to the first sign language video.

[0050] In some examples, once the sign language recognition result is determined, it can be displayed. For instance, for a consistency score, a message like "All correct, very standard!" or "Incorrect, practice again!" can be displayed. Another example is the translation result, which can directly display the semantic text corresponding to the first sign language video. In some examples, audio information can also be output simultaneously.

[0051] In this embodiment, the sign language recognition method can respond to a user's first input by acquiring a first sign language video; inputting the first sign language video into a gesture understanding model, and extracting gesture features of each video frame in the first sign language video through the gesture understanding model; wherein, the gesture understanding model is trained based on multiple training samples, each training sample including multiple sign language video samples corresponding to a word sample; and determining the sign language recognition result based on the gesture features.

[0052] In this way, the gesture understanding model can process the first sign language video to obtain the gesture features corresponding to the first sign language video, thereby determining the sign language recognition results under sign language learning or sign language translation modes. Since this sign language understanding model is trained using multiple sign language video samples of the same word, it helps the sign language understanding model learn the common features of the same word under different expressions, thereby enhancing the adaptability of the sign language understanding model to individual user differences and changes in shooting conditions, and improving the generalization of sign language recognition.

[0053] In some embodiments, acquiring a first sign language video in response to a user's first input may include: In response to the user's first input, retrieve the second sign language video; The second sign language video is input into the key point extraction model to obtain the key point information of each video frame of the second sign language video; Based on key information, determine the start and end video frames of the sign language; The first sign language video is determined from the second sign language video based on the start and end video frames of the sign language.

[0054] In this embodiment, as mentioned above, the function page can provide relevant prompts for recording sign language videos. For example, following the corresponding action prompts and at a fixed shooting distance, the user places their upper body within the detection frame of the function page to ensure the video can capture the complete sign language movement correctly, and then performs the corresponding sign language gesture. The sign language application can automatically start recording sign language videos when it detects that the user's upper body is within the detection frame. The original video recorded at this time is the second sign language video.

[0055] It is understandable that users may not start making sign language gestures immediately. That is, the second sign language video may include not only video frames related to sign language gestures, but also some interfering video frames unrelated to sign language gestures.

[0056] Based on this, the second sign language video can be input into a keypoint extraction model to obtain keypoint information for each frame of the video. Furthermore, based on this keypoint information, the start and end times of the sign language can be determined.

[0057] For example, by feeding the second sign language video frame by frame into an open-source keypoint extraction model, taking MMPose keypoint extraction as an example, the full-body keypoints can be extracted frame by frame, including keypoint information of the body, head and hands. The keypoint information can include the position information of a total of 133 keypoints.

[0058] Based on this key information, the start and end video frames of sign language can be determined. For example, the start of sign language can be considered when the position of the hand key point exceeds the position of the hip key point, and the end of sign language can be considered when the position of the hand key point is lower than the position of the hip key point, thus determining the start and end video frames of sign language.

[0059] The first sign language video can be determined from the second sign language video based on the start and end frames of the sign language video. That is, the starting frame of the first sign language video is the start frame, and the last frame is the end frame.

[0060] In this way, by automatically detecting the start and end frames of sign language actions through the key point extraction model, sign language videos can be cropped, reducing interference from irrelevant frames and making the data input to the gesture understanding model cleaner and more focused. This helps improve the model's efficiency in extracting effective gesture temporal features, indirectly enhancing the model's robustness in handling sign language videos of different lengths and starting positions in real-world scenarios.

[0061] In some embodiments, determining the sign language start video frame and the sign language end video frame based on key point information may include: If the height value of the hand key point in a video frame that indicates a continuous first threshold is higher than the height value of the hip key point, the first video frame in which the height value of the hand key point is higher than the height value of the hip key point is determined as the sign language start video frame. If the height value of the hand key point in a video frame that indicates a second consecutive threshold is lower than the height value of the hip key point, the first video frame in which the height value of the hand key point is lower than the height value of the hip key point is determined as the sign language ending video frame.

[0062] In this embodiment, as mentioned above, the sign language can be judged based on the key point information of the human body, divided into left and right hands. The average height of the left and right hands is used to determine whether the sign language exceeds the average height of the hip bone, thus determining the start and end of the sign language. Taking 91-112 key points for the left hand, 113-133 key points for the right hand, and 11-12 key points for the hip bone as an example, the judgment formula can be as shown in formula (3): in, Let avg represent the height value of the i-th key point.

[0063] If the height value of the hand key point in a video frame that indicates a continuous first threshold is higher than the height value of the hip key point, the first video frame in which the height value of the hand key point is higher than the height value of the hip key point can be determined as the sign language start video frame. If the height value of the hand key point in a video frame that indicates a second consecutive threshold is lower than the height value of the hip key point, the first video frame in which the height value of the hand key point is lower than the height value of the hip key point is determined as the sign language ending video frame.

[0064] The first and second thresholds can be set according to actual needs, and no specific limitation is made here. For example, if both the first and second thresholds are 10, when the start of the action is detected and the hand position is higher than the hip bone for 10 consecutive frames, it means that sign language has started. When the end of the action is detected and the hand position is lower than the hip bone for 10 consecutive frames, it means that sign language has ended.

[0065] In this way, based on the relative height relationship between the key points of the hand and the key points of the hip, and combined with the continuous frame threshold to determine the start and end points of sign language, it has a certain adaptability to changes in the amplitude of user gestures, reduces the error caused by subjective judgment, and improves the accuracy of subsequent feature extraction.

[0066] In some embodiments, the gesture understanding model includes a gesture understanding module, wherein the gesture features are obtained by feature extraction of each video frame in the first sign language video by the gesture understanding module. If the first input is the one that triggers sign language learning, in response to the user's first input, the first sign language video is retrieved, which may include: In response to the user's first input, a reference sign language video is displayed; the reference sign language video includes sign language gestures for at least one word. In response to a second input to a reference sign language video, obtain a first sign language video; Determining sign language recognition results based on gesture features can include: The reference sign language video is input into the gesture understanding module, which then extracts the reference gesture features of each reference video frame from the reference sign language video. Based on gesture features and reference gesture features, the similarity of each video frame is determined; If the similarity of each video frame meets the preset similarity conditions, the consistency score between the first sign language video and the reference sign language video is determined as the sign language learning is correct. If the similarity of each video frame does not meet the preset similarity conditions, the consistency score result is determined to be a sign language learning error. The sign language recognition results include consistency scores.

[0067] In this embodiment, in the sign language learning scenario, users can randomly select or input a word from the sign language vocabulary to learn in the sign language learning function interface, or they can randomly select or input a sentence, that is, a sentence including multiple words to learn.

[0068] At this point, a reference sign language video can be displayed. This means that if the user is learning vocabulary, the reference sign language video can include the sign language gestures for one word. If the user is learning sentences, the reference sign language video can include the sign language gestures for multiple words.

[0069] The system can obtain a first sign language video in response to a second input of a reference sign language video. For example, after the reference sign language video finishes playing, the user can choose to practice following along with it. At this time, the user follows the corresponding action prompts and places their upper body within the detection frame at a fixed shooting distance to ensure that the video can capture the complete sign language movements correctly, and begins to practice. The client then obtains the video of the user's practice movements, thus obtaining the first sign language video.

[0070] The first sign language video can then be input into the gesture understanding module, which extracts the gesture features of each video frame. A reference sign language video can also be input into the gesture understanding module, which extracts the reference gesture features of each reference video frame. The specific implementation process is as described above and will not be repeated here.

[0071] For example, the first sign language video is fed into the gesture understanding model. Based on the word selected by the user, the input is FP frames, and the user's gesture feature P is obtained, with a size of FP×M. Similarly, the reference sign language video corresponding to the word is fed into the gesture understanding model to obtain the gesture feature G, with a size of FG×M, representing the corresponding frame number FG.

[0072] The frame rate of the first sign language video can be aligned with the frame rate of the reference sign language video. For example, a fixed number of video frames can be selected, or a frame rate can be used as a baseline to reduce or insert frames to achieve the same frame rate. The specific method is not limited here.

[0073] The similarity of each video frame can be determined based on gesture features and reference gesture features. The calculation formula is shown in formula (4): Where cos represents the cosine similarity. This represents the reference gesture features in the i-th frame of a reference sign language video. This represents the gesture features of the i-th frame in the first sign language video.

[0074] If the similarity of each video frame meets the preset similarity conditions, the consistency score result is determined to represent correct sign language learning; if the similarity of each video frame does not meet the preset similarity conditions, the consistency score result is determined to represent incorrect sign language learning.

[0075] For example, if the number of frames with similarity exceeding the similarity threshold meets a preset condition, such as the proportion of frames with similarity exceeding the similarity threshold being greater than or equal to 90%, the consistency score result can be determined as indicating correct sign language learning, and the message "All correct, very standard!" can be displayed.

[0076] If the number of frames with similarity exceeding the similarity threshold does not meet the preset conditions, such as the proportion of frames with similarity exceeding the similarity threshold being less than 90%, the consistency score result can be determined as a sign language learning error, and the prompt message "Error, practice again!" can be displayed.

[0077] For example, the average similarity score can be calculated. If the average is greater than or equal to a similarity threshold, the consistency score is considered to represent correct sign language learning. If the average is less than the similarity threshold, the consistency score is considered to represent incorrect sign language learning.

[0078] Specific preset similarity conditions can be set according to actual needs, and no specific limitations are made here.

[0079] In this way, in the sign language learning scenario, by comparing the feature similarity of the user's input sign language video with the reference sign language video frame by frame, a refined evaluation of the learning effect is achieved. It can not only give an overall consistency score, but also make comparisons at the frame level, which helps users understand the subtle differences in gestures and improves the sign language learning experience.

[0080] In some embodiments, when the similarity corresponding to each video frame meets a preset similarity condition, determining the consistency score between the first sign language video and the reference sign language video as a representation of correct sign language learning may include: The average similarity of each video frame is used as the consistency score between the first sign language video and the reference sign language video. If the consistency score is greater than or equal to the score threshold, the consistency score result is determined to represent correct sign language learning; If the similarity of each video frame does not meet the preset similarity criteria, the consistency score result is identified as a sign language learning error, which may include: If the consistency score is less than the scoring threshold, the consistency score result is identified as a sign language learning error.

[0081] In this embodiment, the average similarity of each video frame can be used as the consistency score between the first sign language video and the reference sign language video. The formula for calculating the consistency score is shown in formula (5): in, The cosine similarity score represents the consistency score. This represents the reference gesture features in the i-th frame of a reference sign language video. This represents the gesture features of the i-th frame in the first sign language video.

[0082] If the consistency score is greater than or equal to the scoring threshold α, the consistency score result is determined to represent correct sign language learning. If the consistency score is less than the scoring threshold α, the consistency score result is determined to represent incorrect sign language learning.

[0083] In this way, by calculating the average similarity as a consistency score and comparing the score with a threshold to determine the correctness of learning, the method is simple, effective, easy for users to understand, and the threshold setting can be adjusted according to actual needs, thus balancing the rigor and flexibility of the evaluation.

[0084] In some embodiments, when the number of words corresponding to the sign language gestures included in the reference sign language video is multiple, the method may further include: If the consistency score is less than the score threshold, the first sign language video is divided into multiple video segments according to the preset frame extraction ratio. The consistency score for multiple video segments is determined based on the average similarity of the video frames in each video segment. The erroneous words are identified based on the video segment with the minimum consistency score among multiple video segments; the sign language recognition results also include the sign language gestures for the erroneous words.

[0085] In this embodiment, if the user is learning sentences, and the consistency score is less than the score threshold, the first sign language video can be divided into multiple video segments according to a preset frame extraction ratio.

[0086] The preset frame skipping ratio can be set according to needs. In some examples, the preset frame skipping ratio can be dynamically adjusted based on the number of words and the total duration of the reference sign language video. For example, with a preset frame skipping ratio of β=0.33, the first sign language video can be divided into three video segments at a ratio of one-third.

[0087] By calculating the average similarity of video frames within each video segment, a consistency score can be determined for multiple video segments, thus identifying the low-score intervals. The video segment corresponding to the minimum consistency score is the low-score interval, and specific incorrect words can be identified based on the sign language gestures corresponding to these low-score intervals.

[0088] In some examples, sign language recognition results may also include the sign language gestures for incorrect words. For instance, when the consistency score is below a threshold, the system can simultaneously display the user's action corresponding to the incorrect word and the standard sign language gesture, prompting the user to "relearn the following actions."

[0089] This prompts users to pay attention to the sign language techniques in the lower sections, allowing them to clearly see which movements are not standard enough. Users can then choose to "re-practice" or "go back," forming a closed loop in sign language learning.

[0090] In this way, when multiple words are learned incorrectly using gestures in succession, the system can accurately identify the specific word gestures that the user made mistakes by scoring them in segments and locating the smallest score segment. This provides targeted feedback to the user, improving the effectiveness of the learning system and the user experience.

[0091] In some embodiments, before determining the similarity corresponding to each video frame based on gesture features and reference gesture features, the method may further include: When the number of video frames in the first sign language video is less than the number of reference video frames in the reference sign language video, the gesture features of the video frames in the first sign language video are interpolated to make the number of frames corresponding to the gesture features equal to the number of frames corresponding to the reference gesture features. When the number of video frames in the first sign language video is greater than the number of reference video frames in the reference sign language video, the gesture features of the video frames in the first sign language video are uniformly sampled to make the number of frames corresponding to the gesture features equal to the number of frames corresponding to the reference gesture features.

[0092] In this embodiment, if the number of video frames FP of the first sign language video is not equal to the number of reference video frames FG of the reference sign language video, the gesture features of the video frames in the first sign language video can be sampled or expanded based on the number of reference video frames FG.

[0093] When FP is less than FG, the gesture features of the video frames in the first sign language video are interpolated to frame FG. When FP is greater than FG, the gesture features of the video frames in the first sign language video are uniformly sampled to frame FG. This ensures that the number of frames corresponding to the gesture features is equal to the number of frames corresponding to the reference gesture features.

[0094] In this way, by using interpolation or uniform sampling, the frame numbers of the two video sequences to be compared are aligned, which solves the problem of mismatch in temporal length caused by different user gesture speeds. This ensures the fairness and accuracy of similarity calculation, enables the model to better adapt to the differences in sign language rhythm among different users, and improves the generalization ability of the evaluation.

[0095] In some embodiments, the gesture understanding model includes a gesture understanding module, a temporal attention module, a multi-layer perception module, and a large language module, wherein the gesture features are obtained by the gesture understanding module through feature extraction of each video frame in the first sign language video; When the first input is the one that triggers sign language translation, the sign language recognition result is determined based on gesture features, and may include: The gesture features are input into the temporal attention module, which learns the temporal information of the gesture features to obtain the temporal feature vector. The temporal feature vector is input into the multilayer perceptron module, and the multilayer perceptron module maps the temporal feature vector to the text feature space to obtain the text feature vector; Input the text feature vector into the large language module, and the large language module outputs the translation result corresponding to the first sign language video; The sign language recognition results include translation results.

[0096] In this embodiment, in the sign language translation scenario, the user can adjust to a suitable distance and position so that the detection box on the sign language translation function interface can capture the complete upper body information, so that the user can make the corresponding sign language gestures, and the client can obtain the first sign language video.

[0097] Understandably, for sign language translation, the training of the gesture understanding model can be further based on open-source sign language video-sentence data pairs, as well as self-collected sign language video-sentence data pairs, with one sign language video corresponding to one sentence.

[0098] As mentioned in the model training section above, the cross-entropy loss can be calculated based on the predicted text output by the gesture understanding model and the standard sentence text. The parameters of the gesture understanding module 201 and the large language module 204 remain unchanged; only the temporal attention module 202 and the multilayer perception module 203 are trained. LoRa can also be added during the large model stage for fine-tuning.

[0099] like Figure 4 As shown, based on the trained gesture understanding model, the first sign language video 401 can be fed into the gesture understanding module 201 frame by frame to obtain the gesture features 402 of each video frame. Then, the gesture features 402 are input into the temporal attention module 202. The temporal attention module 202 learns the temporal information of the gesture features 402 to obtain the temporal feature vector 403.

[0100] The temporal feature vector 403 can be input into the multilayer perceptron module 203, and the multilayer perceptron module 203 can map the temporal feature vector 403 to the text feature space to obtain the text feature vector 404.

[0101] The text feature vector 404 can be input into the large language module 204, and the large language module 204 can combine the context semantics to decode it into the translation result 405 corresponding to the first sign language video.

[0102] In this way, in the context of sign language translation, a temporal attention module is introduced to capture the long-term dependencies of gesture features, which are then mapped to the text space through a multilayer perceptron. Finally, the large language module is used to generate natural language semantics, making full use of the powerful language generation and understanding capabilities of the large language module. It can handle complex and continuous sign language sentences and generate fluent and accurate text or speech results, significantly improving the generalized understanding of sign language semantics, especially context-related semantics.

[0103] To facilitate understanding of the sign language recognition method provided in the above embodiments, the following describes the sign language recognition method using two specific scenario embodiments. Figure 5 and Figure 6This is a schematic flowchart of a scenario embodiment of the sign language recognition method provided in this application.

[0104] like Figure 5 As shown, taking a sign language learning scenario as an example, this scenario implementation can specifically include the following steps: Step 501: Enter sign language learning mode and determine the learning object selected by the user, which is vocabulary or sentences; Step 502: Display the reference sign language video corresponding to the learning target; Step 503: Obtain the practice video corresponding to the reference sign language video; Step 504: Extract key information from the practice video and determine the start and end video frames of the sign language based on the key information to obtain the first sign language video; Step 505: Input the first sign language video and the reference sign language video into the trained gesture understanding model to obtain gesture features and reference gesture features; Step 506: Compare the gesture features with the reference gesture features to obtain the consistency score result; Step 507: Display of consistency score results.

[0105] This scenario embodiment proposes a sign language learning method based on a gesture understanding model. The gesture understanding model learns the characteristics of different people performing the same actions. Therefore, in the learning of sign language vocabulary or sign language sentences, it can output a consistency score result of the user's sign language learning based on standard actions, and provide the user with real-time feedback information.

[0106] like Figure 6 As shown, taking a sign language interpretation scenario as an example, this scenario implementation can specifically include the following steps: Step 601: Enter sign language translation mode and obtain the initial sign language video input by the user; Step 602: Extract key point information from the initial sign language video, and determine the start and end video frames of the sign language based on the key point information to obtain the first sign language video; Step 603: Input the first sign language video into the trained gesture understanding model to obtain the translation result corresponding to the first sign language video; Step 604: Translation results displayed.

[0107] This scenario proposes a sign language translation method based on a gesture understanding model. The gesture understanding model learns the characteristics of different people making the same movements, which lays the foundation for the sign language task in terms of visual feature expression. Based on the sign language understanding model, fine-tuning experiments can be carried out on sentence data to achieve the sign language translation task end-to-end.

[0108] The sign language recognition method provided in this application can be executed by a sign language recognition device. This application uses a sign language recognition device to perform the sign language recognition method as an example to illustrate the sign language recognition device provided in this application.

[0109] like Figure 7 As shown, the sign language recognition device 700 provided in this application embodiment may include: Acquisition unit 701 is used to acquire a first sign language video in response to the user's first input; The feature extraction unit 702 is used to input the first sign language video into the gesture understanding model and extract the gesture features of each video frame in the first sign language video through the gesture understanding model; wherein, the gesture understanding model is trained based on multiple training samples, and each training sample includes multiple sign language video samples corresponding to a word sample. The determining unit 703 is used to determine the sign language recognition result based on the gesture features.

[0110] In this way, the gesture understanding model can process the first sign language video to obtain the gesture features corresponding to the first sign language video, thereby determining the sign language recognition results under sign language learning or sign language translation modes. Since this sign language understanding model is trained using multiple sign language video samples of the same word, it helps the sign language understanding model learn the common features of the same word under different expressions, thereby enhancing the adaptability of the sign language understanding model to individual user differences and changes in shooting conditions, and improving the generalization of sign language recognition.

[0111] In some embodiments, the obtaining unit 701 may include: The acquisition sub-unit is used to acquire the second sign language video in response to the user's first input; The extraction sub-unit is used to input the second sign language video into the key point extraction model to obtain the key point information of each video frame in the second sign language video. The first determining subunit is used to determine the sign language start video frame and the sign language end video frame based on key point information; The second determining subunit is used to determine the first sign language video from the second sign language video based on the sign language start video frame and the sign language end video frame.

[0112] In this way, by automatically detecting the start and end frames of sign language actions through the key point extraction model, sign language videos can be cropped, reducing interference from irrelevant frames and making the data input to the gesture understanding model cleaner and more focused. This helps improve the model's efficiency in extracting effective gesture temporal features, indirectly enhancing the model's robustness in handling sign language videos of different lengths and starting positions in real-world scenarios.

[0113] In some embodiments, the gesture understanding model includes a gesture understanding module, which includes a graph convolutional network submodule, a convolutional neural network submodule, and a feature fusion submodule. The feature extraction unit 702 can be specifically used for: Image processing is performed on the first video frame to obtain a hand image, hand key point information, facial key point information, and body key point information; the first video frame is any video frame in the first sign language video; The hand image is input into a convolutional neural network submodule to obtain visual features; Input the key point information of the hands, the key point information of the face, and the key point information of the body into the graph convolutional network submodule to obtain the first key point feature corresponding to the key point information of the hands, the second key point feature corresponding to the key point information of the face, and the third key point feature corresponding to the key point information of the body. Visual features, first keypoint features, second keypoint features, and third keypoint features are input into the feature fusion submodule to obtain the gesture features of the first video frame.

[0114] In this way, by integrating visual features with multiple key point features, a more comprehensive gesture representation is provided, which can capture the visual appearance information of the gesture and the motion information of the skeletal joints. This enables the model to understand the gesture from different dimensions, enhances the model's adaptability to changes in gesture details and different user body characteristics, and thus improves generalization.

[0115] In some embodiments, the gesture understanding model includes a gesture understanding module, wherein the gesture features are obtained by feature extraction of each video frame in the first sign language video by the gesture understanding module. When the first input is the one that triggers sign language learning, Unit 701 can be specifically used for: In response to the user's first input, a reference sign language video is displayed; the reference sign language video includes sign language gestures for at least one word. In response to a second input to a reference sign language video, obtain a first sign language video; Specifically, unit 703 can be used for: The reference sign language video is input into the gesture understanding module, which then extracts the reference gesture features of each reference video frame from the reference sign language video. Based on gesture features and reference gesture features, the similarity of each video frame is determined; If the similarity of each video frame meets the preset similarity conditions, the consistency score between the first sign language video and the reference sign language video is determined as the sign language learning is correct. If the similarity of each video frame does not meet the preset similarity conditions, the consistency score result is determined to be a sign language learning error. The sign language recognition results include consistency scores.

[0116] In this way, in the sign language learning scenario, by comparing the feature similarity of the user's input sign language video with the reference sign language video frame by frame, a refined evaluation of the learning effect is achieved. It can not only give an overall consistency score, but also make comparisons at the frame level, which helps users understand the subtle differences in gestures and improves the sign language learning experience.

[0117] In some embodiments, the gesture understanding model includes a gesture understanding module, a temporal attention module, a multi-layer perception module, and a large language module, wherein the gesture features are obtained by the gesture understanding module through feature extraction of each video frame in the first sign language video; When the first input is the one that triggers sign language translation, the determination unit 703 can specifically be used for: The gesture features are input into the temporal attention module, which learns the temporal information of the gesture features to obtain the temporal feature vector. The temporal feature vector is input into the multilayer perceptron module, and the multilayer perceptron module maps the temporal feature vector to the text feature space to obtain the text feature vector; Input the text feature vector into the large language module, and the large language module outputs the translation result corresponding to the first sign language video; The sign language recognition results include translation results.

[0118] In this way, in the context of sign language translation, a temporal attention module is introduced to capture the long-term dependencies of gesture features, which are then mapped to the text space through a multilayer perceptron. Finally, the large language module is used to generate natural language semantics, making full use of the powerful language generation and understanding capabilities of the large language module. It can handle complex and continuous sign language sentences and generate fluent and accurate text or speech results, significantly improving the generalized understanding of sign language semantics, especially context-related semantics.

[0119] The sign language recognition device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0120] The sign language recognition device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0121] The sign language recognition device provided in this application embodiment can implement all the processes implemented in the method embodiment, and will not be described again here to avoid repetition.

[0122] Optionally, such as Figure 8 As shown, this application embodiment also provides an electronic device 800, including a processor 801 and a memory 802. The memory 802 stores a program or instructions that can run on the processor 801. When the program or instructions are executed by the processor 801, they implement the various steps of the above-described sign language recognition method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0123] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0124] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application.

[0125] The electronic device 900 includes, but is not limited to, components such as: radio frequency unit 901, network module 902, audio output unit 903, input unit 904, sensor 905, display unit 906, user input unit 907, interface unit 908, memory 909, and processor 910.

[0126] Those skilled in the art will understand that the electronic device 900 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 910 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 9 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0127] The processor 910 can be used for: Responding to the user's first input, obtain the first sign language video; The first sign language video is input into the gesture understanding model, and the gesture understanding model extracts the gesture features of each video frame in the first sign language video; the gesture understanding model is trained based on multiple training samples, and each training sample includes multiple sign language video samples corresponding to a word sample; The sign language recognition result is determined based on the gesture characteristics.

[0128] In this way, the gesture understanding model can process the first sign language video to obtain the gesture features corresponding to the first sign language video, thereby determining the sign language recognition results under sign language learning or sign language translation modes. Since this sign language understanding model is trained using multiple sign language video samples of the same word, it helps the sign language understanding model learn the common features of the same word under different expressions, thereby enhancing the adaptability of the sign language understanding model to individual user differences and changes in shooting conditions, and improving the generalization of sign language recognition.

[0129] In some embodiments, the processor 910 can also be used for: In response to the user's first input, retrieve the second sign language video; The second sign language video is input into the key point extraction model to obtain the key point information of each video frame of the second sign language video; Based on key information, determine the start and end video frames of the sign language; The first sign language video is determined from the second sign language video based on the start and end video frames of the sign language.

[0130] In this way, by automatically detecting the start and end frames of sign language actions through the key point extraction model, sign language videos can be cropped, reducing interference from irrelevant frames and making the data input to the gesture understanding model cleaner and more focused. This helps improve the model's efficiency in extracting effective gesture temporal features, indirectly enhancing the model's robustness in handling sign language videos of different lengths and starting positions in real-world scenarios.

[0131] In some embodiments, the gesture understanding model includes a gesture understanding module, which includes a graph convolutional network submodule, a convolutional neural network submodule, and a feature fusion submodule. The 910 processor can also be used for: Image processing is performed on the first video frame to obtain a hand image, hand key point information, facial key point information, and body key point information; the first video frame is any video frame in the first sign language video; The hand image is input into a convolutional neural network submodule to obtain visual features; Input the key point information of the hands, the key point information of the face, and the key point information of the body into the graph convolutional network submodule to obtain the first key point feature corresponding to the key point information of the hands, the second key point feature corresponding to the key point information of the face, and the third key point feature corresponding to the key point information of the body. Visual features, first keypoint features, second keypoint features, and third keypoint features are input into the feature fusion submodule to obtain the gesture features of the first video frame.

[0132] In this way, by integrating visual features with multiple key point features, a more comprehensive gesture representation is provided, which can capture the visual appearance information of the gesture and the motion information of the skeletal joints. This enables the model to understand the gesture from different dimensions, enhances the model's adaptability to changes in gesture details and different user body characteristics, and thus improves generalization.

[0133] In some embodiments, the gesture understanding model includes a gesture understanding module, wherein the gesture features are obtained by feature extraction of each video frame in the first sign language video by the gesture understanding module. The display unit 906 can be used for: In response to the user's first input, a reference sign language video is displayed; the reference sign language video includes sign language gestures for at least one word. The 910 processor can also be used for: In response to a second input to a reference sign language video, obtain a first sign language video; The reference sign language video is input into the gesture understanding module, which then extracts the reference gesture features of each reference video frame from the reference sign language video. Based on gesture features and reference gesture features, the similarity of each video frame is determined; If the similarity of each video frame meets the preset similarity conditions, the consistency score between the first sign language video and the reference sign language video is determined as the sign language learning is correct. If the similarity of each video frame does not meet the preset similarity conditions, the consistency score result is determined to be a sign language learning error. The sign language recognition results include consistency scores.

[0134] In this way, in the sign language learning scenario, by comparing the feature similarity of the user's input sign language video with the reference sign language video frame by frame, a refined evaluation of the learning effect is achieved. It can not only give an overall consistency score, but also make comparisons at the frame level, which helps users understand the subtle differences in gestures and improves the sign language learning experience.

[0135] In some embodiments, the gesture understanding model includes a gesture understanding module, a temporal attention module, a multi-layer perception module, and a large language module, wherein the gesture features are obtained by the gesture understanding module through feature extraction of each video frame in the first sign language video; The 910 processor can also be used for: The gesture features are input into the temporal attention module, which learns the temporal information of the gesture features to obtain the temporal feature vector. The temporal feature vector is input into the multilayer perceptron module, and the multilayer perceptron module maps the temporal feature vector to the text feature space to obtain the text feature vector; Input the text feature vector into the large language module, and the large language module outputs the translation result corresponding to the first sign language video; The sign language recognition results include translation results.

[0136] In this way, in the context of sign language translation, a temporal attention module is introduced to capture the long-term dependencies of gesture features, which are then mapped to the text space through a multilayer perceptron. Finally, the large language module is used to generate natural language semantics, making full use of the powerful language generation and understanding capabilities of the large language module. It can handle complex and continuous sign language sentences and generate fluent and accurate text or speech results, significantly improving the generalized understanding of sign language semantics, especially context-related semantics.

[0137] It should be understood that, in this embodiment, the input unit 904 may include a graphics processing unit (GPU) 9041 and a microphone 9042. The GPU 9041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 906 may include a display panel 9061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 907 includes at least one of a touch panel 9071 and other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0138] The memory 909 can be used to store software programs and various data. The memory 909 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 909 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 909 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0139] Processor 910 may include one or more processing units; optionally, processor 910 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 910.

[0140] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described sign language recognition method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0141] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0142] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described sign language recognition method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0143] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0144] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described sign language recognition method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0145] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0146] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0147] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A sign language recognition method, characterized in that, The method includes: Responding to the user's first input, obtain the first sign language video; The first sign language video is input into the gesture understanding model, and the gesture features of each video frame in the first sign language video are extracted by the gesture understanding model; wherein, the gesture understanding model is trained based on multiple training samples, and each training sample includes multiple sign language video samples corresponding to a word sample; Based on the gesture features, the sign language recognition result is determined.

2. The method according to claim 1, characterized in that, The process of acquiring the first sign language video in response to the user's first input includes: In response to the user's first input, retrieve the second sign language video; The second sign language video is input into the key point extraction model to obtain the key point information of each video frame of the second sign language video; Based on the key information, determine the sign language start video frame and the sign language end video frame; The first sign language video is determined from the second sign language video based on the sign language start video frame and the sign language end video frame.

3. The method according to claim 1, characterized in that, The gesture understanding model includes a gesture understanding module, which includes a graph convolutional network submodule, a convolutional neural network submodule, and a feature fusion submodule. The step of inputting the first sign language video into the gesture understanding model and extracting the gesture features of each video frame in the first sign language video through the gesture understanding model includes: Image processing is performed on the first video frame to obtain a hand image, hand key point information, facial key point information, and body key point information; the first video frame is any one of the video frames in the first sign language video; The hand image is input into the convolutional neural network submodule to obtain visual features; The hand key point information, the facial key point information, and the body key point information are input into the graph convolutional network submodule to obtain the first key point feature corresponding to the hand key point information, the second key point feature corresponding to the facial key point information, and the third key point feature corresponding to the body key point information. The visual features, the first key point features, the second key point features, and the third key point features are input into the feature fusion submodule to obtain the gesture features of the first video frame.

4. The method according to claim 1, characterized in that, The gesture understanding model includes a gesture understanding module, wherein the gesture features are obtained by the gesture understanding module through feature extraction of each video frame in the first sign language video; When the first input is an input that triggers sign language learning, the step of responding to the user's first input and acquiring the first sign language video includes: In response to the user's first input, a reference sign language video is displayed; the reference sign language video includes sign language gestures for at least one word. In response to a second input to the reference sign language video, a first sign language video is acquired; The step of determining the sign language recognition result based on the gesture features includes: The reference sign language video is input into the gesture understanding module, and the gesture understanding module extracts the reference gesture features of each reference video frame in the reference sign language video. Based on the gesture features and the reference gesture features, the similarity of each video frame is determined; If the similarity of each video frame meets the preset similarity conditions, the consistency score between the first sign language video and the reference sign language video is determined to be a sign language learning success. If the similarity of each video frame does not meet the preset similarity condition, the consistency score result is determined to be a sign language learning error. The sign language recognition result includes the consistency score result.

5. The method according to claim 1, characterized in that, The gesture understanding model includes a gesture understanding module, a temporal attention module, a multi-layer perception module, and a large language module. The gesture features are obtained by the gesture understanding module through feature extraction of each video frame in the first sign language video. When the first input is an input that triggers sign language translation, determining the sign language recognition result based on the gesture features includes: The gesture features are input into the temporal attention module, and the temporal attention module learns the temporal information of the gesture features to obtain a temporal feature vector; The temporal feature vector is input into the multilayer perception module, and the multilayer perception module maps the temporal feature vector to the text feature space to obtain the text feature vector. The text feature vector is input into the large language module, and the translation result corresponding to the first sign language video is output through the large language module. The sign language recognition result includes the translation result.

6. A sign language recognition device, characterized in that, The device includes: The acquisition unit is used to acquire the first sign language video in response to the user's first input; The feature extraction unit is used to input the first sign language video into the gesture understanding model and extract the gesture features of each video frame in the first sign language video through the gesture understanding model; wherein, the gesture understanding model is trained based on multiple training samples, and each training sample includes multiple sign language video samples corresponding to a word sample; The determining unit is used to determine the sign language recognition result based on the gesture features.

7. The apparatus according to claim 6, characterized in that, The acquisition unit includes: The acquisition sub-unit is used to acquire the second sign language video in response to the user's first input; An extraction subunit is used to input the second sign language video into the key point extraction model to obtain key point information for each video frame in the second sign language video. The first determining subunit is used to determine the sign language start video frame and the sign language end video frame based on the key point information. The second determining subunit is used to determine the first sign language video from the second sign language video based on the sign language start video frame and the sign language end video frame.

8. The apparatus according to claim 6, characterized in that, The gesture understanding model includes a gesture understanding module, which includes a graph convolutional network submodule, a convolutional neural network submodule, and a feature fusion submodule. The feature extraction unit is specifically used for: Image processing is performed on the first video frame to obtain a hand image, hand key point information, facial key point information, and body key point information; the first video frame is any one of the video frames in the first sign language video; The hand image is input into the convolutional neural network submodule to obtain visual features; The hand key point information, the facial key point information, and the body key point information are input into the graph convolutional network submodule to obtain the first key point feature corresponding to the hand key point information, the second key point feature corresponding to the facial key point information, and the third key point feature corresponding to the body key point information. The visual features, the first key point features, the second key point features, and the third key point features are input into the feature fusion submodule to obtain the gesture features of the first video frame.

9. The apparatus according to claim 6, characterized in that, The gesture understanding model includes a gesture understanding module, wherein the gesture features are obtained by the gesture understanding module through feature extraction of each video frame in the first sign language video; When the first input is the input that triggers sign language learning, the acquisition unit is specifically used for: In response to the user's first input, a reference sign language video is displayed; the reference sign language video includes sign language gestures for at least one word. In response to a second input to the reference sign language video, a first sign language video is acquired; The determining unit is specifically used for: The reference sign language video is input into the gesture understanding module, and the gesture understanding module extracts the reference gesture features of each reference video frame in the reference sign language video. Based on the gesture features and the reference gesture features, the similarity of each video frame is determined; If the similarity of each video frame meets the preset similarity conditions, the consistency score between the first sign language video and the reference sign language video is determined to be a sign language learning success. If the similarity of each video frame does not meet the preset similarity condition, the consistency score result is determined to be a sign language learning error. The sign language recognition result includes the consistency score result.

10. The apparatus according to claim 6, characterized in that, The gesture understanding model includes a gesture understanding module, a temporal attention module, a multi-layer perception module, and a large language module. The gesture features are obtained by the gesture understanding module through feature extraction of each video frame in the first sign language video. When the first input is the input that triggers sign language translation, the determining unit is specifically used for: The gesture features are input into the temporal attention module, and the temporal attention module learns the temporal information of the gesture features to obtain a temporal feature vector; The temporal feature vector is input into the multilayer perception module, and the multilayer perception module maps the temporal feature vector to the text feature space to obtain the text feature vector. The text feature vector is input into the large language module, and the translation result corresponding to the first sign language video is output through the large language module. The sign language recognition result includes the translation result.

11. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the method as described in any one of claims 1-5.

Images