A sign language recognition method and related apparatus, device, and storage medium

Abstract

The application discloses a sign language recognition method and related device, equipment and storage medium. The sign language recognition method comprises the following steps: acquiring a sentence video frame sequence, the sentence video frame sequence is obtained by collecting a sign language action sequence, and the content expressed by the sign language action in the sentence video frame sequence is a sentence; dividing the sentence video frame sequence according to word segmentation to obtain a plurality of word video frame sequences, the content expressed by the sign language action in the sentence video frame sequence is a word; performing action recognition on each word video frame sequence to obtain a word corresponding to each word video frame sequence; and obtaining a sentence corresponding to the sentence video frame sequence by using the word corresponding to each word video frame sequence. The above scheme can improve the communication efficiency of video calls.

Description

Technical Field

[0001] This application relates to the field of video recognition, and in particular to a sign language recognition method and related devices, equipment, and storage media. Background Technology

[0002] In current video call scenarios, text-to-speech conversion is commonly used for communication among hearing-impaired individuals. However, this method does not improve communication efficiency. Furthermore, the immediacy of video calls leads users to have high expectations for efficiency. During video calls, many users do not actively wait for the hearing-impaired person to type, resulting in poor communication effectiveness and low user satisfaction. Summary of the Invention

[0003] This application provides at least one sign language recognition method and related devices, equipment, and storage media, which can improve the efficiency of video call communication.

[0004] The first aspect of this application provides a sign language recognition method, which includes: acquiring a sentence-segmented video frame sequence, wherein the sentence-segmented video frame sequence is obtained by collecting a sign language action sequence, and the sign language action in the sentence-segmented video frame sequence expresses a sentence; dividing the sentence-segmented video frame sequence according to word segmentation to obtain several word-segmented video frame sequences, wherein the sign language action in the word-segmented video frame sequence expresses a word; performing action recognition on each word-segmented video frame sequence to obtain the word corresponding to each word-segmented video frame sequence; and using the word corresponding to each word-segmented video frame sequence to obtain the sentence corresponding to the sentence-segmented video frame sequence.

[0005] The process involves performing action recognition on each segmented video frame sequence to obtain the corresponding word segments, including: extracting the segmented semantic features of each segmented video frame sequence; finding the preset semantic features that match the segmented semantic features from a number of preset semantic features; and using the word segments corresponding to the found preset semantic features as the word segments corresponding to the segmented video frame sequence.

[0006] The method involves identifying preset semantic features that match the word segmentation semantic features from a set of preset semantic features. This includes: obtaining the similarity between the word segmentation semantic features and each preset semantic feature; selecting the preset semantic feature with the highest similarity as the preset semantic feature that matches the word segmentation semantic features; and / or, the word segmentation semantic features are extracted using a visual encoder. The method further includes: acquiring a set of sample word segmentation video frame sequences, which include a first sample video frame sequence, at least one second sample video frame sequence, and at least one third sample video frame sequence. The first and second sample video frame sequences represent different sign language demonstrators performing sign language on the same word segmentation. The video frame sequences are divided into two sets: the first sample video frame sequence and the third sample video frame sequence, which are video frame sequences of sign language actions performed on different word segments. The visual encoder is used to extract features from the sample word segmentation video frame sequences to obtain the sample semantic features corresponding to the sample word segmentation video frame sequences. The feature extraction loss is obtained by using the first similarity between the first sample video frame sequence and the second sample video frame sequence and the second similarity between the first sample video frame sequence and the third sample video frame sequence regarding the sample semantic features. The parameters of the visual encoder are adjusted using the feature extraction loss. At least one preset semantic feature is the sample semantic feature corresponding to the sample word segmentation video frame sequence.

[0007] The process involves dividing the sentence-segment video frame sequence into several word-segment video frame sequences, including: identifying several word-interval video frames that separate two sign language actions from the sentence-segment video frame sequence; and for each pair of adjacent word-interval video frames, obtaining the video frames between the two adjacent word-interval video frames from the sentence-segment video frame sequence as the word-segment video frame sequence corresponding to the two adjacent word-interval video frames.

[0008] The process involves identifying a number of word-interval video frames that separate two sign language actions from a sequence of segmented video frames. This includes: acquiring a first video frame sequence corresponding to each video frame in the segmented video frame sequence; performing word interval recognition on the first video frame sequence corresponding to the video frame to obtain the word interval recognition result of the video frame; wherein the first video frame sequence corresponding to the video frame includes the video frame and a first number of video frames following the video frame; and determining the video frame as a word-interval video frame in response to the word interval recognition result of the video frame meeting the preset word interval requirements.

[0009] The word interval recognition result is obtained by using a word interval recognizer to identify word intervals; and / or, the word interval recognition result of the video frame is the word interval score of the video frame, and the preset word interval requirement is that the word interval score of the video frame is greater than a preset threshold.

[0010] The process of acquiring the segmented video frame sequence includes: for each acquired second video frame sequence, performing motion state recognition on the second video frame sequence to obtain a state recognition result for the second video frame sequence. The state recognition result is used to determine whether there is a sign language action state in the second video frame sequence. The second video frame sequence includes a second number of consecutive video frames acquired from the sign language action sequence, and adjacent second video frame sequences have some identical video frames. Based on the state recognition result, it is determined whether the second video frame sequence is a sign language start sequence or a sign language end sequence. The second video frame sequences between the sign language start sequence, the sign language end sequence, and the sign language start sequence and the sign language end sequence are combined to obtain the segmented video frame sequence.

[0011] Wherein, the state recognition result is obtained by using a motion recognizer to identify the motion state; and / or, determining whether the second video frame sequence is a sign language start sequence or a sign language end sequence based on the state recognition result includes: determining whether the second video frame sequence is in a sign language action state or a non-sign language action state based on the state recognition result; determining the second video frame sequence as a sign language start sequence in response to the second video frame sequence being in a sign language action state and the preceding second video frame sequence being in a non-sign language action state; determining the second video frame sequence as a sign language end sequence in response to the second video frame sequence being in a non-sign language action state and the preceding second video frame sequence being in a sign language action state; and / or, the state recognition result is a sign language action state score, and before determining whether the second video frame sequence is a sign language start sequence or a sign language end sequence based on the state recognition result, the method further includes: converting the sign language action state score into a state identifier, wherein the state identifier is a first identifier representing a sign language action state or a second identifier representing a non-sign language action state.

[0012] Specifically, the process of obtaining sentences corresponding to the segmented video frame sequence by using the segmented words corresponding to each segmented video frame sequence includes: correcting the segmented word sequence composed of the segmented words corresponding to each segmented video frame sequence to obtain the corrected segmented word sequence; and using the corrected segmented word sequence to obtain the sentences corresponding to the segmented video frame sequence.

[0013] The process of correcting the segmented sequence composed of the segments corresponding to each segmented video frame sequence includes: correcting the segmented sequence using an error corrector; and / or, generating an initial sentence using the segmented sequence, displaying the initial sentence, receiving correction information for the initial sentence input by the user, and correcting the segmented sequence based on the correction information.

[0014] The sentence-segmented video frame sequence is acquired during a video call. After obtaining the sentences corresponding to the sentence-segmented video frame sequence by utilizing the words corresponding to each segmented video frame sequence, the process further includes: sending the sentences corresponding to the sentence-segmented video frame sequence to the other end in the video call; and / or, before acquiring the sentence-segmented video frame sequence, the process further includes: identifying the people in the currently acquired video frame to obtain the people's position information in the video frame; determining whether the people in the video frame meet the sign language recognition requirements based on the position information; and prompting the user to adjust their position in response to the people in the video frame not meeting the sign language recognition requirements.

[0015] The second aspect of this application provides a sign language recognition device, comprising: an acquisition module for acquiring a sequence of segmented video frames, wherein the sequence of segmented video frames is obtained by capturing a sequence of sign language actions, and the sign language actions in the sequence of segmented video frames express a sentence; a word segmentation module for dividing the sequence of segmented video frames according to word segmentation to obtain several word segmented video frame sequences, wherein the sign language actions in the sequence of segmented video frames express a word; a recognition module for performing action recognition on each word segmented video frame sequence to obtain the word corresponding to each word segmented video frame sequence; and an encoding module for using the word corresponding to each word segmented video frame sequence to obtain the sentence corresponding to the sequence of segmented video frames.

[0016] A third aspect of this application provides an electronic device including a memory and a processor coupled to each other, the processor being used to execute program instructions stored in the memory to implement the sign language recognition method of the first aspect described above.

[0017] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the sign language recognition method of the first aspect described above.

[0018] The above scheme obtains a sequence of video frames segmented from sign language action sequences. It then divides these segmented video frames into multiple segmented video frame sequences based on word segmentation. Sign language actions within each segmented video frame sequence are identified to obtain the corresponding words. By using the corresponding words, the sentence within each segmented video frame sequence is determined, thus identifying the content expressed by the sign language actions. This approach improves the efficiency of sign language action recognition and consequently enhances the communication efficiency of sign language videos.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0021] Figure 1 This is a flowchart illustrating an embodiment of the sign language recognition method of this application;

[0022] Figure 2 This is a flowchart illustrating another embodiment of the sign language recognition method of this application;

[0023] Figure 3 This is a schematic diagram of the framework of an embodiment of the sign language recognition device of this application;

[0024] Figure 4 This is a schematic diagram of the framework of an embodiment of the electronic device of this application;

[0025] Figure 5 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0026] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0027] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0028] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0029] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of a sign language recognition method. Specifically, it may include the following steps:

[0030] Step S110: Obtain the segmented video frame sequence.

[0031] Among them, the sentence-segment video frame sequence is obtained by collecting sign language action sequences, and the content expressed by the sign language actions in the sentence-segment video frame sequence is a sentence.

[0032] The sign language recognition method presented in this paper can be applied to the recognition of sign language actions in video data. The video data to be recognized can be storable video data, such as sign language instruction videos, or real-time recorded video data, such as video calls. The sign language action sequence is the corresponding sign language action video data, and the sentence-by-sentence video frame sequence is the sentence-by-sentence video data divided from the sign language action video data by a preset number of frames.

[0033] In some embodiments, sign language actions in stored sign language teaching videos are identified. First, the sign language teaching video is divided into multiple sentence-based video frame sequences. The content expressed by the sign language actions in each sentence-based video frame sequence is identified to obtain the corresponding sentence. When the sign language teaching video is played, when a sentence-based video frame sequence is reached, the corresponding sentence appears as a subtitle in the sign language teaching video for the user's learning and reference.

[0034] In other embodiments, when video calls with hearing-impaired individuals require real-time recognition of sign language gestures in the video, video stream data captured by a standard RGB (Red Green Blue) camera can be used as a sequence of segmented video frames. The method for obtaining the segmented video frame sequence may include steps S111 to S113, as detailed below:

[0035] Step S111: For each second video frame sequence acquired, motion state recognition is performed on the second video frame sequence to obtain the state recognition result of the second video frame sequence.

[0036] The state recognition result is used to determine whether there is a sign language action state in the second video frame sequence. The second video frame sequence includes a second number of consecutive video frames obtained by collecting sign language action sequences, and there are partially identical video frames in adjacent second video frame sequences.

[0037] In some embodiments, during video calls with hearing-impaired individuals, identification can be performed based on a second video frame sequence that generates video stream data during the video call. This second video frame sequence comprises a second consecutive number of video frames. Specifically, a motion recognizer is used to perform motion state recognition on the second video frame sequence, yielding a state recognition result. When using the motion recognizer to identify a second video frame sequence, a second consecutive number of video frames from the second video frame sequence are input into the motion recognizer according to a sliding window mechanism. The step size of the sliding window mechanism is the length of the second video frame sequence. For example, the second video frame sequence may include five consecutive video frames. The motion recognizer identifies each video frame and outputs a sign language action state score as the state recognition result for that second video frame sequence. Optionally, the sign language action state score is between 0 and 1, and is used as the state recognition result for the video frame.

[0038] When the motion recognizer outputs the sign language action state score of the second video frame sequence and assigns the sign language action state score to the first video frame of the second video frame sequence, a new video frame needs to be input into the motion recognizer according to the sliding window mechanism to form a second video frame sequence adjacent to the first video frame sequence. After each video frame in the second video frame sequence is assigned a sign language action state score, it is smoothed to prevent abnormal sign language action state scores from affecting the state recognition result of the second video frame sequence. Smoothing is performed once for each subsequent video frame assigned a sign language action state score. Alternatively, smoothing can be performed once for each video frame with a assigned sign language action state score that is part of the second video frame sequence. The sign language action state scores of the smoothed video frames are then thresholded so that the sign language action state score of each video frame is 0 or 1. A state identifier is generated based on the sign language action state score of the second video frame sequence. The state identifier is either a first identifier representing a sign language action state or a second identifier representing a non-sign language action state. For example, if the sign language action state score of the second video frame sequence is 1, then the second video frame sequence can be marked with a first identifier; if the sign language action state score of the second video frame sequence is 0, then the second video frame sequence can be marked with a second identifier. In other embodiments, a motion capture device is used to feature-label the human actions in each second number of video frames of the second video frame sequence. The labeled second number of video frames are then input into an analysis module for action state analysis to obtain the state recognition result of the second video frame sequence, which is the sign language action state score. The motion capture device is a device capable of extracting features from human actions. By combining the human action features of each second number of video frames, the sign language action state score of the second video frame sequence is obtained, which serves as the state recognition result of the second video frame sequence. The sign language action state score of the second video frame sequence is converted into a state identifier, which is either a first identifier representing a sign language action state or a second identifier representing a non-sign language action state. A high score indicates that the current second video frame sequence is in a sign language action state, while a low score indicates that the current second video frame sequence is in a non-sign language action state. It is understandable that the method for comprehensively processing the sign language action state scores of each second number of video frames can be averaging, weighted summation, mean filtering, etc., and no specific limitation is made here.

[0039] Step S112: Determine whether the second video frame sequence is a sign language start sequence or a sign language end sequence based on the state recognition result.

[0040] In some embodiments, each video frame in the second video frame sequence represents the same sign language action or non-sign language action state, and the second video frame sequence can be regarded as a sign language start sequence or a sign language end sequence. The state recognition results based on each second number of video frames are used to determine whether the second video frame sequence is in a sign language action state or a non-sign language action state; if the second video frame sequence is in a sign language action state and the preceding second video frame sequence is in a non-sign language action state, then the second video frame sequence can be determined as a sign language start sequence; if the second video frame sequence is in a non-sign language action state and the preceding second video frame sequence is in a sign language action state, then the second video frame sequence can be determined as a sign language end sequence.

[0041] In other embodiments, the second video frame sequence may include a sign language start video frame or a sign language end video frame, with the sign language start video frame serving as a sign language start signal and the sign language end video frame serving as a sign language end signal. Specifically, if the state recognition result of a video frame in the second video frame sequence is a non-sign language action state, and the state recognition result of the previous video frame is a sign language action state, then that video frame is a sign language end video frame, i.e., a sign language end signal, and the second video frame sequence is considered a sign language end sequence. If the state recognition result of a video frame in the second video frame sequence is a sign language action state, and the state recognition result of the previous video frame is a non-sign language action state, then that video frame is a sign language start video frame, i.e., a sign language start signal, and the second video frame sequence is considered a sign language start sequence.

[0042] Step S113: Combine the sign language start sequence, the sign language end sequence, and the second video frame sequence between the sign language start sequence and the sign language end sequence to obtain the sentence-segmented video frame sequence.

[0043] In some embodiments, a sentence-segmented video frame sequence can be obtained based on a predetermined sign language start sequence and sign language end sequence. Specifically, all second video frame sequences between adjacent sign language start sequences and sign language end sequences can be used as valid video frame sequences for the sentence-segmented video frame sequence, i.e., action state video frame sequences. These sequences are then combined with partial sequences from the sign language start and end sequences to obtain the sentence-segmented video frame sequence. It is understood that whether or not to combine the sign language start and end sequences into the sentence-segmented video frame sequence depends on the specific circumstances and is not specifically limited here.

[0044] In another embodiment, the second video frame sequence may include a sign language start video frame or a sign language end video frame. The second video frame sequence including the sign language start video frame is considered the sign language start sequence, and the second video frame sequence including the sign language end video frame is considered the sign language end sequence. All second video frame sequences between the sign language start sequence and the sign language end sequence are considered valid video frame sequences. Combining partial sequences from the sign language start sequence and the sign language end sequence with the valid video frame sequences yields a sentence-segmented video frame sequence.

[0045] Step S120: Divide the segmented video frame sequence according to word segmentation to obtain several segmented video frame sequences.

[0046] In some embodiments, to improve the accuracy of sign language recognition in sentence-segmented video frame sequences and thus achieve understanding of the content expressed by the sign language actions, the sentence-segmented video frame sequences can be divided into several word-segmented video frame sequences. In some embodiments, step S120 may specifically include the contents of steps S121 to S122, as follows:

[0047] Step S121: From the segmented video frame sequence, find several word-interval video frames that separate two sign language actions.

[0048] In some embodiments, the segmented video frame sequence is identified using a sliding window mechanism. For each segmented video frame sequence, a first video frame sequence corresponding to a video frame is acquired and input into the sliding window for identification. The length of the first video frame sequence is the step size of the sliding window. Word spacing identification is performed on the first video frame sequence corresponding to each video frame to obtain the word spacing identification result for each video frame. The first video frame sequence corresponding to a video frame includes the video frame itself and a first number of video frames following it. If the word spacing identification result of a video frame meets a preset word spacing requirement, the video frame is determined to be a word-spaced video frame.

[0049] In one specific embodiment, by recognizing sign language actions in a video frame, a word interval recognition result corresponding to the video frame is obtained. The word interval recognition result of the video frame is converted into a word interval score of the video frame. If the word interval score of the video frame is greater than a preset threshold, the video frame is determined to be a word interval video frame. The preset threshold can be set empirically or calculated, and is not specifically limited here.

[0050] In another specific embodiment, a word spacing recognizer can be used for word spacing identification. According to a sliding window, the first video frame sequence is input to the word spacing recognizer for identification, and the word spacing score of each video frame in the first video frame sequence is output. Optionally, the word spacing score is between 0 and 1, and the word spacing score is marked on the first frame of the video frame as the word spacing identification result of the video frame. For example, the video frame is 0.2s of video stream data, and the first video frame sequence includes 5 video frames. If the word spacing score of a video frame exceeds a preset threshold of 0.85, then the video frame is considered a word-spaced video frame.

[0051] Step S122: For each pair of adjacent word interval video frames, obtain the video frames between the two adjacent word interval video frames from the sentence video frame sequence, and use them as the word segmentation video frame sequence corresponding to the two adjacent word interval video frames.

[0052] In some embodiments, based on the video frames between two adjacent word intervals, video frames between these two word intervals are cropped from the sentence video frame sequence to form a word segmentation video frame sequence.

[0053] Step S130: Perform action recognition on each segmented video frame sequence to obtain the corresponding word segmentation for each segmented video frame sequence.

[0054] In some embodiments, the specific implementation of recognizing sign language actions in a segmented video frame sequence to obtain the corresponding sign language word segmentation may include the contents of steps S131 to S133, as follows:

[0055] Step S131: For each segmented video frame sequence, extract the segmented semantic features of the segmented video frame sequence.

[0056] In some embodiments, each segmented video frame sequence can be input into a trained visual encoder to extract the segmented semantic features of the segmented video frame sequences. The training method for the visual encoder is as follows: Several sample segmented video frame sequences are acquired. These sequences include a first sample video frame sequence, at least one second sample video frame sequence, and at least one third sample video frame sequence. The first and second sample video frame sequences are video frame sequences of different sign language demonstrators performing sign language actions on the same segmented word. The first and third sample video frame sequences are video frame sequences of sign language actions on different segmented words. After acquiring these sample segmented video frame sequences, the visual encoder is used to extract features from each sequence to obtain the corresponding sample semantic features. A feature extraction loss is obtained using the first similarity between the first and second sample video frame sequences regarding their semantic features, and the second similarity between the first and third sample video frame sequences regarding their semantic features. The parameters of the visual encoder are then adjusted using the feature extraction loss. At least one preset semantic feature is the sample semantic feature corresponding to the segmented video frame sequence. Specifically, the visual encoder can be trained using the Triple-Loss method.

[0057] Step S132: From a number of preset semantic features, find the preset semantic features that match the word segmentation semantic features.

[0058] In some embodiments, the preset semantic features are the semantic features of each sign language segmentation marker. The similarity between each segmentation semantic feature and each preset semantic feature is obtained. The preset semantic feature with the highest similarity is taken as the preset semantic feature that matches the segmentation semantic feature. Specifically, Euclidean distance can be used to calculate the similarity between each segmentation semantic feature and each preset semantic feature. It is understood that in other embodiments, clustering or other methods can also be used to determine the similarity between semantic features. There are many methods for calculating the similarity between each segmentation semantic feature and each preset semantic feature, and no specific limitation is made here.

[0059] Step S133: Use the word segmentation corresponding to the preset semantic features found as the word segmentation corresponding to the segmented video frame sequence.

[0060] In some embodiments, the word segment that corresponds to the preset semantic feature with the highest similarity to the segmented video frame sequence can be used as its corresponding word segment.

[0061] Step S140: Using the word segments corresponding to each segmented video frame sequence, obtain the sentences corresponding to the sentence segmented video frame sequence.

[0062] In some embodiments, the segmentation sequence composed of the segments corresponding to each segmented video frame sequence is corrected to obtain a corrected segmentation sequence. Then, the corrected segmentation sequence is used to obtain the sentences corresponding to the sentence segmentation video frame sequence.

[0063] In one specific embodiment, an error corrector can be used to correct the word segmentation sequence. The error corrector checks each word in the word segmentation sequence to verify whether the words corresponding to the segmented video frame sequence are correct and acceptable. Furthermore, it adjusts the word order in the word segmentation sequence so that the corrected word segmentation sequence becomes a complete and readable sentence.

[0064] In another specific embodiment, to improve the accuracy of sign language gesture recognition, the word segmentation sequence after correction by the error corrector can be further corrected. The specific steps are as follows: generate initial sentences using the word segmentation sequence, display the initial sentences, receive correction information for the initial sentences input by the user, and correct the word segmentation sequence based on the correction information.

[0065] In another specific embodiment, the segmented video frame sequence is acquired during the video call, and the segmented video frame sequence corresponding to the final corrected segment is sent to the other end in the video call.

[0066] In another specific embodiment, the sentences corresponding to the translated and confirmed video frame sequence can be converted into voice data and sent to the other end in the video call, so that the two parties in the video can communicate by voice, thereby improving communication efficiency.

[0067] In some embodiments, when recognizing sign language gestures during a video call, the person in the currently captured video frame is identified to obtain the person's position information within the video frame; based on the position information, it is determined whether the person in the video frame meets the sign language recognition requirements. If the person in the video frame does not meet the sign language recognition requirements, the user is prompted to adjust their position.

[0068] In a specific application scenario, when a hearing-impaired user makes a video call with a hearing person using a smart terminal, a sign language recognition system is used to identify the sign language gestures of the hearing-impaired user during the video call. This sign language recognition system includes a motion recognizer, a word segmentation interval recognizer, a visual encoder, an error corrector, a translator, and a manual error correction module. At the start of the video call, the front-facing RGB camera of the smart terminal device typically captures the video stream data of the sign language gestures. Since the video stream data generated during the video call is dynamic, a sliding window mechanism can be used to control the input and output of the video stream data to the motion recognizer. The sliding window mechanism has a window width of 0.2 and a step size of 1. According to the sliding window mechanism, a consecutive 0.2s video frame sequence from the acquired video stream data is used as the second video frame sequence and input into the motion recognizer. The motion recognizer identifies whether there are sign language gesture states in the second video frame sequence, thereby determining whether the second video frame sequence is a sign language start sequence or a sign language end sequence.

[0069] The motion recognizer performs sign language motion recognition on the input second video frame sequence, which includes the first to fifth video frames. It outputs a sign language motion state score for this second video frame sequence, with a value between 0 and 1. This score is then assigned to the first video frame of the second video frame sequence as the state recognition result. Next, a sixth video frame is input into the motion recognizer, and the second to sixth video frames are combined to form the next adjacent second video frame sequence. This process is repeated to complete the sign language motion recognition of the video stream data. After all video frames in the second video frame sequence have been assigned a sign language motion state score, a mean filter is applied to these scores. This mean filter is repeated for each subsequent sign language motion state score until the video call ends, and the final output is thresholded to 0 or 1. The sign language action state score of the second video frame sequence can be used to identify its state. If the sign language action state score of the second video frame sequence is 0, then the second video frame sequence is in a non-sign language action state; if the sign language action state score of the second video frame sequence is 1, then the second video frame sequence is in a sign language action state. Furthermore, if the second video frame sequence is in a sign language action state and the preceding second video frame sequence is in a non-sign language action state, then the second video frame sequence can be determined as a sign language start sequence; if the second video frame sequence is in a non-sign language action state and the preceding second video frame sequence is in a sign language action state, then the second video frame sequence can be determined as a sign language end sequence. The sign language start sequence, the sign language end sequence, and the second video frame sequences between the sign language start sequence and the sign language end sequence are combined to obtain a sentence-segmented video frame sequence, which is then input into the word segmentation interval recognizer.

[0070] The word segmentation interval recognizer predicts word interval video frames frame by frame from the input sentence-segmented video frame sequence, i.e., whether the current video frame is a segmentation of a complete word. Similarly, a sliding window mechanism is used to control the video stream data input to and output to the word segmentation interval recognizer. The word segmentation interval recognizer performs word segmentation interval recognition on a first video frame sequence in the sentence-segmented video frame sequence, which consists of 0.2 seconds of continuous video frame data. The word segmentation interval recognizer outputs the word interval score of each video frame in the first video frame sequence as the word interval recognition result of the video frame data, with a value between 0 and 1. The word interval score is marked on the first frame of the video frame data. If the word interval score of a video frame is greater than a preset threshold of 0.9, then the video frame is a word interval video frame. The word segmentation interval recognizer obtains the video frames between two adjacent word interval video frames from the sentence-segmented video frame sequence as the word segmented video frame sequence corresponding to the two adjacent word interval video frames, and cuts the recognized word segmented video frame sequence from the corresponding sentence-segmented video frame sequence and sends it to the visual encoder for recognition.

[0071] Based on the obtained segmented video frame sequence, the visual encoder extracts the corresponding segmented semantic features. Then, it calculates the Euclidean distance with all preset semantic features in the word encoding feature table F. After a Softmax operation, all Euclidean distances are normalized to scores between 0 and 1. The data after the Softmax operation is then processed using an Argmax operation to output the preset semantic feature with the highest similarity. The word corresponding to this preset semantic feature is identified according to the word encoding feature table F and used as the corresponding word in the segmented video frame sequence. This process completes the recognition of all segmented video frame sequences in the sentence segmentation video frame sequence. The visual encoder is trained using Triple-Loss. The specific training steps are as follows: Several sample segmented video frame sequences are obtained, including a first sample video frame sequence, at least one second sample video frame sequence, and at least one third sample video frame sequence. The first sample video frame sequence is A, and the second sample video frame sequence is a sign language demonstration sequence where a different sign language demonstrator performs sign language actions for the same segmented word. The third sample video frame sequence B, which performs sign language actions with different word segmentation from the first sample video frame sequence, is formed into a triplet, and supervised training is performed using the following loss:

[0072]

[0073] Where d is the feature similarity measure, which can be calculated using methods such as Euclidean distance, and margin is a constant greater than 0.

[0074] After training is completed, all trained word segments are processed by the visual encoder, and the corresponding encoding features of the word segments are saved in the word segmentation encoding feature table F for subsequent table lookup.

[0075] After identifying the sign language ending sequence using a motion recognizer, a complete sentence-segmented video frame sequence is obtained, and word segmentation and recognition are performed on this sequence. The corresponding word segmentation sequence identified in the sentence-segmented video frame sequence is input into a pre-trained Transformer-based error corrector. Based on the sign language word distribution learned from big data, the word segmentation sequence is updated, thus completing the necessary corrections.

[0076] The corrected word segmentation sequence is input into a Transformer-based translator, which translates the word segmentation sequence into initial sentences and displays them on the user interface for the hearing impaired, prompting them to send the message.

[0077] Hearing-impaired users can open the word segmentation sequence corrected by the error corrector based on the translation results, manually correct individual results, receive correction information for the initial sentence segmentation, correct the word segmentation sequence based on the correction information, and then click "complete." The system will then call the translator again to translate. After the hearing-impaired user confirms that the translation result is correct, they can click "send" to send the content to the recipient's client.

[0078] The system also features an automatic word segmentation table update function. During the correction process described above, as well as the manual correction process, the corresponding word segmentation feature codes will be appended with the updated word segments to form new "word-feature" pairs. These pairs will be added to the word segmentation coding feature table F, thus completing the update of the word segmentation coding feature table and further improving the system's prediction accuracy.

[0079] In addition, the system integrates pose recognition to identify the user's position within the frame, providing necessary prompts to ensure the integrity of the sign language gestures captured by the front-facing camera. Specifically, it identifies the user's current position frame by frame, combining spatial coordinate point position thresholds and the proportion of the user's frame within the image to determine if the user's current position is the optimal recognition location. When an unsuitable user position is detected, a dashed (red) outline of the user's image is displayed in the video frame to prompt the user and help them adjust their position promptly. When the user's position and distance are confirmed to meet the requirements of the sign language recognition algorithm, the dashed line turns green and disappears after a 1-second delay, indicating to the user that the current position meets the optimal recognition requirements.

[0080] To ensure that the sign language information collected throughout the process meets the recognition and analysis standards, the user pose detection module will always be running. If the distance between the user and the camera becomes too short or too large, the user will be prompted to adjust their position and re-enter the sign language content.

[0081] In this embodiment, the sign language recognition system is simple and convenient to operate, using video stream data captured by a common RGB camera as input. It does not rely on specific imaging equipment or wearable hardware, allowing for rapid deployment in common smart terminal devices such as mobile phones and tablets. Furthermore, it does not require conversion into human keypoint data for recognition, and is unaffected by the performance of keypoint detection models. Moreover, the system boasts fast sign language recognition speed. It maintains a word-feature correspondence word segmentation encoding feature table trained with large datasets. During forward inference, it outputs the word segment with the highest similarity score based on the similarity score between the semantic features corresponding to the segmented video frame sequence and the word segmentation encoding feature table. Compared to traditional methods using convolutional neural networks and sequence models, this significantly reduces computational load and resource requirements, ensuring high efficiency and real-time performance. In addition, the system's recognition results are accurate and efficient. The system employs a multi-level error correction mechanism. On one hand, it uses automatic error correction to incorporate large-scale statistical information for post-processing of the results. On the other hand, it introduces a human-computer coupled manual error correction mechanism through user-selected operations to further optimize the recognition results, greatly improving translation accuracy. Meanwhile, the error correction records can self-iterate and be used to update the expanded full-word encoding feature table F, continuously improving the translation accuracy of the system, and the whole process requires no training.

[0082] Please see Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the sign language recognition method of this application. The specific steps are as follows:

[0083] Step S210: Obtain video frame data.

[0084] In some embodiments, video frame data may be video stream data captured in real time during a video call, or stored video data.

[0085] Step S220: Perform sign language action recognition on the video frame data to obtain a segmented video frame sequence.

[0086] In some embodiments, the acquired video frame data is directly input into the word segmentation interval recognizer, which uses the word segmentation interval recognizer to find several word-interval video frames that belong to the separation between two sign language actions. The specific steps can be referred to step S120 above, and will not be described in detail here.

[0087] Step S230: Perform sign language action recognition on each segmented video frame sequence to obtain the corresponding word segmentation for each segmented video frame sequence.

[0088] This step is the same as step S140 above, and will not be described in detail here.

[0089] Step S240: Perform sign language recognition on the video frames to obtain segmented video frames.

[0090] In some embodiments, while identifying word-interval video frames in the video stream data, sentence-interval video frames are also identified. The specific steps can be referred to in step S110 above, and will not be elaborated further here.

[0091] Step S250: Obtain sentences by using the word segments corresponding to the word segmentation video frame sequences between adjacent sentence interval video frames.

[0092] In some embodiments, upon identifying the initial sentence-interval video frame, sign language recognition is performed on subsequent video frame data to obtain a sequence of segmented video frames. The corresponding words are then identified within this sequence. Subsequently, upon identifying an adjacent sentence-interval video frame, a sequence of segmented video frames between this initial and previous sentence-interval video frames can be determined, forming a sentence-based video frame sequence. The corresponding word sequences of these segmented video frame sequences can be combined to form sentences. After obtaining the segmented sequence, it can be corrected and translated, and then provided to the user.

[0093] Please see Figure 3 , Figure 3 This is a schematic diagram of the framework of an embodiment of the sign language recognition device of this application. The sign language recognition device includes: an acquisition module 310, a word segmentation module 320, a recognition module 330, and an encoding module 340. The acquisition module 310 is used to acquire a sequence of segmented video frames, which is obtained by capturing a sequence of sign language actions. The sign language actions in the segmented video frame sequence represent a sentence. The word segmentation module 320 is used to divide the segmented video frame sequence according to words, obtaining several segmented video frame sequences. The sign language actions in the segmented video frame sequences represent a word. The recognition module 330 is used to perform action recognition on each segmented video frame sequence to obtain the corresponding word. The encoding module 340 is used to use the corresponding word in each segmented video frame sequence to obtain the sentence corresponding to the segmented video frame sequence.

[0094] In some embodiments, the recognition module 330 performs action recognition on each segmented video frame sequence to obtain the segmentation corresponding to each segmented video frame sequence, including: for each segmented video frame sequence, extracting the segmentation semantic features of the segmented video frame sequence; finding a preset semantic feature that matches the segmentation semantic feature from a number of preset semantic features; and using the segmentation corresponding to the found preset semantic feature as the segmentation corresponding to the segmented video frame sequence.

[0095] In some embodiments, the recognition module 330 performs the task of finding a preset semantic feature that matches the word segmentation semantic feature from a plurality of preset semantic features, including: obtaining the similarity between the word segmentation semantic feature and each preset semantic feature respectively; taking the preset semantic feature with the highest similarity as the preset semantic feature that matches the word segmentation semantic feature; and / or, the word segmentation semantic feature is extracted using a visual encoder, and the method further includes: obtaining a plurality of sample word segmentation video frame sequences, the plurality of sample word segmentation video frame sequences including a first sample video frame sequence, at least one second sample video frame sequence and at least one third sample video frame sequence, the first sample video frame sequence and the second sample video frame sequence being different sign language demonstrators for the same The video frame sequence for word segmentation and sign language actions, the first sample video frame sequence and the third sample video frame sequence are video frame sequences for sign language actions of different word segments; the visual encoder is used to extract features from the sample word segmentation video frame sequences to obtain the sample semantic features corresponding to the sample word segmentation video frame sequences; the feature extraction loss is obtained by using the first similarity between the first sample video frame sequence and the second sample video frame sequence with respect to the sample semantic features, and the second similarity between the first sample video frame sequence and the third sample video frame sequence with respect to the sample semantic features; the parameters of the visual encoder are adjusted using the feature extraction loss; wherein, at least one preset semantic feature is the sample semantic feature corresponding to the sample word segmentation video frame sequence.

[0096] In some embodiments, the word segmentation module 320 performs word segmentation on the sentence video frame sequence to obtain a plurality of word segmentation video frame sequences, including: finding a plurality of word-interval video frames that belong to the separation between two sign language actions from the sentence video frame sequence; for each pair of adjacent word-interval video frames, obtaining the video frames between the two adjacent word-interval video frames from the sentence video frame sequence as the word segmentation video frame sequence corresponding to the two adjacent word-interval video frames.

[0097] In some embodiments, the word segmentation module 320 performs the following steps: finding a number of word-interval video frames that separate two sign language actions from the segmented video frame sequence. This includes: acquiring a first video frame sequence corresponding to each video frame in the segmented video frame sequence; performing word interval recognition on the first video frame sequence corresponding to the video frame to obtain a word interval recognition result for the video frame; wherein the first video frame sequence corresponding to the video frame includes the video frame and a first number of video frames following the video frame; and determining the video frame as a word-interval video frame in response to the word interval recognition result of the video frame satisfying a preset word interval requirement.

[0098] In some embodiments, the word segmentation module 320 performs word interval recognition by using a word interval recognizer; and / or, the word interval recognition result of the video frame is the word interval score of the video frame, and the preset word interval requirement is that the word interval score of the video frame is greater than a preset threshold.

[0099] In some embodiments, the acquisition module 310 performs the acquisition of a segmented video frame sequence, including: for each acquired second video frame sequence, performing motion state recognition on the second video frame sequence to obtain a state recognition result of the second video frame sequence, the state recognition result being used to determine whether there is a sign language action state in the second video frame sequence, the second video frame sequence including a second number of consecutive video frames obtained by acquiring a sign language action sequence, and adjacent second video frame sequences having partially identical video frames; determining whether the second video frame sequence is a sign language start sequence or a sign language end sequence based on the state recognition result; and combining the sign language start sequence, the sign language end sequence, and the second video frame sequences between the sign language start sequence and the sign language end sequence to obtain a segmented video frame sequence.

[0100] In some embodiments, the acquisition module 310 performs state recognition results to obtain motion state recognition using a motion recognizer; and / or, determines whether the second video frame sequence is a sign language start sequence or a sign language end sequence based on the state recognition results, including: determining whether the second video frame sequence is in a sign language action state or a non-sign language action state based on the state recognition results; determining the second video frame sequence as a sign language start sequence in response to the second video frame sequence being in a sign language action state and the preceding second video frame sequence being in a non-sign language action state; determining the second video frame sequence as a sign language end sequence in response to the second video frame sequence being in a non-sign language action state and the preceding second video frame sequence being in a sign language action state; and / or, the state recognition result is a sign language action state score, and before determining whether the second video frame sequence is a sign language start sequence or a sign language end sequence based on the state recognition results, the module further includes: converting the sign language action state score into a state identifier, wherein the state identifier is a first identifier representing a sign language action state or a second identifier representing a non-sign language action state.

[0101] In some embodiments, the encoding module 340 performs word segmentation based on the word segmentation corresponding to each word segmentation video frame sequence to obtain sentences corresponding to the sentence segmentation video frame sequence, including: correcting the word segmentation sequence composed of the word segmentation corresponding to each word segmentation video frame sequence to obtain a corrected word segmentation sequence; and using the corrected word segmentation sequence to obtain sentences corresponding to the sentence segmentation video frame sequence.

[0102] In some embodiments, the encoding module 340 performs correction of the word segmentation sequence composed of words corresponding to each word segmentation video frame sequence, including: correcting the word segmentation sequence using an error corrector; and / or, generating an initial sentence using the word segmentation sequence, displaying the initial sentence, receiving correction information for the initial sentence input by the user, and correcting the word segmentation sequence based on the correction information.

[0103] In some embodiments, the encoding module 340 executes the segmented video frame sequence, which is acquired during a video call. After obtaining the sentences corresponding to the segmented video frame sequence by utilizing the words corresponding to each segmented video frame sequence, the method further includes: sending the sentences corresponding to the segmented video frame sequence to the other end in the video call; and / or, before acquiring the segmented video frame sequence, the method further includes: identifying the people in the currently acquired video frame to obtain the position information of the people in the video frame; determining whether the people in the video frame meet the sign language recognition requirements based on the position information; and prompting the user to adjust the position in response to the people in the video frame not meeting the sign language recognition requirements.

[0104] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0105] Please see Figure 4 , Figure 4 This is a schematic diagram of a framework of an embodiment of the electronic device 40 of this application. The electronic device 40 includes a memory 41 and a processor 42 coupled to each other. The processor 42 is used to execute program instructions stored in the memory 41 to implement the steps of any of the above-described sign language recognition method embodiments. In a specific implementation scenario, the electronic device 40 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 40 may also include mobile devices such as laptops and tablets, which are not limited here.

[0106] Specifically, processor 42 controls itself and memory 41 to implement the steps of any of the above-described sign language recognition method embodiments. Processor 42 may also be referred to as a CPU (Central Processing Unit). Processor 42 may be an integrated circuit chip with signal processing capabilities. Processor 42 may also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 42 may be implemented using integrated circuit chips.

[0107] Please see Figure 5 , Figure 5This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium 50 of this application. The computer-readable storage medium 50 stores program instructions 501 that can be executed by a processor. The program instructions 501 are used to implement the steps of any of the above-described embodiments of the sign language recognition method.

[0108] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0109] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0111] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0112] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A sign language recognition method, characterized in that, include: A sentence-segmented video frame sequence is obtained by capturing a sign language action sequence, wherein the sign language action in the sentence-segmented video frame sequence expresses a sentence. The segmented video frame sequence is divided according to word segmentation to obtain several word segmented video frame sequences, where the content expressed by the sign language action in the word segmented video frame sequence is a word. Action recognition is performed on each of the segmented video frame sequences to obtain the segmented words corresponding to each segmented video frame sequence, wherein the segmentation is based on the segmented semantic features extracted from the segmented video frame sequences; Using the word segments corresponding to each of the segmented video frame sequences, the sentences corresponding to the segmented video frame sequences are obtained; The word segmentation semantic features are extracted using a visual encoder. The method further includes: acquiring several sample word segmentation video frame sequences, including a first sample video frame sequence, at least one second sample video frame sequence, and at least one third sample video frame sequence. The first and second sample video frame sequences are video frame sequences in which different sign language demonstrators perform sign language actions on the same word, and the first and third sample video frame sequences are video frame sequences in which sign language actions are performed on different words. The visual encoder is used to extract features from the sample word segmentation video frame sequences to obtain sample semantic features corresponding to the sample word segmentation video frame sequences. A feature extraction loss is obtained using a first similarity between the first and second sample video frame sequences regarding the sample semantic features and a second similarity between the first and third sample video frame sequences regarding the sample semantic features. The parameters of the visual encoder are adjusted using the feature extraction loss. At least one of the preset semantic features is a sample semantic feature corresponding to the sample word segmentation video frame sequence.

2. The method according to claim 1, characterized in that, The step of performing action recognition on each of the segmented video frame sequences to obtain the corresponding word segments for each segmented video frame sequence includes: For each segmented video frame sequence, extract the segmented semantic features of the segmented video frame sequence; From a number of preset semantic features, find the preset semantic features that match the word segmentation semantic features; The word segment corresponding to the preset semantic features found is used as the word segment corresponding to the word segmented video frame sequence.

3. The method according to claim 2, characterized in that, The step of finding the preset semantic features that match the word segmentation semantic features from a plurality of preset semantic features includes: Obtain the similarity between the segmented semantic features and each of the preset semantic features; The preset semantic feature with the highest similarity is used as the preset semantic feature that matches the word segmentation semantic feature.

4. The method according to claim 1, characterized in that, The segmented video frame sequence is divided according to word segmentation to obtain several word-segmented video frame sequences, including: From the segmented video frame sequence, find several word-interval video frames that separate two sign language actions; For each pair of adjacent word-interval video frames, the video frames between the two adjacent word-interval video frames are obtained from the sentence-segment video frame sequence, and are used as the word-segment video frame sequence corresponding to the two adjacent word-interval video frames.

5. The method according to claim 4, characterized in that, The step of finding a number of word-interval video frames belonging to the separation between two sign language actions from the segmented video frame sequence includes: For each video frame in the segmented video frame sequence, a first video frame sequence corresponding to a video frame is obtained, and word spacing recognition is performed on the first video frame sequence corresponding to the video frame to obtain the word spacing recognition result of the video frame. The first video frame sequence corresponding to the video frame includes the video frame and a first number of video frames located after the video frame. In response to the word spacing recognition result of the video frame meeting the preset word spacing requirement, the video frame is determined to be the word spacing video frame.

6. The method according to claim 5, characterized in that, The word spacing recognition result is obtained by using a word spacing recognizer to identify word spacing. And / or, the word interval recognition result of the video frame is the word interval score of the video frame, and the preset word interval requirement is that the word interval score of the video frame is greater than a preset threshold.

7. The method according to claim 1, characterized in that, The acquisition of the segmented video frame sequence includes: For each second video frame sequence acquired, motion state recognition is performed on the second video frame sequence to obtain the state recognition result of the second video frame sequence. The state recognition result is used to determine whether there is a sign language action state in the second video frame sequence. The second video frame sequence includes a continuous second number of video frames obtained by acquiring the sign language action sequence. Adjacent second video frame sequences have some identical video frames. Based on the state recognition result, determine whether the second video frame sequence is a sign language start sequence or a sign language end sequence; The sentence-segment video frame sequence is obtained by combining the sign language start sequence, the sign language end sequence, and the second video frame sequence between the sign language start sequence and the sign language end sequence.

8. The method according to claim 7, characterized in that, The state recognition result is obtained by using a motion recognizer to identify the motion state. And / or, determining whether the second video frame sequence is a sign language start sequence or a sign language end sequence based on the state recognition result includes: Based on the state recognition result, it is determined whether the second video frame sequence is in a sign language action state or a non-sign language action state; In response to the second video frame sequence being in a sign language action state and the preceding second video frame sequence being in a non-sign language action state, the second video frame sequence is determined to be the sign language start sequence; In response to the second video frame sequence being in a non-sign language action state and the preceding second video frame sequence being in a sign language action state, the second video frame sequence is determined to be the sign language end sequence; And / or, the state recognition result is a sign language action state score, and before determining whether the second video frame sequence is a sign language start sequence or a sign language end sequence based on the state recognition result, the method further includes: The sign language action state score is converted into a state identifier, which is either a first identifier representing the sign language action state or a second identifier representing the non-sign language action state.

9. The method according to claim 1, characterized in that, The step of obtaining the sentence segmentation corresponding to the sentence segmentation video frame sequence by utilizing the word segmentation corresponding to each of the word segmentation video frame sequences includes: The segmented word sequences corresponding to the segmented video frame sequences are corrected to obtain corrected segmented word sequences; Using the corrected word segmentation sequence, the sentences corresponding to the segmented video frame sequence are obtained.

10. The method according to claim 9, characterized in that, The step of correcting the word segmentation sequence composed of the words corresponding to each of the word segmentation video frame sequences includes: The word segmentation sequence is corrected using an error corrector; and / or, The initial sentence is generated using the word segmentation sequence and displayed. Correction information for the initial sentence is received from the user and the word segmentation sequence is corrected based on the correction information.

11. The method according to claim 1, characterized in that, The segmented video frame sequence is acquired during a video call. After obtaining the sentences corresponding to the segmented video frame sequence by utilizing the word segments corresponding to each segmented video frame sequence, the method further includes: Send the sentences corresponding to the segmented video frame sequence to the other end in the video call; And / or, prior to acquiring the segmented video frame sequence, the method further includes: Identify the people in the currently captured video frame to obtain the position information of the people in the video frame; Based on the location information, determine whether the person in the video frame meets the sign language recognition requirements; If the person in the video frame does not meet the sign language recognition requirements, the user is prompted to adjust their position.

12. A sign language recognition device, characterized in that, The sign language recognition device is used to perform the sign language recognition method according to any one of claims 1 to 11, the sign language recognition device comprising: The acquisition module is used to acquire a sentence-segmented video frame sequence, which is obtained by capturing a sign language action sequence, and the sign language action in the sentence-segmented video frame sequence expresses a sentence. The word segmentation module is used to divide the segmented video frame sequence according to word segmentation to obtain several segmented video frame sequences. The content expressed by the sign language action in the segmented video frame sequence is a word. The recognition module is used to perform action recognition on each of the segmented video frame sequences to obtain the words corresponding to each of the segmented video frame sequences. The encoding module is used to obtain the sentences corresponding to the segmented video frame sequence by utilizing the word segments corresponding to each segmented video frame sequence.

13. An electronic device, characterized in that, The method includes a memory and a processor coupled to each other, the processor being configured to execute program instructions stored in the memory to implement the sign language recognition method according to any one of claims 1 to 11.

14. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the sign language recognition method according to any one of claims 1 to 11.

Images