Sign language recognition method and device, interaction system and electronic device

Abstract

The embodiment of the specification discloses a sign language recognition method, comprising: acquiring a to-be-recognized video stream, wherein the to-be-recognized video stream comprises a continuous sign language action image sequence; acquiring a hand motion posture sequence corresponding to each sign language action image sequence in the to-be-recognized video stream; inputting the to-be-recognized video stream into a pre-trained first recognition model to obtain a first recognition result; the first recognition result comprises a first probability distribution of each target word in a preset word table; inputting the hand motion posture sequence into a pre-trained second recognition model to obtain a second recognition result; the second recognition result comprises a second probability distribution of each target word in the word table; and determining a target text based on the first recognition result and the second recognition result. Accordingly, the application discloses a sign language recognition device and a corresponding interactive system and electronic equipment.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a sign language recognition method, device, interactive system, and electronic device. Background Technology

[0002] Traditional sign language recognition schemes typically use visual signals as input and text information as output, translating visual feature sequences into text feature sequences. However, this recognition method relies solely on visual signals, and during the visual signal acquisition process, factors such as shooting angle and the range of motion may make the movements difficult to recognize, leading to misidentification or missed recognition. Summary of the Invention

[0003] One of the objectives of this invention is to provide a sign language recognition method that combines sign language video streams and hand movement posture sequences to achieve more complete sign language recognition and more accurate sign language translation.

[0004] In accordance with the aforementioned objective, this specification provides an embodiment of a sign language recognition method, comprising:

[0005] Acquire a video stream to be identified, the video stream to be identified comprising a continuous sequence of sign language gesture images;

[0006] Obtain the hand movement posture sequence corresponding to each sign language action image sequence in the video stream to be identified;

[0007] The video stream to be identified is input into a pre-trained first recognition model to obtain a first recognition result; the first recognition result includes a first probability distribution of each target word in a preset vocabulary.

[0008] The hand movement posture sequence is input into a pre-trained second recognition model to obtain a second recognition result; the second recognition result includes a second probability distribution of each target word in the vocabulary;

[0009] Based on the first recognition result and the second recognition result, the target text is determined.

[0010] The sign language recognition method proposed in the embodiments of this specification obtains the visual features of sign language through a sequence of sign language action images and introduces a sequence of sign language movement postures to obtain the movement features of the hands. Then, the sign language recognition results corresponding to the two sequences are determined separately, and the target text corresponding to the sign language is determined by combining the two sign language recognition results. This method can effectively supplement data to address the problems of misidentification and omission caused by improper shooting angle or movement range, and obtain more accurate sign language translation results.

[0011] Furthermore, in some embodiments, the first recognition model is pre-trained in the following manner:

[0012] Acquire a sample video stream, the sample video stream comprising a continuous sequence of sign language action images;

[0013] Determine the first text corresponding to the sign language action image sequence sample, and use the first text as the first tag of the sample video stream;

[0014] The first recognition model is trained based on the sample video stream and the first tag until a first recognition model that meets the preset stopping condition is obtained.

[0015] Furthermore, in some embodiments, the second recognition model is pre-trained in the following manner:

[0016] Determine the second text;

[0017] Obtain a sequence of hand movement postures when performing the sign language action corresponding to the second text;

[0018] The second text is used as the second label for the hand movement posture sequence sample;

[0019] The second model is trained based on the hand movement posture sequence samples and the second label until a second recognition model that meets the preset stopping conditions is obtained.

[0020] Furthermore, in some embodiments, the step of inputting the video stream to be identified into a pre-trained first recognition model to obtain a first recognition result specifically includes:

[0021] When determining the prediction result of the first recognition model for the current target word, the determined text content in the target text is used as the first auxiliary input of the first recognition model;

[0022] Based on the first auxiliary input and the current hand action image sequence, the prediction result of the first recognition model for the current target word is determined.

[0023] Furthermore, in some embodiments, the step of inputting the hand movement posture data into a pre-trained second recognition model to obtain a second recognition result specifically includes:

[0024] When determining the prediction result of the second recognition model for the current target word, the determined text content in the target text is used as the second auxiliary input of the second recognition model;

[0025] Based on the second auxiliary input and the current hand movement posture sequence, the prediction result of the second recognition model for the current target word is determined.

[0026] Furthermore, in some embodiments, determining the target text based on the first recognition result and the second recognition result specifically includes:

[0027] For each target word in the target text, obtain the first probability distribution of the target word predicted by the first recognition model, and obtain the second probability distribution of the target word predicted by the second recognition model;

[0028] The third probability distribution of the target word is obtained by weighted summing of the first probability distribution and the second probability distribution of the target word.

[0029] The target word is determined from the vocabulary based on the third probability distribution of the target word.

[0030] Another objective of this invention is to provide a sign language recognition device that can combine sign language video streams and hand movement posture sequences to achieve more complete sign language recognition and more accurate sign language translation.

[0031] In accordance with the aforementioned objective, this specification provides an embodiment of a sign language recognition device, comprising:

[0032] The first data acquisition module is configured to acquire a video stream to be identified, the video stream to be identified including a continuous sequence of sign language action images;

[0033] The second data acquisition module is configured to acquire a hand movement posture sequence corresponding to each sign language action sequence in the video stream to be identified;

[0034] The first recognition module is configured to input the video stream to be recognized into a pre-trained first recognition model to obtain a first recognition result; the first recognition result includes a first probability distribution of each target word in a preset vocabulary.

[0035] The second recognition module is configured to input the hand movement posture sequence into a pre-trained second recognition model to obtain a second recognition result; the second recognition result includes a second probability distribution of each target word in the vocabulary.

[0036] The text generation module is configured to determine the target text based on the first recognition result and the second recognition result.

[0037] Furthermore, in some embodiments, the first identification model includes:

[0038] The first encoder is configured to extract features from the sign language action image sequence in the video stream to be identified, thereby obtaining visual features;

[0039] The first classifier is configured to classify the visual features to obtain the first recognition result.

[0040] Furthermore, in some embodiments, the second identification model includes:

[0041] The second encoder is configured to extract features from the hand movement posture sequence to obtain hand movement features;

[0042] The second classifier is configured to classify the hand movement features to obtain the second recognition result.

[0043] This specification also provides an interactive system in its embodiments, including:

[0044] The sign language gesture acquisition module is configured to acquire video of the sign language gestures to be recognized.

[0045] The sign language gesture recognition module is configured to recognize the sign language gesture video based on any one of the steps described in the above sign language recognition method to obtain the target text;

[0046] The information generation module is configured to convert the target text into information of a preset type and send the information to the user.

[0047] This specification also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described sign language recognition method.

[0048] This specification also provides an electronic device, including:

[0049] One or more processors; and

[0050] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the sign language recognition method described above.

[0051] The beneficial effects of the sign language recognition method described in this specification are that it obtains the visual features of sign language through a sequence of sign language action images, and introduces a sequence of sign language movement postures to obtain the movement features of the hands. Then, it determines the sign language recognition results corresponding to the two sequences respectively, and combines the two sign language recognition results to determine the target text corresponding to the sign language. This effectively supplements data to address misidentification and missed identification problems caused by improper shooting angles or movement ranges, achieving lightweight sign language recognition. Furthermore, using the determined text content in the target text as auxiliary input allows contextual information to be considered in sign language recognition, resulting in more accurate recognition results.

[0052] The sign language recognition device and interactive system described in the embodiments of this specification also have the above-mentioned beneficial effects. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A schematic flowchart of one implementation of the sign language recognition method described in the embodiments of this specification is shown as an example.

[0055] Figure 2 An exemplary flowchart illustrates the application of the sign language recognition method described in an embodiment of this specification to a scenario.

[0056] Figure 3 An exemplary block diagram of the sign language recognition device described in one embodiment of this specification is shown.

[0057] Figure 4 An exemplary block diagram of the interactive system described in one embodiment of this specification is shown.

[0058] Figure 5 An exemplary structural diagram of an electronic device provided in an embodiment of this specification is shown. Detailed Implementation

[0059] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments in this specification, and not all of the embodiments. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0060] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0061] The hearing-impaired are a significant part of the world's population, and various sectors are making efforts to improve their lives in all aspects. For the hearing-impaired, sign language is an indispensable communication tool in daily life, and technologies such as sign language recognition and translation greatly facilitate communication between the hearing-impaired and other groups by converting sign language gestures into corresponding text. However, the shooting angle and range of movement in sign language can make the gestures difficult to recognize, leading to misidentification or missed recognition. If the visual signals corresponding to the sign language gestures are used as input for translating visual feature sequences into text feature sequences, the problems that may arise during the sign language gesture acquisition process will affect the accuracy of the recognition results.

[0062] Therefore, this specification proposes a new sign language recognition scheme, which, based on the extraction of visual features of sign language movements, introduces motion posture data of sign language movements for calibration, such as hand sensor data, thereby effectively improving the accuracy of sign language recognition.

[0063] The sign language recognition method and apparatus described in this specification will be further described in detail below with reference to the accompanying drawings and specific embodiments. However, this detailed description does not constitute a limitation on the embodiments of this specification.

[0064] It should be noted that the terminology used in the embodiments of the present invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of the invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0065] In one embodiment of this specification, a sign language recognition method is proposed. Figure 1 A schematic flowchart of one implementation of the sign language recognition method described in the embodiments of this specification is shown as an example.

[0066] like Figure 1 As shown, it includes:

[0067] S100: Acquire the video stream to be recognized, which includes a continuous sequence of sign language gesture images.

[0068] The video of the sign language to be recognized can be captured by a device with image acquisition capabilities, such as a mobile phone, camera, or camcorder, and the acquired video stream can be transmitted to a designated location for sign language recognition, for example, by wired transmission or network transmission.

[0069] The video stream to be identified is a video of sign language actions. In sign language, commonly used words are represented by specific sign language actions, which can be static hand movements or a series of continuous hand movements. Therefore, the video stream to be identified is divided into several continuous hand movement image sequences, where each sign language image sequence corresponds to a translated word. The translated words are natural language words, and there are no restrictions on the type of language. All sign language image sequences constitute the video to be identified, and all the translated words constitute the target sentence text.

[0070] S102: Obtain the hand movement posture sequence corresponding to each sign language action image sequence in the video stream to be recognized.

[0071] In some embodiments, the hand movement sequence corresponding to a sign language gesture can be acquired using sensors placed on the hand, such as gyroscopes or accelerometers. When the position or posture of the hand changes, the corresponding sensors capture these changes and reflect them in the sensor data. Therefore, the hand movement sequence can be composed of sensor data, such as gyroscope data or accelerometer data.

[0072] S104: Input the video stream to be identified into the first pre-trained recognition model to obtain the first recognition result; the first recognition result includes the first probability distribution of each target word in the preset vocabulary.

[0073] In some embodiments, the first recognition model can be constructed based on an encoder-decoder structure. A video stream containing at least one continuous sequence of sign language gesture images is input into the encoder of the first recognition model for encoding. Visual features corresponding to each segment of the sign language gesture image sequence are extracted, where each segment of the sign language gesture image sequence translates into a word. Then, the decoder decodes the visual features corresponding to each sign language gesture image sequence to predict the sign language gesture translation result, obtaining the probability distribution of each word in a preset vocabulary as the target word corresponding to the sign language gesture, which serves as the first probability distribution, thereby obtaining the first recognition result.

[0074] The preset vocabulary list records target words that can be expressed through sign language gestures, including but not limited to common greetings such as thank you and goodbye, as well as common pronouns and nouns such as you, me, and him.

[0075] Optionally, a first recognition model can be constructed based on a Transformer network structure, and the encoder and decoder in the first recognition model can also adopt a Transformer network.

[0076] In some embodiments, the first recognition model is pre-trained in the following manner:

[0077] Acquire a sample video stream, which includes a continuous sequence of sign language action images;

[0078] Determine the first text corresponding to the sign language action image sequence sample, and use the first text as the first label of the sample video stream;

[0079] The first recognition model is trained based on the sample video stream and the first label until a first recognition model that meets the preset stopping conditions is obtained.

[0080] The sample video stream contains at least one continuous sequence of sign language gesture images. Each sequence represents a complete word. The corresponding words are combined to obtain the first text corresponding to the sample video stream, which describes the meaning of the sign language gestures in the sample video stream in text form. This first text is then used as the ground truth label for the sample video stream. The sample video stream is input into a first recognition model, and after encoding and decoding, a first recognition result is obtained. The first recognition loss is determined by calculating the difference between the first recognition result and the first label. The first recognition model is trained with the goal of minimizing this recognition loss.

[0081] In some more specific embodiments, the loss function for the first recognition loss can be a total probability distribution difference. This involves calculating the difference between the probability distribution predicted by the first recognition model for each sign language action image sequence sample in the sample video stream and the corresponding word in the first text, and then summing all calculated differences as the first recognition loss. For example, if the first text contains 10 words, the loss function can be calculated by first determining the difference between the probability distribution of each word and the probability distribution predicted by the first recognition model for the corresponding sign language action image sequence, and then summing the differences of the probability distributions of all 10 words to obtain the first recognition loss.

[0082] The first recognition model takes a video of sign language gestures to be recognized as input and outputs the corresponding sign language text. The video stream to be recognized is divided into at least one continuous sequence of sign language gesture images and encoded and decoded. Each independent sign language gesture is translated and output, achieving word-by-word recognition.

[0083] In some embodiments, the video stream to be identified is input into a pre-trained first recognition model to obtain a first recognition result, specifically including:

[0084] When determining the prediction result of the first recognition model for the current target word, the determined text content in the target text is used as the first auxiliary input of the first recognition model;

[0085] Based on the first auxiliary input and the current hand action image sequence, the prediction result of the first recognition model for the current target word is determined.

[0086] Since the first recognition model outputs sign language word by word, when recognizing the second and subsequent words, the probability distribution of the already recognized words and the hand movement image sequence can be input into the first recognition model together to assist in the prediction of subsequent words. Combined with contextual information, a more accurate and reasonable sign language recognition result can be obtained.

[0087] S106: Input the hand movement posture sequence into the pre-trained second recognition model to obtain the second recognition result; the second recognition result includes the second probability distribution of each target word in the vocabulary.

[0088] In some embodiments, the second recognition model can be constructed based on an encoder-decoder structure. The hand movement posture sequence acquired by the sensor is input into the encoder of the second recognition model for encoding, and the hand movement features corresponding to each segment of the hand movement posture sequence are extracted. Each segment of the hand movement posture sequence corresponds to a sign language action, and correspondingly, a word represented by that sign language action. Then, the decoder decodes the hand movement features corresponding to each hand movement posture sequence to predict the sign language translation result, obtaining the probability distribution of each word in a preset vocabulary as the target word corresponding to the sign language action, which serves as the second probability distribution, and thus the second recognition result is obtained.

[0089] To address the issues of inaccurate or incomplete sign language recognition caused by problems such as the shooting angle or range of sign language movements in sign language videos, the introduction of hand movement posture data can supplement the information, thereby improving the accuracy of sign language recognition and reducing misidentification or missed recognition.

[0090] Optionally, a second recognition model can be constructed based on the Transformer network structure, and the encoder and decoder in the second recognition model can also adopt the Transformer network.

[0091] In some embodiments, the second recognition model is pre-trained in the following manner:

[0092] Determine the second text;

[0093] Obtain hand movement posture sequence samples when performing the sign language action corresponding to the second text;

[0094] The second text is used as the second label for the hand movement posture sequence sample;

[0095] The second model is trained based on hand movement posture sequence samples and the second label until a second recognition model that meets the preset stopping conditions is obtained.

[0096] When acquiring samples, it is necessary to first determine the second text as the true label, and then collect hand movement posture sequences when expressing the content of the second text in sign language using sensors and other tools as training samples. Each word in the second text corresponds to a segment of hand movement posture sequence samples. The hand movement posture sequence samples are input into the second recognition model, and the second recognition result is obtained through encoding and decoding. The second recognition loss is determined by calculating the difference between the second recognition result and the second label. The second recognition model is trained with the goal of minimizing this recognition loss.

[0097] In some more specific embodiments, the loss function of the second recognition loss can be a total probability distribution difference, that is, calculating the difference between the probability distribution predicted by the second recognition model for each hand movement posture sequence sample and the corresponding words in the second text, and then determining the sum of all calculated differences as the second recognition loss.

[0098] The second recognition model takes a sequence of hand gestures as input and outputs the corresponding sign language text. After encoding and decoding the hand gesture sequences representing each sign language action, it translates and outputs each independent sign language action, achieving word-by-word recognition.

[0099] In some embodiments, hand movement posture data is input into a pre-trained second recognition model to obtain a second recognition result, specifically including:

[0100] When determining the prediction result of the second recognition model for the current target word, the determined text content in the target text is used as the second auxiliary input of the second recognition model;

[0101] Based on the second auxiliary input and the current hand movement posture sequence, the prediction result of the second recognition model for the current target word is determined.

[0102] Since the second recognition model outputs sign language word by word, when recognizing the second word and subsequent words, the probability distribution of the already recognized words and the hand movement posture sequence samples can be input into the second recognition model together to assist in the prediction of subsequent words. Combined with contextual information, a more accurate and reasonable sign language recognition result can be obtained.

[0103] S108: Based on the first recognition result and the second recognition result, determine the target text.

[0104] The target text is the words or sentences obtained after translating the sign language actions in the video stream to be recognized. It is a natural language word, and the type of language is not limited.

[0105] In some embodiments, determining the target text based on the first recognition result and the second recognition result specifically includes:

[0106] For each target word in the target text, obtain the first probability distribution of the target word predicted by the first recognition model, and obtain the second probability distribution of the target word predicted by the second recognition model;

[0107] The third probability distribution of the target word is obtained by weighted summation of the first and second probability distributions.

[0108] The target word is determined from the vocabulary based on the third probability distribution of the target word.

[0109] Since both the first and second recognition models can identify and output word-by-word, the first and second recognition results respectively contain the probability distribution of each target word. The first recognition result is obtained based on the visual features of sign language movements, which are the primary basis for understanding the meaning of sign language and should be considered more important than hand movement features. Therefore, in some more specific embodiments, when weighted summing the first and second probability distributions of the target words, a higher weight is assigned to the first probability distribution based on visual features, while a lower weight is assigned to the second probability distribution based on hand movement features. This allows the first recognition model to play a primary role, while the second recognition model serves as a fine-tuning tool.

[0110] After obtaining the third probability distribution based on the first and second probability distributions of the target word, the predicted word with the highest probability is selected from the preset word list as the target word.

[0111] By supplementing sign language recognition with hand movement posture data, the lack of visual features during the sign language action acquisition process can be effectively compensated for, such as misidentification or missed identification. This decouples visual data from movement posture data, thereby improving the accuracy and efficiency of sign language recognition.

[0112] The sign language recognition method provided in this specification obtains visual features of sign language through a sequence of sign language action images and introduces a sequence of sign language movement postures to obtain hand movement features. Then, it combines the sign language recognition results obtained based on these two types of sign language features with weighted averages to obtain the corresponding target text. This helps to effectively supplement data to address misidentification and missed identification problems caused by improper shooting angles or movement ranges, achieving decoupling between visual data and movement posture data, thus making sign language recognition lightweight. Furthermore, using identified words from the target text as auxiliary input allows contextual information to be considered in the sign language recognition process, resulting in more accurate recognition results.

[0113] The following description uses an example of the application of a sign language recognition method provided in this embodiment in a specific scenario to further illustrate the sign language recognition method provided in this embodiment. However, this description does not constitute a limitation on the embodiments of this specification.

[0114] Figure 2 An exemplary flowchart illustrates the application of the sign language recognition method described in an embodiment of this specification to a scenario.

[0115] like Figure 2 As shown, the sign language recognition method specifically includes:

[0116] S200: Acquire the sign language video stream to be recognized, which includes three consecutive sequences of sign language action images.

[0117] S202: Obtain the hand movement posture sequence corresponding to each sign language action image sequence in the video stream to be recognized through a hand sensor.

[0118] S204: Input the video stream to be identified into the first pre-trained recognition model. After predicting the first word based on the first segment of sign language action image sequence, predict the second word based on the word and the second segment of sign language action image sequence. Then predict the third word based on the second word and the third segment of sign language action image sequence, thereby obtaining the first recognition result. The first recognition result includes the first probability distribution of each target word in the preset word list.

[0119] S206: Input the hand movement posture sequence into the pre-trained second recognition model. After predicting the first word based on the first segment of the hand movement posture sequence, predict the second word based on the word and the second segment of the hand movement posture sequence. Then predict the third word based on the second word and the third segment of the hand movement posture sequence, thereby obtaining the second recognition result. The second recognition result includes the second probability distribution of each target word in the preset word list.

[0120] S208: For each target word in the target text, obtain its first probability distribution predicted by the first recognition model and its second probability distribution predicted by the second recognition model. Set a weight of 0.8 for the first probability distribution and a weight of 0.2 for the second probability distribution, and perform a weighted sum to obtain the third probability distribution of the target word. Then, determine the target word from the preset word list based on the third probability distribution. Finally, determine the three target words as "I", "very", and "happy", and output the sign language recognition text as "I am very happy".

[0121] In another embodiment, the video stream to be identified and the corresponding hand movement posture sequence can be spliced ​​together and input into a pre-trained third recognition model to obtain a third recognition result. The target text can be determined based on the third recognition result to achieve multimodal sign language recognition.

[0122] Optionally, the third recognition model can adopt a Transformer-based encoder-decoder network structure, where both the encoder and decoder can be built based on Transformer. The video stream to be recognized and the corresponding hand motion posture sequence are input into the encoder, and the visual features and corresponding hand motion features of each sign language action are extracted respectively. The two features are then fused to obtain the fused feature of the sign language action. The fused feature is input into the decoder for prediction to obtain the probability distribution of each word in the preset vocabulary as the target word corresponding to the sign language action, thereby obtaining the third recognition result and outputting the sign language recognized text.

[0123] The third recognition model can be pre-trained using the following methods:

[0124] Acquire a sample video stream, which includes a continuous sequence of sign language action images;

[0125] The third text corresponding to the sign language action image sequence sample is determined, and the third text is used as the third tag of the sample video stream;

[0126] Obtain hand movement posture sequence samples when performing the sign language action corresponding to the third text;

[0127] The sample video stream and the corresponding hand movement posture sequence samples are input into the third recognition model to obtain the sign language prediction results;

[0128] Based on the sign language prediction results and the third label, the third recognition model is trained until a third recognition model that meets the preset stopping conditions is obtained.

[0129] The third recognition model is trained with the goal of minimizing the difference between the sign language prediction result and the third label.

[0130] In another embodiment of this specification, a sign language recognition device is proposed. Figure 3 An exemplary block diagram of the sign language recognition device described in one embodiment of this specification is shown.

[0131] like Figure 3 As shown, it includes:

[0132] The first data acquisition module 30 is configured to acquire a video stream to be identified, which includes a continuous sequence of sign language action images.

[0133] The second data acquisition module 32 is configured to acquire the hand movement posture sequence corresponding to each sign language action sequence in the video stream to be recognized;

[0134] The first recognition module 34 is configured to input the video stream to be recognized into a pre-trained first recognition model to obtain a first recognition result; the first recognition result includes the first probability distribution of each target word in a preset vocabulary.

[0135] The second recognition module 36 is configured to input a hand movement posture sequence into a pre-trained second recognition model to obtain a second recognition result; the second recognition result includes a second probability distribution of each target word in the vocabulary;

[0136] The text generation module 38 is configured to determine the target text based on the first recognition result and the second recognition result.

[0137] The first data acquisition module can capture the sign language video to be recognized using a device with image acquisition capabilities, such as a mobile phone, camera, or camcorder, and transmit the acquired video stream to a designated location for sign language recognition, for example, by using wired transmission or network transmission.

[0138] The video stream to be identified is a video of sign language actions. In sign language, commonly used words are represented by specific sign language actions, which can be static hand movements or a series of continuous hand movements. Therefore, the video stream to be identified is divided into several continuous hand movement image sequences, where each sign language image sequence corresponds to a translated word. The translated words are natural language words, and there are no restrictions on the type of language. All sign language image sequences constitute the video to be identified, and all the translated words constitute the target sentence text.

[0139] In some embodiments, the second data acquisition module can utilize sensors placed on the hand to acquire hand movement posture sequences corresponding to sign language gestures, such as gyroscopes or accelerometers. When the position or posture of the hand changes, the corresponding sensors capture these changes and reflect them in the sensor data. Therefore, the hand movement posture sequence can be composed of sensor data, such as gyroscope data or accelerometer data.

[0140] In some embodiments, the first recognition model includes:

[0141] The first encoder is configured to extract visual features from the sequence of sign language action images in the video stream to be recognized.

[0142] The first classifier is configured to classify visual features to obtain the first recognition result.

[0143] The first recognition module can construct a first recognition model based on an encoder-decoder structure, where the classifier can be an encoder. The first recognition module inputs a video stream containing at least one continuous sequence of sign language gesture images into the encoder of the first recognition model for encoding, extracting the visual features corresponding to each segment of the sign language gesture image sequence. Each segment of the sign language gesture image sequence corresponds to a word after translation. Then, the classifier classifies the visual features corresponding to each sign language gesture image sequence to predict the sign language gesture translation result, obtaining the probability distribution of each word in a preset vocabulary as the target word corresponding to the sign language gesture, which serves as the first probability distribution, thus obtaining the first recognition result.

[0144] The preset vocabulary list records target words that can be expressed through sign language gestures, including but not limited to common greetings such as thank you and goodbye, as well as common pronouns and nouns such as you, me, and him.

[0145] Optionally, a first recognition model can be constructed based on a Transformer network structure, and the encoder and classifier in the first recognition model can also adopt a Transformer network.

[0146] In some embodiments, the first recognition module pre-trains the first recognition model in the following manner:

[0147] Acquire a sample video stream, which includes a continuous sequence of sign language action images;

[0148] Determine the first text corresponding to the sign language action image sequence sample, and use the first text as the first label of the sample video stream;

[0149] The first recognition model is trained based on the sample video stream and the first label until a first recognition model that meets the preset stopping conditions is obtained.

[0150] The sample video stream contains at least one continuous sequence of sign language gesture images. Each sequence represents a complete word. The corresponding words are combined to obtain the first text corresponding to the sample video stream, which describes the meaning of the sign language gestures in the sample video stream in text form. This first text is then used as the true label of the sample video stream. The first recognition module inputs the sample video stream into the first recognition model, and obtains the first recognition result through encoding and decoding. The first recognition loss is determined by calculating the difference between the first recognition result and the first label. The first recognition model is trained with the goal of minimizing this recognition loss.

[0151] In some more specific embodiments, the loss function of the first recognition loss can be a total probability distribution difference, that is, the first recognition module calculates the difference between the probability distribution predicted by the first recognition model for each sign language action image sequence sample in the sample video stream and the corresponding words in the first text, and then determines the sum of all calculated differences as the first recognition loss.

[0152] The first recognition model takes a video of sign language gestures to be recognized as input and outputs the corresponding sign language text. The first recognition module divides the video stream to be recognized into at least one continuous sequence of sign language gesture images and performs encoding and decoding. It translates and outputs each independent sign language gesture, achieving word-by-word recognition.

[0153] In some embodiments, the first recognition module is specifically used to, when determining the prediction result of the first recognition model for the current target word, take the determined text content in the target text as the first auxiliary input of the first recognition model; and determine the prediction result of the first recognition model for the current target word based on the first auxiliary input and the current hand action image sequence.

[0154] Since the first recognition module outputs sign language recognition word by word when using the first recognition model, when recognizing the second word and subsequent words, the probability distribution of the recognized words and the hand movement image sequence can be input into the first recognition model together to assist in the prediction of subsequent words. Combined with contextual information, a more accurate and reasonable sign language recognition result can be obtained.

[0155] In some embodiments, the second identification model includes:

[0156] The second encoder is configured to extract features from the hand motion posture sequence to obtain hand motion features;

[0157] The second classifier is configured to classify hand motion features to obtain a second recognition result.

[0158] The second recognition module can construct a second recognition model based on an encoder-decoder structure, where the classifier can be an encoder. The second recognition module inputs the hand movement posture sequence collected by sensors into the encoder of the second recognition model for encoding, extracting the hand movement features corresponding to each segment of the hand movement posture sequence. Each segment of the hand movement posture sequence corresponds to a sign language action, and correspondingly, a word represented by that sign language action. Then, the classifier classifies the hand movement features corresponding to each hand movement posture sequence to predict the sign language translation result, obtaining the probability distribution of each word in a preset vocabulary as the target word corresponding to the sign language action, which serves as the second probability distribution, thus obtaining the second recognition result.

[0159] To address the issues of inaccurate or incomplete sign language recognition caused by problems such as the shooting angle or range of sign language movements in sign language videos, the second recognition module supplements the information by introducing hand movement posture data, which can improve the problem of misrecognition or missed recognition and improve the accuracy of sign language recognition.

[0160] Optionally, a second recognition model can be constructed based on the Transformer network structure, and the encoder and classifier in the second recognition model can also adopt the Transformer network.

[0161] In some embodiments, the second recognition module pre-trains the second recognition model in the following manner:

[0162] Determine the second text;

[0163] Obtain hand movement posture sequence samples when performing the sign language action corresponding to the second text;

[0164] The second text is used as the second label for the hand movement posture sequence sample;

[0165] The second model is trained based on hand movement posture sequence samples and the second label until a second recognition model that meets the preset stopping conditions is obtained.

[0166] When acquiring samples, the second recognition module first needs to determine the second text as the true label, and then collect hand movement posture sequences when expressing the content of the second text in sign language using sensors and other tools as training samples. Each word in the second text corresponds to a segment of hand movement posture sequence samples. The hand movement posture sequence samples are input into the second recognition model, and the second recognition result is obtained through encoding and decoding. The second recognition loss is determined by calculating the difference between the second recognition result and the second label. The second recognition model is trained with the goal of minimizing this recognition loss.

[0167] In some more specific embodiments, the loss function of the second recognition loss can be a total probability distribution difference, that is, calculating the difference between the probability distribution predicted by the second recognition model for each hand movement posture sequence sample and the corresponding words in the second text, and then determining the sum of all calculated differences as the second recognition loss.

[0168] The second recognition model takes a sequence of hand gestures as input and outputs the corresponding sign language text. The second recognition module encodes and decodes the hand gesture sequences representing each sign language action, then translates and outputs each independent sign language action, achieving word-by-word recognition.

[0169] In some embodiments, the second recognition module is specifically used to, when determining the prediction result of the second recognition model for the current target word, take the determined text content in the target text as the second auxiliary input of the second recognition model; and determine the prediction result of the second recognition model for the current target word based on the second auxiliary input and the current hand movement posture sequence.

[0170] Since the second recognition model outputs sign language word by word, when recognizing the second word and subsequent words, the probability distribution of the already recognized words and the hand movement posture sequence samples can be input into the second recognition model together to assist in the prediction of subsequent words. Combined with contextual information, a more accurate and reasonable sign language recognition result can be obtained.

[0171] The target text is the words or sentences obtained after translating the sign language actions in the video stream to be recognized. It is a natural language word, and the type of language is not limited.

[0172] In some embodiments, the text generation module is specifically used to obtain, for each target word in the target text, a first probability distribution of the target word predicted by a first recognition model and a second probability distribution of the target word predicted by a second recognition model; to perform a weighted summation of the first probability distribution and the second probability distribution of the target word to obtain a third probability distribution of the target word; and to determine the target word from the vocabulary based on the third probability distribution of the target word.

[0173] Since both the first and second recognition models can identify and output word-by-word, the first and second recognition results respectively contain the probability distributions of each target word. The first recognition result is obtained based on the visual features of sign language gestures, which are the primary basis for understanding the meaning of sign language and should be considered more important than hand movement features. Therefore, in some more specific embodiments, when the text generation module performs a weighted sum of the first and second probability distributions of the target words, it assigns a higher weight to the first probability distribution based on visual features and a lower weight to the second probability distribution based on hand movement features, allowing the first recognition model to play a primary role while the second recognition model plays a fine-tuning role.

[0174] After obtaining the third probability distribution based on the first and second probability distributions of the target word, the text generation module selects the predicted word with the highest probability from the preset word list as the target word.

[0175] The text generation module supplements sign language recognition with hand movement posture data, which can effectively make up for the lack of visual features during the sign language action acquisition process, such as misidentification or missed recognition. It decouples visual data from movement posture data, thereby improving the accuracy and efficiency of sign language recognition.

[0176] One embodiment of this specification also proposes an interactive system. Figure 4 An exemplary block diagram of the interactive system described in one embodiment of this specification is shown.

[0177] like Figure 4 As shown, it includes:

[0178] The sign language gesture acquisition module 40 is configured to acquire video of the sign language gestures to be recognized.

[0179] The sign language action recognition module 42 is configured to recognize the sign language action video based on any one of the steps described in the above sign language recognition method to obtain the target text.

[0180] The information generation module 44 is configured to convert the target text into information of a preset type and send the information to the user.

[0181] The sign language motion acquisition module can capture video of the sign language to be recognized using a terminal device with image acquisition capabilities, such as a mobile phone, camera, or camcorder. The acquired video stream is then transmitted to the interactive system via wired or network transmission for sign language recognition. Based on the sign language motion video, the module extracts a continuous sequence of sign language motion images and corresponding hand movement posture sequences, which are then sent to the sign language motion recognition module. Each sign language action represents a word and corresponds to a sequence of sign language motion images and a sequence of hand movement postures.

[0182] The sign language action recognition module inputs the received sign language action image sequence and the corresponding hand movement posture sequence into any pre-trained recognition model in the above sign language recognition method to recognize the sign language action video, obtain the target text, and transmit the target text to the information generation module.

[0183] The information generation module converts the target text into information in various forms, including but not limited to text, speech, vibration, and other suitable combinations of languages, and sends it to the user through terminal devices such as mobile phones and tablets to complete sign language recognition.

[0184] One embodiment of this specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described sign language recognition method.

[0185] One embodiment of this specification also provides an electronic device, comprising:

[0186] One or more processors; and

[0187] A memory associated with one or more processors, the memory being used to store program instructions that, when read and executed by one or more processors, perform the steps of the sign language recognition method described above.

[0188] Figure 5 An exemplary structural diagram of an electronic device provided in an embodiment of this specification is shown, illustrating a structural schematic diagram of a computer system 500 suitable for implementing a terminal device or server of the present invention. Figure 5 The terminal device or server shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0189] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 502, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 504 or programs loaded from storage section 516 into random access memory (RAM) 506. The RAM 506 also stores various programs and data required for the operation of the system 500. The CPU 502, ROM 504, and RAM 506 are interconnected via a bus 508. An input / output (I / O) interface 510 is also connected to the bus 508.

[0190] The following components are connected to I / O interface 510: an input section 512 including a keyboard, mouse, etc.; an output section 514 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 516 including a hard disk, etc.; and a communication section 518 including a network interface card such as a LAN card, modem, etc. The communication section 518 performs communication processing via a network such as the Internet. Drive 520 is also connected to I / O interface 510 as needed. Removable media 522, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 520 as needed so that computer programs read from it can be installed into storage section 516 as needed.

[0191] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 518, and / or installed from removable medium 522. When the computer program is executed by central processing unit (CPU) 502, it performs the functions defined in the methods of this invention.

[0192] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0193] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous. It should also be noted that each block in the drawings, and combinations of blocks in the drawings, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0194] It should be noted that the above examples are merely specific embodiments of the present invention, and the present invention is obviously not limited to the above embodiments, with many similar variations. All modifications that can be directly derived or conceived by those skilled in the art from the content disclosed in this invention should fall within the protection scope of this invention.

Claims

1. A sign language recognition method, comprising: Acquire a video stream to be identified, the video stream to be identified comprising a continuous sequence of sign language gesture images; Obtain the hand movement posture sequence corresponding to each sign language gesture image sequence in the video stream to be identified; the hand movement posture sequence is acquired by sensors installed on the hand. The video stream to be identified is input into a pre-trained first recognition model to obtain a first recognition result; the first recognition result includes a first probability distribution of each target word in a preset vocabulary. The hand movement posture sequence is input into a pre-trained second recognition model to obtain a second recognition result; the second recognition result includes a second probability distribution of each target word in the vocabulary; Based on the first recognition result and the second recognition result, the target text is determined.

2. The method as described in claim 1, wherein the first recognition model is pre-trained in the following manner: Acquire a sample video stream, the sample video stream comprising a continuous sequence of sign language action images; Determine the first text corresponding to the sign language action image sequence sample, and use the first text as the first tag of the sample video stream; The first recognition model is trained based on the sample video stream and the first tag until a first recognition model that meets the preset stopping condition is obtained.

3. The method as described in claim 1, wherein the second recognition model is pre-trained in the following manner: Determine the second text; Obtain a sequence of hand movement postures when performing the sign language action corresponding to the second text; The second text is used as the second label for the hand movement posture sequence sample; The second recognition model is trained based on the hand movement posture sequence samples and the second label until a second recognition model that meets the preset stopping conditions is obtained.

4. The method as described in claim 1, wherein inputting the video stream to be identified into a pre-trained first recognition model to obtain a first recognition result specifically includes: When determining the prediction result of the first recognition model for the current target word, the determined text content in the target text is used as the first auxiliary input of the first recognition model; Based on the first auxiliary input and the current hand action image sequence, the prediction result of the first recognition model for the current target word is determined.

5. The method as described in claim 1, wherein inputting the hand movement posture sequence into a pre-trained second recognition model to obtain a second recognition result specifically includes: When determining the prediction result of the second recognition model for the current target word, the determined text content in the target text is used as the second auxiliary input of the second recognition model; Based on the second auxiliary input and the current hand movement posture sequence, the prediction result of the second recognition model for the current target word is determined.

6. The method as described in claim 1, wherein determining the target text based on the first recognition result and the second recognition result specifically includes: For each target word in the target text, obtain the first probability distribution of the target word predicted by the first recognition model, and obtain the second probability distribution of the target word predicted by the second recognition model; The third probability distribution of the target word is obtained by weighted summing of the first probability distribution and the second probability distribution of the target word. The target word is determined from the vocabulary based on the third probability distribution of the target word.

7. A sign language recognition device, comprising: The first data acquisition module is configured to acquire a video stream to be identified, the video stream to be identified including a continuous sequence of sign language action images; The second data acquisition module is configured to acquire a hand movement posture sequence corresponding to each sign language action sequence in the video stream to be identified; the hand movement posture sequence is acquired by a sensor installed on the hand. The first recognition module is configured to input the video stream to be recognized into a pre-trained first recognition model to obtain a first recognition result; the first recognition result includes a first probability distribution of each target word in a preset vocabulary. The second recognition module is configured to input the hand movement posture sequence into a pre-trained second recognition model to obtain a second recognition result; the second recognition result includes a second probability distribution of each target word in the vocabulary. The text generation module is configured to determine the target text based on the first recognition result and the second recognition result.

8. The apparatus of claim 7, wherein the first identification model comprises: The first encoder is configured to extract features from the sign language action image sequence in the video stream to be identified, thereby obtaining visual features; The first classifier is configured to classify the visual features to obtain the first recognition result.

9. The apparatus of claim 8, wherein the second identification model comprises: The second encoder is configured to extract features from the hand movement posture sequence to obtain hand movement features; The second classifier is configured to classify the hand movement features to obtain the second recognition result.

10. An interactive system, comprising: The sign language gesture acquisition module is configured to acquire video of the sign language gestures to be recognized. A sign language gesture recognition module is configured to recognize the sign language gesture video based on the method described in any one of claims 1 to 6, and obtain target text; The information generation module is configured to convert the target text into information of a preset type and send the information to the user.

11. A computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing the method as claimed in any one of claims 1 to 6.

12. An electronic device, comprising: One or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method as described in any one of claims 1 to 6.

Images