Sign language recognition and interaction

Abstract

Disclosed in embodiments of the present description is a sign language recognition method, comprising: acquiring a video stream to be recognized, said video stream comprising continuous sign language action image sequences; acquiring a hand motion posture sequence corresponding to each sign language action image sequence in said video stream; inputting said video stream into a pre-trained first recognition model to obtain a first recognition result, the first recognition result comprising first probability distribution of each target word in a preset word list; inputting the hand motion posture sequence into a pre-trained second recognition model to obtain a second recognition result, the second recognition result comprising second probability distribution of each target word in the word list; and determining a target text on the basis of the first recognition result and the second recognition result. Correspondingly, disclosed in the present invention are a sign language recognition apparatus, and a corresponding interaction system and electronic device.

Description

Sign language recognition and interaction Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a sign language recognition method, device, interactive system and electronic equipment. Background Art

[0002] Traditional sign language recognition solutions typically use visual signals as input and text as output, translating visual feature sequences into text feature sequences. However, this recognition method relies solely on visual signals. During the visual signal acquisition process, factors such as the shooting angle and the range of the action can make it difficult to identify the action, leading to misidentification or missed recognition.

[0003] Summary of the Invention

[0004] One of the objectives of the present invention is to provide a sign language recognition method that can combine a sign language video stream and a hand motion gesture sequence to achieve more complete sign language recognition and more accurate sign language translation.

[0005] According to the above-mentioned invention purpose, the embodiment of this specification proposes a sign language recognition method, including: obtaining a video stream to be recognized, the video stream to be recognized including a continuous sequence of sign language action images; obtaining a hand movement posture sequence corresponding to each sign language action image sequence in the video stream to be recognized; inputting the video stream to be recognized into a pre-trained first recognition model to obtain a first recognition result; the first recognition result includes a first probability distribution of each target word in a preset vocabulary; inputting the hand movement posture sequence into a pre-trained second recognition model to obtain a second recognition result; the second recognition result includes a second probability distribution of each target word in the vocabulary; and determining the target text based on the first recognition result and the second recognition result.

[0006] The sign language recognition method proposed in the embodiments of this specification obtains the visual features of sign language through a sign language action image sequence, and introduces a sign language motion posture sequence to obtain the motion features of the hand, and then determines the sign language recognition results corresponding to the above two sequences respectively. The two sign language recognition results are combined to determine the target text corresponding to the sign language. It can effectively supplement data for misrecognition and missed recognition problems caused by improper shooting angles or movement ranges, and obtain more accurate sign language translation results.

[0007] Furthermore, in some embodiments, the first recognition model is pre-trained in the following manner: obtaining a sample video stream, the sample video stream including a continuous sign language action image sequence sample; determining a first text corresponding to the sign language action image sequence sample, and using the first text as a first label of the sample video stream; training the first recognition model based on the sample video stream and the first label until the first recognition model that meets a preset stop condition is obtained.

[0008] Furthermore, in some embodiments, the second recognition model is pre-trained in the following manner: determining a second text; obtaining a hand motion gesture sequence sample when making a sign language action corresponding to the second text; using the second text as a second label for the hand motion gesture sequence sample; and training the second model based on the hand motion gesture sequence sample and the second label until the second recognition model that meets a preset stop condition is obtained.

[0009] Furthermore, in some embodiments, the inputting of the video stream to be recognized into a pre-trained first recognition model to obtain a first recognition result specifically includes: when determining the prediction result of the first recognition model for the current target word, using the determined text content in the target text as the first auxiliary input of the first recognition model; based on the first auxiliary input and the current hand movement image sequence, determining the prediction result of the first recognition model for the current target word.

[0010] Furthermore, in some embodiments, the hand motion posture data is input into a pre-trained second recognition model to obtain a second recognition result, specifically including: when determining the prediction result of the second recognition model for the current target word, using the determined text content in the target text as a second auxiliary input of the second recognition model; based on the second auxiliary input and the current hand motion posture sequence, determining the prediction result of the second recognition model for the current target word.

[0011] Furthermore, in some embodiments, determining the target text based on the first recognition result and the second recognition result specifically includes: for each target word in the target text, obtaining the first probability distribution of the target word predicted by the first recognition model, and obtaining the second probability distribution of the target word predicted by the second recognition model; performing weighted summation of the first probability distribution and the second probability distribution of the target word to obtain a third probability distribution of the target word; and determining the target word from the vocabulary based on the third probability distribution of the target word.

[0012] Another object of the present invention is to provide a sign language recognition device that can combine sign language video streams and hand movement posture sequences to achieve more complete sign language recognition and more accurate sign language translation.

[0013] According to the above-mentioned invention purpose, the embodiment of this specification proposes a sign language recognition device, including: a first data acquisition module, configured to acquire a video stream to be recognized, wherein the video stream to be recognized includes a continuous sequence of sign language action images; a second data acquisition module, configured to acquire a hand motion posture sequence corresponding to each sign language action sequence in the video stream to be recognized; a first recognition module, configured to input the video stream to be recognized into a pre-trained first recognition model to obtain a first recognition result; the first recognition result includes a first probability distribution of each target word in a preset vocabulary; a second recognition module, configured to input the hand motion posture sequence into a pre-trained second recognition model to obtain a second recognition result; the second recognition result includes a second probability distribution of each target word in the vocabulary; a text generation module, configured to determine the target text based on the first recognition result and the second recognition result.

[0014] Furthermore, in some embodiments, the first recognition model includes: a first encoder, configured to extract features of the sign language action image sequence in the video stream to be recognized to obtain visual features; and a first classifier, configured to classify the visual features to obtain the first recognition result.

[0015] Furthermore, in some embodiments, the second recognition model includes: a second encoder, configured to extract features of the hand motion posture sequence to obtain hand motion features; and a second classifier, configured to classify the hand motion features to obtain the second recognition result.

[0016] An embodiment of this specification also provides an interactive system, including: a sign language action acquisition module, configured to obtain a sign language action video to be recognized; a sign language action recognition module, configured to recognize the sign language action video based on the steps described in any one of the above-mentioned sign language recognition methods to obtain a target text; an information generation module, configured to convert the target text into information of a preset type and send the information to the user.

[0017] The embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned sign language recognition method when executed by a processor.

[0018] An embodiment of this specification also provides an electronic device, comprising: one or more processors; and a memory associated with the one or more processors, wherein the memory is used to store program instructions, and when the program instructions are read and executed by the one or more processors, the steps of the above-mentioned sign language recognition method are executed.

[0019] The beneficial effect of the sign language recognition method described in the embodiments of this specification lies in obtaining the visual features of sign language through a sequence of sign language action images, and introducing a sequence of sign language motion postures to obtain hand motion features. The sign language recognition results corresponding to the two sequences are then determined separately, and the target text corresponding to the sign language is determined by combining the two sign language recognition results. This method effectively supplements data for misidentification and missed recognition issues caused by improper shooting angles or motion ranges, achieving lightweight sign language recognition. Furthermore, using the identified text content of the target text as auxiliary input allows contextual information to be incorporated into sign language recognition, resulting in more accurate recognition results.

[0020] The sign language recognition device and interactive system described in the embodiments of this specification also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] FIG1 exemplarily shows a flow chart of a sign language recognition method according to an embodiment of this specification in one implementation manner.

[0023] FIG2 exemplarily shows a flow chart of the sign language recognition method described in the embodiments of this specification being applied to a scenario.

[0024] FIG3 exemplarily shows a structural block diagram of a sign language recognition device according to an embodiment of this specification in one implementation manner.

[0025] FIG4 exemplarily shows a structural block diagram of an interactive system according to an embodiment of this specification in one implementation manner.

[0026] FIG5 exemplarily shows a structural diagram of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than those described in this specification. In addition, a single step described in this specification may be broken down into multiple steps for description in other embodiments, and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0029] The hearing-impaired population is an important part of the world's population, and all walks of life are making efforts to make life more convenient for the hearing-impaired in all aspects. For the hearing-impaired, sign language is an indispensable communication tool in daily life. Technologies such as sign language recognition and translation greatly facilitate communication between the hearing-impaired and other groups by converting sign language movements into corresponding text. However, the shooting angle and movement range of sign language movements may make sign language movements difficult to recognize, resulting in misidentification or missed recognition. If the visual signal corresponding to the sign language movement is used as input to translate the visual feature sequence into a text feature sequence, then the problems that may occur during the above-mentioned sign language movement acquisition process will affect the accuracy of the recognition results.

[0030] Therefore, this specification proposes a new sign language recognition solution. Based on the extraction of visual features of sign language movements, the motion posture data of sign language movements, such as hand sensor data, are introduced for calibration, thereby effectively improving the accuracy of sign language recognition.

[0031] The sign language recognition method and device described in the embodiments of this specification will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. However, this detailed description does not constitute a limitation to the embodiments of this specification.

[0032] It should be noted that the terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a," "an," "the," and "the" used in the embodiments of the present invention and the appended claims are intended to include the plural forms, unless the context clearly indicates otherwise.

[0033] In one embodiment of the present specification, a sign language recognition method is proposed. FIG1 exemplarily shows a flow chart of the sign language recognition method described in an embodiment of the present specification in one implementation manner.

[0034] As shown in FIG1 , the process includes the following steps.

[0035] S100: Acquire a video stream to be recognized, where the video stream to be recognized includes a continuous sequence of sign language action images.

[0036] The sign language video to be recognized can be captured by a device with image acquisition function, such as a mobile phone, camera, camcorder, etc., and the obtained video stream to be recognized can be transmitted to a designated location for sign language recognition, for example, by wired transmission or network transmission.

[0037] The video stream to be identified is a sign language action video. In sign language expression, commonly used words are represented by specific sign language actions, which can be static hand actions or continuous hand actions. Therefore, the video stream to be identified is divided into several continuous hand action image sequences, where each sign language action image sequence corresponds to a translated word. The translated word is a natural language word, and the type of language is not restricted. All sign language action image sequences constitute the video to be identified, and all corresponding translated words constitute the target sentence text.

[0038] S102: Obtain a hand movement gesture sequence corresponding to each sign language action image sequence in the video stream to be recognized.

[0039] In some embodiments, the hand motion sequence corresponding to the sign language gesture can be acquired using sensors installed on the hand, such as a gyroscope or accelerometer. When the position or posture of the hand changes, the corresponding sensor captures these changes and reflects them in the sensor data. Therefore, the hand motion sequence can be composed of sensor data, such as gyroscope data or accelerometer data.

[0040] S104: Input the video stream to be recognized into a pre-trained first recognition model to obtain a first recognition result; the first recognition result includes a first probability distribution of each target word in a preset vocabulary.

[0041] In some embodiments, the first recognition model can be constructed based on an encoder-decoder structure. A video stream to be recognized, including at least one continuous sequence of sign language action images, is input into the encoder of the first recognition model for encoding. Visual features corresponding to each sign language action image sequence are extracted, where each sign language action image sequence corresponds to a word after translation. The decoder then decodes the visual features corresponding to each sign language action image sequence to predict the sign language action translation result. The probability distribution of each word in a preset vocabulary being the target word corresponding to the sign language action is obtained as a first probability distribution, thereby obtaining a first recognition result.

[0042] The preset word list records target words that can be expressed with sign language gestures, including but not limited to common greetings such as thank you and goodbye, as well as common pronouns and nouns such as you, me, and him.

[0043] Optionally, the first recognition model can be constructed based on the Transformer network structure, and the encoder and decoder in the first recognition model can also adopt the Transformer network.

[0044] In some embodiments, the first recognition model is pre-trained in the following manner: obtaining a sample video stream, the sample video stream including a continuous sign language action image sequence sample; determining the first text corresponding to the sign language action image sequence sample, and using the first text as the first label of the sample video stream; training the first recognition model based on the sample video stream and the first label until a first recognition model that meets a preset stop condition is obtained.

[0045] The sample video stream contains at least one continuous sequence of sign language action image samples, each of which represents a complete word. The corresponding words are combined to form a first text corresponding to the sample video stream, which describes the meaning of the sign language actions in the sample video stream in text form. The first text is then used as the true label of the sample video stream. The sample video stream is input into a first recognition model, and a first recognition result is obtained through encoding and decoding. A first recognition loss is determined by calculating the difference between the first recognition result and the first label. The first recognition model is trained with the goal of minimizing this recognition loss.

[0046] In some more specific embodiments, the loss function of the first recognition loss can be a total probability distribution difference, that is, the difference between the probability distribution of each sign language action image sequence sample in the sample video stream predicted by the first recognition model and the corresponding word in the first text is calculated, and then the sum of all calculated differences is determined as the first recognition loss. For example, there are a total of 10 words in the first text. Then, when calculating the loss function, the difference between the probability distribution of each word and the probability distribution of the corresponding sign language action image sequence predicted by the first recognition model can be calculated for each word, and then the sum of the probability distribution differences of the 10 words is calculated to obtain the first recognition loss.

[0047] The first recognition model takes as input a video of the sign language gestures to be recognized and outputs the corresponding sign language recognition text. The video stream to be recognized is divided into at least one continuous sequence of sign language gesture images and then encoded and decoded. Each individual sign language gesture is translated and output, achieving word-by-word recognition.

[0048] In some embodiments, the video stream to be recognized is input into a pre-trained first recognition model to obtain a first recognition result, specifically including: when determining the prediction result of the first recognition model for the current target word, using the determined text content in the target text as the first auxiliary input of the first recognition model; based on the first auxiliary input and the current hand movement image sequence, determining the prediction result of the first recognition model for the current target word.

[0049] Since the first recognition model is used for sign language recognition, the output is word by word. When recognizing the second and subsequent words, the probability distribution of the recognized words can be input into the first recognition model together with the hand movement image sequence to assist in the prediction of subsequent words, and combine the context information to obtain more accurate and reasonable sign language recognition results.

[0050] S106: Input the hand movement posture sequence into a pre-trained second recognition model to obtain a second recognition result; the second recognition result includes a second probability distribution of each target word in the vocabulary.

[0051] In some embodiments, the second recognition model can be constructed based on an encoder-decoder structure. A sequence of hand motion gestures collected by a sensor is input into an encoder in the second recognition model for encoding. The hand motion features corresponding to each hand motion gesture sequence are extracted, where each hand motion gesture sequence corresponds to a sign language action and, accordingly, a word represented by the sign language action. The decoder then decodes the hand motion features corresponding to each hand motion gesture sequence to predict the sign language action translation result. The probability distribution of each word in a preset vocabulary being the target word corresponding to the sign language action is obtained as a second probability distribution, thereby obtaining a second recognition result.

[0052] To address the problem of inaccurate or incomplete sign language movement information recognition caused by issues such as the shooting angle of sign language videos or the range of sign language movements, introducing hand movement posture data to supplement it can improve the problem of misrecognition or missed recognition and enhance the accuracy of sign language recognition.

[0053] Optionally, a second recognition model can be constructed based on a Transformer network structure, and the encoder and decoder in the second recognition model can also adopt a Transformer network.

[0054] In some embodiments, the second recognition model is pre-trained in the following manner: determining a second text; obtaining a sequence sample of hand motion gestures when making a sign language action corresponding to the second text; using the second text as a second label for the sequence sample of hand motion gestures; and training the second model based on the sequence sample of hand motion gestures and the second label until a second recognition model that meets a preset stop condition is obtained.

[0055] When acquiring samples, it is necessary to first determine the second text as the true label. Then, using sensors or other tools, collect hand movement sequences when expressing the second text content in sign language as training samples. Each word in the second text corresponds to a hand movement sequence sample. The hand movement sequence samples are input into the second recognition model, and a second recognition result is obtained through encoding and decoding. The second recognition loss is determined by calculating the difference between the second recognition result and the second label. The second recognition model is trained with the goal of minimizing this recognition loss.

[0056] In some more specific embodiments, the loss function of the second recognition loss can be a total probability distribution difference, that is, the difference between the probability distribution of each hand motion posture sequence sample predicted by the second recognition model and the corresponding word in the second text is calculated respectively, and then the sum of all calculated differences is determined as the second recognition loss.

[0057] The second recognition model takes a sequence of hand gestures as input and outputs the corresponding sign language recognition text. After encoding and decoding the hand gesture sequences representing each sign language action, each individual sign language action is translated and output, achieving word-by-word recognition.

[0058] In some embodiments, hand motion posture data is input into a pre-trained second recognition model to obtain a second recognition result, specifically including: when determining the prediction result of the second recognition model for the current target word, the determined text content in the target text is used as the second auxiliary input of the second recognition model; based on the second auxiliary input and the current hand motion posture sequence, the prediction result of the second recognition model for the current target word is determined.

[0059] Since the second recognition model is used for sign language recognition, the output is word by word. When recognizing the second and subsequent words, the probability distribution of the recognized words can be input into the second recognition model together with the hand movement posture sequence samples to assist in the prediction of subsequent words, and combine the context information to obtain more accurate and reasonable sign language recognition results.

[0060] S108: Determine the target text based on the first recognition result and the second recognition result.

[0061] The target text is the words or sentences obtained after translating the sign language movements in the video stream to be recognized. The target text is a natural language word, and the language type is not restricted.

[0062] In some embodiments, based on the first recognition result and the second recognition result, the target text is determined, specifically including: for each target word in the target text, obtaining the first probability distribution of the target word predicted by the first recognition model, and obtaining the second probability distribution of the target word predicted by the second recognition model; performing weighted summation of the first probability distribution and the second probability distribution of the target word to obtain a third probability distribution of the target word; and determining the target word from the vocabulary based on the third probability distribution of the target word.

[0063] Since both the first recognition model and the second recognition model can recognize and output word by word, the first recognition result and the second recognition result respectively contain the probability distribution of each target word. Among them, the first recognition result is obtained based on the visual features of the sign language action. The visual features of the sign language action are the main basis for understanding the meaning of sign language when communicating using sign language, and should be placed in a more important position than the hand movement features. Therefore, in some more specific embodiments, when performing a weighted summation on the first probability distribution and the second probability distribution of the target word, a higher weight is assigned to the first probability distribution based on the visual features, and a lower weight is assigned to the second probability distribution based on the hand movement features, so that the first recognition model plays its main role and the second recognition model plays a fine-tuning role.

[0064] After obtaining a third probability distribution based on the first probability distribution and the second probability distribution of the target word, a predicted word with the highest probability is selected from the preset word list as the target word.

[0065] By supplementing sign language recognition with hand motion posture data, we can effectively make up for the lack of visual features in the process of sign language movement acquisition, such as misidentification or missed recognition, achieve the decoupling of visual data and motion posture data, and improve the accuracy and efficiency of sign language recognition.

[0066] The sign language recognition method provided in the embodiments of this specification obtains visual features of sign language through a sequence of sign language action images and introduces a sequence of sign language motion gestures to obtain hand motion features. The sign language recognition results obtained based on these two sign language features are then weighted to obtain the corresponding target text. This method effectively supplements data for misidentification and missed recognition caused by improper shooting angles or motion ranges, decoupling visual data from motion gesture data and making sign language recognition more lightweight. Furthermore, using identified words in the target text as auxiliary input allows contextual information to be incorporated into the sign language recognition process, resulting in more accurate recognition results.

[0067] The following further illustrates the sign language recognition method provided in this embodiment by taking the application of the sign language recognition method provided in this embodiment in a scenario as an example, but this description does not constitute a limitation on the embodiments of this specification.

[0068] FIG2 exemplarily shows a flow chart of the sign language recognition method described in the embodiments of this specification being applied to a scenario.

[0069] As shown in FIG2 , the sign language recognition method specifically includes the following steps.

[0070] S200: Acquire a sign language video stream to be recognized, where the sign language video stream to be recognized includes three consecutive sign language action image sequences.

[0071] S202: Acquire, through a hand sensor, a hand motion posture sequence corresponding to each sign language action image sequence in the video stream to be recognized.

[0072] S204: Input the video stream to be recognized into a pre-trained first recognition model, and after predicting the first word based on the first sign language action image sequence, continue to predict the second word based on the word and the second sign language action image sequence; then predict the third word based on the second word and the third sign language action image sequence, thereby obtaining a first recognition result; wherein the first recognition result includes a first probability distribution of each target word in the preset vocabulary.

[0073] S206: Input the hand motion posture sequence into a pre-trained second recognition model. After predicting the first word based on the first hand motion posture sequence, continue to predict the second word based on the word and the second hand motion posture sequence; then predict the third word based on the second word and the third hand motion posture sequence, thereby obtaining a second recognition result; wherein the second recognition result includes a second probability distribution of each target word in the preset vocabulary.

[0074] S208: For each target word in the target text, obtain its first probability distribution predicted by the first recognition model and the second probability distribution predicted by the second recognition model, set a weight of 0.8 for the first probability distribution, set a weight of 0.2 for the second probability distribution, and perform weighted summation to obtain the third probability distribution of the target word, and then determine the target word from the preset vocabulary based on the third probability distribution; finally, determine that the three target words are "I", "very", and "happy", and output the sign language recognition text as "I am very happy".

[0075] In another embodiment, the video stream to be recognized and the corresponding hand movement posture sequence can be spliced ​​together and input into a pre-trained third recognition model to obtain a third recognition result. The target text is determined based on the third recognition result to achieve multimodal sign language recognition.

[0076] Optionally, the third recognition model can adopt a Transformer-based encoder-decoder network structure, where both the encoder and decoder can be built based on Transformer. The video stream to be recognized and the corresponding hand movement posture sequence are input into the encoder, and the visual features and corresponding hand movement features of each sign language action are extracted respectively. The two features are fused to obtain the fused features of the sign language action; the fused features are input into the decoder for prediction, and the probability distribution of each word in the preset vocabulary being the target word corresponding to the sign language action is obtained, thereby obtaining the third recognition result and outputting the sign language recognition text.

[0077] The third recognition model can be pre-trained in the following manner: obtain a sample video stream, the sample video stream including a continuous sequence of sign language action image samples; determine a third text corresponding to the sign language action image sequence samples, and use the third text as a third label for the sample video stream; obtain a sequence of hand movement posture samples when making a sign language action corresponding to the third text; input the sample video stream and the corresponding sequence of hand movement posture samples into the third recognition model to obtain a sign language prediction result; train the third recognition model based on the sign language prediction result and the third label until a third recognition model that meets a preset stop condition is obtained.

[0078] The third recognition model is trained with the goal of minimizing the difference between the sign language prediction result and the third label.

[0079] In another embodiment of the present specification, a sign language recognition device is proposed. FIG3 exemplarily shows a structural block diagram of the sign language recognition device described in an embodiment of the present specification in one implementation manner.

[0080] As shown in Figure 3, it includes: a first data acquisition module 30, configured to acquire a video stream to be identified, which includes a continuous sequence of sign language action images; a second data acquisition module 32, configured to acquire a hand motion posture sequence corresponding to each sign language action sequence in the video stream to be identified; a first recognition module 34, configured to input the video stream to be identified into a pre-trained first recognition model to obtain a first recognition result; the first recognition result includes a first probability distribution of each target word in a preset vocabulary; a second recognition module 36, configured to input the hand motion posture sequence into a pre-trained second recognition model to obtain a second recognition result; the second recognition result includes a second probability distribution of each target word in the vocabulary; a text generation module 38, configured to determine the target text based on the first recognition result and the second recognition result.

[0081] The first data acquisition module can shoot the sign language video to be recognized through a device with image acquisition function, such as a mobile phone, camera, camcorder, etc., and transmit the obtained video stream to be recognized to a designated location for sign language recognition, for example, by wired transmission or network transmission.

[0082] The video stream to be identified is a sign language action video. In sign language expression, commonly used words are represented by specific sign language actions, which can be static hand actions or continuous hand actions. Therefore, the video stream to be identified is divided into several continuous hand action image sequences, where each sign language action image sequence corresponds to a translated word. The translated word is a natural language word, and the type of language is not restricted. All sign language action image sequences constitute the video to be identified, and all corresponding translated words constitute the target sentence text.

[0083] In some embodiments, the second data acquisition module can utilize sensors provided on the hand, such as a gyroscope or accelerometer, to acquire a sequence of hand motion gestures corresponding to the sign language gestures. When the position or posture of the hand changes, the corresponding sensor captures these changes and reflects them in the sensor data. Therefore, the sequence of hand motion gestures can be composed of sensor data, such as gyroscope data or accelerometer data.

[0084] In some embodiments, the first recognition model includes: a first encoder configured to extract features from a sequence of sign language action images in a video stream to be recognized to obtain visual features; and a first classifier configured to classify the visual features to obtain a first recognition result.

[0085] The first recognition module can construct a first recognition model based on an encoder-decoder structure, wherein the classifier can be an encoder. The first recognition module inputs a video stream to be recognized, which contains at least one continuous sequence of sign language action images, into the encoder of the first recognition model for encoding. The module extracts the visual features corresponding to each sign language action image sequence, wherein each sign language action image sequence corresponds to a word after translation. The classifier then classifies the visual features corresponding to each sign language action image sequence to predict the sign language action translation result. The probability distribution of each word in the preset vocabulary being the target word corresponding to the sign language action is obtained as the first probability distribution, thereby obtaining the first recognition result.

[0086] The preset word list records target words that can be expressed with sign language gestures, including but not limited to common greetings such as thank you and goodbye, as well as common pronouns and nouns such as you, me, and him.

[0087] Optionally, the first recognition model can be constructed based on the Transformer network structure, and the encoder and classifier in the first recognition model can also adopt the Transformer network.

[0088] In some embodiments, the first recognition module is pre-trained to obtain a first recognition model in the following manner: obtaining a sample video stream, the sample video stream including a continuous sign language action image sequence sample; determining a first text corresponding to the sign language action image sequence sample, and using the first text as the first label of the sample video stream; training the first recognition model based on the sample video stream and the first label until a first recognition model that meets a preset stop condition is obtained.

[0089] The sample video stream contains at least one continuous sequence of sign language action image samples, each of which represents a complete word. The corresponding words are combined to form a first text corresponding to the sample video stream, which describes the meaning of the sign language actions in the sample video stream in text form. The first text is then used as the true label of the sample video stream. The first recognition module inputs the sample video stream into the first recognition model, performs encoding and decoding, and obtains a first recognition result. The difference between the first recognition result and the first label is calculated to determine a first recognition loss. The first recognition model is trained with the goal of minimizing this recognition loss.

[0090] In some more specific embodiments, the loss function of the first recognition loss can be a total probability distribution difference, that is, the first recognition module calculates the difference between the probability distribution of each sign language action image sequence sample in the sample video stream predicted by the first recognition model and the corresponding words in the first text, and then determines the sum of all calculated differences as the first recognition loss.

[0091] The first recognition model takes as input a video of the sign language gestures to be recognized and outputs the corresponding sign language recognition text. The first recognition module divides the video stream into at least one continuous sequence of sign language gesture images and performs encoding and decoding. It translates and outputs each individual sign language gesture, achieving word-by-word recognition.

[0092] In some embodiments, the first recognition module is specifically used to use the determined text content in the target text as the first auxiliary input of the first recognition model when determining the prediction result of the first recognition model for the current target word; based on the first auxiliary input and the current hand movement image sequence, determine the prediction result of the first recognition model for the current target word.

[0093] Since the first recognition module outputs word by word when performing sign language recognition using the first recognition model, when recognizing the second and subsequent words, the probability distribution of the recognized words can be input into the first recognition model together with the hand movement image sequence to assist in the prediction of subsequent words, and more accurate and reasonable sign language recognition results can be obtained by combining contextual information.

[0094] In some embodiments, the second recognition model includes: a second encoder configured to extract features from a hand motion posture sequence to obtain hand motion features; and a second classifier configured to classify the hand motion features to obtain a second recognition result.

[0095] The second recognition module can construct a second recognition model based on an encoder-decoder structure, wherein the classifier can be an encoder. The second recognition module inputs the hand motion posture sequence collected by the sensor into the encoder in the second recognition model for encoding, and extracts the hand motion features corresponding to each hand motion posture sequence, wherein each hand motion posture sequence corresponds to a sign language action and, accordingly, also corresponds to a word represented by the sign language action. The classifier then classifies the hand motion features corresponding to each hand motion posture sequence to predict the sign language action translation result, and obtains the probability distribution of each word in the preset vocabulary being the target word corresponding to the sign language action as the second probability distribution, thereby obtaining the second recognition result.

[0096] To address the problem of inaccurate or incomplete sign language movement information recognition caused by issues such as the shooting angle of the sign language video or the range of sign language movements, the second recognition module introduces hand movement posture data to supplement the information, which can improve the problem of misrecognition or missed recognition and enhance the accuracy of sign language recognition.

[0097] Optionally, a second recognition model can be constructed based on a Transformer network structure, and the encoder and classifier in the second recognition model can also adopt a Transformer network.

[0098] In some embodiments, the second recognition module is pre-trained to obtain a second recognition model in the following manner: determining a second text; obtaining a sequence sample of hand motion gestures when making a sign language action corresponding to the second text; using the second text as a second label of the hand motion gesture sequence sample; and training the second model based on the hand motion gesture sequence sample and the second label until a second recognition model that meets a preset stop condition is obtained.

[0099] When acquiring samples, the second recognition module first determines the second text as the true label. It then uses sensors and other tools to collect sequences of hand gestures used when expressing the second text in sign language as training samples. Each word in the second text corresponds to a sequence of hand gesture samples. The hand gesture sequence samples are input into the second recognition model, and a second recognition result is obtained through encoding and decoding. A second recognition loss is determined by calculating the difference between the second recognition result and the second label. The second recognition model is trained with the goal of minimizing this recognition loss.

[0100] In some more specific embodiments, the loss function of the second recognition loss can be a total probability distribution difference, that is, the difference between the probability distribution of each hand motion posture sequence sample predicted by the second recognition model and the corresponding word in the second text is calculated respectively, and then the sum of all calculated differences is determined as the second recognition loss.

[0101] The second recognition model takes a sequence of hand gestures as input and outputs the corresponding sign language recognition text. The second recognition module encodes and decodes the hand gesture sequences representing each sign language action, then translates and outputs each individual sign language action, achieving word-by-word recognition.

[0102] In some embodiments, the second recognition module is specifically used to use the determined text content in the target text as the second auxiliary input of the second recognition model when determining the prediction result of the second recognition model for the current target word; based on the second auxiliary input and the current hand movement posture sequence, determine the prediction result of the second recognition model for the current target word.

[0103] Since the second recognition model is used for sign language recognition, the output is word by word. When recognizing the second and subsequent words, the probability distribution of the recognized words can be input into the second recognition model together with the hand movement posture sequence samples to assist in the prediction of subsequent words, and combine the context information to obtain more accurate and reasonable sign language recognition results.

[0104] The target text is the words or sentences obtained after translating the sign language movements in the video stream to be recognized. The target text is a natural language word, and the language type is not restricted.

[0105] In some embodiments, the text generation module is specifically used to obtain, for each target word in the target text, a first probability distribution of the target word predicted by the first recognition model, and a second probability distribution of the target word predicted by the second recognition model; perform weighted summation on the first probability distribution and the second probability distribution of the target word to obtain a third probability distribution of the target word; and determine the target word from the vocabulary based on the third probability distribution of the target word.

[0106] Since both the first recognition model and the second recognition model can recognize and output word by word, the first recognition result and the second recognition result respectively contain the probability distribution of each target word. Among them, the first recognition result is obtained based on the visual features of the sign language action. The visual features of the sign language action are the main basis for understanding the meaning of sign language when communicating using sign language, and should be placed in a more important position than the hand movement features. Therefore, in some more specific embodiments, when the text generation module performs a weighted summation of the first probability distribution and the second probability distribution of the target word, a higher weight is assigned to the first probability distribution based on the visual features, and a lower weight is assigned to the second probability distribution based on the hand movement features, so that the first recognition model plays its main role and the second recognition model plays a fine-tuning role.

[0107] After obtaining a third probability distribution based on the first probability distribution and the second probability distribution of the target word, the text generation module selects a predicted word with the highest probability from a preset word list as the target word.

[0108] The text generation module supplements sign language recognition with hand motion posture data, which can effectively make up for the lack of visual features in the sign language action acquisition process, such as misidentification or missed recognition, realize the decoupling of visual data and motion posture data, and improve the accuracy and efficiency of sign language recognition.

[0109] An embodiment of this specification further proposes an interactive system. FIG4 exemplarily shows a structural block diagram of the interactive system described in an embodiment of this specification in one implementation manner.

[0110] As shown in Figure 4, it includes: a sign language action acquisition module 40, configured to obtain a sign language action video to be recognized; a sign language action recognition module 42, configured to recognize the sign language action video based on the steps described in any one of the above-mentioned sign language recognition methods to obtain a target text; an information generation module 44, configured to convert the target text into a preset type of information and send the information to the user.

[0111] The sign language motion acquisition module can capture a video of the sign language to be recognized using a terminal device with image acquisition capabilities, such as a mobile phone, camera, or camcorder. The acquired video stream to be recognized is then transmitted to the interactive system via wired or network transmission for sign language recognition. Based on the sign language motion video, the sign language motion acquisition module extracts a continuous sequence of sign language motion images and corresponding hand movement gesture sequences, which are then sent to the sign language motion recognition module. Each word represented by a sign language motion corresponds to a sequence of sign language motion images and a sequence of hand movement gestures.

[0112] The sign language action recognition module inputs the received sign language action image sequence and the corresponding hand movement posture sequence into any pre-trained recognition model in the above-mentioned sign language recognition method to recognize the sign language action video, obtains the target text, and transmits the target text to the information generation module.

[0113] The information generation module converts the target text into information in the form of text, voice, vibration, and other suitable combinations in various languages, and sends it to the user through terminal devices such as mobile phones and tablets to complete sign language recognition.

[0114] One embodiment of this specification further provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned sign language recognition method when executed by a processor.

[0115] One embodiment of this specification also provides an electronic device, including: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, which, when read and executed by the one or more processors, execute the steps of the above-mentioned sign language recognition method.

[0116] Figure 5 exemplarily illustrates a block diagram of an electronic device provided in accordance with an embodiment of this specification, which illustrates a schematic diagram of the structure of a computer system 500 suitable for implementing a terminal device or server according to an embodiment of the present invention. The terminal device or server shown in Figure 5 is merely an example and should not limit the functionality or scope of use of the embodiments of the present invention.

[0117] As shown in FIG5 , the computer system 500 includes a central processing unit (CPU) 502, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 504 or a program loaded from a storage unit 516 into a random access memory (RAM) 506. Various programs and data required for the operation of the system 500 are also stored in the RAM 506. The CPU 502, the ROM 504, and the RAM 506 are connected to each other via a bus 508. An input / output (I / O) interface 510 is also connected to the bus 508.

[0118] The following components are connected to the I / O interface 510: an input section 512 including a keyboard, a mouse, and the like; an output section 514 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 516 including devices such as a hard disk; and a communication section 518 including a network interface card such as a LAN card or a modem. The communication section 518 performs communication processing via a network such as the Internet. A drive 520 is also connected to the I / O interface 510 as needed. A removable medium 522, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 520 as needed, so that computer programs read therefrom can be installed into the storage section 516 as needed.

[0119] In particular, according to embodiments disclosed herein, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments disclosed herein include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such embodiments, the computer program may be downloaded and installed from a network via the communication portion 518, and / or installed from a removable medium 522. When the computer program is executed by the central processing unit (CPU) 502, the above-described functions defined in the method of the present invention are performed.

[0120] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0121] The foregoing description describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous. It should also be noted that each box in the accompanying drawings, and the combination of boxes in the accompanying drawings, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or may be implemented using a combination of dedicated hardware and computer instructions.

[0122] It should be noted that the above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples, and many similar variations are possible. All variations directly derived from or associating with the present invention by those skilled in the art are intended to fall within the scope of protection of the present invention.

Claims

1. A sign language recognition method, comprising: Acquire a video stream to be identified, wherein the video stream to be identified includes a continuous sequence of sign language action images; Acquire a hand movement posture sequence corresponding to each sign language action image sequence in the video stream to be recognized; Inputting the video stream to be recognized into a pre-trained first recognition model to obtain a first recognition result; the first recognition result includes a first probability distribution of each target word in a preset vocabulary; Inputting the hand motion gesture sequence into a pre-trained second recognition model to obtain a second recognition result; the second recognition result includes a second probability distribution of each target word in the vocabulary; A target text is determined based on the first recognition result and the second recognition result.

2. The method according to claim 1, wherein the first recognition model is pre-trained in the following manner: Acquire a sample video stream, wherein the sample video stream includes continuous sign language action image sequence samples; Determine a first text corresponding to the sign language action image sequence sample, and use the first text as a first label of the sample video stream; The first recognition model is trained according to the sample video stream and the first label until the first recognition model that meets a preset stop condition is obtained.

3. The method according to claim 1, wherein the second recognition model is pre-trained in the following manner: Determine the second text; Acquire a sequence sample of hand movement gestures when making a sign language action corresponding to the second text; Using the second text as a second label of the hand motion posture sequence sample; The second model is trained according to the hand motion posture sequence samples and the second label until the second recognition model that meets the preset stop condition is obtained.

4. The method according to claim 1, wherein inputting the video stream to be recognized into a pre-trained first recognition model to obtain a first recognition result specifically comprises: When determining the prediction result of the first recognition model for the current target word, using the determined text content in the target text as a first auxiliary input of the first recognition model; Based on the first auxiliary input and the current hand motion image sequence, a prediction result of the first recognition model for the current target word is determined.

5. The method according to claim 1, wherein inputting the hand motion posture data into a pre-trained second recognition model to obtain a second recognition result specifically comprises: When determining the prediction result of the second recognition model for the current target word, using the determined text content in the target text as a second auxiliary input of the second recognition model; Based on the second auxiliary input and the current hand movement posture sequence, a prediction result of the second recognition model for the current target word is determined.

6. The method according to claim 1, wherein determining the target text based on the first recognition result and the second recognition result specifically comprises: For each target word in the target text, obtaining the first probability distribution of the target word predicted by the first recognition model, and obtaining the second probability distribution of the target word predicted by the second recognition model; Performing a weighted summation on the first probability distribution and the second probability distribution of the target word to obtain a third probability distribution of the target word; Based on the third probability distribution of the target word, the target word is determined from the vocabulary.

7. A sign language recognition device, comprising: A first data acquisition module is configured to acquire a video stream to be identified, wherein the video stream to be identified includes a continuous sequence of sign language action images; A second data acquisition module is configured to acquire a hand movement posture sequence corresponding to each sign language action sequence in the video stream to be recognized; A first recognition module is configured to input the video stream to be recognized into a pre-trained first recognition model to obtain a first recognition result; the first recognition result includes a first probability distribution of each target word in a preset vocabulary; A second recognition module is configured to input the hand motion gesture sequence into a pre-trained second recognition model to obtain a second recognition result; the second recognition result includes a second probability distribution of each target word in the vocabulary; The text generation module is configured to determine a target text based on the first recognition result and the second recognition result.

8. The apparatus according to claim 7, wherein the first recognition model comprises: A first encoder is configured to extract features from the sign language action image sequence in the video stream to be recognized to obtain visual features; The first classifier is configured to classify the visual features to obtain the first recognition result.

9. The apparatus according to claim 8, wherein the second recognition model comprises: a second encoder configured to extract features from the hand motion posture sequence to obtain hand motion features; The second classifier is configured to classify the hand movement feature to obtain the second recognition result.

10. An interactive system comprising: A sign language action acquisition module, configured to acquire a sign language action video to be recognized; The sign language action recognition module is configured to recognize the sign language based on the method described in any one of claims 1 to 6. Recognize the action video and get the target text; The information generation module is configured to convert the target text into information of a preset type and send the information to the user.

11. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 6.

12. An electronic device comprising: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method as claimed in any one of claims 1 to 6.

Images