Multimodal sign language recognition method and device

Abstract

The present application discloses a multimodal sign language recognition method and device. The method comprises: obtaining the motion information of the user's first hand and at least one set of continuous screen images; performing region segmentation on the at least one set of continuous screen images to obtain at least one set of first continuous facial images corresponding to the facial region, and at least one set of first continuous motion images corresponding to the second hand region; filtering the motion information to obtain sign language feature data, wherein the sign language feature data comprises a first muscle sound signal of at least one channel of the first hand; obtaining a sign language recognition result based on the first muscle sound signal of at least one channel, the at least one set of first continuous facial images, and the at least one set of first continuous motion images.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a multimodal sign language recognition method and device. Background Art

[0002] There are a large number of deaf-mute people in the world. They mainly communicate through sign language, but there are serious obstacles when communicating with people who have no sign language learning foundation. At present, there are many methods of sign language recognition. For example, the sign language made by the deaf-mute is filmed by a camera, and then the sign language movements in the filmed video are recognized to obtain the sign language recognition results. However, when the user's two hands are expressing in sign language at the same time, the user's sign language movements may not be accurately recognized due to the influence of shooting light and shooting angle, which reduces the accuracy of sign language recognition and cannot accurately understand the meaning of the user's expression. Summary of the invention

[0003] The embodiments of the present application provide a multimodal sign language recognition method and apparatus, which respectively identify the motion information of the user's first hand, the motion image of the second hand, and the facial image through a wearable device, and jointly determine the sign language recognition result, thereby improving the accuracy of sign language recognition and accurately understanding the meaning that the user wants to express.

[0004] In a first aspect, an embodiment of the present application provides a multimodal sign language recognition method, which is applied to a wearable device, and the method includes:

[0005] Acquiring motion information of a first hand of a user and at least one set of continuous screen images, wherein the wearable device is worn on the first hand, the motion information is collected by at least one acceleration sensor of the wearable device when the user is performing a sign language motion, and the at least one set of continuous screen images is captured by at least one camera of the wearable device when the user is performing a sign language motion, and the at least one camera corresponds to the at least one set of continuous screen images in a one-to-one manner;

[0006] Performing region segmentation on the at least one group of continuous screen images to obtain at least one group of first continuous facial images corresponding to the facial region and at least one group of first continuous action images corresponding to the second hand region;

[0007] Filtering the motion information to obtain sign language feature data, wherein the sign language feature data includes a first muscle sound signal of at least one channel of a first hand, and the at least one acceleration sensor corresponds one-to-one to the first muscle sound signal of the at least one channel;

[0008] A sign language recognition result is obtained based on the first muscle sound signal of the at least one channel, the at least one group of first continuous facial images, and the at least one group of first continuous action images.

[0009] In a second aspect, an embodiment of the present application provides a wearable device, the wearable device comprising: a transceiver unit and a processing unit;

[0010] The transceiver unit is used to obtain motion information of a first hand of a user and at least one set of continuous screen images, wherein the wearable device is worn on the first hand, the motion information is collected by at least one acceleration sensor of the wearable device when the user is performing a sign language motion, and the at least one set of continuous screen images is captured by at least one camera of the wearable device when the user is performing a sign language motion, and the at least one camera corresponds to the at least one set of continuous screen images in a one-to-one manner;

[0011] The processing unit is used to perform region segmentation on the at least one group of continuous screen images to obtain at least one group of first continuous facial images corresponding to the facial region and at least one group of first continuous action images corresponding to the second hand region;

[0012] The processing unit is configured to filter the motion information to obtain sign language feature data, wherein the sign language feature data includes a first muscle sound signal of at least one channel of a first hand, and the at least one acceleration sensor corresponds one-to-one to the first muscle sound signal of the at least one channel;

[0013] The processing unit is used to obtain a sign language recognition result based on the first muscle sound signal of the at least one channel, the at least one set of first continuous facial images and the at least one set of first continuous action images.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, the processor being connected to the memory, the memory being used to store a computer program, and the processor being used to execute the computer program stored in the memory, so that the electronic device executes the method of the first aspect.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program enables a computer to execute the method of the first aspect.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and is computer-operable to cause the computer to execute the method of the first aspect.

[0017] Implementing the embodiments of the present application has the following beneficial effects:

[0018] It can be seen that in an embodiment of the present application, by obtaining the motion information of the user's first hand and at least one group of continuous screen images, wherein the wearable device is worn on the first hand, the motion information is collected by at least one accelerometer of the wearable device when the user is performing sign language motions, and the at least one group of continuous screen images is taken by at least one camera of the wearable device when the user is performing sign language motions, and the at least one camera corresponds to the at least one group of continuous screen images one by one; the at least one group of continuous screen images is segmented to obtain at least one group of first continuous facial images corresponding to the facial area, and at least one group of first continuous motion images corresponding to the second hand area; the motion information is filtered to obtain sign language feature data, wherein the sign language feature data includes a first muscle sound signal of at least one channel of the first hand, and the at least one accelerometer corresponds to the first muscle sound signal of the at least one channel one by one; based on the first muscle sound signal of the at least one channel, the at least one group of first continuous facial images and the at least one group of first continuous motion images, a sign language recognition result is obtained. The wearable device is used to respectively identify the movement information of the two hands and the facial image of the user during the sign language action to jointly determine the sign language recognition result, which specifically includes the first muscle sound signal of at least one channel of the first hand, at least one group of first continuous facial images and at least one group of first continuous action images of the second hand, to obtain the sign language recognition result, integrate multiple data information, improve the accuracy of the sign language recognition result, and accurately understand the meaning that the user wants to express; further, compared with the traditional method of shooting with a camera or others, the user can directly shoot the sign language action through the wearable device, which improves the convenience of the user's use of sign language recognition, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A schematic diagram of a multimodal sign language recognition system provided in an embodiment of the present application;

[0021] Figure 2 A flowchart of a multimodal sign language recognition method provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of skeleton key point extraction provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of a sign language recognition model training provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of a flow chart of another multimodal sign language recognition method provided in an embodiment of the present application;

[0025] Figure 6 A functional unit composition block diagram of a wearable device provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] The terms "first", "second", "third" and "fourth" etc. in the specification, claims and drawings of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0029] Reference to "embodiments" herein means that a particular feature, result, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0030] In order to facilitate understanding of the technical solution of the present application, the relevant technical terms involved in the present application are first explained.

[0031] It should be noted that the wearable device in the embodiment of the present application refers to a portable device that can be directly worn on the forearm of a user. The wearable device realizes the communication function through a built-in chip to complete the transmission of data with the server. For example, the wearable device may include a smart watch, a smart wristband, etc., and the present invention is not limited to this.

[0032] It should be noted that the user in this application refers to the wearer of the wearable device, the acceleration sensor in this application is a three-axis acceleration sensor, and the first hand in this application refers to the left hand or right hand of the user.

[0033] See also Figure 1 , Figure 1 A schematic diagram of a multimodal sign language recognition system provided in an embodiment of the present application. The multimodal sign language recognition system includes a wearable device 101, a server 102, an acceleration sensor 1011, a camera 1012, and a voice announcer 1013.

[0034] It should be stated that Figure 1 In the description, only one accelerometer 1011 and one camera 1012 are taken as examples, and the present application does not limit the number of accelerometers and cameras. The server 102 receives the motion information of the user's first hand and at least one set of continuous screen images from the wearable device 101, wherein the wearable device 101 is worn on the first hand, and the motion information is collected by at least one accelerometer of the wearable device 101 when the user is performing sign language motions, and the at least one set of continuous screen images is taken by at least one camera of the wearable device when the user is performing sign language motions, and the at least one camera corresponds to the at least one set of continuous screen images one by one; the at least one set of continuous screen images is segmented into regions to obtain at least one set of first images corresponding to the facial region. A continuous facial image, and at least one group of first continuous action images corresponding to a second hand area; filtering the action information to obtain sign language feature data, wherein the sign language feature data includes a first muscle sound signal of at least one channel of the first hand, and the at least one acceleration sensor corresponds one-to-one to the first muscle sound signal of the at least one channel; based on the first muscle sound signal of the at least one channel, the at least one group of first continuous facial images and the at least one group of first continuous action images, obtaining a sign language recognition result; the server 102 sends the sign language recognition result to the wearable device 101; the voice announcer 1013 plays the sign language recognition result.

[0035] It should be noted that in the present application, the wearable device 101 can also execute the method executed by the server 102, and the present application does not limit this. That is, the wearable device 101 obtains the motion information of the user's first hand and at least one group of continuous screen images, wherein the wearable device 101 is worn on the first hand, the motion information is collected by at least one acceleration sensor of the wearable device 101 when the user is performing a sign language motion, and the at least one group of continuous screen images is taken by at least one camera of the wearable device when the user is performing a sign language motion, and the at least one camera corresponds to the at least one group of continuous screen images one by one; the wearable device 101 performs region segmentation on the at least one group of continuous screen images to obtain at least one group of first continuous facial images corresponding to the facial region, and at least one group of first continuous motion images corresponding to the second hand region; the wearable device 101 filters the motion information to obtain sign language feature data, wherein the sign language feature data includes a first muscle sound signal of at least one channel of the first hand, and the at least one acceleration sensor corresponds to the first muscle sound signal of the at least one channel one by one; the wearable device 101 obtains a sign language recognition result based on the first muscle sound signal of the at least one channel, the at least one group of first continuous facial images, and the at least one group of first continuous motion images.

[0036] It can be seen that in an embodiment of the present application, by obtaining the motion information of the user's first hand and at least one group of continuous screen images, wherein the wearable device is worn on the first hand, the motion information is collected by at least one accelerometer of the wearable device when the user is performing sign language motions, and the at least one group of continuous screen images is taken by at least one camera of the wearable device when the user is performing sign language motions, and the at least one camera corresponds to the at least one group of continuous screen images one by one; the at least one group of continuous screen images is segmented into regions to obtain at least one group of first continuous facial images corresponding to the facial region, and at least one group of first continuous motion images corresponding to the second hand region; the motion information is filtered to obtain sign language feature data, wherein the sign language feature data includes a first muscle sound signal of at least one channel of the first hand, and the at least one accelerometer corresponds to the first muscle sound signal of the at least one channel one by one; based on the first muscle sound signal of the at least one channel, the at least one group of first continuous facial images and the at least one first motion image, a sign language recognition result is obtained. The movement information of the first hand, the movement image of the second hand and the facial image of the user during the sign language movement are respectively identified to jointly determine the sign language recognition result, which specifically includes the first muscle sound signal of at least one channel of the first hand, at least one group of first continuous facial images and at least one group of first continuous movement images of the second hand, to obtain the sign language recognition result, integrate multiple data information, improve the accuracy of the sign language recognition result, and accurately understand the meaning that the user wants to express; further, compared with the traditional method of shooting with a camera or others, the user can directly shoot the sign language movement through the wearable device, which improves the convenience of the user in using sign language recognition, thereby improving the user experience.

[0037] See also Figure 2 , Figure 2 A flowchart of a multimodal sign language recognition method provided in an embodiment of the present application, the method includes but is not limited to steps 201-208:

[0038] 201: In response to a touch operation of a user, the wearable device collects first hand motion information of the user through at least one acceleration sensor.

[0039] 202: The wearable device collects at least one set of continuous screen images of the user through at least one camera.

[0040] If the user communicates with someone who has no sign language learning foundation, the user can choose to turn on the sign language recognition function of the wearable device. Specifically, the user can click on the first control, which can be a virtual function button corresponding to the sign language recognition function. In response to the user's touch operation, the wearable device collects the motion information of the first hand and at least one set of continuous screen images through at least one camera and at least one acceleration sensor.

[0041] Furthermore, when the recognition starts, if all cameras among the at least one camera have not captured the facial image of the user, the vibration will be performed with a preset intensity and a prompt message will be displayed until at least one camera among the at least one camera has captured the facial image of the user, wherein the prompt message is used to remind the user to adjust the position of the wearable device, and the prompt message may be "Please adjust the wearing posture of the wearable device so that your face is within the shooting range of the camera."

[0042] A wearable device is worn on the first hand, and motion information is collected by at least one acceleration sensor of the wearable device when the user is performing sign language motions. At least one set of continuous screen images is collected by at least one camera of the wearable device when the user is performing sign language motions, and the at least one camera corresponds one-to-one to the at least one set of continuous screen images.

[0043] 203: The wearable device sends the user's first hand motion information and at least one set of continuous screen images to the server.

[0044] Among them, a wearable device is worn on the first hand, the motion information is collected by at least one acceleration sensor of the wearable device when the user is performing sign language movements, at least one group of continuous screen images is collected by at least one camera of the wearable device when the user is performing sign language movements, at least one camera corresponds to at least one group of continuous screen images, and each camera collects a group of continuous screen images.

[0045] 204: The server performs region segmentation on at least one group of continuous screen images to obtain at least one group of first continuous facial images corresponding to the facial region and at least one group of first continuous action images corresponding to the second hand region.

[0046] Exemplarily, in order to capture complete motion images of the user's second hand and face, multiple cameras can be deployed in the wearable device, and multiple cameras can be used to capture images of the user's second hand and face based on a preset shooting frequency. Each camera will capture a set of continuous screen images to obtain multiple sets of first continuous screen images.

[0047] Optionally, in order to segment the facial area and the second hand area in the multiple groups of first continuous screen images and identify them separately, the Mask R-CNN network model can be used to detect the facial area and the second hand area in the first continuous screen images, perform second hand segmentation and facial segmentation, and obtain at least one group of first continuous facial images corresponding to the facial area, and at least one group of first continuous action images corresponding to the second hand area. If there is no facial image or second hand image in some of the first continuous screen images, a blank image is used instead.

[0048] 205: The server filters the action information to obtain sign language feature data.

[0049] It should be noted that the sign language feature data includes a first muscle sound signal of at least one channel of a first hand, and at least one acceleration sensor corresponds one-to-one to the first muscle sound signal of at least one channel.

[0050] Optionally, the server pre-trains the SVM classifier, specifically including: first, collecting multiple original third muscle sound signals, wherein the multiple original third muscle sound signals are muscle sound signals of the user during daily sign language actions collected in advance after the user wears the wearable device, so as to train the initial SVM classifier. In addition, multiple original third muscle sound signals can also be obtained from the sign language action muscle sound signal library. The original third muscle sound signal is filtered by a filter to obtain a third muscle sound signal. The third muscle sound signal is segmented to obtain multiple fourth muscle sound signals, and each fourth muscle sound signal has its corresponding sign language action description. Specifically. The third muscle sound signal is segmented using the signal secondary envelope method, and each sign language action has a corresponding envelope, and the positions corresponding to the two minimum points before and after the envelope are the start and end positions of the action. And a first label is added to each fourth muscle sound signal, wherein the first label of each fourth muscle sound signal represents the sign language action description corresponding to the fourth muscle sound signal.

[0051] Optionally, for the kth fourth muscle sound signal, wavelet energy feature extraction is performed on the kth fourth muscle sound signal, and the wavelet packet transform decomposes the kth fourth muscle sound signal into subspaces of equal bandwidth in the form of a binary tree, and when the number of signal decomposition layers is z layers, there are a total of 2 z The energy of each sub-band in the last layer after decomposition is expressed by formula (1):

[0052]

[0053] Among them, z is the number of decomposition levels, N z is the number of sub-band signals in the zth layer, E z is the energy of each sub-band in the last layer after decomposition, S z,jIt is the jth signal sub-band in the zth layer after wavelet packet decomposition.

[0054] It should be noted that in this application, the decomposition layer number is 4 layers as an example, then the wavelet packet coefficient energy feature of the muscle sound signal of one channel is 2 4 , i.e. 16 dimensions. The wavelet packet coefficient energy feature is subjected to dimensionality reduction processing to obtain a three-dimensional feature vector corresponding to the kth fourth muscle sound signal. Specifically, the wavelet packet coefficient energy feature can be subjected to dimensionality reduction processing by a linear discriminant analysis method to obtain a three-dimensional feature vector.

[0055] Based on the method of obtaining the three-dimensional feature vector of the kth fourth muscle sound signal, the three-dimensional feature vector corresponding to each fourth muscle sound signal is obtained, and the corresponding first label is added to each three-dimensional feature vector. The three-dimensional feature vector corresponding to each fourth muscle sound signal and its corresponding first label are input into the initial SVM classifier for training until the model converges to obtain a trained SVM classifier.

[0056] Furthermore, the action information is filtered to obtain sign language feature data, wherein the sign language feature data includes a first muscle sound signal of at least one channel of the first hand, and the at least one acceleration sensor corresponds to the first muscle sound signal of the at least one channel. Specifically, taking the first muscle sound signal of a channel as an example, the first muscle sound signal of a channel is segmented to obtain at least one second muscle sound signal, wherein each second muscle sound signal corresponds to a sign language action description; for the i-th second muscle sound signal, the i-th second muscle sound signal is decomposed into 2 equal bandwidths. z sub-band signals, wherein z is the number of decomposition layers, and the i-th second muscle sound signal is any one of the at least one second muscle sound signal; based on each sub-band signal, the signal length of each sub-band signal and the number of decomposition layers, the energy feature of the i-th second muscle sound signal is determined; based on the energy feature of the i-th second muscle sound signal, the energy feature of each second muscle sound signal is determined; based on the linear discriminant analysis method, the energy feature of each second muscle sound signal is reduced in dimension to obtain the energy feature after dimension reduction, wherein the energy feature after dimension reduction is a three-dimensional feature vector; the energy feature after dimension reduction is input into the trained SVM classifier to obtain at least one sign language action description. Specifically, the i-th second muscle sound signal is reduced in dimension to obtain the three-dimensional feature vector of the i-th second muscle sound signal, and the three-dimensional feature vector of the i-th second muscle sound signal is input into the trained SVM classifier to obtain the sign language action description corresponding to the i-th second muscle sound signal. According to the method for obtaining the corresponding sign language action description of the i-th second muscle sound signal, the sign language action description corresponding to each second muscle sound signal in at least one second muscle sound signal is obtained.

[0057] It should be noted that before the user uses the sign language recognition function of the wearable device, at least one accelerometer on the wearable device needs to collect muscle sound signals. In the present application, the number of channels is the same as the number of accelerometers. Two accelerometers are used as an example. If there are two accelerometers, the first muscle sound signals of the two channels will be collected. The processing of the first muscle sound signals of the two channels is similar to the processing method of the first muscle sound signal of the one channel mentioned above, and this application will not go into details here.

[0058] It can be seen that in the embodiment of the present application, the first hand movement classification is completed by measuring the first muscle sound signal of at least one channel of the first hand, and at least one sign language movement description of the first hand is obtained. Classification and recognition are performed based on a pre-trained SVM classifier, which can speed up the recognition efficiency. The user does not need to perform additional operations, but only needs to concentrate on expressing in sign language, and the user's first hand movement can be accurately recognized, which improves the convenience of the user in using the wearable device.

[0059] 206: The server obtains a sign language recognition result based on the first muscle sound signal of at least one channel, at least one set of first continuous facial images, and at least one set of first continuous motion images.

[0060] In order to remove redundant information in multiple groups of first continuous action images, such as removing incomplete action images of the second hand, and removing action images with resolutions lower than a preset resolution, it is necessary to group the multiple groups of first continuous action images again according to time to obtain multiple second action images corresponding to each time. For multiple second action images at the jth time, identify the skeletal key points in each second action image, wherein the jth time is any time when the user is performing a sign language action; determine the number of skeletal key points in each second action image at the jth time; the second action image with the largest number of skeletal key points among the multiple second action images at the jth time is the second action image corresponding to the jth time; based on the second action image corresponding to the jth time, determine the second action image corresponding to each time when the user is performing the sign language action; sort the second action images corresponding to each time in a time series to obtain a second continuous action image. For example, Figure 3 As shown, taking two sets of first continuous action images as an example, two second action images at the jth moment are obtained, which are Figure 3The second action image in the first group of continuous action images, and the second action image in the second group of continuous action images, respectively determine the skeleton key points in the second action image in the first group of continuous action images, and the second action image in the second group of continuous action images, obtain a first skeleton key point map and a second skeleton key point map, determine the number of skeleton key points in the first skeleton key point map and the number of skeleton key points in the second skeleton key point map, such as Figure 3 As shown, the number of bone key points in the first bone key point map is greater than the number of bone key points in the second bone key point map, so the second action image in the first group of continuous action images is the second action image corresponding to the jth moment.

[0061] Optionally, if there are multiple groups of first continuous facial images, obtaining the second continuous facial images based on the second continuous action images and the multiple groups of first continuous facial images means that the second continuous facial images are determined based on the multiple groups of first continuous facial images according to the determination method of the second continuous action images. Specifically, if there are multiple groups of first continuous facial images, the multiple groups of first continuous facial images are grouped again according to time to obtain multiple second facial images corresponding to each time; for the multiple second facial images at the mth time, the skeletal key points in each second facial image at the mth time are identified, wherein the mth time is any time when the user is performing a sign language action; the number of skeletal key points in each second facial image at the mth time is determined; the second facial image with the largest number of skeletal key points among the multiple second facial images at the mth time is the second facial image corresponding to the mth time; based on the second facial image corresponding to the mth time, the second facial image corresponding to each time when the user is performing a sign language action is determined; the second facial images at each time are sorted in time series to obtain the second continuous facial images.

[0062] It can be seen that in the present application, if there are multiple groups of first continuous action images, the multiple groups of first continuous action images are grouped again according to the time to obtain multiple second action images corresponding to each time; for multiple second action images at the jth time, the skeleton key points in each second action image are identified, wherein the jth time is any time when the user is performing the sign language action; the number of skeleton key points in each second action image is determined; the second action image with the largest number of skeleton key points among the multiple second action images is the second action image corresponding to the jth time; based on the second action image, the second action image corresponding to each time when the user is performing the sign language action is determined; the second action image at each time is sorted according to the time series to obtain the second continuous action image; if there are multiple groups of first continuous facial images, the second continuous facial image is obtained based on the method for obtaining the second continuous action image. At the nth time, the image with the most skeleton key points among the multiple groups of first continuous action images and the multiple groups of first continuous facial images is screened out, which is the second action image and the second facial image corresponding to the nth time, which can ensure that the most complete second hand action image and facial image are obtained, thereby improving the accuracy of sign language recognition.

[0063] Furthermore, a sign language recognition result is obtained based on at least one sign language action description, the second continuous facial images and the second continuous action images.

[0064] Obtain at least one third action image corresponding to the end moment of each second muscle sound signal in the multiple second muscle sound signals from the second continuous action images; for the hth end moment, obtain the hth first sign language action description of the second muscle sound signal corresponding to the hth end moment, and the hth third action image, wherein the hth third action image is the third action image corresponding to the hth end moment, and the hth end moment is any one of the end moments of the multiple second muscle sound signals; obtain the first text feature of the hth first sign language action description; obtain the first posture feature of the second hand in the hth third action image; and obtain the sign language recognition result based on the first text feature, the first posture feature and the second continuous facial image.

[0065] Specifically, the first text feature and the first posture feature are fused to obtain the first fused feature. Specifically, the first posture feature is mapped to the text space so that the first posture feature and the first text feature belong to the same dimensional space. Then, by setting a preset weight, the first posture feature and the first text feature belonging to the text space are weightedly calculated to obtain the first fused feature. The first fused feature is classified to obtain the first text corresponding to the hth end moment. Specifically, the first text feature and the first posture feature are input into the sign language recognition model to obtain the first text corresponding to the hth end moment. The subsequent content of the specification will explain in detail the specific training process of the sign language recognition model, and this application does not limit it here.

[0066] It can be seen that in the embodiment of the present application, the motion information of the left hand and the motion information of the right hand are represented by different data methods, specifically, the motion information of the left hand is converted into text features, and the motion information of the right hand is embodied in an image and then the posture features are extracted. When both hands of the user participate in the expression of sign language movements, the text features of the left hand movement expression and the posture features of the right hand movement expression can be integrated to jointly determine the meaning of the user's expression, thereby improving the accuracy of sign language recognition.

[0067] Based on the first text, the text at each end time is determined. Specifically, the method for determining the text at each end time is the same as the method for determining the first text corresponding to the hth end time, and this application will not elaborate on it here.

[0068] Combine multiple texts according to grammatical rules to obtain the text corresponding to the user's sign language. For example, if there are two texts, the first text is "thank you" and the second text is "you", then combine the first text and the second text into "thank you" to obtain the text corresponding to the user's sign language.

[0069] In the second continuous facial image, the first emotion of the hth third facial image corresponding to the hth end moment is obtained; based on the hth third facial image, the user's character features are identified, and the character features include gender and age; based on the first text, the first emotion, gender and age, the first sound parameter corresponding to the hth third facial image is determined, wherein the sound parameter includes loudness parameter, pitch parameter and timbre parameter. Because for different texts, users will have different sound parameters under different emotions, genders and ages, specifically, the type of text can be identified first, for example, the text type can be a statement type, a question type and an exclamation type. Further. A mapping relationship between text type, emotion, gender and age and sound parameters can be established to determine the sound parameters corresponding to the first text, the aforementioned character features and the first emotion. Based on the first text, the text at each end moment is determined, and the text at each end moment is combined based on grammatical rules to obtain the text corresponding to the user's sign language action.

[0070] Based on the first sound parameter, the sound parameter at each end time is determined, and the text at each end time corresponds to the sound parameter at each end time. The sign language recognition result is determined based on the text at each end time and the sound parameter at each end time.

[0071] It can be seen that in the embodiment of the present application, the text and sound parameters at each end moment are determined through the first sign language action description at each end moment, the third action image feature, and the third facial image at each end moment; based on the text and sound parameters at each end moment, the gesture recognition result is determined. In the process of sign language recognition, not only the text meaning of the sign language expression is extracted, but also the sound parameters when the text is played are identified in combination with the facial image, which helps to accurately express the user's sign language meaning and improve the accuracy of recognition.

[0072] See also Figure 4 , Figure 4 A schematic diagram of a sign language recognition model training provided in an embodiment of the present application.

[0073] Furthermore, a training data set is constructed, and based on a preset sign language action image library, a plurality of sample sign language action images and sample texts corresponding to the sign language sample action images are obtained. For a first sample sign language action image, the left hand and right hand actions of the first sample sign language action image are separated, the action of one hand is converted into a sign language action description, and the action of the other hand is segmented from the image. In the present application, the example of converting the left hand action into a sign language action description and segmenting the right hand action from the image to obtain a separate right hand action image is used for explanation. Figure 4 ,Will Figure 4The first sample sign language action image in the example is divided into a left-hand sign language action image and a right-hand sign language action image. Based on the left-hand sign language action image, a left-hand sign language action description "left hand clenched fist, thumb extended" is obtained. The left-hand sign language action description is input into the first neural network to extract text features; the right-hand action image is input into the second neural network to extract posture features. Among them, the first neural network can be a recurrent neural network (RNN), and the second neural network can be an OpenPose neural network. The left-hand sign language action description corresponding to the first sample sign language action image, the right-hand action image, and the first sample text corresponding to the first sample sign language action image constitute the first training data. In this application, the training of the pre-training model with the first training data is explained as an example. Specifically, the left-hand sign language description is converted into text features; the right-hand action image is subjected to hand skeleton key point feature extraction to obtain the posture features of the right-hand action image. The text features and the posture features are subjected to feature fusion through the fully connected layer to obtain the second fused features, and the second fused features are input into the classification layer for training to obtain the predicted sign language text.

[0074] Calculate the first loss of the first training data, specifically, determine the first loss between the predicted sign language text and the first sample text, and convert the predicted sign language text into a first text vector X = (X1, X2, ..., X n ), the first sample text is converted into the second text vector Y = (Y1, Y2, ..., Y n ), where the number of elements in the first text vector and the second text vector is n, and the first loss can be expressed by formula (2):

[0075]

[0076] Among them, L is the first loss, n is the number of elements in the vector, X m is the mth element in the first text vector, Y m is the mth element in the second text vector.

[0077] Furthermore, the pre-trained model is trained based on the loss corresponding to each training data to obtain a sign language recognition model. Taking the first training data as an example, the network parameters of the pre-trained model are adjusted using the loss corresponding to the first training data until the model converges to obtain a sign language recognition model. In practical applications, the losses corresponding to multiple training data can be weighted to obtain a weighted loss, and the weighted loss can be used to adjust the network parameters of the pre-trained model; or the loss of each training data can be used to adjust the network parameters of the pre-trained model one by one, and the present application does not limit this. For example, the training process is explained by taking the example of training the pre-trained model by determining the first loss of the first training data. Figure 4In the method, the left-hand sign language action description, the right-hand sign language action image and the first sample text corresponding to the first sample sign language action image constitute the first training data, and the left-hand sign language action description is converted into text features; the hand bone key point feature extraction is performed on the right-hand sign language action image to obtain the posture feature of the right-hand action image. The text feature and the posture feature are input into the fully connected layer for feature fusion to obtain the second fusion feature, and the second fusion feature is input into the classification layer for classification to obtain the predicted sign language text. The first loss between the predicted sign language text and the first sample text is determined, and the network parameters of the pre-training model are adjusted by using the first loss until the model converges to obtain the sign language recognition model, wherein the pre-training model can be a machine learning model such as a recurrent neural network (RNN) model, a convolutional neural network (CNN) model, a bidirectional long short-term memory recurrent neural network (BiLSTM) model, a variational autoencoder (VAE) model, etc., which is not limited here. The pre-training model can also be a variant or combination of the above-mentioned machine learning models.

[0078] 207: The server sends the sign language recognition result to the wearable device.

[0079] 208: The wearable device plays the sign language recognition result through the voice announcer.

[0080] The wearable device plays the text at each end moment using the sound parameters at each end moment according to the text at each end moment in the sign language recognition result and the sound parameters at each end moment.

[0081] See also Figure 5 , Figure 5 A flowchart of another multimodal sign language recognition method provided in an embodiment of the present application is provided. The method is applied to a wearable device. The method includes but is not limited to steps 501-504:

[0082] 501: Obtain first hand motion information of a user and at least one set of continuous screen images.

[0083] It should be noted that a wearable device is worn on the first hand, the motion information is collected by at least one acceleration sensor of the wearable device when the user is performing sign language movements, at least one set of continuous screen images is captured by at least one camera of the wearable device when the user is performing sign language movements, and at least one camera corresponds one-to-one to at least one set of continuous screen images.

[0084] Optionally, before step 501, the wearable device starts to obtain the user's first hand motion information and at least one set of continuous screen images after responding to the user's touch operation. Furthermore, when starting recognition, if all cameras in the at least one camera have not captured the user's facial image, vibration is performed with a preset intensity and a prompt message is displayed until at least one camera in the at least one camera captures the user's facial image, wherein the prompt message is used to remind the user to adjust the position of the wearable device, and the prompt message may be "Please adjust the wearing posture of the wearable device so that your face is within the shooting range of the camera."

[0085] 502: Performing region segmentation on at least one group of continuous screen images to obtain at least one group of first continuous facial images corresponding to the facial region and at least one group of first continuous action images corresponding to the second hand region.

[0086] Specifically, in step 502, the wearable device performs region segmentation on at least one group of continuous screen images to obtain at least one group of first continuous facial images corresponding to the facial area, and at least one group of first continuous action images corresponding to the second hand area. The implementation method can refer to the implementation method in step 204 in which the server performs region segmentation on at least one group of continuous screen images to obtain at least one group of first continuous facial images corresponding to the facial area, and at least one group of first continuous action images corresponding to the second hand area. This application will not repeat them here.

[0087] 503: Filter the action information to obtain sign language feature data.

[0088] It should be noted that the sign language feature data includes a first muscle sound signal of at least one channel of a first hand, and at least one acceleration sensor corresponds one-to-one to the first muscle sound signal of at least one channel.

[0089] Specifically, the implementation method of the wearable device filtering the motion information to obtain the sign language feature data in step 503 can refer to the implementation method of the server filtering the motion information to obtain the sign language feature data in step 205, and this application will not repeat it here.

[0090] 504: Obtain a sign language recognition result based on the first muscle sound signal of at least one channel, at least one set of first continuous facial images, and at least one set of first continuous action images.

[0091] Specifically, the implementation method for obtaining the sign language recognition result based on the first muscle sound signal of at least one channel, at least one group of first continuous facial images, and at least one group of first continuous motion images in step 504 can refer to the implementation method for obtaining the sign language recognition result based on the first muscle sound signal of at least one channel, at least one group of first continuous facial images, and at least one group of first continuous motion images in step 206, and the present application will not repeat them here.

[0092] In one embodiment of the present application, the wearable device also includes a display screen, and the method also includes: a user clicks a preset control, wherein the preset control may be a voice recognition control; the wearable device responds to the user's touch operation and collects voice information, wherein the voice information is voice information of a target object communicating with the user; the wearable device converts the voice information into a third text and displays it on the display screen; or, the wearable device converts the voice information into a sign language action video corresponding to the voice information; and the wearable device displays the sign language action video on the display screen.

[0093] See also Figure 6 , Figure 6 A functional unit composition block diagram of a wearable device provided in an embodiment of the present application. The wearable device 101 includes: a transceiver unit 601 and a processing unit 602;

[0094] The transceiver unit 601 is configured to receive motion information of a first hand of a user and at least one set of continuous screen images from a wearable device, wherein the wearable device is worn on the first hand, the motion information is collected by at least one acceleration sensor of the wearable device when the user is performing a sign language motion, and the at least one set of continuous screen images is captured by at least one camera of the wearable device when the user is performing a sign language motion, and the at least one camera corresponds to the at least one set of continuous screen images one by one;

[0095] The processing unit 602 is used to perform region segmentation on the at least one group of continuous screen images to obtain at least one group of first continuous facial images corresponding to the facial region and at least one group of first continuous action images corresponding to the second hand region;

[0096] Filtering the motion information to obtain sign language feature data, wherein the sign language feature data includes a first muscle sound signal of at least one channel of a first hand, and the at least one acceleration sensor corresponds one-to-one to the first muscle sound signal of the at least one channel;

[0097] Obtaining a sign language recognition result based on the first muscle sound signal of the at least one channel, the at least one set of first continuous facial images, and the at least one set of first continuous action images;

[0098] The transceiver unit 601 is used to send the sign language recognition result to the wearable device.

[0099] In one embodiment of the present application, in terms of the first muscle sound signal based on the at least one channel, the at least one set of first continuous facial images, and the at least one set of first continuous action images, the processing unit 602 is specifically configured to:

[0100] Based on the first muscle sound signal of the at least one channel, obtaining at least one sign language action description;

[0101] A sign language recognition result is obtained based on the at least one sign language action description, the at least one set of first continuous facial images, and the at least one set of first continuous action images.

[0102] In one embodiment of the present application, in the aspect of obtaining at least one sign language action description based on the first muscle sound signal of the at least one channel, the processing unit 602 is specifically configured to:

[0103] Segmenting the first muscle sound signal of the at least one channel to obtain at least one second muscle sound signal, wherein each second muscle sound signal corresponds to a sign language action description;

[0104] For the i-th second muscle sound signal, the i-th second muscle sound signal is decomposed into 2 equal frequency bandwidths. z sub-band signals, wherein z is the number of decomposition layers, and the i-th second muscle sound signal is any one of the at least one second muscle sound signal;

[0105] Determine the energy feature of the i-th second muscle sound signal based on each sub-band signal, the signal length of each sub-band signal and the number of decomposition layers;

[0106] Determine the energy feature of each second muscle sound signal based on the energy feature of the i-th second muscle sound signal;

[0107] Based on the linear discriminant analysis method, the energy characteristics of each second muscle sound signal are reduced in dimension to obtain the energy characteristics after dimension reduction;

[0108] The energy features after dimensionality reduction are classified and identified to obtain the at least one sign language action description.

[0109] In one embodiment of the present application, the processing unit 602 is specifically configured to:

[0110] If there are multiple groups of first continuous motion images, the multiple groups of first continuous motion images are grouped again according to time to obtain multiple second motion images corresponding to each time;

[0111] For the plurality of second action images at a j-th moment, identifying a skeleton key point in each second action image at the j-th moment, wherein the j-th moment is any moment in the process of the user performing a sign language action;

[0112] Determine the number of skeleton key points in each second action image at the j-th moment;

[0113] The second action image with the largest number of skeleton key points among the plurality of second action images at the j-th moment is used as the second action image corresponding to the i-th moment;

[0114] Determine, based on the second action image corresponding to the jth moment, the second action image corresponding to each moment in the process of the user performing the sign language action;

[0115] Sort the second action images corresponding to each moment in time series to obtain second continuous action images;

[0116] If there are multiple groups of first continuous facial images, the multiple groups of first continuous facial images are grouped again according to time to obtain multiple second facial images corresponding to each time;

[0117] For a plurality of second facial images at an m-th moment, identifying a skeleton key point in each second facial image at the m-th moment, wherein the m-th moment is any moment when the user is performing a sign language action;

[0118] Determine the number of skeleton key points in each second facial image at the mth moment;

[0119] The second facial image having the largest number of skeleton key points among the plurality of second facial images at the m-th moment is used as the second facial image corresponding to the m-th moment;

[0120] Determine, based on the second facial image corresponding to the mth moment, the second facial image corresponding to each moment of the user performing the sign language action;

[0121] Sort the second facial images at each moment in time series to obtain second continuous facial images;

[0122] The obtaining of a sign language recognition result based on the at least one sign language action description, the at least one set of first continuous facial images, and the at least one set of first continuous action images includes:

[0123] A sign language recognition result is obtained based on the at least one sign language action description, the second continuous facial images, and the second continuous action images.

[0124] In one embodiment of the present application, the step of obtaining a sign language recognition result based on the at least one sign language action description, the second continuous facial image, and the second continuous action image includes the following aspects: a processing unit 602 is specifically configured to:

[0125] Acquire at least one third motion image corresponding to the end time of each second muscle sound signal in the plurality of second muscle sound signals from the second continuous motion images;

[0126] For the h-th end time, obtaining the h-th first sign language action description of the second muscle sound signal corresponding to the h-th end time and the h-th third action image corresponding to the h-th end time, wherein the h-th end time is any one of the end times of the multiple second muscle sound signals;

[0127] Obtaining a first text feature of the hth first sign language action description;

[0128] Acquire a first posture feature of the second hand in the hth third action image;

[0129] A sign language recognition result is obtained based on the first text feature, the first posture feature and the second continuous facial image.

[0130] In one embodiment of the present application, in obtaining the sign language recognition result based on the first text feature, the first posture feature and the second continuous facial image, the processing unit 602 is specifically configured to:

[0131] Fusing the first text feature and the first posture feature to obtain a first fused feature;

[0132] Classify the first fusion feature to obtain the first text corresponding to the hth end time;

[0133] In the second continuous facial images, obtaining a first emotion of an h-th third facial image corresponding to an h-th end moment;

[0134] Recognize the user's personal characteristics based on the hth third facial image, where the personal characteristics include gender and age;

[0135] Determining a first sound parameter corresponding to the h-th third facial image based on the first text, the first emotion, the gender, and the age;

[0136] Based on the first text, determining the text at each end moment;

[0137] Based on the first sound parameter, determining the sound parameter at each end moment;

[0138] The sign language recognition result is determined based on the text at each end time and the sound parameters at each end time.

[0139] In one embodiment of the present application, the processing unit 602 is further specifically configured to:

[0140] If all cameras among the at least one camera fail to capture the facial image of the user, vibration is performed with a preset intensity and a prompt message is displayed until at least one camera among the at least one camera captures the facial image of the user, wherein the prompt message is used to remind the user to adjust the position of the wearable device.

[0141] See also Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device 700 includes a transceiver 701, a processor 702 and a memory 703. They are connected via a bus 704. The memory 703 is used to store computer programs and data, and can transmit the data stored in the memory 703 to the processor 702.

[0142] The processor 702 is used to read the computer program in the memory 703 and perform the following operations:

[0143] The control transceiver 701 is used to receive the motion information of the first hand of the user and at least one set of continuous screen images from the wearable device, wherein the wearable device is worn on the first hand, the motion information is collected by at least one acceleration sensor of the wearable device when the user performs the sign language motion, and the at least one set of continuous screen images is captured by at least one camera of the wearable device when the user performs the sign language motion, and the at least one camera corresponds to the at least one set of continuous screen images one by one;

[0144] Performing region segmentation on the at least one group of continuous screen images to obtain at least one group of first continuous facial images corresponding to the facial region and at least one group of first continuous action images corresponding to the second hand region;

[0145] Filtering the motion information to obtain sign language feature data, wherein the sign language feature data includes a first muscle sound signal of at least one channel of a first hand, and the at least one acceleration sensor corresponds one-to-one to the first muscle sound signal of the at least one channel;

[0146] Obtaining a sign language recognition result based on the first muscle sound signal of the at least one channel, the at least one set of first continuous facial images, and the at least one set of first continuous action images;

[0147] The sign language recognition result is sent to the wearable device.

[0148] In one embodiment of the present application, in terms of obtaining a sign language recognition result based on the first muscle sound signal of the at least one channel, the at least one set of first continuous facial images, and the at least one set of first continuous action images, the processor 702 is specifically configured to:

[0149] Based on the first muscle sound signal of the at least one channel, obtaining at least one sign language action description;

[0150] A sign language recognition result is obtained based on the at least one sign language action description, the at least one set of first continuous facial images, and the at least one set of first continuous action images.

[0151] In one embodiment of the present application, in the aspect of obtaining at least one sign language action description based on the first muscle sound signal of the at least one channel, the processor 702 is specifically configured to:

[0152] Segmenting the first muscle sound signal of the at least one channel to obtain at least one second muscle sound signal, wherein each second muscle sound signal corresponds to a sign language action description;

[0153] For the i-th second muscle sound signal, the i-th second muscle sound signal is decomposed into 2 equal frequency bandwidths. z sub-band signals, wherein z is the number of decomposition layers, and the i-th second muscle sound signal is any one of the at least one second muscle sound signal;

[0154] Determine the energy feature of the i-th second muscle sound signal based on each sub-band signal, the signal length of each sub-band signal and the number of decomposition layers;

[0155] Determine the energy feature of each second muscle sound signal based on the energy feature of the i-th second muscle sound signal;

[0156] Based on the linear discriminant analysis method, the energy characteristics of each second muscle sound signal are reduced in dimension to obtain the energy characteristics after dimension reduction;

[0157] The energy features after dimensionality reduction are classified and identified to obtain the at least one sign language action description.

[0158] In one embodiment of the present application, the processor 702 is specifically configured to:

[0159] If there are multiple groups of first continuous motion images, the multiple groups of first continuous motion images are grouped again according to time to obtain multiple second motion images corresponding to each time;

[0160] For a plurality of second action images at a j-th moment, identifying a skeleton key point in each second action image at the j-th moment, wherein the i-th moment is any moment in the process of the user performing a sign language action;

[0161] Determine the number of skeleton key points in each second action image at the j-th moment;

[0162] The second action image with the largest number of skeleton key points among the plurality of second action images at the j-th moment is used as the second action image corresponding to the j-th moment;

[0163] Determine, based on the second action image corresponding to the jth moment, the second action image corresponding to each moment in the process of the user performing the sign language action;

[0164] Sort the second action images corresponding to each moment in time series to obtain second continuous action images;

[0165] If there are multiple groups of first continuous facial images, the multiple groups of first continuous facial images are grouped again according to time to obtain multiple second facial images corresponding to each time;

[0166] For a plurality of second facial images at an m-th moment, identifying a skeleton key point in each second facial image at the m-th moment, wherein the m-th moment is any moment when the user is performing a sign language action;

[0167] Determine the number of skeleton key points in each second facial image at the mth moment;

[0168] The second facial image having the largest number of skeleton key points among the plurality of second facial images at the m-th moment is used as the second facial image corresponding to the m-th moment;

[0169] Determine, based on the second facial image corresponding to the mth moment, the second facial image corresponding to each moment of the user performing the sign language action;

[0170] Sort the third facial image at each moment in time series to obtain a second continuous facial image;

[0171] The obtaining of a sign language recognition result based on the at least one sign language action description, the at least one set of first continuous facial images, and the at least one set of first continuous action images includes:

[0172] A sign language recognition result is obtained based on the at least one sign language action description, the second continuous facial images, and the second continuous action images.

[0173] In one embodiment of the present application, the step of obtaining a sign language recognition result based on the at least one sign language action description, the second continuous facial image, and the second continuous action image includes the following aspects: the processor 702 is specifically configured to:

[0174] Acquire at least one third motion image corresponding to the end time of each second muscle sound signal in the plurality of second muscle sound signals from the second continuous motion images;

[0175] For the h-th end time, obtaining the h-th first sign language action description of the second muscle sound signal corresponding to the h-th end time and the h-th third action image corresponding to the h-th end time, wherein the h-th end time is any one of the end times of the multiple second muscle sound signals;

[0176] Obtaining a first text feature of the hth first sign language action description;

[0177] Acquire a first posture feature of the second hand in the hth third action image;

[0178] A sign language recognition result is obtained based on the first text feature, the first posture feature and the second continuous facial image.

[0179] In one embodiment of the present application, in obtaining the sign language recognition result based on the first text feature, the first posture feature and the second continuous facial image, the processor 702 is specifically configured to:

[0180] Fusing the first text feature and the first posture feature to obtain a first fused feature;

[0181] Classify the first fusion feature to obtain the first text corresponding to the hth end time;

[0182] In the second continuous facial images, obtaining a first emotion of an h-th third facial image corresponding to an h-th end moment;

[0183] Recognize the user's personal characteristics based on the hth third facial image, where the personal characteristics include gender and age;

[0184] Determining a first sound parameter corresponding to the h-th third facial image based on the first text, the first emotion, the gender, and the age;

[0185] Based on the first text, determining the text at each end moment;

[0186] Based on the first sound parameter, determining the sound parameter at each end moment;

[0187] The sign language recognition result is determined based on the text at each end time and the sound parameters at each end time.

[0188] In one embodiment of the present application, the processor 702 is further specifically configured to:

[0189] If all cameras among the at least one camera fail to capture the facial image of the user, vibration is performed with a preset intensity and a prompt message is displayed until at least one camera among the at least one camera captures the facial image of the user, wherein the prompt message is used to remind the user to adjust the position of the wearable device.

[0190] Specifically, the transceiver 701 may be Figure 6 The transceiver unit 601 of the wearable device 101 of the embodiment, the processor 702 may be Figure 6 The processing unit 602 of the wearable device 101 of an embodiment.

[0191] It should be understood that the electronic devices in this application may include smart phones (such as Android phones, iOS phones, Windows Phone phones, etc.), tablet computers, PDAs, laptops, mobile Internet devices MID (Mobile Internet Devices, referred to as: MID) or wearable devices, etc. The above electronic devices are only examples, not exhaustive, and include but are not limited to the above electronic devices. In practical applications, the above electronic devices may also include: smart vehicle terminals, computer equipment, etc.

[0192] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. The computer program is executed by a processor to implement part or all of the steps of any multimodal sign language recognition method recorded in the above method embodiments.

[0193] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps of any multimodal sign language recognition method recorded in the above method embodiments.

[0194] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0195] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0196] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0197] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0198] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software program module.

[0199] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0200] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk or an optical disk, etc.

[0201] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A multimodal sign language recognition method, characterized in that: The method is applied to a wearable device, and the method comprises: Acquiring motion information of a first hand of a user and at least one set of continuous screen images, wherein the wearable device is worn on the first hand, the motion information is collected by at least one acceleration sensor of the wearable device when the user is performing a sign language motion, and the at least one set of continuous screen images is captured by at least one camera of the wearable device when the user is performing a sign language motion, and the at least one camera corresponds to the at least one set of continuous screen images in a one-to-one manner; Performing region segmentation on the at least one group of continuous screen images to obtain at least one group of first continuous facial images corresponding to the facial region and at least one group of first continuous action images corresponding to the second hand region; Filtering the motion information to obtain sign language feature data, wherein the sign language feature data includes a first muscle sound signal of at least one channel of a first hand, and the at least one acceleration sensor corresponds one-to-one to the first muscle sound signal of the at least one channel; Obtaining a sign language recognition result based on the first muscle sound signal of the at least one channel, the at least one set of first continuous facial images, and the at least one set of first continuous action images, including: Based on the first muscle sound signal of the at least one channel, obtaining at least one sign language action description; If there are multiple groups of first continuous action images, the multiple groups of first continuous action images are grouped again according to time to obtain multiple second action images corresponding to each time; for the multiple second action images at the jth time, the skeleton key points in each second action image at the jth time are identified, wherein the jth time is any time when the user is performing the sign language action; the number of skeleton key points in each second action image at the jth time is determined; the second action image with the largest number of skeleton key points among the multiple second action images at the jth time is the second action image corresponding to the jth time; based on the second action image corresponding to the jth time, the second action image corresponding to each time when the user is performing the sign language action is determined; the second action images corresponding to each time are sorted in time series to obtain second continuous action images; If there are multiple groups of first continuous facial images, the multiple groups of first continuous facial images are grouped again according to time to obtain multiple second facial images corresponding to each time; for the multiple second facial images at the mth time, the skeletal key points in each second facial image at the mth time are identified, wherein the mth time is any time when the user is performing a sign language action; the number of skeletal key points in each second facial image at the mth time is determined; the second facial image with the largest number of skeletal key points among the multiple second facial images at the mth time is the second facial image corresponding to the mth time; based on the second facial image corresponding to the mth time, the second facial image corresponding to each time when the user is performing the sign language action is determined; the second facial images at each time are sorted in time series to obtain second continuous facial images; The sign language recognition result is obtained based on the at least one sign language action description, the second continuous facial images and the second continuous action images.

2. The method according to claim 1, characterized in that: The step of obtaining at least one sign language action description based on the first muscle sound signal of the at least one channel includes: Segmenting the first muscle sound signal of the at least one channel to obtain at least one second muscle sound signal, wherein each second muscle sound signal corresponds to a sign language action description; For the i-th second muscle sound signal, the i-th second muscle sound signal is decomposed into 2 equal frequency bandwidths. z sub-band signals, wherein z is the number of decomposition layers, and the i-th second muscle sound signal is any one of the at least one second muscle sound signal; Determine the energy feature of the i-th second muscle sound signal based on each sub-band signal, the signal length of each sub-band signal and the number of decomposition layers; Determine the energy feature of each second muscle sound signal based on the energy feature of the i-th second muscle sound signal; Based on the linear discriminant analysis method, the energy characteristics of each second muscle sound signal are reduced in dimension to obtain the energy characteristics after dimension reduction; The energy features after dimensionality reduction are classified and identified to obtain the at least one sign language action description.

3. The method according to claim 1 or 2, characterized in that: The obtaining the sign language recognition result based on the at least one sign language action description, the second continuous facial images, and the second continuous action images includes: Acquire at least one third motion image corresponding to the end time of each second muscle sound signal in the plurality of second muscle sound signals from the second continuous motion images; For the h-th end time, obtaining the h-th first sign language action description of the second muscle sound signal corresponding to the h-th end time and the h-th third action image corresponding to the h-th end time, wherein the h-th end time is any one of the end times of the multiple second muscle sound signals; Obtaining a first text feature of the hth first sign language action description; Acquire a first posture feature of the second hand in the hth third action image; The sign language recognition result is obtained based on the first text feature, the first posture feature and the second continuous facial image.

4. The method according to claim 3, characterized in that The obtaining the sign language recognition result based on the first text feature, the first posture feature and the second continuous facial image includes: Fusing the first text feature and the first posture feature to obtain a first fused feature; Classify the first fusion feature to obtain the first text corresponding to the h-th end time; In the second continuous facial images, obtaining a first emotion of an h-th third facial image corresponding to the h-th end moment; Recognize the user's personal characteristics based on the hth third facial image, where the personal characteristics include gender and age; Determining a first sound parameter corresponding to the h-th third facial image based on the first text, the first emotion, the gender, and the age; Based on the first text, determining the text at each end moment; Based on the first sound parameter, determining the sound parameter at each end moment; The sign language recognition result is determined based on the text at each end time and the sound parameters at each end time.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: If all cameras among the at least one camera fail to capture the facial image of the user, vibration is performed with a preset intensity and a prompt message is displayed until at least one camera among the at least one camera captures the facial image of the user, wherein the prompt message is used to remind the user to adjust the position of the wearable device.

6. A wearable device, characterized in that: The wearable device comprises: a transceiver unit and a processing unit; The transceiver unit is used to obtain motion information of a first hand of a user and at least one set of continuous screen images, wherein the wearable device is worn on the first hand, the motion information is collected by at least one acceleration sensor of the wearable device when the user is performing a sign language motion, and the at least one set of continuous screen images is captured by at least one camera of the wearable device when the user is performing a sign language motion, and the at least one camera corresponds to the at least one set of continuous screen images in a one-to-one manner; The processing unit is used to perform region segmentation on the at least one group of continuous screen images to obtain at least one group of first continuous facial images corresponding to the facial region and at least one group of first continuous action images corresponding to the second hand region; The processing unit is configured to filter the motion information to obtain sign language feature data, wherein the sign language feature data includes a first muscle sound signal of at least one channel of a first hand, and the at least one acceleration sensor corresponds one-to-one to the first muscle sound signal of the at least one channel; The processing unit is used to obtain a sign language recognition result based on the first muscle sound signal of the at least one channel, the at least one set of first continuous facial images, and the at least one set of first continuous action images, including: Based on the first muscle sound signal of the at least one channel, obtaining at least one sign language action description; If there are multiple groups of first continuous motion images, the multiple groups of first continuous motion images are grouped again according to time to obtain multiple second motion images corresponding to each time; For a plurality of second action images at a j-th moment, identifying a skeleton key point in each second action image at the j-th moment, wherein the j-th moment is any moment when the user is performing a sign language action; Determine the number of skeleton key points in each second action image at the j-th moment; The second action image with the largest number of skeleton key points among the plurality of second action images at the j-th moment is used as the second action image corresponding to the j-th moment; Determine, based on the second action image corresponding to the jth moment, the second action image corresponding to each moment in the process of the user performing the sign language action; Sort the second action images corresponding to each moment in time series to obtain second continuous action images; If there are multiple sets of first continuous facial images, obtaining second continuous facial images based on the second continuous motion images and the multiple sets of first continuous facial images; The sign language recognition result is obtained based on the at least one sign language action description, the second continuous facial images and the second continuous action images.

7. An electronic device, characterized in that: include: A processor and a memory, the processor is connected to the memory, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 5.

Images