Sign language recognition method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a sign language recognition method and device, electronic equipment and storage medium. The method comprises: acquiring visual signals and electromyographic signals collected when a user performs sign language expression, extracting visual features from the visual signals, and extracting electromyographic features from the electromyographic signals; aligning the visual features and the electromyographic features in the time dimension, generating a visual recognition result according to the aligned visual features, and generating an electromyographic recognition result according to the aligned electromyographic features; determining a sign language recognition result according to the visual recognition result and the electromyographic recognition result; and outputting a sign language text and / or synthesized speech according to the sign language recognition result. The embodiments of the present application introduce electromyographic signals to assist in recognition, thereby avoiding the problem of reduced recognition accuracy of a single visual modality under the influence of environmental factors, and enabling the sign language recognition to maintain good recognition effect in a complex environment.

Description

Technical Field

[0001] This application relates to the field of sign language recognition technology, and in particular to a sign language recognition method, device, electronic device and storage medium. Background Technology

[0002] Sign language is an important form of daily communication for hearing-impaired individuals. Converting sign language gestures into text or speech output using automatic recognition technology is crucial for eliminating communication barriers between the hearing-impaired and hearing communities. In recent years, with the development of computer vision technology, solutions that use optical cameras to capture hand movements and then identify the corresponding sign language semantics through models have become the mainstream technological approach in the field of automatic sign language recognition.

[0003] Relevant sign language recognition solutions typically rely on cameras to capture visual signals from the user's hands and face. By analyzing visual features such as key points of the hand skeleton and facial expressions, they identify the sign language words expressed by the user and convert the recognition results into corresponding text or speech output. These solutions can achieve good recognition results in scenarios with ideal visual acquisition conditions, but their recognition performance is highly dependent on the quality and completeness of the visual signals themselves.

[0004] In practical applications, the quality of visual signal acquisition is affected by many environmental factors, leading to a significant decrease in recognition accuracy of single-visual-modality sign language recognition schemes in various complex scenarios. Specifically, in environments with strong light, dim light, or uneven lighting, the acquired visual signals have more noise, reducing the accuracy of visual feature extraction. In scenarios where users are in crowded places or whose hand movements are large and exceed the camera's field of view, key hand points may be obscured or lost, resulting in incomplete visual features. Furthermore, visual acquisition methods based on long-distance cameras also struggle to accurately capture some fine finger movements or subtle facial muscle changes. Summary of the Invention

[0005] This application aims to solve the technical problem that current sign language recognition schemes rely on only a single visual modality, resulting in decreased recognition accuracy when the visual signal quality is affected by environmental factors.

[0006] Firstly, this application provides a sign language recognition method, including: The system acquires visual and electromyographic signals collected when a user performs sign language expression, extracts visual features from the visual signals, and extracts electromyographic features from the electromyographic signals. Align the visual features and the electromyographic features in the time dimension, generate a visual recognition result based on the aligned visual features, and generate an electromyographic recognition result based on the aligned electromyographic features. The sign language recognition result is determined based on the visual recognition result and the electromyographic recognition result; Sign language text and / or synthesized speech are generated based on the sign language recognition results.

[0007] Secondly, this application provides a sign language recognition device, comprising: The acquisition module is used to acquire visual signals and electromyographic signals collected when the user makes sign language expressions, and to extract visual features from the visual signals and electromyographic features from the electromyographic signals. The recognition module is used to align the visual features and the electromyographic features in the time dimension, generate a visual recognition result based on the aligned visual features, and generate an electromyographic recognition result based on the aligned electromyographic features. The determining module is used to determine the sign language recognition result based on the visual recognition result and the electromyography recognition result; The output module is used to generate sign language text and / or synthesized speech based on the sign language recognition results.

[0008] Thirdly, this application provides an electronic device, comprising: At least one processor; At least one network interface, said network interface being communicatively connected to a corresponding processor; and, A memory communicatively connected to the at least one processor; wherein, The network interface is used to establish communication connections between the processor and other external devices; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the sign language recognition method as described in the first aspect.

[0009] Fourthly, this application provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors, causing the one or more processors to perform the sign language recognition method as described in the first aspect.

[0010] At least one advantage of the sign language recognition method, apparatus, electronic device, and storage medium provided in this application is that, addressing the problem that existing sign language recognition schemes rely solely on a single visual modality and suffer significant accuracy degradation when visual signal quality is affected by environmental factors, this application acquires electromyographic (EMG) signals simultaneously with visual signals, generates visual recognition results and EMG recognition results separately, and then fuses them. Since the acquisition of EMG signals is unaffected by environmental factors such as lighting and occlusion, when visual signal quality deteriorates, this application can compensate for the visual recognition results using EMG recognition results, thereby avoiding the problem of decreased recognition accuracy in the aforementioned scenarios with a single visual modality, and enabling sign language recognition to maintain good recognition performance even in complex environments. Attached Figure Description

[0011] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0012] Figure 1 A flowchart illustrating a sign language recognition method provided for an embodiment of the present invention; Figure 2 for Figure 1 A schematic diagram of the sub-process of step S400 in the sign language recognition method shown; Figure 3 A flowchart illustrating another sign language recognition method provided for an embodiment of the present invention; Figure 4 It shows Figure 1 The sign language recognition method shown demonstrates the intention completion process when visual signals are lost. Figure 5 for Figure 1 The diagram shows a sub-process diagram of step S200 in the sign language recognition method, which generates the electromyography recognition result based on the aligned electromyography features. Figure 6 A schematic diagram of the structure of a sign language recognition device provided for an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of an electronic device provided for an embodiment of the present invention. Detailed Implementation

[0013] To facilitate understanding of this application, a more detailed description of this application will be provided below in conjunction with the accompanying drawings and specific embodiments.

[0014] Furthermore, the technical features involved in the different embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0015] This application provides a sign language recognition method applicable to a system comprising smart glasses, an electromyography (EMG) device, and a processing device. The smart glasses are worn on the user's head and are equipped with a camera facing forward to capture the user's visual signals when the user uses sign language.

[0016] Since the user's hands are usually positioned in front of their body and within the camera's field of view when using sign language, the camera can capture images or videos of the user's hands. Additionally, the smart glasses can be equipped with a light sensor to collect ambient light information.

[0017] An electromyography (EMG) device is a device that comes into contact with a specific part of a user's body to collect the user's electromyographic signals when the user performs sign language. The EMG signal refers to the voltage change signal generated by the contraction of relevant muscles during the user's sign language expression.

[0018] Since sign language is primarily expressed through hand movements, the muscles in the upper limbs that control these hand movements contract accordingly when a user uses sign language. Therefore, electromyography (EMG) devices mainly collect EMG signals from these upper limb muscles. In one embodiment, the EMG device is an EMG bracelet worn on the user's upper limb. The bracelet has EMG electrodes on its inner side that contact the user's forearm skin to collect the EMG signals generated by the user's forearm muscles when controlling hand movements.

[0019] In another embodiment, the electromyographic device is an electromyographic patch attached to the user's skin. The aforementioned electromyographic signals can directly reflect the muscle activity corresponding to the user's hand movements, complementing the hand movement information in the visual signals.

[0020] In one alternative implementation, the electromyographic device can also be a hearing aid worn on the user's ear, or electromyographic electrodes integrated into the temples of smart glasses, used to collect electromyographic signals generated by the user's facial muscles when performing sign language. It should be noted that since the range of facial muscle activity is typically small when a user performs sign language, the facial electromyographic signals collected by this type of device are primarily used to assist in recognizing a small number of sign language expressions that rely on facial expressions (such as sign language expressing questions or negations through facial expressions), rather than as the primary basis for recognizing hand gestures corresponding to sign language words. Therefore, this type of electromyographic device that collects facial electromyographic signals can supplement electromyographic devices that collect upper limb electromyographic signals, and the two can be used in combination.

[0021] It should be noted that visual signals are collected by the smart glasses' camera, while electromyographic (EMG) signals are collected by an EMG device that contacts a specific part of the user's body. These are collected by separate, independent devices. This is because EMG signals reflect the weak voltage changes generated by muscle contraction, requiring electrodes in direct contact with the skin of the relevant muscle area for accurate acquisition. The smart glasses' camera cannot collect EMG signals.

[0022] The sign language recognition method provided in this application is executed by a processing device, which is communicatively connected to both smart glasses and an electromyography (EMG) device. This communication connection means that data transmission between the processing device and the smart glasses and the EMG device can be performed via wired or wireless means. The processing device acquires visual signals collected by the smart glasses and EMG signals collected by the EMG device, and processes the visual and EMG signals to obtain the sign language recognition result.

[0023] It should be noted that smart glasses and electromyography (EMG) devices are acquisition devices for visual and EMG signals, respectively, while the processing device is the device that acquires and processes these signals. These three are independent devices; the processing device itself does not contain components for acquiring visual and EMG signals. In practical applications, the processing device can be located inside the smart glasses, such as a built-in processing chip; or it can be a separate device from the smart glasses and EMG device, such as a smartphone, computer, or cloud server.

[0024] In one implementation, the processing device can be a processing chip built into the smart glasses, or it can be a smartphone, computer, or cloud server that is separate from the smart glasses and electromyography device.

[0025] Please refer to Figure 1, which is a flowchart illustrating the sign language recognition method provided in this embodiment. The sign language recognition method includes the following steps: Step S100: Acquire visual and electromyographic signals collected when the user is performing sign language expression, extract visual features from visual signals, and extract electromyographic features from electromyographic signals.

[0026] The processing device acquires visual signals collected by the smart glasses and electromyographic signals collected by the electromyography device. The visual signals carry image or video information of the user's hand movements when expressing sign language, and the electromyographic signals carry information on the voltage changes of the relevant muscles when the user expresses sign language.

[0027] Furthermore, the processing device extracts visual features from the visual signal. These visual features refer to the characteristic data extracted from the visual signal that can characterize the user's hand movements.

[0028] In one implementation, the visual features include positional information of key points on the hand skeleton. Key points on the hand skeleton refer to several preset position points on the user's hand, such as fingertip position points, finger joint position points, wrist position points, etc.

[0029] The processing device can detect the user's hand from each frame of the visual signal using a hand detection algorithm, and identify the position coordinates of each key point of the hand skeleton. The sequence formed by the change of the position coordinates of each key point of the hand skeleton over time is used as a visual feature.

[0030] In one optional implementation, the visual features also include the location information of facial feature points. Facial feature points refer to several preset location points on the user's face, such as the location points of eyebrows, eyes, and corners of the mouth. When the smart glasses are also equipped with an auxiliary camera facing the user's face, or when the user's facial expression information can be collected through other means, the processing device can detect the user's face from the corresponding visual signals, identify the location coordinates of each facial feature point, and include the sequence formed by the change of the location coordinates of each facial feature point over time as part of the visual features.

[0031] Since sign language is primarily expressed through hand gestures, and facial expressions are used in some sign languages ​​to assist in expressing meanings such as questions and negations, visual features are mainly based on the positional information of key points of the hand skeleton, with the positional information of facial feature points serving as an optional supplement.

[0032] While extracting visual features, the processing device also extracts electromyographic (EMG) features from the EMG signal. EMG features refer to characteristic data extracted from the EMG signal that reflects the user's muscle activity state. The processing device can perform bandpass filtering on the EMG signal to remove interference, then use full-wave rectification and envelope extraction to obtain features reflecting muscle contraction intensity, and / or use frequency domain analysis to obtain features reflecting muscle contraction frequency. These features are then used as EMG features.

[0033] Step S200: Align the visual features and electromyographic features in the time dimension, generate a visual recognition result based on the aligned visual features, and generate an electromyographic recognition result based on the aligned electromyographic features.

[0034] Since the frame rate of visual signal acquisition and the sampling frequency of electromyography (EMG) signal are usually different, the processing device timestamps visual features and EMG features based on a unified time reference and resamples them to the same time grid so that visual features and EMG features acquired at the same time correspond to each other, resulting in aligned visual features and aligned EMG features.

[0035] The processing device generates visual recognition results based on the aligned visual features. These visual recognition results refer to the output reflecting the user's sign language expression, obtained from the visual features. This output includes multiple candidate sign language terms and the probability of each candidate term.

[0036] In one implementation, the processing device pre-trains a visual recognition model that takes the positional coordinate sequence of key points on the hand skeleton as input and the probability of each candidate sign language term as output. The processing device inputs the aligned visual features into the visual recognition model, and the visual recognition model outputs each candidate sign language term and its probability as the visual recognition result.

[0037] For example, for a certain visual feature, the visual recognition model outputs "Sorry 0.48, Thank you 0.52, Goodbye 0.00", where "Sorry", "Thank you" and "Goodbye" are candidate sign language words, and the corresponding values ​​are the probabilities of each candidate sign language word.

[0038] The processing device generates electromyography (EMG) recognition results based on the aligned EMG features. These EMG recognition results are outputs reflecting the user's sign language expression, obtained from the EMG features. This output also includes multiple candidate sign language terms and the probability of each term. The processing device determines each candidate sign language term and its probability corresponding to the aligned EMG features based on a pre-established correspondence between EMG features and candidate sign language terms, thus serving as the EMG recognition result.

[0039] For example, for the electromyographic features corresponding to the same time period mentioned above, the processing device generates electromyographic recognition results such as "Sorry 0.65, Thank you 0.30, Hello 0.05".

[0040] It should be noted that the first candidate sign language phrase in the visual recognition result and the second candidate sign language phrase in the electromyography (EMG) recognition result may be the same or not completely identical. Since visual recognition and EMG recognition are performed independently based on visual and EMG features respectively, the sets of candidate sign language phrases output by the two may differ. For example, if the visual recognition result is "Sorry 0.48, Thank you 0.52, Goodbye 0.00", its first candidate sign language phrases are "Sorry", "Thank you", and "Goodbye"; while the EMG recognition result may be "Sorry 0.65, Thank you 0.30, Hello 0.05", its second candidate sign language phrases are also "Sorry", "Thank you", and "Hello". The candidate sign language phrases of the two are not entirely identical.

[0041] In the subsequent fusion process, the processing device takes the union of the first candidate sign language words in the visual recognition results and the second candidate sign language words in the electromyography (EMG) recognition results to obtain all candidate sign language words. For a certain candidate sign language word, if it only appears in the visual recognition results and not in the EMG recognition results, the probability of the candidate sign language word in the EMG recognition results is counted as zero; conversely, if it only appears in the EMG recognition results and not in the visual recognition results, the probability of the candidate sign language word in the visual recognition results is counted as zero.

[0042] Step S300: Determine the sign language recognition result based on the visual recognition result and the electromyography recognition result.

[0043] The sign language recognition result refers to the final recognition output, determined by combining the visual recognition result and the electromyography (EMG) recognition result, reflecting the content expressed by the user's sign language. As described in the aforementioned implementation method, the visual recognition result includes multiple first candidate sign language terms and their probabilities, and the EMG recognition result includes multiple second candidate sign language terms and their probabilities. When determining the sign language recognition result, the processing device takes the union of the first candidate sign language terms in the visual recognition result and the second candidate sign language terms in the EMG recognition result to obtain all candidate sign language terms. Based on the probability of the same candidate sign language term in the visual recognition result and the probability in the EMG recognition result, the device comprehensively determines the probability that the candidate sign language term is the actual content expressed by the user, and selects the candidate sign language term with the highest probability as the sign language recognition result.

[0044] There are several ways for the processing device to determine the sign language recognition result based on the visual recognition result and the electromyography (EMG) recognition result. In one embodiment, the processing device assigns corresponding weights to the visual recognition result and the EMG recognition result, and sums the probabilities of the same candidate sign language word in the two recognition results according to the corresponding weights to obtain the fusion probability of the candidate sign language word. The candidate sign language word with the highest fusion probability is taken as the sign language recognition result. The assigned weights are determined based on at least one of the visual confidence score, which represents the credibility of the visual features, and the EMG confidence score, which represents the credibility of the EMG features. The recognition result with higher credibility is assigned a larger weight, so that the one with higher quality visual signal or EMG signal plays a more important role in determining the sign language recognition result.

[0045] In another implementation, when the probabilities of the first candidate sign language word with the highest probability in the visual recognition result are close to those of the second highest probability first candidate sign language word, and the visual recognition result is semantically ambiguous, the processing device reduces the weight of the visual recognition result and redetermines the sign language recognition result, or directly determines the sign language recognition result based on the electromyography recognition result, thereby correcting the visual recognition result with the help of the electromyography recognition result.

[0046] In this way, when the visual signal quality is good, the sign language recognition result is mainly determined by the visual recognition result; when the visual signal quality decreases due to environmental factors such as strong light, dim light, or occlusion, the role of electromyography (EMG) recognition result in determining the sign language recognition result is correspondingly enhanced, thereby compensating for the visual recognition result with the help of EMG recognition result, so that sign language recognition can still maintain a good recognition effect when the visual signal quality decreases.

[0047] Step S400: Generate sign language text and / or synthesized speech based on the sign language recognition results.

[0048] The processing device generates corresponding sign language text and / or synthesized speech based on the sign language recognition results. Sign language text refers to the written text corresponding to the sign language recognition results, while synthesized speech refers to the speech obtained by converting the sign language recognition results through speech synthesis technology. The processing device can output one or both of these, depending on the actual application scenario.

[0049] To facilitate understanding of the above steps, the following explanation uses the user's sign language expression of "sorry" as an example. The user wears smart glasses and an electromyography (EMG) bracelet. When expressing this in sign language, the hand makes the gesture corresponding to "sorry" in front of the body, causing the relevant upper limb muscles to contract. The smart glasses' camera captures the visual signal of the user's hand movement, while the EMG bracelet captures the electromyographic (EMG) signal of the user's forearm muscles.

[0050] After acquiring visual and electromyographic (EMG) signals, the processing device extracts key point sequences of the hand skeleton from the visual signals as visual features and extracts muscle contraction intensity and other features from the EMG signals as EMG features. After aligning the two, a visual recognition result is generated based on the visual features, and an EMG recognition result is generated based on the EMG features. Visual confidence and EMG confidence are generated. The two recognition results are fused based on the confidence scores to obtain the sign language recognition result "Sorry". Then, the sign language text "Sorry" or the corresponding synthesized speech is generated.

[0051] As mentioned earlier, the visual recognition result includes multiple candidate sign language terms and the probability of each candidate sign language term. Similarly, the electromyography (EMG) recognition result also includes multiple candidate sign language terms and the probability of each candidate sign language term. For ease of distinction, the candidate sign language terms in the visual recognition result are referred to as the first candidate sign language terms, and the candidate sign language terms in the EMG recognition result are referred to as the second candidate sign language terms.

[0052] Please see Figure 2 In one embodiment, step S300 specifically includes the following steps: Step S310: Generate at least one of visual confidence and electromyographic confidence.

[0053] The processing device generates confidence scores, which include at least one of visual confidence scores and electromyographic (EMG) confidence scores. Visual confidence scores are numerical values ​​characterizing the reliability of visual features, while EMG confidence scores are numerical values ​​characterizing the reliability of electromyographic (EMG) features.

[0054] It should be noted that credibility refers to the reliability of the features on which the recognition results are based in reflecting the actual content of the user's sign language expression, while confidence is a numerical value obtained by quantifying credibility: the higher the confidence value, the higher the credibility of the corresponding feature; the lower the confidence value, the lower the credibility of the corresponding feature.

[0055] In one implementation, both the visual confidence score and the electromyographic confidence score range from zero to one.

[0056] The processing device generates visual confidence scores based on the quality of visual signals and / or visual features. Specifically, the processing device determines scores for one or more indicators that reflect the quality of visual signals and / or visual features. The score for each indicator ranges from zero to one, with a higher score indicating better quality. The processing device generates visual confidence scores based on the scores of one or more indicators. For example, when generating visual confidence scores based on multiple indicators, the scores of each indicator are weighted and summed, and the weighted sum is used as the visual confidence score.

[0057] The indicators used for visual confidence assessment include at least one of the following: illumination level of the visual signal, completeness of visual features, and continuity of visual features over time. Illumination level refers to the lighting conditions of the image corresponding to the visual signal at the time of acquisition. The processing device determines the score corresponding to the illumination level by analyzing the brightness distribution of the image in the visual signal. A higher score is given when the image brightness is within a suitable range, and a lower score is given when the image is overexposed or underexposed. This indicator reflects the impact of environmental factors such as strong light and dim light on the quality of visual signal acquisition.

[0058] The completeness of visual features refers to the degree of coverage of visual features extracted from visual signals relative to complete visual features. The processing device can use the ratio between the number of key points of the hand skeleton actually detected and the number of key points of the complete hand skeleton as the score corresponding to the completeness. When the user's hand is not obstructed and is completely within the camera's acquisition range, the score is higher. When the user's hand is obstructed or partially outside the acquisition range, the score is lower. This indicator is used to reflect the impact of factors such as obstruction and being outside the acquisition range on visual features.

[0059] The continuity of visual features in the time dimension refers to whether the changes of visual features between adjacent moments are smooth and conform to the laws of continuous motion. The processing device can determine the score corresponding to the continuity by analyzing the changes of visual features in several consecutive frames. When the changes of visual features are smooth between adjacent frames, the score is higher, and when there are sudden changes or jumps, the score is lower. This index is used to reflect the influence of factors such as fast motion and motion blur on visual features.

[0060] The processing device can generate visual confidence scores based on any one, any two, or all three of the above three indicators. When generating visual confidence scores based on a single indicator, the processing device uses the score corresponding to that indicator as the visual confidence score; when generating visual confidence scores based on two or all three indicators, the processing device performs a weighted summation of the scores corresponding to each indicator and uses the weighted summation result as the visual confidence score.

[0061] The processing device generates an electromyographic confidence score based on the quality of the electromyographic signal and / or electromyographic characteristics. Specifically, the processing device generates an electromyographic confidence score based on scores of one or more indicators reflecting the quality of the electromyographic signal. For example, the corresponding score is determined based on at least one of the following: the signal-to-noise ratio of the electromyographic signal, the contact condition between the electromyographic electrode and the skin, and the degree of interference from motion artifacts in the electromyographic signal. The higher the score, the better the quality of the electromyographic signal, and the higher the generated electromyographic confidence score.

[0062] Step S320: Determine a first weight for the visual recognition result and a second weight for the electromyographic recognition result based on at least one of the visual confidence score and the electromyographic confidence score.

[0063] In the first scenario, the confidence level only includes visual confidence. The processing device uses visual confidence as the first weight and the difference between the visual confidence level and the visual confidence level as the second weight. For example, when the visual confidence level is 0.7, the first weight is 0.7 and the second weight is 1. 0.7 = 0.3. This scenario is suitable for applications where the electromyographic signal quality is relatively stable and the weights of the two channels are mainly adjusted based on the visual signal quality.

[0064] In the second scenario, the confidence level only includes the EMG confidence level. The processing device uses the EMG confidence level as the second weight and the difference between the two as the first weight. For example, when the EMG confidence level is 0.4, the second weight is 0.4 and the first weight is 1. 0.4 = 0.6. This scenario is suitable for applications where the visual signal quality is relatively stable and the weights of the two channels are mainly adjusted based on the quality of the electromyographic signal.

[0065] In the third scenario, the confidence level includes both visual confidence and electromyography (EMG) confidence. The processing device normalizes the visual and EMG confidence levels, using the normalized visual confidence as the first weight and the normalized EMG confidence as the second weight. Normalization involves dividing the visual confidence by the sum of the visual and EMG confidence levels to obtain the normalized visual confidence as the first weight; and dividing the EMG confidence by the sum of the visual and EMG confidence levels to obtain the normalized EMG confidence as the second weight, ensuring that the sum of the first and second weights equals one. For example, when the visual confidence is 0.8 and the EMG confidence is 0.4, the first weight is approximately 0.8 ÷ (0.8 + 0.4) ≈ 0.67, and the second weight is approximately 0.4 ÷ (0.8 + 0.4) ≈ 0.33. In this scenario, the weights of both signals are adjusted based on both visual and EMG signal quality.

[0066] It should be noted that, regardless of which scenario is used, the first weight is positively correlated with visual confidence, meaning the higher the visual confidence, the greater the first weight; the second weight is positively correlated with electromyography (EMG) confidence, meaning the higher the EMG confidence, the greater the second weight. Therefore, the more reliable the recognition result, the greater its weight in the fusion process.

[0067] Step S330: For the same candidate sign language word, the probability of the candidate sign language word in the visual recognition result and the probability in the electromyography recognition result are weighted and summed according to the first weight and the second weight to obtain the fusion probability of the candidate sign language word.

[0068] The processing device takes the union of the first candidate sign language phrase from the visual recognition result and the second candidate sign language phrase from the electromyography (EMG) recognition result to obtain all candidate sign language phrases. For a given candidate sign language phrase, the processing device multiplies its probability in the visual recognition result by a first weight and its probability in the EMG recognition result by a second weight, then sums the results to obtain the fusion probability of that candidate sign language phrase. Let P1 be the probability of a candidate sign language phrase in the visual recognition result, P2 be the probability in the EMG recognition result, W1 be the first weight, and W2 be the second weight. Then the fusion probability of the candidate sign language phrase is P = W1 × P1 + W2 × P2. If the candidate sign language phrase only appears in the visual recognition result, its probability P2 in the EMG recognition result is considered zero; if the candidate sign language phrase only appears in the EMG recognition result, its probability P1 in the visual recognition result is considered zero.

[0069] Step S340: Determine whether all candidate sign language entries have been traversed.

[0070] The processing device determines whether the fusion probability has been calculated for all candidate sign language entries. If not all candidate sign language entries have been traversed, it returns to step S330 to calculate the fusion probability for the next candidate sign language entry; if all candidate sign language entries have been traversed, it proceeds to step S350.

[0071] Step S350: Select the candidate sign language word with the highest fusion probability as the sign language recognition result.

[0072] The processing device compares the fusion probabilities of each candidate sign language word and takes the candidate sign language word with the highest fusion probability as the sign language recognition result.

[0073] To facilitate understanding of the above weighted fusion process, the following description continues with an example where the user expresses the sign language entry "sorry", and the confidence levels include a visual confidence level and an electromyographic confidence level. Assume that the processing device generates a visual confidence level of 0.7 and an electromyographic confidence level of 0.3 in step S300, and after normalization processing, the first weight is 0.7 and the second weight is 0.3. The visual recognition result is "sorry 0.48, thank you 0.52, goodbye 0.00", and the electromyographic recognition result is "sorry 0.65, thank you 0.30, hello 0.05".

[0074] The processing device takes the union of the two candidate sign language entries, and obtains all candidate sign language entries as "sorry", "thank you", "goodbye" and "hello", wherein the probability of "goodbye" in the electromyographic recognition result is counted as zero, and the probability of "hello" in the visual recognition result is counted as zero. The processing device sequentially calculates the fusion probability of each candidate sign language entry: For the candidate sign language entry "sorry", P = 0.7 × 0.48 + 0.3 × 0.65 = 0.531; For the candidate sign language entry "thank you", P = 0.7 × 0.52 + 0.3 × 0.30 = 0.454; For the candidate sign language entry "goodbye", P = 0.7 × 0.00 + 0.3 × 0.00 = 0.000; For the candidate sign language entry "hello", P = 0.7 × 0.00 + 0.3 × 0.05 = 0.015.

[0075] The processing device compares the above fusion probabilities, and takes the candidate sign language entry "sorry" with the highest fusion probability as the sign language recognition result. It can be seen that the judgment of the visual recognition result for "sorry" and "thank you" is somewhat ambiguous (the two probabilities are close and "thank you" is slightly higher). By introducing the electromyographic recognition result for weighted fusion, the sign language recognition result "sorry" is finally correctly obtained.

[0076] In practical applications, there is a situation where the visual signal collection quality is good, but the probabilities of a plurality of first candidate sign language entries in the visual recognition result are close to each other. For example, both the sign language entries "sorry" and "thank you" involve similar movements of the hands in front of the body. When the user expresses "sorry", the probabilities of the two first candidate sign language entries "sorry" and "thank you" in the visual recognition result may be very close. At this time, even if the visual signal quality is good and the visual confidence level is high, it is difficult to accurately distinguish the sign language entry actually expressed by the user only based on the visual recognition result.

[0077] In view of the above situation, the embodiments of the present application provide another sign language recognition method, the schematic flowchart of which is shown in Figure 3 As shown, the method is in Figure 1 Based on steps S100 and S200, after aligning visual features and electromyographic features in the time dimension, a conflict arbitration judgment and processing is added, specifically including the following steps: Steps S100 and S200 and Figure 1 The sign language recognition method shown is the same and will not be described again here. After aligning the visual features and electromyographic features in the time dimension in step S200, step S310 is executed.

[0078] Step S310: Generate at least one of visual confidence and electromyographic confidence.

[0079] Step S320: Determine a first weight for the visual recognition result and a second weight for the electromyographic recognition result based on at least one of the visual confidence score and the electromyographic confidence score.

[0080] Step S330: For the same candidate sign language term, the probability of the candidate sign language term in the visual recognition result and the probability in the electromyography recognition result are weighted and summed according to the first weight and the second weight to obtain the fusion probability of the candidate sign language term. Steps S310, S320 and S330 have been described in the foregoing embodiments and will not be repeated here.

[0081] Step S360: Determine the probability difference between the first candidate sign language word with the highest probability and the second highest probability first candidate sign language word in the visual recognition results.

[0082] The processing device sorts the probabilities of each first candidate sign language word in the visual recognition results, records the probability of the first candidate sign language word with the highest probability as the highest probability, records the probability of the first candidate sign language word with the second highest probability as the second highest probability, and calculates the probability difference between the highest probability and the second highest probability.

[0083] It should be noted that, as described in the aforementioned embodiments, the visual recognition result itself includes multiple first candidate sign language words and the probability of each first candidate sign language word. Therefore, the processing device can directly obtain the probability of each first candidate sign language word from the visual recognition result, and determine the highest probability and the second highest probability by comparing the probabilities numerically, without the need for additional processing.

[0084] Step S370: Determine whether the probability difference is less than a preset threshold.

[0085] The processing device compares the probability difference with a preset threshold. The preset threshold is a critical value used to determine whether there is semantic ambiguity in the visual recognition result. The preset threshold can be preset according to the actual application scenario, for example, set to 0.1.

[0086] The smaller the probability difference, the closer the probabilities of the highest-probability first candidate sign language word and the second-highest-probability first candidate sign language word are, the lower the visual recognition result's distinguishability between the two, and the more likely semantic ambiguity exists. When the probability difference is less than a preset threshold, the processing device determines that the visual recognition result has semantic ambiguity and executes step S380; when the probability difference is greater than or equal to the preset threshold, the processing device determines that the visual recognition result has sufficient distinguishability and no semantic ambiguity, and executes step S350.

[0087] Step S350: Select the candidate sign language word with the highest fusion probability as the sign language recognition result.

[0088] The processing device uses the weighted fusion method described in the foregoing implementation method to take the candidate sign language word with the highest fusion probability among the obtained candidate sign language words as the sign language recognition result, which will not be elaborated here.

[0089] Step S380: Reduce the first weight, and re-weight and fuse the visual recognition result and the electromyography recognition result according to the reduced first weight to obtain the sign language recognition result; or, take the second candidate sign language word with the highest probability in the electromyography recognition result as the sign language recognition result.

[0090] After determining that the visual recognition result has semantic ambiguity, the processing device adjusts the fusion process, specifically including any one of the following two methods.

[0091] The first approach is to reduce the first weight and then re-weight and fuse the visual recognition result and the electromyography recognition result based on the reduced first weight. The candidate sign language word with the highest fusion probability after re-weighting and fusion is taken as the sign language recognition result.

[0092] Specifically, in one implementation, the processing device multiplies a determined first weight by a preset reduction factor to obtain a reduced first weight, where the reduction factor is greater than zero and less than one; accordingly, it increases a second weight so that the sum of the reduced first weight and the increased second weight remains one. For example, if the first weight is 0.85 and the second weight is 0.15 before reduction, multiplying the first weight by a reduction factor of 0.3 results in a reduced first weight of 0.255, and the second weight is correspondingly increased to 0.745.

[0093] In another implementation, the processing device reduces the first weight without changing the second weight. In this case, the sum of the reduced first weight and the second weight is less than one. After weighted summation, the processing device normalizes the fusion probability of each candidate sign language entry.

[0094] The processing device, following the weighted summation method described in the aforementioned embodiment, recalculates the fusion probability of each candidate sign language term based on the reduced first weight and the increased second weight, and takes the candidate sign language term with the highest fusion probability after reweighted fusion as the sign language recognition result. By reducing the first weight, the weight of the electromyography (EMG) recognition result in the fusion is increased, thereby using the EMG recognition result to correct visual recognition results with semantic ambiguity.

[0095] The second approach involves using the second candidate sign language word with the highest probability from the electromyography (EMG) results as the sign language recognition result. Specifically, when there is semantic ambiguity in the visual recognition results, the processing device no longer relies on the visual recognition results but directly selects the second candidate sign language word with the highest probability from the EMG results as the sign language recognition result.

[0096] Regardless of which method is used, the processing device obtains the sign language recognition result in step S380.

[0097] After obtaining the sign language recognition result by performing step S350 or step S380, the processing device performs step S400 to generate sign language text and / or synthesized speech based on the sign language recognition result. Step S400 is the same as in the aforementioned implementation method and will not be described again here.

[0098] It should be noted that steps S350 and S380 are two branches to be executed based on the judgment result of step S370: step S350 is executed when the probability difference is greater than or equal to the preset threshold, and step S380 is executed when the probability difference is less than the preset threshold. After obtaining the sign language recognition result by executing either branch, step S400 is entered.

[0099] To better understand the conflict arbitration process described above, we will continue with the example of a user expressing the sign language phrase "sorry." Assume the user is in a well-lit environment with unobstructed hands. The visual confidence score is 0.85, the first weight is 0.85, the electromyographic confidence score is 0.15, and the second weight is 0.15. The visual recognition result is "thank you 0.52, sorry 0.48, goodbye 0.00," while the electromyographic recognition result is "sorry 0.70, thank you 0.25, hello 0.05."

[0100] The processing device sorts the probabilities of each first candidate sign language word in the visual recognition results, determining that the highest probability is 0.52 for "thank you" and the second highest probability is 0.48 for "sorry," calculating the probability difference as 0.52. 0.48 = 0.04.

[0101] The processing device compares the probability difference of 0.04 with the preset threshold of 0.1. Since 0.04 is less than 0.1, the processing device determines that the visual recognition result has semantic ambiguity and executes step S380.

[0102] Assuming the processing device uses the first method, multiplying the first weight of 0.85 by a reduction factor of 0.3 yields a reduced first weight of 0.255, and correspondingly increasing the second weight to 0.745. The processing device then recalculates the fusion probability of each candidate sign language term: For "I'm sorry", P = 0.255 × 0.48 + 0.745 × 0.70 = 0.644; For "thank you", P = 0.255 × 0.52 + 0.745 × 0.25 = 0.319; For "Goodbye", P = 0.255 × 0.00 + 0.745 × 0.05 = 0.037.

[0103] The processing device uses the candidate sign language word "sorry" with the highest fusion probability as the sign language recognition result. It can be seen that, when the probability of "thank you" is slightly higher than that of "sorry" in the visual recognition result, the conflict arbitration reduces the first weight and enhances the role of electromyography recognition result, ultimately correctly obtaining the sign language recognition result of "sorry" actually expressed by the user.

[0104] In practical applications, visual signals may be lost at certain moments during sign language expression. For example, the smart glasses' camera may be blocked by other people or objects when the user is in a crowded place, or the user's hand movements may be too large and exceed the camera's capture range. In the event of lost visual signals, the processing device cannot extract valid visual features from the visual signals, and therefore cannot generate valid visual recognition results.

[0105] For the above situation, please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating the process of intention completion when visual signals are lost, as provided in this embodiment.

[0106] During the normal visual signal segment before the loss of visual signal, the processing device processes the visual signal and electromyographic signal in the manner described in any of the aforementioned embodiments. Simultaneously, the processing device caches visual features to form visual history features. These visual history features refer to the visual features cached by the processing device prior to the loss of visual signal. As shown in Figure 4, the visual history features include visual features F1, F2, F3, F4, F5, and F6 extracted from the normal visual signal segment.

[0107] In one implementation, the processing device determines whether the visual signal is lost based on whether effective visual features can be extracted from the visual signal: when the processing device fails to detect effective hand skeleton key points from the visual signal for a preset number of consecutive frames (e.g., several consecutive frames), or when the number of detected hand skeleton key points is less than a preset number, the processing device determines that the visual signal is lost.

[0108] In another implementation, the processing device determines whether a visual signal is lost based on the detection confidence level of the hand skeleton key points output by the hand detection algorithm: when the detection confidence level of the detected hand skeleton key points is continuously lower than a preset detection threshold, the processing device determines that the visual signal is lost. When the visual signal recovers from the lost state to a state where effective visual features can be extracted normally, the processing device determines that the visual signal has returned to normal.

[0109] When the processing device determines that the visual signal is lost, it enters the visual signal loss segment. In this segment, the processing device cannot extract valid visual features from the visual signal, but the electromyography (EMG) device is unaffected by visual acquisition conditions, and the processing device can still acquire EMG signals and extract EMG features from them. At this point, the processing device performs intent completion processing.

[0110] Specifically, the processing device inputs the cached visual history features prior to the loss of visual signals, as well as the real-time electromyographic features extracted during the period of visual signal loss, into the prediction model. The prediction model refers to a model that predicts subsequent sign language actions based on temporal input data, such as a sequence prediction model based on a long short-term memory network or a self-attention mechanism.

[0111] The predictive model uses hand movement trends reflected in visual history features and current muscle contraction status reflected in electromyography (EMG) features to predict the sign language content expressed by the user during periods of visual signal loss, and outputs the sign language intention. The sign language intention refers to the output predicted by the predictive model based on visual history features and EMG features, reflecting the content expressed by the user during periods of visual signal loss.

[0112] The processing device generates sign language recognition results based on the predicted sign language intentions, and then generates sign language text and / or synthesized speech based on the sign language recognition results. Once the visual signal returns to normal, the processing device resumes processing according to... Figure 1 or Figure 3 The sign language recognition method shown performs sign language recognition and continues to cache visual features for intention completion processing when visual signals are lost again.

[0113] The following example illustrates how a user expresses the sign language phrase "sorry." Suppose that while the user is making the "sorry" gesture, a pedestrian crosses between the user and the smart glasses' camera, causing the visual signal to be lost midway through the action. Before the visual signal is lost, the processing device has cached visual history features F1 to F6, which reflect the trend of the user's hand moving towards the chin area in front of the body. If the processing device fails to detect valid hand skeletal key points for several consecutive frames, it determines that the visual signal is lost and enters the visual signal loss segment.

[0114] During the visual signal loss segment, the processing device acquires continuously collected electromyographic (EMG) signals from the EMG device and extracts real-time EMG features, which reflect the continuous contraction of the user's relevant muscles. The processing device inputs the visual history features F1 to F6 and the real-time EMG features into a prediction model. Based on the hand's movement trend towards the chin and the continuous muscle contraction, the prediction model predicts the user's sign language gesture as "sorry" and outputs the sign language intention "sorry." The processing device then generates the sign language recognition result "sorry." It can be seen that even if the visual signal is lost during the user's expression, the sign language recognition result can still be correctly obtained through intention completion processing.

[0115] In one implementation, the visual history features are the visual features of the last frame before the visual signal was lost. This implementation requires a smaller amount of data to be cached by the processing device.

[0116] In another implementation, the visual history features are multiple frames of visual features within a preset time window before the visual signal is lost. This implementation provides richer information on hand movement trends reflected by the visual history features, resulting in higher accuracy of the prediction model.

[0117] Since electromyography (EMG) signals reflect voltage changes caused by muscle contraction, there is no direct correspondence between EMG signals and sign language entries. A mapping relationship between EMG features and candidate sign language entries needs to be established beforehand so that the processing device can generate EMG recognition results based on the EMG features. This mapping relationship refers to the correspondence between EMG features and candidate sign language entries.

[0118] Considering that different users may have different electromyographic signals when expressing the same sign language words due to differences in physiological factors such as subcutaneous fat thickness, skin dryness and wetness, and muscle mass, this embodiment establishes a mapping relationship for the user through user calibration.

[0119] Please see Figure 5 In one implementation, generating electromyography (EMG) recognition results based on aligned EMG features specifically includes the following steps: During the calibration phase, the processing equipment performs the following steps to establish a mapping relationship.

[0120] Step S211: Collect electromyographic feature samples of the user for the preset sign language words.

[0121] Specifically, the processing device presents the user with preset sign language phrases and guides the user to sequentially perform the sign language actions corresponding to each preset phrase. The preset sign language phrases refer to a set of pre-selected sign language phrases used for calibration, such as "hello," "thank you," "sorry," "yes," and "no." As the user sequentially performs the sign language actions corresponding to each preset phrase, the processing device acquires electromyographic (EMG) signals collected by an EMG device and extracts EMG features. The EMG features extracted when the user performs a sign language action corresponding to a specific preset phrase are used as the user's EMG feature sample for that preset phrase.

[0122] Step S212: Establish the correspondence between electromyographic feature samples and preset sign language entries to obtain the mapping relationship.

[0123] Specifically, the processing device associates each electromyographic (EMG) feature sample with a corresponding preset sign language phrase, thus establishing a mapping relationship. For example, it associates the EMG feature sample collected when the user makes the "sorry" gesture with the preset sign language phrase "sorry," the EMG feature sample collected when the user makes the "thank you" gesture with the preset sign language phrase "thank you," and so on. The processing device stores the established mapping relationships for later retrieval during the recognition phase.

[0124] During the recognition phase, the processing device performs the following steps to generate electromyography (EMG) recognition results.

[0125] Step S213: Obtain the pre-established mapping relationship.

[0126] In this step, the processing device acquires the mapping relationship established and stored during the calibration phase.

[0127] Step S214: Obtain the aligned electromyographic features.

[0128] In this step, the processing device acquires the aligned electromyographic features.

[0129] Step S215: Determine the candidate sign language terms and their probabilities corresponding to the electromyographic features based on the mapping relationship, and use them as the electromyographic recognition results.

[0130] Specifically, the processing device matches the aligned electromyographic features with the electromyographic feature samples corresponding to each preset sign language term in the mapping relationship. Based on the degree of matching, it determines each candidate sign language term and its probability corresponding to the electromyographic feature, which is then used as the electromyographic recognition result. For example, the processing device calculates the similarity between the aligned electromyographic features and each electromyographic feature sample, uses each preset sign language term as a second candidate sign language term, and normalizes the similarity between the aligned electromyographic features and the electromyographic feature samples corresponding to the preset sign language term to use as the probability of the candidate sign language term.

[0131] The following explanation uses the user's expression of the sign language phrase "sorry" as an example. During the calibration phase, the processing device sequentially presents the user with preset sign language phrases such as "hello," "thank you," "sorry," "yes," and "no," collecting electromyographic (EMG) feature samples when the user makes the "sorry" gesture and associating these EMG feature samples with the preset sign language phrase "sorry." During the recognition phase, when the user makes the "sorry" gesture, the processing device obtains the mapping relationship, acquires the aligned EMG features, and matches the aligned EMG features with each EMG feature sample in the mapping relationship. It finds that the EMG feature sample corresponding to "sorry" has the highest similarity, thus generating the EMG recognition result: "sorry 0.78, thank you 0.18, hello 0.04."

[0132] Because the mapping relationship is established specifically for this user, the processing device can generate electromyography (EMG) recognition results more accurately based on the user's EMG characteristics compared to using a general mapping relationship.

[0133] It should be noted that the calibration phase is usually performed when the user uses the device for the first time. After the calibration phase establishes the mapping relationship, the recognition phase can repeatedly call the established mapping relationship without having to recalibrate before each recognition.

[0134] In an optional implementation, the mapping relationship can also be established based on a pre-set general sample library. The general sample library refers to a set of samples formed by pre-collecting electromyographic (EMG) feature samples of multiple sample users expressing various preset sign language terms, and associating each EMG feature sample with its corresponding preset sign language term. After the processing device establishes the mapping relationship based on the general sample library, the user can use the sign language recognition method provided in this application without calibration. This application does not limit the specific method of establishing the mapping relationship.

[0135] The present invention also provides a sign language recognition device based on the above-described sign language recognition method. Please refer to [link to relevant documentation]. Figure 6Figure 6 is a schematic diagram of the sign language recognition device provided in this embodiment. The sign language recognition device 200 includes an acquisition module 201, a recognition module 202, a determination module 203, and an output module 204. The acquisition module 201 is connected to the recognition module 202, the recognition module 202 is connected to the determination module 203, and the determination module 203 is connected to the output module 204. The connection means that data can be transmitted between the modules.

[0136] The acquisition module 201 is used to acquire visual and electromyographic (EMG) signals collected when the user performs sign language expression, and extracts visual features from the visual signals and EMG features from the EMG signals. As described in Embodiment 1, the visual signals are collected by the camera of the smart glasses, and the EMG signals are collected by an EMG device that is in contact with a specific part of the user's body. The acquisition module 201 acquires the above signals through a communication connection with the smart glasses and the EMG device. The acquisition module 201 outputs the extracted visual and EMG features to the recognition module 202.

[0137] The recognition module 202 is used to align visual features and electromyographic features in the time dimension, generate a visual recognition result based on the aligned visual features, and generate an electromyographic recognition result based on the aligned electromyographic features. The recognition module 202 outputs the generated visual recognition result and electromyographic recognition result to the determination module 203.

[0138] The determining module 203 is used to determine the sign language recognition result based on the visual recognition result and the electromyography (EMG) recognition result. In one embodiment, the determining module 203 generates at least one of the visual confidence score and the EMG confidence score as described in Embodiment 2, determines a first weight for the visual recognition result and a second weight for the EMG recognition result based on the confidence score, and for the same candidate sign language word, weights and sums its probabilities in the two recognition results according to the first weight and the second weight to obtain a fusion probability, and takes the candidate sign language word with the highest fusion probability as the sign language recognition result. In another embodiment, the determining module 203, as described in Embodiment 3, reduces the first weight and re-weights and fuses the sign language word when the probability difference between the first candidate sign language word with the highest probability and the second highest probability first candidate sign language word in the visual recognition result is less than a preset threshold, or takes the second candidate sign language word with the highest probability in the EMG recognition result as the sign language recognition result. In another embodiment, the determining module 203, as described in Embodiment 4, predicts the user's sign language expression intention based on visual history features and electromyographic features acquired during the visual signal loss, and uses the recognition result generated based on the sign language expression intention as the sign language recognition result. The determining module 203 outputs the sign language recognition result to the output module 204.

[0139] Output module 204 is used to generate sign language text and / or synthesized speech based on the sign language recognition results.

[0140] It should be noted that the above modules are divided according to function. Each module can be implemented by software, hardware, or a combination of both. Furthermore, the above module division method is only an example. In actual implementation, two or more modules can be merged into one module, or a module can be further divided into multiple sub-modules. This application does not limit the specific method of module division.

[0141] The sign language recognition device 200 can be installed inside smart glasses, or it can be a smartphone, computer, or cloud server that is separate from the smart glasses and electromyography device. This application does not limit the specific form of the sign language recognition device 200.

[0142] The present invention also provides an electronic device based on the above-described sign language recognition method, the schematic diagram of which is shown below. Figure 7 As shown, the electronic device 100 includes: One or more processors 101, a network interface 102, and a memory 103, Figure 7 The example consists of a processor 101, a network interface 102, and a memory 103.

[0143] The network interface 102 is communicatively connected to the corresponding processor 101, and the processor 101 and the memory 102 can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0144] The network interface 102 is used to establish communication connections between the processor 101 and other external devices, including the following types: RJ-45 interface, SC fiber optic interface, AUI interface, FDDI interface and Console interface.

[0145] The memory 103, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The processor 101 executes various functional applications and data processing of the electronic device by running the non-volatile software programs, instructions, and units stored in the memory 103, thereby implementing the sign language recognition method of the above-described method embodiment.

[0146] The memory 103 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 103 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 103 may optionally include memory remotely located relative to the processor 101, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0147] The one or more units are stored in the memory 103, and when executed by one or more processors 101, they execute the sign language recognition method in any of the above method embodiments.

[0148] The aforementioned electronic device can execute the sign language recognition method provided in the embodiments of the present invention, and has the corresponding program modules and beneficial effects for executing the method. Technical details not described in detail in the electronic device embodiments can be found in the sign language recognition method provided in the embodiments of the present invention.

[0149] This invention also provides a non-volatile computer-readable storage medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The non-volatile computer-readable storage medium carries one or more programs, which, when executed, implement the sign language recognition method of this disclosure.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A sign language recognition method, characterized in that, include: The system acquires visual and electromyographic signals collected when a user performs sign language expression, extracts visual features from the visual signals, and extracts electromyographic features from the electromyographic signals. Align the visual features and the electromyographic features in the time dimension, generate a visual recognition result based on the aligned visual features, and generate an electromyographic recognition result based on the aligned electromyographic features. The sign language recognition result is determined based on the visual recognition result and the electromyographic recognition result; Sign language text and / or synthesized speech are generated based on the sign language recognition results.

2. The method according to claim 1, characterized in that, The visual recognition result includes multiple first candidate sign language terms and the probability of each first candidate sign language term; the electromyography recognition result includes multiple second candidate sign language terms and the probability of each second candidate sign language term. The step of determining the sign language recognition result based on the visual recognition result and the electromyographic recognition result includes: Generate at least one of a visual confidence score and an electromyographic confidence score; the visual confidence score characterizes the reliability of the visual feature, and the electromyographic confidence score characterizes the reliability of the electromyographic feature. A first weight and a second weight of the visual recognition result are determined based on at least one of the visual confidence score and the electromyographic recognition result; For the same candidate sign language word, the probability of the candidate sign language word in the visual recognition result and the probability in the electromyography recognition result are weighted and summed according to the first weight and the second weight to obtain the fusion probability of the candidate sign language word. The candidate sign language word with the highest fusion probability is taken as the sign language recognition result.

3. The method according to claim 2, characterized in that, Before using the candidate sign language word with the highest fusion probability as the sign language recognition result, the method further includes: Determine the probability difference between the first candidate sign language word with the highest probability and the second highest probability first candidate sign language word in the visual recognition results; When the probability difference is greater than or equal to a preset threshold, the candidate sign language word with the highest fusion probability is used as the sign language recognition result. When the probability difference is less than a preset threshold, the first weight is reduced, and the visual recognition result and the electromyography recognition result are re-weighted and fused according to the reduced first weight to obtain the sign language recognition result. Alternatively, the second candidate sign language word with the highest probability in the electromyography recognition results can be used as the sign language recognition result.

4. The method according to claim 1, characterized in that, During the acquisition of the visual signal, the visual features are cached as visual history features. The method further includes: When the visual signal is lost, the user's sign language expression intention is predicted based on the visual history features and the electromyographic features acquired during the period of visual signal loss, and the recognition result generated based on the sign language expression intention is used as the sign language recognition result.

5. The method according to claim 1, characterized in that, The step of generating electromyography (EMG) recognition results based on the aligned EMG features includes: Obtain a pre-established mapping relationship; the mapping relationship represents the correspondence between electromyographic features and candidate sign language entries; Based on the mapping relationship, candidate sign language terms and their probabilities corresponding to the electromyographic features are determined as the electromyographic recognition result.

6. The method according to claim 5, characterized in that, The mapping relationship is established in advance through a calibration process, which includes: Collect electromyographic feature samples of the user for preset sign language entries; Establish the correspondence between the electromyographic feature samples and the preset sign language entries to obtain the mapping relationship.

7. A sign language recognition device, characterized in that, include: The acquisition module is used to acquire visual signals and electromyographic signals collected when the user makes sign language expressions, and to extract visual features from the visual signals and electromyographic features from the electromyographic signals. The recognition module is used to align the visual features and the electromyographic features in the time dimension, generate a visual recognition result based on the aligned visual features, and generate an electromyographic recognition result based on the aligned electromyographic features. The determining module is used to determine the sign language recognition result based on the visual recognition result and the electromyography recognition result; The output module is used to generate sign language text and / or synthesized speech based on the sign language recognition results.

8. An electronic device, characterized in that, include: At least one processor; At least one network interface, which is communicatively connected to a corresponding processor; as well as, A memory communicatively connected to the at least one processor; wherein, The network interface is used to establish communication connections between the processor and other external devices; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the sign language recognition method as described in any one of claims 1-6.

9. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, which are executed by one or more processors to cause the one or more processors to perform the sign language recognition method as described in any one of claims 1-6.

Images