Sign language recognition method and device

Abstract

The invention relates to a sign language recognition method and device, and belongs to the technical field of computer vision and voice signal processing, the method is used for sign language recognition based on a sign language recognition model, and the sign language recognition model comprises a feature extraction module, a feature fusion module and a recognition module; the method comprises the following steps: extracting an action part image feature and a voice feature of a sign language person from an obtained sign language video through a feature extraction module; fusing the action part image features and the voice features through a feature fusion module to obtain fused features; and recognizing the fusion features through a recognition module to obtain a sign language recognition result. The sign language recognition model is used for recognizing the fusion feature combining the image feature of the action part of the sign language person and the voice feature of the sign language person, the intention of the sign language person can be fully understood, and the sign language recognition accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and speech signal processing, and in particular to a sign language recognition method and device. Background Art

[0002] Sign language is the main way of communication for the deaf and hearing-impaired. The research on sign language recognition technology is of great significance to promoting barrier-free communication. Existing sign language recognition technology mainly relies on visual signals, capturing the hand movements and gestures of the signer through a camera, and using computer vision algorithms for recognition. In simple and static environments, these methods can achieve good results. However, in actual applications, sign language communication often occurs in complex scenarios, such as outdoors, public places, and multi-person environments. At this time, the background is complex and changeable, the lighting conditions are unstable, and there are various interference factors, which leads to a significant decrease in recognition accuracy. Summary of the invention

[0003] In view of this, it is necessary to provide a sign language recognition method and device to solve the problem of reduced sign language recognition accuracy in complex scenarios in the prior art.

[0004] In order to solve the above problems, in a first aspect, the present invention provides a sign language recognition method for performing sign language recognition based on a sign language recognition model, wherein the sign language recognition model includes a feature extraction module, a feature fusion module and a recognition module; the method includes: Extracting the image features of the signer's motion parts and the voice features from the acquired sign language video by the feature extraction module; The action part image feature and the speech feature are fused by the feature fusion module to obtain a fused feature; The fusion feature is recognized by the recognition module to obtain a sign language recognition result.

[0005] Optionally, the sign language recognition model is obtained by training sample sign language videos, and the sample sign language videos include a first sample sign language video without dubbing processing, and a second sample sign language action video with dubbing processing.

[0006] Optionally, the sign language recognition model includes a data preprocessing module, and the data preprocessing module includes an encoder, a dilated space pyramid pooling unit, and a decoder; and the method further includes: Extracting features of a first image in the acquired sign language video by using MobileNet in the encoder to obtain first image features; Processing the first image feature by the atrous spatial pyramid pooling unit to obtain a second image feature; Decoding the second image feature by the decoder to obtain a second image; The extracting of the image features of the action parts of the signer from the acquired sign language video by the feature extraction module includes: The feature extraction module extracts the image features of the signer's action parts from the second image.

[0007] Optionally, the data preprocessing module further includes a background removal unit, a key image extraction and cropping unit, and the method further includes: Removing the foreground image from the second image by the background removal unit to obtain a third image; The key image extraction and cropping unit extracts a key third image from the third image according to the degree of difference between the third images, and crops the key third image to remove other areas except the main action area in the key third image, thereby obtaining a fourth image; wherein the main action area includes the hand action area of ​​the signer; The step of extracting the image features of the sign language user's action parts from the second image by the feature extraction module includes: The feature extraction module extracts the image features of the sign language user's action parts from the fourth image.

[0008] Optionally, the feature extraction module includes a first three-dimensional residual convolutional neural network combined with a spatial attention mechanism; extracting the image features of the signer's action parts from the fourth image by the feature extraction module includes: The third image feature is obtained by performing feature extraction on the fourth image through the first three-dimensional residual convolutional neural network combined with the spatial attention mechanism.

[0009] Optionally, the feature fusion module includes a first alignment unit and a feature fusion unit; fusing the action part image feature and the speech feature through the feature fusion module to obtain a fusion feature includes: Performing time alignment on the third image feature and the speech feature by using a dynamic time warping algorithm through the first alignment unit; The aligned third image feature and the speech feature are fused by the feature fusion unit to obtain a fused feature.

[0010] Optionally, the feature fusion module further includes a second alignment unit; and the method further includes: The third image feature and the speech feature are dimensionally aligned by the second alignment unit.

[0011] Optionally, the feature fusion unit fuses the aligned third image feature and the speech feature to obtain a fused feature, including: The feature fusion unit determines the weights of the third image feature and the speech feature based on the multimodal attention mechanism, and performs weighted summation of the aligned third image feature and the speech feature according to the weights to obtain a fused feature.

[0012] Optionally, the recognition module includes a second three-dimensional residual convolutional neural network and a long short-term memory network; the recognition module recognizes the fusion feature to obtain a sign language recognition result, including: Encoding the fusion features through the second three-dimensional residual convolutional neural network; The encoded fusion features are decoded through the long short-term memory network to obtain text information corresponding to the sign language actions in the sign language video.

[0013] In a second aspect, the present invention further provides a sign language recognition device for performing sign language recognition based on a sign language recognition model, wherein the sign language recognition model includes a feature extraction module, a feature fusion module and a recognition module; the device includes: A feature recognition functional component is used to extract the image features of the signer's movement parts and the voice features from the acquired sign language video through the feature extraction module; A feature fusion functional component, used to fuse the action part image feature and the voice feature through the feature fusion module to obtain a fusion feature; The sign language recognition functional component is used to recognize the fusion feature through the recognition module to obtain a sign language recognition result.

[0014] The beneficial effects of the present invention are: The present invention extracts the image features of the signer's movement parts and the voice features in the sign language video through a sign language recognition model, and then recognizes the sign language content in combination with the image features of the movement parts and the voice features, so as to fully understand the signer's intention and improve the accuracy of sign language recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of a flow chart of an embodiment of a sign language recognition method provided by the present invention; Figure 2 A schematic diagram of a sign language video recognition architecture provided by the present invention; Figure 3 A schematic diagram of a data preprocessing process provided by the present invention; Figure 4 A schematic diagram of a visual signal processing flow provided by the present invention; Figure 5 A schematic diagram of a speech signal processing flow provided by the present invention; Figure 6 A schematic diagram of a multimodal feature fusion process provided by the present invention; Figure 7 A schematic diagram of a sequence learning and time synchronization process provided by the present invention; Figure 8 A schematic structural diagram of an embodiment of a sign language recognition device provided by the present invention. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0017] In the description of the embodiments of the present invention, unless otherwise specified, the meaning of "plurality" is two or more. The "first", "second", etc. involved in the embodiments of the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence, nor are they used to indicate or imply their relative importance or implicitly indicate the number of technical features indicated. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more.

[0018] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0019] Reference Figure 1 , shows a flow chart of an embodiment of a sign language recognition method provided by the present invention, the method is used to perform sign language recognition based on a sign language recognition model, the sign language recognition model includes a feature extraction module, a feature fusion module and a recognition module; the method includes: S101, extracting the image features of the movement parts and the voice features of the signer from the acquired sign language video through a feature extraction module.

[0020] S102, fusing the action part image features and the speech features through a feature fusion module to obtain fused features.

[0021] S103, identifying the fused features through a recognition module to obtain a sign language recognition result.

[0022] In some cases, the signer may emit some voice signals, such as breath sounds, while making the sign language movements. These voice signals can also serve as auxiliary information to help the model understand the sign language content more accurately. Especially in complex scenes, relying solely on the recognition of the signer's hand movements without fully utilizing these potential auxiliary information will lead to limited sign language recognition performance.

[0023] Based on this, this embodiment uses a sign language recognition model to extract the image features of the signer's movement parts and voice features in the sign language video, and then recognizes the sign language content in combination with the image features of the movement parts and the voice features, so as to fully understand the signer's intention and improve the accuracy of sign language recognition.

[0024] In one embodiment, the sign language recognition model is trained by sample sign language videos, where the sample sign language videos include a first sample sign language video without dubbing processing and a second sample sign language action video with dubbing processing.

[0025] During the model training phase, if the signer does not speak in the sample sign language video obtained, the sample sign language video can be dubbed by manual dubbing or text-to-speech technology to obtain a second sample sign language action video. Manual dubbing can be performed by professionals who record synchronized voice descriptions based on the sample sign language video. Text-to-speech technology can use the TTS engine to convert the text content corresponding to the sign language into voice signals, ensuring multiple languages ​​and voice styles. During dubbing, in order to ensure the precise synchronization of the audio signal and the video signal on the timeline, timestamp or synchronization signal proofreading can be used to handle possible delays and offsets in the audio and video to ensure the consistency of multimodal data.

[0026] In one embodiment, the sign language recognition model includes a data preprocessing module, which includes an encoder, an atrous spatial pyramid pooling (ASPP) unit and a decoder; the sign language recognition method may also include: extracting features of a first image in the acquired sign language video through MobileNet in the encoder to obtain first image features; processing the first image features through the atrous spatial pyramid pooling unit to obtain second image features; decoding the second image features through the decoder to obtain a second image. The step of extracting the image features of the signer's action parts from the acquired sign language video through the feature extraction module may include: extracting the image features of the signer's action parts from the second image through the feature extraction module.

[0027] In the encoder part, the depth-wise separable convolution and batch normalization layers of MobileNet are used to extract multi-scale first image features from images in the sign language video. The ASPP module contains multiple dilated convolution layers, and the dilation rate can be set to 3, 6, and 9 respectively. The first image features are further processed by the ASPP module to expand the receptive field and capture spatial information of different scales. The decoder can restore the second image feature map to the original image size through bilinear upsampling and convolution layers. The encoder and decoder layers are connected through jumps to fuse shallow and deep features to improve the segmentation effect.

[0028] This embodiment proposes a complex background removal method based on deep learning, which uses a lightweight MobileNet combined with ASPP and an encoder-decoder structure to effectively remove the complex background in the first image and obtain a second image. Afterwards, the image features of the signer's action parts are extracted from the second image to reduce background interference and improve feature extraction efficiency.

[0029] In one embodiment, the data preprocessing module further includes a background removal unit, a key image extraction and cropping unit, and the sign language recognition method further includes: removing the foreground image in the second image by the background removal unit to obtain a third image; extracting a key third image from the third image according to the degree of difference between each third image by the key image extraction and cropping unit, and cropping the key third image to remove other areas in the key third image except the main action area to obtain a fourth image; wherein the main action area includes the hand action area of ​​the signer. The step of extracting the image features of the signer's action parts from the second image by the feature extraction module may include: extracting the image features of the signer's action parts from the fourth image by the feature extraction module.

[0030] The second image is an Alpha image. By extracting the foreground image of the sign language video and removing the complex background in the Alpha image, a third image with a pure background can be obtained.

[0031] Next, the third image with a pure background can be subjected to key image extraction and center cropping. Key image extraction calculates the difference between the third images and selects the key third image with significant changes to reduce redundant information. Center cropping is to focus on the main action area of ​​the signer and further remove irrelevant background. The fourth image obtained after cropping is uniformly adjusted to a fixed size for subsequent model processing.

[0032] In this embodiment, the foreground image in the second image is further removed to obtain the third image, and then the third image is subjected to key image extraction and center cropping to obtain the fourth image. Afterwards, the image features of the signer's action parts are extracted from the fourth image, which can reduce background interference, reduce data redundancy, and improve feature extraction efficiency.

[0033] In this embodiment, the fourth image may be further enhanced, normalized, and standardized. The enhancement process includes: using histogram equalization to adjust the brightness and contrast of the fourth image and enhance the hand details in the fourth image. The normalization process includes: normalizing the pixel values ​​of the fourth image to the interval [0,1]. The standardization process includes: subtracting the mean of the pixels of the fourth image and dividing by the standard deviation, thereby eliminating data distribution differences and improving the convergence speed and stability of the model.

[0034] In one embodiment, the feature extraction module also includes a first three-dimensional residual convolutional neural network combined with a spatial attention mechanism; the step of extracting image features of the signer's movement parts from the fourth image through the feature extraction module may also include: extracting features of the fourth image through the first three-dimensional residual convolutional neural network combined with the spatial attention mechanism to obtain third image features.

[0035] The main body of the three-dimensional residual convolutional neural network (ACN encoder) is composed of 3D-ResNet18. The main body of the three-dimensional residual convolutional neural network includes residual blocks. In this embodiment, a spatial attention module is added after each residual block to enhance the model's attention to the sign language action area. The spatial attention module performs global average pooling (GAP) and global maximum pooling (GMP) on the input feature map F to obtain two feature maps Favg and Fmax. It is concatenated in the channel dimension to form a feature map Fs, and a 7*7*1 convolution is performed to generate a spatial attention weight map Ms :

[0036] in, is the Sigmoid activation function, Represents convolution. Ms Multiplying with the original feature map obtains the weighted feature map.

[0037] This embodiment adopts the first three-dimensional residual convolutional neural network combined with the spatial attention mechanism, which can obtain the spatiotemporal fusion features of the fourth image, capture the key action information in the fourth image, and reduce redundant data.

[0038] In one embodiment, the feature fusion module includes a first alignment unit, a second alignment unit and a feature fusion unit; the step of fusing the action part image features and the voice features through the feature fusion module to obtain a fused feature may also include: using a dynamic time warping algorithm through the first alignment unit to time align the third image features and the voice features; using the second alignment unit to dimensionally align the third image features and the voice features; and using the feature fusion unit to fuse the aligned third image features and the voice features to obtain a fused feature.

[0039] This embodiment ensures the synchronization of multimodal data by aligning the speech features and the third image features in time and dimension. In terms of time alignment, the dynamic time warping (DTW) algorithm is used to align the speech features and the third image features in time to eliminate the time deviation between different modalities. Among them, the DTW algorithm calculates the distance matrix between the speech features and the third image features to find the best matching path and achieve alignment. In terms of dimensional alignment, the speech features and the third image features are linearly transformed or dimensionality reduced, such as using principal component analysis (PCA) or linear discriminant analysis (LDA), so that the speech features and the third image features have the same dimension, which is convenient for subsequent feature fusion.

[0040] In this embodiment, the data preprocessing module can also be used to: extract the voice signal in the sign language video and preprocess the voice signal; then the feature extraction model can also be used to extract features from the preprocessed voice signal to obtain voice features.

[0041] The preprocessing of speech signals may include: first, using short-time Fourier transform (STFT) to convert speech signals from time domain to frequency domain. For example, the speech signal is divided into short-time frames (25ms frame length, 10ms frame shift), and each frame is Fourier transformed to obtain a spectrum representation. Then, the spectrum of the background noise is estimated using the spectral subtraction method, and the noise spectrum is subtracted from the speech spectrum of each frame to obtain the net speech spectrum. In order to further enhance the speech signal, a linear adaptive filter (LMS) algorithm can be used to dynamically adjust the filter coefficients according to the error between the expected signal and the actual signal to suppress residual noise. In addition, voice activity detection (VAD) can also be performed to detect effective speech segments in the speech signal and remove silent and noisy segments through features such as energy threshold and zero crossing rate. According to the acquisition environment and noise level, the parameters of the noise reduction and filtering algorithms are adjusted to achieve the best effect. In order to meet the needs of real-time processing and optimize the computational efficiency of the algorithm, FFT can be used to accelerate STFT calculation.

[0042] Feature extraction of the preprocessed speech signal may include: first, extracting Mel frequency cepstral coefficients (MFCC) and log-Mel energy spectrum (Log-Mel spectrum) from the preprocessed speech signal. These acoustic features can effectively simulate the auditory perception of the human ear and capture the acoustic features of speech. Then, deep feature extraction of these acoustic features is performed using a convolutional neural network (CNN). For example, the Log-Mel spectrum is regarded as a two-dimensional image, input into the CNN model, and high-level speech features are extracted through multiple convolutional layers, pooling layers, and fully connected layers. The model is trained on a corpus with speech content annotations to extract speech features with semantic information. The extracted acoustic features are normalized in terms of mean and variance to eliminate differences under different recording conditions and improve the stability of the features.

[0043] In one embodiment, the step of fusing the aligned third image features and speech features through a feature fusion unit to obtain a fused feature includes: determining the weights of the third image features and the speech features based on a multimodal attention mechanism through the feature fusion unit, and performing weighted summation of the aligned third image features and speech features according to the weights to obtain a fused feature. In the feature fusion process, feature selection or regularization methods may be used to prevent collinearity and redundancy between features. As part of the model, the fusion strategy may be jointly optimized during the model training process to improve the fusion effect.

[0044] In one embodiment, the speech recognition model also includes a second three-dimensional residual convolutional neural network and a long short-term memory network; the step of identifying the fused features through the recognition module to obtain the sign language recognition result may include: encoding the fused features through the second three-dimensional residual convolutional neural network; decoding the encoded fused features through the long short-term memory network to obtain text information corresponding to the sign language actions in the sign language video.

[0045] The second three-dimensional residual convolutional neural network may be the same as or different from the first three-dimensional residual convolutional neural network.

[0046] In this embodiment, an encoder-decoder architecture is used to encode and decode the fused features to obtain text information corresponding to the sign language action in the sign language video. The encoder can be composed of the aforementioned ACN encoder, and the decoder uses a long short-term memory network (LSTM) combined with a temporal attention mechanism.

[0047] In the decoding process, the temporal attention mechanism is used to make the decoder pay more attention to the important fragments in the entire sign sentence and enhance the decoding performance. Specifically, the decoder calculates the attention score et,i between the decoder hidden state ht and the encoder hidden state sequence {h1,h2,…,h3} at each time step t:

[0048] The attention score is normalized by the Softmax function to obtain the attention weight :

[0049] Using the attention weights, we perform a weighted summation of the encoder hidden states to obtain the context vector :

[0050] The current hidden state of the decoder From the previous moment , the output of the previous moment and context vector Determine, the update formula is:

[0051] Among them, LSTMCell represents the update function of the LSTM unit. The decoder outputs the prediction at the current moment , the probability distribution of generated vocabulary is calculated through the Softmax function.

[0052] In order to avoid the propagation of erroneous information during decoding, a teacher forcing strategy can be used during training. According to a certain ratio, the real target sequence is used as the input of the decoder to guide the model to learn the correct sequence generation. The teacher forcing ratio is a hyperparameter, and the optimal value is determined through experiments.

[0053] In one embodiment, the sign language recognition model further includes an output module, which is used to obtain text information corresponding to the sign language action in the sign language video, output the text information for the user to view, or use the text-to-speech technology ChatTTS to convert the recognized text information into speech, and output the speech for the user to listen to. The output module supports multi-language output.

[0054] Reference Figure 2, showing a schematic diagram of a sign language video recognition architecture provided by the present invention. The sign language video recognition architecture includes: a data acquisition module, a data preprocessing module, a feature extraction module, a feature fusion module, a sequence learning and time synchronization module, and an output module. The data acquisition module is used to capture the sign language video of the signer in real time through a camera and other devices. The data preprocessing module is used to preprocess the voice in the sign language video to obtain a preprocessed voice signal, and to preprocess the first image in the sign language video to obtain a fourth image. The feature extraction module is used to extract voice features from the preprocessed voice signal, and to extract third image features from the fourth image. The feature fusion module is used to fuse the voice features with the third image features to obtain fused features. The sequence learning and time synchronization module is used to pre-adopt an encoder-decoder architecture to perform sequence modeling on the sequence of fused features, and then obtain text information corresponding to the sign language action in the sign language video according to the fused features and sequence modeling in actual application. The output module is used to output text information or speech converted from text information.

[0055] Reference Figure 3 , showing a schematic diagram of a data preprocessing process provided by the present invention. Data preprocessing includes preprocessing of a visual signal (first image) in a sign language video, and preprocessing of a speech signal in a sign language video. Preprocessing of visual signals mainly includes complex background removal, key image extraction, cropping, image enhancement and normalization. Preprocessing of speech signals mainly includes: noise reduction, speech enhancement and speech feature normalization.

[0056] Reference Figure 4 , showing a schematic diagram of a visual signal processing flow provided by the present invention. The input of the sign language recognition model is a first image, that is, a video frame in a sign language video. The input first image is sequentially passed through an encoder, a dilated space pyramid pooling module, a decoder, a background removal module, a key image extraction and cropping module, an ACN encoder, and an ACN decoder to obtain text information corresponding to the sign language action in the sign language video.

[0057] Reference Figure 5 , shows a schematic diagram of a speech signal processing flow provided by the present invention. The pre-processed speech signal can be subjected to feature extraction through a CNN network to obtain speech features.

[0058] Reference Figure 6 , showing a schematic diagram of a multimodal feature fusion process provided by the present invention. The feature fusion module performs time alignment and dimension alignment on the third image feature and the speech feature respectively, and then determines the weights of the third image feature and the speech feature based on the multimodal attention mechanism, and performs weighted summation on the aligned third image feature and the speech feature according to the weights to obtain the fusion feature F.

[0059] Reference Figure 7 , showing a schematic diagram of a sequence learning and time synchronization process provided by the present invention. The ACN encoder is used to encode the fusion features, and the LSTM network combined with the time attention mechanism is used to decode the encoded fusion features to generate a text sequence.

[0060] In summary, the beneficial effects of the present invention include: 1. Improved the accuracy of sign language recognition. The present invention extracts voice signals from sign language videos and deeply fuses voice signals with visual signals, making full use of the complementarity of multimodal data. Through multimodal data fusion, the model can understand the sign language content more comprehensively, reduce recognition errors caused by insufficient single modality information, and significantly improve recognition accuracy.

[0061] 2. Enhanced the robustness of the model in complex backgrounds. This paper uses a lightweight MobileNet combined with dilated spatial pyramid pooling and an encoder-decoder structure to remove the background in the image, which can effectively remove the complex background in the sign language video, allowing the model to accurately capture the hand movements of the signer in various complex scenes and reduce the impact of background interference on recognition.

[0062] 3. Enhanced the effect of multimodal feature fusion: The present invention uses a dynamic time warping algorithm to time-align visual and speech features, and uses a multimodal attention mechanism to dynamically adjust the weights of each modal feature. In this way, the model effectively solves the problems of time deviation and dimensionality differences between different modal features, and enhances the fusion effect of multimodal features.

[0063] 4. Improved the effectiveness of feature extraction: The present invention introduces a spatial attention mechanism into the three-dimensional residual convolutional neural network, thereby enhancing the model's attention to the sign language action area, reducing the interference of redundant information, and improving the quality of feature representation.

[0064] 5. Improved real-time and efficiency of model recognition: In the data preprocessing stage, the present invention uses key image extraction and center cropping to reduce the redundancy of image data and the computational burden of subsequent processing. Combined with a lightweight network structure and optimized algorithms, the model achieves real-time processing while ensuring high accuracy, meeting the timeliness requirements of practical applications.

[0065] 6. Improved user experience: The model can not only convert sign language into text, but also convert text into speech output through text-to-speech technology. This meets the needs of different users, facilitates communication between sign language and spoken language, and improves the practicality of the model.

[0066] Reference Figure 8, showing a schematic diagram of the structure of an embodiment of a sign language recognition device provided by the present invention. The device 80 is used to perform sign language recognition based on a sign language recognition model, the sign language recognition model includes a feature extraction module, a feature fusion module and a recognition module; the device 80 includes: The feature recognition function component 801 is used to extract the image features of the signer's movement parts and the voice features from the acquired sign language video through the feature extraction module; The feature fusion functional component 802 is used to fuse the action part image feature and the voice feature through the feature fusion module to obtain a fusion feature; The sign language recognition function component 803 is used to recognize the fusion features through the recognition module to obtain the sign language recognition result.

[0067] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0068] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A sign language recognition method, characterized in that: Used to perform sign language recognition based on a sign language recognition model, the sign language recognition model includes a feature extraction module, a feature fusion module and a recognition module; the method includes: Extracting the image features of the signer's motion parts and the voice features from the acquired sign language video by the feature extraction module; The action part image feature and the speech feature are fused by the feature fusion module to obtain a fused feature; The fusion feature is recognized by the recognition module to obtain a sign language recognition result.

2. The sign language recognition method according to claim 1, characterized in that: The sign language recognition model is obtained by training sample sign language videos, wherein the sample sign language videos include a first sample sign language video without dubbing processing and a second sample sign language action video with dubbing processing.

3. The sign language recognition method according to claim 1, characterized in that: The sign language recognition model includes a data preprocessing module, and the data preprocessing module includes an encoder, a dilated space pyramid pooling unit, and a decoder; the method also includes: Extracting features of a first image in the acquired sign language video by using MobileNet in the encoder to obtain first image features; Processing the first image feature by the atrous spatial pyramid pooling unit to obtain a second image feature; Decoding the second image feature by the decoder to obtain a second image; The extracting of the image features of the action parts of the signer from the acquired sign language video by the feature extraction module includes: The feature extraction module extracts the image features of the signer's action parts from the second image.

4. The sign language recognition method according to claim 3, characterized in that: The data preprocessing module further includes a background removal unit, a key image extraction and clipping unit, and the method further includes: Removing the foreground image from the second image by the background removal unit to obtain a third image; The key image extraction and cropping unit extracts a key third image from the third image according to the degree of difference between the third images, and crops the key third image to remove other areas except the main action area in the key third image, thereby obtaining a fourth image; wherein the main action area includes the hand action area of ​​the signer; The step of extracting the image features of the sign language user's action parts from the second image by the feature extraction module includes: The feature extraction module extracts the image features of the sign language user's action parts from the fourth image.

5. The sign language recognition method according to claim 4, characterized in that: The feature extraction module includes a first three-dimensional residual convolutional neural network combined with a spatial attention mechanism; extracting the image features of the signer's action parts from the fourth image by the feature extraction module includes: The third image feature is obtained by performing feature extraction on the fourth image through the first three-dimensional residual convolutional neural network combined with the spatial attention mechanism.

6. The sign language recognition method according to claim 5, characterized in that: The feature fusion module includes a first alignment unit and a feature fusion unit; the feature fusion module is used to fuse the action part image feature and the speech feature to obtain a fusion feature, including: Performing time alignment on the third image feature and the speech feature by using a dynamic time warping algorithm through the first alignment unit; The aligned third image feature and the speech feature are fused by the feature fusion unit to obtain a fused feature.

7. The sign language recognition method according to claim 6, characterized in that: The feature fusion module further includes a second alignment unit; and the method further includes: The third image feature and the speech feature are dimensionally aligned by the second alignment unit.

8. The sign language recognition method according to claim 6, characterized in that: The step of fusing the aligned third image feature and the voice feature by the feature fusion unit to obtain a fused feature includes: The feature fusion unit determines the weights of the third image feature and the speech feature based on the multimodal attention mechanism, and performs weighted summation of the aligned third image feature and the speech feature according to the weights to obtain a fused feature.

9. The sign language recognition method according to claim 1, characterized in that: The recognition module includes a second three-dimensional residual convolutional neural network and a long short-term memory network; the recognition module recognizes the fusion feature to obtain a sign language recognition result, including: Encoding the fusion features through the second three-dimensional residual convolutional neural network; The encoded fusion features are decoded through the long short-term memory network to obtain text information corresponding to the sign language actions in the sign language video.

10. A sign language recognition device, characterized in that: Used to perform sign language recognition based on a sign language recognition model, wherein the sign language recognition model includes a feature extraction module, a feature fusion module and a recognition module; the device includes: A feature recognition functional component is used to extract the image features of the signer's movement parts and the voice features from the acquired sign language video through the feature extraction module; A feature fusion functional component, used to fuse the action part image feature and the voice feature through the feature fusion module to obtain a fusion feature; The sign language recognition functional component is used to recognize the fusion feature through the recognition module to obtain a sign language recognition result.

Images