Continuous sign language recognition method, electronic equipment and storage medium

Abstract

The invention relates to the technical field of sign language recognition, and provides a continuous sign language recognition method, electronic equipment and a storage medium. The method comprises the following steps: firstly, collecting and segmenting a continuous sign language video, extracting an isolated sign language action boundary through a BIO tag, and constructing a sign language library after data enhancement; then key point information of hands, faces and upper bodies is extracted, and comprehensive feature vectors are generated in combination with RGB features; and extracting double-flow sequence information by using an online CSLR model and an offline ISLR model, and translating in a text network to generate a sign language text sequence. According to the method, the accuracy and real-time performance of continuous sign language recognition can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision and natural language processing, and in particular to a continuous sign language recognition method based on key points and spatiotemporal dual-stream fusion, an electronic device and a storage medium. Background Art

[0002] Existing sign language recognition methods mostly focus on isolated sign language motion recognition (ISLR), and lack support for more complex continuous sign language recognition (CSLR) in actual scenarios. In continuous sign language recognition, gestures usually appear in a coherent manner, making it difficult to accurately distinguish the start and end of the action. In addition, existing continuous sign language recognition methods often have problems with insufficient accuracy in action boundary segmentation and key point extraction, and the model accuracy is prone to decline when faced with changing expressions, head postures, and upper body movements. Therefore, in order to achieve recognition and translation in more complex sign language environments, improvements need to be made in segmenting sign language movements, extracting sign language semantic information, and constructing data enhancement to improve the accuracy and real-time performance of continuous sign language recognition, thereby improving the user experience.

[0003] At present, the commonly used sign language recognition model is the continuous sign language recognition model (CSLR) combined with multimodality. The CSLR model is trained by using connection time classification loss and sentence-level annotations. However, during the inference process, these models usually need to process the entire sign language video to make predictions, resulting in high latency and large memory consumption. This method is called offline recognition. Offline recognition actually uses the isolated sign language recognition model (ISLR). First, the ISLR model is pre-trained to segment the sign language boundary, and then the processed data is input into the new CSLR model for training. Due to the lack of practical online recognition solutions, the application of current technology in real-time conversations or emergency situations is still insufficient. Online recognition is crucial in these scenarios, and existing methods generally lack the fusion of key points and RGB information, and it is difficult to combine online recognition and global feature extraction in the same framework. In order to better balance complex action connection and real-time processing, the present invention will combine online CSLR and offline ISLR through the Two-Stream-SLR model (or other networks that can obtain optical flow and RGB dual-stream information) to solve the above problems.

[0004] In the prior art, publication number CN117152828A discloses a method and device for continuous sign language recognition, which trains global temporal features, wherein the trained global temporal features include the correspondence between continuous sign language video frames and sign language words under the global temporal sequence; trains local temporal features, wherein the trained local temporal features include the correspondence between continuous sign language video frames and sign language words under the local temporal sequence; utilizes a cross-context knowledge distillation objective function to fuse the trained local temporal features and continuous sign language text features into the trained global temporal features, and trains the fused global temporal features to generate the correspondence between continuous sign language video frames and continuous sign language words.

[0005] The main problems with the above method are: the introduction of cross-up and down knowledge distillation for feature fusion increases the complexity of the model, and the training time and resource consumption increase significantly when training on large-scale data sets, which is not conducive to real-time sign language recognition needs and is difficult to adapt to sign language recognition scenarios that require timely feedback; after introducing multiple feature training and fusion steps, the parameter adjustment of the model becomes more complicated, requiring more expert knowledge and trial and error to find the optimal parameter configuration, resulting in reduced versatility of the model.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0007] The object of the present invention is to provide a continuous sign language recognition method, an electronic device and a storage medium to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solutions: A continuous sign language recognition method, the specific steps comprising: Step 1: Collect continuous sign language videos, segment each continuous sign language video through the Two-Stream-SLR model, divide the continuous sign language video into several sliding windows, predict the BIO label of each frame in each sliding window, determine the start, duration and end stages of the action through the predicted BIO label, extract the boundaries of isolated sign language actions, divide the continuous sign language actions into individual isolated sign language actions, perform data enhancement to generate enhanced sign language actions, and merge the isolated sign language actions and enhanced sign language actions to generate a sign language library; Step 2: Extract key point information of hands, face and upper body, and generate key point feature vectors based on the key point information; Step 3: Combine the key point features with the RGB features to generate a comprehensive feature vector; Step 4: Extract the dual-stream sequence information in the comprehensive feature vector through the online CSLR model and the offline ISLR model, fuse the dual-stream sequence information to generate the final sequence, translate the final sequence in the text network, and generate a sign language text sequence.

[0009] Furthermore, the principle for dividing the continuous sign language movements into individual isolated sign language movements is as follows: The continuous sign language video is divided into single-frame images. The motion information is captured by the optical flow in the Two-Stream-SLR model, and the appearance information is captured by the RGB stream. The boundary of the action is determined based on the optical flow and RGB streams, and the features of each frame are extracted. The start, inner and outer parts of the sign language action in each frame feature are identified by the Two-Stream-SLR model. The start part indicates the start of the action, the inner part indicates the continuation of the started action, and the outer part indicates the part that does not belong to any action, that is, the end of the action, which are marked as Begin, Inside and Outside respectively. All parts marked as Begin and Inside are extracted to constitute an isolated sign language action.

[0010] Furthermore, the principle for generating enhanced sign language actions is: The enhancement measures include: speeding up or slowing down the video frames of the continuous sign language video, changing the rhythm of the sign language movements, and randomly rotating the images in the continuous sign language video; After the enhancement measures are taken, the enhanced continuous sign language video is processed in the Two-Stream-SLR model in the same way as extracting isolated sign language movements, and the extracted sign language movements are the enhanced sign language movements.

[0011] Furthermore, the principle for generating key point features is: Based on the data of the sign language library, the images extracted from the sign language action video frames are input into the COCO-HRNet model to extract the coordinates of key points including the hands, face and upper body. The coordinate features are generated based on the coordinates of the key points. The formula is: ; in, represents the coordinate features of the i-th key point, represents the horizontal coordinate of the i-th key point, i represents the index of the key point, Represents the ordinate of the i-th key point; The key points include: the midpoint of the palm and the tip of the five fingers, the facial features, and the midpoint of the shoulders and chest of the upper body; The differential features are generated according to the Euclidean distance of the key point coordinates of adjacent frames, and the formula is as follows: ; in, represents the differential feature of the i-th key point, represents the horizontal coordinate of the i-th key point in the j+1-th frame image, Represents the index of the image, represents the horizontal coordinate of the i-th key point in the j-th frame image, represents the ordinate of the i-th key point in the j+1-th frame image, Represents the ordinate of the i-th key point in the j-th frame image; Generate key point feature vector based on coordinate features and differential features, expressed as: ; in, represents the feature vector of the ith key point. Furthermore, the principle for generating the comprehensive feature vector is: ; in, represents the comprehensive feature vector of the i-th key point, represents the RGB feature of the i-th key point, and Includes the i-th key red channel value, green channel value and blue channel value; Calculate the average value of the comprehensive feature vectors of all key points to generate the comprehensive feature vector of a frame of image. The formula is: ; in, Represents the comprehensive feature vector of the image.

[0012] Furthermore, the principle for generating the final sequence is: The two-stream representation models a continuous sign language video and a keypoint sequence generated by an off-the-shelf keypoint estimator; The online CSLR model processes the comprehensive feature vector as follows: ; in, represents the internal state of the online CSLR model after processing the input features at time t, represents the function of the online CSLR model, Q express The comprehensive feature vector of the image at that moment, represents the internal state of the online CSLR model after processing the input features at time t-1; The offline ISLR model processes the comprehensive feature vector as follows: ; in, Represents the internal state of the offline ISLR model after processing the input features. A function representing an offline ISLR model, Represents the comprehensive feature vector of the image; The generated time flow information is: ; in, Represents time flow information, represents the time flow information extraction function; The generated spatial stream information is: ; in, Indicates spatial stream information, represents the spatial stream information extraction function; The formula used to generate the final sequence is: ; in, represents the final sequence, Represent the weight coefficients of temporal stream information and spatial stream information respectively, and .

[0013] Furthermore, the principle for generating the sign language text sequence is as follows: The annotation-to-text network implements the translation process as follows: ; in, Represents the translated text sequence, represents the neural network decoder; The final generated sign language text sequence is represented as: ; in, represents a sequence of sign language text, Indicates that from all possible text sequences X, select the conditional probability The largest sequence.

[0014] The present invention also provides an electronic device, which is used to implement the above-mentioned sign language recognition method, and specifically includes: at least one processor and a memory connected to the processor, wherein: the memory is used to store computer programs or instructions, and the processor is used to execute the computer program or instructions.

[0015] The present invention also provides a storage medium, which is applied to an electronic device, specifically comprising: the storage medium carries one or more computer programs, and the one or more computer programs can be executed by the electronic device to enable the electronic device to implement the above-mentioned continuous sign language recognition method.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The present invention segments and processes continuous sign language videos, extracts the boundaries of each isolated sign language action, and generates an enhanced sign language technology to form a sign language library after data enhancement, so that the system can fully cope with different sign language expressions and process more complex continuous sign language expressions; extracts and integrates key point information, so that the system can capture the most representative action features in sign language expressions, avoids background interference, and improves the accuracy of sign language action recognition; combines key point features with RGB features, integrates the time information and spatial information of sign language actions, thereby better capturing the semantic information of sign language and improving the accuracy of translation.

[0017] The present invention also combines an online CSLR model and an offline ISLR model. The online model processes input data in real time to adapt to the fluidity and timeliness of sign language, while the offline model reuses the fusion of the overall data double-stream sequence information to help the model understand the contextual relationship of sign language during recognition, thereby improving the accuracy and fluency of sign language recognition, especially when faced with complex sign language expressions, it can better restore the semantics, make the translation result more in line with the rhythm of natural human communication, and achieve the purpose of accurate and real-time translation, further improving the user experience. Compared with the prior art, the present invention can not only use all time step information for recognition under offline conditions, but also support online CSLR models to meet real-time scene requirements; at the same time, through the combination of key points and RGB features, the ability to capture and understand complex sign language movements is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a schematic diagram of a method flow chart of an embodiment of the present invention; Figure 2 A line graph showing how the accuracy of the sign language feature model extracted by the embodiment of the present invention varies with the length of the video; Figure 3 The scatter plots are of the changing trends of the accuracy of the five sign language recognition models according to the embodiments of the present invention as a function of the number of experiments; Figure 4 It is a line graph showing the changing trend of the accuracy of the five sign language recognition models according to the embodiment of the present invention as a function of the number of experiments; Figure 5 The scatter plot of the F1 scores of the five sign language recognition models according to the embodiment of the present invention versus the number of experiments; Figure 6 This is a line graph showing the changing trend of the F1 scores of the five sign language recognition models according to the embodiments of the present invention as a function of the number of experiments. DETAILED DESCRIPTION

[0019] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0020] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0021] Example: See also Figure 1-Figure 6 , the present invention provides a technical solution: A continuous sign language recognition method, the specific steps comprising: Step 1: Collect continuous sign language videos, segment each continuous sign language video through the Two-Stream-SLR model, divide the continuous sign language video into several sliding windows, predict the BIO label of each frame in each sliding window, determine the start, duration and end stages of the action through the predicted BIO label, extract the boundaries of isolated sign language actions, divide the continuous sign language actions into individual isolated sign language actions, perform data enhancement to generate enhanced sign language actions, and merge the isolated sign language actions and enhanced sign language actions to generate a sign language library; In this embodiment, the principle for dividing the continuous sign language action into individual isolated sign language actions is as follows: The continuous sign language video is segmented into single-frame images. In the Two-Stream-SLR model, motion information is captured by the optical flow stream, and appearance information is captured by the RGB stream. The boundary of the action is determined based on the optical flow stream and the RGB stream, and the features of each frame are extracted. The start, inner and outer parts of the sign language action in each frame feature are identified by the Two-Stream-SLR model. The start part indicates the start of the action, the inner part indicates the continuation of the started action, and the outer part indicates the part that does not belong to any action, that is, the end of the action, which are marked as Begin, Inside and Outside respectively. All the parts marked as Begin and Inside are extracted to form an isolated sign language action. First, use a two-stream network to separate the sign language actions of the continuous, undivided sign language video, and then use BIO in each sliding window to predict the BIO label of each frame through the model. The predicted BIO label is associated with the real-time action sequence to determine the current action start, duration, and end stage; after processing each window, the action boundary and state are dynamically updated according to the newly predicted BIO label. Repeat the above process for the newly input frame to continuously generate BIO labels to form a seamless prediction sequence The principles for generating enhanced sign language actions are: Enhancement measures include: speeding up or slowing down video frames of continuous sign language videos, analyzing the acceleration, deceleration, and pause characteristics of sign language when normal hearing-impaired people communicate, and randomly rotating images in continuous sign language videos; After the enhancement measures are taken, the enhanced continuous sign language video is processed in the Two-Stream-SLR model in the same way as the method of extracting isolated sign language movements, and the extracted sign language movements are the enhanced sign language movements; The generated sign language library is used as the training set of the model.

[0022] In addition to the Two-Stream-SLR model, other commonly used models for extracting sign language action features include the C3D model, the I3D model, the LSTM-RNN model, and the TSN model, among which: C3D model: 3D convolutional neural network, which can capture the spatial and temporal information of video data at the same time. It is suitable for processing short-term dynamic features, but has limited effect on long videos. I3D model: dilated 3D convolutional neural network, which processes 3D data by dilating 2D convolution and can fully capture the spatiotemporal features in the video, but has high computational cost and long training time; LSTM-RNN model: Long short-term memory-recursive neural network, which is good at processing sequence data and capturing long-term dependencies, but it is difficult to train and may have problems with gradient disappearance or explosion; TSN model: Time segment network, which improves the recognition ability of long-term actions by segmenting videos, but does not adequately handle the dependencies of long-term actions.

[0023] The performance comparison of other models for extracting sign language action features and the Two-Stream-SLR model is shown in Table 1: Table 1. Comparison of model performance for extracting sign language action features ; Under different continuous sign language video lengths, the accuracy comparison of the other four models for extracting sign language action features and the Two-Stream-SLR model under different video lengths is shown in Table 2: Table 2. Comparison of model accuracy for extracting sign language action features ; Tables 1 and 2 reflect the comparison between several models for extracting sign language action features in continuous sign language videos and the Two-Stream-SLR model. It can be seen that the Two-Stream-SLR model is superior to other models in terms of accuracy, enhanced data effect, and sign language library coverage.

[0024] Step 2: Extract key point information of hands, face and upper body, and generate key point feature vectors based on the key point information; In this embodiment, the principle for generating key point features is: The continuous sign language action video is divided into frame sequences, each frame is resized to adapt to the input size of the COCO-HRNet model, each frame image is input into the COCO-HRNet model, the positions of the hands, face and upper body in each frame image are detected, and the coordinates of the key points are extracted.

[0025] Based on the data of the sign language library, the images extracted from the sign language action video frames are input into the COCO-HRNet model to extract the coordinates of key points including the hands, face and upper body. The coordinate features are generated based on the coordinates of the key points. The formula is: ; in, represents the coordinate features of the i-th key point, represents the horizontal coordinate of the i-th key point, i represents the index of the key point, Represents the ordinate of the i-th key point; The key points include: the midpoint of the palm and the tip of the five fingers, the facial features, and the midpoint of the shoulders and chest of the upper body; The coordinate feature reflects the position of the key point in the same frame image, and each key point in the same frame image has a unique coordinate feature.

[0026] The differential features are generated according to the Euclidean distance of the key point coordinates of adjacent frames, and the formula is as follows: ; in, represents the differential feature of the i-th key point, represents the horizontal coordinate of the i-th key point in the j+1-th frame image, Represents the index of the image, represents the horizontal coordinate of the i-th key point in the j-th frame image, represents the ordinate of the i-th key point in the j+1-th frame image, Represents the ordinate of the i-th key point in the j-th frame image; The differential feature reflects the position change of the same key point in adjacent frame images and reflects the dynamic information of the key point. The larger the value of the differential feature, the greater the position change of the key point in adjacent frames and the greater the movement amplitude of the action, which reflects the change characteristics of the sign language action over time.

[0027] Generate key point feature vector based on coordinate features and differential features, expressed as: ; in, Represents the feature vector of the i-th key point.

[0028] Step 3: Combine the key point features with the RGB features to generate a comprehensive feature vector; In this embodiment, the principle for generating the comprehensive feature vector is: ; in, represents the comprehensive feature vector of the i-th key point, represents the RGB feature of the i-th key point, and Includes the i-th key red channel value, green channel value and blue channel value; RGB features mainly contain color and texture information in the video, reflect the appearance characteristics of sign language, and provide transition information between consecutive frames. Key point features reflect the movement trajectory and spatial position of the hands, face, and upper body, and embody the changing characteristics of key points in consecutive frames over time. Combining RGB features and key point features, a comprehensive feature that contains both spatial and temporal information is generated.

[0029] Calculate the average value of the comprehensive feature vectors of all key points to generate the comprehensive feature vector of a frame of image. The formula is: ; in, Represents the comprehensive feature vector of the image; In a frame of image, the comprehensive feature vector of each key point is calculated. The comprehensive feature vector of the entire image is the average value of the comprehensive feature vectors of all key points, which reflects the comprehensive features of this frame of image.

[0030] Step 4: Extract the dual-stream sequence information in the comprehensive feature vector through the online CSLR model and the offline ISLR model, fuse the dual-stream sequence information to generate the final sequence, translate the final sequence in the text network, and generate a sign language text sequence.

[0031] In this embodiment, the principle for generating the final sequence is: The two-stream representation models a continuous sign language video and a keypoint sequence generated by an off-the-shelf keypoint estimator; The online CSLR model processes the comprehensive feature vector as follows: ; in, represents the internal state of the online CSLR model after processing the input features at time t, represents the function of the online CSLR model, Q express The comprehensive feature vector of the image at that moment, represents the internal state of the online CSLR model after processing the input features at time t-1; The function is responsible for converting the current input features and the hidden state of the previous moment into the hidden state of the current moment, that is, the internal state of the model after processing the input features, and capturing the time dependency in the comprehensive feature vector based on the gated recurrent unit.

[0032] The purpose of the online CSLR model is to process sign language input in real time. It processes the information of the current frame at each time step and combines it with the information of the previous frame for recognition, thereby achieving a continuous, streaming recognition process. Therefore, it is necessary to consider the feature changes in the time dimension. The offline ISLR model processes the comprehensive feature vector as follows: ; in, Represents the internal state of the offline ISLR model after processing the input features. A function representing an offline ISLR model, Represents the comprehensive feature vector of the image; The function processes the input features of the entire time to generate hidden states and captures spatial features through the convolutional neural network CNN; The offline ISLR model processes the entire sign language video after it is fully captured, and can utilize information from all time steps, so there is no need to emphasize the time dimension, and the comprehensive features are processed as a whole; The generated time flow information is: ; in, Represents time flow information, represents the time flow information extraction function; The function is to extract time series information, use a model that can process time series, and use gated recurrent units to extract time flow information in online CSLR models and offline ISLR models; The generated spatial stream information is: ; in, Indicates spatial stream information, represents the spatial stream information extraction function; The role of is to extract spatial feature information and use convolutional neural network to extract spatial features and local features.

[0033] The formula used to generate the final sequence is: ; in, represents the final sequence, Represent the weight coefficients of temporal stream information and spatial stream information respectively, and .

[0034] The final sequence combines the temporal flow information and the spatial flow information, reflecting the characteristics of sign language movements from two levels. In the sign language recognition process, the temporal flow information and the spatial flow information are equally important, so the two have the same weight. . The principle for generating sign language text sequences is: The annotation-to-text network implements the translation process as follows: ; in, Represents the translated text sequence, represents the neural network decoder; The final generated sign language text sequence is represented as: ; in, represents a sequence of sign language text, Indicates that from all possible text sequences X, select the conditional probability The largest sequence.

[0035] The text sequence is translated through the sequence-to-sequence model (Seq2Seq), which includes an encoder and a decoder. The encoder uses a gated recurrent unit to convert the input translated text sequence X into a hidden state and generate a context vector. When decoding X, one character is generated each time, and the probability distribution of the current output is calculated based on the previous character and the context vector. The decoder selects the character with the highest generation probability at each step, so that The current maximum value can be obtained and the corresponding character is added to the sign language text sequence In , the translation process of the entire text sequence is regarded as a process of searching for the maximum probability; The final sequence is input into the decoder network and translated into a text sequence. It means that from all possible text sequences X, the sequence with the largest conditional probability is selected as the output. That is, when the model generates a sequence given the input features, it predicts multiple possible output characters and selects the most likely character to output to the sequence.

[0036] In addition to the online CSLR model and the offline ISLR model, other commonly used sign language recognition models include HMM, CRF, RNN and CNN, specifically: Hidden Markov Models (HMM): Hidden Markov models are suitable for tasks with sparse annotations or no large-scale data, but they assume that the current state is only related to the previous state, which is limited in capturing the complex time-dependent information in sign language; Conditional Random Fields (CRF): Conditional random fields perform global optimization on the entire sequence through conditional probability modeling. It does not require assumptions about the independence of observed data or state variables, making it more suitable for handling complex dependencies. However, CRF requires manual feature design, which increases the complexity of sign language recognition tasks and requires higher computing resources. Recurrent Neural Networks (RNN): Recurrent neural networks can effectively model long-distance dependencies in time series and are suitable for complex dynamic changes before and after. However, traditional RNNs may have problems with gradient vanishing and gradient exploding, and the amount of computation required for training and inference is large, which affects the real-time performance of recognition. Convolutional Neural Network (CNN): Convolutional neural network is good at extracting spatial features and capturing visual information in sign language. However, it cannot capture dynamic information in time series. It is only suitable for processing single frames or short-term contexts and tends to ignore global contexts.

[0037] The variation trend of the accuracy of the online CSLR model and the offline ISLR model combined with the other four models for sign language recognition with the number of experiments is shown in Table 3: Table 3. Trend of the accuracy of the sign language model with the number of experiments ; It can be seen that the combination of the online CSLR model and the offline ISLR model has a higher recognition accuracy than other sign language recognition models, and the accuracy rate tends to increase as the number of experiments increases.

[0038] The F1 scores of the online CSLR model and the offline ISLR model combined with the other four models for sign language recognition vary with the number of experiments as shown in Table 4: Table 4. F1 score trend of sign language recognition model with the number of experiments ; The F1 score of the model is the harmonic mean of the precision and recall, which takes into account the accuracy and completeness of the model. The accuracy measures the accuracy of the model in positive prediction. A high accuracy means that the model has a low proportion of incorrect predictions when predicting positive classes. The recall rate represents the ratio of the number of samples correctly predicted by the model as positive to the total number of actual positive samples, which measures the model's ability to capture positive samples. A high recall rate indicates that the model can identify most of the positive samples, and the positive samples represent the target categories that the model hopes to identify or predict. The F1 score reflects the performance of the model when processing unbalanced data. The value range is between 0 and 1, and the higher the value, the better the model performance.

[0039] The present invention also provides an electronic device, which is used to implement the above-mentioned sign language recognition method, and specifically includes: at least one processor and a memory connected to the processor, wherein: the memory is used to store computer programs or instructions, and the processor is used to execute the computer program or instructions.

[0040] The present invention also provides a storage medium, which is applied to an electronic device, specifically comprising: the storage medium carries one or more computer programs, and the one or more computer programs can be executed by the electronic device to enable the electronic device to implement the above-mentioned continuous sign language recognition method.

[0041] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0042] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. Those skilled in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software methods depends on the specific application and design constraints of the technical solution.

[0043] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0044] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A method for continuous sign language recognition, characterized in that: The specific steps include: Step 1: Collect continuous sign language videos, segment each continuous sign language video through the Two-Stream-SLR model, divide the continuous sign language video into several sliding windows, predict the BIO label of each frame in each sliding window, determine the start, duration and end stages of the action through the predicted BIO label, extract the boundaries of isolated sign language actions, divide the continuous sign language actions into individual isolated sign language actions, perform data enhancement to generate enhanced sign language actions, and merge the isolated sign language actions and enhanced sign language actions to generate a sign language library; Step 2: Extract key point information of hands, face and upper body, and generate key point feature vectors based on the key point information; Step 3: Combine the key point features with the RGB features to generate a comprehensive feature vector; Step 4: Extract the dual-stream sequence information in the comprehensive feature vector through the online CSLR model and the offline ISLR model, fuse the dual-stream sequence information to generate the final sequence, translate the final sequence in the text network, and generate a sign language text sequence.

2. A method for continuous sign language recognition according to claim 1, characterized in that: The principle for dividing the continuous sign language action into individual isolated sign language actions in step 1 is: The continuous sign language video is divided into single-frame images. The motion information is captured by the optical flow in the Two-Stream-SLR model, and the appearance information is captured by the RGB stream. The boundary of the action is determined based on the optical flow and RGB streams, and the features of each frame are extracted. The start, inner and outer parts of the sign language action in each frame feature are identified by the Two-Stream-SLR model. The start part indicates the start of the action, the inner part indicates the continuation of the started action, and the outer part indicates the part that does not belong to any action, that is, the end of the action, which are marked as Begin, Inside and Outside respectively. All parts marked as Begin and Inside are extracted to constitute an isolated sign language action.

3. A method for continuous sign language recognition according to claim 1, characterized in that: The principle for generating the enhanced sign language action in step 1 is: The enhancement measures include: speeding up or slowing down the video frames of the continuous sign language video, changing the rhythm of the sign language movements, and randomly rotating the images in the continuous sign language video; After the enhancement measures are taken, the enhanced continuous sign language video is processed in the Two-Stream-SLR model in the same way as extracting isolated sign language movements, and the extracted sign language movements are the enhanced sign language movements.

4. The method for continuous sign language recognition according to claim 1, characterized in that: The principle for generating key point features in step 2 is: Based on the data of the sign language library, the images extracted from the sign language action video frames are input into the COCO-HRNet model to extract the coordinates of key points including the hands, face and upper body. The coordinate features are generated based on the coordinates of the key points. The formula is: ； in, represents the coordinate features of the i-th key point, represents the horizontal coordinate of the i-th key point, i represents the index of the key point, Represents the ordinate of the i-th key point; The key points include: the midpoint of the palm and the tip of the five fingers, the facial features, and the midpoint of the shoulders and chest of the upper body; The differential features are generated according to the Euclidean distance of the key point coordinates of adjacent frames, based on the formula: ； in, represents the differential feature of the i-th key point, represents the horizontal coordinate of the i-th key point in the j+1-th frame image, Represents the index of the image, represents the horizontal coordinate of the i-th key point in the j-th frame image, represents the ordinate of the i-th key point in the j+1-th frame image, Represents the ordinate of the i-th key point in the j-th frame image; Generate key point feature vector based on coordinate features and differential features, expressed as: ； in, Represents the feature vector of the i-th key point.

5. A method for continuous sign language recognition according to claim 4, characterized in that: The principle on which the step 3 generates the comprehensive feature vector is: ； in, represents the comprehensive feature vector of the i-th key point, represents the RGB feature of the i-th key point, and Includes the i-th key red channel value, green channel value and blue channel value; Calculate the average value of the comprehensive feature vectors of all key points to generate the comprehensive feature vector of a frame of image. The formula is: ； in, Represents the comprehensive feature vector of the image.

6. The method for continuous sign language recognition according to claim 1, characterized in that: The principle for generating the final sequence in step 4 is: The two-stream representation models a continuous sign language video and a keypoint sequence generated by an off-the-shelf keypoint estimator; The online CSLR model processes the comprehensive feature vector as follows: ； in, represents the internal state of the online CSLR model after processing the input features at time t, represents the function of the online CSLR model, Q express The comprehensive feature vector of the image at that moment, represents the internal state of the online CSLR model after processing the input features at time t-1; The offline ISLR model processes the comprehensive feature vector as follows: ； in, Represents the internal state of the offline ISLR model after processing the input features. A function representing an offline ISLR model, Represents the comprehensive feature vector of the image; The generated time flow information is: ； in, Represents time flow information, represents the time flow information extraction function; The generated spatial stream information is: ； in, Represents spatial stream information, represents the spatial stream information extraction function; The formula used to generate the final sequence is: ； in, represents the final sequence, Represent the weight coefficients of temporal stream information and spatial stream information respectively, and , .

7. A method for continuous sign language recognition according to claim 6, characterized in that: The principle for generating the sign language text sequence in step 4 is: The translation process implemented by the annotation-to-text network is: ； in, Represents the translated text sequence, represents the neural network decoder; The final generated sign language text sequence is represented as: ； in, represents a sequence of sign language text, Indicates that from all possible text sequences X, select the conditional probability The largest sequence.

8. An electronic device, characterized in that: The electronic device comprises at least one processor and a memory connected to the processor, wherein: the memory is used to store computer programs or instructions, and the processor is used to execute the computer programs or instructions, so that the electronic device implements the continuous sign language recognition method according to any one of claims 1 to 7.

9. A storage medium, applied to an electronic device, characterized in that: The storage medium carries one or more computer programs, and the one or more computer programs can be executed by the electronic device so that the electronic device implements the continuous sign language recognition method according to any one of claims 1 to 7.

Images