Sign language recognition method based on multi-level state feature optimization

Abstract

The present invention relates to a sign language recognition method based on multi-level state feature optimization. A sign language recognition model is constructed to recognize sign language. The sign language recognition model includes a Resnet-18 network, a two-stage Gloss encoder, and a CTC decoder connected in sequence. The model is more efficient during the processing process and does not restrict the user's input image frame size, making it more suitable for actual deployment and application. The method refers to the composition rules of sign language semantics and uses a two-stage Gloss encoder to reconstruct the image frame sequence according to the semantic rules, deeply exploring the correlation between sign language features. Through a decision fusion mechanism optimized by multiple feature states, the local features and global output of the model are jointly optimized to achieve the final recognition effect.

Description

Technical Field

[0001] The present invention relates to a recognition technology, and in particular to a sign language recognition method based on multi-level state feature optimization. Background Art

[0002] Sign language is a visual language composed of gestures, body movements, and facial expressions, following specific sign language grammar rules. It is currently the primary means of communication for people with hearing impairments. In linguistic terms, gloss is considered the basic unit of sign language, and each gloss is composed of one or several gestures.

[0003] Sign language recognition is divided into isolated sign language recognition (ISLR) and continuous sign language recognition (CSLR), which convert sign language images or videos into annotated sequences. ISLR has distinct boundaries between gestures and annotations, while CSLR only provides the temporal order of the sign sequence without any segmentation information, further increasing the difficulty of the task. Current continuous sign language datasets are limited by production costs and typically use sentence-level annotations. In this study, CSLR based on these datasets is considered a weakly supervised problem.

[0004] Traditional CSLR methods model feature sequences based on methods such as Hidden Markov Model (HMM), Conditional Random Field (CRF), and Dynamic Time Warping (DTW). These methods offer low recognition accuracy and are complex to implement, resulting in several deficiencies in practical deployments. Furthermore, CSLR tasks are affected by multiple factors, including lighting conditions, background complexity, and motion amplitude. Traditional solutions are no longer sufficient for recognition tasks in these complex environments, and the robustness of the models has reached a bottleneck. Summary of the Invention

[0005] To address the problems existing in continuous sign language recognition, a sign language recognition method based on multi-level state feature optimization is proposed to break the bottleneck of existing methods and enable the model to maintain high accuracy in more complex scenarios.

[0006] The technical solution of the present invention is: a sign language recognition method based on multi-level state feature optimization, which constructs a sign language recognition model for sign language recognition. The sign language recognition model includes a Resnet-18 network, a two-stage Gloss encoder, and a CTC decoder connected in sequence; n consecutive video frames are fed into the Resnet-18 network to extract local image features based on the Resnet-18 network. The extracted features are subjected to secondary feature extraction and fusion based on the spatial variation and temporal correlation of the sign language by the G1 feature extractor and the G2 feature extractor in the two-stage Gloss encoder, respectively, to form Gloss features. Finally, the Gloss features are recognized by the CTC decoder;

[0007] During sign language recognition model training, a decision fusion mechanism for optimizing multiple feature states is used. The CTC loss function is used to locally optimize the multiple feature states that pass through the Resnet-18 network and the two-stage Gloss encoder. The KLDiv loss function is also used to correct the feature states that pass through the Resnet-18 network and the G2 feature extractor. Through continuous iterative training, the local features and global output of the model are jointly optimized to achieve the final recognition effect.

[0008] Furthermore, the Resnet-18 network image local feature extraction specifically includes: n The video stream of RGB pictures is extracted according to the time sequence of the pictures. i The characteristics of an RGB image are F i ，i ∈{1,2,…, n}; Since the number of features extracted from different video streams is not fixed, features with the same dimension as the extracted features but with all values ​​0 are used O i Expand it to a fixed number of m Extended feature set S ={ F 1, F 2,…, F n , O 1,… O i}, i ∈{1,2,…, mn}; Extended feature set S Sent to the subsequent two-stage Gloss encoder.

[0009] Furthermore, the two-stage Gloss encoder is composed of a G1 feature extractor and a G2 feature extractor; the G1 feature extractor processes features in the time dimension, and the G2 feature extractor processes features in the spatial dimension;

[0010] The G1 feature extractor is composed of two double-layer structures including a 1D-CNN network and a maximum pooling layer MP, which processes the input features in the time dimension. Its network structure is 1D-CNN-MP-1D-CNN-MP;

[0011] The G2 feature extractor follows the G1 feature extractor and is composed of only a 1D-CNN network, which extracts input features in the spatial dimension.

[0012] Furthermore, the two-stage Gloss encoder processing steps include:

[0013] Step 1: The G1 feature extractor uses a sliding window idea to process the input extended feature set S , forming a first-level Gloss feature set G1={ g 1, g 2,…, g x}; After being processed by the G1 feature extractor, the number of features contained in the feature set G1 will be the same as that in the extended feature set S The number of features included varies; assuming Q 0= m , m is the number of features contained in the extended feature set S, then the number of features contained in the feature set G1 x The calculation method is as follows:

[0014] T i = ( Q i -1 +2 p - k ) / s + 1, i ∈{1,2}(1)

[0015] Q i = ( T i +2 pm - km ) / sm +1, i ∈{1,2} (2)

[0016] x = Q 2(3)

[0017] in, T i and Q i Respectively represent i The number of features after 1D-CNN processing and the i The number of features after MP processing, p 、 k 、 s Represent the padding size, filter size and sliding step size of 1D-CNN respectively, pm 、 km 、 sm Represent the padding size, filter size and sliding step size of MP respectively.

[0018] Step 2: The G1 feature extractor uses 1D-CNN to further extract the spatial features of the first-level Gloss feature set G1 and output the second-level Gloss feature set G2 = {g 1 ’ , g 2 ’ ,…, g x ’}, this operation does not change the size of the feature set.

[0019] Furthermore, the decision fusion mechanism for optimizing multiple feature states specifically includes:

[0020] 1) Use the CTC loss function to locally optimize the multi-feature states after the Resnet-18 network, G1 feature extractor, and G2 feature extractor, respectively, denoted as L1, L2, and L3;

[0021] 2) The KLDiv loss function is used to correct the feature state after the Resnet-18 network and the G2 feature extractor, which is recorded as L4.

[0022] L4 = KLDiv( softmax(F), softmax(G2)),

[0023] 3) The total loss function of the optimization decision is: L=w1L1+w2L2+L3+w3L4. Through continuous iterative training, the weight coefficients w1, w2, and w3 of the local and final attention layers of the model are optimized.

[0024] The beneficial effects of the present invention are as follows: the present invention is based on a sign language recognition method with a multi-level state feature optimization and a fully convolutional structure, which is more efficient in the processing process. At the same time, the structure does not limit the size of the user's input image frame, and is more suitable for actual deployment applications; the method refers to the composition rules of sign language semantics, uses a two-level Gloss encoder to reconstruct the image frame sequence according to the semantic rules, and deeply explores the correlation between sign language features; through a decision fusion mechanism of multiple feature state optimization, the local features and global output of the model are jointly optimized to obtain the final recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Schematic diagram of the framework principle of the method of the present invention;

[0026] Figure 2 This is a schematic diagram of the decision fusion mechanism involved in the present invention. DETAILED DESCRIPTION

[0027] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0028] A sign language recognition method based on multi-level state feature optimization, such as Figure 1 As shown in the figure, in response to the uncertainty of complex environments and the complexity of gesture changes in sign language recognition, a recognition model is constructed according to the composition rules of sign language semantics: n consecutive video frames are fed into the Resnet-18 network for local image feature extraction based on the Resnet-18 network; the extracted features are subjected to secondary feature extraction and fusion of the spatial changes and temporal associations of sign language through a two-stage Gloss encoder G1 feature extractor and G2 feature extractor to form Gloss features; finally, the CTC decoder is used to recognize the Gloss features.

[0029] During model training, a decision fusion mechanism for optimizing multiple feature states is used to locally optimize the multiple feature states that have passed through the Resnet-18 network and the two-stage Gloss encoder using the CTC loss function. At the same time, the KLDiv loss function is used to correct the feature states that have passed through the Resnet-18 network and the G2 feature extractor. Through continuous iterative training, the local features and global output of the model are jointly optimized to achieve the final recognition effect.

[0030] The local image feature extraction part of the Resnet-18 network includes:

[0031] Step 1: For the input containing n The video stream contains RGB images (the number of RGB images in each video stream is not fixed, so the size of n is not fixed). Feature extraction is performed according to the time sequence of the images. The feature of the i-th RGB image is recorded as F i ，i ∈{1,2,…, n};

[0032] Step 2: Since the number of features extracted from different video streams is not fixed, use features with the same dimension as the extracted features but with all values ​​0 O i Expand it to a fixed number of m Extended feature set S ={ F 1, F 2,…, F n , O 1,… O i}, i ∈{1,2,…, mn};

[0033] Step 3: Expand the feature set S Sent to the subsequent two-stage Gloss encoder.

[0034] The two-stage Gloss encoder part:

[0035] 1) Design a two-stage Gloss encoder structure, including:

[0036] The two-stage Gloss encoder structure consists of a G1 feature extractor and a G2 feature extractor. The G1 feature extractor processes features in the temporal dimension, while the G2 feature extractor processes features in the spatial dimension.

[0037] 2) The G1 feature extractor and the G2 feature extractor include:

[0038] The G1 feature extractor is composed of two double-layer structures, one containing a 1D-CNN network and one maximum pooling layer (denoted as MP). It uses the sliding window idea to process input features in the time dimension. Its network structure is 1D-CNN-MP-1D-CNN-MP.

[0039] The G2 feature extractor follows the G1 feature extractor and consists of only a 1D-CNN network that extracts the input features in the spatial dimension.

[0040] 3) Two-stage Gloss encoder processing steps, including:

[0041] Step 1: The G1 feature extractor uses a sliding window idea to process the input extended feature set S , forming a first-level Gloss feature set G1={ g 1, g 2,…, g x After being processed by the G1 feature extractor, the number of features contained in the feature set G1 will be the same as that in the extended feature set S The number of features included varies. Assume Q 0= m ( m is the number of features included in the extended feature set S), then the number of features included in the feature set G1 x The calculation method is as follows:

[0042] T i = ( Q i -1 +2 p - k ) / s +1, i ∈{1,2}(1)

[0043] Q i = ( T i+ 2 pm - km ) / sm + 1, i ∈{1,2} (2)

[0044] x = Q 2(3)

[0045] in, T i and Q i Respectively represent i The number of features after 1D-CNN processing and the i The number of features after MP processing, p 、 k 、 s Represent the padding size, filter size and sliding step size of 1D-CNN respectively, pm 、 km 、 sm They represent the padding size, filter size and sliding step size of MP respectively.

[0046] Step 2: The G1 feature extractor uses 1D-CNN to further extract the spatial features of the first-level Gloss feature set G1 and output the second-level Gloss feature set G2 = { g 1 ’ , g 2 ’ ,…, g x ’}, this operation does not change the size of the feature set;

[0047] Step 3: The secondary Gloss feature set G2 is sent to the subsequent CTC decoder.

[0048] The decision fusion method of multi-level state feature optimization is as follows: Figure 2 Shown, including:

[0049] Step 1: Use the CTC loss function to locally optimize the multi-feature states after the Resnet-18 network, G1 feature extractor, and G2 feature extractor, respectively, and record them as L1, L2, and L3.

[0050] Step 2: In order to solve the model contribution allocation problem caused by the multi-feature state optimization, the KLDiv loss function is used to correct the feature state after the Resnet-18 network and the G2 feature extractor, which is recorded as L4.

[0051] L4 = KLDiv( softmax(F), softmax(G2)) (4)

[0052] Step 3: The total loss function of the optimization decision is: L=w1L1+w2L2+L3+w3L4. Through continuous iterative training, the weight coefficients w1, w2, and w3 of the local and final attention layers of the model are optimized.

[0053] The features processed by the Resnet-18 network, G1 feature extractor, and G2 feature extractor are optimized independently. Independent optimization operations will introduce model contribution distribution problems and imbalance phenomena. On this basis, the KLDiv loss function is used to distill and correct the features processed by the Resnet-18 network and G2 feature extractor to stabilize the model performance.

[0054] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A sign language recognition method based on multi-level state feature optimization, characterized in that: A sign language recognition model was constructed to identify sign language. The model consisted of a Resnet-18 network, a two-stage Gloss encoder, and a CTC decoder connected in sequence. N consecutive video frames were fed into the Resnet-18 network for local image feature extraction. The extracted features were then used by the G1 and G2 feature extractors in the two-stage Gloss encoder to perform secondary feature extraction and fusion based on the spatial variation and temporal correlation of the sign language, forming Gloss features. Finally, the CTC decoder was used to recognize the Gloss features. During sign language recognition model training, a decision fusion mechanism for optimizing multiple feature states was used. The CTC loss function was used to locally optimize the multiple feature states that passed through the Resnet-18 network and the two-stage Gloss encoder. The KLDiv loss function was also used to correct the feature states that passed through the Resnet-18 network and the G2 feature extractor. Through continuous iterative training, the local features and global output of the model were jointly optimized to achieve the final recognition effect. The two-stage Gloss encoder consists of a G1 feature extractor and a G2 feature extractor; the G1 feature extractor processes features in the time dimension, and the G2 feature extractor processes features in the spatial dimension; The G1 feature extractor is composed of two double-layer structures including a 1D-CNN network and a maximum pooling layer MP, which processes the input features in the time dimension. Its network structure is 1D-CNN-MP-1D-CNN-MP; The G2 feature extractor follows the G1 feature extractor and consists of only a 1D-CNN network, which extracts the input features in the spatial dimension. The two-stage Gloss encoder processing steps include: Step 1: The G1 feature extractor uses a sliding window idea to process the input extended feature set S , forming a first-level Gloss feature set G1={ g 1, g 2,…, g x }; After being processed by the G1 feature extractor, the number of features contained in the feature set G1 will be the same as that of the extended feature set S The number of features included varies; assuming Q 0 = m , m is the number of features contained in the extended feature set S, then the number of features contained in the feature set G1 x The calculation method is as follows: T i = ( Q i -1 +2 p - k ) / s + 1 ， i ∈{1,2}(1) Q i = ( T i +2 pm - km ) / sm +1， i ∈{1,2} (2) x = Q 2(3) in, T i and Q i Respectively represent i The number of features after 1D-CNN processing and the i The number of features after MP processing, p 、 k 、 s Represent the padding size, filter size and sliding step size of 1D-CNN respectively, pm 、 km 、 sm Represent the padding size, filter size, and sliding step size of MP respectively; Step 2: The G1 feature extractor uses 1D-CNN to further extract the spatial features of the first-level Gloss feature set G1 and output the second-level Gloss feature set G2 = { g 1 ’ , g 2 ’ ,…, g x ’ }, this operation does not change the size of the feature set.

2. The sign language recognition method based on multi-level state feature optimization according to claim 1, characterized in that: The Resnet-18 network image local feature extraction specifically includes: n The video stream of RGB pictures is extracted according to the time sequence of the pictures. i The characteristics of an RGB image are F i ，i ∈{1,2,…, n }; Since the number of features extracted from different video streams is not fixed, features with the same dimension as the extracted features but with all values ​​0 are used O i Expand it to a fixed number of m Extended feature set S ={ F 1, F 2,…, F n , O 1,… O i }, i ∈{1,2,…, mn }; Extended feature set S Sent to the subsequent two-stage Gloss encoder.

3. The sign language recognition method based on multi-level state feature optimization according to claim 1 or 2, characterized in that: The decision fusion mechanism for optimizing multiple feature states specifically includes: 1) Use the CTC loss function to locally optimize the multi-feature states after the Resnet-18 network, G1 feature extractor, and G2 feature extractor, respectively, denoted as L1, L2, and L3; 2) The KLDiv loss function is used to correct the feature state after the Resnet-18 network and the G2 feature extractor, which is recorded as L4. L4 = KLDiv( softmax(F), softmax(G2)), 3) The total loss function of the optimization decision is: L=w1L1+w2L2+L3+w3L4. Through continuous iterative training, the weight coefficients w1, w2, and w3 of the local and final attention layers of the model are optimized.

Images