Continuous sign language recognition method based on multi-cue mutual distillation and self-distillation

By employing multi-cue mutual distillation and self-distillation methods, and utilizing facial and hand information from sign language videos, this approach addresses the issues of insufficient utilization of local information and inadequate training of the feature extraction module in existing technologies, thereby improving the accuracy of continuous sign language recognition and the generalization ability of neural networks.

CN114821802BInactive Publication Date: 2024-11-01HEBEI UNIV OF TECH +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210528751.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-16
Publication Date
2024-11-01
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing continuous sign language recognition methods, when utilizing deep learning technology, suffer from problems such as insufficient utilization of local information, inadequate training of feature extraction modules, and low accuracy in recognizing similar sign language words.

Method used

We employ a method based on multi-cue mutual distillation and self-distillation, utilizing facial and hand information from sign language videos. Spatial features are extracted through a residual network, and a total loss function is constructed by combining a one-dimensional convolutional module and a bidirectional gated recurrent unit for training, thereby enhancing the generalization ability and recognition performance of the neural network.

Benefits of technology

It improves the accuracy of sign language recognition, enhances the generalization ability of neural networks, can better handle local information and temporal correlation in sign language videos, and improves the convergence of the feature extraction part of hybrid neural networks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114821802B_ABST
    Figure CN114821802B_ABST
Patent Text Reader

Abstract

The present invention is a continuous sign language recognition method based on multi-cue mutual distillation and self-distillation, including obtaining a sign language video, extracting face, left hand and right hand videos from the sign language video and preprocessing, extracting spatial features from the video, then extracting sign language word-level features with short-term temporal connections from the spatial features, extracting context-level features with long-term temporal connections from the sign language word-level features, and then using a fully connected layer and an activation function to obtain a normalized classification probability matrix; constructing a total loss function based on the obtained classification probability matrix, and training a neural network; using the trained neural network to predict sign language and output a sign language label. The method utilizes the face information and hand information in the original sign language image, fully utilizes the knowledge of face information and hand information by mutual distillation, solves the problem of insufficient utilization of local information, and improves the generalization ability of the neural network; and increases the coordination between different modules in the neural network by self-distillation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The technical solution of the present invention relates to the technical field of deep learning image processing and image recognition, and specifically is a continuous sign language recognition method based on multi-cue mutual distillation and self-distillation. Background Art

[0002] The purpose of continuous sign language recognition is to translate a complete sign language sentence into a complete text sentence. A complete sign language sentence consists of several words. The task of continuous sign language recognition is to segment the complete sign language sentence into several sign language words, and then synthesize them into a complete text sentence through sentences. Continuous sign language recognition involves multiple technical fields such as computer vision, human-computer interaction, pattern recognition and natural language processing.

[0003] Sign language is not just a change of gestures. It is a body language composed of gestures, upper body movements and facial expressions. It is an important way of communication between the hearing-impaired and normal people. Since most normal people do not understand sign language, continuous sign language recognition technology enables communication between the hearing-impaired and normal people. Sign language recognition is generally divided into two categories: isolated sign language recognition and continuous sign language recognition. Continuous sign language recognition is usually based on isolated sign language recognition, which recognizes isolated sign language words and then performs word splicing. Traditional continuous sign language recognition methods use manual features for modeling. In recent years, deep learning technology has been used in continuous sign language recognition, which has significantly improved the defects of low recognition accuracy in continuous sign language recognition due to factors such as gesture occlusion and large gesture changes.

[0004] However, in existing sign language datasets, the label corresponding to a continuous sign language video is the label of the entire sentence. The alignment of isolated sign language words contained in the video and their corresponding word labels is unknown, so continuous sign language recognition is a weakly supervised problem. The Connectionist Temporal Classification (CTC) network can solve the problem of weak supervision of video labels and can automatically complete the alignment task between the input video sequence and the text sequence. Zhou et al. used CTC as the loss for sequence learning in "Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition" to establish a mapping relationship between sign language video frames and sign language vocabulary, but the sequence learning part is more prone to overfitting than the feature extraction part. The patent application with publication number CN112487951A discloses a method for sign language recognition and translation. The method extracts key points from the output feature map of the third basic network layer of the backbone network by deconvolution upsampling, obtains vectors formed by joint nodes, and then connects them with the vectors of the backbone network. A two-layer bidirectional long short-term memory network is used to learn temporal features, and a CTC classifier is used as the optimization loss function, and a smooth-L1 function is used as the key point matching loss function. The disadvantage of this method is that it ignores the fact that sign language is a dynamic process, does not extract short-term spatiotemporal features, and loses motion information. The patent application with publication number CN112329690A discloses a continuous sign language recognition method based on a spatiotemporal residual network and a temporal convolutional network. The method uses a spatiotemporal residual network to extract features, enhances short-term temporal connections through a temporal convolutional network, and then establishes long-term connections through a bidirectional gated recurrent unit, and uses a CTC classifier as the final loss function of the network. The disadvantage of this method is that it only focuses on the image itself and does not make full use of the local information in the image. Since the gesture expressions of some words are very similar, such as "repay" and "revenge", "computer" and "piano", "surprise" and "strange" (see the document "Hu H, Zhou W, Pu J, et al. Global-local enhancement network for NMF-aware sign language recognition [J]. ACM transactions on multimedia computing, communications, and applications (TOMM), 2021, 17 (3): 1-19. "), it is often necessary to combine facial expressions to determine the specific meaning. Therefore, the recognition accuracy of this method for similar sign language words is low.

[0005] In previous studies, deep learning-based continuous sign language recognition usually uses a hybrid model that combines a feature extraction module and a sequence learning module. Min et al. proposed in "Visual Alignment Constraint for Continuous Sign Language Recognition" that the hybrid model will lead to insufficient training of the feature extraction module, resulting in poor convergence ability of the feature extraction module. Summary of the invention

[0006] In view of the shortcomings of the prior art, the technical problem to be solved by the present invention is: to provide a continuous sign language recognition method based on multi-cue mutual distillation and self-distillation. The method utilizes the facial information and hand information in the original sign language image, and through mutual distillation, fully utilizes the knowledge of facial information and hand information, solves the problem of insufficient utilization of local information, and improves the generalization ability of the neural network, thereby achieving the purpose of improving recognition accuracy; through self-distillation, the coordination between different modules in the neural network is increased, and to a certain extent, the problem that the hybrid neural network formed by the combination of multiple modules will lead to insufficient training of some feature extraction modules is solved, the convergence of the feature extraction module is enhanced, and the recognition ability of the entire neural network for sign language is enhanced.

[0007] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0008] A continuous sign language recognition method based on multi-cue mutual distillation and self-distillation, characterized in that the method includes the following contents:

[0009] Obtain a sign language video, and extract a face, a left hand, and a right hand video from the sign language video;

[0010] Preprocessing including random cropping, random flipping and normalization is performed on the sign language video, face video, left hand video and right hand video;

[0011] The sign language video is input into the residual network Resnet50 for feature extraction to obtain global spatial features; the face video, left hand video and right hand video are input into the residual network Resnet18 to obtain face spatial features, left hand spatial features and right hand spatial features;

[0012] The left hand spatial features and the right hand spatial features are concatenated according to the second dimension to obtain the hand spatial features; the global sign language word-level features, facial sign language word-level features and hand sign language word-level features with short-term temporal connections are extracted from the global spatial features, facial spatial features and hand spatial features using a one-dimensional convolution module; the one-dimensional convolution module is composed of a one-dimensional convolution and a temporal maximum pooling, and the structure of the one-dimensional convolution module is Conv1D , MaxPooling 1D , Conv 1D , MaxPooling 1D , Conv 1D It is a one-dimensional convolution with a convolution kernel size of 5, MaxPooling 1D is the temporal maximum pooling of size 2;

[0013] Extracting global context-level features, facial context-level features and hand context-level features with long-term temporal connections from global sign language word-level features, facial sign language word-level features and hand sign language word-level features, and then using a fully connected layer and a softmax activation function to obtain normalized global classification probability matrices, facial classification probability matrices, hand classification probability matrices and global sign language word-level classification probability matrices;

[0014] Based on the obtained classification probability matrix, a total loss function is constructed to train the neural network; the total loss function includes three parts: link temporal classification loss function, multi-cue mutual distillation loss and self-distillation loss; the trained neural network is used to predict sign language and output sign language labels.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] The outstanding essential features of the present invention are as follows:

[0017] (1) In this application, original sign language images, facial images, and hand images are used as multi-cue inputs of the neural network. Based on the same semantic information of the corresponding images, the multiple cues learn from each other through mutual knowledge distillation, which enhances the generalization ability of the neural network and improves the recognition ability of each clue for sign language. In actual application, sign language words are inferred and generated only through original sign language images without the need to input facial images and hand images. This application increases the convergence of the network part by using sign language word-level features and context-level features through self-local self-distillation, helping the overall neural network to improve the recognition effect of sign language words.

[0018] (2) This application uses the Openpose algorithm to locate the hand joints and face positions from the original sign language images of each frame of the sign language video, and then crops the face image, left hand image and right hand image according to the coordinate points, and then obtains the face video, left hand video and right hand video; the four clues of the sign language video, face video, left hand video and right hand video are respectively extracted spatial features through the neural network, and the two hand clues are spliced ​​together. The subsequent three clues are respectively obtained through the one-dimensional convolution module, the bidirectional gating unit and the fully connected layer to obtain their respective classification probability matrices. In the process of constructing the loss function to train the neural network, the multi-clue mutual distillation loss and self-distillation method proposed in this application enhance the generalization ability of each clue by using the KL divergence loss. Finally, the CTC loss is used to impose label alignment constraints on each clue for training. In the final sign language recognition process, the classification probability matrix is ​​converted into sign language words through the beam search decoding strategy to realize the recognition of continuous sign language.

[0019] The significant improvements of the present invention are:

[0020] (1) Compared with CN112329690A, the method of the present invention can fully utilize the facial and hand information in the original sign language image, improve the problem of insufficient training of the feature extraction part in the hybrid neural network through self-distillation, and improve the recognition effect.

[0021] (2) The method of the present invention uses the residual network Resnet50 to extract the spatial features of the sign language video, and uses the residual network Resnet18 to extract the spatial features of the face video, the left hand video and the right hand video. Different depths of networks are used for feature extraction for images of different sizes. The residual structure in Resnet can effectively avoid the problems of gradient explosion and gradient disappearance, making the network converge faster, easier to optimize, and having fewer parameters than the previous structure.

[0022] (3) Compared with CN112487951A, the method of the present invention has the advantage of extracting short-term spatiotemporal features through a one-dimensional convolution module to capture the motion information of sign language, thereby obtaining word-level features of sign language for subsequent operations, which can fully strengthen the temporal correlation between gestures and improve the recognition effect.

[0023] (4) The method of the present invention uses a combination of two-dimensional convolution and one-dimensional convolution to replace three-dimensional convolution, which can effectively reduce the number of parameters of the neural network, speed up the calculation speed of the neural network, and control the number of frames included in the sign language word-level features according to the size of the one-dimensional convolution kernel and the temporal pooling, while the three-dimensional convolution cannot effectively control the number of frames included in the sign language word-level features.

[0024] (5) The method of the present invention uses KL divergence loss to enable the three clues to learn useful knowledge from each other through mutual distillation, making full use of the characteristics of multiple clues and improving the generalization ability of each clue.

[0025] (6) The method of the present invention proposes a self-distillation method, which uses KL divergence loss to enable sign language word-level features and context-level features to learn useful knowledge from each other through mutual distillation. This can effectively change the problem of insufficient training of the feature extraction part in the hybrid model, thereby improving the recognition effect of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is the overall flow chart of the present invention;

[0027] Figure 2 It is a schematic diagram of mutual distillation and self-distillation. DETAILED DESCRIPTION

[0028] like Figure 1 As shown, the process of the continuous sign language recognition method based on multi-cue mutual distillation and self-distillation of the present invention is as follows:

[0029] Get the sign language video V and extract the face video V from the sign language video face , Left hand video V lhand And right hand video V rhand →Sign language video V, face video V face , Left hand video V lhand And right hand video V rhand Preprocess → Extract sign language video V and face video V face , Left hand video V lhand And right hand video V rhand The spatial features of → use the one-dimensional convolution module to transform the spatial features F full 、F face 、F rhand and F lhand Convert to sign language word-level features with short-term temporal connections → Extract sign language word-level features F′ full , F′ face and F′ hand Context-level features of long-term temporal connections E full 、E face and E hand →Get the normalized classification probability matrix Y full , Y face , Y hand and Y gloss →Construct the total loss function and train the neural network →Predict the sign language based on the trained neural network and output the sign language label G →Complete continuous sign language recognition.

[0030] like Figure 2 As shown, using the classification probability matrix Y full , Y face , Y hand Perform mutual knowledge distillation and use the classification probability matrix Y full and Y gloss The process of self-distillation is as follows:

[0031] Multi-cue mutual distillation part: using KL divergence loss D KL Calculate the classification probability matrix Y full and Y face The difference between face and Y full Difference → Use KL divergence loss D KL Calculate the classification probability matrix Y full and Y hand The difference between hand and Y full Difference → Use KL divergence loss D KL Calculate the classification probability matrix Y hand and Y face The difference between face and Y hand The difference → According to the above three parts KL divergence loss D KL Calculate multi-cue mutual distillation loss ;

[0032] Self-distillation part: using KL divergence loss D KL Calculate the classification probability matrix Y full and Y gloss The difference between gloss and Y full The difference → According to these two parts of the divergence loss D KL Calculating self-distillation losses .

[0033] In this application, the classification probability matrix Y full , Y face , Y hand and Y gloss The dimensions are all (T′, 1296), which means that the feature blocks of T′ are classified into 1296 categories, using the KL divergence loss D KL As a process of knowledge distillation, in the multi-cue mutual distillation, Y full , Y face , Y hand Use KL divergence loss D for each pair KLConstraints are imposed, so that the classification probability matrix of its own part can be adjusted through the classification probability matrix of other clues, and the neural network can be adjusted through back propagation, thereby enhancing the generalization ability of the neural network. full and Y gloss Use KL divergence loss D KL To constrain the two, they adjust each other according to each other’s classification probability matrices, which enhances the convergence of the local neural network while enhancing the generalization ability of the model.

[0034] In this application, the sign language word-level features obtained by controlling the one-dimensional convolution module contain information of sixteen frames of images in the video, which is basically the same as the number of frames required for an isolated sign language word. This can greatly enhance the processing of local time information and enhance the recognition ability of the neural network.

[0035] Example 1

[0036] The continuous sign language recognition method based on multi-cue mutual distillation and self-distillation in this embodiment has the following specific steps:

[0037] The first step is to obtain a sign language video and extract the face, left hand, and right hand videos from the sign language video;

[0038] Assume that the sign language video V = (v1, ..., v t ,…v T ), the sign language video is a video formed by the original sign language image sequence, T is the number of frames of the sign language video V, representing the 1st frame, ..., tth frame, ..., Tth frame of the original sign language image sequence, the size of the original sign language image in the sign language video is 210×260 pixels, and each frame of the original sign language image is converted into 256×256 pixels using formula (1);

[0039] v t =Resize(256, 256) (1)

[0040] In formula (1), Resize(·) represents the function of adjusting the image size;

[0041] The OpenPose algorithm is used to obtain the human joint point set P in the sign language video V = (p1, ..., p t ,…p T ), p t represents the set of human joint points in the original sign language image of the tth frame, p tcontains 18 two-dimensional arrays, i.e., the coordinates of 18 joint points; each frame of the original sign language image is cropped according to the coordinate points in the image to obtain the face image, the left hand image and the right hand image, respectively. The face image size is 64×64 pixels, and the left and right hand images are 96×96 pixels; all frames of face images, left hand images and right hand images are arranged in time series to obtain face videos, respectively. Left Hand Video And right hand video and They represent the face image, left hand image and right hand image cropped from the original sign language image of frame t respectively;

[0042] Get face video V face The expression is as follows, and the same is true for obtaining the left and right hand videos;

[0043] V face =Crop(V,P) (2)

[0044] In formula (2), Crop(·) represents the function of cropping with the coordinate point as the center;

[0045] Step 2: Sign language video V, face video V face , Left hand video V lhand And right hand video V rhand Perform pre-processing;

[0046] The sign language video V = (v1, ..., v i ,…v T ) are adjusted to 128×128 pixels in each frame of the original sign language image, and randomly cropped to 112×112 pixels, and then randomly flipped horizontally, and then the pixel values ​​of the image are normalized to (-1, 1). The dimension of the sign language video V input to the neural network is (T, 3, 112, 112); the facial video V face , Left hand video V lhand And right hand video V rhand Each frame image in the image is randomly flipped horizontally, and then the pixel values ​​of the image are normalized to (-1, 1). The face video V face , Left hand video V lhand And right hand video V rhand The dimensions of the input network are (T, 3, 64, 64), (T, 3, 96, 96) and (T, 3, 96, 96) respectively. Formula (3) is the preprocessing expression of the sign language video V, and formulas (4) to (6) are the preprocessing expressions of the face video V respectively. face , Left hand video V lhand And right hand video V rhandPreprocessing expression of ;

[0047]

[0048]

[0049]

[0050]

[0051] In formulas (3) to (6), Resize(·) represents a function for adjusting the image size, RandomCrop(·) represents a function for randomly cropping an image to a fixed size, RandomFlip(·) represents a function for randomly flipping an image horizontally, and Normalize(·) represents a normalization function for normalizing the pixel values ​​of an image from (0, 255) to (-1, 1).

[0052] The third step is to use the neural network to extract the sign language video V and the face video V face , Left hand video V lhand And right hand video V rhand The spatial features of the global spatial features F full , face spatial feature F face , left hand spatial feature F lhand and the right hand spatial feature F rhand ;

[0053] The preprocessed sign language video V is input into the residual network Resnet50 (Residual Network 50) for spatial feature extraction to obtain the global spatial feature F full The dimension is (T, 1024), and the expression for extracting global spatial features is:

[0054]

[0055] In formula (7), Conv resnet50 (·) represents the function of the residual network Resnet50, Represents the global spatial feature F full The t-th vector in is the spatial feature extracted from the preprocessed sign language image of the t-th frame;

[0056] The pre-processed face video V face , Left hand video V lhand And right hand video V rhand The facial spatial features are obtained by inputting them into the residual network Resnet18 (Residual Network 18) for spatial feature extraction. Left hand space characteristics and right hand spatial features F face 、F rhand and F lhand The dimensions are all (T, 1024), and the expressions of extracted spatial features are shown in the following formulas (8) to (10):

[0057]

[0058]

[0059]

[0060] In formulas (8) to (10), Conv resnet18 (·) represents the function of the residual network Resnet18, f t full 、f t lhand and f t rhand Represent the facial spatial features F face , left hand spatial feature F lhand and the right hand spatial feature F rhand The tth vector in ;

[0061] The fourth step is to use the one-dimensional convolution module to extract the global spatial features F full , face spatial feature F face , left hand spatial feature F lhand and the right hand spatial feature F rhand Extract global sign language word-level features, facial sign language word-level features and hand sign language word-level features with short-term temporal connections;

[0062] The left hand spatial feature F lhand and the right hand spatial feature F rhand According to the second dimension, we can concatenate and obtain the hand space feature F hand , F hand The dimension is (T, 2048), and the concatenation expression is shown in the following formula (11):

[0063] F hand =Cat(F lhand , F rhand ) (11)

[0064] In formula (11), Cat(·) represents the function of feature concatenation according to the second dimension.

[0065] Use one-dimensional convolution modules to transform the global spatial features F full , face spatial feature F face and hand space feature F hand Perform feature extraction of short-term time connections to obtain the global sign language word-level feature F′ full , word-level feature of facial sign language F′ face and the word-level feature F′ of hand sign language hand ; The one-dimensional convolution module consists of one-dimensional convolution and temporal maximum pooling. The specific structure of the one-dimensional convolution module is Conv 1D , MaxPooling 1D , Conv 1D , MaxPooling 1D , Conv 1D It is a one-dimensional convolution with a convolution kernel size of 5, MaxPooling 1D The maximum pooling of the size is 2, and the one-dimensional convolution module is used to extract the word-level features of the sign language as shown in the following formulas (12) to (14);

[0066]

[0067]

[0068]

[0069] In formulas (12) to (14), Conv1D(·) represents a one-dimensional convolution module. and Represents the global sign language word-level features F′ full , word-level feature of facial sign language F′ face and the word-level feature F′ of hand sign language hand For the t-th vector in , the word-level feature dimension of sign language is (T′, 1024), T′=(N-4) / 2=(((T-4) / 2)-4) / 2, where N represents the time dimension;

[0070] The one-dimensional convolution module can be divided into two stages. First, the global spatial feature F full , face spatial feature F face and hand space feature F hand The dimensions are converted to (1, 1024, T), (1, 1024, T) and (1, 2048, T) respectively, and the one-dimensional convolution module and the temporal maximum pooling are used to extract features to obtain the intermediate features. and The specific operation is expressed as shown in the following formulas (15) to (17):

[0071]

[0072]

[0073]

[0074] In formulas (15) to (17), reshape(·) represents the function of transforming feature dimensions, Conv 1D (·) represents a one-dimensional convolution function with a convolution kernel size of 5, MaxPooling 1D (·) represents the temporal maximum pooling function, and Represents the intermediate features and The tth vector in, the intermediate feature and The dimensions are all (1, N, 1024), N = (T-4) / 2;

[0075] Then the intermediate features and They are respectively used as input through another one-dimensional convolution in the one-dimensional convolution module and the temporal maximum pooling to be converted into a global sign language word-level feature F′ with short-term temporal connection. full , word-level feature of facial sign language F′ face and the word-level feature F′ of hand sign language hand , the specific operation is expressed as shown in the following formula (18)(19)(20);

[0076]

[0077]

[0078]

[0079] The fifth step is to extract the global sign language word-level features F′ full , word-level feature of facial sign language F′ face and the word-level feature F′ of hand sign language hand Extracting global context-level features E with long-term temporal connections full , facial contextual features E face and hand context level feature E hand ;

[0080] The global sign language word-level feature F′ obtained in the fourth step above full , word-level feature of facial sign language F′ face and the word-level feature F′ of hand sign language handThe dimensions of are converted to (T′, 1, 1024) and are input into two layers of bidirectional gated recurrent units in series. The bidirectional gated recurrent unit (BGRU) is an existing well-known algorithm that extracts global context-level features with long-term temporal connections. Facial context-level features and hand context-level features The expression of extracting context-level features is shown in the following formulas (21) to (23);

[0081]

[0082]

[0083]

[0084] In formulas (21) to (23), RNNGroup(·) represents a function for extracting context-level features with long-term temporal connections. RNNGroup(·) consists of two layers of bidirectional gated recurrent units connected in series. and Represent the global context level features E full , facial contextual features E face and hand context level feature E hand The tth vector in ;

[0085] Step 6: Get the normalized global classification probability matrix Y full , face classification probability matrix Y face , hand classification probability matrix Y hand And the global sign language word-level classification probability matrix Y gloss ;

[0086] The global context-level feature E obtained in the fifth step full , facial contextual features E face , hand context level feature E hand and the global sign language word-level feature F′ obtained in the fourth step full After passing through a fully connected layer respectively, the fully connected layer output feature FC is obtained full , FC face , FC hand and FC gloss , the size of the output feature of the fully connected layer is the vocabulary size 1296, which is expressed by the fully connected layer as shown in the following formulas (24) to (27);

[0087]

[0088]

[0089]

[0090]

[0091] In formulas (24) to (27), fc(·) represents the fully connected layer. and Represents the fully connected layer output feature FC full , FC face , FC hand and FC gloss The tth vector in FC full , FC face , FC hand and FC gloss The dimensions of are all (T′, 1296);

[0092] The fully connected layer outputs the feature FC full , FC face , FC hand and FC gloss The global classification probability matrix is ​​obtained by the softmax activation function full , face classification probability matrix Y face , hand classification probability matrix Y hand And the global sign language word-level classification probability matrix Y gloss ; The softmax activation function converts each vector in the feature into a probability distribution. The sum of the probability values ​​corresponding to each vector is 1. The classification probability matrix is ​​expressed as shown in the following formulas (28) to (31);

[0093]

[0094]

[0095]

[0096]

[0097] In formulas (28) to (31), softmax(·) represents the activation function, and Represents the global classification probability matrix Y full , face classification probability matrix Y face , hand classification probability matrix Y hand And the global sign language word-level classification probability matrix Y gloss The tth vector in ; where the global classification probability matrix Y full The mth probability value of the tth vector in The calculation process of is expressed as shown in the following formula (32):

[0098]

[0099] In formula (32), exp(·) represents an exponential function with the natural constant e as the base. Represents the fully connected layer output feature FC full The mth probability value of the tth vector in , where M represents the number of probability values;

[0100] Step 7: Construct the total loss function and train the neural network, including the residual network Resnet50, the residual network Resnet18, the one-dimensional convolution module and the two-layer bidirectional gated recurrent unit in series;

[0101] The total loss function constructed in this step consists of three parts: link temporal classification loss function Multi-cue mutual distillation loss and self-distillation losses The specific operations are as follows:

[0102] Step 7.1: Constructing the Link Time Classification Loss Function

[0103] The global classification probability matrix Y obtained in the sixth step full , face classification probability matrix Y face , hand classification probability matrix Y hand And the global sign language word-level classification probability matrix Y gloss The real labels l corresponding to the sign language are input into their respective connectionist temporal classification networks (CTC). The connectionist temporal classification network introduces an empty label <blank>As an auxiliary marker, the predicted probability is obtained by adding the conditional probabilities of all possible paths; the input is the global classification probability matrix Y full The predicted probability P(l|Y full ), input is the face classification probability matrix Y face The predicted probability P(l|Y face ), the input is the hand classification probability matrix Y hand The predicted probability P(l|Y hand ) and the input is the global sign language word-level classification probability matrix Y gloss The predicted probability P(l|Y gloss ) is expressed as shown in the following formulas (33) to (36);

[0104]

[0105]

[0106]

[0107]

[0108] In formulas (33) to (36), Π=(π1,...,π i ,…π s ) represents the set of all possible paths, p(Π|Y full )、p(Π|Y face )、p(Π|Y hand ) and p(Π|Y gloss ) represent the input as Y full , Y head , Y hand and Y gloss The probability of outputting path Π when , B is a many-to-one mapping used to remove empty labels and duplicate labels in path Π;

[0109] In order to maximize the predicted probability, the link temporal classification network uses the negative of the logarithm of the predicted probability as the loss function, so the link temporal global classification loss is Linking temporal face classification loss Linking temporal hand classification loss and link temporal global sign language word-level classification loss It is expressed as shown in the following formulas (37) to (40);

[0110]

[0111]

[0112]

[0113]

[0114] In formulas (37) to (40), log(·) represents a logarithmic function with a natural number e as the base;

[0115] In summary, the link temporal classification network loss function There are four parts of loss in total, which are expressed as shown in the following formula (41):

[0116]

[0117] Step 7.2: Construct multi-cue mutual distillation loss

[0118] Using the global classification probability matrix Y obtained in the sixth step full , face classification probability matrix Y face and the hand classification probability matrix Y hand Construct KL divergence (Kullback-Leibler divergence) loss D KL , expressed as shown in the following formulas (42) to (47);

[0119]

[0120]

[0121]

[0122]

[0123]

[0124]

[0125] In formulas (42) to (47), and Represents the global classification probability matrix Y full , face classification probability matrix Y face and the hand classification probability matrix Y hand The t-th vector in corresponds to the m-th probability value in the probability distribution, and M represents the number of probability values, that is, the output feature size of the fully connected layer;

[0126] In summary, multi-cue mutual distillation loss It is expressed as shown in the following formula (48);

[0127]

[0128] Step 7.3: Constructing the self-distillation loss

[0129] Using the global classification probability matrix Y obtained in the sixth step full And the global sign language word-level classification probability matrix Y gloss Constructing the self-distillation loss Self-distillation losses It is expressed as shown in the following formula (49):

[0130]

[0131] In formula (49), D KL (p full ||p gloss ) and D KL (p gloss ||p full ) are KL divergence losses, Represents the global sign language word-level classification probability matrix Y gloss The t-th vector in corresponds to the m-th probability value in the probability distribution;

[0132] Therefore, the total loss function of the entire neural network is is the sum of the three partial losses, expressed as shown in the following formula (50);

[0133]

[0134] In formula (50), α and β represent hyperparameters used to adjust the loss ratio;

[0135] Step 8: Use the trained neural network to predict sign language and output the sign language label G;

[0136] Using the neural network trained in step 7, beam search decoding (Beam) is used to decode the global classification probability matrix Y obtained in step 6 full Decode and complete the mapping from the classification probability matrix to the sign language sequence to obtain the sign language label G, which is expressed as shown in the following formula (51):

[0137] G=Beam(Y full ) (51)

[0138] In formula (51), Beam(·) represents the beam search decoding function;

[0139] At this point, continuous sign language recognition based on multi-cue mutual distillation and self-distillation is completed.

[0140] A total of three clues are used in the present invention, namely global space features, face space features and hand space features.

[0141] Any matters not described in the present invention are applicable to the prior art.< / blank>

Claims

1. A continuous sign language recognition method based on multi-cue mutual distillation and self-distillation, It is characterized in that The method includes the following: Obtain a sign language video, and extract a face, a left hand, and a right hand video from the sign language video; Preprocessing including random cropping, random flipping and normalization is performed on the sign language video, face video, left hand video and right hand video; The sign language video is input into the residual network Resnet50 for feature extraction to obtain global spatial features; the face video, left hand video and right hand video are input into the residual network Resnet18 to obtain face spatial features, left hand spatial features and right hand spatial features; The left hand spatial features and the right hand spatial features are concatenated according to the second dimension to obtain the hand spatial features; the global sign language word-level features, facial sign language word-level features and hand sign language word-level features with short-term temporal connections are extracted from the global spatial features, facial spatial features and hand spatial features using a one-dimensional convolution module; the one-dimensional convolution module is composed of a one-dimensional convolution and a temporal maximum pooling, and the structure of the one-dimensional convolution module is Conv 1D , MaxPooling 1D , Coonv 1D , MaxPooling 1D , Conv 1D It is a one-dimensional convolution with a convolution kernel size of 5, MaxPooling 1D is the temporal maximum pooling of size 2; Extracting global context-level features, facial context-level features and hand context-level features with long-term temporal connections from global sign language word-level features, facial sign language word-level features and hand sign language word-level features, and then using a fully connected layer and a softmax activation function to obtain normalized global classification probability matrices, facial classification probability matrices, hand classification probability matrices and global sign language word-level classification probability matrices; Based on the obtained classification probability matrix, a total loss function is constructed to train the neural network; the total loss function includes three parts: link temporal classification loss function, multi-cue mutual distillation loss and self-distillation loss; the trained neural network is used to predict sign language and output the sign language label; Linking temporal classification loss function The construction process is: The global classification probability matrix Y full , face classification probability matrix Y face , hand classification probability matrix Y hand And the global sign language word-level classification probability matrix Y gloss The real labels l corresponding to the sign language are input into their respective linked temporal classification networks to obtain the predicted probabilities; the input is the global classification probability matrix Y full The predicted probability P(l|Y full ), input is the face classification probability matrix Y face The predicted probability P(l|Y face ), the input is the hand classification probability matrix Y hand The predicted probability P(l|Y hand ) and the input is the global sign language word-level classification probability matrix Y gloss The predicted probability P(l|Y gloss ) is expressed as shown in the following formulas (33) to (36); In formulas (33) to (36), Π=(π 1 ,…,π i ,…π s ) represents the set of all possible paths, p(Π|Y full )、p(Π|Y face )、p(Π|Y hand ) and p(Π|Y gloss ) represent the input as Y full , Y head , Y hand and Y gloss The probability of outputting path Π when , B is a many-to-one mapping used to remove empty labels and duplicate labels in path Π; In order to maximize the predicted probability, the link temporal classification network uses the negative of the logarithm of the predicted probability as the loss function, so the link temporal global classification loss is Linking temporal face classification loss Linking temporal hand classification loss and link temporal global sign language word-level classification loss It is expressed as shown in the following formulas (37) to (40); In formulas (37) to (40), log(·) represents a logarithmic function with a natural number e as the base; In summary, the link temporal classification network loss function There are four parts of loss in total, which are expressed as shown in the following formula (41): Multi-cue mutual distillation loss The construction process is: Using the global classification probability matrix Y full , face classification probability matrix Y face and the hand classification probability matrix Y hand Construct KL divergence loss D KL , expressed as shown in the following formulas (42) to (47); In formulas (42) to (47), and Represents the global classification probability matrix Y full , face classification probability matrix Y face and the hand classification probability matrix Y hand The t-th vector in corresponds to the m-th probability value in the probability distribution, and M represents the number of probability values, that is, the output feature size of the fully connected layer; In summary, multi-cue mutual distillation loss It is expressed as shown in the following formula (48); Constructing the self-distillation loss The construction process is: Using the global classification probability matrix Y full And the global sign language word-level classification probability matrix Y gloss Constructing the self-distillation loss Self-distillation losses It is expressed as shown in the following formula (49): In formula (49), D KL (p full ||p gloss ) and D KL (p gloss ||p full ) are KL divergence losses, Represents the global sign language word-level classification probability matrix Y aloss The t-th vector in corresponds to the m-th probability value in the probability distribution; Therefore, the total loss function of the entire neural network is is the sum of the three partial losses, expressed as shown in the following formula (50); In formula (50), α and β represent hyperparameters used to adjust the loss ratio.

2. The continuous sign language recognition method based on multi-cue mutual distillation and self-distillation according to claim 1, It is characterized in that The dimension of the sign language video input network is (T, 3, 112, 112), and the dimensions of the face video, left hand video and right hand video input networks are (T, 3, 64, 64), (T, 3, 96, 96) and (T, 3, 96, 96), respectively, where T represents the number of video frames.

3. The continuous sign language recognition method based on multi-cue mutual distillation and self-distillation according to claim 1, It is characterized in that The OpenPose algorithm is used to obtain the human joint point set P in the sign language video. 1 ,…,p t ,…p T ), p t represents the set of human joint points in the original sign language image of the tth frame, p t contains 18 two-dimensional arrays, i.e., the coordinates of 18 joint points; each frame of the original sign language image is cropped to obtain a face image, a left hand image, and a right hand image; all frames of face images, left hand images, and right hand images are arranged in time series to obtain a face video, a left hand video, and a right hand video.

4. A continuous sign language recognition method based on multi-cue mutual distillation and self-distillation, It is characterized in that The specific steps of this method are: The first step is to obtain a sign language video and extract the face, left hand, and right hand videos from the sign language video; Assume that the sign language video V = (v 1 , …, v t ,…v T ), the sign language video is a video formed by the original sign language image sequence, T is the number of frames of the sign language video V, representing the 1st frame, ..., tth frame, ..., Tth frame of the original sign language image sequence, and each frame of the original sign language image is converted into 256×256 pixels; The OpenPose algorithm is used to obtain the human joint point set P in the sign language video V = (p 1 ,…,p t ,…p T ), p t represents the set of human joint points in the original sign language image of the tth frame, p t contains 18 two-dimensional arrays, i.e., the coordinates of 18 joint points; each frame of the original sign language image is cropped according to the coordinate points in the image to obtain the face image, the left hand image, and the right hand image; all the frames of the face image, the left hand image, and the right hand image are arranged in time series to obtain the face video Left Hand Video And right hand video and They represent the face image, left hand image and right hand image cropped from the original sign language image of frame t respectively; Step 2: Sign language video V, face video V face , Left hand video V lhand And right hand video V rhand Perform pre-processing; The sign language video V=(v 1 , …, v i ,…v T ) are adjusted to 128×128 pixels in each frame of the original sign language image, and randomly cropped to 112×112 pixels, and then randomly flipped horizontally, and then the pixel values ​​of the image are normalized to (-1, 1). The dimension of the sign language video V input to the neural network is (T, 3, 112, 112); the facial video V face , Left hand video V lhand And right hand video V rhand Each frame image in the image is randomly flipped horizontally, and then the pixel values ​​of the image are normalized to (-1,1). The face video V face , Left hand video V lhand And right hand video V rhand The dimensions of the input network are (T, 3, 64, 64), (T, 3, 96, 96), and (T, 3, 96, 96). The preprocessing expression is: In formulas (3) to (6), Resize(·) represents a function for adjusting the image size, RandomCrop(·) represents a function for randomly cropping an image to a fixed size, RandomFlip(·) represents a function for randomly flipping an image horizontally, and Normalize(·) represents a normalization function for normalizing the pixel values ​​of an image from (0, 255) to (-1, 1). The third step is to use the neural network to extract the sign language video V and the face video V face , Left hand video V lhand And right hand video V rhand The spatial features of the global spatial features F full , face spatial feature F face , left hand spatial feature F lhand and the right hand spatial feature F rhand ; The preprocessed sign language video V is input into the residual network Resnet50 (Residual Network 50) for spatial feature extraction to obtain the global spatial feature F full The dimension is (T, 1024), and the expression for extracting global spatial features is: In formula (7), Conv resnet50 (·) represents the function of the residual network Resnet50, Represents the global spatial feature F full The t-th vector in is the spatial feature extracted from the preprocessed sign language image of the t-th frame; The pre-processed face video V face , Left hand video V lhand And right hand video V rhand The facial spatial features are obtained by inputting them into the residual network Resnet18 (Residual Network 18) for spatial feature extraction. Left hand space characteristics and right hand spatial features F face 、F rhand and F lhand The dimensions are all (T, 1024), and the expressions of extracted spatial features are shown in the following formulas (8) to (10): In formulas (8) to (10), Conv resnet18 (·) represents the function of the residual network Resnet18, and Represent the facial spatial features F face , left hand spatial feature F lhand and the right hand spatial feature F rhand The tth vector in ; The fourth step is to use the one-dimensional convolution module to extract the global spatial features F full , face spatial feature F face , left hand spatial feature F lhand and the right hand spatial feature F rhand Extract global sign language word-level features, facial sign language word-level features and hand sign language word-level features with short-term temporal connections; The left hand spatial feature F lhand and the right hand spatial feature F rhand According to the second dimension, we can concatenate and obtain the hand space feature F hand , F hand The dimension is (T, 2048), and the concatenation expression is shown in the following formula (11): F hand =Cat(F lhand ,F rhand )#(11) In formula (11), Cat(·) represents the function of feature concatenation according to the second dimension. Use one-dimensional convolution modules to transform the global spatial features F full , face spatial feature F face and hand space feature F hand Perform feature extraction of short-term time connections to obtain the global sign language word-level feature F′ full , word-level features of facial sign language F' face and hand sign language word-level feature F' hand ; The one-dimensional convolution module consists of one-dimensional convolution and temporal maximum pooling. The specific structure of the one-dimensional convolution module is Conv 1D , MaxPooling 1D , Conv 1D , MaxPooling 1D , Conv 1D It is a one-dimensional convolution with a convolution kernel size of 5, MaxPooling 1D The maximum pooling of the size is 2, and the one-dimensional convolution module is used to extract the word-level features of the sign language as shown in the following formulas (12) to (14); In formulas (12) to (14), Conv1D(·) represents a one-dimensional convolution module. and Represents the global sign language word-level features F′ full , word-level feature of facial sign language F′ face and hand sign language word-level feature F' hand For the t-th vector in , the word-level feature dimension of sign language is (T′, 1024), T′=(N-4) / 2=(((T-4) / 2)-4) / 2, where N represents the time dimension; The one-dimensional convolution module can be divided into two stages. First, the global spatial feature F full , face spatial feature F face and hand space feature F hand The dimensions are converted to (1,1024, T), (1,1024, T) and (1,2048, T) respectively, and the one-dimensional convolution module and the temporal maximum pooling are used to extract features to obtain the intermediate features. and The specific operation is expressed as shown in the following formulas (15) to (17): In formulas (15) to (17), reshape(·) represents the function of transforming feature dimensions, Conv 1D (·) represents a one-dimensional convolution function with a convolution kernel size of 5, MaxPooling 1D (·) represents the temporal maximum pooling function, and Represents the intermediate features and The tth vector in, the intermediate feature and The dimensions are all (1, N, 1024), N = (T-4) / 2; Then the intermediate features and They are respectively used as input through another one-dimensional convolution in the one-dimensional convolution module and the temporal maximum pooling to be converted into a global sign language word-level feature F′ with short-term temporal connection. full , word-level feature of facial sign language F′ face and the word-level feature F′ of hand sign language hand , the specific operation is expressed as shown in the following formula (18)(19)(20); The fifth step is to extract the global sign language word-level features F′ full , word-level feature of facial sign language F′ face and the word-level feature F′ of hand sign language hand Extracting global context-level features E with long-term temporal connections full , facial contextual features E face and hand context level feature E hand ; The global sign language word-level feature F′ obtained in the fourth step above full , word-level feature of facial sign language F′ face and the word-level feature F′ of hand sign language hand The dimensions of are converted to (T′, 1, 1024) and are input into two layers of bidirectional gated recurrent units in series to extract global context-level features with long-term temporal connections. Facial context-level features and hand context-level features The expression of extracting context-level features is shown in the following formulas (21) to (23); In formulas (21) to (23), RNNGroup(·) represents a function for extracting context-level features with long-term temporal connections. RNNGroup(·) consists of two layers of bidirectional gated recurrent units connected in series. and Represent the global context level features E full , facial contextual features E face and hand context level feature E hand The tth vector in ; Step 6: Get the normalized global classification probability matrix Y full , face classification probability matrix Y face , hand classification probability matrix Y hand And the global sign language word-level classification probability matrix Y gloss ; The global context-level feature E obtained in the fifth step full , facial contextual features E face , hand context level feature E hand and the global sign language word-level feature F′ obtained in the fourth step full After passing through a fully connected layer respectively, the fully connected layer output feature FC is obtained full , FC face , FC hand and FC gloss , the size of the output feature of the fully connected layer is the vocabulary size 1296, which is expressed by the fully connected layer as shown in the following formulas (24) to (27); In formulas (24) to (27), fc(·) represents the fully connected layer. and Represents the fully connected layer output feature FC full , FC face , FC hand and FC gloss The tth vector in FC full , FC face , FC hand and FC gloss The dimensions of are all (T′, 1296); The fully connected layer outputs the feature FC full , FC face , FC hand and FC gloss The global classification probability matrix Y is obtained by the softmax activation function full , face classification probability matrix Y face , hand classification probability matrix Y hand And the global sign language word-level classification probability matrix Y gloss ; The softmax activation function converts each vector in the feature into a probability distribution. The sum of the probability values ​​corresponding to each vector is 1. The classification probability matrix is ​​expressed as shown in the following formulas (28) to (31); In formulas (28) to (31), softmax(·) represents the activation function, and Represents the global classification probability matrix Y full , face classification probability matrix Y face , hand classification probability matrix Y hand And the global sign language word-level classification probability matrix Y gloss The tth vector in ; where the global classification probability matrix Y full The mth probability value of the tth vector in The calculation process of is expressed as shown in the following formula (32): In formula (32), exp(·) represents an exponential function with the natural constant e as the base. Represents the fully connected layer output feature FC full The mth probability value of the tth vector in , where M represents the number of probability values; Step 7: Construct a total loss function and train the neural network; the total loss function includes the link time series classification loss function Multi-cue mutual distillation loss and self-distillation losses Step 7.1: Constructing the Link Time Classification Loss Function The global classification probability matrix Y obtained in the sixth step full , face classification probability matrix Y face , hand classification probability matrix Y hand And the global sign language word-level classification probability matrix Y gloss The real labels l corresponding to the sign language are input into their respective linked temporal classification networks to obtain the predicted probabilities; the input is the global classification probability matrix Y full The predicted probability P(l|Y full ), input is the face classification probability matrix Y face The predicted probability P(l|Y face ), the input is the hand classification probability matrix Y hand The predicted probability P(l|Y hand ) and the input is the global sign language word-level classification probability matrix Y gloss The predicted probability P(l|Y gloss ) is expressed as shown in the following formulas (33) to (36); In formulas (33) to (36), Π=(π 1 ,…,π i ,…π s ) represents the set of all possible paths, p(Π|Y full )、p(Π|Y face )、p(Π|Y hand ) and p(Π|Y gloss ) represent the input as Y full , Y head , Y hand and Y gloss The probability of outputting path Π when , B is a many-to-one mapping used to remove empty labels and duplicate labels in path Π; In order to maximize the predicted probability, the link temporal classification network uses the negative of the logarithm of the predicted probability as the loss function, so the link temporal global classification loss is Linking temporal face classification loss Linking temporal hand classification loss and link temporal global sign language word-level classification loss It is expressed as shown in the following formulas (37) to (40); In formulas (37) to (40), log(·) represents a logarithmic function with a natural number e as the base; In summary, the link temporal classification network loss function There are four parts of loss in total, which are expressed as shown in the following formula (41): Step 7.2: Construct multi-cue mutual distillation loss Using the global classification probability matrix Y obtained in the sixth step full , face classification probability matrix Y face and the hand classification probability matrix Y hand Construct KL divergence loss D KL , expressed as shown in the following formulas (42) to (47); In formulas (42) to (47), and Represents the global classification probability matrix Y full , face classification probability matrix Y face and the hand classification probability matrix Y hand The t-th vector in corresponds to the m-th probability value in the probability distribution, and M represents the number of probability values, that is, the output feature size of the fully connected layer; In summary, multi-cue mutual distillation loss It is expressed as shown in the following formula (48); Step 7.3: Constructing the self-distillation loss Using the global classification probability matrix Y obtained in the sixth step full And the global sign language word-level classification probability matrix Y gloss Constructing the self-distillation loss Self-distillation losses It is expressed as shown in the following formula (49): In formula (49), D KL (p full ||p gloss ) and D KL (p gloss| |p full ) are KL divergence losses, Represents the global sign language word-level classification probability matrix Y gloss The t-th vector in corresponds to the m-th probability value in the probability distribution; Therefore, the total loss function of the entire neural network is is the sum of the three partial losses, expressed as shown in the following formula (50); In formula (50), α and β represent hyperparameters used to adjust the loss ratio; Step 8: Use the trained neural network to predict sign language and output the sign language label G; Using the neural network trained in step 7, beam search decoding (Beam) is used to decode the global classification probability matrix Y obtained in step 6 full Decode and complete the mapping from the classification probability matrix to the sign language sequence to obtain the sign language label G, which is expressed as shown in the following formula (51): G=Beam(Y full )#(51) In formula (51), Beam(·) represents the beam search decoding function; At this point, continuous sign language recognition based on multi-cue mutual distillation and self-distillation is completed.