Continuous sign language recognition method and device based on resnet34 network-attention mechanism

Abstract

The application provides a continuous sign language recognition method and device based on a ResNet34 network-attention mechanism, and relates to the technical field of artificial intelligence recognition, and comprises the following steps: S1, acquiring a first video data training set, and acquiring a second video data training set by using a KFE clustering algorithm; S2, constructing a ResNet34 network topology, fusing a PSA channel attention mechanism and a RCC space attention mechanism into a PR attention mechanism, and integrating the PR attention mechanism with the ResNet34 network topology to extract feature information of the second video data training set; S3, constructing a BiLSTM network topology to encode the feature information of the second video data training set, and adopting an LSTM-CTC end-to-end network structure topology to decode the second video data training set after encoding; and S4, constructing an objective function to adjust network topology parameters of the ResNet34 network topology and the PR attention mechanism, BiLSTM network topology parameters and LSTM-CTC end-to-end network structure topology parameters. The application can alleviate the technical problem of overfitting of a neural network structure caused by video redundancy in the prior art.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence recognition technology, and in particular to a continuous sign language recognition method and apparatus based on the ResNet34 network-attention mechanism. Background Technology

[0002] Sign language, as a specific communication language for deaf and hard-of-hearing individuals, integrates knowledge from the fields of natural language processing and computer vision. Sign language recognition, as a subtask within this field, has attracted considerable attention from researchers. Generally, sign language recognition is divided into isolated word recognition and continuous sign language recognition tasks. Although significant progress has been made in isolated word recognition, continuous sign language recognition is gaining increasing importance because it neglects the underlying semantic relationships and long-term temporal dependencies within a sign language sentence.

[0003] In recent years, various sign language recognition methods have been designed to improve the accuracy of continuous sign language recognition. Early sign language recognition research often relied on data gloves and other sensor devices to collect motion changes and temporal information of gestures in real time, and modeled the temporal information of sign language through traditional hidden Markov models, or extracted hand information through conditional random fields.

[0004] With the rise of deep learning (DL), more and more researchers have used neural networks for continuous sign language recognition. The rapid development of neural networks has opened new research doors for sign language recognition, translation, and generation. Currently, researchers are using convolutional neural networks, recurrent convolutional neural networks, and graph convolutional neural networks combined with skeleton information to recognize sign language. Others are using continuous-time classification methods to compare sign language videos with the recognized text, thus improving network convergence. However, current sign language recognition work rarely focuses on multimodal input. For a neural network, inputting a complete sign language sequence, the redundancy of the video can easily lead to overfitting. Therefore, designing a novel continuous sign language recognition method is essential. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a continuous sign language recognition method and apparatus based on the ResNet34 network-attention mechanism, so as to alleviate the technical problem of overfitting of the neural network structure with video redundancy and improve the generalization ability of the continuous sign language recognition method.

[0006] This invention discloses a continuous sign language recognition method based on a ResNet34 network-attention mechanism, comprising:

[0007] S1: Obtain a first video data training set, which includes RGB video and depth video, and use the KFE clustering algorithm to extract keyframes from the first video data training set to obtain a second video data training set, which has labels.

[0008] S2: Construct a ResNet34 network topology, fuse the PSA channel attention mechanism and the RCC spatial attention mechanism into a PR attention mechanism, and integrate the PR attention mechanism with the ResNet34 network topology to extract feature information from the second video data training set;

[0009] S3: Construct a BiLSTM network topology to encode the feature information of the second video data training set, and use an LSTM-CTC end-to-end network structure topology to decode the encoded second video data training set;

[0010] S4: Construct an objective function to adjust the network topology parameters after the ResNet34 network topology is integrated with the PR attention mechanism, the BiLSTM network topology parameters, and the LSTM-CTC end-to-end network structure topology parameters.

[0011] Preferably, the method further includes:

[0012] Obtain the first video data test set and test the completed ResNet34 network topology integrated with the PR attention mechanism, the BiLSTM network topology, and the LSTM-CTC end-to-end network structure topology.

[0013] Preferably, the step of extracting keyframes from the first video data training set using the KFE clustering algorithm to obtain the second video data training set includes:

[0014] Obtain the initial threshold, the frame set of the first video data training set, and the cluster centroids of all clusters;

[0015] The frames of the first video data training set are obtained based on the frame set of the first video data training set, and the nearest distance from the frame of the first video data training set to the cluster centroid is obtained based on the cluster centroid of all the clusters.

[0016] Determine whether the nearest distance from a frame to the cluster centroid in the first video data training set is less than an initial threshold;

[0017] If so, the frames of the first video data training set are assigned to the class of the nearest cluster centroid, the frames of the first video data training set are removed from the frame set of the first video data training set, and the steps of obtaining the initial threshold, the frame set of the first video data training set, and the cluster centroids of all clusters are executed.

[0018] If not, the frames of the first video data training set are assigned to a new category, the frames of the first video data training set are removed from the frame set of the first video data training set, and the steps of obtaining the initial threshold, the frame set of the first video data training set, and the cluster centroids of all clusters are executed.

[0019] Preferably, the ResNet34 network topology includes an initial layer, a first residual layer, a second residual layer, a third residual layer, a fourth residual layer, and a global average pooling layer;

[0020] The number of convolution kernels in the first residual layer is 64, and the number of the first residual layers is 3;

[0021] The second residual layer has 128 convolution kernels and 4 layers.

[0022] The number of convolution kernels in the third residual layer is 256, and the number of third residual layers is 6;

[0023] The fourth residual layer has 512 convolution kernels and 3 layers.

[0024] The steps for integrating the PR attention mechanism with the ResNet34 network topology include:

[0025] The PR attention mechanism is introduced between the fourth residual layer and the global average pooling layer.

[0026] Preferably, in the step of fusing the PSA channel attention mechanism and the RCC spatial attention mechanism into a PR attention mechanism, the PSA channel attention mechanism is:

[0027] [X0, X1, ..., X S-1 ] = Split(X);

[0028] F i =Conv(K i ×K i G i (X) i );

[0029] F = Cat([F0, F1, ..., F)) S-1 ]);

[0030] X∈R C×W×H- The first feature map obtained after the second video training set is processed by the first four residual layers of the ResNet34 network;

[0031] C, W, and H - the channels, width, and height of the first feature map;

[0032] Split - on the channel dimension of the first feature map X∈R C×W×H Perform the operation of dividing the data equally into S parts;

[0033] X i ∈R C / S×W×H -The first feature map, after being equally divided, has a channel ratio of C / S;

[0034] K i -Different convolution kernel parameters;

[0035] G i - Parameters of grouped convolution;

[0036] F i ∈R C / S×W×H - Multi-scale features after multi-scale feature extraction;

[0037] Cat- splices multi-scale features from different receptive fields along the channel dimension;

[0038] F∈R C×W×H -Feature vector after multi-scale feature concatenation;

[0039] The following formula is used to extract the weights from the feature vector obtained after concatenating the multi-scale features:

[0040] g i =AvgPool(F i );

[0041] Z i =σ(W1δ(W0(g) i )));

[0042] Z = Cat([Z0, Z1, ..., Z)) S-1 ]);

[0043] AvgPool(·) - represents global average pooling;

[0044] σ(·) is the sigmoid activation function;

[0045] δ(·) is the ReLU activation function;

[0046] g i ∈R C / S×1×1 - Feature vectors that undergo global average pooling for multi-scale features;

[0047] W0 and W1 are weight matrices with dimensions [C / S / r, C / S] and [C / S, C / S / r], respectively, where r represents the reduction rate;

[0048] Z i - Attention weights for different parts of dimension [C / S, 1, 1];

[0049] Z-dimensional cross-dimensional channel attention feature weight map with dimension [C, 1, 1];

[0050] The attention weights are normalized using the following formula, and then tensor products are performed between the weights and the feature vectors extracted from multi-scale features:

[0051] att = Softmax(Z);

[0052] Y = att⊙F;

[0053] att - Normalized channel attention weights.

[0054] The RCC attention mechanism involves concatenating the Criss-Cross module twice to obtain rich contextual information. The Criss-Cross channel attention mechanism is as follows:

[0055] Q = W Q Y;

[0056] K = W K Y;

[0057] V = W V Y;

[0058] W Q and W K Both are weight matrices of dimension [C′, C].

[0059] W V It is a weight matrix with dimensions [C, C].

[0060] The PR attention mechanism is integrated with the ResNet34 network topology using the following formula to extract feature information from the second video data training set:

[0061] Perform an Affinity operation to obtain the relationship between each pixel in a feature map of size [W, H] and its corresponding pixels in the same row and column:

[0062] D = Affinity(Q, K);

[0063]

[0064] Affinity-Q and K are feature maps of dimension [C′, W, H].

[0065] Ω u - In the spatial dimension of Q, there exists a feature vector Q for each position u. u ∈R C′ ;

[0066] Where Ω i，u ∈R C′ It is Ω u The i-th element,

[0067] d i，u ∈D-feature Q u and Ω i，u The correlation of i = [1, ..., H+W-1].

[0068] Based on the relationship D between each pixel in a feature map of size [W, H] and its corresponding pixels in the same row and column, a softmax layer is applied along the channel dimension to compute feature map A:

[0069] A = softmax(D);

[0070] Aggregation operation is performed on feature map A to collect contextual information Y′:

[0071]

[0072] Aggregation - In the spatial dimension of V, there exists a feature vector V for each position u. u ∈R C set Φ u ∈R (H+W-1)×C It is the set of feature vectors extracted from matrix V that are in the same row and column as position u;

[0073] Y′ - This refers to the context information of long connections in the vertical and horizontal directions that have been captured;

[0074] Repeatedly concatenate the context information of the captured long connections in the vertical and horizontal directions, as shown in the following formula:

[0075] Y″ = CrissCross(Y′);

[0076] Y′=CrissCross(Y);

[0077] Y″ - This refers to the feature vector for which global pixel information has been obtained.

[0078] Preferably, the step of decoding the encoded second video data training set using the LSTM-CTC end-to-end network topology and the labels of the second video data training set includes:

[0079] The calculation of the CTC loss function for an LSTM-CTC end-to-end network includes:

[0080] Define a many-to-one mapping function β(·) to correspond to its target sequence y:

[0081] in

[0082]

[0083] In the formula, π n The π label at time -n;

[0084] - The probability of it occurring at time n;

[0085] The CTC loss function is:

[0086]

[0087] Preferably, the objective function is constructed using the following formula:

[0088]

[0089] The objective function constructed by L is used to adjust the network topology parameters of the ResNet34 network topology integrated with the PR attention mechanism, the network topology parameters of the BiLSTM encoder, and the network structure topology of the LSTM-CTC decoder.

[0090] S - the dimension of the given second video data training set;

[0091] ||ω|| 2 - Regularization terms to avoid overfitting;

[0092] λ - Hyperparameter of the regularization term.

[0093] Preferably, the step of acquiring the first video data test set and testing the constructed ResNet34 network topology integrated with the PR attention mechanism, the BiLSTM network topology, and the LSTM-CTC end-to-end network structure topology includes:

[0094] The WER value indicates the accuracy of the recognition:

[0095]

[0096] S, I (Ins), and D (Del) represent the minimum replacement, insertion, and deletion operations, respectively.

[0097] N represents the total number of words in the tag.

[0098] Preferably, the step of acquiring the first video data test set and testing the constructed ResNet34 network topology integrated with the PR attention mechanism, the BiLSTM network topology, and the LSTM-CTC end-to-end network structure topology includes:

[0099] Accuracy represents the precision of the recognition process.

[0100]

[0101] S, I (Ins), and D (Del) represent the minimum replacement, insertion, and deletion operations, respectively.

[0102] N represents the total number of words in the tag.

[0103] On the other hand, a continuous sign language recognition device based on a ResNet34 network-attention mechanism includes:

[0104] Video acquisition module: used to acquire a first video data training set, the first video dataset including RGB video and depth video, and to extract keyframes from the first video data training set using the KFE clustering algorithm to obtain a second video data training set, the second video training set having labels;

[0105] Feature extraction module: used to construct the ResNet34 network topology and integrate the PR attention mechanism with the ResNet34 network topology to extract feature information from the second video data training set;

[0106] Decoding module: Used to construct a BiLSTM network topology to encode the feature information of the second video data training set, and to use an LSTM-CTC end-to-end network structure topology to decode the encoded second video data training set;

[0107] Parameter tuning module: Used to construct objective functions to adjust the network topology parameters after the integration of the ResNet34 network topology and the PR attention mechanism, the BiLSTM network topology parameters, and the LSTM-CTC end-to-end network structure topology parameters.

[0108] The embodiments of this invention bring the following beneficial effects: This invention provides a continuous sign language recognition method and apparatus based on a ResNet34 network-attention mechanism, comprising: S1: acquiring a first video data training set, the first video dataset including RGB video and depth video, and using the KFE clustering algorithm to extract keyframes from the first video data training set to obtain a second video data training set, the second video training set having labels; S2: constructing a ResNet34 network topology, and integrating the PR attention mechanism with the ResNet34 network topology to extract feature information from the second video data training set; S3: constructing a BiLSTM network topology to encode the feature information of the second video data training set, and using an LSTM-CTC end-to-end network structure topology to decode the encoded second video data training set; S4: constructing an objective function to adjust the network topology parameters after integrating the ResNet34 network topology and the PR attention mechanism, the BiLSTM network topology parameters, and the LSTM-CTC end-to-end network structure topology parameters. The method and apparatus provided by this invention can alleviate the technical problem of overfitting of neural network structures caused by video redundancy in the prior art, and improve the generalization ability of continuous sign language recognition methods.

[0109] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0110] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0111] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0112] Figure 1 This is a flowchart of a continuous sign language recognition method based on a ResNet34 network-attention mechanism provided in an embodiment of the present invention.

[0113] Figure 2 This is a network structure diagram based on the ResNet34 network and attention mechanism provided in an embodiment of the present invention;

[0114] Figure 3The flowchart of the ResNet34 residual module network for the continuous sign language recognition method based on the ResNet34 network-attention mechanism provided in the embodiments of the present invention is as follows:

[0115] Figure 4 The network structure diagram of the ResNet34 residual module for the continuous sign language recognition method based on the ResNet34 network attention mechanism provided in this embodiment of the invention is shown below.

[0116] Figure 5 A network diagram of the LSTM basic unit module of the continuous sign language recognition method based on ResNet34 network-attention mechanism provided in the embodiments of the present invention;

[0117] Figure 6 The network structure diagram of the BiLSTM encoder for the continuous sign language recognition method based on the ResNet34 network-attention mechanism provided in the embodiments of the present invention is shown. Detailed Implementation

[0118] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0119] Currently, sign language recognition work rarely focuses on multimodal input. For a neural network, inputting a complete sign language sequence can easily lead to overfitting due to video redundancy. Based on this, the present invention provides a continuous sign language recognition method and apparatus based on the ResNet34 network-attention mechanism, which can alleviate the technical problem of overfitting of neural network structures due to video redundancy and improve the generalization ability of continuous sign language recognition methods.

[0120] To facilitate understanding of this embodiment, the continuous sign language recognition method and apparatus based on the ResNet34 network-attention mechanism disclosed in this embodiment will first be described in detail.

[0121] Example 1:

[0122] Combination Figure 1 Embodiment 1 of the present invention provides a continuous sign language recognition method based on a ResNet34 network-attention mechanism, comprising:

[0123] S1: Obtain a first video data training set, which includes RGB video and depth video, and use the KFE clustering algorithm to extract keyframes from the first video data training set to obtain a second video data training set, which has labels.

[0124] S2: Construct a ResNet34 network topology and integrate the PR attention mechanism with the ResNet34 network topology to extract feature information from the second video data training set;

[0125] Combination Figure 6 S3: Construct a BiLSTM network topology to encode the feature information of the second video data training set, and use an LSTM-CTC end-to-end network structure topology to decode the encoded second video data training set; it should be noted that LSTM, as a special type of RNN network, is capable of learning long-distance temporal dependencies. Figure 5 As shown, at time t, the LSTM has three inputs: the current input value x of the network at time t. t The output value h of the LSTM at the previous time step t-1 and the cell state C from the previous moment. t-1 The LSTM output has two parts: the current LSTM output value h. t The current cell state C t Its operating mechanism consists of internal input gates, forget gates, and output gates.

[0126] Combination Figure 5 LSTM has the following characteristics:

[0127] 1) The forget gate determines how much of the cell state from the previous time step needs to be retained in the current time step. f can be obtained through the forget gate. t ;

[0128] f t =σ(W f ·[h t-1 x t ]+b f );

[0129] 2): The input gate determines how much of the network's input data needs to be stored to the cell state at the current moment. The input gate can be used to obtain i... t The temporary state at the current moment can be obtained through the cell state. Then, the unit state C of the previous LSTM cell structure is applied. t-1 The output f of the forget gate t The output i of the input gate t and temporary status The current cell state C can be obtained.t .

[0130] i t =σ(W i ·[h t-1 x t ]+b i )

[0131]

[0132] 3): The output gate controls how much of the current cell state needs to be output to the current output value. The output gate can be used to obtain the output value. t Combined with the current cell state C t and o t The final output h can be obtained. t ;

[0133] o t =σ(W o ·[h t-1 x t ]+b o );

[0134] h t =o t *tan(C t )

[0135] BiLSTM is a combination of forward LSTM and backward LSTM to obtain long-term contextual information. Specifically, it uses LSTM to compute the inverted input sequence, and finally stacks the results of the forward LSTM and the backward LSTM.

[0136] Furthermore, the high-dimensional sign language video feature vector extracted by the PR attention mechanism and the ResNet34 network topology integration module is fed into the BiLSTM-LSTM encoder and decoder. Specifically, we input the sign language images X = {x1, x2, ..., x...} i , ..., x T The input is fed into the PR attention mechanism and integrated with the ResNet34 network topology to extract feature information from the second video data training set;

[0137] Combination Figure 6 The network will output E = {e1, e2, ..., e}. i , ..., e T}, where xi∈R C×H×W e i ∈R C′ T represents the number of frames in the sign language video. The outputs of the feature extraction network are stacked to obtain E′∈R. T×C′It can be used as input to a BiLSTM encoder;

[0138] S4: Construct an objective function to adjust the network topology parameters after the ResNet34 network topology is integrated with the PR attention mechanism, the BiLSTM network topology parameters, and the LSTM-CTC end-to-end network structure topology parameters.

[0139] Preferably, the method further includes:

[0140] Obtain the first video data test set and test the completed ResNet34 network topology integrated with the PR attention mechanism, the BiLSTM network topology, and the LSTM-CTC end-to-end network structure topology.

[0141] Preferably, the step of using the KFE clustering algorithm to extract keyframes from the first video data training set to obtain the second video data training set includes:

[0142] Obtain the initial threshold, the frame set of the first video data training set, and the cluster centroids of all clusters;

[0143] The frames of the first video data training set are obtained based on the frame set of the first video data training set, and the nearest distance from the frame of the first video data training set to the cluster centroid is obtained based on the cluster centroid of all the clusters.

[0144] Determine whether the nearest distance from a frame to the cluster centroid in the first video data training set is less than an initial threshold;

[0145] If so, the frames of the first video data training set are assigned to the class of the nearest cluster centroid, the frames of the first video data training set are removed from the frame set of the first video data training set, and the steps of obtaining the initial threshold, the frame set of the first video data training set, and the cluster centroids of all clusters are executed.

[0146] If not, the frames of the first video data training set are assigned to a new category, the frames of the first video data training set are removed from the frame set of the first video data training set, and the steps of obtaining the initial threshold, the frame set of the first video data training set, and the cluster centroids of all clusters are executed.

[0147] Combination Figures 2 to 4 Preferably, the ResNet34 network topology includes an initial layer, a first residual layer, a second residual layer, a third residual layer, a fourth residual layer, and a global average pooling layer.

[0148] The number of convolution kernels in the first residual layer is 64, and the number of the first residual layers is 3;

[0149] The second residual layer has 128 convolution kernels and 4 layers.

[0150] The number of convolution kernels in the third residual layer is 256, and the number of third residual layers is 6;

[0151] The fourth residual layer has 512 convolution kernels and 3 layers.

[0152] Furthermore, as a backbone network of CNNs, ResNet has long been widely used in various scenarios in computer vision.

[0153] ResNet (Deep Residual Network) is characterized by its use of numerous residual units with identical parameters, forming BasicBlocks. Multiple BasicBlocks are combined, along with a preprocessing layer and a final fully connected classification layer, to form a ResNet network. The residual units are as follows: Figure 3 As shown. Considering the significant impact of neural network depth, this paper chooses to use ResNet to address this problem. Specifically, this application uses... Figure 4 The ResNet34 shown is used as the backbone network of this invention. By incorporating spatial and channel attention mechanisms, it can effectively extract image feature information from sign language videos while avoiding the gradient vanishing problem in deep networks.

[0154] More specifically, the steps for integrating the PR attention mechanism with the ResNet34 network topology include:

[0155] The PR attention mechanism is introduced between the fourth residual layer and the global average pooling layer.

[0156] It should be noted that in the embodiments provided by this invention, because the feature maps contain redundant information, this paper highlights the important regions of spatial hand movements and uses different weights to represent different levels of importance. Since each channel represents a unique detector, the purpose of channel attention is to focus on which features are meaningful. Furthermore, this paper introduces a spatial attention module that focuses on which spatial features are more meaningful.

[0157] Preferably, in the step of fusing the PSA channel attention mechanism and the RCC spatial attention mechanism into a PR attention mechanism, the PSA channel attention mechanism is:

[0158] [X0, X1, ..., X S-1 ] = Split(X);

[0159] F i=Conv(K i ×K i G i (X) i );

[0160] F = Cat([F0, F1, ..., F)) S-1 ]);

[0161] X∈R C×W×H - The first feature map obtained after the second video training set is processed by the first four residual layers of the ResNet34 network;

[0162] C, W, and H - the channels, width, and height of the first feature map;

[0163] Split - on the channel dimension of the first feature map X∈R C×W×H Perform the operation of dividing the data equally into S parts;

[0164] X i ∈R C / S×W×H -The first feature map, after being equally divided, has a channel ratio of C / S;

[0165] K i -Different convolution kernel parameters;

[0166] G i - Parameters of grouped convolution;

[0167] F i ∈R C / S×W×H - Multi-scale features after multi-scale feature extraction;

[0168] Cat- splices multi-scale features from different receptive fields along the channel dimension;

[0169] F∈R C×W×H -Feature vector after multi-scale feature concatenation;

[0170] The following formula is used to extract the weights of the feature vector obtained after concatenating the multi-scale features:

[0171] g i =AvgPool(F i );

[0172] Z i =σ(W1δ(W0(g) i )));

[0173] Z = Cat([Z0, Z1, ..., Z)) S-1 ]);

[0174] AvgPool(·) - represents global average pooling;

[0175] σ(·) is the sigmoid activation function;

[0176] δ(·) is the ReLU activation function;

[0177] g i ∈R C / S×1×1 -Feature vectors that undergo global average pooling for multi-scale features;

[0178] W0 and W1 are weight matrices with dimensions [C / S / r, C / S] and [C / S, C / S / r], respectively, where r represents the reduction rate;

[0179] Z i - Attention weights for different parts of dimension [C / S, 1, 1];

[0180] Z-dimensional cross-dimensional channel attention feature weight map with dimension [C, 1, 1];

[0181] The attention weights are normalized using the following formula, and then tensor products are performed between the weights and the feature vectors extracted from multi-scale features:

[0182] att = Softmax(Z);

[0183] Y = att⊙F;

[0184] att - Normalized channel attention weights;

[0185] It should be noted that, in the embodiments provided by this invention, the RCC attention mechanism involves concatenating the Criss-Cross module twice to obtain rich contextual information, wherein the Criss-Cross channel attention mechanism is as follows:

[0186] Q = W Q Y;

[0187] K = W K Y;

[0188] V = W V Y;

[0189] W Q and W K Both are weight matrices of dimension [C′, C].

[0190] W V It is a weight matrix with dimensions [C, C].

[0191] The PR attention mechanism is integrated with the ResNet34 network topology using the following formula to extract feature information from the second video data training set:

[0192] Perform an Affinity operation to obtain the relationship between each pixel in a feature map of size [W, H] and its corresponding pixels in the same row and column:

[0193] D = Affinity(Q, K);

[0194]

[0195] Affinity-Q and K are feature maps of dimension [C′, W, H].

[0196] Ω u - In the spatial dimension of Q, there exists a feature vector Q for each position u. u ∈R C′ ;

[0197] Where Ω i，u ∈R C′ It is Ω u The i-th element,

[0198] d i，u ∈D-feature Q u and Ω i，u The correlation of i = [1, ..., H+W-1].

[0199] Furthermore, in the spatial dimension of Q, there exists a feature vector Q for each position u. u ∈R C′ At the same time, the set of eigenvectors Ω that are in the same row and column as position u can be extracted from the K matrix. u ∈R (H+W-1)×C′ , where Ω i，u ∈R C′ It is Ω u The i-th element, d i，u ∈D is a feature Q u and Ω i，u， The relevance of i = [1, ..., H+W-1];

[0200] Based on the relationship D between each pixel in a feature map of size [W, H] and its corresponding pixels in the same row and column, a softmax layer is applied along the channel dimension to compute feature map A:

[0201] A = softmax(D);

[0202] Aggregation operation is performed on feature map A to collect contextual information Y′:

[0203]

[0204] Aggregation - In the spatial dimension of V, there exists a feature vector V for each position u. u ∈R C set Φ u ∈R (H+W-1)×C It is the set of feature vectors extracted from matrix V that are in the same row and column as position u;

[0205] Y′ - This refers to the context information of long connections captured in the vertical and horizontal directions;

[0206] However, at this point, the pixel-level feature information is still somewhat sparse. Therefore, it is necessary to repeatedly concatenate the context information of the long connections in the vertical and horizontal directions that have been captured, as shown in the following formula:

[0207] Y″ = CrissCross(Y′);

[0208] Y′=CrissCross(Y);

[0209] Y″ - This refers to the feature vector for which global pixel information has been obtained.

[0210] Preferably, the step of decoding the encoded second video data training set using the LSTM-CTC end-to-end network topology and the labels of the second video data training set includes:

[0211] The calculation of the CTC loss function for an LSTM-CTC end-to-end network includes:

[0212] Define a many-to-one mapping function β(·) to correspond to its target sequence y:

[0213] in

[0214]

[0215] In the formula, π n The π label at time -n;

[0216] - The probability of it occurring at time n;

[0217] The CTC loss function is:

[0218]

[0219] Furthermore, CTC is an objective function that integrates all possibilities for alignment between the input and target sequences. In the dataset used in this invention, the label (-) has been added to the sign language annotations to accurately simulate the transition between two adjacent sign language words;

[0220] The intermediate label path of the input sequence is represented as π = (π1, π2, ..., π). t , ..., π T ), where π t ∈{V∪{-}*}; V is the sign language vocabulary, and the asterisk is a wildcard;

[0221] For a given input X, the probability of path π is calculated as follows:

[0222]

[0223] π n It is the π label at time n. It is the probability of occurring at time n.

[0224] Due to the various subdivisions of sign language annotation labels, the same input sequence and target sequence can easily produce different alignments. Therefore, CTC defines a many-to-one mapping function β(·) to align with the target sequence y, i.e., y = β(π). The probability of y can be defined as the sum of probabilities of all matching alignments, calculated as follows:

[0225]

[0226] Therefore, the above formula can be replaced with the CTC loss function, defined as follows:

[0227]

[0228] Preferably, the objective function is constructed using the following formula:

[0229]

[0230] The objective function constructed by L is used to adjust the network topology parameters of the ResNet34 network topology integrated with the PR attention mechanism, the network topology parameters of the BiLSTM encoder, and the network structure topology of the LSTM-CTC decoder.

[0231] S - the dimension of the given second video data training set;

[0232] ||ω|| 2 - Regularization terms to avoid overfitting;

[0233] λ - Hyperparameter of the regularization term.

[0234] Preferably, the step of acquiring the first video data test set and testing the constructed ResNet34 network topology integrated with the PR attention mechanism, the BiLSTM network topology, and the LSTM-CTC end-to-end network structure topology includes:

[0235] The WER value indicates the accuracy of the recognition:

[0236]

[0237] S, I (Ins), and D (Del) represent the minimum replacement, insertion, and deletion operations, respectively.

[0238] N represents the total number of words in the tag.

[0239] It should be noted that Word Error Rate (WER) is frequently used as an evaluation metric in speech recognition and machine translation tasks. In this invention, we evaluate network quality using different experimental metrics, WER and Accuracy. While observing evaluation metrics can determine model performance, this invention does not involve learning the network based on metrics; therefore, the CRB-Net designed in this invention still uses gradient descent for network training.

[0240] WER measures sequence transitions using a combination of substitution, deletion, and insertion operations. For sign language recognition tasks, a low WER value often indicates high recognition accuracy. The formula for calculating WER is as follows:

[0241]

[0242] Preferably, the step of acquiring the first video data test set and testing the constructed ResNet34 network topology integrated with the PR attention mechanism, the BiLSTM network topology, and the LSTM-CTC end-to-end network structure topology includes:

[0243] Accuracy represents the precision of the recognition process.

[0244]

[0245] S, I (Ins), and D (Del) represent the minimum replacement, insertion, and deletion operations, respectively.

[0246] N represents the total number of words in the tag;

[0247] Example 2:

[0248] This invention provides a continuous sign language recognition device based on a ResNet34 network-attention mechanism, comprising:

[0249] Video acquisition module: used to acquire a first video data training set, the first video dataset including RGB video and depth video, and to extract keyframes from the first video data training set using the KFE clustering algorithm to obtain a second video data training set, the second video training set having labels;

[0250] Feature extraction module: used to construct the ResNet34 network topology and integrate the PR attention mechanism with the ResNet34 network topology to extract feature information from the second video data training set;

[0251] Decoding module: Used to construct a BiLSTM network topology to encode the feature information of the second video data training set, and to use an LSTM-CTC end-to-end network structure topology to decode the encoded second video data training set;

[0252] Parameter tuning module: Used to construct objective functions to adjust the network topology parameters after the integration of the ResNet34 network topology and the PR attention mechanism, the BiLSTM network topology parameters, and the LSTM-CTC end-to-end network structure topology parameters.

[0253] Example 3:

[0254] In Embodiment 3 of this invention, to demonstrate the generalization ability of the proposed network model on the test set under strongly supervised learning, the method of this invention was evaluated on two large sign language datasets: the CSL dataset from the University of Science and Technology of China, and our own TJUT sign language recognition and translation dataset (TJUT-SLRT). Experimental results show that the algorithm proposed in this invention has high accuracy in evaluating the WER index on different datasets, thus proving that this invention has extremely high generalization ability.

[0255] This invention has the following advantages:

[0256] 1) The algorithm was evaluated on two large-scale sign language datasets. Experiments were conducted on the test set divided by the TJUT-SLRT dataset. The results are shown in the table below. It can be seen that the method provided in this application has a lower word error rate (WER) than the continuous sign language recognition method based on the CBAM attention mechanism, which also verifies that the generalization of this method is good.

[0257] WER training results of this method and a sign language recognition method based on the CBAM attention mechanism on the Tjut test dataset:

[0258] ResNet34+CBAM+BiLSTM ResNet34+PRR+BiLSTM (Ours) TJUT 11.45％ 11.26％

[0259] WER training results of this method on the CSL dataset:

[0260] DataSet DEV TEST CSL 2.01％ 1.76％

[0261] Accuracy of this method on the CSL dataset under different settings:

[0262] Method Accuray CNN+LSTM 0.873 ResNet18+LSTM 0.905 ResNet34+LSTM 0.926 CNN+BiLSTM 0.896 ResNet18+BiLSTM 0.928 ResNet34+BiLSTM 0.943 PRR-ResNet34+BiLSTM 0.982

[0263] 2) Using RGB video and depth video data helps the network learn image feature representations better;

[0264] 3) The KFE clustering algorithm based on K-means algorithm is used to reduce video redundancy, further alleviate the technical problem of overfitting of neural network structure, and improve the generalization ability of continuous sign language recognition method.

[0265] 4) The PR attention module is introduced into the existing ResNet34 network topology to make the network more focused on hand features. The PR attention module captures spatial information at different scales in the channel dimension to enrich the feature space and establishes long-distance dependencies based on global spatial information. In the spatial dimension, second-order attention is used to generate feature maps with dense and rich contextual information. Because it uses two consecutive sparse self-attention calculations to approximate one dense self-attention calculation, it not only reduces memory consumption and time complexity while ensuring advanced performance, but also reduces the interference of redundant information in the global pixels.

[0266] 5) An encoder-decoder network model is used to achieve end-to-end continuous sign language recognition. While capturing the semantic information of image features in the temporal dimension, it can also map source and target sequences of unequal length. From a vertical perspective, the hidden layers contain more information from previous nodes, especially in sign language sequences with a large number of video frames, where the model exhibits better memorization and effectively avoids information loss in long sequences.

[0267] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0268] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0269] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0270] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0271] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0272] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0273] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0274] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A continuous sign language recognition method based on ResNet34 network-attention mechanism, characterized in that, include: S1: Obtain a first video data training set, which includes RGB video and depth video, and use the KFE clustering algorithm to extract keyframes from the first video data training set to obtain a second video data training set, which has labels. S2: Construct a ResNet34 network topology, fuse the PSA channel attention mechanism and the RCC spatial attention mechanism into a PR attention mechanism, and integrate the PR attention mechanism with the ResNet34 network topology to extract feature information from the second video data training set; S3: Construct a BiLSTM network topology to encode the feature information of the second video data training set, and use the LSTM-CTC end-to-end network structure topology and the labels of the second video data training set to decode the encoded second video data training set; S4: Construct an objective function to adjust the network topology parameters after the ResNet34 network topology is integrated with the PR attention mechanism, the BiLSTM network topology parameters, and the LSTM-CTC end-to-end network structure topology parameters; S5: Obtain the first video data test set and test the completed ResNet34 network topology integrated with the PR attention mechanism, the BiLSTM network topology, and the LSTM-CTC end-to-end network structure topology. In the step of fusing the PSA channel attention mechanism and the RCC spatial attention mechanism into a PR attention mechanism, the PSA channel attention mechanism is: [X0,X1,…,X S-1 ]=Split(X); F i =Conv(K i ×K i ,G i )(X i ); F=Cat([F0,F1,…,F S-1 ]); X∈R C×W×H - The first feature map obtained after the second video data training set is processed by the first four residual layers of the ResNet34 network; C, W, and H – channels, width, and height of the first feature map; Split - on the channel dimension of the first feature map X∈R C×W×H Perform the operation of dividing the data equally into S parts; X i ∈R C / S×W×H -The first feature map, after being equally divided, has a channel ratio of C / S; K i -Different convolution kernel parameters; G i - Parameters of grouped convolution; F i ∈R C / S×W×H - Multi-scale features after multi-scale feature extraction; Ca t- splices multi-scale features from different receptive fields along the channel dimension; F∈R C×W×H -Feature vector after multi-scale feature concatenation; The following formula is used to extract the weights from the feature vector obtained after concatenating the multi-scale features: g i =AvgPool(F i ); Z i =σ(W1δ(W0(g i ))); Z=Cat([Z0,Z1,…,Z S-1 ]); AvgPool(·) - represents global average pooling; σ(·) is the sigmoid activation function; δ(·) is the ReLU activation function; g i ∈R C / S×1×1 -Feature vectors that undergo global average pooling for multi-scale features; W0 and W1 are weight matrices with dimensions [C / S / r, C / S] and [C / S, C / S / r], respectively, where r represents the reduction rate; Z i - Attention weights for different parts of dimension [C / S, 1, 1]; Z-dimensional cross-dimensional channel attention feature weight map with dimension [C, 1, 1]; The attention weights are normalized using the following formula, and then tensor products are performed between the weights and the feature vectors extracted from multi-scale features: att = Softmax(Z); Y = αtt⊙F; att --- Normalized channel attention weights; The RCC spatial attention mechanism concatenates the Criss-Cross module twice to obtain rich contextual information. The Criss-Cross channel attention mechanism is as follows: Q=W Q Y; K=W K Y; V=W V Y; W Q and W K Both are weight matrices of dimension [C′, C]. W V It is a weight matrix with dimensions [C, C]. The PR attention mechanism is integrated with the ResNet34 network topology using the following formula to extract feature information from the second video data training set: Perform an Affinity operation to obtain the relationship between each pixel in a feature map of size [W, H] and its corresponding pixels in the same row and column: D = Affinity(Q, K); Affinity-Q and K are feature maps of dimension [C′, W, H]. Q u - The feature vector at position u in the spatial dimension of Q, Q u ∈R C′ ; Where Ω i，u ∈R C′ It is Ω u The i-th element, d i，u ∈D-feature Q u and Ω i，u The relevance of i to [1, ..., H+W-1] is given by the following formula: Based on the relationship D between each pixel in a feature map of size [W, H] and its corresponding pixels in the same row and column, a softmax layer is applied along the channel dimension to compute feature map A: A = softmax(D); Aggregation operation is performed on feature map A to collect contextual information Y′u: Aggregation - In the spatial dimension of V, there exists a feature vector V for each position u. u ∈R C set Φ u ∈R (H +W-1)×C It is the set of feature vectors extracted from matrix V that are in the same row and column as position u; Y′- represents the context information of the long connections captured in the vertical and horizontal directions; Repeatedly concatenate the context information of the captured long connections in the vertical and horizontal directions, as shown in the following formula: Y″ = CrissCross(Y′); Y′=CrissCross(T); Y″- is the feature vector for which global pixel information has been obtained.

2. The method according to claim 1, characterized in that, The step of extracting keyframes from the first video data training set using the KFE clustering algorithm to obtain the second video data training set includes: Obtain the initial threshold, the frame set of the first video data training set, and the cluster centroids of all clusters; The frames of the first video data training set are obtained based on the frame set of the first video data training set, and the nearest distance from the frame of the first video data training set to the cluster centroid is obtained based on the cluster centroid of all the clusters. Determine whether the nearest distance from a frame to the cluster centroid in the first video data training set is less than an initial threshold; If so, the frames of the first video data training set are assigned to the class of the nearest cluster centroid, the frames of the first video data training set are removed from the frame set of the first video data training set, and the steps of obtaining the initial threshold, the frame set of the first video data training set, and the cluster centroids of all clusters are executed. If not, the frames of the first video data training set are assigned to a new category, the frames of the first video data training set are removed from the frame set of the first video data training set, and the steps of obtaining the initial threshold, the frame set of the first video data training set, and the cluster centroids of all clusters are executed.

3. The method according to claim 1, characterized in that, The ResNet34 network topology includes an initial layer, a first residual layer, a second residual layer, a third residual layer, a fourth residual layer, and a global average pooling layer. The number of convolution kernels in the first residual layer is 64, and the number of the first residual layers is 3; The second residual layer has 128 convolution kernels and 4 layers. The number of convolution kernels in the third residual layer is 256, and the number of third residual layers is 6; The fourth residual layer has 512 convolution kernels and 3 layers. The steps for integrating the PR attention mechanism with the ResNet34 network topology include: The PR attention mechanism is introduced between the fourth residual layer and the global average pooling layer.

4. The method according to claim 1, characterized in that, The steps of acquiring the first video data test set and testing the constructed ResNet34 network topology integrated with the PR attention mechanism, the BiLSTM network topology, and the LSTM-CTC end-to-end network structure topology include: The WER value indicates the accuracy of the recognition: S′, I, and D′ represent the minimum replacement, insertion, and deletion operations, respectively. N represents the total number of words in the tag.

5. The method according to claim 1, characterized in that, The steps of acquiring the first video data test set and testing the constructed ResNet34 network topology integrated with the PR attention mechanism, the BiLSTM network topology, and the LSTM-CTC end-to-end network structure topology include: Accuracy represents the precision of the recognition process. S′, I, and D′ represent the minimum replacement, insertion, and deletion operations, respectively. N represents the total number of words in the tag.

6. A continuous sign language recognition device based on a ResNet34 network-attention mechanism, characterized in that, include: Video acquisition module: used to acquire a first video data training set, which includes RGB video and depth video, and to extract keyframes from the first video data training set using the KFE clustering algorithm to acquire a second video data training set, which has labels. Feature extraction module: used to construct the ResNet34 network topology, integrate the PSA channel attention mechanism and the RCC spatial attention mechanism into the PR attention mechanism, and integrate the PR attention mechanism with the ResNet34 network topology to extract feature information from the second video data training set; Decoding module: Used to construct a BiLSTM network topology to encode the feature information of the second video data training set, and to decode the encoded second video data training set using an LSTM-CTC end-to-end network structure topology and the labels of the second video data training set; Parameter tuning module: Used to construct objective functions to adjust the network topology parameters after the ResNet34 network topology is integrated with the PR attention mechanism, the BiLSTM network topology parameters, and the LSTM-CTC end-to-end network structure topology parameters; Test module: Used for the first video data test set to test the completed ResNet34 network topology integrated with the PR attention mechanism, the BiLSTM network topology, and the LSTM-CTC end-to-end network structure topology; The PSA channel attention mechanism is as follows: [X0,X1,…,X S-1 ]=Split(X); F i =Conv(K i ×K i ,G i )(X i ); F=Cat([F0,F1,…,F S-1 ])、 X∈R C×W×H - The first feature map obtained after the second video data training set is processed by the first four residual layers of the ResNet34 network; C, W, and H - the channels, width, and height of the first feature map; Split --- split the first feature map X∈R along the channel dimension C×W×H Perform the operation of dividing the data equally into S parts; X i ∈R C / S×W×H -The first feature map, after being equally divided, has a channel ratio of C / S; K i -Different convolution kernel parameters; G i - Parameters of grouped convolution; F i ∈R C / S×W×H - Multi-scale features after multi-scale feature extraction; Cat -- stitches together multi-scale features from different receptive fields along the channel dimension; F∈R C×W×H -Feature vector after multi-scale feature concatenation; The following formula is used to extract the weights from the feature vector obtained after concatenating the multi-scale features: g i =AvgPool(F i ); Z i =σ(W1δ(W0(g i ))); Z=Cat([Z0,Z1,…,Z S-1 ]); AvgPool(·) - represents global average pooling; σ(·) is the sigmoid activation function; δ(·) is the ReLU activation function; g i ∈R C / S×1×1 -Feature vectors that undergo global average pooling for multi-scale features; W0 and W1 are weight matrices with dimensions [C / S / r, C / S] and [C / S, C / S / r], respectively, where r represents the reduction rate; Z i - Attention weights for different parts of dimension [C / S, 1, 1]; Z-dimensional cross-dimensional channel attention feature weight map with dimension [C, 1, 1]; The attention weights are normalized using the following formula, and then tensor products are performed between the weights and the feature vectors extracted from multi-scale features: att = Softmax(Z); Y = αtt⊙F; att - Normalized channel attention weights; The RCC spatial attention mechanism concatenates the Criss-Cross module twice to obtain rich contextual information. The Criss-Cross channel attention mechanism is as follows: Q=W Q T; K=W K Y; V=W V Y; W Q and W K Both are weight matrices of dimension [C′, C]. W V It is a weight matrix with dimensions [C, C]. The PR attention mechanism is integrated with the ResNet34 network topology using the following formula to extract feature information from the second video data training set: Perform an Affinity operation to obtain the relationship between each pixel in a feature map of size [W, H] and its corresponding pixels in the same row and column: D = Affinity(Q, K); Affinity-Q and K are feature maps of dimension [C′, W, H]. Q u - The feature vector at position u in the spatial dimension of Q, Q u ∈R C′ ; Where Ω i，u ∈R C′ It is Ω u The i-th element, d i，u ∈D-feature Q u and Ω i，u The relevance of i to [1, ..., H+W-1] is given by the following formula: Based on the relationship D between each pixel in a feature map of size [W, H] and its corresponding pixels in the same row and column, a softmax layer is applied along the channel dimension to compute feature map A: A = softmax(D); Aggregation operations are performed on feature map A to collect contextual information Y′u; Aggregation - In the spatial dimension of V, there exists a feature vector V for each position u. u ∈R C set Φ u ∈R (H +W-1)×C It is the set of feature vectors extracted from matrix V that are in the same row and column as position u; T′- represents the context information of the long connections captured in the vertical and horizontal directions; Repeatedly concatenate the context information of the captured long connections in the vertical and horizontal directions, as shown in the following formula: Y″ = CrissCross(Y′); Y′=CrissCross(Y); Y″- is the feature vector for which global pixel information has been obtained.

Images