Continuous sign language recognition method based on i3d-tcp network structure

Abstract

A continuous sign language recognition method based on I3D-TCP network structure adopts an I3D model to extract sign language information features from an RGB sign language video, captures dynamic information of the sign language video by using a time attention module, then converts sign language video time sequences into a covariance pool matrix, the correlation of the content of frames and the correlation between frames, finally utilizes matrix power normalization to accelerate network training and improve the generalization ability of the network, so that the sign language can be better recognized in a complex scene.The present application seamlessly connects with the existing I3D model, produces an effective sign language video recognition architecture, can accurately recognize sign language, improves the generalization ability of the network, solves the problem of recognizing sign language in a complex scene, and can promote the communication between hearing people and hearing-impaired people.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, artificial intelligence, data mining, and deep learning. Specifically, it relates to a continuous sign language recognition method based on the I3D-TCP network structure. Technical Background

[0002] With the development of artificial intelligence, computer vision or machine vision has attracted much attention from researchers. Computer vision has many practical applications in artificial intelligence, such as image recognition, object detection / tracking, face detection, object detection, and pose recognition. Before the rise of deep learning technology, traditional computer vision relied heavily on manual feature extraction. With the development of the internet, the amount of data has grown exponentially, while GPU hardware costs have become increasingly expensive, leading to the rise of deep learning, represented by deep neural networks. To date, sign language recognition has become one of the important research areas in computer vision. It can bring convenience to people with hearing impairments and language communication difficulties, promoting their participation in social interactions and access to external information. Because deep learning has powerful representation learning capabilities, it can learn and understand knowledge without the need for manual feature extraction and rule formulation. Therefore, we utilize deep learning technology in computer vision to design a continuous sign language recognition method for the I3D-TCP network structure, which can effectively solve the problem of sign language recognition in complex scenarios. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies and solve the problem of sign language recognition in complex scenarios, the inventors provide a continuous sign language recognition method for I3D-TCP network structures. This method addresses the issue of sign language recognition in complex scenarios, enabling computers to learn and understand the meaning expressed by sign language users, thus facilitating communication between hearing and deaf individuals.

[0004] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0005] A continuous sign language recognition method for I3D-TCP network architecture includes the following steps:

[0006] 1) Define the sign language video dataset as x∈R B×T×H×W×C Where B represents the batch size, T is defined as the video frame length, i.e., the video segment size, H and W represent the length and width respectively, and C represents the number of channels; the original RGB video segment size is x∈R B ×T×H×W×C Perform matrix transpose to obtain x∈R B×C×T×H×W The input is used to extract features from the I3D model, and the final output feature map size is z∈R. B×D×T’×H’×W’Where D represents the output feature dimension, T' represents the size of the output frame, and H' and W' represent the output length of H and the output width of W, respectively;

[0007] 2) Set the output feature map size z∈R of the I3D model. B×D×T’×H’×W’ As input, the dimension of z is first compressed, z∈R (B·T)×D’×H’×W’ The calculation is performed using a time attention module, defined as:

[0008] z′=TCA(z)⊙TSA(z) (1)

[0009] Where z′ represents the output of the temporal attention module, TCA(·) represents the time-based channel attention module calculation of z, TSA(·) represents the time-based spatial attention module calculation of z, and "⊙" is defined as element-wise multiplication. Formula (2) is used to extract features from the covariance matrix of the temporal sequence of the sign language video. The formula is:

[0010] z″=TCP(z′) (2)

[0011] Where z″ represents the output of the temporal covariance pooling, and TCP(·) represents the temporal covariance pooling calculation of z′, realizing the correlation of frame content and inter-frame correlation of sign language video, and then calculating through matrix power normalization, the formula is as follows:

[0012] y = MPN(z″) (3)

[0013] Finally, the output result y is calculated by matrix power normalization. Compared with batch normalization, the advantage of matrix power normalization is that it can speed up the convergence of the neural network, making the training time shorter and more efficient. It can also improve the generalization ability of the neural network and make the model perform better on the test set.

[0014] Furthermore, during training, a clip of the sign language video was 32 frames in size, with each frame measuring 224×224 pixels. On the sign language dataset, the I3D-TCP network architecture was trained using two GPUs with a batch size of 32. PyTorch's default SGD optimizer and cross-entropy loss function were used, with a learning rate of 0.001, momentum of 0.9, and weight decay of 0.00005.

[0015] The technical approach of this invention is as follows: Deep learning has powerful representation learning capabilities and a large-scale sign language dataset. A TCP module is used to generate powerful video representations, which are seamlessly integrated with existing I3D models to create an effective sign language video recognition architecture. This architecture can accurately recognize sign language, improve the generalization ability of the network, and solve the problem of sign language recognition in complex scenarios.

[0016] The beneficial effects of this invention are mainly reflected in the following aspects: In this method, the I3D-TCP network architecture first uses the I3D model to extract features from the sign language video, then uses the temporal attention module to capture the dynamic information of the sign language video, then transforms the temporal sequence of the sign language video into a covariance pooling matrix, the correlation of the content of the frames and the correlation between frames, and finally uses matrix power normalization to accelerate the training of the network and improve the generalization ability of the network. The end-to-end deep neural network training makes the model perform better on the test set. Attached Figure Description

[0017] Figure 1 This is a structural diagram of the I3D-TCP network architecture.

[0018] Figure 2 This is a structural diagram of the I3D model.

[0019] Figure 3 This is a structural diagram of the TCP model. Detailed Implementation

[0020] Reference Figures 1-3 A continuous sign language recognition method for I3D-TCP network architecture includes the following steps:

[0021] 1) Define the sign language video dataset as x∈R B×T×H×W×C Where B represents the batch size, T is defined as the video frame length, i.e., the video segment size, H and W represent the length and width respectively, and C represents the number of channels; the original RGB video segment size is x∈R B ×T×H×W×C Perform matrix transpose to obtain x∈R B×C×T×H×W The input is used to extract features from the I3D model, and the final output feature map size is z∈R. B×D×T’×H’×W’ Where D represents the output feature dimension, T' represents the size of the output frame, and H' and W' represent the output length of H and the output width of W, respectively;

[0022] 2) Set the output feature map size z∈R of the I3D model. B×D×T’×H’×W’ As input, the dimension of z is first compressed, z∈R (B·T)×D’×H’×W’ The calculation is performed using a time attention module, defined as:

[0023] z′=TCA(z)⊙TSA(z) (1)

[0024] Where z′ represents the output of the temporal attention module, TCA(·) represents the time-based channel attention module calculation of z, TSA(·) represents the time-based spatial attention module calculation of z, and "⊙" is defined as element-wise multiplication. Formula (2) is used to extract features from the covariance matrix of the temporal sequence of the sign language video. The formula is:

[0025] z″=TCP(z′) (2)

[0026] Where z″ represents the output of the temporal covariance pool, and TCP(·) represents the temporal covariance pooling calculation of z′, realizing the correlation of frame content and inter-frame correlation of sign language video. Then, the calculation is performed through matrix power normalization, which can improve the matrix operation capability. The formula is as follows:

[0027] y = MPN(z″) (3)

[0028] Finally, the output result y is calculated by matrix power normalization. Compared with batch normalization, the advantage of matrix power normalization is that it can speed up the convergence of the neural network, making the training time shorter and more efficient. It can also improve the generalization ability of the neural network and make the model perform better on the test set.

[0029] Furthermore, during training, a clip of the sign language video was 32 frames in size, with each frame measuring 224×224 pixels. On the sign language dataset, the I3D-TCP network architecture was trained using two GPUs with a batch size of 32. PyTorch's default SGD optimizer and cross-entropy loss function were used, with a learning rate of 0.001, momentum of 0.9, and weight decay of 0.00005.

[0030] In this embodiment, the I3D-TCP network architecture first uses the I3D model to extract features from the sign language video, then uses the temporal attention module to capture the dynamic information of the sign language video, then transforms the temporal sequence of the sign language video into a covariance pooling matrix, the correlation of the content of the frames and the correlation between frames, and finally uses matrix power normalization to accelerate the training of the network and improve the generalization ability of the network. The end-to-end deep neural network training makes the model perform better on the test set.

[0031] Thus, the continuous sign language recognition method based on the I3D-TCP network structure proposed in this embodiment can perform sign language recognition in complex scenarios, enabling the network to learn and understand the meaning expressed by the sign language speaker from the sign language video.

Claims

1. A continuous sign language recognition method based on a 13D-TCP network structure, characterized in that, The method includes the following steps: Define the sign language video dataset as ,in B Indicates batch size, T Defined as the length of a video frame, i.e., the size of a video segment. H and W They represent length and width respectively. C Indicates the number of channels; [the original text is incomplete and requires further context to translate accurately.] RGB The video clip size is Perform matrix transpose to obtain The input is used to extract features from the I3D model, and the final output feature map size is [size missing]. ,in D Indicates the output feature dimension. T’ Indicates the size of the output frame. H’ and W’ They represent H Output length and W ; output width; Size of the output feature map of the I3D model As input, first put z Compressing the dimensions, The calculation is performed using a time attention module, defined as: (1)； in This represents the output of the temporal attention module. Indicates to z Perform time-based channel attention module computation. Indicates to z The spatial attention module is calculated based on time. "⊙" is defined as element-wise multiplication. Formula (2) is used to extract features from the covariance matrix of the temporal sequence of the sign language video. The formula is: (2)； in This represents the output of the time covariance pool. Indicates to Temporal covariance pooling is performed to calculate the correlation between frame content and inter-frame correlation in the sign language video. Then, matrix power normalization is used for calculation, and the final output is the predicted value for video classification. The formula is as follows: (3)； Finally, the output result is calculated using matrix power normalization. y。 2. The continuous sign language recognition based on the I3D-TCP network structure as described in claim 1, characterized in that, During training, a clip from a sign language video was 32 frames in size, with each frame measuring 224×224 pixels. On the sign language video dataset, the I3D-TCP network architecture was trained using two GPUs with a batch size of 32. PyTorch's default SGD optimizer and cross-entropy loss function were used, with a learning rate of 0.001, momentum of 0.9, and weight decay of 0.00005.

Images