A multi-view sign language recognition method based on a multi-stream 3D convolutional neural network

Abstract

The application discloses a multi-view sign language recognition method based on a multi-stream 3D convolutional neural network. The method is based on a multi-stream 3D convolutional neural network to process multi-view sign language recognition, uses a 3D convolutional neural network to extract the space-time features of sign language data, and is based on a multi-stream convolutional neural network structure to simultaneously input multi-view sign language data for training. The recognition results of each stream of the network are fused through a multi-view fusion module, so that the complementary information of different view sign language data is fully utilized, and multi-view sign language recognition is more accurate.

Description

TECHNICAL FIELD

[0001] The application relates to a multi-view sign language recognition method based on a multi-stream 3D convolutional neural network and belongs to the technical field of gesture recognition. BACKGROUND

[0002] Sign language is the main way of communication between the deaf and the hearing, and sign language recognition involves video acquisition and processing, computer vision, natural language processing and other research fields, and is a challenging and social issue. In previous research, people have drawn on the research experience in the field of action recognition, and widely used 3D convolutional neural networks in the field of sign language recognition. The 3D convolutional neural network can simultaneously learn image features and complex temporal features between adjacent frames from sign language data, so that the performance of sign language recognition is greatly improved. Although people have carried out a large amount of research in the field of sign language recognition, these researches mainly focus on single-view sign language recognition, while sign language communication in life usually involves various views.

[0003] Therefore, the application provides a multi-view sign language recognition method based on a multi-stream 3D convolutional neural network, which fuses the complementary information of sign language data of different views to improve the performance of multi-view sign language recognition. SUMMARY

[0004] The application provides a multi-view sign language recognition method based on a multi-stream 3D convolutional neural network, which uses a 3D convolutional neural network to extract the spatio-temporal features of sign language data based on a multi-stream 3D convolutional neural network to process multi-view sign language recognition; based on a multi-stream convolutional neural network structure to simultaneously input multi-view sign language data for training; and through a multi-view fusion module to fuse the recognition results of each stream of the network, so that the complementary information of sign language data of different views is fully utilized, and the multi-view sign language recognition is more accurate.

[0005] The application better extracts the spatio-temporal features of sign language data through a 3D convolutional neural network, and the multi-stream convolutional neural network model and the multi-view fusion module can effectively fuse the complementary information of sign language data of different views.

[0006] The application mainly focuses on a multi-view sign language recognition method, but the current multi-view sign language dataset is relatively scarce, and there is no dataset that can be used for multi-view sign language recognition. Therefore, the application independently collects sign language data, processes the sign language data, and prepares two multi-view sign language datasets, IWS11 and IWS30.

[0007] An Intel RealSense D435i camera was used as the acquisition device to capture RGB video of sign language data. Five cameras were used to capture sign language data from five different angles: directly in front of the subject, diagonally upwards, diagonally downwards, diagonally to the left, and diagonally to the right, obtaining sign language data from different perspectives.

[0008] After obtaining sign language data from different perspectives, the data was processed to form a multi-view original sign language dataset. First, the sign language data was extracted from the camera as RGB video. Then, the sign language data from different perspectives was aligned, and noise generated during the data acquisition process was denoised. Next, the sign language data was segmented into word-level short videos, and the frequency of each word was statistically analyzed. Finally, the top 11 and top 30 words by frequency were selected to form the IWS11 and IWS30 datasets. Both the IWS11 and IWS30 datasets contain sign language data collected from five different perspectives. The difference lies in that the IWS11 dataset contains 11 categories of sign language data, while the IWS30 dataset contains 30 categories. Additionally, the IWS11 dataset has a longer duration of sign language data, while the IWS30 dataset has a shorter duration. Therefore, for sign language recognition on the IWS11 and IWS30 datasets, different sign language data segmentation strategies are adopted before the sign language data is input into the multi-stream 3D convolutional neural network.

[0009] Sign language data is defined as a T-frame image sequence X = [x1, x2, ..., x...]. T ], x t This represents the t-th frame in an image sequence. The N class label sequences corresponding to the image sequence are represented as L = [l1, l2, ..., ln]. N ], l n This represents the nth class label corresponding to the image sequence. Sign language data from different perspectives are processed by a multi-stream 3D convolutional neural network to obtain recognition results corresponding to the respective class labels.

[0010] Sign language recognition identifies the word categories represented by people's gestures, facial expressions, and body postures in sign language data. It is defined as a function model that maps sign language data to different word categories. A 3D convolutional neural network is used as the feature extractor for each stream in a multi-stream neural network model to extract visual features F = [f1, f2, ..., f] from the input image sequence. T, ], f t' This represents the visual features extracted from the t-th frame of the image sequence. After passing through a Softmax layer, the recognition result for each stream is obtained: Q = [q1, q2, ..., q5]. iThe identification result of the i-th stream is represented, and the identification of each stream is weighted and averaged to obtain a final sign language identification result q RGB .

[0011] The multi-view sign language recognition model realizes better recognition effect on multi-view sign language data through expansion and optimization of existing sign language recognition models. Firstly, for sign language data, 2D convolutional neural network cannot extract time and space features of sign language data at the same time, therefore, 3D convolutional neural network is used as a feature extractor of the multi-view sign language recognition model, which can more accurately capture the motion information of sign language data. Secondly, for multi-view sign language data, a single stream convolutional neural network cannot effectively identify the sign language data of each view, therefore, the multi-view sign language recognition model is used to input multi-view sign language data at the same time, and the sign language data of each view is identified respectively. Finally, in order to better utilize the complementary information between multi-view sign language data, a multi-view fusion module is used to fuse the identification results of the sign language data of each view, so that the sign language data of different views can provide motion information to each other, and better multi-view sign language recognition results are achieved. The multi-view sign language recognition model proposed in the application mainly consists of three parts: a 3D convolutional neural network model, a multi-stream convolutional neural network model and a multi-view fusion module.

[0012] The 3D convolutional neural network model consists of 8 convolutional layers, 5 pooling layers, 2 fully connected layers and 1 Softmax layer. The main function of the convolutional layer is to extract the features of the input data through convolution operation, and the main function of the pooling layer is to select the features extracted in the convolutional layer through pooling operation. Therefore, by combining the convolutional layer and the pooling layer, the input data can be better abstractly represented. The main function of the fully connected layer is to convert the selected features of the pooling layer into a one-dimensional feature vector, and to effectively identify the input data. The main function of the Softmax layer is to normalize the one-dimensional feature vector output by the fully connected layer to obtain the probability value of the input data belonging to each class, so as to output the final recognition result. The 3D convolutional neural network learns the features of three dimensions (x, y, t) in the time domain and the spatial domain through three-dimensional convolution and three-dimensional pooling, and captures the motion information of the sign language data, and the specific expression is as follows:

[0013]

[0014] Wherein, f x,y,t represents a unit at the space-time position (x, y, t) on the feature map, v (x+i)(y+j)(t+k) represents an input unit at the space-time position (x+i, y+j, t+k), w i,j,k represents the weight parameter of the three-dimensional convolution kernel, b represents the bias of the feature map, and a(·) represents the activation function.

[0015] The 3D convolutional neural network takes sign language data as input and takes the recognition result of different sign language classes as output. First, the size of each video frame is cropped, and the size of all video frames is adjusted to 128x117; then, the sign language data is segmented into non-overlapping sign language segments by sliding window, and the segmented sign language segments are slightly different for sign language data of different time lengths, such as segmenting the sign language data in the IWS11 dataset into 16-frame sign language segments and segmenting the sign language data in the IWS30 dataset into 8-frame sign language segments, and inputting them into the 3D convolutional neural network; then, the 3D convolutional neural network extracts the spatiotemporal features of the sign language segments through multiple layers of three-dimensional convolution and pooling operations, and in the training process, the input sign language segments are randomly cropped to a size of 112x112 to obtain better output results; finally, the classification results of different sign language classes are obtained through the full connection layer and the Softmax layer, and the final recognition result is output.

[0016] The multi-stream convolutional neural network model extends the original single-stream convolutional neural network to a multi-stream convolutional neural network, changes the original single input of the convolutional neural network to multiple inputs, to meet the simultaneous input and training of multi-view sign language data, and prepares for the fusion of different view sign language data later.

[0017] Each view of the multi-view sign language data contains rich complementary information. In the single-stream convolutional neural network, the features of different view sign language data are learned simultaneously, so that these complementary information cannot be fully utilized. In the multi-stream convolutional neural network, each stream corresponds to a view, and the features of different view sign language data are learned respectively, and then the learned results are fused, which can fully utilize these complementary information and obtain better recognition results.

[0018] Specifically, the multi-view sign language data is simultaneously input into multiple 3D convolutional neural networks with the same structure and processed in parallel. The 3D convolutional neural networks of different streams extract the spatiotemporal features of different view sign language data through convolution layers, pooling layers and full connection layers respectively; then the extracted spatiotemporal features are input into the Softmax layer to classify the sign language data and obtain the recognition results of different streams; finally, the recognition results of different streams are simultaneously input into the multi-view fusion module for corresponding fusion.

[0019] The multi-view fusion module is mainly used to fuse the recognition results of different streams of the 3D convolutional neural network taking different view sign language data as input, and to complete the fusion of multi-view sign language data by weighted average calculation of multiple recognition results, so as to obtain the final recognition result of the multi-stream 3D convolutional neural network.

[0020] This invention considers two placement options for the multi-view fusion module, with different emphases in the fusion process. One option is to place it at the input position of the multi-stream convolutional neural network (CNN), located at the bottom layer, primarily for fusing local features of the multi-view sign language data. The other option is to place it at the output position of the CNN, located at the top layer, primarily for fusing global features of the multi-view sign language data. Analysis shows that complementary information from different viewpoints of sign language data mainly exists within the global features; therefore, placing the multi-view fusion module at the output position of the CNN achieves better fusion results.

[0021] Specifically, the multi-stream 3D convolutional neural network uses a relatively simple fusion method to fuse the recognition results of the sign language data from the five perspectives above. It saves the results of the Softmax layer for each perspective. The Softmax layer normalizes the results of the fully connected layer. The mathematical expressions for the output results and the loss function are as follows:

[0022]

[0023]

[0024] Among them, z j p represents the output of class j in the fully connected layer. j Let l represent the probability of class j in the Softmax layer. n This represents the label of the actual sample.

[0025] Weights are assigned to each viewpoint based on its accuracy on the validation set, and the weighted average of the five viewpoint accuracies is used as the final recognition result of the multi-stream 3D convolutional neural network. The specific calculation formula is as follows:

[0026] q RGB =α1q1+α2q2+α3q3+α4q4+α5q5

[0027] Where, q 1,2,3,4,5 This represents the recognition results from five perspectives, α 1,2,3,4,5 q represents the hyperparameter that is adjusted based on the accuracy on the validation set. RGB This represents the final recognition result of the multi-stream 3D convolutional neural network.

[0028] The technical advantages of this invention are as follows: 1. Addressing the current scarcity of multi-view sign language datasets in the field of sign language recognition, this invention independently collects and processes multi-view sign language data, creating two multi-view sign language datasets, IWS11 and IWS30. 2. It employs a 3D convolutional neural network as the feature extractor for the multi-view sign language recognition model, simultaneously extracting the temporal and spatial features of the sign language data to better capture the motion information of the sign language data. 3. It uses a multi-stream 3D convolutional neural network, taking sign language data from each viewpoint as input to each stream, and recognizing sign language data from different views separately. A multi-view fusion module is also added to perform a weighted average calculation of the recognition results from different viewpoints to obtain the final recognition result. Attached Figure Description

[0029] Figure 1 This is a diagram showing the camera placement.

[0030] Figure 2 This is a chart of sign language data.

[0031] Figure 3 This is a schematic diagram of an image sequence.

[0032] Figure 4 This is a diagram of a multi-view sign language recognition model based on a multi-stream 3D convolutional neural network.

[0033] Figure 5 This is a diagram of a 3D convolutional neural network structure.

[0034] Figure 6 The diagram shows the operations for 2D and 3D convolution.

[0035] Figure 7 A diagram showing the placement of the multi-view fusion module. Detailed Implementation

[0036] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0037] This invention uses 3D convolutional neural networks to better extract the spatiotemporal features of sign language data. The proposed multi-stream convolutional neural network model and multi-view fusion module can effectively fuse complementary information from sign language data from different perspectives.

[0038] This invention focuses on multi-view sign language recognition methods. However, there is a lack of multi-view sign language datasets available for use in multi-view sign language recognition. Therefore, this invention independently collects and processes sign language data to create two multi-view sign language datasets, IWS11 and IWS30.

[0039] An Intel RealSense D435i camera was used as the acquisition device to capture RGB video of sign language data. Five cameras were used to capture data from five different angles: directly in front of the subject, diagonally upwards, diagonally downwards, diagonally to the left, and diagonally to the right. Figure 1 As shown, raw sign language data from different perspectives were obtained, such as... Figure 2 As shown.

[0040] After obtaining the raw sign language data from different perspectives, the data was processed to create a multi-view sign language dataset. First, the sign language data was extracted from the camera as RGB video to obtain the raw data. Then, the sign language data from different perspectives was aligned, and noise generated during the data acquisition process was denoised. Next, the sign language data was segmented into word-level short videos, and the frequency of each word was statistically analyzed. Finally, the top 11 and top 30 most frequent words were selected to form the IWS11 and IWS30 datasets, respectively. Both the IWS11 and IWS30 datasets contain sign language data collected from five different perspectives. The difference lies in that the IWS11 dataset contains 11 categories of sign language data, while the IWS30 dataset contains 30 categories. Additionally, the IWS11 dataset has a longer duration of sign language data, while the IWS30 dataset has a shorter duration. Therefore, for sign language recognition on the IWS11 and IWS30 datasets, different sign language data segmentation strategies are adopted before the sign language data is input into the multi-stream 3D convolutional neural network.

[0041] In this invention, sign language data is defined as a T-frame image sequence X = [x1, x2, ..., x...]. T ], x t This represents the t-th frame in an image sequence, such as... Figure 3 As shown. The sequence of N class labels corresponding to the image sequence is represented as L=[l1,l2,...,l N ], l n This represents the nth class label corresponding to the image sequence. Sign language data from different perspectives are processed by a multi-stream 3D convolutional neural network to obtain recognition results corresponding to the respective class labels.

[0042] Sign language recognition identifies the word categories represented by people's gestures, facial expressions, and body postures in sign language data. Sign language recognition can be defined as a function model that maps sign language data to different word categories. In this invention, a 3D convolutional neural network is used as the feature extractor for each stream in a multi-stream neural network model to extract visual features F = [f1, f2, ..., f...] from the input image sequence. T, ], f t'This represents the visual features extracted from the t-th frame of the image sequence. After passing through a Softmax layer, the recognition result for each stream is obtained: Q = [q1, q2, ..., q5]. i Let q represent the recognition result of the i-th stream. The final sign language recognition result q is obtained by weighted averaging the recognition results of each stream. RGB .

[0043] The multi-view sign language recognition model achieves better recognition results for multi-view sign language data by extending and optimizing existing sign language recognition models. First, for sign language data, 2D convolutional neural networks cannot simultaneously extract the temporal and spatial features of the data. Therefore, a 3D convolutional neural network is used as the feature extractor for the multi-view sign language recognition model to more accurately capture the motion information of the sign language data. Second, for multi-view sign language data, single-stream convolutional neural networks cannot effectively recognize sign language data from different perspectives. Therefore, a multi-view sign language recognition model is adopted, simultaneously inputting multi-view sign language data and recognizing the sign language data from each perspective separately. Finally, to better utilize the complementary information between multi-view sign language data, a multi-view fusion module is used to fuse the recognition results of sign language data from different perspectives, allowing the sign language data from different perspectives to provide motion information to each other, achieving better multi-view sign language recognition results. The multi-view sign language recognition model proposed in this invention mainly consists of three parts: a 3D convolutional neural network model, a multi-stream convolutional neural network model, and a multi-view fusion module. The multi-stream 3D convolutional neural network model is as follows: Figure 4 As shown.

[0044] The 3D convolutional neural network model consists of 8 convolutional layers, 5 pooling layers, 2 fully connected layers, and 1 softmax layer. The structure of the 3D convolutional neural network is as follows: Figure 5 As shown. The main function of convolutional layers is to extract features from the input data through convolutional operations, while the main function of pooling layers is to select features extracted by the convolutional layers through pooling operations. Therefore, by combining convolutional and pooling layers, the input data can be better abstracted and represented. The main function of fully connected layers is to transform the features selected by the pooling layers into a one-dimensional feature vector and to effectively identify the input data. The main function of the softmax layer is to normalize the one-dimensional feature vector output by the fully connected layers, obtain the probability value of the input data belonging to each class, and thus output the final recognition result. 3D convolutional neural networks can effectively extract the spatiotemporal features of sign language data, laying the foundation for producing good recognition results.

[0045] For sign language data, capturing motion information between consecutive video frames is crucial for achieving good recognition results. Compared to 2D convolutional neural networks, which can only learn features in the spatial domain (x, y), 3D convolutional networks can learn features in both the temporal and spatial domains (x, y, t) simultaneously through 3D convolution and 3D pooling. Figure 6 As shown, to better capture the motion information of sign language data, the specific mathematical expression is as follows:

[0046]

[0047] Among them, f x,y,t v represents the unit at the spatiotemporal location (x, y, t) on the feature map. (x+i)(y+j)(t+k) w represents the input unit at the spatiotemporal location (x+i, y+j, t+k). i,j,k denoted by , b represents the weight parameters of the 3D convolution kernel, b represents the bias of the feature map, and a(·) represents the activation function.

[0048] The 3D convolutional neural network (CNN) takes sign language data as input and outputs the recognition results of different sign language class labels. First, the size of each video frame is cropped, with all frames resized to 128×117. Then, a sliding window is used to segment the sign language data into non-overlapping segments. The segmentation varies slightly depending on the duration of the data; for example, sign language data from the IWS11 dataset is segmented into 16 frames, and sign language data from the IWS30 dataset is segmented into 8 frames. These segments are then input into the 3D CNN. Next, the 3D CNN extracts the spatiotemporal features of the sign language segments through multiple layers of 3D convolution and pooling operations. During training, to obtain better output results, the input sign language segments are randomly cropped to 112×112 pixels. Finally, after passing through fully connected layers and a softmax layer, the classification results of different sign language class labels are obtained, and the final recognition result is output.

[0049] Multi-stream convolutional neural network models extend the original single-stream convolutional neural network into a multi-stream convolutional neural network, which can change the original single input of the convolutional neural network into multiple inputs, so as to meet the simultaneous input and training of sign language data from multiple perspectives, and prepare for the fusion of sign language data from different perspectives in the future.

[0050] Each perspective in multi-view sign language data contains rich complementary information. In single-stream convolutional neural network models, features from different perspectives are learned simultaneously, preventing the full utilization of this complementary information. However, in multi-stream convolutional neural network models, each stream corresponds to one perspective, learning features from different perspectives separately, and then fusing the learned results. This allows for the full utilization of this complementary information, resulting in better recognition outcomes.

[0051] Specifically, multi-view sign language data is simultaneously input into multiple identical 3D convolutional neural networks for parallel processing. The 3D convolutional neural networks of different streams extract spatiotemporal features from the sign language data from different perspectives through convolutional layers, pooling layers, and fully connected layers, respectively. These extracted spatiotemporal features are then input into a Softmax layer to classify the sign language data, yielding recognition results for different streams. Finally, the recognition results from different streams are simultaneously input into a multi-view fusion module for fusion.

[0052] The multi-view fusion module is mainly used to fuse the recognition results of different streams of 3D convolutional neural network with sign language data from different perspectives as input. By performing a weighted average calculation on multiple recognition results, the fusion of multi-view sign language data is completed, thereby obtaining the final recognition result of the multi-stream 3D convolutional neural network model.

[0053] This invention considers two placement positions for the multi-view fusion module, with different emphases in the fusion process. One position is at the input location of the multi-stream convolutional neural network, such as... Figure 7 As shown in the left image, this position is at the bottom layer of a multi-stream convolutional neural network, primarily used for fusing local features from multi-view sign language data. Another approach is to place it at the output position of the multi-stream convolutional neural network, such as... Figure 7 As shown in the right figure, this position is at the top layer of the multi-stream convolutional neural network, mainly used to fuse global features of multi-view sign language data. Analysis reveals that complementary information from different viewpoints of sign language data primarily resides within the global features. Therefore, placing the multi-view fusion module at the output position of the multi-stream convolutional neural network yields better fusion results.

[0054] Specifically, the multi-stream 3D convolutional neural network model uses a relatively simple fusion method to fuse the recognition results of the sign language data from the five perspectives above. It saves the results of the Softmax layer for each perspective. The Softmax layer normalizes the results of the fully connected layer. The mathematical expressions for the output results and loss function are as follows:

[0055]

[0056]

[0057] Among them, z jp represents the output of class j in the fully connected layer. j Let l represent the probability of class j in the Softmax layer. n This represents the label of the actual sample.

[0058] Weights are assigned to each viewpoint based on its accuracy on the validation set, and the weighted average of the five viewpoint accuracies is used as the final recognition result of the multi-stream 3D convolutional neural network. The specific calculation formula is as follows:

[0059] q RGB =α1q1+α2q2+α3q3+α4q4+α5q5

[0060] Where, q 1,2,3,4,5 This represents the recognition results from five perspectives, α 1,2,3,4,5 q represents the hyperparameter that is adjusted based on the accuracy on the validation set. RGB This represents the final recognition result of the multi-stream 3D convolutional neural network.

[0061] Experiments and Analysis

[0062] The multi-stream 3D convolutional neural network model achieved good results on both the IWS11 and IWS30 datasets. Compared with previous 3D convolutional neural network models, the multi-stream 3D convolutional neural network model proposed in this invention achieves better performance. Related ablation experiments were also conducted, and the experiments ultimately demonstrated that the multi-view sign language recognition method based on multi-stream 3D convolutional neural networks proposed in this invention can achieve optimal performance in multi-view sign language recognition.

Claims

1. A multi-view sign language recognition method based on a multi-stream 3D convolutional neural network, characterized in that, Using cameras as acquisition devices, RGB video of sign language data was acquired; using 5 cameras, sign language data was acquired from 5 different angles: directly in front of the subject, diagonally in front of the subject, diagonally in front of the subject from the subject from the subject from the subject's front of the subject, diagonally in front of the subject from the subject's front of the subject, and diagonally in front of the subject's front of the subject from the subject's front of the subject, and sign language data from different perspectives was acquired; after acquiring sign language data from different perspectives, sign language data was processed to form a multi-perspective raw sign language dataset. First, sign language data is extracted from the camera as RGB video. Then, the sign language data from different perspectives is aligned, and noise generated during data acquisition is removed. Next, the sign language data is segmented into word-level short videos, and the frequency of each word is statistically analyzed. Finally, the top 11 and top 30 words by frequency are selected to form the IWS11 and IWS30 datasets, respectively. Both the IWS11 and IWS30 datasets contain sign language data collected from five different perspectives. The IWS11 dataset contains 11 categories of sign language data, while the IWS30 dataset contains 30 categories. Additionally, the IWS11 dataset has a longer duration of sign language data, while the IWS30 dataset has a shorter duration. For sign language recognition on the IWS11 and IWS30 datasets, different sign language data segmentation strategies are adopted before the sign language data is input into the multi-stream 3D convolutional neural network; The model for implementing this multi-view sign language recognition method consists of three parts: a 3D convolutional neural network model, a multi-stream convolutional neural network model, and a multi-view fusion module. The 3D convolutional neural network model consists of 8 convolutional layers, 5 pooling layers, 2 fully connected layers, and 1 softmax layer. The convolutional layers extract features from the input data through convolutional operations, while the pooling layers select features extracted from the convolutional layers through pooling operations. The fully connected layers transform the selected features from the pooling layers into a one-dimensional feature vector and effectively identify the input data. The softmax layer normalizes the one-dimensional feature vector output from the fully connected layers, obtaining the probability value of the input data belonging to each class, thus outputting the final recognition result. The 3D convolutional neural network learns features in both the time and spatial domains (x, y, t) simultaneously through three-dimensional convolution and three-dimensional pooling, capturing the motion information of sign language data. The specific expression is as follows: Among them, f x,y,t v represents the unit at the spatiotemporal location (x, y, t) on the feature map. (x+i)(y+j)(t+k) w represents the input unit at the spatiotemporal location (x+i, y+j, t+k). i,j,k represents the weight parameters of the 3D convolution kernel, b represents the bias of the feature map, and a(·) represents the activation function; The 3D convolutional neural network (CNN) takes sign language data as input and outputs the recognition results of different sign language class labels. First, the size of each video frame is cropped, and all video frames are resized to 128×117. Then, a sliding window is used to segment the sign language data into non-overlapping sign language segments. For sign language data of different durations, different segments are created. The sign language data from the IWS11 dataset is segmented into 16-frame segments, and the sign language data from the IWS30 dataset is segmented into 8-frame segments, which are then input into the 3D CNN. Next, the 3D CNN extracts the spatiotemporal features of the sign language segments through multi-layer 3D convolution and pooling operations. During training, to obtain better output results, the input sign language segments are randomly cropped to 112×112 pixels. Finally, after passing through fully connected layers and softmax layers, the classification results of different sign language class labels are obtained, and the final recognition result is output. The multi-stream convolutional neural network model extends the single-stream convolutional neural network into a multi-stream convolutional neural network, turning the original single input of the convolutional neural network into multiple inputs, so as to meet the simultaneous input and training of sign language data from multiple perspectives, and to prepare for the fusion of sign language data from different perspectives in the future. Multi-view sign language data are simultaneously input into multiple 3D convolutional neural networks with identical structures for parallel processing. Different streams of 3D convolutional neural networks extract spatiotemporal features of the sign language data from different perspectives through convolutional layers, pooling layers, and fully connected layers. The extracted spatiotemporal features are then input into a Softmax layer to classify the sign language data, yielding recognition results for different streams. Finally, the recognition results from different streams are simultaneously input into a multi-view fusion module for fusion. In order to fuse the recognition results of different streams of 3D convolutional neural network with sign language data from different perspectives as input, the multi-view fusion module performs a weighted average calculation on multiple recognition results to complete the fusion of multi-view sign language data, thereby obtaining the final recognition result of the multi-stream 3D convolutional neural network. Consider two placement options for the multi-view fusion module, each with a different focus: one is placed at the input of the multi-stream convolutional neural network (CNN), at its bottom layer, to fuse local features of the multi-view sign language data; the other is placed at the output of the CNN, at its top layer, to fuse global features of the multi-view sign language data. Since complementary information from different viewpoints exists within the global features, placing the multi-view fusion module at the output of the CNN yields better fusion results. The recognition results of a multi-stream 3D convolutional neural network fusing sign language data from five perspectives are stored, and the results of the Softmax layer for each perspective are saved. The Softmax layer normalizes the results of the fully connected layer. The mathematical expressions for the output results and the loss function are as follows: Among them, z j p represents the output of class j in the fully connected layer. j Let l represent the probability of class j in the Softmax layer. n Indicates the labels of real samples; Weights are assigned to each viewpoint based on its accuracy on the validation set, and the weighted average of the five viewpoint accuracies is used as the final recognition result of the multi-stream 3D convolutional neural network. The specific calculation formula is as follows: q RGB =&α1q1+α2q2+α3q3+α4q4+α5q5 Where, q 1,2,3,4,5 This represents the recognition results from five perspectives, α 1,2,3,4,5 q represents the hyperparameter that is adjusted based on the accuracy on the validation set. RGB This represents the final recognition result of the multi-stream 3D convolutional neural network.

2. The multi-view sign language recognition method based on a multi-stream 3D convolutional neural network according to claim 1, characterized in that, Sign language data is defined as a T-frame image sequence X = [x1, x2, ..., x...]. T ], x t This represents the t-th frame in an image sequence; the N class label sequences corresponding to the image sequence are represented as L = [l1, l2, ..., l...]. N ], l n This represents the nth class label corresponding to the image sequence; sign language data from different perspectives are learned by a multi-stream 3D convolutional neural network to obtain recognition results corresponding to the respective class labels; Sign language recognition identifies the word categories represented by people's gestures, facial expressions, and body postures in sign language data; Sign language recognition is defined as a function model that maps sign language data to different word categories; a 3D convolutional neural network is used as the feature extractor for each stream in the multi-stream neural network model to extract visual features F = [f1, f2, ..., f] from the input image sequence. T, ], f t' This represents the visual features extracted from the t-th frame of the image sequence; after... The Softmax layer obtains the recognition result Q = [q1, q2, ..., q5] for each stream. i Let q represent the recognition result of the i-th stream. The final sign language recognition result q is obtained by weighted averaging the recognition results of each stream. RGB .

Images