Human body behavior recognition method based on multi-modal data and deep learning

Abstract

The invention discloses a human body behavior recognition method based on multi-modal data and deep learning, and relates to the technical field of computer vision, and the method comprises the steps: carrying out the preprocessing of skeleton data and RGB data; carrying out feature extraction by adopting a Resnet3D 18 network RGB (Red, Green and Blue) video stream; inputting the proposed RGB features and skeleton features into a feature fusion network to obtain fusion features; the feature fusion module comprises a feature association module, a feature weighting module and a feature confidence module; the fusion features realize division of different types of actions through a softmax classifier; interference of noise and background on classification performance is reduced through preprocessing, a Resnet3D 18 network and an adaptive graph convolutional network double-flow feature extraction network are adopted to effectively integrate and extract feature information, and a fusion network performs dynamic confidence weighting on two heterogeneous features, so that a model can dynamically adjust importance of each modal according to actual conditions.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method for human behavior recognition based on multimodal data and deep learning. Background Art

[0002] Human behavior recognition is an important research direction in the field of computer vision. Its purpose is to simulate human visual perception functions in order to accurately identify the category of behavior in different environments. The current mainstream human behavior recognition methods usually use deep learning methods. The human behavior recognition methods based on deep learning use a trainable feature extraction model to automatically learn behavior representations from videos in an end-to-end manner to complete classification. At present, there are many network structures for human behavior recognition methods based on deep learning, such as those based on convolutional neural networks and those based on graph convolutional neural networks.

[0003] Human behavior recognition based on convolutional neural networks can extract the appearance features of human behavior very well, but it can never make good use of the motion information in human data, and the noise in RGB data makes it impossible to stimulate the full performance of the convolutional network. There are still some complex background interference and overfitting problems. Human behavior recognition based on skeletons of graph convolutional neural networks can capture the dependency of joints and extend it to structural connections to obtain high-order dependencies, but the performance of skeletons is often unsatisfactory in some categories with subtle hand movements.

[0004] In order to make full use of the complementarity of RGB data and skeleton data, some recent works have studied deep learning architectures to fuse RGB and skeleton data in human action recognition. Since different modal data have their own advantages and disadvantages in terms of main features such as appearance, posture, geometry, lighting and perspective, their advantages and disadvantages can complement each other. For example, RGB modal data is sensitive to environmental changes and performs poorly in complex scenes, while the skeleton modality is robust to background changes, has a simple structure and rich information. Therefore, the fusion of RGB data and skeleton data is a relatively effective strategy. Multimodal data can provide more opportunities for action recognition, but it also brings new problems and deficiencies. For example, Moon et al. [3] The work of used 2D convolutional neural network and temporal convolution to model RGB and 2D skeleton modalities, but the fusion of 2D convolutional neural network and temporal convolution cannot effectively integrate the 3D information of human motion and the temporal information of 2D skeleton; Das et al. [1] An end-to-end spatiotemporal attention network was designed with RGB data and 3D skeleton data as input, but the network could not achieve the weight distribution of the two different modalities, and the data-guided performance could not be fully utilized; Xu et al. [9]The designed human action recognition model based on multimodal data extracts features through a deep learning model, but it relies too much on RGB features, resulting in a large amount of RGB background noise in the model features.

[0005] Based on this, the purpose of the present invention is to provide a human behavior recognition method based on multimodal data and deep learning to solve the problems of background noise and integration of different extracted network features. Summary of the invention

[0006] In order to make up for the shortcomings of the prior art problems, the present invention provides a human behavior recognition method based on multimodal data and deep learning. Preprocessing is used to reduce the interference of noise and background on classification performance. Resnet3D 18 network and adaptive graph convolution network dual-stream feature extraction network are adopted to effectively integrate and extract feature information. The fusion network dynamically weights the confidence of two heterogeneous features, so that the model can dynamically adjust the importance of each modality according to the actual situation, and can more flexibly adapt to the needs of different tasks, so that the fused features are more representative and discriminative.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A human behavior recognition method based on multimodal data and deep learning comprises the following steps:

[0009] (1) Preprocessing skeleton and RGB data;

[0010] (2) Resnet3D 18 network RGB video stream is used for feature extraction; an adaptive graph convolution block is added to the AGCN network. The output feature dimension of the improved AGCN network is the same as that of the Resnet3D 18 network. The improved AGCN network is used to extract features from the skeleton data;

[0011] (3) Input the proposed RGB features and skeleton features into the feature fusion network to obtain fusion features; the feature fusion module includes a feature association module, a feature weighting module and a feature confidence module;

[0012] (4) The fused features are used to classify actions of different categories through the softmax classifier.

[0013] In the present invention, the preprocessing of RGB data includes:

[0014] (a) Crop the video V at equal time intervals and retain only K frames related to the human body to obtain a set of video frames;

[0015] (b) The Crop(·) method is used to crop the task center activity area of ​​all image sequences to ensure that all samples of the dataset input to the network focus on the action of the person as much as possible and reduce the interference of useless background;

[0016] (c) All image sequences in the dataset are processed by the Resize(·) method to adjust the size of the image sequence to the size required by the deep learning network;

[0017] (d) After resizing, the data needs to be normalized;

[0018] (e) New sample images are randomly generated through data augmentation methods, which not only enhances the generalization ability of the model but also avoids overfitting and improves the robustness of the model.

[0019] In the present invention, the preprocessing of the skeleton data is specifically as follows:

[0020] Skeleton DataX S keleton∈R B × C ×T×N× M Dimensions are adjusted.

[0021]

[0022] Where B represents the number of batches, C represents the number of channels, T represents the frame in the skeleton data sequence, N represents the number of joints, and M represents the skeleton in the frame; after adjustment, the B and M dimensions are merged into a new dimension BM, where BM = B × M.

[0023] In the present invention, the feature association associates each feature with all feature data. When calculating the output of each pixel position, the correlation is calculated not only with the neighborhood, but also with all positions in the image. The calculated response value is then used to re-assign the original feature weights:

[0024] For the current position feature vector x i ∈X Fusion The operations are:

[0025]

[0026] Among them, x j ∈X Fusion For all possible other position feature vectors, y i represents the output feature, y i and x i The feature size is the same as that of g(·), and for each position x j The features of x are transformed into a 1×1 convolution operation, and f(·) is used to calculate xj and x i Embedded Gaussian function of the correlation:

[0027]

[0028] Here θ(x i )=W θ x i β(x j )=W β x i , θ(·) and β(·) are 1×1 convolution operations, W θ and W β is the weight matrix. In order to include feature similarity information after the non-local mean operation without destroying the initial feature form and information, a residual connection is inserted in the non-local mean operation:

[0029] z i =W z y i +x i (11)

[0030] Among them, "+x i " represents residual connection, W z Its weight matrix, x i and i Given in formula (6), we get the characteristic z i , the x corresponding to all possible positions i i After processing, feature stacking is performed to finally obtain Z Fusion ∈R B × M .

[0031] In the present invention, the feature weighting module processing process is as follows:

[0032] First, by fusion feature Z Fusion Regularization is performed by calculating the sum of squares:

[0033]

[0034] Then, Z Fusion Put in the fully connected layer and the ReLU layer to obtain a more expressive new fusion feature:

[0035] Z Fusion =ReLU(FC(W f Z Fusion ))∈R B×C (14)

[0036] Where FC(·) represents the fully connected layer, W f As its weight matrix, then ZFusion Use one-dimensional convolution to calculate the attention weight F Fusion :

[0037] F Fusion =σ(Conv1D(W C Z Fusion ))∈R B×C (15)

[0038] where σ(·) represents the sigmoid activation function, Conv1D(·) represents a one-dimensional convolution; finally, the attention weight F Fusion Linearly multiply the two input feature vectors to output the attention-weighted feature map Y RGB and Y Skeleton :

[0039]

[0040] In the present invention, the feature confidence is determined by taking a certain modality feature as the decision modality. The reliability of the modality data will directly affect its contribution to the feature integration and the degree of use of the other modality information:

[0041] The skeleton mode is used as the decision mode, and the input of the feature confidence module is the output Y of the feature weighting module. Skeleton , first through a fully connected layer, and then through a sigmoid activation function, to obtain the gating matrix Y' Skeleton , so that each element is located in [0,1], as shown in formula (15).

[0042] Y' Skeleton =σ(Fc(W·Y Skeleton ))∈R B×C (18)

[0043] To be used for Y Skeleton and Y RGB Perform gate operation: Y Skeleton ⊙Y′ Skeleton and Y RGB ⊙(1-Y′ Skeleton )∈R B×C , where ⊙ represents element-wise multiplication; finally, the gated features of RGB and skeleton are simply combined by element-wise addition to produce the comprehensive action feature, as shown below:

[0044] Y Fusion =Y Skeleton ⊙Y′ Skeleton +Y RGB ⊙(1-Y′ Skeleton )∈R B×C (19)

[0045] In order to make the feature confidence module play a role in deciding which modality to use and how much modality information to use in the entire fusion network, a loss function L is set here gate To train the gating module to enforce the priority of learning decision modes;

[0046] L gate = -log(1-Y' Skeleton ) (20);

[0047] Taking RGB mode as the decision mode, the input of the feature confidence module is Y RGB , first through a fully connected layer, and then through a sigmoid activation function, to obtain the gating matrix Y' RGB As shown in formula (21).

[0048] Y' RGB =σ(Fc(W·Y RGB ))∈R B×C (twenty one)

[0049] To be used for Y Skeleton and Y RGB Perform gate operation: Y RGB ⊙Y′ RGB and Y Skeleton ⊙*(1-Y′ RGB ); Finally, the RGB and skeleton gated features are simply combined by element-wise addition to produce the comprehensive action feature as follows:

[0050] Y Fusion =Y RGB ⊙Y′ RGB +Y skeleton ⊙(1-Y′ RGB ) (twenty two)

[0051] Set the loss function L gate To train the gating module, we enforce the learning priority of the decision mode.

[0052] L gate = -log(1-Y' RGB ) (twenty three).

[0053] In the present invention, a joint loss function is used to comprehensively optimize the entire fusion network framework.

[0054] L total =L fushion +λL gate (twenty four)

[0055] Where L fushion is the cross entropy loss function, λ represents the balance factor, and its initial value is set to 1.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] (1) The present invention preprocesses RGB data and skeleton data according to the differential expression of different data types, and focuses on the human targets in the data through preprocessing, reduces the interference of noise and background on classification performance, reduces computing memory consumption, and improves the efficiency and practicality of the algorithm. It can also improve the accuracy of classification results, and enhance the generalization ability of the model and its performance in complex scenes.

[0058] (2) The present invention adopts Resnet3D 18 network and adaptive graph convolutional network dual-stream feature extraction network to solve the problem that the feature extraction of single modality has achieved a relatively good extraction level, but there are still deficiencies in the feature extraction of multi-modality and the problem that the features of different modalities cannot be extracted well. The present invention realizes the use of heterogeneous networks to extract the features of different modal data separately, and improves the fit of the fusion of features of different modal data.

[0059] (3) The present invention designs a fusion network for two features, and adopts feature association to eliminate the problem of lack of global information due to the limitation of the information receptive field size output by the heterogeneous network, which is conducive to better construction of long-range dependency relationships; since the feature reliability of different heterogeneous features at the same position is not the same, the two heterogeneous features are dynamically confidence weighted, and the weight of each modal feature in the decision-making process is dynamically adjusted according to the reliability of each modal feature in the current situation, so that the model can dynamically adjust the importance of each modality according to the actual situation, and can more flexibly adapt to the needs of different tasks, so that the fused features are more representative and discriminative; better realize the complementarity of RGB features and skeleton data, make up for the problem of weak complementarity of multimodal features in previous methods, obtain more expressive fusion features to achieve accurate classification of different categories of actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a schematic diagram of the Resnet3D 18 network structure of the present invention.

[0061] Figure 2 This is a schematic diagram of the AGCN network structure of the present invention.

[0062] Figure 3 This is a framework diagram of the feature fusion module of the present invention.

[0063] Figure 4 This is a schematic diagram of the characteristic association module of the present invention.

[0064] Figure 5 This is a schematic diagram of the feature weighting module of the present invention.

[0065] Figure 6This is a schematic diagram of the feature confidence module of the present invention.

[0066] Figure 7 This is a t-SNE visualization diagram based on single modality and fusion modality of the present invention. DETAILED DESCRIPTION

[0067] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0068] like Figure 1-7 As shown, the present invention proposes a method for human behavior recognition based on multimodal data and deep learning, and the specific technical solution is as follows:

[0069] (1) Preprocessing technology in multimodal human motion data

[0070] In the multimodal data human behavior recognition and classification task, the presence of noise and background in the original data will affect the classification accuracy, increase the computational burden and increase the risk of overfitting. To solve this problem, a series of preprocessing operations are performed on the two types of human data, and the following processing techniques are used for RGB and skeleton data:

[0071] (a) Crop the video V at equal time intervals and retain only K frames related to the human body. The obtained video frame set is represented as X RGB ={X1,X2,...,X K}, the total number of frames of video V is T, and the default frame extraction interval is Frame, get an image sequence with a height of H, a width of W, and a number of channels of C Where B is the number of batches input into the network each time.

[0072] (b) In order to ensure that all samples of the data set input to the network focus on the action of the person as much as possible and reduce the interference of useless background, the Crop(·) method is used to crop the task center activity area of ​​all image sequences. The size of the cropped data image is 1000×1000. The processing method is shown in formula (1):

[0073]

[0074] (c) All image sequences in the data set are processed by the Resize(·) method to adjust the size of the image sequence to the size required by the deep learning network. The size of the sample image data after processing is 112×112. As shown in formula (2):

[0075]

[0076] (d) In order to effectively improve the overall training process speed, accelerate the convergence of the model and improve the stability of the model, the data needs to be normalized after the size is adjusted. The maximum and minimum normalization is used for normalization, and the data with the same size after processing is normalized. Perform linear transformation, let X min and X max are the minimum and maximum values ​​of the attribute respectively. The original value of is mapped to a value in the interval [0,1] through maximum and minimum normalization, as shown in formula (3).

[0077]

[0078] (e) In order to alleviate the overfitting of the data in the RGB training set, new sample images are randomly generated through data augmentation. The specific operation is to The training set of image data is randomly rotated 90°, 180°, 270° counterclockwise, flipped horizontally or vertically, scaled, and other random operations are performed to enhance the generalization ability of the model while avoiding overfitting and improving the robustness of the model.

[0079] (f) In order to obtain better skeleton features, the skeleton data X S keleton∈R B × C ×T×N× M , where B represents the number of batches, C represents the number of channels, T represents the frames in the skeleton data sequence, N represents the number of joints, and M represents the skeleton in the frame.

[0080] The skeleton data dimension is adjusted, and the B and M dimensions are merged into a new dimension BM, where BM = B × M.

[0081]

[0082] (2) Feature extraction technology of heterogeneous networks

[0083] (2.1) For RGB video stream, the Resnet3D 18 network framework is as follows Figure 1 (a) As shown. The first three layers of the network have 64 output channels, the middle two layers have 128 output channels, the last two layers have 256 output channels, and the number of channels in the last two layers is 512. Each layer of the Resnet3D 18 network contains a residual structure. The Resnet3D 18 network uses Basic Block as a residual block. The Basic Block structure is as follows Figure 1As shown in (b), the Basic Block consists of two 3×3×3 convolutional layers, each of which is followed by a batch normalization (BN) layer and a ReLU activation function layer. Input feature vector of Basic Block The output features are:

[0084]

[0085] Where f(·) is a residual module, which is the preprocessed data. The final output obtained by inputting continuous Basic Block residual blocks is: The features of , where C represents the number of RGB channels, and the output is a one-dimensional vector of 512 features.

[0086] (2.2) Adaptive graph convolutional network (AGCN) is used for skeleton data. The network model is as follows: Figure 2 As shown in (a), the number of channels of the network is 64 for the first three layers, 128 for the middle three layers, and 256 for the last three layers. In order to ensure that the skeleton data features are consistent with the RGB features, an adaptive graph convolution block with a channel number of 512 is added. AGCN is a classic ST-GCN-based model, which replaces the original spatial graph convolution with an adaptive graph convolution block. The adaptive convolution block is as follows Figure 2 As shown in (b), each adaptive graph convolution block is composed of adaptive convolution layers Convs, BN layers, Relu layers and Dropout. The adaptive convolution layers are as follows: Figure 2 As shown in (c), the adaptive convolution layer is a data-driven network layer that combines the parameterized and similar adjacency matrix with the original adjacency matrix and trains the network together to update the parameters. The feature vector of the input network is The output network features are:

[0087]

[0088] The AGCN network adopts the space division strategy in ST-GCN, dividing the joint K into three parts: root node, centripetal node and centrifugal node. The three parts are updated with different parameters during the training process and have different importance. Finally, they are superimposed as the output network features. Each part of the parameter update contains three adjacency matrices A k , B k and C k , A k The adjacency matrix representing the physical information of the human body, where the weights are not updated; B k It is also an N×N matrix whose initial values ​​are all zero, and its elements are optimized together with other parameters during the training process; C kis a data correlation graph, which learns a unique data correlation graph for each sample. Similar to the Resnet 3D network, Enter the AGCN network and get After Resnet3D 18 and AGCN networks and dimension transformation, the RGB features and skeleton features have the same dimension.

[0089] (3) Feature Fusion

[0090] Feature fusion is the RGB features processed by the feature extraction module and skeleton features As input. First, the feature maps of two different modalities are used to generate new global correlation features; second, the global correlation features are used to calculate the attention feature weights, and the attention feature weights are further used to weight the original features; the modulated features are then dynamically assigned the confidence of the two features through a gating mechanism; finally, the two features after confidence assignment are added together for classification for action recognition. The feature fusion module has three main components, such as Figure 3 As shown in the figure: feature association, feature weighting and feature confidence. Next, we will introduce them separately.

[0091] In order to integrate information from different sources into one feature vector, reduce the computational complexity of the model training process and retain the information contained in the two different modal features as much as possible, the different modal features are first combined by splicing.

[0092]

[0093] Concatenation(·) is a feature concatenation operation. They represent the RGB features and skeleton features processed by the feature extraction module, respectively, where M = 2×C.

[0094] (3.1) Feature association

[0095] The feature association module associates each feature with all feature data. When calculating the output of each pixel position, it not only calculates with the neighborhood, but also calculates the correlation with all positions in the image. Then, the calculated response value is used to re-assign the feature weights of the original feature, such as Figure 4 shown.

[0096] For the current position feature vector x i ∈X Fusion The operations are:

[0097]

[0098] Among them, x j ∈XFusion For all possible other position feature vectors, y i represents the output feature, y i and x i The feature size is the same as that of g(·), and for each position x j The features of x are transformed into a 1×1 convolution operation, and f(·) is used to calculate x j and x i Embedded Gaussian function of the correlation:

[0099]

[0100] Here θ(x i )=W θ x i β(x j )=W β x i , θ(·) and β(·) are 1×1 convolution operations, W θ and W β is the weight matrix. In order to include feature similarity information after the non-local mean operation without destroying the initial feature form and information, a residual connection is inserted in the non-local mean operation:

[0101] z i =W z y i +x i (11)

[0102] Among them, "+x i " represents residual connection, W z Its weight matrix, x i and i Given in formula (6), we get the characteristic z i , the x corresponding to all possible positions i i After processing, feature stacking is performed to finally obtain Z Fusion ∈R B×M .

[0103] (3.2) Feature weighting

[0104] In order to eliminate the different dimensions and ranges between the two different features, and to speed up the model convergence and improve the model stability, the fusion feature Z is first Fusion Normalization is performed via L2 regularization:

[0105]

[0106] Then Z Fusion Put in the fully connected layer and the ReLU layer to obtain a more expressive new fusion feature:

[0107] Z Fusion =ReLU(FC(W f Z Fusion ))∈R B×C (14)

[0108] Where FC(·) represents the fully connected layer, W f As its weight matrix, then Z Fusion Use one-dimensional convolution to calculate the attention weight F Fusion :

[0109] F Fusion =σ(Conv1D(W C Z Fusion ))∈R B×C (15)

[0110] where σ(·) represents the sigmoid activation function and Conv1D(·) represents a one-dimensional convolution. Finally, the attention weight F Fusion Linearly multiply the two input feature vectors to output the attention-weighted feature map Y RGB and Y Skeleton , the network structure is as follows Figure 5 shown.

[0111]

[0112] (3.3) Feature confidence

[0113] Feature confidence is determined by taking a certain modality feature as the decision modality. The reliability of the modality data will directly affect its contribution to feature integration and the degree of use of the other modality information. For example, when the skeleton information itself is accurate enough to independently complete the action recognition task, the fusion network will reduce its dependence on RGB modality features and pay more attention to the human motion information in the skeleton features. The specific framework structure is as follows: Figure 6 shown.

[0114] The skeleton mode is used as the decision mode, and the input of the feature confidence module is the output Y of the feature weighting module. Skeleton , first through a fully connected layer, and then through a sigmoid activation function, to obtain the gating matrix Y' Skeleton , so that each element is located in [0,1], as shown in formula (18).

[0115] Y' Skeleton =σ(Fc(W·Y Skeleton ))∈R B×C (18)

[0116] To be used for Y Skeleton and Y RGB Perform gate operation: Y Skeleton⊙Y′ Skeleton and Y RGB ⊙(1-Y′ Skeleton )∈R B×C , where ⊙ represents element-wise multiplication; finally, the gated features of RGB and skeleton are simply combined by element-wise addition to produce the comprehensive action feature, as shown below:

[0117] Y Fusion =Y Skeleton ⊙Y′ Skeleton +Y RGB ⊙(1-Y′ Skeleto n)∈R B×C (19)

[0118] In order to make the feature confidence module play a role in deciding which modality to use and how much modality information to use in the entire fusion network, a loss function L is set here gate To train the gating module, we enforce the learning priority of the decision mode.

[0119] L gate = -log(1-Y' Skeleton ) (20)

[0120] If the RGB mode is used as the decision mode, the process is the same as the skeleton mode, that is, the input of the feature confidence module is Y RGB , first through a fully connected layer, and then through a sigmoid activation function, to obtain the gating matrix Y' RGB As shown in formula (21).

[0121] Y' RGB =σ(Fc(W·Y RGB ))∈R B×C (twenty one)

[0122] To be used for Y Skeleton and Y RGB Perform gate operation: Y RGB ⊙Y′ RGB and Y Skeleton ⊙(1-Y′ RGB ); Finally, the RGB and skeleton gated features are simply combined by element-wise addition to produce the comprehensive action feature as follows:

[0123] Y Fusion =Y RGB ⊙Y′ RGB +Y Skeleton ⊙(1-Y′ RGB ) (twenty two)

[0124] Set the loss function L gate To train the gating module, we enforce the learning priority of the decision mode.

[0125] L gate = -log(1-Y' RGB ) (twenty three).

[0126] Joint training

[0127] Two deep learning models are used to extract deep features from RGB stream and skeleton stream, namely and It is used to represent the implicit semantics of the same type of action corresponding to different modal data. and Input feature fusion module for training to obtain the final fusion feature Y Fusion The prediction is then normalized to a probability distribution through the softmax function. In order to promote the collaborative training of two deep learning networks with different modalities, a joint loss function is used to comprehensively optimize the entire fusion network framework.

[0128] L total =L fushion +λL gate (twenty four)

[0129] Where L fushion is the cross entropy loss function, λ represents the balance factor, and its initial value is set to 1. This design can make full use of the complementary information between different modalities to maximize the model performance during the training process.

[0130] (4) Experimental proof

[0131] (4.1) Dataset Introduction

[0132] The mainstream human behavior recognition dataset NTU-RGB+D was selected to verify the proposed method. [8] It is one of the most widely used datasets in the field of human action recognition. It contains a total of 60 types of actions performed by 40 subjects, with a total of 56,880 action sequences. All 60 action categories are divided into two categories, one is single-person action, and the other is two-person interactive action. Each action contains RGB data, depth data, and skeleton data of 25 joints. The dataset has two evaluation methods, namely Cross Subject (CS) and Cross View (CV). The CS benchmark divides 40 subjects into two groups. The first group has 40,320 action sequences for training samples, and the second group has 16,560 action sequences for testing samples. The CV benchmark uses the action sequences recorded by cameras 2 and 3 for training samples, totaling 37,920 samples, and the action sequences recorded by camera 1 for testing samples, totaling 18,960 samples.

[0133] (4.2) Experimental environment configuration and parameter setting

[0134] The experimental environment and configuration are listed in Table 1. The training process is as follows: First, a model with only RGB modality and a model with only skeleton modality are trained. For the RGB stream, the RGB frame size is adjusted to 112*112, the learning rate and weight decay are set to 0.1 and 0.0005 respectively, and the length of the video sequence is set to 16; for the skeleton stream, the learning rate and weight decay are set to 0.1 and 0.0005 respectively. The network is trained using the SGD optimizer, and the optimal training model is retained only for feature extraction of RGB stream data and skeleton data. Training is performed only by minimizing the loss L fushion Then load the pre-trained RGB-only and skeleton-only models of the two modalities to initialize the fusion network, train the fusion network while freezing the pre-trained part of the network, and then minimize the overall loss L total To train the entire feature fusion part. The number of training rounds for the fusion network is 6, the learning rate is set to 0.01, and it is divided by 10 in the third epoch. The accuracy that can intuitively express the generalization ability of the model is selected as the evaluation indicator of the model performance, as shown in the formula (19):

[0135]

[0136] Where TP represents the number of samples predicted correctly, and Total represents the total number of samples.

[0137] Table 1 Experimental environment and configuration

[0138]

[0139] (4.3) Impact of Fusion Network

[0140] In order to verify the impact of the fusion network on the accuracy, a detailed comparative analysis was conducted to compare the performance differences between single-modal and multi-modal features in recognition accuracy. From the data in Table 2, it can be clearly observed that the proposed multi-modal feature fusion scheme improves the accuracy by 7.15% and 2.58% compared with the single RGB modality and skeleton modality, significantly improving the recognition accuracy. The results fully demonstrate the effectiveness of multi-modal feature fusion in improving recognition performance.

[0141] For the test sample, the feature vector Y of the fusion feature, RGB mode and skeleton mode is calculated respectively. Fusion , and Using t-distributed stochastic neighbor embedding (t-SNE) for visualization, we can find that the feature aggregation of the fusion feature is better than that of the RGB mode and the skeleton mode. The visualization results are as follows: Figure 7 As shown, Figure 7(a) is the t-SNE visualization of RGB mode. Figure 7 (b) is the t-SNE visualization of the skeleton mode. Figure 7 (c) t-SNE visualization of fused features.

[0142] Table 2 The impact of multimodality on accuracy

[0143]

[0144] (4.4) Impact of feature fusion scheme

[0145] In order to explore the contribution of feature fusion technology to accuracy, some traditional fusion technologies were used to fuse RGB features and skeleton features, including average, sum, multiplication, maximum and concatenation operations. Table 4 reveals the specific impact of different fusion strategies on recognition accuracy. According to the data in Table 3, it can be observed that compared with the average, multiplication, maximum and concatenation in traditional fusion, the recognition accuracy of the fusion scheme proposed in the present invention is improved by 1.17%, 1.3%, 1.19% and 0.83% respectively. This result fully confirms the superiority of the fusion scheme proposed in the present invention over the traditional fusion scheme.

[0146] Table 3. Impact of feature fusion scheme on accuracy

[0147]

[0148] (4.5) Impact of weight factors

[0149] The network consists of three serial modules, which are used for the association of skeleton and RGB features, feature weighting and feature confidence determination. fushion and L gate Two loss functions, where L fushion represents the loss of fused modality features, L gate represents the loss of confidence in the confidence modality. The total loss function is defined as L total =L fushion +λL gate , λ is a weight factor used to adjust the relative importance between the two losses. When λ is set to different values, it will have a slight impact on the overall accuracy. Table 4 reflects the impact of different λ settings on the fusion accuracy. From Table 5, it can be seen that when the value of λ is 1.5, the accuracy of the overall loss is optimal.

[0150] Table 4 Impact of weight factors on accuracy

[0151]

[0152] (4.6) Impact of decision-making module

[0153] In order to verify the role of decision modality in feature confidence, RGB modality and skeleton modality are used as decision modalities to carry out verification experiments. The experimental results are shown in Table 5. The accuracy of skeleton modality as decision modality is 7.56% higher than that of RGB. The results show that the appropriate decision modality plays a relatively important role in the entire fusion process.

[0154] Table 5 Impact of decision module on accuracy

[0155]

[0156] (4.7) Comparison with other methods

[0157] In order to verify the effectiveness and advancement of the method proposed in this invention, a detailed comparative analysis was conducted on the NTU-RGB+D dataset. A variety of advanced methods in the current field were selected for comparison. As shown in Table 6, the results of these comparative analyses fully demonstrate the superiority of the fusion strategy of this invention. Compared with the existing methods, the method of this invention has achieved significant improvement in accuracy.

[0158] Table 6 Comparison of models with existing technologies on the NTU-RGB+D dataset

[0159]

[0160] What is described above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

[0161] References

[0162] [1].Das,S.,et al.,Toyota Smarthome: Real-WorldActivities ofDailyLiving,in in Proceedings of the 17th IEEE / CVFInternational Conference onComputer Vision.2020.

[0163] [2].Joze,H.R.V.,et al.MMTM:Multimodal TransferModulefor CNNFusion.inin Proceedings oftheIEEE / CVF Conference on Computer Vision andPatternRecognition.2020.

[0164] [3].Moon,G.,et al.Integralaction:Pose-Driven FeatureIntegrationforRobustHuman Action Recognition in Videos.in in Proceedingsofthe IEEE / CVF Conference on Computer Vision andPattern Recognition.2021.

[0165] [4].Liu,G.,et al.Action Recognition Basedon 3D Skeleton AndRGBFrameFusion.in Proceedings of2019 IEEE / RSJInternational Conference onIntelligentRobots andSystems.2019.IEEE.

[0166] [5].De Boissiere,A.M.and R.Noumeir,Infrared and 3D Skeleton FeatureFusionfor RGB-DAction Recognition.Institute ofElectrical and ElectronicsEngineers Inc,2020.8:p.168297-168308.

[0167] [6].Su,L.,et al.,MSAF:Multimodal SplitAttention Fusion.arXiv preprintarXiv:1409.1556,2020.

[0168] [7].Liu,M.and J.Yuan.RecognizingHuman Actions as theEvolution ofPoseEstimation Maps.in in Proceedings oftheIEEE Conference on Computer VisionandPattern Recognition.2018.

[0169] [8].Li,J.,et al.,SGM-Net:Skeleton-GuidedMultimodalNetworkforActionRecognition.Pattern Recognition.104.

[0170] [9]WEIYAO Xu,MUQING W,MIN Z,TING X.Fusion ofSkeleton and RGBFeaturesfor RGB-D Human Action Recognition[J].IEEE Sensors Journal,2021,21(17):19157-19164.

Claims

1. A human behavior recognition method based on multimodal data and deep learning, characterized in that: The following steps are involved: (1) Preprocessing of skeleton and RGB data; (2) Using Resnet3D 18 network RGB video stream for feature extraction; An adaptive graph convolution block is added to the AGCN network. The output feature dimension of the improved AGCN network is the same as that of the Resnet3D 18 network. The improved AGCN network is used to extract features from the skeleton data. (3) Input the proposed RGB features and skeleton features into the feature fusion network to obtain the fusion features; The feature fusion module includes a feature association module, a feature weighting module and a feature confidence module; (4) The fused features are used to classify actions of different categories through the softmax classifier.

2. The human behavior recognition method based on multimodal data and deep learning according to claim 1, characterized in that: The preprocessing of RGB data includes: (a) Crop the video V at equal time intervals and retain only K frames related to the human body to obtain a set of video frames; (b) The Crop(·) method is used to crop the task center activity area of ​​all image sequences to ensure that all samples of the dataset input to the network focus on the action of the person as much as possible and reduce the interference of useless background; (c) All image sequences in the dataset are processed by the Resize(·) method to adjust the size of the image sequence to the size required by the deep learning network; (d) After resizing, the data needs to be normalized; (e) New sample images are randomly generated through data augmentation methods, which not only enhances the generalization ability of the model but also avoids overfitting and improves the robustness of the model.

3. The human behavior recognition method based on multimodal data and deep learning according to claim 1, characterized in that: The preprocessing of skeleton data is as follows: For skeleton data X Skeleton ∈R B×C×T×N×M Dimensions are adjusted. Where B represents the number of batches, C represents the number of channels, T represents the frame in the skeleton data sequence, N represents the number of joints, and M represents the skeleton in the frame; after adjustment, the B and M dimensions are merged into a new dimension BM, where BM = B × M.

4. The human behavior recognition method based on multimodal data and deep learning according to claim 1, characterized in that: The feature association associates each feature with all feature data. When calculating the output of each pixel position, it not only calculates with the neighborhood, but also calculates the correlation with all positions in the image, and then uses the calculated response value to re-assign the feature weights of the original feature: For the current position feature vector x i ∈X Fusion The operations are: Among them, x j ∈X Fusion For all possible other position feature vectors, y i represents the output feature, y i and x i The feature size is the same as that of g(·), and for each position x j The features of x are transformed into a 1×1 convolution operation, and f(·) is used to calculate x j and x i Embedded Gaussian function of the correlation: Here θ(x i )=W θ x i β(x j )=W β x i , θ(·) and β(·) are 1×1 convolution operations, W θ and W β is the weight matrix. In order to include feature similarity information after the non-local mean operation without destroying the initial feature form and information, a residual connection is inserted in the non-local mean operation: z i =W z y i +x i (11) Among them, "+x i " represents residual connection, W z Its weight matrix, x i and i Given in formula (6), we get the characteristic z i , the x corresponding to all possible positions i i After processing, feature stacking is performed to finally obtain Z Fusion ∈R B×M .

5. The human behavior recognition method based on multimodal data and deep learning according to claim 1, characterized in that: The feature weighting module processing process is as follows: First, by fusion feature Z Fusion Regularization is performed by calculating the sum of squares: Then, Z Fusion Put in the fully connected layer and the ReLU layer to obtain a more expressive new fusion feature: Z Fusion =ReLU(FC(W f WITH Fusion ))∈R B×C (14) Where FC(·) represents the fully connected layer, W f As its weight matrix, then Z Fusion Use one-dimensional convolution to calculate the attention weight F Fusion : F Fusion =σ(Conv1D(W C WITH Fusion ))∈R B×C (15) where σ(·) represents the sigmoid activation function, Conv1D(·) represents a one-dimensional convolution; finally, the attention weight F Fusion Linearly multiply the two input feature vectors to output the attention-weighted feature map Y RGB and Y Skeleton :

6. The human behavior recognition method based on multimodal data and deep learning according to claim 1, characterized in that: The feature confidence is based on a certain modality feature as the decision modality. The reliability of the modality data will directly affect its contribution to the feature integration and the degree of use of the other modality information: The skeleton mode is used as the decision mode, and the input of the feature confidence module is the output Y of the feature weighting module. Skeleton , first through a fully connected layer, and then through a sigmoid activation function, to obtain the gating matrix Y' Skeleton , so that each element is located in [0,1], as shown in formula (15). Y' Skeleton =σ(Fc(W·Y Skeleton ))∈R B×C (18) To be used for Y Skeleton and Y RGB Perform gate operation: Y Skeleton ⊙Y′ Skeleton and Y RGB ⊙(1-Y′ Skeleton )∈R B×C , where ⊙ represents element-wise multiplication; finally, the gated features of RGB and skeleton are simply combined by element-wise addition to produce the comprehensive action feature, as shown below: AND Fusion =And Skeleton ☉And' Skeleton +Y RGB ⊙(1-Y' Skeleton )∈R B×C (19) In order to make the feature confidence module play a role in deciding which modality to use and how much modality information to use in the entire fusion network, a loss function L is set here gate To train the gating module to enforce the priority of learning decision modes; L gate =-log(1-Y' Skeleton ) (20); Taking RGB mode as the decision mode, the input of the feature confidence module is Y RGB , first through a fully connected layer, and then through a sigmoid activation function, to obtain the gating matrix Y' RGB As shown in formula (21). Y' RGB =σ(Fc(W·Y RGB ))∈R B×C (21) To be used for Y Skeleton and Y RGB Perform gate operation: Y RGB ⊙Y' RGB and Y Skeleton ⊙(1-Y' RGB ); Finally, the RGB and skeleton gated features are simply combined by element-wise addition to produce the comprehensive action feature as follows: AND Fusion =And RGB ☉And' RGB +Y Skeleton ☉(1-Y' RGB ) (22) Set the loss function L gate To train the gating module, we enforce the learning priority of the decision mode. L gate =-log(1-Y' RGB ) (23).

7. The method for human behavior recognition based on multimodal data and deep learning according to claim 6, characterized in that: A joint loss function is used to comprehensively optimize the entire fusion network framework. THE total =L fushion +λL gate (24) Where L fushion is the cross entropy loss function, λ represents the balance factor, and its initial value is set to 1.

Images