A cross-attention-based multi-modal fusion wavelet knowledge distillation video behavior recognition method and system

Abstract

The application relates to a video behavior recognition method and system based on cross-attention multi-modal fusion wavelet knowledge distillation, which comprises the following steps: wearable sensor data is acquired and processed; one-dimensional time sequence signals of wearable sensor data are converted into two-dimensional image representation; a multi-modal attention hollow convolution kernel residual network is built; a teacher network is trained; a student network is built; a wavelet distillation loss is constructed; the student network is trained by minimizing the wavelet distillation loss and cross-entropy loss; the wearable sensor-video behavior is recognized through the trained student network, and a recognition result is obtained. The application fully fuses different modal information through multi-modal cross-attention, and guides video recognition through wavelet knowledge distillation. The application filters noise through wavelet transformation, and has strong robustness.

Description

TECHNICAL FIELD

[0001] The application relates to a video behavior recognition method and system based on cross-attention multi-modal fusion wavelet knowledge distillation, and belongs to the technical fields of behavior recognition, sensor technology, machine learning and the like. BACKGROUND

[0002] With the development of deep learning and computer vision, behavior recognition has attracted extensive attention due to its wide application in video monitoring systems, video retrieval, human-computer interaction and the like, and especially a behavior recognition method and model based on video and image are widely applied to real life. However, first, a method based on a visual sensor is largely dependent on the quality of video and image. If the resolution of a camera is low or is shielded, the quality of the obtained image or video is poor, and the effect of behavior recognition is easily affected, which is also a bottleneck of the video behavior recognition method in actual application. Secondly, a behavior recognition method based on video has a large amount of calculation and needs to be supported by a hardware device with complex computing capacity, thereby increasing the cost. In addition, in the case that visual sensor data is unavailable in places such as banks that need privacy protection, a behavior recognition method based on video cannot be implemented. A behavior recognition method based on a wearable sensor can solve the problem of the video behavior recognition method. Time-based one-dimensional data such as three-axis acceleration, gyroscope and direction signal obtained through a sensor in a smart watch or a smart phone is used to obtain the behavior change of a person. The behavior recognition method based on the wearable sensor does not depend on video and image, and provides the possibility of behavior recognition in a privacy-protected scene.

[0003] With the popularity and increasing demand of smart cities and smart medical treatment, human behavior recognition based on a wearable sensor has become a key research field of human activity cognition. Although some behavior recognition methods based on a wearable sensor have been proposed and good effects have been achieved, most of these methods only consider time series data of the wearable sensor, and the complementary relationship between visual sensor data and wearable sensor data is less considered. Therefore, it is quite important to use motion information from different modalities of visual sensor and wearable sensor to improve the recognition performance of behavior. SUMMARY

[0004] In view of the deficiencies of the prior art, the application provides a video behavior recognition method based on cross-attention multi-modal fusion wavelet knowledge distillation.

[0005] The application provides a cross-attention multi-modal wavelet knowledge distillation algorithm, fully utilizes wearable sensor and visual sensor information, fully fuses different modal information through multi-modal cross-attention, and guides video recognition through wavelet knowledge distillation.

[0006] The application provides a cross-attention-based multi-modal wavelet knowledge distillation model for wearable sensors and video data. The contributions of the model are: 1) feature extraction of wearable sensor data is performed by a convolutional network constructed by a hollow convolution and an attention mechanism, and the receptive field is increased; 2) cross-attention modules are used to fuse wearable sensor data of multiple modalities to obtain complementary information of different modalities, and more prior knowledge is provided for the next distillation; 3) the last pooling layer of the teacher network and the student network is replaced by discrete wavelet transform, only low-frequency features are retained to reduce noise and extract useful information, and then distillation is used to guide the student network to recognize behaviors from video data and improve the recognition performance.

[0007] Term explanation:

[0008] 1、ImageNet dataset, a computer vision dataset, was created by Professor Li Feifei of Stanford University. The dataset contains 14,197,122 images and 21,841 Synset indexes. Synset is a node in the WordNet hierarchy, which is also a set of synonymous words. The ImageNet dataset has always been a benchmark for evaluating the performance of image classification algorithms. The ImageNet dataset is a large image dataset established to promote the development of computer image recognition technology. In 2016, the ImageNet dataset contained more than 1 million images, each of which was manually labeled with a class. The images in the ImageNet dataset cover most of the image categories that can be seen in life. ImageNet was originally a dataset with more than 1 million images.

[0009] 2、BNInception network, an important milestone in the history of CNN classifier development. Before the appearance of Inception, most popular CNNs simply stacked more and more convolutional layers to make the network deeper in the hope of better performance. The second version of Inception, also known as BN-Inception, mainly introduces an important technology of deep learning, Batch Normalization (BN). The use of BN technology normalizes the data before it enters another layer of the network, which can achieve higher accuracy and training speed.

[0010] 3、VGG network, is the Oxford University computer vision group and Google DeepMind researchers instrument developed deep convolutional neural network. VGG mainly explores the relationship between the depth of convolutional neural network and its performance, through repeatedly stacking 3*3 small convolution kernel and 2*2 maximum pooling layer, VGGNet successfully builds 16-19 layers of deep convolutional neural network. Compared with the previous network structure, the error rate is greatly reduced; at the same time, the generalization ability of VGG is very good, and it has good performance on different picture data sets. So far, VGG is still often used to extract feature images.

[0011] 4、ResNet network, proposed by Microsoft Research Institute He Kai Ming, Zhang Xiangyu, Ren Shaoqing, Sun Jian. The research motivation is to solve the degradation problem of deep network, unlike the past network is to fit a distribution by learning, ResNet is to fit the residual relative to the output of the last layer by learning.

[0012] The technical scheme of the application is:

[0013] A video behavior recognition method based on cross attention multi-modal fusion wavelet knowledge distillation, comprising the following steps:

[0014] Obtaining wearable sensor data and processing: converting one-dimensional time series signal of wearable sensor data into two-dimensional image representation;

[0015] Building a multi-modal attention hole convolution kernel residual network, namely a teacher network;

[0016] Training the teacher network;

[0017] Building a student network;

[0018] Minimizing wavelet distillation loss and cross entropy loss to train the student network;

[0019] Recognizing wearable sensor-video behavior through the trained student network to obtain the recognition result.

[0020] According to the application, the wearable sensor data processing comprises:

[0021] Firstly, the wearable sensor data has three axial time series signals (x, y, z), define one of the axial signals as X={x1,…,x n} using the minimum-maximum normalization method to normalize the original signal X to the interval [-1, 1] to obtain the normalized signal As shown in formula (I):

[0022]

[0023] The normalized signal is then transformed to a polar coordinate system with a transformation function g, which represents the cosine angle of the normalized amplitude and the radius from time t, as shown in equation (II):

[0024]

[0025] t i is a time stamp, and θ i is the cosine angle of the normalized amplitude;

[0026] The correlation coefficient between time intervals is obtained by the dot product and the triangle sum between the dots, and the correlation coefficient is calculated by the cosine of the angle between the vectors, the correlation coefficient between time i and j is calculated by cos(φ i +φ j ), and φ i and φ j are the cosine angles of the normalized amplitudes of time i and j, respectively;

[0027] The matrix definition of the GAF angle field is G, as shown in equation (III):

[0028]

[0029] Assuming that each three-axis sensor data of length m is converted into a GAF matrix of size m x m, the GAF matrix G of the three-axis sensor data is assembled into a three-channel image representation P = {G x ,G y ,G z} of size m x m x 3; m is the length of the three-axis sensor data, G x , G y , and G z are the GAF matrices of the x-axis, y-axis, and z-axis of the three-axis sensor data, respectively;

[0030] According to the application, the multi-modal attentional hole convolution kernel residual network preferably comprises one convolution layer, one BN layer, one activation function layer, four Max Pooling layers, five SADRM modules, one DWT module, one modal fusion module, and one full connection layer; the modal fusion module comprises a plurality of CVA modules;

[0031] The convolution layer, the BN layer, the activation function layer, the SADRM module, the Max Pooling layer, and the DWT module are built to extract features of the wearable sensor data, the Max Pooling layer performs down-sampling operation, the DWT module performs wavelet transform, and the plurality of CVA modules process the fusion between different modalities; the fused features obtained from the plurality of CVA modules are input to the full connection layer through cascading.

[0032] ​Further preferably, the SADRM module comprises a 1*1 convolution layer, a stack of a 1*1 convolution layer and a 3*3 convolution layer, an LKA module, a stack of two 3*3 dilated convolution layers with a dilation rate of 2, and a stack of two 3*3 dilated convolution layers with a dilation rate of 4.

[0033] The last layer feature map of the teacher network is input into the SADRM module, and after the calculation and processing of each part, the obtained five calculation results and the input feature map are added to form a shortcut connection, and finally the output Stage of the SADRM module is obtained, as shown in formula (VI):

[0034]

[0035] In formula (VI), x is the input of the last stage, Conv 1×1 represents a 1*1 convolution operation, Conv 3×3 represents a 3*3 convolution operation, LKA represents a Large Kernel Attention operation, Dila 3×3 represents a 3*3 dilated convolution operation.

[0036] Further preferably, in the SADRM module, for the stacked convolution layers, assuming that the receptive field of the ith layer is RF i =a 1,i ×a 2,i , a 1,i , a 2,i respectively represent the length and width of the receptive field, the convolution kernel size of the i+1th layer is k i+1 ×k i+1 , and the dilation rate of the dilated convolution of the i+1th layer is d i+1 , then the receptive field RF i+1 of the i+1th layer is as shown in formula (V):

[0037] RF i+1 =[a 1,i +(k i+1 -1)d i+1 ]×[a 2,i +(k i+1 -1)d i+1 ] (V)

[0038] In formula (V), RF0=1*1 represents the receptive field of the input layer.

[0039] Further preferably, the LKA module is as shown in formula (VI) and formula (VII):

[0040] Atten=Conv 1×1 (DW-D-Conv(DW-Conv(X))) (VI)

[0041]

[0042] (VI), (VII) in formula, is the input feature, DW-Conv is the deep convolution, DW-D-Conv is the deep dilated convolution, Conv 1×1 represents a one-dimensional channel convolution, is the attention mapping, which represents the importance of each feature, represents the element product, Y refers to the output of the large kernel attention module.

[0043] According to the present application, the DWT module decomposes the data into different components according to different frequency intervals for anti-aliasing in signal processing, given an input image P, using Haar wavelet to perform wavelet transform along the row and column respectively, decomposed into formula (VIII), formula (IX), formula (X), formula (X) :

[0044] P ll = LPL T (VIII)

[0045] P lh = HPL T (IX)

[0046] P hl = LPH T (X)

[0047] P hh = HPH T (XI)

[0048] In formula (VIII), (IX), (X), (XI), L is a low-pass filter matrix, H is a high-pass filter matrix, P refers to a given input image, P ll , P lh , P hl and P hh are the low-frequency component, horizontal high-frequency component, vertical high-frequency component and diagonal high-frequency component of the image after decomposition respectively; the low-frequency component P ll is selected as the output of the DWT module; the wavelet transform is used to replace the maximum pooling layer with a step of 2 in the last stage of the teacher network, as shown in formula (XII) :

[0049] MaxPool s＝2 →DWT ll (XII)

[0050] In formula (XII), MaxPool s＝2 represents the maximum pooling layer with a step of 2, DWT ll represents the transformation mapping of the feature mapping to the low-frequency component.

[0051] According to the application, preferably, the CVA module performs information fusion between different modalities through a cross-attention method, as shown in formula (XIII):

[0052]

[0053] In formula (XIII), x and y represent different modalities, W Q , W K , W V is a weight matrix for calculating query, key and value in the CVA module, CVA(x, y) represents cross-view attention, d k represents the dimension of the key, and softmax() represents a softmax function.

[0054] According to the application, preferably, the teacher network is trained, including:

[0055] The wearable sensor data is converted into a two-dimensional image, and the two-dimensional image is input into the built teacher network for iterative training 100 times to obtain a behavior classification result.

[0056] According to the application, preferably, the student network uses a BNInception network, a VGG network or a ResNet network pre-trained through an ImageNet data set.

[0057] According to the application, preferably, the student network is trained by minimizing wavelet distillation loss and cross-entropy loss, including the following steps:

[0058] (1) Obtain the wavelet distillation loss of the teacher network and the student network;

[0059] (2) Obtain the cross-entropy loss of the teacher network and the student network;

[0060] (3) Train the student network through a gradient descent method with the total loss as the target to obtain the trained student network.

[0061] According to the application, preferably, the wavelet distillation loss L WKD of the teacher network and the student network is the L1 loss between the output of the DWT module of the teacher network and the output of the DWT module of the student network, as shown in formula (XIV):

[0062]

[0063] In formula (XIV), Ψ S is the output of the wavelet transform layer of the student network, is the output of the wavelet transform layer of the teacher network i, and ||·||1 represents the L1 norm;

[0064] The cross-entropy loss L of the teacher network and the student network CE As shown in formula (XV):

[0065]

[0066] In formula (XV), q c is the true label of the cth class, is the probability of the corresponding behavior class c output by the softmax layer of the student network;

[0067] The total loss L is as shown in formula (XVI):

[0068] L=(1-α)L CE +αL WKD (XVI)

[0069] In formula (XVI), α is a hyperparameter.

[0070] A video behavior recognition system based on cross-attention multi-modal fusion wavelet knowledge distillation, comprising:

[0071] A wearable sensor data acquisition and processing module configured to acquire and process wearable sensor data, and convert one-dimensional time series signals of the wearable sensor data into two-dimensional image representations;

[0072] A multi-modal attentional dilated convolution kernel residual network building module configured to build a multi-modal attentional dilated convolution kernel residual network;

[0073] A teacher network training module configured to train a teacher network;

[0074] A student network building module configured to build a student network;

[0075] A student network training module configured to minimize wavelet distillation loss and cross-entropy loss to train the student network;

[0076] A recognition module configured to recognize wearable sensor-video behaviors through the trained student network to obtain a recognition result.

[0077] A computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the video behavior recognition method based on cross-attention multi-modal fusion wavelet knowledge distillation when executing the computer program.

[0078] A computer-readable storage medium having a computer program stored thereon, the computer program implementing the steps of the video behavior recognition method based on cross-attention multi-modal fusion wavelet knowledge distillation when executed by a processor.

[0079] The beneficial effects of this invention are as follows:

[0080] 1. To consider the complementary relationship between visual sensor and wearable sensor data, this invention proposes a multimodal wavelet knowledge distillation algorithm based on cross-attention. Compared with most existing behavior recognition methods based on wearable sensors, this invention fully utilizes information from both wearable and visual sensors, achieves comprehensive fusion of different modal information through multimodal cross-attention, and uses wavelet knowledge distillation to guide video recognition.

[0081] 2. This invention proposes a multimodal attention-dilated convolution kernel residual network, which extends the attention mechanism from a single modality to multimodal feature fusion. It integrates a large kernel attention module, dilated convolution, and cross attention into the multimodal attention-dilated convolution kernel residual module, thereby increasing the fusion of modal information and information from different modalities.

[0082] 3. This invention proposes the idea of ​​using wearable sensor data to guide video data for behavior recognition. It learns supplementary knowledge in wearable sensor data through wavelet distillation and filters noise through wavelet transform, which is highly robust.

[0083] 4. This invention uses the knowledge distillation method to train small networks, which has low hardware requirements, high portability, and great application prospects.

[0084] 5. The knowledge distillation network of this invention can not only be applied to direction recognition, but can also be extended to other deep learning fields. Attached Figure Description

[0085] Figure 1 This is a schematic diagram of the SADRM module of the present invention;

[0086] Figure 2 This is a schematic diagram illustrating the information fusion between different modalities using the cross-attention method in the CVA module of this invention;

[0087] Figure 3 This is a schematic diagram of the structure of the multimodal attention-drilled convolution kernel residual network of the present invention;

[0088] Figure 4 This is a general block diagram of a video behavior recognition method based on cross-attention multimodal fusion wavelet knowledge distillation according to the present invention;

[0089] Figure 5 This is a schematic diagram of the teacher network and student network training process of the present invention. Detailed Implementation

[0090] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0091] Example 1

[0092] A video behavior recognition method based on cross-attention multimodal fusion wavelet knowledge distillation, such as Figure 4 As shown, the steps are as follows:

[0093] Acquire and process wearable sensor data: In order to maintain the local temporal relationship of the one-dimensional motion data of wearable sensors, the one-dimensional time series signal of wearable sensor data is converted into a two-dimensional image representation;

[0094] Construct a multimodal attention-diffuse convolution kernel residual network, i.e., a teacher network;

[0095] Training teacher network;

[0096] Build a student network;

[0097] Minimize wavelet distillation loss and cross-entropy loss to train the student network;

[0098] The trained student network identifies behaviors in wearable sensor videos, yielding recognition results. These results categorize the input video according to the behavior; for example, if the behavior in the video is "throwing something," the trained network will classify it as "throwing something."

[0099] Example 2

[0100] The video behavior recognition method based on cross-attention multimodal fusion wavelet knowledge distillation described in Example 1 differs in that:

[0101] Wearable sensor data processing includes:

[0102] First, wearable sensor data (including data acquired from accelerometers, gyroscopes, orientation sensors, etc.) consists of three axial time-series signals (x, y, z). One of these axial signals is defined as X = {x1, ..., x...} n The original signal X is normalized to the interval [-1, 1] using the min-max normalization method, resulting in the normalized signal. As shown in equation (I):

[0103]

[0104] Then, the normalized signal is transformed using the transformation function g. Transforming to a polar coordinate system, it represents the cosine angle of the normalized amplitude and the radius starting from time t, as shown in equation (II):

[0105]

[0106] t i It refers to the timestamp, θi It refers to the cosine angle of the normalized amplitude;

[0107] The correlation coefficient between time intervals is obtained by triangulation between points. The correlation coefficient is calculated using the cosine of the angle between vectors. The correlation coefficient between time intervals i and j is expressed as cos(φ). i +φ j To calculate φ i φ j The cosine angles of the normalized amplitudes at times i and j are respectively;

[0108] The matrix of the Grammy angle field (GAF) is defined as G, as shown in equation (III):

[0109]

[0110] This method provides a novel representation style for Grammy angle fields, preserving local temporal relationships in the form of temporal correlations through timestamp augmentation. Data from wearable sensors, such as accelerometer, gyroscope, and orientation signals, are typically triaxial. Therefore, assuming each triaxial sensor data point of length m is transformed into an m×m GAF matrix, the GAF matrix G of the triaxial sensor data is assembled into a three-channel image representation P = {G} of size m×m×3. x G y G z}; m is the length of the triaxial sensor data, G x G y G z These are the GAF matrices for the x, y, and z axes of the triaxial sensor data, respectively.

[0111] like Figure 3 As shown, the multimodal attention-dilated convolutional kernel residual network includes one convolutional layer (7×7Stride 2), one batch normalization (BN) layer, one activation function layer (Leaky ReLU), four max pooling layers, five SADRM modules, one DWT module, one modality fusion module, and one fully connected layer; the modality fusion module includes multiple CVA modules;

[0112] like Figure 3 As shown, feature extraction of wearable sensor data is performed by constructing convolutional layers, BN layers, activation function layers, SADRM modules, Max Pooling layers, and DWT modules. Max Pooling layers perform downsampling operations, DWT modules perform wavelet transforms, and multiple CVA modules handle the fusion between different modalities. The fused features obtained from multiple CVA modules are cascaded and input into a fully connected layer.

[0113] Because sensor data includes multiple modalities, it is necessary to extract and fuse the most representative features from these modalities. To fully utilize the complementary information of each modality in multiple teacher networks, the attention mechanism is extended from a single modality to multimodal feature fusion. This involves integrating the Large Kernel Attention (LKA) module, dilated convolution layers, and cross-attention into a multimodal attention dilated convolution kernel residual network.

[0114] The self-attention dilated convolution module is similar to the Inception structure, such as... Figure 1 As shown, the SADRM module (Self-Attention Dilation Convolution Residual Model) includes a 1*1 convolutional layer, a stacked 1*1 convolutional layer and a 3*3 convolutional layer, an LKA module (Large Kernel Attention), two stacked 3*3 dilation convolutional layers with a dilation rate of 2, and two stacked 3*3 dilation convolutional layers with a dilation rate of 4. Using dilated convolutional layers in the network can extract more comprehensive action feature information while maintaining the spatial resolution of the feature map.

[0115] The feature map of the previous layer of the teacher network is input into the SADRM module. After undergoing the calculations and processing described above, the five calculation results are added to the input feature map to form a shortcut connection, ultimately yielding the output Stage of the SADRM module, as shown in equation (VI):

[0116]

[0117] In equation (VI), x is the input of the previous stage, Conv 1×1 Represents a 1×1 convolution operation, Conv 3×3 LKA represents a 3×3 convolution operation, and LKA represents a Large Kernel Attention operation. 3×3 This represents a 3×3 dilated convolution operation. In the SADRM module, by simultaneously using the LKA module, multiple normal convolutional layers, and dilated convolutional layers, the network's width and adaptability are increased, improving its ability to extract features at different scales. Simultaneously, by avoiding the use of large convolutional kernels, the model's parameters are effectively controlled, reducing computational cost. Furthermore, the introduction of shortcut connections from residual networks into the SADRM module suppresses common deep neural network degradation problems and gradient vanishing phenomena, increasing network depth and improving the model's fitting ability.

[0118] Traditional convolutional network architectures often lose motion details due to stride and deconvolution operations, affecting the final action recognition performance. Detailed spatial information is crucial for action recognition. One strategy to address this is to maintain the feature map resolution, but this increases the number of model parameters and computational cost. Dilated convolutions, however, can increase the receptive field of the convolutional kernel without reducing feature resolution. Using dilated convolutions in the network allows for the extraction of more comprehensive action feature information while preserving the spatial resolution of the feature map, which is significant for action recognition. Therefore, this invention uses dilated convolutions in the teacher network to extract more complementary information.

[0119] Add 4 dilated convolutional layers to each SADRM module. Figure 1 In this context, four 3x3 dilated convolutional layers are added to each SADRM module. Figure 1 The dilation rate of the dilated convolution on the fourth branch is 2; the dilation rate of the dilated convolution on the fifth branch is 4. In the SADRM module, for stacked convolutional layers, it is assumed that the receptive field of the i-th layer is RF. i =a 1,i ×a 2,i a 1,i a 2,i These refer to the length and width of the receptive field, respectively. For example, 3×3 means a receptive field with a length and width of 3. The kernel size of the (i+1)th layer is k. i+1 ×k i+1 The dilation rate of the dilated convolution in the (i+1)th layer is d. i+1 Then the receptive field RF of the (i+1)th layer i+1 As shown in equation (V):

[0120] RF i+1 =[a 1,i +(k i+1 -1)d i+1 ]×[a 2,i +(k i+1 -1)d i+1 (V)

[0121] In equation (V), RF0 = 1 × 1 represents the receptive field of the input layer.

[0122] Self-attention mechanisms, when applied to 2D images, simply treat them as 1D sequences, neglecting the 2D structure of the image. For high-resolution images, the computational cost explodes exponentially, placing high demands on the processor. This approach only considers spatial dimensional information, ignoring channel-level information. To address this issue, this invention employs a large kernel attention module combining attention and convolution. This module not only possesses the adaptability and long-term dependency of self-attention mechanisms but also the advantages of convolution, allowing the utilization of contextual information. The LKA module is shown in equations (VI) and (VII):

[0123] Atten=Conv 1×1 (DW-D-Conv(DW-Conv(X))) (VI)

[0124]

[0125] In equations (VI) and (VII), The input features are DW-Conv (depthwise convolution), DW-D-Conv (depthwise dilated convolution), and Conv. 1×1 Represents one-dimensional channel convolution. It's an attention map, representing the importance of each feature. Y represents the element-wise product, and Y refers to the output of the large kernel attention module.

[0126] The DWT (Discrete Wavelet Transform) module decomposes data into different components according to different frequency intervals, which is used for anti-aliasing in signal processing. Given an input image P, the Haar wavelet is used to perform wavelet transforms along the rows and columns respectively, decomposing it into equations (VIII), (IX), (X), and (XI):

[0127] P ll =LPL T (VIII)

[0128] P lh =HPL T (IX)

[0129] P hl =LPH T (X)

[0130] P hh =HPH T (XI)

[0131] In equations (VIII), (IX), (X), and (XI), L is the low-pass filter matrix, H is the high-pass filter matrix, and P refers to the given input image. ll P lh P hl and Phh These are the low-frequency components, horizontal high-frequency components, vertical high-frequency components, and diagonal high-frequency components after image decomposition; the low-frequency component P ll This represents a position where the grayscale or brightness changes relatively smoothly; in the high-frequency component, P... lh P hl and P hh This corresponds to locations in the image where grayscale or brightness changes are significant, such as noise. Therefore, this invention considers selecting the low-frequency component P. ll The low-frequency components are used as the output of the DWT module; on the one hand, the low-frequency components maintain the structure of the input layer according to the prescribed rules, so that the input image patch has a better representation. On the other hand, by losing the high-frequency components, some noise can be suppressed. In the last stage of the teacher network, wavelet transform is used to replace the max pooling layer with a step size of 2, as shown in equation (XII):

[0132] MaxPool s＝2 →DWT ll (XII)

[0133] In equation (XII), MaxPool s＝2 DWT represents a max-pooling layer with a step size of 2. ll This represents the transformation mapping from the feature map to the low-frequency component.

[0134] After feature extraction from the preceding modules, the CVA (Cross-View Attention) module fuses information between different modalities using a cross-attention method, as shown in Equation (XIII):

[0135]

[0136] In equation (XIII), x and y represent different modes, and W Q W K W V In the CVA module, the weight matrix for query, key, and value is calculated, and CVA(x,y) represents cross-view attention. k The dimension representing the key, and `softmax()` representing the softmax function. Unlike previous feature fusion methods, this method determines the importance of modal information through attention weights. During cross-view fusion, it maintains similar embedding dimensions and depths for modalities such as acceleration and angular velocity. Multiple CVA modules are used... Figure 2 The interaction takes three data modalities (M1, M2, M3) as an example. The cross-view attention (CVA) fusion representation of any two modalities (M1 and M2, M1 and M3, M2 and M3) is shown in Equation (XIII). Then, the three CVA results are concatenated and input into the fully connected layer.

[0137] Training teacher networks, such as Figure 5 As shown, it includes:

[0138] Wearable sensor data (accelerometer and gyroscope data) is converted into two-dimensional images using the Grammy Angular Field (GAF) method. These two-dimensional images are then fed into a pre-built teacher network for iterative training 100 times to obtain behavior classification results.

[0139] The student network uses a BNInception network, a VGG network, or a ResNet network pre-trained on the ImageNet dataset.

[0140] The student network is trained by minimizing wavelet distillation loss and cross-entropy loss, such as... Figure 5 As shown, the steps are as follows:

[0141] (1) Obtain the wavelet distillation loss of the teacher network and the student network;

[0142] (2) Obtain the cross-entropy loss of the teacher network and the student network;

[0143] (3) Train the student network with the total loss as the target by using the gradient descent method to obtain a well-trained student network.

[0144] The general knowledge distillation loss is the KL (Kullback-Leibler divergence) between the student network output and the teacher network output. Compared to ordinary knowledge distillation, the wavelet distillation loss La of the teacher and student networks... WKD The L1 loss between the outputs of the DWT modules of the teacher network and the student network is shown in Equation (XIV):

[0145]

[0146] In formula (XIV), Ψ S For the output of the wavelet transform layer of the student network, Ψ i T Let ||·||1 represent the L1 norm, where ||·||1 represents the output of the wavelet transform layer of teacher network i.

[0147] When training a student network (such as a BNInception network), the total loss includes not only the wavelet knowledge distillation loss, but also the cross-entropy loss between the student network output and the ground truth label, and the cross-entropy loss L between the teacher network and the student network. CE As shown in equation (XV):

[0148]

[0149] In equation (XV), qc For the c-th type of real label, The probability of the corresponding behavior category c output by the softmax layer of the student network;

[0150] The total loss L is as shown in equation (XVI):

[0151] L=(1-α)L CE +αL WKD (XVI)

[0152] In equation (XVI), α is a hyperparameter.

[0153] We selected RGB video data from the publicly available UTD-MHAD dataset and modal data from two different wearable sensors (accelerometer and gyroscope data). The dataset contains 27 different motion categories, repeated 4 times by 8 subjects. This dataset has 5 modes: RGB, depth, skeleton, Kinect, and inertial data. The visual sensor data was acquired using the Kinect camera, while the wearable sensor data was acquired using the inertial sensor.

[0154] The method in this invention is compared with TSN, TSM, MFLF, Logits, AT, and CC methods.

[0155] TSN stands for Temporal Segment Networks. It uses a sparse sampling strategy to divide the video into three segments using information from the entire video. For each segment, a video clip is randomly sampled evenly, and a two-stream network is used for classification.

[0156] TSM stands for Temporal Shift Module. It shifts part of the channel along the time dimension to enhance information exchange between adjacent frames, and is characterized by high efficiency and high performance.

[0157] MFLF stands for Multimodal Feature-Level Fusion, which provides a feature-level fusion method based on RGB, depth, and inertial sensors to improve the accuracy of human motion classification.

[0158] Logits, AT, and CC are three methods for knowledge distillation.

[0159] The method proposed in this invention is configured as follows: initial learning rate of 0.0002, 160 iterations, and α set to 0.1.

[0160] The UTD-MHAD dataset was used, and the recognition performance (%) of each method on the UTD-MHAD dataset is shown in Table 1.

[0161] Table 1

[0162]

[0163] The method of this invention achieves an accuracy rate of 98.61%, which is significantly better than the methods listed.

[0164] Example 3

[0165] A video behavior recognition system based on cross-attention multimodal fusion wavelet knowledge distillation includes:

[0166] The wearable sensor data acquisition and processing module is configured to: acquire wearable sensor data and process it, converting the one-dimensional time series signal of the wearable sensor data into a two-dimensional image representation;

[0167] The multimodal attention-dilated convolutional kernel residual network construction module is configured to: construct a multimodal attention-dilated convolutional kernel residual network;

[0168] The teacher network training module is configured as: Training teacher networks;

[0169] The student network setup module is configured to: set up a student network;

[0170] The student network training module is configured to train the student network by minimizing wavelet distillation loss and cross-entropy loss.

[0171] The recognition module is configured to recognize video behaviors through a trained student network and obtain recognition results.

[0172] Example 4

[0173] A computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the video behavior recognition method based on cross-attention multimodal fusion wavelet knowledge distillation as described in Embodiment 1 or 2.

[0174] Example 5

[0175] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the video behavior recognition method based on cross-attention multimodal fusion wavelet knowledge distillation as described in Embodiment 1 or 2.

Claims

1. A wearable sensor-video behavior recognition method based on cross-attention multimodal fusion wavelet knowledge distillation, characterized in that, The steps include the following: Acquire and process wearable sensor data: convert the one-dimensional time series signal of wearable sensor data into a two-dimensional image representation; Construct a multimodal attention-diffuse convolution kernel residual network, i.e., a teacher network; Training teacher network; Build a student network; Minimize wavelet distillation loss and cross-entropy loss to train the student network; The trained student network identifies wearable sensor-video behavior to obtain recognition results; The multimodal attention-dilated convolutional kernel residual network consists of 1 convolutional layer, 1 batch normalization (BN) layer, 1 activation function layer, 4 max-pooling layers, 5 SADRM modules, 1 DWT module, 1 modality fusion module, and 1 fully connected layer; the modality fusion module includes multiple CVA modules. Feature extraction from wearable sensor data is achieved by constructing convolutional layers, batch normalization (BN) layers, activation function layers, SADRM modules, Max Pooling layers, and DWT modules. Max Pooling layers perform downsampling operations, DWT modules perform wavelet transforms, and multiple CVA modules handle the fusion between different modalities. The fused features obtained from multiple CVA modules are cascaded and input into a fully connected layer. The SADRM module includes a 1x1 convolutional layer, a stacked 1x1 convolutional layer and a 3x3 convolutional layer, an LKA module, two stacked 3x3 dilated convolutional layers with a dilation of 2, and two stacked 3x3 dilated convolutional layers with a dilation of 4. The feature map of the previous layer of the teacher network is input into the SADRM module. After being processed by the above-mentioned parts, the five calculation results are added to the input feature map to form a shortcut connection, and finally the output Stage of the SADRM module is obtained, as shown in Equation (VI): In equation (VI), x is the input of the previous stage, Conv 1×1 Represents a 1×1 convolution operation, Conv 3×3 LKA represents a 3×3 convolution operation, and LKA represents a Large Kernel Attention operation. 3×3 This represents a 3×3 dilated convolution operation; In the SADRM module, for stacked convolutional layers, it is assumed that the receptive field of the i-th layer is RF. i =a 1,i ×a 2,i a 1,i a 2,i These refer to the length and width of the receptive field, respectively, and the kernel size of the (i+1)th layer is k. i+1 ×k i+1 The dilation rate of the dilated convolution in the (i+1)th layer is d. i+1 Then the receptive field RF of the (i+1)th layer i+1 As shown in equation (V): RF i+1 =[a 1,i +(k i+1 -1)d i+1 ]×[a 2,i +(k i+1 -1)d i+1 ] (Ⅴ) In equation (V), RF0 = 1 × 1 represents the receptive field of the input layer; The LKA module is shown in equations (VI) and (VII): Atten=Conv 1×1 (DW-D-Conv(DW-Conv(X))) (VI) In equations (VI) and (VII), The input features are DW-Conv (depthwise convolution), DW-D-Conv (depthwise dilated convolution), and Conv. 1×1 Represents one-dimensional channel convolution. It's an attention map, representing the importance of each feature. Y represents the element-wise product, and Y refers to the output of the large kernel attention module. The DWT module decomposes data into different components according to different frequency intervals for anti-aliasing in signal processing. Given an input image P, it uses Haar wavelets to perform wavelet transforms along the rows and columns respectively, decomposing it into equations (VIII), (IX), (X), and (XI): P ll =LPL T (Ⅷ) P lh =HPL T (Ⅸ) P hl =LPH T (Ⅹ) P hh =HPH T (Ⅺ) In equations (VIII), (IX), (X), and (XI), L is the low-pass filter matrix, H is the high-pass filter matrix, and P refers to the given input image. ll P lh P hl and P hh These are the low-frequency components, horizontal high-frequency components, vertical high-frequency components, and diagonal high-frequency components after image decomposition; the low-frequency component P is selected. ll The output of the DWT module is used; in the last stage of the teacher network, wavelet transform is used instead of the max pooling layer with a step size of 2, as shown in equation (XII): MaxPool s＝2 →DWT ll (Ⅻ) In equation (XII), MaxPool s＝2 DWT represents a max-pooling layer with a step size of 2. ll The transformation mapping represents the feature map to the low-frequency component; The CVA module uses a cross-attention method to fuse information between different modalities, as shown in equation (XIII): In equation (XIII), x and y represent different modes, and W Q W K W V In the CVA module, the weight matrix for query, key, and value is calculated, and CVA(x,y) represents cross-view attention. k The dimension represents the key, and softmax() represents the softmax function.

2. The wearable sensor-video behavior recognition method based on cross-attention multimodal fusion wavelet knowledge distillation according to claim 1, characterized in that, Wearable sensor data processing includes: First, wearable sensor data consists of three axial time-series signals (x, y, z), one of which is defined as X = {x1, ..., xz}. n The original signal X is normalized to the interval [-1, 1] using the min-max normalization method, resulting in the normalized signal. As shown in equation (I): Then, the normalized signal is transformed using the transformation function g. Transforming to a polar coordinate system, it represents the cosine angle of the normalized amplitude and the radius starting from time t, as shown in equation (II): t i It refers to the timestamp, θ i It refers to the cosine angle of the normalized amplitude; The correlation coefficient between time intervals is obtained by triangulation between points. The correlation coefficient is calculated using the cosine of the angle between vectors. The correlation coefficient between time intervals i and j is expressed as cos(φ). i +φ j To calculate φ i φ j The cosine angles of the normalized amplitudes at times i and j are respectively; The matrix of the Gramm-angle field is defined as G, as shown in equation (Ⅲ): Assuming each triaxial sensor data point of length m is converted into a GAF matrix of size m×m, the GAF matrix G of the triaxial sensor data is assembled into a three-channel image representation P = {G} of size m×m×3. x G y G z }; m is the length of the triaxial sensor data, G x G y G z These are the GAF matrices for the x-axis, y-axis, and z-axis data from the three-axis sensor.

3. The wearable sensor-video behavior recognition method based on cross-attention multimodal fusion wavelet knowledge distillation according to claim 1, characterized in that, Training a network of teachers, including: Wearable sensor data is converted into two-dimensional images, which are then fed into a pre-built teacher network for iterative training 100 times to obtain behavior classification results.

4. The wearable sensor-video behavior recognition method based on cross-attention multimodal fusion wavelet knowledge distillation according to claim 1, characterized in that, To train a student network by minimizing wavelet distillation loss and cross-entropy loss, the following steps are included: (1) Obtain the wavelet distillation loss of the teacher network and the student network; (2) Obtain the cross-entropy loss of the teacher network and the student network; (3) Train the student network using the gradient descent method with the total loss as the objective, and obtain a well-trained student network; The student network uses a BNInception network, a VGG network, or a ResNet network pre-trained on the ImageNet dataset.

5. The wearable sensor-video behavior recognition method based on cross-attention multimodal fusion wavelet knowledge distillation according to claim 4, characterized in that, Wavelet distillation loss L of teacher network and student network WKD The L1 loss between the outputs of the DWT modules of the teacher network and the student network is shown in Equation (XIV): In formula (XIV), Ψ S For the output of the wavelet transform layer of the student network, Let ||·||1 represent the L1 norm, where ||·||1 represents the output of the wavelet transform layer of teacher network i. Cross-entropy loss L of teacher network and student network CE As shown in equation (XV): In equation (XV), q c For the c-th class of real labels, P c S The probability of the corresponding behavior category c output by the softmax layer of the student network; The total loss L is as shown in equation (XVI): L=(1-α)L CE +αL WKD (XVI) In equation (XVI), α is a hyperparameter.

6. A wearable sensor-video behavior recognition system based on cross-attention multimodal fusion wavelet knowledge distillation, used to implement the wearable sensor-video behavior recognition method based on cross-attention multimodal fusion wavelet knowledge distillation as described in any one of claims 1-5, characterized in that, include: The wearable sensor data acquisition and processing module is configured to: acquire wearable sensor data and process it, converting the one-dimensional time series signal of the wearable sensor data into a two-dimensional image representation; The multimodal attention-dilated convolutional kernel residual network construction module is configured to: construct a multimodal attention-dilated convolutional kernel residual network; The teacher network training module is configured as: Training teacher networks; The student network setup module is configured to: set up a student network; The student network training module is configured to train the student network by minimizing wavelet distillation loss and cross-entropy loss. The recognition module is configured to recognize wearable sensor-video behavior through a trained student network and obtain recognition results.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the wearable sensor-video behavior recognition method based on cross-attention multimodal fusion wavelet knowledge distillation as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the wearable sensor-video behavior recognition method based on cross-attention multimodal fusion wavelet knowledge distillation as described in any one of claims 1-5.

Images