Human behavior recognition method based on fusion of multi-modal data acquired by unmanned aerial vehicle

Abstract

The invention provides a human behavior recognition method based on fusion of multi-modal data acquired by an unmanned aerial vehicle, and the method comprises the following steps: splicing a joint Tokens with a space CLS Tokens of the same frame after feature extraction is completed by using a space Transform, and obtaining a feature fusion module; performing structure coding on the joint point sequence based on a human anatomy structure, and constructing a human topological structure association module; the method comprises the following steps of: constructing a time sequence cross Transform module; the space Transform, the feature fusion module, the human body topological structure association module and the time sequence crossing Transform module are stacked to obtain a trunk feature extraction network; a 10-layer trunk feature extraction network is adopted to perform deep feature extraction on input early-stage fusion data and feature fusion data, and classification is performed by using a classification head; the features output by the backbone network are mapped to the dimensions with the same number as the behavior categories, a prediction score is output for each category, and a human body behavior recognition result is obtained according to the prediction scores. According to the method, the human body behavior in the multi-modal data acquired by the unmanned aerial vehicle can be effectively identified.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method for human behavior recognition based on the fusion of multimodal data collected by an unmanned aerial vehicle. Background Art

[0002] Human behavior recognition is an important research area in computer vision, with widespread applications in security, rescue, and transportation. While data collected by traditional fixed equipment can achieve high recognition accuracy, it suffers from limitations such as a single viewing angle, poor environmental adaptability, and weak model generalization. Furthermore, the equipment is expensive and requires fixed installation locations, making it difficult to scale up. Drones are low-cost and highly maneuverable, capable of carrying a variety of data acquisition devices to capture multimodal data. This data, which is diverse, rich in information, and complementary, can be fused to significantly improve the robustness and generalization of models, making it more suitable for practical applications. Therefore, human behavior recognition based on the fusion of multimodal data collected by drones is gaining increasing attention.

[0003] At present, human action recognition methods based on multimodal data fusion are mainly divided into three types: data-level fusion, feature-level fusion, and decision-level fusion. Data-level fusion aims to align data of different modalities in the data preprocessing stage and then perform data splicing, stacking, and other operations, which can maintain the integrity and authenticity of the original data. Prior art literature Jing Y, Wang F. Tp-vit: A two-pathway vision transformer for video action recognition [C] IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022: 2185-2189. In response to the problem that the single-modal input of the existing Transformer model is difficult to take into account the spatiotemporal features, a dual-pathway collaborative architecture based on multimodal input is proposed. This architecture splices the serialized skeleton tokens generated by human joints with the dual-path RGB visual tokens at the input layer to construct a multimodal data embedding and input it into the dual-pathway Transformer. The dual-pathway Transformer works collaboratively through two parallel processing paths: the slow path uses a high resolution (224×224) and a low frame rate (8 fps) configuration to capture fine static spatial details, while the fast path uses a low resolution (112×112) and a high frame rate (32 fps) configuration to effectively extract dynamic temporal information. The two paths share encoder parameters to reduce computational redundancy. However, this strategy of only achieving shallow interaction through data splicing does not fully explore deep cross-modal semantic associations. While the strategy of sharing encoder parameters reduces computational redundancy, it also limits the ability to learn the differentiating features of different modalities, affecting the collaborative modeling of spatiotemporal relationships.

[0004] Feature-level fusion achieves multimodal feature interaction through neural network intermediate layers. It uses feature concatenation, mapping, or attention mechanisms to integrate deep semantic associations across modalities and construct a joint feature representation, effectively capturing cross-modal complementary information. Prior Art Reference: Das S, Sharma S, Dai R, et al. VPN: Learning video-pose embedding for activities of daily life [C] / / European conference on computer vision. Cham: Springer International Publishing, 2020: 72-90. To improve fine-grained action recognition, a video-pose embedding network (VPN) is proposed. This network achieves cross-modal feature complementarity by jointly modeling RGB video and 3D skeleton data. For video streams, an I3D network is used as the visual backbone to extract spatiotemporal features, and a spatiotemporal attention coupler is designed to dynamically focus on key regions. For joint point streams, a graph convolutional network (GCN) is used to model the topological relationships between human joints and generate highly semantic pose features. A spatial embedding module maps the two feature types into a unified semantic space, strengthening modal alignment and interaction. A normalized Euclidean loss is introduced to optimize embedding consistency. However, the superposition of graph convolutional networks and attention mechanisms results in insufficient model integration, which increases the number of redundant parameters in the network structure. At the same time, relying solely on spatial embedding mapping for feature fusion limits the utilization of complementary information.

[0005] Decision-level fusion involves independently extracting and classifying features from each modal data, then integrating the decision results at the model output layer through weighted averaging, voting mechanisms, or probabilistic fusion to generate the final classification result. Prior Art: Ardianto S, Hang H M. Multi-view and multi-modal action recognition with learned fusion [C] / / 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). IEEE, 2018: 1601-1604. To improve the robustness of multimodal action recognition, a multimodal fusion framework based on the Temporal Segmentation Network (TSN) was proposed. This framework integrates multiple sources of information, including RGB, depth, infrared, and optical flow, by extending the data fusion stage. For multi-view feature fusion, a learned weight allocation network (Learned Fusion Net) was designed to dynamically aggregate spatiotemporal features from different viewpoints, effectively addressing the problem of single-view misclassification. However, this approach relies on independent training of multiple branches and a complex fusion strategy, resulting in high computational overhead.

[0006] In summary, based on the characteristics of multimodal human behavior data collected by drones, building an integrated fusion processing framework to improve the real-time and robustness of the network has important practical significance and research value. Summary of the Invention

[0007] In view of this, the present invention discloses a human behavior recognition method based on the fusion of multimodal data collected by drones, so as to effectively identify human behavior in multimodal data collected by drones.

[0008] The technical solution provided by the present invention is a method for human behavior recognition based on the fusion of multimodal data collected by drones, comprising the following steps:

[0009] S1: Obtain the brightness weights of RGB data and infrared data through the brightness decoupling and perception module, and embed them into the input sequence respectively. Then, perform a preliminary fusion of the two modalities as the network input;

[0010] S2: Multi-frame encoding of the joint point sequence is performed using a multi-layer perceptron to generate joint tokens, which are then concatenated with the spatial CLS tokens of the same frame after feature extraction using a spatial transformer to obtain a feature fusion module.

[0011] S3: Structural encoding of the joint sequence based on the human anatomical structure is performed, and it is added to the attention matrix after feature fusion to construct a human topological structure association module;

[0012] S4: Group the fused features of S3 along the time dimension and perform average pooling and maximum pooling respectively to generate the time difference correlation matrix and construct the time series cross Transformer module;

[0013] S5: Stack the spatial Transformer, feature fusion module, human topology association module, and temporal cross-Transformer module to obtain the backbone feature extraction network;

[0014] S6: A 10-layer backbone feature extraction network is used to perform deep feature extraction on the input preliminary fusion data and feature fusion data, and a classification head is used for classification. The features output by the backbone network are mapped to the same dimension as the number of behavior categories, and a prediction score is output for each category. The human behavior recognition result is obtained based on the prediction score.

[0015] Preferably, S1 specifically includes:

[0016] S11: Perform spatial conversion on the RGB data to obtain the corresponding HSL color space, then perform channel decoupling on the HSL color space to retain the brightness channel L, and stack the RGB three channels with the corresponding brightness channel L to obtain the input data of the brightness perception network;

[0017] S12: The input data obtained in S11 is divided into strong and weak categories according to the brightness value and the data is labeled. The data is then sent to a brightness perception network consisting of two 3*3 convolutional layers and an average pooling layer to extract brightness features. After that, two fully connected layers and a softmax function are used to obtain the strong / weak classification probability in the range of 0-1.

[0018] S13: The Softmax value of the strong category obtained in S12 is used as a continuous brightness value and input into the gating function. The nonlinear weight curve is constructed by using the continuity of the Softmax probability and the gating function of the exponential function to achieve adaptive weight distribution. The gating function is shown in formula (1);

[0019]

[0020] Where l is the continuous brightness value, α and β are smooth transition coefficients, which are used to adjust the smoothness of the weight in the brightness transition area l≈0.5; the output of the gate function is used as the weight of the RGB mode, that is, w=w RGB , then the infrared modal weight is w IR =1-w RGB .

[0021] Preferably, the step of performing multi-frame encoding of the joint point sequence by a multi-layer perceptron to generate joint tokens in S2 includes:

[0022] Use the human pose estimation algorithm to extract human joint point data from RGB sequences and infrared sequences, and stack multiple frames of data in chronological order to generate a joint point sequence;

[0023] The joint point sequence is multi-frame encoded through a multi-layer perceptron to generate joint tokens with the same dimension as the spatial CLS Tokens feature.

[0024] Preferably, S3 specifically includes:

[0025] S31: Based on the topological structure of the human skeleton, the joints are abstracted as nodes in the graph structure. The edges of adjacent joints are assigned a distance value of 1. The distances of non-adjacent joints are calculated by accumulating the shortest paths to generate a physical distance matrix S, where the elements S i,j represents the anatomical distance between joints i and j;

[0026] S32: Take the physical distance matrix S generated in step S31 as prior knowledge, and for each distance value d∈{0,1,2,…,D max}, assigning a learnable scalar parameter θ d , generating the parameter vector The elements of the structural encoding matrix B are generated by table lookup Thus, each element in the structural encoding matrix B is indexed by the anatomical distance of the corresponding joint pair to a scalar value in the parameter vector θ; through training, the model automatically adjusts the importance of each distance component in θ;

[0027] S33: In the standard self-attention mechanism, the attention weight matrix is ​​calculated by the dot product of the query and the key to obtain semantic similarity. After introducing the structural encoding matrix B, this matrix is ​​directly superimposed on the dot product result as a bias term to correct the calculation of the attention weight;

[0028] A i,j It is represented as the (i, j)th element of each element A of the attention score matrix. The calculation formula of the attention weight associated with the human topology structure is shown in formula (2):

[0029]

[0030] Among them, Q i and represent the i-th element of the query matrix Q and the j-th element of the key matrix K, respectively. The content correlation between the feature vectors of response joints i and j, d k is the dimension of the key matrix k, whose square root is used to scale the dot product result, B i,j The elements of the structure encoding matrix B are directly added to the attention score as a bias term, so that the model is subject to the hard constraints of anatomical structure while paying attention to semantic similarity.

[0031] Preferably, S4 includes:

[0032] S41: Compress the input feature channel to 1 / 5 and transpose the time dimension to the first position to obtain the time dimension feature after time and space separation Where T is the time dimension, is the number of channels after dimensionality reduction, V is the number of human joints;

[0033] S42: The time dimension feature X′ obtained in S41 T Divided into two groups along the channel dimension Among them, split2 means that the channel dimension Equally divided into two groups, the number of channels in each group becomes but

[0034] right Perform average pooling and maximum pooling respectively to generate the global time series trend vector and the local saliency vector Among them, pool a (·),pool m (·) are global average pooling and maximum pooling operations, respectively, which compress the features of each time frame into a scalar to obtain pool a , linear(·) is a fully connected layer that adjusts the feature distribution, and σ(·) is a sigmoid operation that normalizes the weights to [0, 1];

[0035] S43: The local time series trend vector Q obtained in S42 t and the local saliency vector K t Perform the difference attention dot product operation to generate the difference perception matrix. The calculation formula of the difference perception matrix is ​​shown in formula (3);

[0036] A T =sigmoid(atten(Q T -K t )) (3)

[0037] in, It is a difference perception item, and the difference is automatically expanded into a T×T matrix, which represents the difference weight between frames.

[0038] Preferably, the classification head in S6 is composed of two fully connected layers and a Softmax function.

[0039] The invention proposes a method for human behavior recognition based on the fusion of multimodal data collected by drones. This method pre-fuse the RGB and infrared sequences collected by drones through brightness weight embedding, linear mapping, and position embedding. Joint features are then used for feature fusion with multimodal features extracted from the data. While maintaining the integrity and authenticity of the original data, the model adapts to the brightness of different modalities and compensates for the insufficient feature extraction problem of a single data source. Simultaneously, the invention also constructs a human topology association module and a temporal cross-Transformer module, thereby enhancing the model's action semantic modeling capabilities and temporal difference perception capabilities, effectively extracting the spatiotemporal features of the multimodal human behavior data collected by drones.

[0040] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0043] Figure 1 This is a diagram showing the overall framework of a method for human behavior recognition based on the fusion of multimodal data collected by drones, provided in the embodiments disclosed in the present invention;

[0044] Figure 2 A schematic diagram of a brightness decoupling and perception module provided in an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of a human body topology association module provided in an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of a time-series crossover Transformer module provided in an embodiment of the present invention;

[0047] Figure 5 An example diagram of some multimodal data of the UAV-Human dataset provided in the embodiments disclosed in the present invention;

[0048] Figure 6 This is the confusion matrix of some behavior recognition results of the UAV-Human dataset provided by the disclosed embodiment of the present invention. DETAILED DESCRIPTION

[0049] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of systems consistent with certain aspects of the present invention, as detailed in the appended claims.

[0050] To solve the problems of insufficient fusion network integration and poor adaptability and robustness of multimodal human behavior data in practical applications, this embodiment provides a human behavior recognition method based on the fusion of multimodal data collected by drones, including the following steps:

[0051] S1: Use the DJI Matrice 100 drone platform equipped with a Microsoft Azure Kinect DK camera to obtain RGB data and infrared data, both of which are saved in the form of video frames.

[0052] The brightness decoupling and perception modules obtain the brightness weights of RGB and infrared data and embed them into the input sequence, performing a preliminary fusion of the two modalities as the network input. The input sequence is obtained through video segmentation, linear mapping, and position embedding.

[0053] The brightness decoupling and perception module spatially transforms the RGB sequence to obtain the HSL color space, decouples its channels, and retains the brightness channel L. The three RGB channels and the brightness channel L are fed into a brightness perception network consisting of two 3*3 convolutional layers and an average pooling layer to extract brightness features. The network then passes through two fully connected layers and a Softmax function to output strong / weak classification probabilities. The probability of the strong category is used as the brightness value, and finally, a gating function is used to calculate the brightness weight. These are then embedded into the input sequence obtained through video segmentation, linear mapping, and position embedding, and the two modalities are pre-fused and used as network input.

[0054] Specifically, S1 includes:

[0055] S11: Perform spatial conversion on the RGB data to obtain the corresponding HSL color space, then perform channel decoupling on the converted HSL color space to retain the brightness channel L, and stack the RGB three channels with the corresponding brightness channel L to obtain the input data of the brightness perception network;

[0056] S12: The input data is divided into strong and weak categories according to the brightness value and the data is labeled. It is then sent to the brightness perception network composed of two 3*3 convolution and average pooling layers to extract brightness features. After that, two fully connected layers and a softmax function are used to obtain the classification probability in the range of 0-1.

[0057] S13: The Softmax value of the strong category obtained in S12 is used as a continuous brightness value and input into the gating function. The nonlinear weight curve is constructed by using the continuity of the Softmax probability and the gating function with the introduction of the exponential function to achieve adaptive weight distribution. The gating function is shown in formula (1).

[0058]

[0059] Where l is the continuous brightness value, α and β are smooth transition coefficients, which are used to adjust the smoothness of the weight in the brightness transition area (l≈0.5). The output of the gate function is used as the weight of the RGB mode, that is, w=w RGB , then the infrared modal weight is w IR =1-w RGB .

[0060] S2: Multi-frame encoding of the joint point sequence is performed using a multi-layer perceptron to generate joint tokens, which are then concatenated with the spatial CLS tokens of the same frame after feature extraction using a spatial transformer to achieve feature fusion.

[0061] A human body pose estimation algorithm is used to extract human joint data from RGB and infrared sequences, and multiple frames of data are stacked in chronological order to generate a joint sequence. The joint sequence is then encoded using a multi-layer perceptron to generate joint tokens with the same dimension as the spatial CLS Tokens feature. Each spatial CLS Token is then concatenated with the joint tokens of its corresponding frame in the feature dimension to achieve feature fusion. The spatial CLS Tokens described refer to the process of self-attention of the spatial Transformer, which gradually aggregates the global spatial information of each frame through the interaction of all tokens, and encodes the spatial distribution pattern and contextual association features of objects in each frame using a single vector.

[0062] Specifically, S2 includes:

[0063] S21: Use the human body posture estimation algorithm to extract the human body joint point data in the RGB sequence and infrared sequence. The joint point data of each frame is saved in the form of three-dimensional coordinates. The number of joint points of each human body is 17, that is, the i-th frame is represented as The acquired multi-frame joint point data are stacked in time order to generate a joint point sequence of F frames, which is represented as {S1,…,S F};

[0064] S22: Multi-frame encoding of joint point sequences through multi-layer perceptron Generate joint tokens of the same dimension (D dimension) as the spatial CLS tokens features generated by the input sequence after passing through the spatial transformer, which contain the global spatial information of each frame image;

[0065] S23: Concatenate each spatial CLS Tokens with the joint Tokens of the corresponding frame in the feature dimension to achieve feature fusion.

[0066] S3: Structural encoding of joint point sequences based on human anatomy is performed, and this encoding is combined with the attention matrix after feature fusion to construct a human topology association module that can focus on human physical associations and improve the model's ability to capture semantic motion.

[0067] Structural encoding maps the path length of each joint pair into a learnable scalar bias term based on the topological connection relationship between joints in human anatomy, and adds it to the attention score when calculating the attention matrix. This constructs a human topological structure association module that can guide the model to strengthen the attention distribution of anatomically related joints during feature fusion and enhance the ability of action semantic modeling.

[0068] Specifically include:

[0069] S31: Based on the topological structure of the human skeleton, the joints are abstracted as nodes in the graph structure. The edges of adjacent joints are assigned a distance value of 1. The distances of non-adjacent joints are calculated by accumulating the shortest paths to generate a physical distance matrix S, where the elements S i,j represents the anatomical distance between joints i and j;

[0070] S32: Take the physical distance matrix S generated in step S31 as prior knowledge, and for each distance value d∈{0,1,2,…,D max}, assigning a learnable scalar parameter θ d , generating the parameter vector The elements of the structural encoding matrix B are generated by table lookup Thus, each element in the structural encoding matrix B is indexed by the anatomical distance of the corresponding joint pair to the scalar value in the parameter vector θ. Through training, the model can automatically adjust the importance of each distance component in θ;

[0071] S33: In the standard self-attention mechanism, the attention weight matrix is ​​calculated by the dot product of query and key to obtain semantic similarity. After the structural encoding matrix B is introduced, the matrix is ​​directly superimposed on the dot product result as a bias term to correct the calculation of attention weight. i,jThe (i, j)th element of each element A of the attention score matrix is ​​expressed as follows: the calculation formula of the attention weight associated with the human topology structure is shown in formula (2).

[0072]

[0073] Among them, Q i and represent the i-th element of the query matrix Q and the j-th element of the key matrix K, respectively. The content correlation between the feature vectors of response joints i and j, d k is the dimension of the key matrix k, whose square root is used to scale the dot product result, B i,j The elements of the structure encoding matrix B are directly added to the attention score as a bias term, so that the model is subject to the hard constraints of anatomical structure while paying attention to semantic similarity.

[0074] S4: Group the features output by S3 along the time dimension and perform average pooling and maximum pooling respectively to generate query keys and key keys. Use QK cross attention to generate a time difference perception matrix to capture the difference between the time point and the global average state, and construct a time series cross Transformer module.

[0075] The input features are divided into two groups along the time dimension. One group undergoes average pooling to generate query keys, which extract global features of the time series and reflect the overall trend of the action. The other group undergoes max pooling to generate keys, which capture local extreme features of the time series and highlight keyframe information. Finally, the query keys and keys are input into the cross-attention method to generate a temporal difference perception matrix, which is then constructed into a time-series cross-Transformer module.

[0076] Specifically include:

[0077] The features output in step 3 are grouped along the time dimension and average pooled and max pooled respectively to generate a time difference correlation matrix, which is then constructed into a time series cross-Transformer module. The steps of the time series cross-Transformer are:

[0078] S41: Compress the input feature channel to 1 / 5 and transpose the time dimension to the first position to obtain the time dimension feature after time and space separation Where T is the time dimension, is the number of channels after dimensionality reduction, V is the number of human joints;

[0079] S42: The time dimension feature X′ obtained in S31 T Divided into two groups along the channel dimension Among them, split2 means that the channel dimension Equally divided into two groups, the number of channels in each group becomes but Then, yes Perform average pooling and maximum pooling respectively to generate the global time series trend vector and the local saliency vector Among them, pool a (·),pool m (·) are global average pooling and maximum pooling operations, respectively, which compress the features of each time frame into a scalar to obtain pool a , linear(·) is a fully connected layer that adjusts the feature distribution, and σ(·) is a sigmoid operation that normalizes the weights to [0, 1];

[0080] S43: The local time series trend vector Q obtained in S42 t and the local saliency vector K t Perform the difference attention dot product operation to generate the difference perception matrix. The calculation formula of the difference perception matrix is ​​shown in formula (3).

[0081] A T =sigmoid(atten(Q T -K t )) (3)

[0082] in, It is a difference perception item, and the difference is automatically expanded into a T×T matrix, which represents the difference weight between frames.

[0083] S5: Stack the spatial Transformer and feature fusion module with the human topology structure association module and the temporal cross Transformer to obtain the backbone feature extraction network;

[0084] The backbone feature extraction network is composed of a spatial Transformer module, a feature fusion module, a human topology structure association module, and a temporal cross Transformer module stacked in sequence.

[0085] S6: The multimodal data is fed into a 10-layer backbone network through pre-fusion and feature fusion methods. Deep feature extraction and classification are performed using a classification head to obtain human action recognition results. The classification head, consisting of two fully connected layers and a softmax function, maps the features output by the backbone network to the same number of action categories and outputs a prediction score for each category. The human action recognition results are obtained based on the prediction scores.

[0086] In order to demonstrate the effectiveness of the proposed method, a large-scale multimodal dataset UAV-Human for UAV behavior understanding was used for verification. The dataset contains 67,428 annotated video sequences of 119 objects, covering 155 types of actions, and includes 6 modal data types: RGB, joints, depth, infrared, fisheye, and night vision. The data covers 45 scenes, including complex conditions such as day and night, wind and rain, and multiple flight postures of UAVs. Figure 6 As shown in Figure 3, the partial confusion matrix obtained from the test shows that the method proposed in this invention can effectively identify human behavior in multimodal data collected by drones.

[0087] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These changes and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A human behavior recognition method based on multimodal data fusion collected by drones, characterized by: The steps include: S1: Obtain the brightness weights of RGB data and infrared data through the brightness decoupling and perception module, and embed them into the input sequence respectively, and then perform preliminary fusion of the two modalities as network input; S2: Multi-frame encoding of the joint point sequence is performed using a multi-layer perceptron to generate joint tokens, which are then concatenated with the spatial CLS tokens of the same frame after feature extraction using a spatial transformer to obtain a feature fusion module. S3: Structural encoding of the joint sequence based on the human anatomical structure is performed, and it is added to the attention matrix after feature fusion to construct a human topological structure association module; S4: Group the fused features of S3 along the time dimension and perform average pooling and maximum pooling respectively to generate the time difference correlation matrix and construct the time series cross Transformer module; S5: Stack the spatial Transformer, feature fusion module, human topology association module, and temporal cross-Transformer module to obtain the backbone feature extraction network; S6: A 10-layer backbone feature extraction network is used to perform deep feature extraction on the input preliminary fusion data and feature fusion data, and a classification head is used for classification. The features output by the backbone network are mapped to the same dimension as the number of behavior categories, and a prediction score is output for each category. The human behavior recognition result is obtained based on the prediction score.

2. The human behavior recognition method based on multimodal data fusion collected by drone according to claim 1 is characterized in that: S1 specifically includes: S11: Perform spatial conversion on the RGB data to obtain the corresponding HSL color space, then perform channel decoupling on the HSL color space to retain the brightness channel L, and stack the RGB three channels with the corresponding brightness channel L to obtain the input data of the brightness perception network; S12: The input data obtained in S11 is divided into strong and weak categories according to the brightness value and the data is labeled. The data is then sent to a brightness perception network consisting of two 3*3 convolutional layers and an average pooling layer to extract brightness features. After that, two fully connected layers and a softmax function are used to obtain the strong / weak classification probability in the range of 0-1. S13: The Softmax value of the strong category obtained in S12 is used as a continuous brightness value and input into the gating function. The nonlinear weight curve is constructed by using the continuity of the Softmax probability and the gating function of the exponential function to achieve adaptive weight distribution. The gating function is shown in formula (1); Where l is the continuous brightness value, α and β are smooth transition coefficients, which are used to adjust the smoothness of the weight in the brightness transition area l≈0.5; the output of the gate function is used as the weight of the RGB mode, that is, w=w RGB , then the infrared modal weight is w IR =1-w RGB .

3. The human behavior recognition method based on multimodal data fusion collected by drone according to claim 1 is characterized in that: As described in S2, the multi-layer perceptron is used to encode the joint point sequence into multiple frames to generate joint tokens, including: Use the human pose estimation algorithm to extract human joint point data from RGB sequences and infrared sequences, and stack multiple frames of data in chronological order to generate a joint point sequence; The joint point sequence is multi-frame encoded through a multi-layer perceptron to generate joint tokens with the same dimension as the spatial CLS Tokens feature.

4. The human behavior recognition method based on multimodal data fusion collected by drone according to claim 1 is characterized in that: S3 specifically includes: S31: Based on the topological structure of the human skeleton, the joints are abstracted as nodes in the graph structure. The edges of adjacent joints are assigned a distance value of 1. The distances of non-adjacent joints are calculated by accumulating the shortest paths to generate a physical distance matrix S, where the elements S i,j represents the anatomical distance between joints i and j; S32: Take the physical distance matrix S generated in step S31 as prior knowledge, and for each distance value d∈{0,1,2,…,D max }, assigning a learnable scalar parameter θ d , generating the parameter vector The elements of the structural encoding matrix B are generated by table lookup Thus, each element in the structural encoding matrix B is indexed by the anatomical distance of the corresponding joint pair to a scalar value in the parameter vector θ; through training, the model automatically adjusts the importance of each distance component in θ; S33: In the standard self-attention mechanism, the attention weight matrix is ​​calculated by the dot product of the query and the key to obtain semantic similarity. After introducing the structural encoding matrix B, this matrix is ​​directly superimposed on the dot product result as a bias term to correct the calculation of the attention weight; A i,j It is represented as the (i, j)th element of each element A of the attention score matrix. The calculation formula of the attention weight associated with the human topology structure is shown in formula (2): Among them, Q i and represent the i-th element of the query matrix Q and the j-th element of the key matrix K, respectively. The content correlation between the feature vectors of response joints i and j, d k is the dimension of the key matrix k, whose square root is used to scale the dot product result, B i,j The elements of the structure encoding matrix B are directly added to the attention score as a bias term, so that the model can focus on semantic similarity while being subject to the hard constraints of anatomical structure.

5. The human behavior recognition method based on multimodal data fusion collected by drone according to claim 1 is characterized in that S4 include: S41: Compress the input feature channel to 1 / 5 and transpose the time dimension to the first position to obtain the time dimension feature after time and space separation Where T is the time dimension, is the number of channels after dimensionality reduction, V is the number of human joints; S42: The time dimension feature X′ obtained in S41 T Divided into two groups along the channel dimension Among them, split2 means that the channel dimension Equally divided into two groups, the number of channels in each group becomes but right Perform average pooling and maximum pooling respectively to generate the global time series trend vector and the local saliency vector Among them, pool a (·),pool m (·) are global average pooling and maximum pooling operations, respectively, which compress the features of each time frame into a scalar and obtain linear(·) is a fully connected layer that adjusts the feature distribution, and σ(·) is a Sigmoid operation that normalizes the weights to [0, 1]; S43: The local time series trend vector Q obtained in S42 t and the local saliency vector K t Perform the difference attention dot product operation to generate the difference perception matrix. The calculation formula of the difference perception matrix is ​​shown in formula (3); A T =sigmoid(atten(Q T -K t )) (3) in, It is a difference perception item, and the difference is automatically expanded into a T×T matrix, which represents the difference weight between frames.

6. The human behavior recognition method based on multimodal data fusion collected by drone according to claim 1 is characterized in that: The classification head described in S6 consists of two fully connected layers and a Softmax function.

Images