Human behavior recognition method based on unmanned aerial vehicle collected multi-modal data fusion

Abstract

This invention provides a method for human behavior recognition based on the fusion of multimodal data collected by UAVs, comprising the following steps: concatenating joint tokens with spatial CLS tokens extracted using a spatial Transformer in the same frame to obtain a feature fusion module; structurally encoding the joint sequence based on human anatomical structure to construct a human topology association module; constructing a temporal cross-Transformer module; stacking the spatial Transformer, feature fusion module, human topology association module, and temporal cross-Transformer module to obtain a backbone feature extraction network; employing a 10-layer backbone feature extraction network to perform deep feature extraction on the input pre-fusion data and feature fusion data, and using a classification head for classification; mapping the features output by the backbone network to a dimension equal to the number of behavior categories, and outputting a prediction score for each category; and obtaining the human behavior recognition result based on the prediction score. This method can effectively recognize human behavior in multimodal data collected by UAVs.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for human behavior recognition based on the fusion of multimodal data collected by unmanned aerial vehicles. Background Technology

[0002] Human behavior recognition is an important research direction in computer vision, with wide applications in security, rescue, and transportation. While data collected by traditional fixed equipment can achieve high recognition accuracy, it suffers from limitations such as a single perspective, poor environmental adaptability, and weak model generalization ability. Furthermore, the high cost and fixed installation locations of these devices hinder large-scale application. Unmanned aerial vehicles (UAVs), on the other hand, are low-cost and highly mobile, capable of carrying various data acquisition devices to obtain multimodal data. This data is diverse in environment, rich in information, and complementary. Through fusion processing, the robustness and generalization ability of the model can be significantly improved, making it more suitable for practical applications. Therefore, human behavior recognition based on the fusion of multimodal data collected by UAVs is attracting increasing attention.

[0003] Currently, human behavior recognition methods based on multimodal data fusion are mainly divided into three types: data-level fusion, feature-level fusion, and decision-level fusion. Data-level fusion aims to align data from different modalities during the data preprocessing stage and then perform data splicing, stacking, and other operations, which can maintain the integrity and authenticity of the original data. Existing technical literature Jing Y, Wang F. Tp-vit: A two-pathway vision transformer for video action recognition [C] IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022: 2185-2189. To address the problem that existing Transformer models cannot take into account spatiotemporal features with a single modal input, a dual-pathway collaborative architecture based on multimodal input is proposed. This architecture splices the serialized skeleton tokens generated from human joints with dual-pathway RGB visual tokens at the input layer to construct multimodal data embedding and input it into the dual-pathway Transformer. The dual-path Transformer works collaboratively through two parallel processing paths: the slow path uses a high resolution (224×224) and low frame rate (8 frames) configuration to capture fine static spatial details, while the fast path uses a low resolution (112×112) and high frame rate (32 frames) configuration to effectively extract dynamic temporal information. The two paths share encoder parameters to reduce computational redundancy. However, this approach only achieves shallow interaction through data concatenation and does not fully explore deep semantic relationships across modalities. Furthermore, while the strategy of sharing encoder parameters reduces computational redundancy, it limits the learning ability of different modal features, affecting the collaborative modeling effect of spatiotemporal relationships.

[0004] Feature-level fusion achieves multimodal feature interaction through intermediate layers of neural networks. It integrates deep semantic associations of different modalities by using feature concatenation, mapping, or attention mechanisms to construct joint feature representations, which can effectively capture cross-modal complementary information. Existing technical literature Das S, Sharma S, Dai R, et al. Vpn: Learning video-pose embedding for activities of daily living[C] / / European conference on computer vision.Cham:Springer International Publishing,2020:72-90. To improve the ability to recognize fine-grained actions, a video-pose embedding network (VPN) is proposed. It achieves cross-modal feature complementarity by jointly modeling RGB video and 3D skeleton data. For video streams, the I3D network is used as the visual backbone to extract spatiotemporal features, and a spatiotemporal attention coupler is designed to dynamically focus on key regions. For joint streams, graph convolutional networks (GCN) are used to model the topological relationship of human joints to generate high semantic pose features. The two types of features are mapped to a unified semantic space through a spatial embedding module to enhance modal alignment and interaction, and normalized Euclidean loss is introduced to optimize embedding consistency. However, the combined use of graph convolutional networks and attention mechanisms leads to insufficient model integration, resulting in an increase in redundant parameters in the network structure. At the same time, relying solely on spatial embedding mapping for feature fusion limits the utilization of complementary information.

[0005] Decision-level fusion refers to independently extracting and classifying features from each modality, and then integrating the decision results at the model output layer through weighted averaging, voting mechanisms, or probability fusion to generate the final classification result. Existing technical literature includes Ardianto S, Hang H M. Multi-view and multi-modal action recognition with learned fusion[C] / / 2018Asia-Pacific Signal and Information ProcessingAssociation Annual Summit and Conference(APSIPA ASC).IEEE,2018:1601-1604. To improve the robustness of multimodal action recognition, a multimodal fusion framework based on a temporal segmentation network (TSN) is proposed, integrating multi-source information such as RGB, depth, infrared, and optical flow by extending the data fusion stage. For multi-view feature fusion, a learned fusion net is designed to dynamically aggregate spatiotemporal features from different perspectives, effectively solving the problem of misjudgment from a single perspective. However, this method relies on independent training of multiple branches and complex fusion strategies, resulting in high computational overhead.

[0006] In summary, based on the characteristics of multimodal human behavior data collected by UAVs, constructing an integrated fusion processing framework to improve the real-time performance and robustness of the network has significant practical implications and research value. Summary of the Invention

[0007] In view of this, the present invention discloses a method for human behavior recognition based on the fusion of multimodal data collected by UAVs, so as to effectively identify human behavior in multimodal data collected by UAVs.

[0008] The technical solution provided by this invention, a human behavior recognition method based on the fusion of multimodal data collected by unmanned aerial vehicles, includes the following steps:

[0009] S1: Obtain the brightness weights of RGB and infrared data through brightness decoupling and sensing modules, embed them into the input sequence, and perform preliminary fusion of the two modes as network input;

[0010] S2: Multi-frame encoding of the joint sequence using a multi-layer perceptron generates joint tokens, which are then concatenated with the spatial CLS tokens extracted by the spatial Transformer in the same frame to obtain the feature fusion module.

[0011] S3: Based on the human anatomical structure, the joint sequence is structurally encoded, and then added to the attention matrix after feature fusion to construct a human topological structure association module;

[0012] S4: The fused features of S3 are grouped along the time dimension and average pooling and max pooling are performed respectively to generate a time difference correlation matrix, and a time-series cross-transformer module is constructed.

[0013] S5: Stack the spatial Transformer, feature fusion module, human topology association module, and temporal cross Transformer module to obtain the backbone feature extraction network;

[0014] S6: A 10-layer backbone feature extraction network is used to perform deep feature extraction on the input pre-fusion data and feature fusion data, and a classification head is used for classification. The features output by the backbone network are mapped to the same dimension as the number of behavior categories, and a prediction score is output for each category. The human behavior recognition result is obtained based on the prediction score.

[0015] Preferably, S1 specifically includes:

[0016] S11: Perform space conversion on the RGB data to obtain the corresponding HSL color space, then decouple the channels of the HSL color space and retain the luminance channel L. Stack the three RGB channels with the corresponding luminance channel L to obtain the input data of the luminance sensing network.

[0017] S12: Based on the brightness value, the input data obtained in S11 is divided into two categories, strong and weak, and the data is labeled. Then, it is fed into a brightness perception network consisting of two 3*3 convolutional layers and an average pooling layer to extract brightness features. After that, the strong / weak classification probability in the 0-1 interval is obtained through two fully connected layers and the Softmax function.

[0018] S13: The Softmax value of the strong category obtained in S12 is used as the continuous brightness value and input into the gating function. The continuousness of the Softmax probability and the gating function with the introduction of the exponential function are used to construct a nonlinear weight curve to realize the adaptive allocation of weight. The gating function is shown in Equation (1).

[0019]

[0020] Where l is the continuous luminance value, α, β are smooth transition coefficients used to adjust the smoothness of the weights in the luminance transition region l≈0.5; the output of the gating function is used as the weight of the RGB mode, i.e., w = w RGB Then the infrared mode weight is w IR =1-w RGB .

[0021] Preferably, the step S2, which involves multi-frame encoding of the joint sequence using a multilayer perceptron to generate joint tokens, includes:

[0022] Human pose estimation algorithm is used to extract human joint point data from RGB and infrared sequences, and multiple frames of data are stacked in chronological order to generate joint point sequence.

[0023] By using a multilayer perceptron to encode the joint sequence across multiple frames, joint tokens with the same dimension as the spatial CLS tokens features are generated.

[0024] Preferably, S3 specifically includes:

[0025] S31: Based on the topological structure of the human skeleton, joints are abstracted as nodes in a graph structure. Edges between adjacent joints are assigned a distance value of 1, and the distances between non-adjacent joints are calculated by accumulating the shortest path distances, generating a physical distance matrix S, where elements S... i,j This represents the anatomical distance between joints i and j;

[0026] S32: Using the physical distance matrix S generated in step S31 as prior knowledge, for each distance value d∈{0,1,2,…,D max Assign a learnable scalar parameter θ d Generate parameter vector The elements of the structure encoding matrix B are generated by a lookup table. Thus, each element in the structure encoding matrix B is indexed by the anatomical distance of the corresponding joint pair to a scalar value in the parameter vector θ; through training, the model automatically adjusts the importance of each distance component in θ;

[0027] S33: In the standard self-attention mechanism, the attention weight matrix is ​​calculated by the dot product of the query and the key to obtain semantic similarity. After introducing the structure encoding matrix B, this matrix is ​​directly superimposed on the dot product result as a bias term to correct the calculation of attention weights.

[0028] A i,j Let A be the (i,j)th element of each element A in the attention score matrix. Then, the formula for calculating the attention weight associated with the human topology is shown in Equation (2):

[0029]

[0030] Among them, Q i and These represent the i-th element of the query matrix Q and the j-th element of the key matrix K, respectively. The correlation between the content of the feature vectors of joints i and j, d k Let B be the dimension of the key matrix k, and its square root is used to scale the dot product result. i,j The elements of the structural encoding matrix B are directly added to the attention score as bias terms, so that the model is subject to the hard constraints of anatomical structure while focusing on semantic similarity.

[0031] Preferably, S4 includes:

[0032] S41: Compress the input features to 1 / 5 of their channels and transpose the time dimension to the first position to obtain the time dimension features after spatiotemporal separation. Where T represents the time dimension. V represents the number of channels after dimensionality reduction, and V represents the number of human joints.

[0033] S42: The time dimension feature X′ obtained in S41 T Divided into two groups along the channel dimension Where split2 represents splitting the channel dimension Divide into two equal groups, with the number of channels in each group becoming but

[0034] right Global time series trend vectors are generated by performing average pooling and max pooling respectively. and local significance vector Among them, pool a (·),pool m (·) represents the global average pooling operation and the max pooling operation, respectively, which compress the features of each time frame into a scalar, resulting in the pool... a , linear(·) is a fully connected layer that adjusts the feature distribution, and σ(·) is a Sigmoid operation that normalizes the weights to [0,1].

[0035] S43: The local time-series trend vector Q obtained from S42 t and the local saliency vector K t Perform the difference attention dot product operation to generate the difference perception matrix. The calculation formula of the difference perception matrix is ​​shown in Equation (3).

[0036] A T =sigmoid(atten(Q) T -K t (3)

[0037] in, For the difference-aware term, the difference is automatically expanded into a T×T matrix to represent the inter-frame difference weights.

[0038] Preferably, the classification head in S6 consists of two fully connected layers and a Softmax function.

[0039] This invention proposes a human behavior recognition method based on the fusion of multimodal data collected by UAVs. The method first fuses RGB and infrared sequences collected by the UAV through brightness weight embedding, linear mapping, and position embedding. Then, it fuses keypoint features with the extracted multimodal features. While maintaining the integrity and authenticity of the original data, this method enables the model to adapt to brightness variations across different modalities and compensates for insufficient feature extraction from a single data source. Furthermore, this invention constructs a human topology association module and a temporal cross-transformer module to enhance the model's action semantic modeling ability and temporal difference perception ability, effectively extracting the spatiotemporal features of the multimodal human behavior data collected by the UAV.

[0040] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit the disclosure of the present invention. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is an overall framework diagram of a human behavior recognition method based on the fusion of multimodal data collected by UAV, provided by an embodiment of the present invention.

[0044] Figure 2 This is a schematic diagram of the brightness decoupling and sensing module provided in an embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram of the human body topology association module provided in an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of the timing crossover Transformer module provided in the embodiments of the present invention;

[0047] Figure 5 This is an example diagram of a portion of the multimodal data from the UAV-Human dataset provided in an embodiment of the present invention.

[0048] Figure 6 This is a confusion matrix of partial behavior recognition results from the UAV-Human dataset provided in the embodiments of this invention. Detailed Implementation

[0049] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of systems consistent with some aspects of the invention as detailed in the appended claims.

[0050] To address the shortcomings of integrated networks and the poor adaptability and robustness of multimodal human behavior data in practical applications, this implementation plan provides a human behavior recognition method based on the fusion of multimodal data collected by unmanned aerial vehicles (UAVs), comprising the following steps:

[0051] S1: Use the DJI Matrice 100 drone platform equipped with a Microsoft Azure Kinect DK camera to acquire RGB and infrared data, both of which are saved in the form of video frames.

[0052] The brightness weights of RGB and infrared data are obtained through a brightness decoupling and perception module and embedded into the input sequence. These two modalities are then fused in the initial stage and used as network input. The input sequence is obtained through video segmentation, linear mapping, and position embedding.

[0053] The luminance decoupling and perception module involves spatially transforming the RGB sequence to obtain the HSL color space, decoupling its channels, and retaining the luminance channel L. The RGB three channels and the luminance channel L are then fed into a luminance perception network consisting of two 3x3 convolutional layers and an average pooling layer to extract luminance features. Two fully connected layers and a Softmax function then output strong / weak classification probabilities, using the probability of the strong class as the luminance value. Finally, a gating function is used to calculate the luminance weights. These probabilities are then embedded into input sequences obtained through video segmentation, linear mapping, and positional embedding, respectively, to perform preliminary fusion of the two modalities, serving as the network input.

[0054] Specifically, S1 includes:

[0055] S11: Perform space conversion on the RGB data to obtain the corresponding HSL color space, then decouple the channels of the converted HSL color space and retain the luminance channel L. Stack the three RGB channels with the corresponding luminance channel L to obtain the input data of the luminance sensing network.

[0056] S12: Based on the brightness value, the obtained input data is divided into two categories, strong and weak, and the data is labeled. Then, it is fed into a brightness perception network consisting of two 3*3 convolutional layers and an average pooling layer to extract brightness features. After that, the classification probability of the 0-1 interval is obtained through two fully connected layers and the Softmax function.

[0057] S13: The Softmax value of the strong category obtained in S12 is used as a continuous brightness value and input into the gating function. The continuousness of the Softmax probability and the gating function with an exponential function are used to construct a nonlinear weight curve to achieve adaptive weight allocation. The gating function is shown in Equation (1).

[0058]

[0059] Where l is the continuous luminance value, and α and β are smoothing transition coefficients used to adjust the smoothness of the weights in the luminance transition region (l≈0.5). The output of the gating function is used as the weights for the RGB modes, i.e., w = w RGB Then the infrared mode weight is w IR =1-w RGB .

[0060] S2: Multi-frame encoding of the joint sequence is performed by a multi-layer perceptron to generate joint tokens. These tokens are then concatenated with the spatial CLS tokens extracted by the spatial Transformer in the same frame to achieve feature fusion.

[0061] Human pose estimation algorithms are used to extract human joint point data from RGB and infrared sequences. Multiple frames of data are stacked chronologically to generate a joint point sequence. Then, a multilayer perceptron is used to encode the joint point sequence across multiple frames, generating joint tokens with the same dimension as the spatial CLS tokens. Each spatial CLS token is then concatenated with the joint tokens of its corresponding frame along the feature dimension to achieve feature fusion. The spatial CLS tokens refer to the global spatial information of each frame image gradually aggregated through the interaction of all tokens during the self-attention process of the spatial Transformer, encoding the spatial distribution pattern and contextual features of objects within each frame image using a single vector.

[0062] Specifically, S2 includes:

[0063] S21: Human pose estimation algorithm is used to extract human joint point data from RGB and infrared sequences. Joint point data for each frame is saved in three-dimensional coordinates. Each human body has 17 joint points, i.e., the i-th frame is represented as... The acquired multi-frame keypoint data are stacked in chronological order to generate a keypoint sequence of F frames, denoted as {S1,…,S...}. F};

[0064] S22: Multi-frame encoding of the keypoint sequence using a multilayer perceptron. Generate joint tokens with the same dimension (D-dimensional) as the spatial CLS tokens features generated after the input sequence passes through a spatial Transformer and contains global spatial information of each frame of the image.

[0065] S23: Concatenate each spatial CLS Tokens with the corresponding frame's joint Tokens along the feature dimension to achieve feature fusion.

[0066] S3: Based on the human anatomical structure, the joint sequence is structurally encoded and added to the attention matrix after feature fusion to construct a human topological structure association module that can focus on the physical association of the human body and improve the model's ability to capture the semantics of action.

[0067] Structural encoding maps the path length of each joint pair to a learnable scalar bias term based on the topological connections between joints in human anatomy. This bias term is then added to the attention score when calculating the attention matrix, thus constructing a human topological structure association module that can guide the model to strengthen the attention distribution of anatomically related joints during feature fusion and enhance the ability to model action semantics.

[0068] Specifically, it includes:

[0069] S31: Based on the topological structure of the human skeleton, joints are abstracted as nodes in a graph structure. Edges between adjacent joints are assigned a distance value of 1, and the distances between non-adjacent joints are calculated by accumulating the shortest path distances, generating a physical distance matrix S, where elements S... i,j This represents the anatomical distance between joints i and j;

[0070] S32: Using the physical distance matrix S generated in step S31 as prior knowledge, for each distance value d∈{0,1,2,…,D max Assign a learnable scalar parameter θ d Generate parameter vector The elements of the structure encoding matrix B are generated by a lookup table. Thus, each element in the structure encoding matrix B is indexed by the anatomical distance of the corresponding joint pair to a scalar value in the parameter vector θ. Through training, the model can automatically adjust the importance of each distance component in θ;

[0071] S33: In the standard self-attention mechanism, the attention weight matrix is ​​calculated by the dot product of the query and the key to obtain semantic similarity. After introducing the structural encoding matrix B, this matrix is ​​directly superimposed on the dot product result as a bias term, correcting the calculation of the attention weights. (The rest of the text appears to be a continuation of the previous sentence and can be left as is.) i,jLet A be the (i,j)th element of each element A in the attention score matrix. Then, the formula for calculating the attention weight associated with the human topology is shown in Equation (2).

[0072]

[0073] Among them, Q i and These represent the i-th element of the query matrix Q and the j-th element of the key matrix K, respectively. The correlation between the content of the feature vectors of joints i and j, d k Let B be the dimension of the key matrix k, and its square root is used to scale the dot product result. i,j The elements of the structural encoding matrix B are directly added to the attention score as bias terms, so that the model is subject to the hard constraints of anatomical structure while focusing on semantic similarity.

[0074] S4: Group the features output by S3 along the time dimension and perform average pooling and max pooling respectively to generate query keys and key keys. Use QK cross attention to generate a time difference perception matrix to capture the difference between the time point and the global average state, and construct a time-series cross Transformer module.

[0075] The input features are divided into two groups along the time dimension. One group undergoes average pooling to generate a query key to extract global features of the time series and reflect the overall trend of the action. The other group undergoes max pooling to generate a key to capture local extreme features of the time series and highlight keyframe information. Finally, the query and key keys are input into cross-attention to generate a temporal difference-aware matrix, which is used to construct a temporal cross-transformer module.

[0076] Specifically, it includes:

[0077] The features output from step 3 are grouped along the time dimension and then subjected to average pooling and max pooling respectively to generate a time difference correlation matrix, which is then used to construct a temporal cross-transformer module. The steps of the temporal cross-transformer are as follows:

[0078] S41: Compress the input features to 1 / 5 of their channels and transpose the time dimension to the first position to obtain the time dimension features after spatiotemporal separation. Where T represents the time dimension. V represents the number of channels after dimensionality reduction, and V represents the number of human joints.

[0079] S42: The time dimension feature X′ obtained in S31 T Divided into two groups along the channel dimension Where split2 represents splitting the channel dimension Divide into two equal groups, with the number of channels in each group becoming but Then, to Global time series trend vectors are generated by performing average pooling and max pooling respectively. and local significance vector Among them, pool a (·),pool m (·) represents the global average pooling operation and the max pooling operation, respectively, which compress the features of each time frame into a scalar, resulting in the pool... a , linear(·) is a fully connected layer that adjusts the feature distribution, and σ(·) is a Sigmoid operation that normalizes the weights to [0,1].

[0080] S43: The local time-series trend vector Q obtained from S42 t and the local saliency vector K t A difference perception matrix is ​​generated by performing an interpolation attention dot product operation. The formula for calculating the difference perception matrix is ​​shown in equation (3).

[0081] A T =sigmoid(atten(Q) T -K t (3)

[0082] in, For the difference-aware term, the difference is automatically expanded into a T×T matrix to represent the inter-frame difference weights.

[0083] S5: Stack the spatial Transformer and feature fusion module with the human body topology association module and the temporal cross Transformer to obtain the backbone feature extraction network;

[0084] The backbone feature extraction network is composed of spatial Transformer module, feature fusion module, human body topology association module and temporal cross Transformer module stacked in sequence.

[0085] S6: Multimodal data is fed into a 10-layer backbone network through pre-fusion and feature fusion methods for deep feature extraction and classification using a classification head to obtain human behavior recognition results. The classification head consists of two fully connected layers and a Softmax function, which maps the features output by the backbone network to a dimension equal to the number of behavior categories and outputs a prediction score for each category. The human behavior recognition result is obtained based on the prediction score.

[0086] To demonstrate the effectiveness of the proposed method, the large-scale multimodal dataset UAV-Human, designed for understanding UAV behavior, was used for validation. This dataset contains 67,428 annotated video sequences representing 119 objects, covering 155 action categories and including six modalities: RGB, keypoint, depth, infrared, fisheye, and night vision. The data covers 45 scenarios, including complex conditions such as day / night cycles, wind, and rain, as well as various UAV flight attitudes. Figure 6 As shown in the test results, the partial confusion matrix indicates that the method proposed in this invention can effectively identify human behavior in multimodal data collected by UAVs.

[0087] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these changes and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for human behavior recognition based on the fusion of multimodal data collected by unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: S1: Obtain the brightness weights of RGB and infrared data through brightness decoupling and sensing modules, embed them into the input sequence, and perform preliminary fusion of the two modes as network input; S2: Multi-frame encoding of the joint sequence using a multi-layer perceptron generates joint tokens, which are then concatenated with the spatial CLS tokens extracted by the spatial Transformer in the same frame to obtain the feature fusion module. S3: Based on the human anatomical structure, the joint sequence is structurally encoded, and then added to the attention matrix after feature fusion to construct a human topological structure association module; S4: The fused features of S3 are grouped along the time dimension and average pooling and max pooling are performed respectively to generate a time difference correlation matrix, and a time-series cross-transformer module is constructed. S5: Stack the spatial Transformer, feature fusion module, human topology association module, and temporal cross Transformer module to obtain the backbone feature extraction network; S6: A 10-layer backbone feature extraction network is used to perform deep feature extraction on the input pre-fusion data and feature fusion data, and a classification head is used for classification. The features output by the backbone network are mapped to the same dimension as the number of behavior categories, and a prediction score is output for each category. The human behavior recognition result is obtained based on the prediction score. S1 specifically includes: S11: Perform space conversion on the RGB data to obtain the corresponding HSL color space, then decouple the channels of the HSL color space and retain the luminance channel L. Stack the three RGB channels with the corresponding luminance channel L to obtain the input data of the luminance sensing network. S12: Based on the brightness value, the input data obtained in S11 is divided into two categories, strong and weak, and the data is labeled. Then, it is fed into a brightness perception network consisting of two 3*3 convolutional layers and an average pooling layer to extract brightness features. After that, the strong / weak classification probability in the 0-1 interval is obtained through two fully connected layers and the Softmax function. S13: The Softmax value of the strong category obtained in S12 is used as the continuous brightness value and input into the gating function. The continuousness of the Softmax probability and the gating function with the introduction of the exponential function are used to construct a nonlinear weight curve to realize the adaptive allocation of weight. The gating function is shown in Equation (1). in, Continuous brightness value It is a smooth transition coefficient used to adjust the weight in the brightness transition region. The smoothness of the RGB modes; using the output of the gating function as the weights. Then the infrared mode weights are .

2. The human behavior recognition method based on multimodal data fusion collected by UAV according to claim 1, characterized in that, The method described in S2 for generating joint tokens by encoding the joint sequence across multiple frames using a multilayer perceptron includes: Human pose estimation algorithm is used to extract human joint point data from RGB and infrared sequences, and multiple frames of data are stacked in chronological order to generate joint point sequence. By using a multilayer perceptron to encode the joint sequence across multiple frames, joint tokens with the same dimension as the spatial CLS tokens features are generated.

3. The human behavior recognition method based on multimodal data fusion collected by UAV according to claim 1, characterized in that, S3 specifically includes: S31: Based on the topological structure of the human skeleton, joints are abstracted as nodes in a graph structure. Edges between adjacent joints are assigned a distance value of 1, and the distances between non-adjacent joints are calculated by accumulating the shortest paths to generate a physical distance matrix. , of which elements Indicates joint and The anatomical distance between them; S32: The physical distance matrix generated in step S31 is... As prior knowledge, for each distance value Assign a learnable scalar parameter Generate parameter vector ;Structure encoding matrix The elements are generated by looking up a table. Thus, the structure encoding matrix Each element in the vector is indexed to a parameter vector by the anatomical distance of the corresponding joint pair. The scalar value in the model; through training, the model automatically adjusts. The importance of each distance component; S33: In the standard self-attention mechanism, the attention weight matrix is ​​used to calculate semantic similarity by the dot product of the query and the key, and a structural encoding matrix is ​​introduced. Then, this matrix is ​​directly superimposed on the dot product result as a bias term to correct the calculation of attention weights; Will Represented as each element of the attention score matrix The The formula for calculating the attention weight associated with the human body topology is shown in equation (2): in, and Representing the query matrix The element and key matrix The One element, Reactive joint and Content relevance of feature vectors Key matrix The dimension of is used to scale the dot product result, and its square root is used for scaling. For the structure encoding matrix The elements are directly added to the attention score as bias terms, so that the model is subject to the hard constraints of anatomical structure while focusing on semantic similarity.

4. The human behavior recognition method based on multimodal data fusion collected by UAV according to claim 1, characterized in that, S4 include: S41: Compress the input feature channels to... Then, the time dimension is transposed to the first position to obtain the time dimension features after spatiotemporal separation. ,in, In terms of time dimension, The number of channels after dimensionality reduction. The number of joints in the human body; S42: The time dimension features obtained in S41 Divided into two groups along the channel dimension ,in, This indicates the channel dimension. Divide into two equal groups, with the number of channels in each group becoming ,but ; right Global time series trend vectors are generated by performing average pooling and max pooling respectively. and local significance vector ;in, These are global average pooling and max pooling operations, respectively, which compress the features of each time frame into a scalar, resulting in... , To adjust the feature distribution of the fully connected layer, The Sigmoid operation normalizes the weights to [0,1]. S43: Global time series trend vector obtained from S42 and local significance vector Perform the difference attention dot product operation to generate the difference perception matrix. The calculation formula of the difference perception matrix is ​​shown in Equation (3). in, For difference-perceived items, the difference is automatically expanded to... The matrix represents the inter-frame difference weights.

5. The human behavior recognition method based on multimodal data fusion collected by UAV according to claim 1, characterized in that, The classification head described in S6 consists of two fully connected layers and a Softmax function.

Images