Unmanned aerial vehicle space-time behavior recognition method based on graph neural network

Abstract

The invention provides an unmanned aerial vehicle space-time behavior recognition method based on a graph neural network, and relates to the technical field of artificial intelligence and computer vision, and the method comprises the steps: obtaining behavior action video data of a to-be-recognized object, carrying out the key point recognition of the video data, and generating a behavior key point data set; dividing the behavior key point data set into a training data set, a verification data set and a test data set; constructing a space-time behavior recognition model based on the graph neural network; wherein the spatio-temporal behavior recognition model comprises a spatial information aggregation module, a time sequence information aggregation module and a spatio-temporal information focusing module; using the training data set and the verification data set to train and verify the space-time behavior recognition model, adjusting and optimizing model parameters, and storing an optimal model weight; and loading the optimal model weight, and inputting the test data set into the space-time behavior recognition model for behavior detection to obtain a behavior recognition result.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence and computer vision technology, specifically relating to a method for recognizing the spatiotemporal behavior of unmanned aerial vehicles based on graph neural networks. Background Technology

[0002] With the widespread application of drone technology in scenarios such as intelligent inspection, urban security, industrial monitoring, and emergency rescue, accurate identification of human behavior in drone aerial videos has become a key technology. Currently, the mainstream methods in the field of behavior recognition are mainly based on deep learning, especially the framework combining convolutional neural networks (CNNs) and recurrent neural networks (RNNs). CNNs are responsible for extracting spatial features from video frames, while RNNs are used to model dynamic information in the temporal dimension. Such methods have achieved good results in traditional monitoring scenarios with fixed viewpoints and stable backgrounds. However, in drone aerial videos, due to factors such as high viewpoints, drastic changes in target scale, significant human posture distortion, complex backgrounds, and frequent occlusion, directly applying existing models faces severe challenges. Specifically, CNNs struggle to capture the non-local spatial topology of human joints, and RNNs' implicit modeling of temporal sequences is insufficient to effectively capture complex motion patterns involving long distances and multi-joint collaboration. Although subsequent graph neural networks (GNNs) can model joint spatial relationships, the fixed topology graphs they rely on cannot adapt to the non-local dynamic interactions between joints caused by occlusion and viewpoint changes in aerial scenarios, resulting in insufficient accuracy and robustness in behavior recognition. Summary of the Invention

[0003] This application provides a method for recognizing the spatiotemporal behavior of unmanned aerial vehicles (UAVs) based on graph neural networks to solve one of the aforementioned technical problems.

[0004] The technical solution adopted in this application is as follows: This application provides a method for spatiotemporal behavior recognition of unmanned aerial vehicles (UAVs) based on graph neural networks, including: Acquire video data of the behavior of the object to be identified, and perform key point recognition on the video data to generate a behavior key point dataset; The behavioral key point dataset is divided into a training dataset, a validation dataset, and a test dataset. A spatiotemporal behavior recognition model based on graph neural networks is constructed; wherein, the spatiotemporal behavior recognition model includes a spatial information aggregation module, a temporal information aggregation module, and a spatiotemporal information focusing module; The spatiotemporal behavior recognition model is trained and validated using the training dataset and validation dataset, the model parameters are adjusted and optimized, and the optimal model weights are saved. The optimal model weights are loaded, and the test dataset is input into the spatiotemporal behavior recognition model for behavior detection to obtain the behavior recognition result.

[0005] According to one embodiment of this application, the process of key point recognition of video data includes: The video data is used to predict key points using a key point detection model to obtain the spatial coordinates and confidence information of the key points.

[0006] According to one embodiment of this application, the process of constructing a spatiotemporal behavior recognition model based on a graph neural network includes: Input skeleton sequence information, standardize the input through a batch normalization layer, and map the features to a high-dimensional space through a linear layer; The mapped features are input into multiple spatiotemporal graph convolutional layers; each spatiotemporal graph convolutional layer includes the spatial information aggregation module, the temporal information aggregation module, and the spatiotemporal information focusing module.

[0007] According to one embodiment of this application, the spatial information aggregation module uses a dynamic spatiotemporal adjacency matrix to aggregate spatial information; wherein, the dynamic spatiotemporal adjacency matrix includes a physically constrained adjacency matrix, a globally learnable adjacency matrix, and a spatiotemporal semantic association dynamic adjacency matrix.

[0008] According to one embodiment of this application, the time-series information aggregation module uses multi-scale temporal convolution; wherein, the multi-scale temporal convolution includes multiple branches, each branch using a one-dimensional temporal convolution with a different dilation rate.

[0009] According to one embodiment of this application, the spatiotemporal information focusing module uses a spatiotemporal attention mechanism; including: Temporal and spatial information are obtained through pooling operations, and then features are concatenated. The concatenated features are subjected to convolution and activation processing, and then split into temporal and spatial features. Convolutional processing is performed on temporal and spatial features respectively to generate attention scores, and the results are output by combining residual connections.

[0010] According to one embodiment of this application, the training and verification process includes: The model parameters are optimized using the cross-entropy loss function, and the model is trained using the SGD optimizer. Top-1 accuracy was used as the performance evaluation metric for the model.

[0011] A second aspect of this application provides a UAV spatiotemporal behavior recognition device based on graph neural networks, comprising: The key point recognition module is used to acquire video data of the behavior and actions of the object to be recognized, and to perform key point recognition on the video data to generate a behavior key point dataset. The dataset partitioning module is used to divide the behavioral key point dataset into a training dataset, a validation dataset, and a test dataset. The model building module is used to build a spatiotemporal behavior recognition model based on graph neural networks; wherein, the spatiotemporal behavior recognition model includes a spatial information aggregation module, a temporal information aggregation module, and a spatiotemporal information focusing module; The training module is used to train and validate the spatiotemporal behavior recognition model using the training dataset and validation dataset, adjust and optimize the model parameters, and save the optimal model weights. The behavior detection module is used to load the optimal model weights, input the test dataset into the spatiotemporal behavior recognition model for behavior detection, and obtain behavior recognition results.

[0012] A third aspect of this application provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps described in the method.

[0013] A fourth aspect of this application provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method as described.

[0014] Due to the adoption of the above technical solution, the beneficial effects achieved by this application are as follows: This application generates a behavior keypoint dataset by acquiring video data of behavioral actions and performing keypoint recognition. This effectively reduces the impact of background information and environmental noise on behavior recognition, improves the robustness of the model to pose changes, and provides a clean, structured input for subsequent modeling.

[0015] By constructing a spatiotemporal behavior recognition model that includes a spatial information aggregation module, a temporal information aggregation module, and a spatiotemporal information focusing module, efficient modeling of spatiotemporal features in complex aerial photography scenarios was achieved. Specifically: The spatial information aggregation module adaptively learns the connection relationships and strengths between nodes through a dynamic adjacency matrix, overcoming the limitations of fixed topology graphs. It can explicitly model the nonlocal cooperative motion between remote joints, improving the structured understanding of complex postures.

[0016] The temporal information aggregation module employs multi-scale temporal convolution to extract features at different time scales in parallel, while capturing the contextual dependencies of short-term action units and long-term behavioral patterns, thereby enhancing the ability to characterize complex temporal patterns with varying action rhythms and durations.

[0017] The spatiotemporal information focusing module adaptively calculates importance weights in the spatial and temporal dimensions through an attention mechanism, enabling the model to accurately focus on the key nodes and key frames most relevant to behavior discrimination, reducing information redundancy and interference, thereby improving recognition accuracy and robustness.

[0018] By dividing the dataset into training, validation, and test sets, and training, validating, and optimizing the model's parameters while preserving the optimal model weights, the model's generalization ability and stability were ensured. Finally, by loading the optimal weights for behavior detection, a significant improvement in the accuracy and efficiency of behavior recognition in complex drone aerial photography environments was achieved. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a method for recognizing the spatiotemporal behavior of unmanned aerial vehicles based on graph neural networks, provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0020] Figure label: 810, Processor; 820, Communication interface; 830, Memory; 840, Communication bus. Detailed Implementation

[0021] To more clearly illustrate the overall concept of this application, a detailed explanation is provided below with reference to the accompanying drawings.

[0022] Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application may also be implemented in other ways different from those described herein. Therefore, the scope of protection of this application is not limited to the specific embodiments disclosed below. It should be noted that, unless otherwise specified, the embodiments of this application and the features thereof can be combined with each other.

[0023] In this application, unless otherwise expressly specified and limited, the "above" or "below" of the second feature can mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediate medium. In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples.

[0024] Example 1 like Figure 1 As shown, a method for recognizing the spatiotemporal behavior of unmanned aerial vehicles (UAVs) based on graph neural networks includes: Acquire video data of the behavior of the object to be identified, and perform key point recognition on the video data to generate a behavior key point dataset.

[0025] As mentioned above, "acquiring" behavioral video data encompasses the process of capturing continuous frame image sequences in real-world inspection, security, or simulated scenarios using imaging devices mounted on drones (such as RGB cameras, depth cameras, or infrared cameras). "Keypoint recognition" refers to using computer vision models to locate and extract the spatial coordinates of major human joints (such as shoulders, elbows, wrists, hips, knees, and ankles) from each frame of video image. This process abstracts the human body into a topological structure composed of keypoints, thereby stripping away superficial features susceptible to environmental interference (such as clothing color and lighting changes). "Generating a behavioral keypoint dataset" refers to standardizing and organizing the extracted, time-sequential keypoint sequences (which may include two-dimensional coordinates x, y, or three-dimensional coordinates x, y, z, and possible keypoint confidence levels) into a standardized dataset suitable for model training and inference. This step fundamentally elevates the behavioral recognition problem from pixel-level understanding to the analysis of structured spatiotemporal sequence data, providing a suitable input for the efficient processing of subsequent graph neural networks.

[0026] To illustrate, let's consider a specific application example: Suppose this method is applied to a drone-based power line inspection scenario, requiring the identification of whether inspection personnel are engaging in specific behaviors such as "waving a warning" or "climbing a tower." First, the drone captures a video containing inspection personnel along its inspection route. Then, the system processes this video frame by frame, calling a keypoint detection model (such as HRNet or OpenPose) pre-trained on large human datasets like COCO. For each frame, the model outputs a list of coordinates for, for example, 17 keypoints (such as the nose, left and right eyes, left and right shoulders, left and right elbows). For instance, in a certain frame, the model identifies the coordinates of the "right wrist" keypoint as (320, 150), with a confidence score of 0.95. By performing this operation on the entire video sequence, a dataset is ultimately generated. This dataset is essentially a multidimensional array, where the dimension represents [number of video sequence frames] × [number of human keypoints per frame] × [feature dimensions of each keypoint (e.g., x, y, confidence score)]. This dataset serves as the direct input to the subsequent graph neural network model. The key points naturally form the nodes of the graph, while the connections (edges) between nodes are obtained based on the innate structure of the human body or through dynamic learning.

[0027] It should be noted that, in specific implementation scenarios, the "key point recognition" can be extended beyond the above-mentioned solution, and is not limited to real-time detection from RGB video. Data sources can be expanded to include: directly reading public datasets with existing key point annotations (such as NTU-RGB+D); fusing multimodal information, such as combining depth information obtained from a depth camera to generate 3D key points with depth coordinates (z-axis); or extracting key points from infrared video sequences to adapt to nighttime or low-light environments.

[0028] In specific implementation scenarios, the keypoint dataset generated based on the above scheme can contain information beyond basic coordinates. To enhance feature representation capabilities, the feature dimensions of each keypoint can be expanded, for example, to include: the motion velocity vector of the keypoint (calculated by the coordinate difference with the previous frame), motion acceleration, or the relative position with the human body's center of mass.

[0029] In specific implementation scenarios, this step can also, by default, handle the behavior of a single human being, building upon the above approach. The method is also applicable to complex scenarios involving multiple interacting individuals. In this case, "generating a behavior keypoint dataset" involves detecting and tracking multiple human targets in each frame of the video and extracting their keypoint sequences, providing a data foundation for subsequent model recognition of multi-person collaborative or interactive behaviors (such as "passing items" or "handshaking"). This emphasizes the scalability of this solution from recognizing single-person behavior to group behavior.

[0030] The behavioral key point dataset is divided into a training dataset, a validation dataset, and a test dataset.

[0031] As mentioned above, "partitioning" refers to deconstructing the complete behavioral keypoint dataset generated in the preceding steps into three functionally independent subsets according to a pre-defined proportion or rule designed to simulate real-world application challenges. The training dataset serves as the direct learning material for the model, used to adjust millions or even billions of learnable parameters in the graph neural network (such as weights in the dynamic adjacency matrix, convolutional kernel parameters, etc.) through forward and backward propagation algorithms, enabling it to initially grasp the mapping pattern from keypoint sequences to behavioral categories. The validation dataset acts as an "examiner" and "guide" during training. It does not directly participate in parameter updates but is used to periodically evaluate the model's stage-specific performance, serving as a basis for adjusting model hyperparameters (such as learning rate, network depth, etc.), implementing early stopping strategies to prevent overfitting, and selecting the best model iteration version. The test dataset is used for the final, one-time performance evaluation. This dataset must remain "completely unknown" throughout the model development and tuning phases to simulate the model's performance when encountering entirely new data in real-world scenarios such as drone inspections and security. Its evaluation results (such as Top-1 accuracy) are considered an objective measure of the model's generalization ability.

[0032] To illustrate, let's take a specific implementation process as an example: Assume this solution uses the publicly available NTU-RGB+D dataset as its base data source. This dataset contains a large number of human behavior samples captured by multiple cameras, each sample labeled with the 3D coordinates of 25 body joints. One specific implementation method is to follow the official recommended benchmark splitting criteria of this dataset when preparing data for the model of this invention. For example, under the "cross-subject" benchmark, all behavior samples from a subset of subjects (e.g., the first 20 actors) are assigned to the training dataset, while samples from the remaining subjects (e.g., the last 20 actors) are assigned to the test dataset. The validation dataset can then be randomly selected from the training data proportionally. This splitting method aims to evaluate the model's ability to recognize new objects with different individual movement habits that it has not previously learned from. Another "cross-viewpoint" benchmark uses data from several camera perspectives as the training and validation datasets, while all data from a completely different perspective is used as the test dataset. This splitting rigorously simulates the real challenge of drastic changes in viewpoint altitude and angle during drone flight, used to test the model's robustness to viewpoint changes.

[0033] It should be noted that, in specific implementation scenarios, the dataset can be divided based on the above-mentioned "cross-subject" or "cross-perspective" criteria, in addition to the above-mentioned scheme. To comprehensively evaluate the model's performance under different application pressures, the division strategy can be extended to: "cross-scenario," that is, using data collected in one background environment (such as an indoor office) for training and validation, while using data in another significantly different environment (such as an outdoor square) for testing, in order to test the model's resistance to background interference; or "cross-behavioral complexity," that is, using simple, single-person behavioral samples for training, while using complex, multi-person interactive behavioral samples for testing.

[0034] In specific implementation scenarios, based on the above solution, when processing continuous video data with strong temporal correlation, to prevent "data leakage" from distorting performance evaluation, the partitioning operation must ensure isolation in the temporal dimension. An extended implementation method is to include all behavioral video samples collected in earlier time periods into the training and validation datasets, while using samples collected in later time periods as the test dataset. This "time-series partitioning" strategy ensures that the model learns from "past" data and predicts "future" behavior, which is more in line with actual deployment scenarios and enhances the reliability and persuasiveness of the evaluation results.

[0035] In specific implementation scenarios, the specific proportions of the training, validation, and testing datasets can be flexible beyond the fixed limits described above. This solution protects all implementation methods that dynamically adjust the partitioning proportions based on the total size of the dataset, the number of behavior categories, and sample balance. For example, for massive datasets, the proportion of the training set can be appropriately increased to facilitate learning; for small datasets, a strategy of reserving more data for testing can be adopted. Furthermore, stratified sampling techniques can be used during the partitioning process to ensure that the proportion of each behavior category in each subset is essentially consistent with the original dataset, preventing evaluation bias caused by insufficient samples of a certain behavior category in a particular subset. This demonstrates the flexibility and rigor of this method in engineering practice.

[0036] A spatiotemporal behavior recognition model based on graph neural networks is constructed; wherein the spatiotemporal behavior recognition model includes a spatial information aggregation module, a temporal information aggregation module, and a spatiotemporal information focusing module.

[0037] As described above, this model treats the human skeletal sequence as a graph structure that evolves over time, where human joints constitute the nodes of the graph, and natural connections or dynamically learned associations between nodes constitute edges. The model uses three collaborative core modules to hierarchically extract and refine behavioral features from spatial, temporal, and spatiotemporal dimensions. The core function of the spatial information aggregation module is to model the intrinsic spatial topological relationships between human joints within each time frame. It abandons the fixed adjacency matrix used in traditional methods, instead employing a dynamic mechanism to learn the connection strength between nodes based on the real-time features of the joints (such as position and movement trends). This allows it to adaptively capture non-local joint collaborations caused by changes in viewpoint and occlusion (e.g., the remote association between the hand and torso when waving), greatly enhancing the structured understanding of complex and unconventional postures. The temporal information aggregation module is responsible for capturing the evolution of node features along the time axis. This module employs a multi-scale parallel processing approach, utilizing temporal convolutional operations with different receptive fields to simultaneously extract motion context information from neighboring and distant frames. This effectively characterizes the complete temporal dynamics from brief action units (such as rapid hand waving) to long-term behavioral patterns (such as the transition from walking to running). The spatiotemporal information focusing module, acting as a feature optimizer, is placed after the aforementioned two modules. Its role is to automatically evaluate and highlight key joint nodes and key time frames that contribute most to behavior discrimination across the joint spatiotemporal dimension through an attention mechanism. Simultaneously, it suppresses information redundancy and interference from background or irrelevant motion, thereby guiding the model to concentrate limited computational resources on the most discriminative spatiotemporal regions, ultimately improving the model's recognition accuracy and robustness in complex environments.

[0038] For example, let's take a specific drone security scenario to illustrate how the model works collaboratively to identify "climbing" behavior. The model receives a dataset of keypoints representing the suspect's skeletal sequence, generated by the preprocessing stage. First, the spatial information aggregation module begins its work: in each frame, it not only establishes basic connections based on prior human anatomy (such as the natural connection between the elbow and wrist), but more importantly, it learns dynamically. For instance, when a suspect makes a "reaching out to grab" motion, the module might establish a strong connection between the "right hand" node and the "left foot" node (even though they are not physically adjacent), because these two nodes move collaboratively during the climbing motion to maintain body balance. This dynamic connectivity is encoded in a variable adjacency matrix for information aggregation. Next, the temporal information aggregation module begins to analyze how these spatial features change over time: its multi-scale convolutional structure allows the model to simultaneously capture short-term, impulsive movements like "rapidly extending the arm" and long-term, slow trends like "the body's center of gravity continuously shifting upwards," thus coherently identifying a series of discrete actions as the complete behavior of "climbing." Finally, the spatiotemporal information focusing module processes the entire sequence: it may calculate that the "wrist" node at the moment of "grasping" and the "knee" node at the moment of "stepping" have extremely high spatial attention weights, while the frames in which these key actions occur have extremely high temporal attention weights. By multiplying these spatiotemporal attention maps with the feature maps, the model amplifies these key signals, enabling it to ultimately eliminate interference such as swaying leaves in the background and accurately classify the sequence as a "climbing" behavior.

[0039] It should be noted that, in specific implementation scenarios, the internal structure of the three core modules can be implemented in various ways based on the above scheme. For example, the dynamic adjacency matrix in the spatial information aggregation module can be extended to be composed of a weighted sum of a physical constraint matrix, a globally learnable matrix, and a spatiotemporal semantic association matrix, where the weights can be fixed or learnable. The multi-scale convolution in the temporal information aggregation module can be extended to include a combination of multiple parallel paths, including dilated convolution, max pooling branches, and identity mapping branches, to capture richer temporal context. The attention mechanism of the spatiotemporal information focusing module can be extended to generate spatial and temporal attention maps separately and then fuse them, or directly generate a unified spatiotemporal attention cube.

[0040] In specific implementation scenarios, the model is not limited to using only one set of modules, building upon the above scheme. In an extended implementation, the spatial, temporal, and focusing modules can constitute a basic unit, and the entire recognition model is composed of multiple such basic units cascaded together to form a deep network architecture, thereby enabling hierarchical feature learning from shallow local features to deep global semantics. Furthermore, residual connections or skip connections can be introduced between modules to promote gradient flow and alleviate the training challenges of deep networks.

[0041] In specific implementation scenarios, the model can be further expanded beyond the above approach to accommodate the dimensionality of input node features. In addition to basic keypoint coordinates, node features can be extended to include derived features such as the movement speed, acceleration, or position relative to the root node of the human body, providing the model with richer initial information. Similarly, the features ultimately used for classification can be global features obtained through global average pooling, or higher-order features further fused through a multilayer perceptron, demonstrating the model's adaptability at the output level.

[0042] The spatiotemporal behavior recognition model is trained and validated using the training and validation datasets, the model parameters are adjusted and optimized, and the optimal model weights are saved.

[0043] As described above, "training" refers to batch inputting skeleton sequence samples from the training dataset into the spatiotemporal behavior recognition model. Forward propagation is used to calculate the model's predicted behavior categories. Then, a pre-defined loss function (such as cross-entropy loss) is used to quantify the difference between the predicted values ​​and the true labels. Next, using backpropagation, the loss value is backpropagated layer by layer from the output layer to the input layer, calculating the gradient of the loss function with respect to millions of learnable parameters in the model (including but not limited to elements in the dynamic adjacency matrix, weights and biases of each convolutional layer, and parameters in the attention mechanism). Finally, optimization algorithms such as stochastic gradient descent or its variants are used to fine-tune the parameters along the inverse direction of the gradient, aiming to gradually reduce the loss value in each iteration and continuously enhance the model's predictive ability. "Validation," on the other hand, involves intermittently pausing parameter updates during training, inputting a validation dataset that the model has never seen before into the model, and evaluating its performance metrics (such as Top-1 accuracy). The core purpose of this process is to monitor the model's generalization ability and prevent the model from overfitting (i.e., excessively memorizing noise and specific patterns from the training data) and losing its ability to judge new data. Based on feedback from the validation set, "tuning and optimizing model parameters" refers not only to adjusting trainable parameters through gradient descent, but also to manual or automatic tuning of hyperparameters (such as learning rate, batch size, and regularization strength). "Saving optimal model weights" means continuously tracking the model's performance on the validation set throughout the training cycle and persistently storing the instantaneous state of all model parameters corresponding to the peak performance. This set of weights is considered to have achieved the best balance between fitting ability and generalization ability, and represents the final version for subsequent deployment and testing.

[0044] For example, consider a specific implementation process: Suppose we are training a model to identify "falling" behavior during drone inspections. An untrained spatiotemporal behavior recognition model is initialized with randomly set parameters. During training, a training dataset containing numerous "walking," "running," "bending," and "falling" samples is used. The model first learns to recognize basic postures, for example, by initially establishing joint connections through a spatial information aggregation module. In early training epochs, the model might incorrectly identify "slipping and sitting down" as "sitting down." The calculated loss value is high, so the optimizer significantly adjusts the model parameters. As training progresses, model performance improves, and the parameter adjustments become smaller. Simultaneously, after each training epoch, a validation dataset (containing another batch of people's falling behavior) is used for evaluation. It might be observed that after 50 training epochs, the model's accuracy on the validation set is highest; however, in subsequent epochs, although the training set loss continues to decrease, the validation set accuracy begins to stagnate or decline, indicating overfitting. At this point, the model is determined to be in optimal state at the end of the 50th cycle, and all model weights (i.e., the values ​​of all parameters) at that time point are immediately saved to the storage medium. This saved model has fully learned the essential spatiotemporal features of "falling" from the training set, and has maintained its discriminative power against new objects through the "supervision" of the validation set, thus laying the foundation for reliable application in real-world scenarios.

[0045] It should be noted that, in specific implementation scenarios, the training process can incorporate various advanced training techniques to improve the final model performance, building upon the above-mentioned approach. This includes employing learning rate scheduling strategies, such as gradually reducing the learning rate according to a predetermined plan during training, or dynamically reducing the learning rate when the validation set performance plateaus, to achieve more refined parameter convergence. Furthermore, the training process can include data augmentation operations, i.e., applying random, physically consistent affine transformations (such as small-amplitude rotations or translations) or adding noise to the skeleton sequence before inputting the training data into the model. This artificially increases the diversity and quantity of data, serving as an effective extension method to improve model robustness and prevent overfitting.

[0046] In specific implementation scenarios, the "adjustment and optimization" mentioned above can be extended beyond a single loss function and optimizer. Besides cross-entropy loss and the SGD optimizer, this step can be extended to use other types of loss functions (such as label smoothing loss) and other more advanced optimization algorithms (such as Adam and AdamW). Furthermore, the criterion of "preserving the optimal model weights" can be expanded beyond simply relying on the highest Top-1 accuracy. It can also comprehensively consider the model's computational efficiency, parameter count, or employ more complex early stopping strategies. For example, training can be terminated early and the optimal weights preserved when the validation set loss no longer decreases for several consecutive epochs, thereby saving computational resources.

[0047] In specific implementation scenarios, training can be performed on a single computing node, or it can be extended to a distributed training architecture, processing data in parallel across multiple GPUs or computing nodes to significantly shorten training time. Furthermore, the process of "adjusting and optimizing model parameters" can be expanded to include automated hyperparameter optimization. For example, algorithms such as grid search, random search, or Bayesian optimization can be used to automatically find a set of hyperparameters that best performs the model on the validation set, thereby reducing reliance on human experience and improving the automation and intelligence of the model development process.

[0048] The optimal model weights are loaded, and the test dataset is input into the spatiotemporal behavior recognition model for behavior detection to obtain the behavior recognition result.

[0049] As described above, "loading the optimal model weights" is a model state recovery process. It re-injects the fully trained and validated set of optimal parameters stored in persistent media (such as hard drives or cloud storage) into the architecture of the spatiotemporal behavior recognition model. This operation enables the model to possess the previously learned "knowledge" and "ability" to identify behavioral patterns from keypoint sequences, putting it in a state capable of performing inference tasks. "Inputting the test dataset" refers to feeding completely "unfamiliar" skeleton sequence data, which has never participated in the model training and hyperparameter optimization process, in batches or one by one, into the model with loaded weights. This data is used to fairly and objectively evaluate the model's generalization performance. "Performing behavior detection" refers to the forward computation process within the model: the input keypoint data flows sequentially through the spatial information aggregation module, the temporal information aggregation module, and the spatiotemporal information focusing module, undergoing a series of complex feature transformations and nonlinear activations, ultimately forming a probability distribution for all possible behavior categories at the output layer. "Obtaining the behavior recognition result" involves interpreting the output probability distribution. Typically, the category with the highest probability is selected as the final recognition label, thereby completing the automated mapping from raw video data to high-level behavioral semantics. The final output of this step is not only the final test of the model's practical value but also the value realization point of the entire methodology.

[0050] For example, consider a specific application in a city security scenario. Suppose a security department needs to use drones to automatically identify abnormal behaviors such as "sudden running" and "fighting" in public places. The development team has trained and saved the optimal weights of a recognition model using a large amount of historical data, following this method. Now, the effectiveness of the system needs to be tested. First, technicians load this optimal weight file, representing the model's best state, from the storage server and deploy it to the drone's ground control station or edge computing device. Then, they process a recently acquired drone video containing various suspicious behaviors that has never been used in model development, extracting its skeleton sequence to form a test dataset. When the system is running, a real-time or recorded skeleton sequence (e.g., a sequence depicting two people rapidly approaching each other with violent arm swings) is input into the model with the loaded weights. The various modules within the model work together: the spatial module captures the violent dynamic relationship between the arms and torso, the temporal module identifies rapid and conflicting movement patterns, and the focusing module amplifies keyframes of hand strikes and body shoving. Ultimately, the model outputs a probability vector showing a 95% probability for the "fighting" category. Based on this, the system obtains and reports the behavior recognition result, triggering an alert notification in the background. By performing this operation on a complete test set and statistically demonstrating a high recognition accuracy, the effectiveness and reliability of this method in real-world scenarios are ultimately proven.

[0051] It should be noted that, in specific implementation scenarios, the "loading" and "behavior detection" processes described above are not limited to specific hardware or software environments. This solution covers the loading and execution of model weights on various computing platforms, including but not limited to: large-scale, batch test data analysis on cloud servers; low-latency real-time inference on edge computing devices (such as drone onboard computers and smart camera built-in processors); and offline behavior recognition on mobile terminals.

[0052] In specific implementation scenarios, the "behavior recognition result" can be expanded beyond a single category label, building upon the above approach. Its output format can be diversified to adapt to the needs of different application scenarios. For example, the result can be output as a probability distribution, providing the confidence level for all possible behavior categories, allowing the upper-level system to make more complex decisions (such as combining information from other sensors for comprehensive judgment). Furthermore, the result can be associated with a timestamp and spatial location information to pinpoint the time and area of ​​the behavior in the original video. Even further, the result can trigger a specific control command; for example, upon recognizing a "fall" behavior, automatically controlling the drone to hover and zoom in on the camera, achieving a closed-loop intelligent response.

[0053] In specific implementation scenarios, based on the above solutions, the deployed system can operate in a continuous learning mode. That is, in the production environment, recognition results deemed low-confidence by the model or samples that have undergone manual review are collected to form an incremental test dataset. This dataset can be used for subsequent model retraining and fine-tuning, thereby achieving continuous model optimization and version iteration. Furthermore, this step can also include a performance monitoring and reporting mechanism to automatically record metrics such as recognition accuracy and response time on the test set or real data stream, providing data support for system maintenance and upgrades.

[0054] According to one embodiment of this application, the process of key point recognition of video data includes: The video data is used to predict key points using a key point detection model to obtain the spatial coordinates and confidence information of the key points.

[0055] As described above, the process of key point identification in video data is specifically implemented in the following ways: First, raw video data containing human behavior, captured by a drone, is acquired. This video is a time series composed of consecutive image frames.

[0056] The video data is then fed into a pre-trained keypoint detection model. This model is a computer vision model based on a deep learning architecture, which extracts features from a single frame image through a convolutional neural network and regresses the positional information of the main joints of the human body.

[0057] The keypoint detection model processes each video frame independently and outputs prediction results for several predetermined joints of the human body (such as head, neck, shoulder, elbow, wrist, hip, knee, ankle, etc.) in that frame. Each prediction result contains two parts of information: One is the spatial coordinates of the key points. In a two-dimensional scene, these coordinates are represented by pixel positions, usually horizontal and vertical coordinates relative to the top left corner of the image; if three-dimensional spatial positioning is involved, these coordinates can further include depth information.

[0058] Secondly, there is the confidence level information for key points. This confidence level is a value between 0 and 1, representing the model's degree of confidence in the accuracy of its predicted key point locations. The higher the confidence level, the greater the likelihood that the key point is clearly visible and accurately located.

[0059] Finally, by performing the above operations frame by frame on the video sequence, a behavioral keypoint dataset corresponding to the original video is generated, containing the spatial locations and confidence scores of all keypoints in each frame. This provides structured input data for subsequent graph neural network models. The confidence score information can be used for weighting or filtering in subsequent modeling processes to improve data quality and model robustness.

[0060] According to one embodiment of this application, the process of constructing a spatiotemporal behavior recognition model based on a graph neural network includes: Input skeleton sequence information, standardize the input through a batch normalization layer, and map the features to a high-dimensional space through a linear layer; The mapped features are input into multiple spatiotemporal graph convolutional layers; each spatiotemporal graph convolutional layer includes the spatial information aggregation module, the temporal information aggregation module, and the spatiotemporal information focusing module.

[0061] As described above, the process of constructing a spatiotemporal behavior recognition model based on graph neural networks is specifically implemented through the following steps: First, the model receives the input skeleton sequence information. This sequence information represents the changes in the positional data of human joints over time.

[0062] Next, the input data is standardized using a batch normalization layer. This operation adjusts the input skeleton sequence data to a stable distribution with a mean of zero and a variance of one, thereby accelerating the subsequent model training convergence process and improving training stability.

[0063] Subsequently, a linear layer performs feature transformation on the standardized data. This linear layer projects the original, relatively low-dimensional joint features into a higher-dimensional feature space through a linear mapping matrix. This process enhances the expressive power of the features, laying the foundation for subsequent complex spatiotemporal feature extraction.

[0064] Then, the high-dimensional features obtained after mapping are input into a series of sequentially connected spatiotemporal graph convolutional layers for deep feature learning. Each spatiotemporal graph convolutional layer is a composite computational unit, which contains three core processing modules in sequence: The spatial information aggregation module is used to model the spatial topological relationship between key points within a single frame and aggregate the feature information of neighboring nodes through a dynamically generated adjacency matrix.

[0065] The temporal information aggregation module is used to capture the continuity of joint motion in the time dimension and extract the pattern of node feature evolution over time through multi-scale temporal convolution operations.

[0066] The spatiotemporal information focusing module is used to calculate attention weights in a joint spatiotemporal dimension, adaptively highlighting key nodes and key frames that contribute more to behavior recognition, and suppressing redundant information.

[0067] By stacking multiple such spatiotemporal graph convolutional layers, the model is able to progressively extract highly discriminative spatiotemporal feature representations from the original skeleton sequence.

[0068] According to one embodiment of this application, the spatial information aggregation module uses a dynamic spatiotemporal adjacency matrix to aggregate spatial information; wherein, the dynamic spatiotemporal adjacency matrix includes a physically constrained adjacency matrix, a globally learnable adjacency matrix, and a spatiotemporal semantic association dynamic adjacency matrix.

[0069] As described above, the spatial information aggregation module uses a dynamic spatiotemporal adjacency matrix for spatial information aggregation. This matrix defines the connection relationships and strengths between nodes in the graph structure and can adaptively capture spatial topological changes at key points. The dynamic spatiotemporal adjacency matrix is ​​constructed collaboratively from the following three components: First, the physical constraint adjacency matrix is ​​defined based on prior knowledge of human anatomy, reflecting the inherent physical connections between human joints (such as the natural connection between the elbow and wrist). By introducing learnable mask parameters, this matrix allows the network to dynamically strengthen or weaken the strength of these inherent connections during training, thereby adapting to subtle changes in specific movements while maintaining physiological plausibility.

[0070] Secondly, the globally learnable adjacency matrix is ​​a fully trainable parameter matrix that does not depend on any predefined human body topology. This matrix automatically learns general joint interaction patterns across samples and actions from data using the gradient descent algorithm, enabling it to capture global spatial associations that transcend physical constraints and are related to behavioral semantics.

[0071] Finally, the spatiotemporal semantic association dynamic adjacency matrix is ​​dynamically generated by fusing the similarity of joints in the spatial and temporal dimensions. Specifically, it calculates the spatial position similarity and motion trajectory similarity between joints, balances the contributions of both through learnable coefficients, and then processes them through a nonlinear activation function to generate an adjacency matrix that can reflect the nonlocal joint dynamic associations caused by changes in viewpoint, occlusion, or specific actions.

[0072] By combining the three matrices in a weighted or summed manner, the dynamic spatiotemporal adjacency matrix can simultaneously utilize prior knowledge, data-driven learning, and spatiotemporal context information, enabling the spatial information aggregation module to more accurately model joint spatial relationships in complex scenarios and improve the robustness of behavior recognition.

[0073] According to one embodiment of this application, the time-series information aggregation module uses multi-scale temporal convolution; wherein, the multi-scale temporal convolution includes multiple branches, each branch using a one-dimensional temporal convolution with a different dilation rate.

[0074] As described above, the temporal information aggregation module uses a multi-scale temporal convolution structure to capture the dynamic patterns of joint motion at different time scales.

[0075] This module consists of multiple parallel processing branches. First, the input features are compressed in channel dimension by a convolutional layer with a kernel size of 1, and the number of feature channels is evenly distributed to each branch.

[0076] Each branch uses a different temporal receptive field configuration: The first branch uses max pooling to extract salient motion features within the local neighborhood.

[0077] The second to fifth branches use one-dimensional temporal convolutions with the same kernel size but dilation rates of 1 to 4, respectively. By gradually increasing the dilation rate, the model can sequentially expand its receptive field for the time series while keeping the number of parameters constant, thereby simultaneously capturing the contextual dependencies of both short-term action units and long-term behavioral patterns.

[0078] The last branch maintains an identity connection, directly passing the original features to preserve the initial information.

[0079] The outputs of all branches are concatenated along the channel dimension to form a comprehensive feature that integrates multi-scale temporal information. Finally, a convolutional layer with a kernel size of 1 is used to fuse and adjust the dimensions of the concatenated feature, achieving effective integration of multi-scale temporal features and enhancing the model's ability to model complex action patterns with different rhythms and durations.

[0080] According to one embodiment of this application, the spatiotemporal information focusing module uses a spatiotemporal attention mechanism; including: Temporal and spatial information are obtained through pooling operations, and then features are concatenated. The concatenated features are subjected to convolution and activation processing, and then split into temporal and spatial features. Convolutional processing is performed on temporal and spatial features respectively to generate attention scores, and the results are output by combining residual connections.

[0081] As described above, the spatiotemporal information focusing module achieves feature optimization through a spatiotemporal attention mechanism, and the specific processing flow is as follows: First, pooling operations are performed on the input features along both the spatial and temporal dimensions. Spatially, pooling is performed on all channel features at each keypoint to obtain spatial information features; temporally, pooling is performed on all channel features for each time frame to obtain temporal information features. The pooling operation employs a combination of max pooling and mean pooling.

[0082] Next, the obtained spatial and temporal information features are concatenated to form a fused feature. This fused feature is then subjected to dimensionality reduction using a 1×1 convolutional layer and a nonlinear transformation is performed using an activation function.

[0083] Then, the processed fused features are re-splitted into spatial feature components and temporal feature components. These two components are then processed using convolutional layers with a kernel size of 1×1 to generate attention score matrices in the spatial dimension and the temporal dimension, respectively.

[0084] Finally, the two attention score matrices are multiplied to obtain a spatiotemporal attention map. After normalization by an activation function, this map is multiplied element-wise with the original input features to enhance key features. Simultaneously, a residual connection is introduced to add the enhanced features to the original input features, preserving the original information while ensuring the stability of gradient propagation, ultimately outputting the optimized features.

[0085] According to one embodiment of this application, the training and verification process includes: The model parameters are optimized using the cross-entropy loss function, and the model is trained using the SGD optimizer. Top-1 accuracy was used as the performance evaluation metric for the model.

[0086] As described above, the specific implementation of the training and verification process is as follows: During the model training phase, the cross-entropy loss function is used as the optimization objective. This loss function provides direction for optimizing model parameters by calculating the difference between the probability distribution predicted by the model and the true behavior category label. The calculation process is as follows: for each training sample, the current prediction error value is calculated based on the predicted probability of each category output by the model and the true category label of the sample; the average of the error values ​​of all training samples is then calculated to obtain the overall loss value.

[0087] The model parameters are updated using a stochastic gradient descent optimizer. This optimizer iteratively adjusts the weight parameters in the model based on the gradient direction calculated from the loss function, with a set learning rate step size. A momentum parameter can also be configured during the optimization process to accelerate convergence and improve training stability by incorporating directional information from the previous update.

[0088] During the model validation phase, Top-1 accuracy is used as the performance evaluation metric. This metric is calculated as follows: the validation dataset is input into the trained model, the number of samples whose predicted probability matches the true class is counted, and this number is then divided by the total number of samples in the validation set to obtain the recognition accuracy. By periodically calculating this metric on the validation set, the changing trend of the model's generalization ability can be monitored, and the model weights that perform best on the validation set can be selected and saved accordingly.

[0089] Throughout the training process, based on the changes in Top-1 accuracy on the validation set, hyperparameters such as the learning rate can be adjusted accordingly. Training can be terminated early when the accuracy no longer improves to prevent overfitting and ensure that a model with the best generalization performance is obtained.

[0090] A second aspect of this application provides a UAV spatiotemporal behavior recognition device based on graph neural networks, comprising: The key point recognition module is used to acquire video data of the behavior and actions of the object to be recognized, and to perform key point recognition on the video data to generate a behavior key point dataset. The dataset partitioning module is used to divide the behavioral key point dataset into a training dataset, a validation dataset, and a test dataset. The model building module is used to build a spatiotemporal behavior recognition model based on graph neural networks; wherein, the spatiotemporal behavior recognition model includes a spatial information aggregation module, a temporal information aggregation module, and a spatiotemporal information focusing module; The training module is used to train and validate the spatiotemporal behavior recognition model using the training dataset and validation dataset, adjust and optimize the model parameters, and save the optimal model weights. The behavior detection module is used to load the optimal model weights, input the test dataset into the spatiotemporal behavior recognition model for behavior detection, and obtain behavior recognition results.

[0091] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any of the first aspects above.

[0092] Figure 2 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 2 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call logical instructions in the memory 830 to execute the method in any of the embodiments of the first aspect described above, the method including: Acquire video data of the behavior of the object to be identified, and perform key point recognition on the video data to generate a behavior key point dataset; The behavioral key point dataset is divided into a training dataset, a validation dataset, and a test dataset. A spatiotemporal behavior recognition model based on graph neural networks is constructed; wherein, the spatiotemporal behavior recognition model includes a spatial information aggregation module, a temporal information aggregation module, and a spatiotemporal information focusing module; The spatiotemporal behavior recognition model is trained and validated using the training dataset and validation dataset, the model parameters are adjusted and optimized, and the optimal model weights are saved. The optimal model weights are loaded, and the test dataset is input into the spatiotemporal behavior recognition model for behavior detection to obtain the behavior recognition result.

[0093] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0094] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer being able to perform the methods provided by the above methods, the method comprising: Acquire video data of the behavior of the object to be identified, and perform key point recognition on the video data to generate a behavior key point dataset; The behavioral key point dataset is divided into a training dataset, a validation dataset, and a test dataset. A spatiotemporal behavior recognition model based on graph neural networks is constructed; wherein, the spatiotemporal behavior recognition model includes a spatial information aggregation module, a temporal information aggregation module, and a spatiotemporal information focusing module; The spatiotemporal behavior recognition model is trained and validated using the training dataset and validation dataset, the model parameters are adjusted and optimized, and the optimal model weights are saved. The optimal model weights are loaded, and the test dataset is input into the spatiotemporal behavior recognition model for behavior detection to obtain the behavior recognition result.

[0095] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the methods provided by the above methods, the method comprising: Acquire video data of the behavior of the object to be identified, and perform key point recognition on the video data to generate a behavior key point dataset; The behavioral key point dataset is divided into a training dataset, a validation dataset, and a test dataset. A spatiotemporal behavior recognition model based on graph neural networks is constructed; wherein, the spatiotemporal behavior recognition model includes a spatial information aggregation module, a temporal information aggregation module, and a spatiotemporal information focusing module; The spatiotemporal behavior recognition model is trained and validated using the training dataset and validation dataset, the model parameters are adjusted and optimized, and the optimal model weights are saved. The optimal model weights are loaded, and the test dataset is input into the spatiotemporal behavior recognition model for behavior detection to obtain the behavior recognition result.

[0096] Example 2 Step S1: Perform key point recognition on human behavior and action video data to generate corresponding behavior key point datasets; Step S2: Divide the video corresponding behavior key point dataset into training dataset, validation dataset and test dataset; Step S3: Construct a spatiotemporal behavior recognition model for UAVs based on graph neural networks; Step S4: Use the training dataset and validation dataset from step S2 to train and validate the model constructed in step S3, and adjust and optimize the model parameters, and save the optimal model weights obtained from the training. Step S5: Load the optimal model weights saved in step S4, and input the test dataset from step S2 into the model constructed in step S3 for behavior detection; Further, step S1 first extracts key point information of the human body in the video. Human key point recognition is performed on the original video data. If the original data already contains accurate key point annotations (such as 3D joint coordinates provided by the NTU-RGB+D dataset), this information is directly extracted and utilized, and the key point information is defined as (x, y, z). If the original data is only RGB video, a key point detection model (e.g., HRNet) pre-trained on a large dataset (such as COCO) is used for key point prediction, and the key point is defined as (x, y, conf). Here, x, y, z represent the spatial coordinates of the key point, and conf represents the confidence information of the key point location predicted by the HRNet network.

[0097] Further, step S2 specifically involves using the NTU-RGB+D dataset as the base dataset for model training. The NTU-RGB+D dataset is currently the largest dataset containing 3D joint annotations for human behavior recognition tasks. This dataset contains 56,000 action segments across 60 behavior categories, all recorded simultaneously using three cameras to provide multi-view data. The annotation information provided by the dataset consists of 3D joint positions detected by the Kinect depth sensor in the camera coordinate system. Each subject in each skeleton sequence has 25 joints. The authors of this dataset recommend two benchmark standards: 1) Cross-Subject Benchmark (X-Sub): 39,889 segments are used for training, and 16,390 segments are used for testing. In this setting, the training segments come from a subset of actors, while the model is evaluated on segments from the remaining actors; 2) Cross-View Benchmark (X-View): 37,462 segments are used for training, and 18,817 segments are used for testing. In this setting, the training segments come from camera views 2 and 3, while the evaluation segments all come from camera view 1. This approach follows this convention and reports Top-1 recognition accuracy on both of the aforementioned benchmark tests. Furthermore, step S3 constructs a UAV spatiotemporal behavior recognition model based on graph neural networks, including the following steps: S31: Input skeleton sequence information representation C represents the feature dimension, T represents the number of frames in the skeleton sequence, and V represents the number of keypoints. In the backbone network, the input is first standardized through a batch normalization layer to stabilize the data distribution, which is beneficial for subsequent training. S32: A linear layer maps skeleton joint features from a low-dimensional space to a high-dimensional space, i.e. , in, , The feature dimension representing the skeleton information in a high-dimensional space; S33: The feature X1 output from S32 is input into nine consecutive spatiotemporal graph convolutional layers. Each spatiotemporal graph convolutional layer contains three parts: a spatial information aggregation module, a temporal information aggregation module, and a spatiotemporal information focusing module. The spatial information aggregation module is used to capture the spatial connection relationships between skeleton nodes in the dynamic spatiotemporal semantic perception graph convolution. The mathematical expression is as follows: ,in It is the first The spatial information aggregation module of the layer outputs features. It is the first The nodes of the layer input feature matrix, It is the first The learnable weight matrix of the layer, It is an activation function. This represents the aggregation matrix, where the first dimension indicates that the adjacency matrix is ​​divided into three subsets: the root node itself, the centripetal set (neighboring nodes closer to the centroid of the skeleton than the root node), and the eccentric set (neighboring nodes farther from the centroid of the skeleton than the root node). The calculation formula is as follows: Where D is the degree matrix and A is the dynamic spatiotemporal adjacency matrix. The degree matrix is ​​a diagonal matrix, and its diagonal elements are... The degree of a node is used to describe the number of edges connected to each node in the graph. The dynamic spatiotemporal adjacency matrix A can be broken down into three parts. ,in Represents the physical constraint adjacency matrix. Represents the globally learnable adjacency matrix. Represents a dynamic adjacency matrix for spatiotemporal semantic association. These are learnable coefficients. The physical constraint adjacency matrix is ​​defined as follows: ,in The intrinsic adjacency matrix is ​​defined based on prior knowledge of human anatomy, where ⊙ denotes element-wise matrix multiplication. The mask matrix is ​​the sum of the learnable matrix M and the identity matrix I. This design allows the network to dynamically strengthen or weaken connections. Represents the activation function. The globally learnable matrix is ​​defined as... , It is a learnable parameter matrix used to capture general interaction patterns across samples and actions. The spatiotemporal semantic association adjacency matrix is ​​defined as follows: ,in It is a learnable coefficient used to balance spatial and temporal similarity, the tanh activation function. It is the similarity of motion trajectories, and the calculation formula is: . This represents the difference between consecutive frames and is used to clarify the motion trajectory information of the encoded nodes. It is spatial similarity, and the calculation formula is: Conv represents a convolutional layer with a 1x1 kernel; S34: The temporal information aggregation module uses multi-scale temporal convolution to capture the temporal motion relationships of keypoints between frames. Multi-scale The temporal convolution consists of six branches. Each branch first undergoes a 1x1 convolution to reduce the channel dimension, where the output channel dimension is the input channel dimension divided by the total number of branches. The first branch is then set as a max-pooling branch with a 3x1 kernel. The second to fifth branches are one-dimensional temporal convolutions, each with a 3 kernel and dilation rates set to 1 to 4, allowing the model to capture features from different time scales. The last branch undergoes no additional processing. The results from all branches are concatenated along the channel dimension, and further feature fusion and dimensionality adjustment are performed using a 1x1 convolution to obtain the aggregated temporal information. S35: The spatiotemporal information focusing section introduces spatiotemporal attention, enabling the model to focus on important spatiotemporal nodes. Let the feature assumptions output by step S34 be... The input X is divided into time-series information by combining max pooling and mean pooling. and spatial information Two branches, of which , Then, the temporal information and node information are concatenated along the feature dimensions. ,in Represents splicing operation The number of channels is reduced using a 1x1 convolution, where r represents the channel reduction factor. This represents the Hardswish activation function. The obtained features are then re-split into temporal information and spatial node information. Split represents the split operation. Finally, 1x1 convolutions are used to restore the original dimensions, and the outer product of the two attention score matrices is taken to obtain the spatiotemporal focusing attention score. ,in This represents the Sigmoid activation function. Finally, residual connections are added to ensure gradient stability. ,in represents the ReLU activation function, and BN() represents batch normalization. S36: The deep feature representation of the skeleton sequence information obtained after nine spatiotemporal attention layers is as follows: Global average pooling is used to aggregate features in both time and space. The feature dimension F is mapped from C to the number of classes K through a fully connected layer, and the normalized class probability distribution is obtained by using the Softmax function. Further, in step S4, the model constructed in step S3 is trained, validated, and optimized using the training and validation datasets from step S2, and the resulting optimal model weights are saved. Specifically: S41: Construct a loss function for optimizing and updating model parameters. The model is selected using the cross-entropy function as the classification loss function. Update the model's parameters regarding the target class prediction; the cross-entropy function expression is as follows: , Where N is the number of training samples, This represents the true class label of the sample. If the true class of sample i is equal to c, it is set to 1; otherwise, it is set to 0. This is the predicted probability that sample i belongs to class c, as predicted by the model. The model adjusts its parameters in the opposite direction of the gradient of the loss function, thereby gradually reducing the loss value. S42: Set the model training parameters, using the SGD optimization algorithm to optimize model parameter learning. The initial learning rate is set to 0.01, and a step-decay strategy is adopted, multiplying the learning rate by 0.1 every ten epochs. The momentum is set to 0.9. During training, data augmentation methods such as random rotation, translation, and scaling are used. The above steps are repeated to iterate the model until the training iterations are completed, and the optimal model weights are saved. S43: Use Top-1 accuracy as a performance metric for the model. Where N is the total number of samples in the dataset, and i represents the i-th sample. It is the predicted category of the i-th sample. It is the true label of the i-th sample. Further, in step S5, the optimal model weights saved in step S4 are loaded, and the test dataset from step S2 is input into the model constructed in step S3 to identify behavior categories. Specifically: S51: Input the test dataset from step S2 into the model constructed in step S3; S52: Load the optimal model weights from step S4 into the model constructed in step S3 to detect behavioral action videos.

[0098] For any parts not mentioned in this application, existing technologies may be used or referenced.

[0099] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0100] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A method for spatiotemporal behavior recognition of unmanned aerial vehicles (UAVs) based on graph neural networks, characterized in that, include: Acquire video data of the behavior of the object to be identified, and perform key point recognition on the video data to generate a behavior key point dataset; The behavioral key point dataset is divided into a training dataset, a validation dataset, and a test dataset. A spatiotemporal behavior recognition model based on graph neural networks is constructed; wherein, the spatiotemporal behavior recognition model includes a spatial information aggregation module, a temporal information aggregation module, and a spatiotemporal information focusing module; The spatiotemporal behavior recognition model is trained and validated using the training dataset and validation dataset, the model parameters are adjusted and optimized, and the optimal model weights are saved. The optimal model weights are loaded, and the test dataset is input into the spatiotemporal behavior recognition model for behavior detection to obtain the behavior recognition result.

2. The method according to claim 1, characterized in that, The process of key point identification in video data includes: The video data is used to predict key points using a key point detection model to obtain the spatial coordinates and confidence information of the key points.

3. The method according to claim 1, characterized in that, The process of constructing a spatiotemporal behavior recognition model based on graph neural networks includes: Input skeleton sequence information, standardize the input through a batch normalization layer, and map the features to a high-dimensional space through a linear layer; The mapped features are input into multiple spatiotemporal graph convolutional layers; each spatiotemporal graph convolutional layer includes the spatial information aggregation module, the temporal information aggregation module, and the spatiotemporal information focusing module.

4. The method according to claim 3, characterized in that, The spatial information aggregation module uses a dynamic spatiotemporal adjacency matrix to aggregate spatial information; wherein, the dynamic spatiotemporal adjacency matrix includes a physically constrained adjacency matrix, a globally learnable adjacency matrix, and a spatiotemporal semantic association dynamic adjacency matrix.

5. The method according to claim 3, characterized in that, The time-series information aggregation module uses multi-scale temporal convolution; wherein, the multi-scale temporal convolution includes multiple branches, each branch using a one-dimensional temporal convolution with a different dilation rate.

6. The method according to claim 3, characterized in that, The spatiotemporal information focusing module uses a spatiotemporal attention mechanism; including: Temporal and spatial information are obtained through pooling operations, and then features are concatenated. The concatenated features are subjected to convolution and activation processing, and then split into temporal and spatial features. Convolutional processing is performed on temporal and spatial features respectively to generate attention scores, and the results are output by combining residual connections.

7. The method according to claim 1, characterized in that, The training and validation process includes: The model parameters are optimized using the cross-entropy loss function, and the model is trained using the SGD optimizer. Top-1 accuracy was used as the performance evaluation metric for the model.

8. A spatiotemporal behavior recognition device for unmanned aerial vehicles based on graph neural networks, characterized in that, include: The key point recognition module is used to acquire video data of the behavior and actions of the object to be recognized, and to perform key point recognition on the video data to generate a behavior key point dataset. The dataset partitioning module is used to divide the behavioral key point dataset into a training dataset, a validation dataset, and a test dataset. The model building module is used to build a spatiotemporal behavior recognition model based on graph neural networks; wherein, the spatiotemporal behavior recognition model includes a spatial information aggregation module, a temporal information aggregation module, and a spatiotemporal information focusing module; The training module is used to train and validate the spatiotemporal behavior recognition model using the training dataset and validation dataset, adjust and optimize the model parameters, and save the optimal model weights. The behavior detection module is used to load the optimal model weights, input the test dataset into the spatiotemporal behavior recognition model for behavior detection, and obtain behavior recognition results.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1-7.

Images