Adaptive refined graph convolution action recognition method based on enhanced features

Abstract

According to the self-adaptive refined image convolution action recognition method based on enhanced features, modeling is carried out through multi-order features and motion features, and joint positions, skeleton information and angle codes are synthesized, so that the diversity and expression ability of input features are enhanced; a self-adaptive graph convolution process based on data driving is designed, a complex interaction relation between joints is dynamically modeled, a self-adaptive refined feature activation mechanism is introduced, information flow between nodes is optimized, and refined modeling of skeleton space information is achieved. Meanwhile, modeling is carried out through frame index semantic time features, time sequence evolution features of the actions are captured, space-time semantic association of the actions is comprehensively analyzed, and the recognition capacity of the model for complex ground service command actions is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and deep learning, and particularly relates to an adaptive refinement graph convolution action recognition method based on enhanced features. BACKGROUND

[0002] In the existing air transportation system, airport surface operation is the key link to ensure flight safety. Pilots rely on the command actions of ground crew and the information transmitted by the command rod to complete operations such as taxiing and parking. The International Civil Aviation Organization has established a standardized ground command action system to ensure the uniformity and safety of ground operation worldwide. With the rapid development of low-altitude economy, unmanned aerial vehicle technology is gradually becoming an important part of future transportation systems. New unmanned aerial vehicles represented by electric vertical take-off and landing aircraft and unmanned cargo aircraft are driving profound changes in many fields such as urban air travel, logistics transportation, and emergency rescue. Under the background of low-altitude economy, enabling unmanned aerial vehicles to understand and respond to these standardized ground command actions and effectively coordinate with the traditional ground command system is a core problem and important challenge for unmanned aerial vehicles to smoothly integrate into the existing airport operation system. However, the automatic recognition of ground command actions still faces many difficulties in practical applications, and existing action recognition methods still face multiple bottlenecks in practical applications of ground command.

[0003] First, in the field of action recognition, mainstream methods include technologies based on human joint coordinates, skeletal vectors, and graph convolution networks, which are generally plagued by the problem of similarity between classes and difference within classes in practical applications. Specifically, different command actions often have highly similar joint motion trajectories, for example, "normal parking" and "emergency parking" are very similar in form, making it difficult for the model to effectively classify the class boundary. At the same time, the same class of actions can have significant differences due to the body size, habits, action amplitude of the command personnel, as well as external factors such as shooting angle, distance, and clothing, making it difficult for low-dimensional features such as joint coordinates or skeletal lengths to maintain consistency, thereby limiting recognition accuracy. In addition, the existing methods rely on limited feature dimensions, which are difficult to capture subtle action differences and are prone to feature confusion between similar actions.

[0004] Secondly, the fixed graph topology based on the natural skeleton of the human body has obvious limitations. For example, the adjacency matrix used in the graph convolution network is often static, only reflecting the physical connection between joints, and cannot express the dynamic interaction relationship between joints in actual motion. For example, when running, the arms and legs are not directly connected by bones, but there is an important cooperative motion pattern. The static topology cannot model the cross-region joint coupling relationship. In addition, the same joint may have different functions in different actions, such as the shoulder joint being highly related to the arm when waving, and needing to coordinate with the legs when jumping. The fixed topology is difficult to adaptively adjust the connection relationship according to the specific action.

[0005] Furthermore, the feature activation mechanism in existing models is still not perfect. For example, although the channel attention mechanism can extract global information, it often ignores the interaction of local features, leading to inaccurate weight distribution between key joints and non-key joints. At the same time, the lack of multi-granularity information fusion mechanism also limits the model's ability to fine-grained modeling of action features at different scales.

[0006] In addition, existing methods generally lack sufficient constraints on temporal semantics in time modeling. Many actions, such as "jumping" and "squatting", are very similar in joint trajectories, but their temporal order (such as upward or downward motion) has important distinguishing significance. Due to the failure to effectively integrate such semantic information, the model's ability to understand long-time actions is limited, affecting the recognition effect.

[0007] In summary, the fundamental limitations of existing technologies are: single or simply combined features cannot cope with the contradiction between inter-class similarity and intra-class difference; fixed graph topology and static activation mechanism cannot adapt to the diversity and dynamic changes of actions; at the same time, the lack of effective spatiotemporal semantic constraints also restricts the model's discriminative ability in complex long-time action recognition tasks. SUMMARY

[0008] To solve the problems existing in the prior art, the present application proposes an adaptive refinement graph convolution action recognition method based on enhanced features, comprising the following steps:

[0009] Step 1: Obtain an aviation ground command action video sequence, and perform human body detection and skeleton sequence extraction;

[0010] Step 2: Perform multi-granularity feature encoding and feature fusion on the skeleton sequence obtained in step 1 to obtain fused multi-order features;

[0011] Step 3: Perform spatial dimension modeling on the fused multi-order features output by step 2 to obtain features representing the spatial interaction relationship between joints;

[0012] Step 4: The frame index of the skeleton sequence is semantically encoded using a One-hot vector, and the feature representing the spatial interaction relationship between joints obtained in step 3 is spliced with the frame index semantic encoding and dimension conversion to obtain joint dimension information; the joint dimension information is sequentially subjected to spatial average pooling, three-layer time convolution, and time maximum pooling to obtain a time aggregation feature vector representing the comprehensive time information of the entire sequence;

[0013] Step 5: Action classification prediction is performed according to the time aggregation feature vector to obtain an action recognition result.

[0014] Further, in step 1, an aviation ground command action video sequence is obtained, and human body detection is performed on each frame of the video sequence to locate the key regions of the human body in the video; the key regions are expanded and then cropped from the original video frame and adjusted to a standard size to serve as normalized human region images; human joint information extraction is performed on the normalized human region images, and the joint coordinates of all frames are combined into a skeleton sequence

[0015]

[0016] wherein T represents the total number of frames in the video sequence; V represents the total number of joint nodes in each frame; represents the three-dimensional coordinates of joint v at the t-th frame, and is represented as:

[0017]

[0018] wherein , , are the three-dimensional spatial coordinate components of joint v at the t-th frame, respectively.

[0019] Further, step 2 includes the following steps:

[0020] Step 2.1: Based on the joint coordinate information output in step 1, angle encoding features are calculated by measuring the included angles between three joints.

[0021] Step 2.2: Based on the joint coordinate information output in step 1, static domain features and dynamic domain features of joints and bones are obtained; the static domain features include joint positions and bone data, and the dynamic domain features include joint speeds and bone speeds.

[0022] Step 2.3: The static domain features, dynamic domain features, and angle encoding features are fused to obtain fused multi-order features.

[0023] Further, the specific process of step 2.1 is as follows:

[0024] Given three joints v, ui and u2, where v is the target joint for which the angle feature is computed, ui and u2 are two end points in the skeleton; define the vector from joint v to the end point of the skeleton at the t-th frame as:

[0025]

[0026] Let a be the angle between the vectors and , then define the angle encoding of joint v as:

[0027] .

[0028] Further, in step 2.1, for a certain target joint, select one of the following three angle encoding methods:

[0029] Local angle encoding: represent the local relative motion relationship by calculating the angle between the target joint and its two adjacent joints; when the target joint has only one adjacent joint, its angle encoding is set to zero; when the target joint has more than two adjacent joints, select the two most active adjacent joints for calculation;

[0030] Center-oriented angle encoding: measure the relative position relationship between the target joint and the core region of the body; the core region of the body consists of the neck joint and the pelvic joint;

[0031] Pair joint angle encoding: describe the angle relationship between the target joint and four groups of key end points, which are joints, elbow joints, knee joints and foot joints, respectively.

[0032] Further, step 3 specifically includes the following process:

[0033] Step 3.1: using the fused multi-order features obtained in step 2 , use two learnable mapping functions θ and φ to measure the similarity between joints by inner product to obtain the time-varying adjacency matrix A t :

[0034]

[0035] Where θ and φ are two learnable mapping functions, and the softmax function is used to normalize the results;

[0036] Step 3.2: according to the fused multi-order features obtained in step 2 , obtain the feature activation weight vector by information cross-mapping and dynamic fusion based on learnable factors;

[0037] ​Step 3.3: Encode the joint type by One-hot method, and encode the fused multi-order features obtained in step 2 After temporal concatenation, the joint type semantic code is After concatenation, the input of the adaptive graph convolution network is obtained after inputting a single-layer convolution , and the adaptive graph convolution network with three-layer residual structure is used to obtain the features representing the spatial interaction relationship between joints .

[0038] Further, the specific process of step 3.2 is as follows:

[0039] Information cross mapping:

[0040] The fused multi-order features obtained in step 2 are After temporal concatenation, it is expressed as , where C is the number of channels, T represents the total number of frames in the video sequence, and V represents the total number of joint nodes in each frame. First, the features containing global information are converted into channel descriptors by global average pooling :

[0041]

[0042] where is the global feature representation of the nth channel, is the fused feature corresponding to the nth channel;

[0043] Use the strip matrix B = [ b 1 , b 2 , b 3 , . . . , b m ] to perform local channel interaction, and m is the number of local channels:

[0044]

[0045] where is the local channel descriptor;

[0046] Use the diagonal matrix D = [ d 1 , d 2 , d 3 , . . . , d C ] to capture the dependency relationship between channels as global information:

[0047]

[0048] where is the global channel descriptor;

[0049] According to the formula:

[0050]

[0051] Perform cross-scale association to obtain the cross-correlation matrix ;

[0052] Dynamic fusion based on learnable factors:

[0053] According to the formula:

[0054]

[0055]

[0056] Get the fused global channel weight And the local channel weight , And then use the learnable parameter β∈[0,1] to realize the adaptive fusion of global and local features:

[0057]

[0058] Where σ is the Sigmoid function, which limits the value to (0,1); β is the parameter that needs to be trained; The feature activation weight vector of the output.

[0059] Further, in step 3.3, the calculation process of each layer in the adaptive graph convolutional network with three-layer residual structure is:

[0060]

[0061]

[0062]

[0063] Where And are transformation matrices, shared by all time steps; Indicates element-wise multiplication; is the intermediate feature of graph convolution; is the feature adjusted by feature activation; is the output of this layer, which is also the input of the next layer.

[0064] In addition, the present application also proposes an electronic device and a readable storage medium:

[0065] An electronic device, comprising a processor, a memory, the memory being used to store one or more programs; characterized in that: when the one or more programs are executed by the processor, the above method is implemented.

[0066] A readable storage medium stores a computer program, characterized in that: when the computer program is executed by the processor, the above method is implemented.

[0067] Advantages

[0068] The application aims at the action recognition problem based on skeleton sequence in the unmanned aerial vehicle ground command task, and proposes an adaptive refinement graph convolution action recognition method based on enhanced features.

[0069] In the method innovation aspect, the application models multi-order features and motion features, comprehensively encodes joint positions, skeleton information and angles, enhances the diversity and expression ability of input features, designs a data-driven adaptive graph convolution process, dynamically models the complex interaction relationship between joints, introduces an adaptive refinement feature activation mechanism, optimizes the information flow between nodes, and realizes the fine modeling of skeleton space information.

[0070] In the performance verification aspect, through comparative experiments on the NTU-RGB+D 60 and NTU-RGB+D 120 data sets, it is proved that the proposed method has achieved significant performance improvement under various evaluation settings.

[0071] In the application verification aspect, the application collects 34 types of command actions regulated by ICAO as the unmanned aerial vehicle ground command data set in the unmanned aerial vehicle ground command scene.

[0072] The application provides an effective solution for the application of the action recognition technology based on skeleton sequence in the unmanned aerial vehicle ground command field, and provides technical support for the efficient cooperation of unmanned aerial vehicles and ground command systems.

[0073] Additional aspects and advantages of the application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0074] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the following drawings in which:

[0075] Figure 1 : Adaptive refinement graph convolutional framework for ground crew command action recognition based on enhanced features;

[0076] Figure 2 : Angle encoding diagram; (a) Local angle feature; (b) Center guided angle feature; (c) Pairwise joint angle feature;

[0077] Figure 3 : Data-driven adaptive graph generation process;

[0078] Figure 4 : Adaptive refinement feature activation mechanism diagram;

[0079] Figure 5 : Adaptive graph convolutional network structure;

[0080] Figure 6 : Time convolution module diagram;

[0081] Figure 7 : UAV ground crew command action specification and action meaning;

[0082] Figure 8 : UAV ground crew action dataset class distribution;

[0083] Figure 9 : UAV ground crew command action recognition process;

[0084] Figure 10 : UAV ground crew command action recognition result. DETAILED DESCRIPTION

[0085] Embodiments of the present application are described in detail below, which are exemplary and intended to explain the present application, and cannot be understood as a limitation of the present application.

[0086] To improve the recognition ability of unmanned aerial vehicles for complex ground service instructions, an adaptive refinement graph convolution action recognition method based on enhanced features is provided, and the core process includes multi-order feature and motion feature modeling, data-driven adaptive refinement graph convolution, and joint frame index semantic time feature modeling; the multi-order feature and motion feature modeling process models the motion pattern of the sequence, thereby enhancing feature diversity and expression ability; the data-driven adaptive refinement graph convolution process dynamically captures the complex interaction relationship between joints, finely models the spatial information of the ground service commander's skeleton, and strengthens the information flow and interaction between the joints through an adaptive feature activation mechanism; the joint frame index semantic time feature modeling process focuses on modeling the dynamic features of the action evolution over time, and the specific process is as shown in Figure 1 as follows:

[0087] Step 1: Video frame acquisition, human body detection, and skeleton sequence extraction.

[0088] Step 1.1: Input the video sequence and perform human body detection:

[0089] An original aviation ground service command action video sequence is obtained, and human body detection is performed on each frame of the video sequence to locate the key regions of interest (ROI) in the video.

[0090] The human body detection network outputs the detection results in the form of a four-tuple:

[0091] [ x i , y i , w i , h i ] i = 1 n

[0092] Wherein: n is the total number of detected targets; x i , y i are the horizontal and vertical coordinates (in pixels) of the center point of the i-th target; w i , h i represent the width and height (in pixels) of the i-th detection box, respectively. The human body detection network is a conventional method in the art, which will not be described here.

[0093] Step 1.2: Detection result screening and ROI processing:

[0094] A confidence screening mechanism is set to improve the reliability of the detection results, and detection results with a confidence of not less than 0.5 are retained to filter out low-confidence false detections. To ensure that the detection box completely covers the target personnel area, the ROI area is expanded to 1.25 times the size of the original detection box, the expanded detection box is cropped from the original video frame to obtain the region of interest, and the cropped image is uniformly adjusted to a standard size of 256x192 pixels as a standardized human region image, which is used as input for subsequent pose estimation.

[0095] Step 1.3: Pose Estimation and Skeleton Sequence Extraction:

[0096] A pose estimation network based on decoupled coordinate classification strategy is used to extract information of human joint points. For each normalized human region image of a frame, the pose estimation network outputs the three-dimensional coordinates of all joint points in the frame.

[0097] Combining the joint point coordinates of all frames into a skeleton sequence:

[0098]

[0099] where T represents the total number of frames in the video sequence; V represents the total number of joint points in each frame; represents the three-dimensional coordinates of joint point v at the t-th frame, which is represented as:

[0100]

[0101] where , , are the three-dimensional spatial coordinate components of joint point v at the t-th frame, respectively.

[0102] Step 2: Multi-order Feature Fusion and Motion Feature Modeling:

[0103] The multi-order feature and motion feature modeling step aims to improve the recognition ability of the skeleton action recognition model through more rich feature encoding methods. Existing models mostly use joint and skeleton representations to analyze human skeleton motion patterns, but still have deficiencies in handling actions with large inter-class similarity and intra-class difference. Especially when human actions have similar joint motion trajectories, existing models have difficulty in accurately distinguishing these actions based on a single feature. In addition, different individuals have body size and action execution differences, and the same class of actions may also exhibit significant joint point coordinate differences under this influence.

[0104] This step performs multi-granularity feature encoding and feature fusion on the skeleton sequence obtained in step 1, mainly including three sub-steps: angle encoding feature extraction, joint and skeleton static and dynamic domain modeling, feature embedding and fusion:

[0105] Step 2.1: Extract Angle Encoding Features:

[0106] Angle information describes the relative motion of body parts in skeleton-based action recognition, and this feature is achieved by measuring the angle between three joints. Based on the joint point coordinate information output by step 1, angle encoding features are calculated by measuring the angle between three joints. Angle encoding is a third-order feature, which pays more attention to the relative relationship of motion compared to joint coordinates (first-order) and skeleton vectors (second-order).

[0107] Given three joints v, u1 and u2, where v is the target joint for which the angle feature is computed, and u1 and u2 are two end points in the skeleton. Define the vector from joint v to the end point of the skeleton as:

[0108]

[0109] Let α be the angle between the vectors and , then define the angle encoding of joint v as:

[0110]

[0111] When α ranges from 0 to π radians, the feature value monotonically increases, which makes the fully parallel vector (α = 0) correspond to the feature value 0, and the fully opposite vector (α = π) correspond to the feature value 2.

[0112] If all angle feature combinations are used indiscriminately, the computational complexity will reach O(V 3 T), where V represents the number of joints and T represents the number of time frames. The cubic complexity means that when the number of joints and the number of frames are large, the amount of calculation will quickly expand, resulting in a significant decrease in the training and inference speed of the model. In order to reduce the computational complexity, this step selects three types of representative angle encodings, as shown in: Figure 2

[0113] (1) Local angle encoding: Local relative motion relationship is represented by calculating the angle between the target joint and its two adjacent joints. When the target joint has only one adjacent joint, its angle encoding is set to zero; when the target joint has more than two adjacent joints, the two joints with the most active motion are selected for calculation. For example, the neck joint is usually associated with two shoulder joints, while the head and abdominal joints have less activity and are not considered as the main calculation object. Local angle encoding can effectively capture the relative motion pattern between two bones, and is suitable for analyzing the subtle motion of local areas. Local angle encoding can more finely describe the local motion pattern between joints, and is advantageous in analyzing the action changes of local areas of the human body, such as the relative motion of the shoulder, elbow, etc. Compared with using only joint or bone data, local angle encoding can provide more rich motion information and improve the ability to distinguish small differences in complex actions.

[0114] ​​(2) Center orientation angle coding: measures the relative position between the target joint and the core region of the body. Two angles are constructed: the first is the non-fixed axis angle, which is defined by the neck, the target joint, and the pelvis, and the second is the fixed axis angle, which is defined by the neck, the pelvis, and the target joint. For the neck and pelvis joints, since they are located on the reference axis of angle calculation, the center orientation angle coding is set to zero. This angle coding can effectively describe the relative motion between the target joint and the body center, such as the center orientation angle between the target arm joint and the core joint, which is more suitable for analyzing coordinated motion, and the center orientation angle coding helps to reveal the relationship between the target joint and the overall structure of the body.

[0115] (3) Pair joint angle coding: describes the included angle relationship between the target joint and four groups of key endpoints (hand joint, elbow joint, knee joint, and foot joint) to capture the angle information related to motion in the skeletal structure. The basis for selecting these four groups of endpoints is that they have high relevance during motion execution and can effectively reflect the dynamic characteristics of the overall motion trajectory and joint coordination relationship. When the target joint belongs to one of the four groups of endpoints, the corresponding pair joint angle feature is set to zero.

[0116] For a target joint, any of the above angle coding can be selected.

[0117] Step 2.2: Static and dynamic domain feature modeling of joints and bones:

[0118] (1) Joint position (static domain)

[0119] The joint position data is derived from the original three-dimensional coordinates extracted in step 1. The three-dimensional coordinates of joint v at the t-th frame are represented as:

[0120]

[0121] This static domain feature directly reflects the absolute spatial position of each joint at the t-th frame, depicting the basic posture structure of human motion.

[0122] (2) Joint velocity (dynamic domain)

[0123] To enhance the description of joint motion trends, the joint velocity is calculated by the coordinate difference of the same joint in adjacent two frames:

[0124]

[0125] where is the velocity vector of joint v at the t-th frame, capturing the motion trend and dynamic change of the joint.

[0126] (3) Bone data (second-order static domain)

[0127] Skeleton data represents the relative position relationship between joints, and depicts the spatial coherence of each part of the human body. Skeleton data is a second-order static domain feature, and the length and direction of the skeleton are represented by the vector between joints, i.e., the vector from the source joint to the target joint. The calculation of skeleton information is based on the coordinate difference between the source joint and the target joint, and the direction points to the position away from the center of gravity of the skeleton:

[0128]

[0129] wherein represents the vector representation of the skeleton with the source joint v and the target joint u in the t-th frame.

[0130] (4) Skeleton velocity (second-order dynamic domain)

[0131] Further capture the dynamic changes of the skeleton in the time dimension, and define the skeleton velocity as:

[0132]

[0133] wherein is the velocity of the skeleton from the source joint v to the target joint u in the t-th frame.

[0134] Step 2.3: Feature embedding and fusion

[0135] Embed the above multi-order features into the same high-dimensional space respectively to improve the expression ability of the features; wherein the multi-order features include joint position features, joint velocity features, skeleton data features, skeleton velocity features and angle encoding features.

[0136] Taking the joint position feature as an example, two fully connected layers are used for encoding:

[0137]

[0138] obtain the joint position embedding feature wherein and are weight matrices; b1 and b2 are bias vectors; σ is a ReLU activation function; C represents the feature dimension; C1 is the joint embedding dimension.

[0139] Similarly, the remaining features are encoded through two connection layers respectively, and the corresponding embedding features are , , and .

[0140] Finally, these embedding features are spliced, and batch normalization and fully connected layer fusion are performed to obtain the fused multi-order feature representation:

[0141] F t v = σ ( B N ( W F [ J ˜ t ν , V ˜ t ν , B ˜ t u ν , M ˜ t u ν , E ˜ t ν ] ) + b F )

[0142] Among them: W F and b F Here are the weight matrix and bias vector; BN represents the batch normalization operation; [·] represents the concatenation operation.

[0143] Step 3: Perform data-driven adaptive thinning graph convolution:

[0144] This step models the spatial dimensions of the fused multi-level features output from step 2, capturing the complex spatial interactions between joints. This step comprises three sub-steps: adaptive topology learning, adaptive refinement of feature activation mechanisms, and spatial modeling using joint semantic adaptive graph convolution.

[0145] Step 3.1: Based on data-driven adaptive topology learning, obtain the time-varying adjacency matrix sequence {A}. t |t=1,2,...,T}.

[0146] The human skeleton is a natural, non-Euclidean data structure. Traditional graph convolution methods based on fixed topology (such as ST-GCN) can only reflect the static physical structural relationships of the human skeleton. However, the interactions between joints in actual movements are far more complex, often involving the coordination of distant joints. For example, during running, the legs, arms, and torso need to coordinate their movements. Traditional graph convolution methods based on fixed topology, using fixed adjacency matrices, can only capture local relationships such as knee and hip, elbow and shoulder, neglecting important collaborations such as between the left leg and right hand. This localized receptive field limits the network's understanding of global motion patterns, leading to the gradual attenuation of signals between long-distance joints and insufficient understanding of the overall movement. Furthermore, fixed adjacency matrices are also difficult to adapt to the dynamic changes in joint relationships in different movements. For example, the shoulder joint mainly interacts with the arm when waving, while in jumping, it needs to coordinate with the legs. Therefore, the natural connections between joints cannot fully represent the coupling relationships between skeletal joints during movement.

[0147] To more accurately characterize this complex coupling relationship in skeletal motion, this step uses data-driven adaptive learning of the adjacency matrix. By learning the dynamic topology between joints through data-driven learning, it more accurately reflects the complex interactions between joints during motion execution. Based on the fusion features obtained in step 2... Using two learnable mapping functions θ and φ, the similarity between joints is measured by the inner product, such as... Figure 3 As shown:

[0148]

[0149] in Let be the normalized time-varying adjacency matrix learned at frame t; θ and φ are two linear transformation functions whose parameters need to be learned during training; the softmax function normalizes the result so that the sum of each row is 1; To be The resulting matrix after concatenation, (·) T This represents the matrix transpose. The advantages of this design are: the model learns a unique graph structure for each frame, adapting to changes in joint interactions during action execution; moreover, the inner product operation can capture the similarity of joints in the feature space, transcending the limitations of physical connections; and furthermore, softmax normalization ensures the rationality of the adjacency matrix.

[0150] Step 3.2: Adaptive refinement of feature activation mechanism:

[0151] While adaptive graph convolution captures spatial relationships between joints through dynamic adjacency matrices, it doesn't fully exploit the interdependencies between feature channels. Inspired by channel attention mechanisms, this step introduces an adaptive refinement feature activation mechanism to more fully utilize global and local action representation information during network construction, optimize information flow between nodes, and effectively allocate and activate high-dimensional motion features. Feature representation is optimized at two levels: information cross-mapping and dynamic fusion based on learnable factors.

[0152] Information cross-mapping:

[0153] The fusion features obtained in step 2 are represented as follows: Where C is the number of channels, T represents the total number of frames in the video sequence, and V represents the total number of joints in each frame. First, the features containing global information are converted into channel descriptors through global average pooling. Average pooling is then performed on the time dimension T and the joint dimension V to calculate the global features of each channel:

[0154]

[0155] in This represents the global feature representation of the nth channel. For the fusion feature corresponding to the nth channel, global information of all channels in the time and joint dimensions is aggregated through global average pooling operation.

[0156] Using striped matrices B = [ b 1 , b 2 , b 3 , . . . , b m ] Perform local channel interaction, where m is the number of local channels:

[0157]

[0158] in It is a local channel descriptor.

[0159] To enhance the representation of global information, a diagonal matrix is ​​used. D = [ d 1 , d 2 , d 3 , . . . , d C ] Capture the dependency between each channel as global information:

[0160]

[0161] wherein is a global channel descriptor. The diagonal matrix D can capture the remote dependency information, and the global information obtained by the diagonal matrix is combined with the local information obtained by the strip matrix through dynamic fusion based on the learnable factor to facilitate the effective interaction of global information and local information. Through the cross-scale correlation operation, the adaptive refinement feature mechanism obtains deep correlation at different granularity levels:

[0162]

[0163] wherein is a cross-correlation matrix.

[0164] Dynamic fusion based on learnable factors:

[0165] Dynamic fusion based on learnable factors assigns features and reduces computational complexity, extracts row and column information from the cross-correlation matrix and its transpose as global and local information vectors, and realizes optimized fusion through learnable parameters, according to the formula:

[0166]

[0167]

[0168] The global channel weight after fusion is obtained and the local channel weight , and the adaptive fusion of global and local features is realized by using the learnable parameter β∈[0,1]:

[0169]

[0170] wherein σ is a Sigmoid function, limiting the value to (0,1); β is a parameter that needs to be trained; is the output feature activation weight vector. Through information cross mapping and dynamic fusion based on learnable factors, the adaptive graph convolution module can more efficiently integrate global and local action representation information at different granularities, and optimize the information flow process.

[0171] Step 3.3: Spatial modeling of joint semantic adaptive graph convolution:

[0172] In the scenario of drone ground command action recognition, many command gestures exhibit highly similar joint point trajectories in their skeletal representation, especially with significant overlap in joint point position changes and movement directions. For example, the heading commands "Normal stop" and "Emergency stop" both appear as a trajectory of arms raised and crossed above the head; "Identify gate" and "Move upwards" may both show a spatial distribution of arms raised high. Relying solely on the spatial position and movement trajectory information of joint points makes it difficult to visually distinguish similar actions.

[0173] Semantics can help understand specific information and more accurately locate specific joints. Therefore, to improve the model's understanding of the role of different joints in action, joint type semantic information is introduced. Joint type semantic information contains its functional characteristics in the human body structure. These functional characteristics can semantically enhance the model's understanding of the role of different joints in action execution.

[0174] The semantic information of spatial joint types is encoded using a one-hot encoding method. For the k-th joint, its type is represented by a one-hot vector:

[0175]

[0176] The k-th dimension is 1, and the other dimensions are 0; d j For semantic vector dimensions.

[0177] Using formula

[0178] Z i n = σ ( B N ( W 3 [ F , J ˜ ] ) + b 3 )

[0179] The fused features are concatenated with the semantic encoding of joint types, and the number of parameters is controlled by a single-layer convolution. To be The fusion features obtained after splicing To be The semantic encoding of the joint type obtained after concatenation; and Here are the weight matrix and the bias.

[0180] Information interaction between key points is achieved by constructing an adaptive graph convolutional network with a three-layer residual structure. The input to the adaptive graph convolutional network is given, and the output of the resulting adaptive graph convolutional network is given. , as the features representing the spatial interaction between joints. The network is based on a data-driven adaptive topology, and through an adaptive refinement feature activation mechanism based on information cross-mapping and dynamic fusion based on learnable factors, a residual graph convolutional network is constructed to enhance the expression ability of key features. The structure of each layer is as shown in Figure 5

[0181]

[0182]

[0183]

[0184]

[0185] Step 4: Time feature modeling based on joint frame index semantics

[0186] In the time dimension, the frame index as semantic information can represent the dynamic information of the joint nodes changing over time, and clearly define the time sequence order of the action occurrence, thereby providing additional distinguishing information for the time dimension. For example, the two actions of "Chocks inserted" and "Chocks removed" both show a "stabbing" action in the spatial trajectory, but there is a significant difference in their directionality: the former represents the arms gradually moving inward until they touch, while the latter represents the arms moving outward from the inside. By introducing the frame index, the model can capture the directional difference of the action evolving over time. Therefore, this step models the dynamic features of the action evolving over time, and captures the time evolution law of the action through a time convolution network.

[0187] In the time dimension, the frame index as semantic information can represent the dynamic information of the joint nodes changing over time, and clearly define the time sequence order of the action occurrence. For the t-th frame, use the One-hot vector to represent its time position

[0188]

[0189] ​​​​​​​​​​The t-th dimension is 1, and the other dimensions are 0; d f The semantic vector dimension of the frame index.

[0190] According to the formula:

[0191] I = σ ( B N ( W 4 [ Z o u t , T ˜ ] ) + b 4 )

[0192] Output the spatial model obtained in step 3 and The data is spliced ​​together and then transformed to obtain the joint dimension information. ,in To be The semantic encoding of the frame index obtained after concatenation and Here are the weight matrix and the bias.

[0193] According to the formula:

[0194]

[0195] Joint dimension information Spatial average pooling is performed to aggregate the joint-dimensional information into a single time-series representation. ,in The dimension is reduced from T×V×C to T×C.

[0196] Then, a three-layer temporal convolutional structure is used, with different kernel sizes applied to a single time series. Multi-scale temporal feature extraction:

[0197] Layer 1: Temporal feature extraction is performed using a 3×1 convolutional kernel, and the output is... ;

[0198] Layer 2: Uses a 1×1 convolutional kernel for dimensionality transformation and feature enhancement, outputting... ;

[0199] Layer 3: Further temporal feature extraction is performed using a 3×1 convolutional kernel, and the output is... .

[0200] in These represent the number of output channels for each layer. Batch normalization and ReLU activation functions are used after each convolution operation.

[0201] After the temporal convolution, a temporal max pooling layer is used to aggregate the information from all time steps, reducing the time scale to 1.

[0202]

[0203] in , which contains the comprehensive temporal information of the whole sequence.

[0204] Step 5: Action classification prediction:

[0205] The time-aggregated feature vector output by step 4 is input into a fully connected layer to map to the action class space:

[0206] wherein

[0207] is the weight matrix of the fully connected layer; is the bias vector; is the total number of action classes; is the unnormalized prediction score of each class. The output of the fully connected layer is

[0208] Apply the softmax function to obtain the probability distribution of each class:

[0209]

[0210] wherein is the probability vector of each class, satisfying .

[0211] The final action recognition prediction class is the class with the highest probability:

[0212]

[0213] The above model uses the cross-entropy loss function when training:

[0214]

[0215] wherein: N is the batch size; is the true label of the i-th sample (One-hot encoding); is the probability that the i-th sample predicted by the model belongs to the j-th class.

[0216] During training, the stochastic gradient descent optimizer is used to update the model parameters. The following training parameters are set: the initial learning rate is set to 0.1; the cosine annealing learning rate scheduling strategy is used, so that the learning rate gradually decays to 0.001 during training; the total number of training rounds is 80 rounds; the batch size is 32; the Nesterov momentum is used, with a momentum coefficient of 0.9; the weight decay coefficient is 0.0001, which is used for regularization to prevent overfitting.

[0217] ​And data augmentation is performed on the input skeleton sequence to improve the generalization ability of the model, including: random rotation: randomly rotating the skeleton sequence in three-dimensional space, with a rotation angle range of [-15°, 15°]; random scaling: randomly scaling the skeleton sequence, with a scaling ratio range of [0.9, 1.1]; random time cropping: randomly cropping the time sequence at the starting point while maintaining a fixed length of input; joint noise: adding Gaussian noise to the joint coordinates with a standard deviation of 0.01.

[0218] Experimental verification:

[0219] The embodiment first evaluates the proposed motion recognition method on two widely used skeleton motion recognition public datasets (NTU-RGB+D 60 and NTU-RGB+D 120) to verify its effectiveness and universality on standard benchmarks.

[0220] NTU-RGB+D 60 contains 56,000 samples, covering 60 action categories, including daily actions, health-related actions, and interactive actions. This dataset is collected using a Microsoft Kinect V2 sensor, which can accurately capture the three-dimensional position information of 25 human joints. Data collection uses three different camera angles: front view (0°) and left and right oblique view (±45°). NTU-RGB+D 120, as an extended version, contains 113,945 samples and 120 action categories, with participants from 15 different countries, collected under 32 different camera settings, further enriching the complexity and diversity of actions.

[0221] In terms of experimental setup, for NTU-RGB+D 60, two standard evaluation protocols are used: (1) Cross-Subject (X-Sub): 40 participants are evenly divided into training and testing groups to generate training and testing samples, respectively; (2) Cross-View (X-View): data from the front and +45° angle are used for training, and data from the -45° angle are used for testing. For NTU-RGB+D 120, Cross-Subject (X-Sub) and Cross-Setup (X-Set) evaluation protocols are used, where X-Sub divides 106 participants into two groups, and X-Set uses even-numbered camera settings for training and odd-numbered settings for testing.

[0222] In terms of network architecture configuration, due to the data fusion stage, the spliced and encoded data is raised in dimension, and a convolutional layer is used to reduce the dimension of the data to 128. The three-layer adaptive graph convolutional neural network accepts data-driven graph structure, and the data dimension is 128, 256 and 256 in turn. In the process of time-based TCN module processing, the data dimension is 256, 512 and 512 in turn, and finally the time dimension is compressed to 1 dimension through the time pooling layer. Finally, a fully connected layer and a Softmax function are used for classification, and the dimension corresponds to the number of action categories to generate the final action classification result.

[0223] In terms of training configuration, the experiment is based on Python3.8 development environment and implemented in PyTorch2.0.1 deep learning framework. All model training parameters are consistent, the batch size is set to 64, the initial learning rate is 0.001, and the Adam optimizer is selected. The total training is performed for 120 rounds, and the learning rate is decayed by 10 times at the 60th, 90th and 110th rounds. In order to prevent overfitting, Dropout regularization mechanism is added and set to 0.2, and ReLU activation function is selected to ensure the expression ability of nonlinear features. The whole training process uses cross-entropy loss function to optimize the network.

[0224] In the data preprocessing stage, missing data is removed and existing data is denoised. The denoising process includes two parts based on frame length and diffusion. In the data preprocessing process, if a frame contains two human bodies, it is divided into two frames, so that each sequence contains only one person. In addition, the skeleton sequence is randomly divided into 20 segments, and a frame at a random time point in each segment is selected to finally form a new sequence containing 20 frames.

[0225] The comparison results of the present application with existing mainstream methods on the NTU-RGB+D 60 dataset are shown in Table 1. The present application method has performance advantages compared with action recognition methods based on recurrent neural networks, convolutional neural networks and graph convolutional networks. The Top-1 accuracy of the present application method is improved by 10.0% and 6.6% compared with the VA-LSTM method in X-Sub and X-View settings, respectively, proving the significant advantage of the present application method in time series feature modeling capability; the Top-1 accuracy is improved by 2.9% and 3.1% compared with the HCN method in X-Sub and X-View settings, respectively, showing the superiority of the present application method in capturing action details and spatial modeling; the Top-1 accuracy is improved by 7.9% and 5.9% compared with the classic ST-GCN method in X-Sub and X-View settings, respectively, further verifying the ability of the present application method in modeling skeleton feature relationships and distinguishing complex action categories. Finally, the Top-1 accuracy of the present application algorithm on the NTU-RGB+D 60 dataset reaches 89.4% (X-Sub) and 94.2% (X-View), and the present application algorithm obtains the optimal effect among all comparison algorithms.

[0226] Table 1 Comparison of NTU-RGB+D 60 with current mainstream methods

[0227]

[0228] The comparison results of the present application with other algorithms on the NTU-RGB+D 120 dataset are shown in Table 2. The present application algorithm exhibits obvious competitiveness under different evaluation settings. In the X-Sub setting, the Top-1 accuracy of the present application algorithm is 81.7%, which is improved by 2.5% compared with the Top-1 accuracy of the SGN method, showing the good ability of the present application algorithm in processing different individual recognition tasks. In the X-Set setting, the performance of the present application algorithm is also outstanding, with a Top-1 accuracy of 83.3%, which is further improved by 2% compared with the 81.5% of the SGN. Compared with the Twostream GCA-LSTM method, the Top-1 accuracy of the present application algorithm is improved by 20.5% in the X-Sub setting and 20.0% in the X-Set setting. Overall, the present application algorithm exhibits good performance for inter-class similarity and intra-class difference actions.

[0229] Table 2 Comparison of NTU-RGB+D 120 with current mainstream methods

[0230]

[0231] After verifying the effectiveness of the method on a general dataset, the embodiment further applies the proposed model to the recognition task of UAV ground command actions. By constructing a UAV ground command action dataset that conforms to ICAO specifications, the method is tested and analyzed in real-world scenarios to verify its applicability and reliability in practical applications.

[0232] UAV ground command actions are standardized non-verbal communication methods in aviation ground operations, conveying information through specific actions and batons to make up for the lack of language communication and achieve efficient interaction between pilots and ground personnel. UAV ground command actions are standardized by ICAO, Figure 7 The UAV ground command action specifications and action meanings are listed.

[0233] To solve the problem of data scarcity in the field of UAV ground command action recognition, the embodiment constructs a representative UAV ground command action dataset. As shown in Figure 7 Based on the UAV ground command actions specified by ICAO as the standard, this paper constructs a set of UAV ground action dataset. Five trained actors are invited to participate in the data collection, performing 34 standardized command actions. The collection scene is set in a spacious room, containing diverse background elements such as greenery, buildings, and runways. A fixed USB camera (1280x720, 25 FPS) is used, and the actors perform various actions facing different directions within a range of 2-10 m from the camera. A total of 1883 videos are recorded, covering all 34 actions, and the category distribution is shown in Figure 8 To evaluate the model performance, two settings are used: (1) Cross-subject evaluation (X-Sub): dividing the actors into training / test sets to simulate different object recognition; (2) Hold-out evaluation (HO): dividing the training / test set in an 8:2 ratio and repeating the random division five times to take the average result to improve the reliability of the evaluation.

[0234] Figure 9 The complete process of UAV ground command action recognition is shown, with the original input being a series of continuous UAV ground command action video frames. Human body detection is performed on each frame to locate the key areas of the human body in the video, and all joint nodes are combined into a skeleton sequence. The skeleton sequence is input into the graph convolution action recognition network to predict the action category of the UAV ground command personnel in the video.

[0235] To verify the effectiveness of the method proposed in the present application in the unmanned aerial vehicle ground command action recognition task, the present application respectively adopts two experimental settings of X-Sub and HO on the constructed data set to compare mainstream methods. The X-Sub setting is mainly used to measure the generalization ability of the model when facing unseen subjects, and the HO setting is more concerned about the performance stability of the model under the condition of random division of data. The performance of the method in the present application is evaluated through the two setting methods, and is compared with three representative methods ST-GCN, Shift-GCN and 2s-AGCN.

[0236] The experimental results of the unmanned aerial vehicle ground action data set in the X-Sub setting are shown in Table 3. ST-GCN mainly relies on static spatial structure for correlation modeling, and its feature modeling for unmanned aerial vehicle ground command action has a certain effectiveness to a certain extent. However, due to the lack of deep analysis of dynamic features and complex interaction, it has certain limitations in modeling ability and generalization ability, and its accuracy is 87.76%, the Jaccard coefficient is 80.62%, and the F1 value is 87.52%. The local feature modeling method of Shift-GCN is insufficient to capture complex action features between classes. The experimental results show that both of them have certain ability to model human skeleton action features, but the generalization ability is limited in the cross-subject scene. In contrast, the dual-stream network 2s-AGCN using skeleton flow and joint flow models the action features through the skeleton features, improves the action feature discrimination ability, and its performance reaches an accuracy of 88.30%, a Jaccard coefficient of 82.88%, and an F1 value of 89.29%. However, the method is relatively coarse-grained in modeling the dynamic coupling relationship between different joints, and there is still some confusion between similar actions in classes. The method in the present application models the sequence based on multi-order features and motion features, and optimizes the information flow and interaction between joints based on the enhanced data-driven adaptive refinement graph convolution. The results show that the method in the present application is superior to 2s-AGCN in accuracy (90.71%), Jaccard coefficient (84.32%) and F1 value (90.23%).

[0237] Table 3 Experimental results of unmanned aerial vehicle ground action data set in X-Sub setting

[0238]

[0239] The HO setting experiment results are shown in Table 4. The method of the present application also achieves the best performance, with an accuracy (96.09%) and F1 value (96.22%) increased by 1.97% and 1.84% respectively compared with 2s-AGCN, fully embodying the accuracy of the model under the condition of random data division. In the unmanned aerial vehicle ground command action task, the significant inter-class similarity and intra-class difference pose challenges to action recognition. For example, the "normal parking" and "emergency parking" actions are similar in dynamic features, while the "turn left" and "turn right" actions show greater variability due to differences in individual execution methods and posture angles. The method of the present application optimizes the analysis of features and the interaction between nodes through multi-stage feature modeling and adaptive refinement mechanisms, effectively solving the inter-class similarity and intra-class difference problems in the unmanned aerial vehicle ground command action task. These improvements enhance the performance of the model in the unmanned aerial vehicle ground command action recognition task, demonstrating the high accuracy and application potential of the algorithm in this paper in dynamic environments.

[0240] Table 4 HO setting experiment results of unmanned aerial vehicle ground action data set

[0241]

[0242] Figure 10 Part of the action test results of the unmanned aerial vehicle ground command action data set are shown, and the action sequence is consistent with the ICAO standard action category. The left side of each part is an example of the ICAO standard action, and the right side is the video key frame recognition result of the corresponding action in the data set. In each action key frame, the pose estimation result is displayed as a blue point in an orange detection box, clearly identifying the key point position of the commander. The top left corner of the key frame lists the prediction results of the unmanned aerial vehicle ground command action category, including the Top-10 categories and their prediction confidence. From the experimental results, it can be seen that the method of the present application can accurately identify all the unmanned aerial vehicle ground command actions in the data set, demonstrating the reliability of the model in the unmanned aerial vehicle ground command action classification task.

[0243] Although the embodiments of the present application have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and should not be construed as limiting the present application. Those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments without departing from the principles and spirit of the present application within the scope of the present application.

Claims

1. An adaptive refinement graph convolution action recognition method based on enhanced features, characterized in that: The method comprises the following steps: Step 1: Obtain an aviation ground service command action video sequence, and perform human body detection and skeleton sequence extraction on the video sequence; Step 2: Perform multi-granularity feature coding and feature fusion on the skeleton sequence obtained in step 1 to obtain fused multi-order features; Step 3: Perform spatial dimension modeling on the fused multi-order features output in step 2 to obtain features representing the spatial interaction relationship between joints; Step 4: Use a One-hot vector to perform semantic coding on the frame index of the skeleton sequence, concatenate the features representing the spatial interaction relationship between joints obtained in step 3 with the frame index semantic coding, and perform dimension conversion to obtain joint dimension information; after sequentially performing spatial average pooling, three-layer time convolution and time maximum pooling on the joint dimension information, a time aggregation feature vector reflecting the overall sequence comprehensive time information is obtained; Step 5: Perform action classification prediction according to the time aggregation feature vector to obtain an action recognition result.

2. The method of claim 1, wherein the method is based on an enhanced feature adaptive refinement graph convolutional action recognition. In step 1, an aviation ground service command action video sequence is obtained, and human body detection is performed on each frame of the video sequence to locate the key regions of the human body in the video; the key regions are expanded and then cropped from the original video frame and adjusted to a standard size to obtain normalized human region images; human joint information is extracted from the normalized human region images, and the joint coordinates of all frames are combined into a skeleton sequence where T represents the total number of frames in the video sequence; V represents the total number of junctions within each frame; represents the three-dimensional coordinates of the junction v in the tth frame, which is represented as: wherein , , are the three-dimensional spatial coordinate components of the joint v at the t-th frame, respectively.

3. The method of claim 1, wherein the method further comprises: Step 2 comprises the following steps: Step 2.1: Based on the joint coordinate information output in step 1, the angle coding features are calculated by measuring the included angle between three joints; Step 2.2: Based on the joint coordinate information output in step 1, obtain the static domain features and dynamic domain features of the joints and the skeleton; the static domain features include joint position and skeleton data, and the dynamic domain features include joint speed and skeleton speed; Step 2.3: Fuse the static domain features, dynamic domain features and angle coding features to obtain fused multi-order features.

4. The method of claim 3, wherein the method further comprises: The specific process of step 2.1 is as follows: Given three joints v, u1, and u2, where v is the target joint for calculating angular features, and u1 and u2 are two endpoints in the skeleton; define the distance from joint v to the skeleton endpoints at frame t. The vector is: Let α be the angle between the vectors and The angle coding of joint v is defined as: 。 5. The method of claim 4, wherein the method further comprises: In step 2.1, for a target joint, one of the following three angle coding methods is selected: Local angle coding: the relative motion relationship is represented by calculating the included angle between the target joint and its two adjacent joints; when the target joint has only one adjacent joint, the angle coding is set to zero; when the target joint has more than two adjacent joints, the two most active adjacent joints are selected for calculation; Center-directed angle coding: measures the relative position relationship between the target joint and the core region of the body; The core region of the body is composed of the neck joint and the pelvic joint; Pairwise joint angle coding: describes the included angle relationship between the target joint and four groups of key end points, which are joints, elbow joints, knee joints and foot joints, respectively. 6.The method of claim 1, wherein: Step 3 specifically comprises the following process: Step 3.1: Utilize the fused multi-order features obtained from Step 2 The time-varying adjacency matrix A is obtained by measuring the similarity between joints using inner product with two learnable mapping functions θ and φ t : Where θ and φ are two learnable mapping functions, and the softmax function is used to normalize the results; Step 3.2: merged multi-order features according to step 2 , obtaining feature activation weight vectors through information cross mapping and dynamic fusion based on learnable factors ; Step 3.3: Encode the joint type by One-hot method, and encode the fused multi-order features obtained in step 2 After splicing in time sequence, encode the joint type semantics Splicing, after inputting a single-layer convolution, obtain the input of the adaptive graph convolution network , and obtain the features representing the spatial interaction relationship between joints by using the adaptive graph convolution network with three-layer residual structure .

7. The method of claim 6, wherein the method further comprises: The specific process of step 3.2 is as follows: Information cross mapping: The fused multi-order features obtained in step 2 are represented as After being spliced in time sequence, represented as where C is the number of channels, T represents the total number of frames in the video sequence, and V represents the total number of joints in each frame. First, the features containing global information are converted into channel descriptors by global average pooling : wherein is a global feature representation of the nth channel, is a fusion feature corresponding to the nth channel. Using a strip matrix Local channel interaction is performed, m is the number of local channels: wherein is a local channel descriptor; Utilizing diagonal matrices Capture dependencies between channels as global information: wherein is a global channel descriptor; According to the formula: Perform cross-scale correlation to get cross-correlation matrix ; Dynamic fusion based on learnable factors: According to the formula: The global channel weight value after fusion is obtained and the local channel weight value The adaptive fusion of global and local features is realized by using a learnable parameter β∈[0,1]: where σ is a sigmoid function that limits the values between (0, 1); and β is a parameter that needs to be trained; is the output feature activation weight vector. 8.The method of claim 7, wherein: In step 3.3, the calculation process of each layer in the three-layer residual structure adaptive graph convolutional network is as follows: wherein and is a transformation matrix, shared for all time steps; denotes element-wise multiplication; is the intermediate feature of the graph convolution; is the feature after activation adjustment; is the output of this layer, while serving as the input of the next layer. 9.An electronic device comprising a processor, a memory storing one or more programs; characterized in that: The one or more programs, when executed by the processor, implement the method of any one of claims 1-8.

10. A readable storage medium, storing a computer program, characterized in that: The computer program, when executed by the processor, implements the method of any one of claims 1-8.

Images