A robust behavior recognition method against target occlusion

Abstract

The application discloses a kind of robust behavior recognition methods for target occlusion, it is related to the technical field of behavior recognition based on non-complete skeleton in computer vision.The technical scheme includes the following steps: S1, construct symbiotic space-time graph SSTG for non-complete skeleton sequence;S2, the extraction of skeleton sequence spatial feature using adaptive cross graph convolution ACGC;S3, the fusion complement of skeleton sequence spatial feature and time feature in symbiotic attention encoder SAE;S4, knowledge sharing and behavior recognition based on cross-graph mutual learning.The application is applicable to classroom, street, high-speed rail station, community and other complex intelligent monitoring scenes, in the case where part of the body of target human body is occluded, the behavior can still be accurately understood.

Description

Technical Field

[0001] This invention relates to the field of behavior recognition technology based on incomplete skeletons in computer vision, and particularly to a robust behavior recognition method for target occlusion. Background Technology

[0002] With the widespread application of intelligent monitoring technology in classrooms, stations, communities, streets, and other scenarios, behavior recognition based on human skeleton sequences has become one of the important technical routes in the field of behavior analysis due to its advantages such as high data volume, high computational efficiency, and insensitivity to changes in lighting and environment. Human skeleton sequences are usually composed of multiple frames of human skeletons arranged in chronological order. Each frame contains multiple joint points and two-dimensional or three-dimensional coordinate information, which can effectively describe the spatiotemporal variation characteristics of human movements.

[0003] However, in real-world intelligent surveillance scenarios, occlusion of the target human body is very common. For example, in a classroom setting, the lower limbs or torso are easily obscured by desks; in densely populated areas such as streets and stations, pedestrians easily obscure each other's view; and in residential areas and around buildings, blind spots may be created due to buildings, billboards, vegetation, and the camera's field of view. All of these factors can lead to problems such as missing joints, incomplete local structures, or the entire human target being invisible in the human skeleton sequence. This causes the original complete skeleton to degenerate into an incomplete skeleton sequence, thus posing significant challenges to subsequent behavior recognition.

[0004] Compared to intensive data such as images, videos, or point cloud data, human skeleton data consists of only a few key joints. While it has the advantage of lower storage and computational overhead, it is more dependent on joint information. Once occlusion occurs, some key nodes or skeletal connections become missing, disrupting the original topological structure of the skeleton, weakening the integrity of the action representation, and causing the loss of behavioral semantic information, thus affecting the accuracy of behavior recognition. Therefore, behavior recognition of incomplete skeletons in occluded scenarios is one of the key problems that urgently need to be solved in the field of intelligent surveillance.

[0005] Existing technologies have conducted limited research on behavior recognition using incomplete skeletons, primarily employing multi-stream feature modeling, missing joint reconstruction, or utilizing left-right symmetry to fill in missing nodes, attempting to mitigate information degradation caused by occlusion. However, existing methods still suffer from the following shortcomings: First, most methods rely heavily on a single skeleton graph structure for feature extraction and topology optimization, making them highly dependent on a complete skeleton structure. When key nodes are missing, model performance tends to degrade. Second, existing methods mostly focus on filling in missing nodes or local structures, rarely modeling spatiotemporal information holistically from a sequence-level behavioral semantic perspective, thus making it difficult to fully recover dynamic behavioral features lost due to occlusion. Third, during information propagation in graph convolution, missing nodes lead to a reduction in the neighborhood aggregation range, resulting in insufficient contextual information available for node updates, further impacting feature extraction quality and recognition robustness.

[0006] How to solve the above-mentioned technical problems is the challenge facing this invention. Summary of the Invention

[0007] The purpose of this invention is to provide a robust behavior recognition method for target occlusion, addressing the technical problem of decreased behavior recognition accuracy caused by topological structure disruption, information transmission obstruction, and impaired spatiotemporal continuity in incomplete skeleton sequences under partial occlusion conditions. The method first constructs a symbiotic spatiotemporal graph from the input human skeleton sequence, enhancing information transmission between key points by establishing two complementary dynamic graph structures. Then, adaptive cross-graph convolution is used to extract, enhance, and fuse spatial features under different graph structures, reducing the graph convolutional network's dependence on the complete skeleton structure. Based on this, a symbiotic attention encoder is used to encode spatiotemporal features of the skeleton sequence from both long-term temporal and instantaneous spatial dimensions, achieving sequence-level spatiotemporal feature fusion and complementarity through cross-encoder connections. Finally, a cross-graph mutual learning mechanism applies sequence-level semantic consistency constraints to the two symbiotic branches, achieving knowledge sharing and collaborative learning to obtain the final behavior representation and output the recognition result.

[0008] To achieve the above-mentioned objectives, the present invention employs the following technical solution: a robust behavior recognition method for target occlusion, comprising the following steps:

[0009] S10. Construct a symbiotic spatiotemporal graph (SSTG) for incomplete skeleton sequences. Model the skeleton sequence of human behavior as a spatiotemporal graph. Assume that the time edge is the connection between the same joint points between adjacent frames. Model the incomplete skeleton sequence based on the symbiotic spatiotemporal graph (SSTG) and mine known joint information to establish two dynamic graph structures.

[0010] S20. Using Adaptive Cross-Graph Convolution (ACGC) to extract spatial features of the skeleton sequence, input the incomplete skeleton sequence after SSTG modeling, construct co-occurrence graph convolutional layers. Each graph convolutional layer contains an Adaptive Cross-Graph Convolution (ACGC) module and a co-occurrence attention encoder (SAE) module. The ACGC module is used to generate the dynamic topology of the co-occurrence spatiotemporal graph. Based on the topology, spatial features of different graph structures are extracted and fused. The fused spatial behavior features are input into the SAE module.

[0011] S30, the fusion and complementation stage of spatial and temporal features of the skeleton sequence in the co-occurring attention encoder (SAE);

[0012] S40. Knowledge sharing and behavior recognition based on cross-graph mutual learning.

[0013] Further, step S10 includes:

[0014] S101. Construct an adaptive symbiotic spatiotemporal graph (SSTG) given a human skeleton sequence. , , and Corresponding to the number of frames, nodes, and channels respectively, to characterize the spatial dependencies between different key points, the human skeleton sequence is constructed as a symbiotic spatiotemporal skeleton graph set. , Let be the number of spacetime skeleton diagrams contained therein. Let each spacetime skeleton diagram be denoted as . , and These are the sets of nodes and edges, respectively. The symbiotic spatiotemporal graph SSTG constructs two adaptive spatiotemporal skeleton graphs. and The adjacency matrices of the two are respectively and . No. A spatiotemporal skeleton diagram and Adjacency matrix , any node The neighborhood is denoted as , For neighborhood set, For the spatial neighborhood, the two adaptive spatiotemporal skeleton diagrams are further combined. and The spatial neighborhood is merged, and the nodes The neighborhood is expanded to .

[0015] Further, step S20 includes:

[0016] S201. Construct the basic graph convolution structure of Adaptive Cross-Graph Convolution (ACGC), given an original skeleton sequence modeled by the co-occurrence spatiotemporal graph SSTG. , its first The characteristics of a layer are denoted as , Indicates the first Number of channels in the layer features. Spatiotemporal skeleton diagram. adjacency matrix ,make ,in Let be the identity matrix, let yes The corresponding degree matrix, let It is the first The first convolutional layer performs the... The weights corresponding to each graph during convolution. It is the number of graphs. For weighted fusion function, If it is a non-linear activation function that adds features, then the multi-graph cross-graph convolution under the composite graph structure is... The characteristics of the layer are ;

[0017] S202. Because target occlusion reduces the number of effective joints in incomplete skeleton sequences, resulting in insufficient node neighborhood information, ACGC expands the scope of node information aggregation by introducing higher-order neighborhoods. For In terms of any node The The order neighborhood is denoted as Because SSTG contains two adaptive spatiotemporal skeleton graphs. and , The neighborhood was further expanded to Furthermore, learnable weights for dynamically adjusting edge connection strength are introduced. Learnable parameters for adjusting the proportion of the learned graph structure and the original graph structure in the fusion process. ,but Dynamic adjacency matrix .make for The corresponding degree matrix, then graph convolution based on the co-occurrence spatiotemporal graph can be expressed as: ;

[0018] S203. To further mitigate the impact of occlusion-induced joint motion information loss on behavior recognition, Adaptive Cross-Graph Convolution (ACGC) introduces adaptive topology to mine context-based node dependencies. and The corresponding node features are as follows: and and and Constructing a correlation modeling function Used for computing nodes and Correlation between features ,in and It is a linear transformation function used to map features to a low-dimensional space to reduce computational complexity. Furthermore, an adaptive topological matrix is ​​constructed through cascaded functions. ,in It is a cascade function, meaning to cascade sequentially. Features of each channel. Used to restore low-dimensional features to a linear layer of the initial dimension. For nodes and The adaptive topological relationships between them. All node pairs. Constituting the first The adaptive topology matrix corresponding to each graph structure A learnable parameter is introduced to weigh the contribution of adaptive topology. Adaptive topology matrix Embedded into dynamic adjacency matrix In the process, the final node topology is obtained. .make for The corresponding degree matrix is ​​then the multi-graph cross-convolution after fusing higher-order neighborhood, dynamic adjacency relations, and adaptive topology. .

[0019] Further, step S30 includes:

[0020] S301. Construct a co-occurrence attention encoder (SAE). The SAE consists of two parallel branches: Long-Term Coding (LTE), which treats the global motion trajectory of each node as a coding unit to extract the motion trajectory of different nodes in the entire sequence; and Instantaneous Spatial Coding (ISE), which divides the complete skeleton sequence into several action segments as coding units and unfolds them sequentially according to the keypoint number and time order to form an instantaneous spatial sequence. Assume the action sampling interval is... The initial encoding of each instantaneous spatial sequence is denoted as... Each branch treats keypoints as tokens, introducing multi-head attention. By cascading the sequences encoded by each attention head, co-attention encoding extracts two co-existing sequence-level behavioral features, denoted as... and ,and , ;

[0021] S302, Sequence-level behavioral features obtained from the long-time coding LTE branch by the co-occurring attention encoder SAE. Sequence-level behavioral features obtained from the instantaneous spatial coding ISE branch Adaptive two-dimensional average pooling is performed separately. This is embedded into global information, and the motion of key points is compressed into channel descriptors to achieve cross-context feature aggregation. Then, a channel-compressed linear mapping is used. and channel recovery linear mapping Implement feature transformation and linear mapping in channel compression. and channel recovery linear mapping Set the ReLU activation function between To increase the non-linearity of the features, finally, the Sigmoid activation function is used. Obtain the weights of the non-one-hot encoding. Assume sequence-level behavioral features. , For branch type, The feature processing procedure is denoted as the channel weight generation function. Then the branch Corresponding channel importance weights , Weighting the channels and the sequence-level behavioral characteristics of the corresponding branches Element-wise multiplication is performed to reflect the feature dimensions of the current behavioral pattern. Therefore, the instantaneous spatial encoding ISE branch yields the following fused features: ,in To instantaneously spatially encode the channel importance weights corresponding to the ISE branch, The convolution kernel is Two-dimensional convolution operation, Represents the size of the convolution kernel. The instantaneous spatial encoding uses the ISE branch fusion feature. The co-occurrence attention encoder (SAE) uses residual connections to... Features fused with encoding The summation serves as the final fusion feature of the instantaneous spatial encoding ISE branch.

[0022] Further, step S40 includes:

[0023] S401. Introducing the unnormalized score Logits and the normalized exponential function to convert the unnormalized score into predicted probabilities for each behavior category. Given an input human skeleton sequence sample. Assuming The unnormalized score Logits obtained after the instantaneous spatial coding ISE branch is Let the total number of behavior categories be... Then the sample Belongs to the The probability of class behavior is , Summation index for behavior categories. The calculation method for the Long Time Coding (LTC) branch is similar, yielding samples. Long-time coding LTE branch prediction Probability of class behavior .

[0024] S402, Introducing KL divergence as a measure of the difference between the prediction distributions of two co-occurring branches. Let The predicted probability distribution of the ISE branch output is used for instantaneous spatial coding. This is the predicted probability distribution for the output of the Long Time Coding (LTC) LTE branch. Given a set of samples... The sample size is , If the distance function is KL, then the KL divergence from the long-time coding LTE branch to the instantaneous spatial coding ISE branch is expressed as: Similarly, we can obtain Then consistency loss Assuming the sample The final output obtained after the cross-co-occurrence graph convolutional network based on mutual learning is , It is a variable parameter that balances the contributions of the two types of losses. It is an indicator variable that indicates whether the classification is correct, and is used to explicitly control category selection. The overall mutual learning loss Based on the above overall mutual learning loss During training, the final classification result and the predicted probability distribution of the two co-occurring branches are simultaneously constrained to ensure consistency between the Long Time Coding (LTE) branch and the Instantaneous Spatial Coding (ISE) branch at the behavioral semantic level. Through this cross-graph mutual learning mechanism, the two co-occurring branches can perform collaborative learning and knowledge sharing, mining richer behavioral patterns from incomplete skeleton sequences and obtaining more discriminative behavioral representations. The final behavioral representation is input into a classification head, which includes a Global Average Pooling (GAP) layer, a Fully Connected Layer (FCL) layer, and a SoftMax classifier. The classification head obtains the predicted scores for each behavioral category, and the category label with the highest predicted score is used as the behavioral recognition result of the input human skeleton sequence.

[0025] Meanwhile, this invention proposes a robust behavior recognition system for target occlusion, and the system includes the following steps for applying the method described in this invention:

[0026] The Symbiotic Spatiotemporal Graph (SSTG) building module is configured to perform the following process: constructing the Symbiotic Spatiotemporal Graph (SSTG) for incomplete skeleton sequences;

[0027] The extraction module is configured to perform the following process: extracting spatial features of the skeleton sequence using Adaptive Cross-Graph Convolution (ACGC);

[0028] The fusion complementation module is configured to perform the following process: fusion complementation of spatial and temporal features of the backbone sequence in the co-attention encoder SAE;

[0029] The knowledge sharing and behavior recognition module is configured to perform the following process: knowledge sharing and behavior recognition based on cross-graph mutual learning.

[0030] Meanwhile, the present invention proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed, it implements the steps of the method described in the present invention.

[0031] Furthermore, the present invention proposes a computer-readable storage medium having a computer program stored thereon, the computer program being configured to implement the steps of the method described in the present invention when invoked by a processor.

[0032] Finally, the present invention provides a computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method described in the present invention.

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0034] (1) This invention constructs a symbiotic spatiotemporal graph and utilizes the complementarity between different graph structures to enable key points to transmit information to more key points, enhance communication between key points, reduce the impact of blocked information flow between nodes on feature extraction, thereby weakening the dependence of graph convolutional networks on the complete skeleton graph structure.

[0035] (2) By setting an adaptive cross-graph convolution module, the present invention embeds the complementarity of graph structure into the feature extraction process, and further reduces the model’s dependence on the complete skeleton structure through spatial feature fusion, adaptive topology and dynamic adjacency generation, thereby weakening the negative impact of loss of joint motion information while improving the discriminativeness of behavioral features.

[0036] (3) By setting up a symbiotic attention encoder, the present invention extracts sequence-level behavioral features at different levels using long-term coding branches and instantaneous spatial coding branches, and achieves the complementarity, iterative fusion and knowledge sharing of sequence-level spatiotemporal features through cross-encoder connection, so that the reconstructed spatiotemporal features can be used as a supplement to the behavioral information of occluded joints, thereby enriching the behavioral patterns of incomplete skeleton sequences under part occlusion.

[0037] (4) By introducing a cross-graph mutual learning mechanism, this invention incorporates sequence-level semantic similarity constraints into the network training process, prompting co-existing branches to learn collaboratively and share knowledge, extracting more discriminative behavior representations, achieving consistency constraints of sequence-level behavior semantics between two co-existing branches, and thus improving the robustness of behavior recognition under partial occlusion.

[0038] (5) This invention is designed for the practical application needs of intelligent monitoring scenarios. In complex environments such as classrooms, streets, high-speed rail stations, and residential areas where objects and people are obstructed, people are obstructed, and there are blind spots in monitoring, it can perform behavior recognition on incomplete skeleton sequences caused by obstruction, providing more robust algorithm support for intelligent monitoring systems and has good practical application value. Attached Figure Description

[0039] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0040] Figure 1 The flowchart illustrates a robust behavior recognition method for target occlusion provided by this invention.

[0041] Figure 2 This is a system model diagram of a robust behavior recognition method for target occlusion according to the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Of course, the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0043] Example 1: See Figure 1 and Figure 2 The present invention provides a technical solution as follows: a robust behavior recognition method for target occlusion, such as... Figure 1 As shown, it includes: the construction stage of the symbiotic spatiotemporal graph, the extraction stage of spatial features, the fusion and complementarity stage of spatial and temporal features, and knowledge sharing and behavior recognition based on cross-graph mutual learning. Figure 2As shown, the system model of a robust behavior recognition method for target occlusion consists of four entities: a co-occurrence spatiotemporal graph, an adaptive cross-graph convolution, a co-occurrence attention encoder, and a cross-graph mutual learning layer. The co-occurrence spatiotemporal graph models the skeleton sequence of human behavior as a spatiotemporal graph. The human skeleton sequence is sequentially passed through the co-occurrence graph convolutional layer and the cross-graph mutual learning layer for behavioral feature extraction and recognition. The output of the co-occurrence attention encoder module serves as the input to the next co-occurrence graph convolutional layer, and the output of the last SAE module serves as the input to the cross-graph mutual learning layer. The cross-graph mutual learning layer module achieves knowledge sharing of the features obtained from the two co-occurrence encoders. Finally, the final prediction score is obtained through the classifier's classification head, and the label with the highest score is taken as the recognition result of the occluded human skeleton sequence.

[0044] This example provides a robust behavior recognition method for target occlusion, which includes the following steps:

[0045] S10. Construct a symbiotic spatiotemporal graph (SSTG) for incomplete skeleton sequences. Model the skeleton sequence of human behavior as a spatiotemporal graph. Assume that the time edge is the connection between the same joint points between adjacent frames. Model the incomplete skeleton sequence based on the symbiotic spatiotemporal graph (SSTG) and mine known joint information to establish two dynamic graph structures.

[0046] S20. Using Adaptive Cross-Graph Convolution (ACGC) to extract spatial features of the skeleton sequence, input the incomplete skeleton sequence after SSTG modeling, construct co-occurrence graph convolutional layers. Each graph convolutional layer contains an Adaptive Cross-Graph Convolution (ACGC) module and a co-occurrence attention encoder (SAE) module. The ACGC module is used to generate the dynamic topology of the co-occurrence spatiotemporal graph. Based on the topology, spatial features of different graph structures are extracted and fused. The fused spatial behavior features are input into the SAE module.

[0047] S30, the fusion and complementation stage of spatial and temporal features of the skeleton sequence in the co-occurring attention encoder (SAE);

[0048] S40. Knowledge sharing and behavior recognition based on cross-graph mutual learning.

[0049] Step S10 specifically includes the following steps:

[0050] S101. Construct an adaptive symbiotic spatiotemporal graph (SSTG) given a human skeleton sequence. , , and Corresponding to the number of frames, nodes, and channels respectively, to characterize the spatial dependencies between different key points, the human skeleton sequence is constructed as a symbiotic spatiotemporal skeleton graph set. , Let be the number of spacetime skeleton diagrams contained therein. Let each spacetime skeleton diagram be denoted as . , and These are the sets of nodes and edges, respectively. The symbiotic spatiotemporal graph SSTG constructs two adaptive spatiotemporal skeleton graphs. and The adjacency matrices of the two are respectively and . No. A spatiotemporal skeleton diagram and Adjacency matrix , any node The neighborhood is denoted as , For neighborhood set, For the spatial neighborhood, the two adaptive spatiotemporal skeleton diagrams are further combined. and The spatial neighborhood is merged, and the nodes The neighborhood is expanded to .

[0051] Step S20 specifically includes the following steps:

[0052] S201. Construct the basic graph convolution structure of Adaptive Cross-Graph Convolution (ACGC), given an original skeleton sequence modeled by the co-occurrence spatiotemporal graph SSTG. , its first The characteristics of a layer are denoted as , Indicates the first Number of channels in the layer features. Spatiotemporal skeleton diagram. adjacency matrix ,make ,in Let be the identity matrix, let yes The corresponding degree matrix, let It is the first The first convolutional layer performs the... The weights corresponding to each graph during convolution. It is the number of graphs. For weighted fusion function, If it is a non-linear activation function that adds features, then the multi-graph cross-graph convolution under the composite graph structure is... The characteristics of the layer are ;

[0053] S202. Because target occlusion reduces the number of effective joints in incomplete skeleton sequences, resulting in insufficient node neighborhood information, ACGC expands the scope of node information aggregation by introducing higher-order neighborhoods. For In terms of any node The The order neighborhood is denoted as Because SSTG contains two adaptive spatiotemporal skeleton graphs. and , The neighborhood was further expanded to Furthermore, learnable weights for dynamically adjusting edge connection strength are introduced. Learnable parameters for adjusting the proportion of the learned graph structure and the original graph structure in the fusion process. ,but Dynamic adjacency matrix .make for The corresponding degree matrix, then graph convolution based on the co-occurrence spatiotemporal graph can be expressed as: ;

[0054] S203. To further mitigate the impact of occlusion-induced joint motion information loss on behavior recognition, Adaptive Cross-Graph Convolution (ACGC) introduces adaptive topology to mine context-based node dependencies. and The corresponding node features are as follows: and and and Constructing a correlation modeling function Used for computing nodes and Correlation between features ,in and It is a linear transformation function used to map features to a low-dimensional space to reduce computational complexity. Furthermore, an adaptive topological matrix is ​​constructed through cascaded functions. ,in It is a cascade function, meaning to cascade sequentially. Features of each channel. Used to restore low-dimensional features to a linear layer of the initial dimension. For nodes and The adaptive topological relationships between them. All node pairs. Constituting the first The adaptive topology matrix corresponding to each graph structure A learnable parameter is introduced to weigh the contribution of adaptive topology. Adaptive topology matrix Embedded into dynamic adjacency matrix In the process, the final node topology is obtained. .make for The corresponding degree matrix is ​​then the multi-graph cross-convolution after fusing higher-order neighborhood, dynamic adjacency relations, and adaptive topology. .

[0055] Step S30 specifically includes the following steps:

[0056] S301. Construct a co-occurrence attention encoder (SAE). The SAE consists of two parallel branches: Long-Term Coding (LTE), which treats the global motion trajectory of each node as a coding unit to extract the motion trajectory of different nodes in the entire sequence; and Instantaneous Spatial Coding (ISE), which divides the complete skeleton sequence into several action segments as coding units and unfolds them sequentially according to the keypoint number and time order to form an instantaneous spatial sequence. Assume the action sampling interval is... The initial encoding of each instantaneous spatial sequence is denoted as... Each branch treats keypoints as tokens, introducing multi-head attention. By cascading the sequences encoded by each attention head, co-attention encoding extracts two co-existing sequence-level behavioral features, denoted as... and ,and , ;

[0057] S302, Sequence-level behavioral features obtained from the long-time coding LTE branch by the co-occurring attention encoder SAE. Sequence-level behavioral features obtained from the instantaneous spatial coding ISE branch Adaptive two-dimensional average pooling is performed separately. This is embedded into global information, and the motion of key points is compressed into channel descriptors to achieve cross-context feature aggregation. Then, a channel-compressed linear mapping is used. and channel recovery linear mapping Implement feature transformation and linear mapping in channel compression. and channel recovery linear mapping Set the ReLU activation function between To increase the non-linearity of the features, finally, the Sigmoid activation function is used. Obtain the weights of the non-one-hot encoding. Assume sequence-level behavioral features. , For branch type, The feature processing procedure is denoted as the channel weight generation function. Then the branch Corresponding channel importance weights , Weighting the channels and the sequence-level behavioral characteristics of the corresponding branches Element-wise multiplication is performed to reflect the feature dimensions of the current behavioral pattern. Therefore, the instantaneous spatial encoding ISE branch yields the following fused features: ,in To instantaneously spatially encode the channel importance weights corresponding to the ISE branch, The convolution kernel is Two-dimensional convolution operation, Represents the size of the convolution kernel. The instantaneous spatial encoding uses the ISE branch fusion feature. The co-occurrence attention encoder (SAE) uses residual connections to... Features fused with encoding The summation serves as the final fusion feature of the instantaneous spatial encoding ISE branch.

[0058] Step S40 specifically includes the following steps:

[0059] S401. Introducing the unnormalized score Logits and the normalized exponential function to convert the unnormalized score into predicted probabilities for each behavior category. Given an input human skeleton sequence sample. Assuming The unnormalized score Logits obtained after the instantaneous spatial coding ISE branch is Let the total number of behavior categories be... Then the sample Belongs to the The probability of class behavior is , Summation index for behavior categories. The calculation method for the Long Time Coding (LTC) branch is similar, yielding samples. Long-time coding LTE branch prediction Probability of class behavior .

[0060] S402, Introducing KL divergence as a measure of the difference between the prediction distributions of two co-occurring branches. Let The predicted probability distribution of the ISE branch output is used for instantaneous spatial coding. This is the predicted probability distribution for the output of the Long Time Coding (LTC) LTE branch. Given a set of samples... The sample size is , If the distance function is KL, then the KL divergence from the long-time coding LTE branch to the instantaneous spatial coding ISE branch is expressed as: Similarly, we can obtain Then consistency loss Assuming the sample The final output obtained after the cross-co-occurrence graph convolutional network based on mutual learning is , It is a variable parameter that balances the contributions of the two types of losses. It is an indicator variable that indicates whether the classification is correct, and is used to explicitly control category selection. The overall mutual learning loss Based on the above overall mutual learning loss During training, the final classification result and the predicted probability distribution of the two co-occurring branches are simultaneously constrained to ensure consistency between the Long Time Coding (LTE) branch and the Instantaneous Spatial Coding (ISE) branch at the behavioral semantic level. Through this cross-graph mutual learning mechanism, the two co-occurring branches can perform collaborative learning and knowledge sharing, mining richer behavioral patterns from incomplete skeleton sequences and obtaining more discriminative behavioral representations. The final behavioral representation is input into a classification head, which includes a Global Average Pooling (GAP) layer, a Fully Connected Layer (FCL) layer, and a SoftMax classifier. The classification head obtains the predicted scores for each behavioral category, and the category label with the highest predicted score is used as the behavioral recognition result of the input human skeleton sequence.

[0061] Example 2: This example proposes a robust behavior recognition system for target occlusion, including:

[0062] The Symbiotic Spatiotemporal Graph (SSTG) building module is configured to perform the following process: constructing the Symbiotic Spatiotemporal Graph (SSTG) for incomplete skeleton sequences;

[0063] The extraction module is configured to perform the following process: extracting spatial features of the skeleton sequence using Adaptive Cross-Graph Convolution (ACGC);

[0064] The fusion complementation module is configured to perform the following process: fusion complementation of spatial and temporal features of the backbone sequence in the co-attention encoder SAE;

[0065] The knowledge sharing and behavior recognition module is configured to perform the following process: knowledge sharing and behavior recognition based on cross-graph mutual learning.

[0066] Example 3: This example proposes an electronic system, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method steps of the present invention.

[0067] Example 4: This example proposes a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the method described in this invention, which will not be repeated here.

[0068] Example 5: This example proposes a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, they implement the steps of the method described in this invention, which will not be repeated here.

[0069] It should be noted that the processing flow of embodiments 2-5 corresponds to the specific steps of the method provided in embodiment 1 of the present invention, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in this embodiment can be found in the method provided in embodiment 1 of the present invention.

[0070] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0071] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A robust behavior recognition method for target occlusion, characterized in that, Includes the following steps: Step S1: Construct a symbiotic spatiotemporal graph SSTG for incomplete skeleton sequences; Step S2: Extract spatial features of the skeleton sequence using Adaptive Cross-Graph Convolution (ACGC); Step S2 includes the following steps: S21. Construct the basic graph convolution structure of Adaptive Cross-Graph Convolution (ACGC), given an original skeleton sequence modeled by the co-occurrence spatiotemporal graph SSTG. , its first The characteristics of a layer are denoted as , Indicates the first The number of channels in the layer features, and the spatiotemporal skeleton diagram. adjacency matrix ,make ,in Let be the identity matrix, let yes The corresponding degree matrix, let It is the first The first convolutional layer performs the... The weights corresponding to each graph during convolution. It is the number of graphs. For weighted fusion function, If it is a non-linear activation function that adds features, then the multi-graph cross-graph convolution under the composite graph structure is... The characteristics of the layer are ; S22, Adaptive Cross-Graph Convolution (ACGC) expands the scope of node information aggregation by introducing higher-order neighborhoods. In terms of any node The The order neighborhood is denoted as Since the symbiotic spatiotemporal graph SSTG contains two adaptive spatiotemporal skeleton graphs... and , The neighborhood is expanded to Learnable weights are introduced to dynamically adjust the connection strength of edges. Learnable parameters for adjusting the proportion of the learned graph structure and the original graph structure in the fusion process. ,but Dynamic adjacency matrix ,make for The corresponding degree matrix is ​​then represented by graph convolution based on the co-occurrence spatiotemporal graph SSTG as follows: ; S23. Adaptive Cross-Graph Convolution (ACGC) introduces adaptive topology to mine context-based node dependencies. and The corresponding node features are as follows: and and and Construct a correlation modeling function Used for computing nodes and Correlation between features ,in and It is a linear transformation function used to map features to a low-dimensional space, and constructs an adaptive topological matrix through cascaded functions. ,in It is a cascade function, meaning to cascade sequentially. Characteristics of each channel Used to restore low-dimensional features to a linear layer of the initial dimension. For nodes and The adaptive topological relationship between them, all node pairs Constituting the first The adaptive topology matrix corresponding to each graph structure A learnable parameter is introduced to weigh the contribution of adaptive topology. Adaptive topology matrix Embedded into dynamic adjacency matrix In the process, the final node topology is obtained. ,make for The corresponding degree matrix is ​​then the multi-graph cross-convolution after fusing higher-order neighborhood, dynamic adjacency relations, and adaptive topology. ; Step S3: Fusion and complementation of spatial and temporal features of the backbone sequence in the co-occurring attention encoder (SAE); Step S4: Knowledge sharing and behavior recognition based on cross-graph mutual learning.

2. The robust behavior recognition method for target occlusion according to claim 1, characterized in that, Step S1 includes: constructing a symbiotic spatiotemporal graph SSTG, given a human skeleton sequence. , , and Corresponding to the number of frames, nodes, and channels respectively, to characterize the spatial dependencies between different key points, the human skeleton sequence is constructed as a symbiotic spatiotemporal skeleton graph set. , Let be the number of spacetime skeleton diagrams contained therein, and denote each spacetime skeleton diagram as . , and These are the sets of nodes and edges, respectively. The symbiotic spatiotemporal graph SSTG constructs two adaptive spatiotemporal skeleton graphs. and Two adaptive spatiotemporal skeleton diagrams and The corresponding adjacency matrices are respectively and , No. A spatiotemporal skeleton diagram , Adjacency Matrix , any node The neighborhood is denoted as , For neighborhood set, For spatial neighborhood, two adaptive spatiotemporal skeleton graphs are combined. and The spatial neighborhood is merged, and the nodes The neighborhood is expanded to .

3. The robust behavior recognition method for target occlusion according to claim 1, characterized in that, Step S3 includes the following steps: S31. Construct a co-occurrence attention encoder (SAE). The SAE consists of two parallel branches: Long-Term (LTE) coding, which treats the global motion trajectory of each node as a coding unit to extract the motion trajectory of different nodes throughout the sequence; and Instantaneous Spatial (ISE) coding, which divides the complete skeleton sequence into several action segments as coding units and unfolds them sequentially according to keypoint numbers and time order into an instantaneous spatial sequence. The sampling interval for the actions is... The initial encoding of each instantaneous spatial sequence is denoted as... Each branch treats keypoints as tokens, introducing multi-head attention. By cascading the sequences encoded by each attention head, co-occurring attention encoding extracts two co-occurring sequence-level behavioral features, denoted as... and ,and , ; S32. Sequence-level behavioral features obtained by the co-occurring attention encoder (SAE) from the long-time coding LTE branch. Sequence-level behavioral features obtained from the instantaneous spatial coding ISE branch Adaptive two-dimensional average pooling is performed separately. It is embedded into global information, and the motion of the joints is compressed into channel descriptors to achieve cross-context feature aggregation, through channel compression linear mapping. and channel recovery linear mapping Implement feature transformation and linear mapping in channel compression. and channel recovery linear mapping Set the ReLU activation function between To increase the non-linearity of the features, the Sigmoid activation function is then used. Obtain the weights of the non-one-hot encoding, and assume sequence-level behavioral features. , For branch type, The feature processing procedure is denoted as the channel weight generation function. Then the branch Corresponding channel importance weights , The channel's important weight and the sequence-level behavioral characteristics of the corresponding branches Element-wise multiplication is performed to reflect the feature dimensions of the current behavioral pattern. The instantaneous spatial encoding of the ISE branch is used, and the fused features are... ,in To instantaneously spatially encode the channel importance weights corresponding to the ISE branch, The convolution kernel is Two-dimensional convolution operation, Represents the size of the convolution kernel. To instantaneously spatially encode the features fused from the ISE branches, the co-occurrence attention encoder (SAE) uses residual connections to... Features fused with encoding The summation serves as the final fusion feature of the instantaneous spatial encoding ISE branch.

4. The robust behavior recognition method for target occlusion according to claim 1, characterized in that, Step S4 includes the following steps: S41. Introduce unnormalized score Logits and normalized exponential function to convert the unnormalized score Logits into predicted probabilities for each behavior category, given an input human skeleton sequence sample. , The unnormalized score Logits obtained after the instantaneous spatial coding ISE branch is Let the total number of behavior categories be... Then the sample Belongs to the The probability of class behavior is , The summation index for behavior categories is calculated similarly for the Long Time Encoding (LTE) branch, yielding the sample. Long-time coding LTE branch prediction Probability of class behavior ; S42. Introducing KL divergence as a measure of the difference between the prediction distributions of two co-occurring branches, let The predicted probability distribution of the ISE branch output is used for instantaneous spatial coding. Given a set of samples, the predicted probability distribution of the output of the Long Time Coding (LTC) LTE branch is calculated. The sample size is , If the distance function is KL, then the KL divergence from the long-time coding LTE branch to the instantaneous spatial coding ISE branch is expressed as: ,have to Then consistency loss ,sample The final output obtained after the cross-co-occurrence graph convolutional network based on mutual learning is , It is a variable parameter that balances the contributions of the two types of losses. It is an indicator variable that indicates whether the classification is correct, and is used to explicitly control category selection. The overall mutual learning loss Based on overall mutual learning loss During training, the predicted probability distributions of the final classification result and the two co-occurring branches are constrained simultaneously.

5. A robust behavior recognition system for target occlusion, characterized in that, The system comprising the steps of applying the method according to any one of claims 1 to 4, wherein the system includes: The Symbiotic Spatiotemporal Graph (SSTG) building module is configured to perform the following process: constructing the Symbiotic Spatiotemporal Graph (SSTG) for incomplete skeleton sequences; The extraction module is configured to perform the following process: extracting spatial features of the skeleton sequence using Adaptive Cross-Graph Convolution (ACGC); The fusion complementation module is configured to perform the following process: fusion complementation of spatial and temporal features of the backbone sequence in the co-attention encoder SAE; The knowledge sharing and behavior recognition module is configured to perform the following process: knowledge sharing and behavior recognition based on cross-graph mutual learning.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed, it implements the steps of the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is configured to implement the steps of the method according to any one of claims 1 to 4 when invoked by a processor.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 4.

Images