A Behavior Recognition Method and System Based on Attribute-Aware Semantic Collaborative Networks

Abstract

This invention discloses a behavior recognition method and system based on an attribute-aware semantic collaborative network, comprising: acquiring a skeletal sequence from a video; extracting geometric statistical information from the skeletal sequence to construct a geometric perception cue vector; acquiring modulation parameters based on the geometric perception cue vector, and performing linear modulation on the spatiotemporal features extracted by the backbone network according to the modulation parameters, inputting the modulated parameters into the attribute-aware semantic collaborative network, and collaboratively modeling the spatial structure information and temporal dynamic information of the skeletal sequence through spatial semantic channels and temporal semantic channels, and obtaining a behavior recognition model through semantic consistency optimization training; inputting the video to be recognized into the behavior recognition model, outputting the behavior category, and simultaneously outputting the geometric interpretation basis corresponding to the recognition result. This invention can explicitly inject skeletal geometric structure information into the spatiotemporal representation learning process, realize the integrated output of recognition results and judgment basis, and improve the stability and interpretability of the model in complex scenarios.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and in particular to a behavior recognition method and system based on attribute-aware semantic collaborative networks. Background Technology

[0002] Human behavior recognition has been widely used in fields such as medical rehabilitation and sports. Although it has high performance, existing methods still suffer from insufficient interpretability in high-reliability application scenarios: traditional models only output the action category and cannot explain "why the judgment was made", lacking interpretability and semantic transparency. For example, in rehabilitation training, doctors cannot know the specific basis for the model's judgment of posture violations.

[0003] Prior Art (Lu, Mingqi, Xiaobo Lu, and Jun Liu. Momentum ContrastiveTeacher for Semi-Supervised Skeleton Action Recognition. IEEE Transactions on Image Processing (2025).) By designing a student-teacher dual encoder, positive sample pairs are generated through skeletal cloud coloring data enhancement, and spatiotemporal features are extracted through dual paths (spatial GCN and temporal convolution), thereby optimizing classification accuracy and feature clustering effect.

[0004] Chinese patent document CN120708290A discloses a spatiotemporally decoupled human behavior recognition method based on dynamic semantic guidance mask. It performs semantic guidance masking on the skeletal sequence in the spatial and temporal dimensions, and obtains a recognition model by combining encoder and cross-domain contrastive learning training. Finally, it outputs human behavior prediction results for the input video.

[0005] However, while the aforementioned existing technologies and action recognition methods combining self-supervised contrastive learning and spatiotemporal decoupling networks (such as SkeleMoCLR and ActCLR) have improved recognition accuracy, they still have the following shortcomings: (1) Lack of geometric-semantic mapping: The model does not establish an explicit mapping relationship between geometric features such as joint distance and bone segment structure relationship and action semantics, resulting in uninterpretable decision-making; (2) Insufficient coordination of spatiotemporal features driven by geometric attributes: Spatial branches rely on preset skeletal topology for local relationship modeling, while temporal branches focus on dynamic change modeling between adjacent frames. The two branches usually extract features separately and then fuse them. They have not yet achieved coordinated adjustment of spatial structural information and temporal dynamic information around unified geometric attribute prompts (such as how the joint angle change rate affects decision-making over time). (3) Lack of interpretation mechanism for generating results: Existing methods usually only output behavior categories and lack a mechanism for outputting skeletal geometric basis corresponding to the recognition results.

[0006] Therefore, there is an urgent need to provide a new skeletal behavior recognition method that encodes the geometric statistical distribution of bones into geometric perception cues, and conditionally modulates the spatiotemporal features of bones based on the geometric perception cues. At the same time, it realizes the collaborative modeling of spatial semantics and temporal semantics, so that the model can output the geometric interpretation basis corresponding to the recognition result while outputting the behavior category, thereby improving the interpretability, traceability and application reliability of the behavior recognition method. Summary of the Invention

[0007] This invention provides a behavior recognition method and system based on attribute-aware semantic collaborative networks, which can achieve behavior recognition with both recognition performance and interpretability.

[0008] The technical solution of the present invention is as follows: A behavior recognition method based on attribute-aware semantic cooperative networks includes the following steps: (1) Extract the human skeleton key point coordinate sequence from the video containing human behavior, and perform coordinate alignment and scale normalization on the human skeleton key point coordinate sequence to obtain a standardized skeleton sequence. (2) Extract geometric statistical information from the standardized skeletal sequence and construct a geometric perception cue vector; (3) Obtain modulation parameters based on geometric perception cue vectors, and perform linear modulation on the spatiotemporal features extracted by the backbone network according to the modulation parameters to obtain the spatiotemporal features after attribute modulation; (4) Input the spatiotemporal features modulated by the attributes into the attribute-aware semantic collaborative network, and perform collaborative modeling of the spatial structure information and temporal dynamic information of the skeleton sequence through the spatial semantic channel and the temporal semantic channel. Obtain the behavior recognition model through semantic consistency optimization training. (5) Input the video to be identified into the behavior recognition model, output the behavior category, and simultaneously output the geometric interpretation basis corresponding to the recognition result.

[0009] This invention focuses on human skeletal sequences. By extracting geometric statistical information from the skeletal structure, it constructs a geometric perception cue vector for characterization enhancement and uses the geometric perception cue to conditionally modulate the spatiotemporal features of the skeleton. Spatial semantics and temporal semantics are co-modeled to achieve behavior recognition that combines recognition performance and interpretability.

[0010] In step (1), the coordinates of the main joints of the human body are extracted from the video containing human behavior using a pose estimation algorithm, forming a set of human skeleton key point coordinate sequences that change with the frame number. ,in C Indicates the number of channels.T Indicates the number of time frames. V Indicates the number of joints.

[0011] Step (2) aims to transform the geometric structural information in the human skeletal sequence into a numerical structural semantic representation, providing an interpretable geometric prior for subsequent feature modulation. This invention uses distance spectrum geometric cues to uniformly encode the skeletal structure, that is, extracting geometric statistical features from two levels: joint-to-joint distance relationships and bone segment-perceived distance relationships, and mapping them into frame-level cue vectors.

[0012] Step (2) includes: extracting joint pair distance distribution and bone segment perception distance distribution from the standardized bone sequence, constructing a continuous geometric statistical distribution representation through kernel function mapping, and performing time aggregation on the continuous geometric statistical distribution representation to obtain a sequence-level geometric perception cue vector.

[0013] Preferably, step (2) includes: (2-1) For the standardized skeletal sequence, construct the first... The joint pair distance matrix of the frame is smoothed and saturated normalized to obtain the joint pair distance distribution; (2-2) The radial basis function (RBF) kernel set is used to perform soft quantization encoding on the joint pair distance distribution to obtain the global distance spectrum attribute vector. ; (2-3) Based on the bone segment set, construct the first Bone segment perception distance spectrum attribute vector of the frame : (2-4) will and By concatenating the data, a frame-level geometric attribute vector is obtained. ; For all frame-level geometric attribute vectors Perform time-dimensional mean pooling to obtain sequence-level geometry-aware cue vectors. .

[0014] In a further preferred embodiment, step (2-1) includes: For standardized skeletal sequences, construct the first Joint pair distance matrix of the frame:

[0015] in, C Indicates the number of coordinate channels. T Indicates the number of time frames. Indicates the number of joints; Indicates the first Frame number The joint and the first Euclidean distance between joints; , The first Frame number The, the The standardized coordinate vectors of each joint; The joint pair distance matrix is ​​smoothed and saturated normalized to obtain the normalized distance, thus yielding the joint pair distance distribution:

[0016] in, This represents the normalized joint pair distance; For a robust reference scale (the high 99th percentile of all joint pair distances in the training set can be used).

[0017] Order No. Frame number The original coordinates of each joint are Taking the central joint of the pelvis as the reference origin, its coordinates are denoted as... Then the centralized first The coordinates of each joint are ; The Euclidean distance between the cervical joint and the central pelvic joint is used as the first... Reference torso length of the frame :

[0018] ; For the entire skeletal sequence The average of the reference torso lengths is used to obtain the scale normalization factor. :

[0019] No. Frame number The standardized coordinates of a joint are defined as follows:

[0020] in, To prevent extremely small positive numbers with a denominator of zero; No. The normalized set of joints for a frame is .

[0021] In a further preferred embodiment, step (2-2) includes: The distance distribution is soft-quantized and encoded using a radial basis function (RBF) kernel set to obtain the first... Global distance spectrum attribute vector of a frame : Suppose that the radial basis function kernel set contains The distance kernel, the first A distance from the core center Defined as:

[0022] bandwidth Defined as:

[0023] in, This is the bandwidth adjustment factor; Based on the distances of all joint pairs, construct the first... Global distance spectrum attribute vector of a frame :

[0024] in, Indicates the first Frame in Global response value on a distance kernel; .

[0025] In a further preferred embodiment, step (2-3) includes: Define bone segment set ,in Indicates by the first The joint and the first A bone segment is connected by a set of joints; based on the set of bone segments, the first... Bone segment perception distance spectrum attribute vector of the frame :

[0026] in, Indicates the number of bone segments; Indicates the first Frame in The bone segment response value on the distance core; .

[0027] Further preferably, in step (2-4), for all frame-level geometric attribute vectors Perform time-dimensional mean pooling to obtain sequence-level geometry-aware cue vectors. ,include:

[0028] in, This represents the time aggregation operator.

[0029] Preferably, step (3) includes: (3-1) Sequence-level geometric perception cue vector Normalization is performed to obtain the attribute hint vector. ; (3-2) Attribute cue vectors are processed through a two-layer perceptron. By performing nonlinear mapping and dimensionality reduction, the embedding vector is obtained. :

[0030] in, and Here is the weight matrix for the two-layer perceptron. It is a non-linear activation function. and For the corresponding bias vector; (3-3) Using the FiLM conditional generator to process the embedding vector Perform a linear transformation to generate modulation parameters. :

[0031] in, Represents the weights of the linear transformation. For linear transformation parameters, This represents the scaling parameter. For offset parameters; (3-4) Extracting the spatiotemporal features of the skeletal sequence using a graph convolutional network Based on the modulation parameters, the spatiotemporal characteristics are recorded. Perform linear modulation to obtain the spatiotemporal characteristics after linear modulation. :

[0032] in, This is element-wise multiplication; It is a vector consisting entirely of 1s; and Dimensions and spatiotemporal characteristics The channel dimensions are consistent.

[0033] Preferably, step (4) includes: (4-1) Construct spatial semantic channels and obtain the output features of spatial semantic channels. ; (4-2) Construct a temporal semantic channel and obtain the output features of the temporal semantic channel. ; (4-3) Projecting the projector head into the projector head. , Mapped to spatial alignment vectors in the contrastive learning space Time alignment vector ,Will and Fusion to obtain global fusion features ; (4-4) Set up the momentum encoder for momentum updates. The output of the momentum encoder is respectively the corresponding... , , momentum branch representation , , ; (4-5) Construct cross-domain contrast loss, perform semantic consistency optimization training, and obtain the behavior recognition model.

[0034] In a further preferred embodiment, step (4-1) includes: Construct a spatial semantic channel to capture the semantic relationships of the human skeleton in its spatial topology; obtain the output features of the spatial semantic channel. :

[0035] in, This indicates a channel-by-channel multiplication operation; Represents a skeleton-based topological adjacency matrix Spatial graph convolution operation; , These are the scaling and offset parameters for the spatial semantic channel, respectively, generated by the FiLM condition generator. Obtained through mapping.

[0036] In a further preferred embodiment, step (4-2) includes: Construct a temporal semantic channel to capture the dynamic patterns of actions changing over time; obtain the output features of the spatial semantic channel. :

[0037] in, Indicates the use of temporal adjacency matrix Perform graph convolution; , These are the scaling and offset parameters for the temporal semantic channels, respectively, generated by the FiLM condition generator. Obtained through mapping.

[0038] The modulation parameters in the spatial channel are determined by the embedding vector. Generated through independent linear mappings, i.e.

[0039] in, , For learnable mapping matrix, , This is the bias vector.

[0040] The modulation parameters in the time channel are determined by the embedding vector. Generated through independent linear mappings, i.e.

[0041] in, , For learnable mapping matrix, , This is the bias vector.

[0042] Further preferred, for Introducing a multi-head self-attention mechanism Perform weighted aggregation and hierarchical normalization:

[0043] in, Indicates the first The output of each attention head; , , They represent the first Each attention head's query, key, and value; This represents the feature dimension of each attention head. This is the scaling factor; This indicates the number of attention heads.

[0044] In a further preferred embodiment, step (4-3) includes: Through the projection head , Will , Mapped to spatial alignment vectors in the contrastive learning space Time alignment vector :

[0045] Fusion , Obtain global fusion features :

[0046] in, For fusion operators, This is a global projection head.

[0047] In a further preferred embodiment, step (4-4) also includes: The property momentum update mechanism of the momentum encoder is as follows:

[0048] in, This is the momentum smoothing coefficient. Indicates the first The encoder parameters are queried during each training iteration; Indicates the first The parameters of the momentum encoder during the next training iteration; This indicates the updated momentum encoder parameters.

[0049] In a further preferred embodiment, step (4-5) includes: Skeletal samples from different action sequences form negative sample pairs, while multi-view sequences of the same action form positive sample pairs. The optimization objective is:

[0050] in, Temperature coefficient; Indicates the index of the current sample; ) represents a positive sample pair; for any and , Negative sample pairs; Candidate key vector set; This represents the similarity function.

[0051] Let the set of sample pairs be The overall goal is:

[0052] in, This is the balance coefficient; Represents a set A pair of aligned vectors in the array.

[0053] Preferably, step (5) includes: (5-1) Input the video to be identified into the behavior recognition model to obtain the human behavior category in the video to be identified; (5-2) Obtain modulation parameters through steps (1)-(4) Calculate the modulation contribution score of each feature channel to the current recognition result. :

[0054] in, For the first Scaling and modulation parameters for each feature channel; For the first Offset modulation parameters for each characteristic channel; These are the weighting coefficients; For the first The response intensity of each feature channel in the modulated spatiotemporal features;

[0055] in, For the first The feature channel in the first Frame, First Modulated spatiotemporal features corresponding to each joint position The amount; Calculate the contribution of each joint. :

[0056] in, Indicates the first The first feature channel and the second The association weights between the joints; (5-3) Sort the contribution of all joints in descending order and select the joints with the largest contribution to form a set of key contribution units; output the joint geometric parameters corresponding to the set of key contribution units as the basis for geometric interpretation.

[0057] The joint geometric parameters include joint pair distance parameters, bone segment length parameters, and distance spectrum distribution parameters.

[0058] The present invention also provides a behavior recognition system based on an attribute-aware semantic collaborative network, including a memory, a processor, and a computer program stored in the memory and executable on the processor; The memory stores a behavior recognition model; the video to be recognized is input into the behavior recognition model, and the processor executes the computer program to predict the behavior type in the video to be recognized; The behavior recognition model is obtained through training the steps in the method.

[0059] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) Enhanced interpretability. This invention constructs a geometric perception cue vector by statistically encoding the distance relationships between joint pairs and bone segments in the skeletal sequence, and further generates modulation parameters corresponding to intermediate features, so that the recognition results can be traced back to the corresponding geometric cue and modulation basis, thereby improving the interpretability of the model output.

[0060] (2) Improved stability. This invention adopts a geometric statistical representation based on relative distance, and combines robust normalization and distance spectrum coding, which can reduce the impact of absolute coordinate changes, scale changes and local jitter on feature representation, thereby improving the stability of skeletal behavior recognition in complex scenes.

[0061] (3) Easy to deploy. The geometric cue coding and conditional modulation mechanism of the present invention can be used as a front-end geometric coding module and an intermediate feature modulation module, and can be combined with existing skeletal spatiotemporal feature extraction networks, which facilitates deployment in different skeletal data sources and application scenarios. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating the behavior recognition method based on attribute-aware semantic collaborative networks. Figure 2 This is a schematic block diagram of a behavior recognition system based on an attribute-aware semantic collaborative network. Figure 3 A flowchart illustrating the process of encoding geometric perception prompts and modulating attribute conditions; Figure 4 This is a flowchart illustrating the semantic collaborative consistency learning process. Detailed Implementation

[0063] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not limit it in any way.

[0064] The core of this invention lies in: firstly, statistically encoding the distance relationships between joint pairs and bone segments in the skeletal sequence to construct a sequence-level geometric perception cue vector; then, generating feature modulation parameters based on the geometric perception cue vector to conditionally modulate the spatiotemporal features of the backbone network; finally, achieving behavior recognition through collaborative modeling of spatial semantic channels and temporal semantic channels, and outputting the geometric interpretation basis corresponding to the recognition result.

[0065] Unlike traditional graph convolution or implicit attention mechanisms that rely solely on topological connections, this invention constructs an attribute-aware semantic collaborative network to achieve collaborative modeling of spatial morphology and temporal dynamics in a geometric attribute-driven manner.

[0066] Building upon the spatiotemporal feature modeling achieved through a unified backbone network, an attribute semantic cue modulation mechanism is further introduced to enable semantic-driven and interpretable feature learning of human movements at the structural level. This mechanism includes: quantizing and encoding joint pair distance information and bone segment distance information in the skeletal sequence to obtain a sequence-level geometric perception cue vector; generating scaling and offset parameters from the cue vector; and performing channel-by-channel linear modulation on the spatiotemporal features extracted by the backbone network to enhance the responsiveness of feature representation to differences in skeletal structure and dynamic changes in movement.

[0067] like Figure 1 and Figure 2 As shown, the behavior recognition method based on attribute-aware semantic collaborative network of the present invention includes the following steps: Step S101: Skeletal sequence extraction Step S101 is performed based on the bone sequence extraction unit.

[0068] Obtaining 3D skeleton sequences from video data ,in C Indicates the number of coordinate channels. T Indicates the number of time frames. V Indicates the number of joints.

[0069] Step S102: Skeletal sequence standardization and enhancement Step S102 is performed based on skeletal standardized units.

[0070] Perform coordinate unification and scale normalization. Perform data augmentation on the skeletal sequence to improve generalization ability; augmentation may include joint-level random occlusion, frame-level random occlusion, and geometric transformations such as rotation, mirroring, or shearing.

[0071] Step S103: Geometric perception cue encoding Step S103 aims to transform the geometric structural information in the human skeletal sequence into a numerical geometric statistical distribution representation, providing an interpretable geometric prior for subsequent feature modulation. This invention employs distance spectrum geometric cues to uniformly encode the skeletal structure, extracting geometric statistical features from two levels: joint-to-joint distance relationships and bone segment-perceived distance relationships, and mapping them to frame-level cue vectors.

[0072] like Figure 3 As shown, geometry-aware cue encoding specifically includes the following steps: ① Joint distance matrix construction and robust normalization Order No. Frame number The original coordinates of each joint are Taking the central joint of the pelvis as the reference origin, its coordinates are denoted as... Then the centered joint coordinates are The Euclidean distance between the cervical joint and the central pelvic joint is used as the first... Reference torso length of the frame : , For the entire sequence The average of the reference torso lengths is used to obtain the scale normalization factor:

[0073] Therefore, the first Frame number The standardized coordinates of a joint are defined as follows:

[0074] in, To prevent extremely small positive numbers with a denominator of zero.

[0075] No. The normalized set of joints for a frame is .in, Indicates the first Frame number The standardized coordinate vector of each joint. C Indicates the number of coordinate channels. Indicates the number of joints.

[0076] Based on the standardized joint set, construct the first... Joint pair distance matrix of the frame:

[0077] in, Indicates the first Frame number The joint and the first The Euclidean distance between the joints.

[0078] To mitigate the impact of abnormally large distances on the statistical results, a robust reference scale is introduced. . The highest quantile of the distances between all joint pairs in the training set can be taken as 99%. Further, the distance matrix is ​​smoothed and saturated normalized to obtain the normalized distance:

[0079] in, This represents the normalized joint pair distance. This normalization method can suppress extreme values ​​while preserving relative distance relationships, thus improving the stability of geometric cue coding.

[0080] ② Global distance spectrum attribute To transform the normalized distance relationships into a continuous statistical representation that can be used for network modeling, a radial basis function (RBF) kernel set is used to perform soft quantization encoding on the distance distribution.

[0081] Suppose that the RBF core set contains There are distance kernels, the center of which is defined as:

[0082] Bandwidth is defined as:

[0083] in, Indicates the first One distance from the core center, Indicates kernel bandwidth. This is the bandwidth adjustment coefficient. Based on the distances of all joint pairs, the first... Global distance spectrum attribute vector of the frame:

[0084] in, Indicates the first Frame in Global response value on a distance kernel; This represents the global distance spectrum attribute vector. This vector is used to describe the overall structural distribution of human posture at different distance scales.

[0085] ③ Bone segment perception distance spectrum attributes To further highlight the local structural information in the human kinematic topology, distance relationships on the bone segment set are modeled in addition to the global distance spectrum. The bone segment set is... ,in Indicates by the first The joint and the first A bone segment connected by several joints. This can be given by the skeletal topology.

[0086] Based on the bone segment set, construct the first Frame bone segment perception distance spectrum attribute vector:

[0087] in, Indicates the number of bone segments; Indicates the first Frame in The bone segment response value on the distance core; This represents the bone segment sensing distance spectrum attribute vector. This vector is used to describe the morphological changes of local limb segments at different distance scales.

[0088] Furthermore, the first Global distance spectrum attribute vector of a frame Bone segment perception distance spectrum attribute vector By concatenating the first and last elements according to the feature dimension, a frame-level geometric attribute vector is obtained. For all frame-level geometric attribute vectors Perform time-dimensional mean pooling to obtain sequence-level geometry-aware cue vectors. .

[0089] Step S104: Attribute Condition Modulation Generation Step S104 is based on the attribute-conditional modulation unit, using the FiLM conditional generator for feature-wise linear modulation. This step maps the geometrically aware cue vector obtained in step S103 to modulation parameters and performs linear modulation on the intermediate features of the backbone network extracted from the skeletal spatiotemporal features. The backbone network is the pre-spatiotemporal coding network in the attribute-aware semantic collaborative network, and its output intermediate spatiotemporal features are denoted as... .

[0090] The first obtained in step S103 The geometric attribute vector of a frame is denoted as ,in , For attribute dimensions. For all frame-level geometric attribute vectors. Perform time-based statistical aggregation to obtain sequence-level geometric cue vectors. :

[0091] in, This represents a temporal aggregation operator used for statistical aggregation of frame-level geometric attributes. The temporal aggregation operator is mean pooling. To stabilize the attribute scale distribution among different samples, the sequence-level geometric cue vector is... Perform layer normalization to obtain the normalized geometric hint vector:

[0092] in, This represents the attribute hint vector after layer normalization; The representation layer normalization operator is used to standardize the input vector to zero mean and unit variance, thereby reducing the impact of attribute scale differences between different samples on the generation of modulation parameters. It is calculated from the joint pair distance statistics and bone segment distance statistics of the skeletal sequence, without the need for manual semantic annotation.

[0093] Subsequently, a two-layer perceptron is used for nonlinear mapping and dimensionality reduction to obtain the embedding vector. :

[0094] in, and Here is the weight matrix for the two-layer perceptron. It is a non-linear activation function. and This corresponds to the bias vector. Embedded vector. The statistical information from the geometrically perceived cues is synthesized and used as an intermediate representation for generating conditional modulation parameters. A linear mapping layer is used to transfer the conditional embedding vector. Scaling and offset parameters required for mapping to feature modulation :

[0095] in, Represents the weights of the linear transformation. For linear transformation parameters, This represents the scaling parameter. Offset parameter. Scaling parameter. With offset parameter Used for global attribute condition modulation of intermediate features in attribute-aware semantic collaborative networks.

[0096] Spatiotemporal feature modeling of skeletal sequences is performed based on graph convolutional networks. The intermediate spatiotemporal features generated during the encoding process are denoted as... The attribute-conditional modulation module generates scaling parameters based on semantic attributes. With bias parameters And the intermediate spatiotemporal features of attribute-aware semantic collaborative networks. By performing channel-by-channel linear scaling and translation, the modulated intermediate features are obtained. :

[0097] in, This is element-wise multiplication; It is a vector consisting entirely of 1s; and Dimensions and modulated features The channel dimension is consistent. This operation enables semantically driven linear scaling and translation in the feature space, allowing the model to adapt to different poses and structural changes.

[0098] Steps S105-S106: Attribute-aware semantic collaborative network modeling Steps S105 and S106 are executed by the attribute-aware semantic collaboration unit to perform semantic collaboration consistency learning. Spatial semantic channels and temporal semantic channels are constructed respectively to collaboratively model the linearly modulated spatiotemporal features, and the consistency of spatial and temporal representations in a unified semantic space is constrained by semantic consistency optimization. In a preferred implementation, semantic consistency optimization is achieved through cross-domain contrastive learning.

[0099] To achieve simultaneous optimization of spatial topological semantics and temporal evolution semantics, feature collaborative modeling is performed through a dual semantic channel mapping structure under attribute modulation.

[0100] like Figure 4 As shown, semantic collaborative consistency learning includes the following steps: For skeletal sequences ,inC Indicates the number of channels. T Indicates the number of time frames. V This represents the number of key points. In this embodiment, the normalized geometric cue vector obtained in step S103 is uniformly denoted as... Its dimensions are In step S104, the conditional embedding vector The statistical information of the geometric perception cues is comprehensively expressed by vectors. The corresponding scaling and offset parameters are generated by mapping the FiLM condition generators corresponding to the spatial and temporal channels to achieve conditional modulation of spatial and temporal semantic features.

[0101] The spatial semantic channel is responsible for capturing the semantic relationships of the human skeleton in its spatial topology. The modulated intermediate spatiotemporal features output in step S104 are: The output features of the spatial semantic channel are denoted as Its definition

[0102] in: Represents the output features of the spatial semantic channel; This represents the spatial adjacency matrix corresponding to the skeleton topology; : Represents a spatial adjacency matrix Spatial graph convolution operations are used to model the local topological relationships between joints and bone segments; : indicates that it is composed of conditional embedding vectors The scaling parameters obtained by mapping through the spatial channel FiLM condition generator are used to adjust the response intensity of each feature channel in spatial semantic modeling; : From conditional embedding vectors The offset parameters obtained by mapping through the spatial channel FiLM condition generator are used to perform conditional offset compensation for spatial semantic features; : indicates a channel-by-channel multiplication operation.

[0103] Furthermore, the modulation parameters in the spatial channel are determined by the conditional embedding vector. Generated through independent linear mappings, i.e.

[0104] in, , For learnable mapping matrix, , This is the bias vector. Through the above spatial semantic modeling, the model can adaptively adjust the information propagation intensity between different joints and bone segments based on the geometric structure information of the current sample, under the constraints of skeleton topology.

[0105] Temporal semantic channels are used to capture the dynamic patterns of actions changing over time. This is achieved by constructing a temporal sequence graph adjacency matrix. This allows for the establishment of connections between adjacent frames for the same joint, enabling cross-time relationship modeling.

[0106]

[0107] in: This represents the output features of the temporal semantic channel; : Represents adjacency matrix based on temporal sequence The temporal graph convolution operation is used to aggregate the continuous dynamic information of the same joint in adjacent time frames; Represents the conditional embedding vector The scaling parameters obtained by mapping through the temporal channel FiLM condition generator are used to adjust the response intensity of each feature channel in temporal semantic modeling; : indicates that it is composed of conditional embedding vectors The offset parameters obtained by mapping through the FiLM condition generator in the time channel are used to perform conditional offset compensation on the temporal semantic features.

[0108] Furthermore, the modulation parameters in the time channel are determined by the conditional embedding vector. Generated through independent linear mappings, i.e.

[0109] in, , For learnable mapping matrix, , This is the bias vector. Through the above temporal semantic modeling, the model can adaptively adjust the representation of temporal dynamic features based on the geometric attribute information of the current skeletal sequence.

[0110] In addition, to enhance the ability to model long-term dependencies, this module... Introducing a multi-head self-attention mechanism:

[0111] in, Indicates the first The output of each attention head; , , They represent the first Each attention head's query, key, and value; This represents the feature dimension of each attention head. This is the scaling factor; This indicates the number of attention heads.

[0112] By employing multi-head attention to weighted aggregate and layer-normalize temporal features, a global temporal semantic representation is obtained. Ultimately, the outputs of the spatial and temporal channels achieve a unified representation in the semantic space, providing a foundation for subsequent semantic coherence and consistency optimization.

[0113] The semantic collaborative optimization mechanism continues the theoretical framework of contrastive learning during the training phase, but does not rely on manual labels or external semantic annotations. Instead, it achieves consistent collaboration of spatial-temporal semantics through attribute modulation.

[0114] The spatial and temporal semantic channel outputs after attribute modulation: , in , This refers to the projection head, which is used to map the channel outputs to vector representations in the contrastive learning space. , These are the spatial alignment vector and the time alignment vector, respectively.

[0115] Let the global fusion feature be ,in For fusion operators, This is a global projection head. To build stable consistency constraints, one implementation sets up a momentum encoder for momentum updates, whose outputs are denoted as... , , ,correspond , , The momentum branch is represented.

[0116] The query encoder is used to output the query representation of the current sample, and its parameters are updated through backpropagation of the overall loss function; the momentum encoder is used to output the corresponding target representation, and its parameters do not directly participate in backpropagation, but are updated by the query encoder parameters through an exponential moving average.

[0117] Let the first During the next training iteration, the momentum encoder parameters are updated as follows:

[0118] in, This is the momentum smoothing coefficient. Indicates the first The encoder parameters are queried during each training iteration; Indicates the first The parameters of the momentum encoder during the next training iteration; This represents the updated momentum encoder parameters. Through the above update method, the momentum encoder parameters evolve smoothly relative to the query encoder parameters, thereby improving the stability of the target representation and enhancing the consistency of the semantic consistency optimization training process.

[0119] Skeletal samples from different action sequences naturally form negative sample pairs, while multi-view sequences of the same action form positive sample pairs. The optimization objective can be expressed as:

[0120] in, Temperature coefficient; Indicates the index of the current sample; ) represents a positive sample pair; for any and , Negative sample pairs; Represents the set of candidate key vectors; This represents the similarity function.

[0121] This loss function maximizes the similarity of positive samples and minimizes the similarity of negative samples in the embedding space, thereby achieving structurally self-supervised discriminative optimization.

[0122] Let the set of sample pairs be The overall goal is:

[0123] in, This is a balancing coefficient used to adjust the importance of different cross-domain consistency terms; Represents a set A pair of aligned vectors in the embedding space. Through this mechanism, features from different domains (spatial-temporal-global) are uniformly constrained within the embedding space, thereby achieving cross-domain semantic consistency and structural alignment.

[0124] Step S107: Action Recognition and Interpretation Generation Based on the fused semantic features obtained in step S105 Input to the classification layer. The classification layer bases its input on the fused semantic feature vector. The corresponding behavior category is output. To provide interpretable evidence for the behavior recognition results, this invention further calculates a key contribution set based on geometric cue information and feature modulation response.

[0125] Specifically, based on the characteristic modulation parameters obtained in step S104 Calculate the modulation contribution score of each feature channel to the current recognition result. Modulation contribution score This measure of the influence of each feature channel in the current sample can be expressed as:

[0126] in, , Indicates the number of channels. Indicates the first Modulation contribution score of each characteristic channel; For the first Scaling and modulation parameters for each feature channel; For the first Offset modulation parameters for each characteristic channel; These are weighting coefficients used to balance the influence of the two types of modulation parameters; Indicates the first The response intensity of each feature channel in the modulated spatiotemporal features. Defined as the average absolute response value of the channel in the time and space dimensions, i.e.:

[0127] Among them, the modulated spatiotemporal features Corresponding element Represented as the first The first channel in the Frame, First Response values ​​at each joint location.

[0128] Then, based on the correspondence between feature channels and skeletal joints, the modulation contribution score of each feature channel is mapped to the contribution of the corresponding joint; the contribution of the joint can be expressed as:

[0129] in, Indicates the first The contribution of each joint to the current recognition result. Indicates the first The first feature channel and the second The correlation weights between joints are derived from the modulated spatiotemporal features. The response intensity is normalized at the joint node dimension and used to characterize the proportion of the contribution of this feature channel among the joint nodes.

[0130] The contributions of all joints are sorted in descending order, and the joints with the highest contributions are selected to form a set of key contribution units. Based on the geometric cue features corresponding to the set of key contribution units, the key joint pair distance relationships, key bone segment structural relationships, and key distance spectrum response patterns related to the current behavior category are determined. This information is output as structured data in the form of joint indices, corresponding contribution values, and corresponding geometric parameters as the basis for geometric interpretation. Geometric parameters include joint pair distance parameters, bone segment length parameters, and distance spectrum distribution parameters.

[0131] In practical applications, the video to be identified is input into the skeleton extraction module to obtain the skeleton sequence, which is then processed through steps (1) to (5) for inference. During the training phase, the query encoder and momentum encoder are used for comparative learning and optimization. During the inference phase, only the skeleton extraction module, geometric cue encoding module, backbone feature extraction module, attribute semantic co-modeling module, classification output module, and interpretation output module are retained. The momentum encoder, negative sample queue, and contrastive loss calculation module do not participate in the inference.

[0132] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A behavior recognition method based on attribute-aware semantic collaborative networks, characterized in that, Includes the following steps: (1) Extract the human skeleton key point coordinate sequence from the video containing human behavior, and perform coordinate alignment and scale normalization on the human skeleton key point coordinate sequence to obtain a standardized skeleton sequence. (2) Extract geometric statistical information from the standardized skeletal sequence and construct a geometric perception cue vector, including: (2-1) For the standardized skeletal sequence, construct the first... The joint pair distance matrix of the frame is smoothed and saturated normalized to obtain the joint pair distance distribution; (2-2) The radial basis function kernel set is used to perform soft quantization encoding on the joint pair distance distribution to obtain the global distance spectrum attribute vector. ; (2-3) Based on the bone segment set, construct the first Bone segment perception distance spectrum attribute vector of the frame : (2-4) will and By concatenating the data, a frame-level geometric attribute vector is obtained. ; For all frame-level geometric attribute vectors Perform time-dimensional mean pooling to obtain sequence-level geometry-aware cue vectors. , Indicates the number of time frames; (3) Obtain modulation parameters based on geometric perception cue vectors, and perform linear modulation on the spatiotemporal features extracted by the backbone network according to the modulation parameters to obtain the spatiotemporal features after attribute modulation; (4) Input the linearly modulated spatiotemporal features into the attribute-aware semantic collaborative network, and perform collaborative modeling of the spatial structure information and temporal dynamic information of the skeleton sequence through the spatial semantic channel and the temporal semantic channel. Obtain the behavior recognition model through semantic consistency optimization training. (5) Input the video to be identified into the behavior recognition model, output the behavior category, and simultaneously output the geometric interpretation basis corresponding to the recognition result.

2. The behavior recognition method based on attribute-aware semantic collaborative network according to claim 1, characterized in that, Step (2-1) includes: For standardized skeletal sequences, construct the first Joint pair distance matrix of the frame: in, Indicates the number of joints; Indicates the first Frame number The joint and the first Euclidean distance between joints; , The first Frame number The, the The standardized coordinate vectors of each joint; The joint pair distance matrix is ​​smoothed and saturated normalized to obtain the normalized distance, thus yielding the joint pair distance distribution: in, This represents the normalized joint pair distance; For robust reference standards.

3. The behavior recognition method based on attribute-aware semantic collaborative network according to claim 1, characterized in that, Step (2-2) includes: The distance distribution is soft-quantized and encoded using a radial basis function kernel set to obtain the first... Global distance spectrum attribute vector of a frame : Suppose that the radial basis function kernel set contains The distance kernel, the first A distance from the core center Defined as: bandwidth Defined as: in, This is the bandwidth adjustment factor; Based on the distances of all joint pairs, construct the first... Global distance spectrum attribute vector of a frame : in, Indicates the first Frame in Global response value on a distance kernel; .

4. The behavior recognition method based on attribute-aware semantic collaborative network according to claim 1, characterized in that, Steps (2-3) include: Define bone segment set ,in Indicates by the first The joint and the first A bone segment is connected by a set of joints; based on the set of bone segments, the first... Bone segment perception distance spectrum attribute vector of the frame : in, Indicates the number of bone segments; Indicates the first Frame in The bone segment response value on the distance core; .

5. The behavior recognition method based on attribute-aware semantic collaborative network according to claim 1, characterized in that, Step (3) includes: (3-1) Sequence-level geometric perception cue vector Normalization is performed to obtain the attribute hint vector. ; (3-2) Attribute cue vectors are processed through a two-layer perceptron. By performing nonlinear mapping and dimensionality reduction, the embedding vector is obtained. ; (3-3) Using the FiLM conditional generator to process the embedding vector Perform a linear transformation to generate modulation parameters; (3-4) Extracting the spatiotemporal features of the skeletal sequence using a graph convolutional network Based on the modulation parameters, the spatiotemporal characteristics are recorded. Perform linear modulation to obtain the spatiotemporal characteristics after linear modulation. .

6. The behavior recognition method based on attribute-aware semantic cooperative network according to claim 5, characterized in that, Step (4) includes: (4-1) Construct spatial semantic channels and obtain the output features of spatial semantic channels. ; (4-2) Construct a temporal semantic channel and obtain the output features of the temporal semantic channel. ; (4-3) Projecting the projector head into the projector head. , Mapped to spatial alignment vectors in the contrastive learning space Time alignment vector ,Will and Fusion to obtain global fusion features ; (4-4) Set up the momentum encoder for momentum updates. The output of the momentum encoder is respectively the corresponding... , , momentum branch representation , , ; (4-5) Construct cross-domain contrast loss, perform semantic consistency optimization training, and obtain the behavior recognition model.

7. The behavior recognition method based on attribute-aware semantic cooperative network according to claim 6, characterized in that, Step (4-1) includes: Construct a spatial semantic channel to capture the semantic relationships of the human skeleton in its spatial topology; obtain the output features of the spatial semantic channel. : in, This indicates a channel-by-channel multiplication operation; Represents a skeleton-based topological adjacency matrix Spatial graph convolution operation; , These are the scaling and offset parameters for the spatial semantic channel, respectively, generated by the FiLM condition generator. Obtained through mapping.

8. The behavior recognition method based on attribute-aware semantic cooperative network according to claim 6, characterized in that, Step (4-2) includes: Construct a temporal semantic channel to capture the dynamic patterns of actions changing over time; obtain the output features of the temporal semantic channel. : in, Indicates the use of temporal adjacency matrix Perform graph convolution; , These are the scaling and offset parameters for the temporal semantic channels, respectively, generated by the FiLM condition generator. Obtained through mapping.

9. A behavior recognition system based on an attribute-aware semantic collaborative network, characterized in that, Includes a memory, a processor, and a computer program stored in the memory and executable on the processor; The memory stores a behavior recognition model; when the video to be recognized is input into the behavior recognition model, the processor executes the computer program to predict the behavior type in the video to be recognized, and simultaneously outputs the geometric interpretation basis corresponding to the recognition result; The behavior recognition model is obtained by training through the steps of the method described in any one of claims 1-8.

Images