Behavior recognition detection method and system based on structured behavior semantic space

Abstract

A behavior recognition detection method and system based on a structured behavior semantic space, in an offline stage, a structured behavior semantic space is established according to a hierarchical verb tree in VerbNet, after a unified database with a unified label system is constructed through consistency processing, semantic representation and geometric representation of nodes in the semantic space are extracted from a general behavior data set and a physical-semantic mapping model is established, in an online stage, semantic behavior categories of a to-be-tested sample are recognized according to the trained physical-semantic mapping model, and in a specific behavior understanding scene, a specified behavior category is recognized through migration. After the structured behavior semantic space and the unified database are established, the semantic information and the geometric information of the structured semantic space are extracted, and according to the mapping between the physical space and the semantic space, migration of the specific behavior understanding scene is carried out, so that the detection of the behavior category is realized.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of artificial intelligence, and particularly relates to a behavior recognition detection method and system based on a structured behavior semantic space. BACKGROUND

[0002] Human behavior understanding is an important branch of computer vision, and the goal is to infer the behavior of people in images or videos and the interaction with the environment. Behavior understanding is widely used in intelligent driving, security, and robot fields, and is one of the most important artificial intelligence technologies for the industry, and is increasingly attracting people's attention. Machine learning mainly studies computer algorithms that can automatically improve through experience, and usually obtains, abstracts, and summarizes key information and knowledge from a large amount of experience data, and an artificial neural network is an important branch of machine learning, and is currently widely used in artificial intelligence related tasks. The technical problem of the existing image classification method is that there are ambiguity and ambiguity problems of behavior semantic labels in task design, and different behavior data sets are incompatible with each other due to the above-mentioned semantic label ambiguity problem when being constructed, and the trained models are also difficult to migrate to each other, which brings great challenges to truly realize general and generalizable behavior understanding. SUMMARY

[0003] The application proposes a behavior recognition detection method and system based on a structured behavior semantic space to solve the above problems in the prior art. After establishing a structured behavior semantic space and a unified database, semantic information and geometric information of the structured semantic space are extracted, and according to the mapping of the physical space and the semantic space, the migration of the specific behavior understanding scene is performed to realize the detection of the behavior category.

[0004] The application is implemented by the following technical scheme:

[0005] The application relates to a behavior recognition detection method based on a structured behavior semantic space. In an offline stage, a structured behavior semantic space is established according to a hierarchical verb tree in VerbNet, and after a unified database with a unified label system is constructed through consistency processing, semantic representations and geometric representations of nodes in the semantic space are extracted from a general behavior data set and a physical-semantic mapping model is established. In an online stage, the semantic behavior category of a to-be-detected sample is recognized according to the trained physical-semantic mapping model, and in a specific behavior understanding scene, a specified behavior category is recognized through migration.

[0006] The structured behavior semantic space takes each node in the hierarchical verb tree in VerbNet as a category in the semantic space; each node contains a plurality of verbs with common semantics and provides rich textual descriptions for explaining the meaning of the node; the semantic relationship between the nodes is established through hierarchical organization of the nodes, and structured knowledge is formed.

[0007] The unified database with a unified label system refers to: through behavior category mapping, the behavior label definitions of each existing behavior dataset are mapped through the structured behavior semantic space, so as to have a unified behavior definition and a unified format database.

[0008] The existing behavior dataset refers to: a variety of modal behavior recognition task datasets, including: pictures, videos, human skeletons, human motion capture data, etc. However, due to the problem of non-uniform semantic labels, the behavior category task design of these datasets is isolated from each other, and does not support unified training.

[0009] The behavior category mapping refers to: mapping the behavior categories defined in any behavior dataset to the corresponding verbs under any node of the structured behavior semantic space; preferably, the process is realized through semi-automatic crowdsourcing annotation, and the behavior categories of each dataset only need to be aligned once.

[0010] The semantic representation of the node in the semantic space refers to: encoding the multiple behavior verbs and rich textual descriptions contained in the node, specifically: after integrating the verbs and example sentences contained in the node m j into a sentence, the text encoder of the CLIP (CLIP: Connecting Text and Images) model encodes it into a semantic information vector t j .

[0011] The geometric representation refers to: modeling the hierarchical relationship between nodes through hyperbolic space, specifically: after randomly initializing the geometric information vectors of all nodes in the structured semantic space, 600 cycles of optimization of the target hyperbolic space are performed using Riemannian stochastic gradient descent, to obtain the geometric information vector g j of each node: the distance between nodes with the same parent node should be close; the distance between nodes with different parent nodes should be far; the shortest path between sibling nodes should be close to their parent nodes.

[0012] The physical space refers to: physical visual signals carrying semantic information, including pictures, videos, skeletons, motion capture data, etc.

[0013] The physical-semantic mapping model includes: a node decoupling unit, a pseudo-label enhancement unit, a semantic alignment unit, and a geometric alignment unit. Specifically: the node decoupling unit decouples the physical representation based on the semantic information of the nodes to obtain a visual representation conditioned on the nodes; the pseudo-label enhancement unit utilizes the structural and linguistic prior information of the semantic space to calculate the correlation matrix C between nodes, and calculates labels for verb nodes with low certainty (i.e., fewer training samples) based on the high certainty of the verb node labels; the semantic alignment unit aligns the semantic and physical representations of nodes in the semantic space and outputs the semantic behavior category of the test sample, i.e., the node in the structured behavioral semantic space corresponding to the test sample; the geometric alignment unit aligns the geometric and physical representations of nodes in the semantic space and outputs the semantic behavior category of the test sample.

[0014] The alignment refers to: encoding the semantic representation p of a node in the semantic space into an object representation v using a deep neural network, and then labeling the verb node based on this semantic representation. Calculate the loss function L = L cls (D T (v i ,t i ),y i )+L cls (D G (v i ,g i ),y i ), where: N is the number of target nodes in the structured semantic space S, v i ,t i ,g i These represent the physical, semantic, and geometric representations of a node, respectively. T (·,·) is the distance metric function between the object representation and semantic representation of a node, D G (·,·) is the distance metric function between the object representation and the geometric representation of a node, L cls (·,·) is the classification loss function.

[0015] The decoupling mentioned above refers to: based on the semantic information v of the node raw The network f(·) is decomposed into node-conditional visual representations. Where: v i It is calculated using node index i as a condition.

[0016] The aforementioned identification of a specified behavior category through transfer learning refers to: based on the output of the trained physical-semantic mapping model, predicting the behavior category S in the transfer dataset. ori And the prediction of node category S node The prediction of node categories is further improved through a learnable network. nodeTransform into node-conditional behavior category prediction S ′ ori Ultimately, by combining S ori and S ′ ori Output the predicted behavior category to complete the transfer of specific behavior understanding in the scenario.

[0017] This invention relates to a system for implementing the above-mentioned method, comprising: a structured behavioral semantic space establishment unit, a unified database establishment unit, a node semantic information extraction unit, a node geometric information extraction unit, a physical-semantic mapping model unit, and a specific behavior understanding scenario transfer unit, wherein: the structured behavioral semantic space establishment unit establishes a structured behavioral semantic space by organizing corpus and processing geometric information based on the hierarchical verb tree in VerbNet; the unified database establishment unit performs consistency processing based on existing behavior understanding datasets to obtain a unified database with a unified label system; the semantic information extraction unit performs natural language encoding based on multiple behavioral verbs and rich textual descriptions contained in nodes to obtain a semantic information vector for each node; the geometric information extraction unit obtains a geometric information vector for each node by modeling and optimizing hyperbolic space based on the hierarchical relationships between nodes defined in VerbNet; the physical-semantic mapping model unit establishes a physical-semantic mapping model based on the unified database, node semantic information, and node geometric information to identify the semantic behavior category of a given sample; and the specific behavior understanding scenario transfer unit outputs the recognition result for the new specified behavior category based on the physical-semantic mapping model through label transfer mapping.

[0018] Technical effect

[0019] This invention utilizes the hierarchical verb tree in VerbNet to organize the corpus and process geometric information, establishing a structured behavioral semantic space. The semantic information extraction unit performs natural language encoding based on the multiple behavioral verbs and rich textual descriptions contained in each node, obtaining a semantic information vector for each node. The geometric information extraction unit, based on the hierarchical relationships between nodes defined in VerbNet, obtains a geometric information vector for each node through hyperbolic space modeling and optimization. Based on the semantic information of the nodes, the physical representation is transformed to obtain a new, decoupled representation conditioned on the nodes. Utilizing the structural and linguistic prior information of the semantic space, the correlation matrix C between nodes is calculated. Based on the labels of verb nodes with high determinism, labels of verb nodes with low determinism are calculated for pseudo-label enhancement. Compared to existing technologies, this invention significantly improves behavioral understanding performance. In the tree-structured hierarchical classification task, the model proposed in this invention achieves an accuracy of 34.46 mAP, with particularly significant performance under few-shot learning, reaching an accuracy of 21.84 mAP. On multiple large-scale behavioral understanding benchmarks, transfer learning results have also achieved significant performance improvements compared to the current state of the field: 48.50 mAP on the HOI detection benchmark HICO, 79.80% accuracy on the video behavioral data benchmark HAA500, 87.46% accuracy on HMDB51, 60.55% accuracy on Charades, and 89.05% accuracy on Kinetics-400. Attached Figure Description

[0020] Figure 1 This is a flowchart of the present invention;

[0021] Figure 2 This is a schematic diagram of the system of the present invention;

[0022] Figure 3 This is a schematic diagram of the structured behavioral semantic space of the present invention;

[0023] Figure 4 and Figure 5 for Figure 3 Enlarged view of a specific area;

[0024] Figure 6 This is a schematic diagram illustrating the effect of the behavior understanding of the present invention. Detailed Implementation

[0025] like Figure 1 As shown, this embodiment illustrates a behavior recognition and detection method based on structured behavior semantic space, comprising:

[0026] Step 1: Establish a structured behavioral semantic space. Based on the hierarchical verb tree in VerbNet, each node in the verb tree is treated as a category in the structured behavioral semantic space. Each node contains multiple behavioral verbs with shared semantics. Rich textual descriptions are provided to explain the meaning of each node. Semantic connections between nodes are established through hierarchical organization, forming structured knowledge. This structured behavioral semantic space contains approximately 5800 behavior-related verbs, broadly covering various behavioral categories.

[0027] Step 2: Construct a unified database with a unified tagging system, specifically including:

[0028] Step 2.1, Unified Labeling System: After behavior category mapping, the behavior label definitions of each existing behavior dataset can be defined using the structured behavior semantic space, thus achieving a unified behavior definition based on the behavior label set. and the set of nodes in the structured behavioral semantic space Among them: the j-th node has its verb members. When behavior label a i With any If there is a synonym relationship, then mark the relationship as follows: otherwise This process is done manually: each annotator reads the meaning explanation of a node and all its verb members, and then assigns a label to each behavior. value.

[0029] Step 2.2, Unifying Data Modal: Perform temporal, spatial, and 2D-3D consistency processing on data of different modalities. When a sample is a time series, it is sampled at 3fps, and the label corresponding to the sampled frame is used as its behavior label 'a'. When a sample is a single-frame image / human skeleton, its original behavior label 'a' is retained. For a single human body (human-object pair) in an image, if a label for that person / person-object pair is provided, that label is used; otherwise, the behavior label of the entire image is used. For 3D skeleton / motion capture datasets, they are uniformly converted to SMPL format. For image and video data lacking 3D human body annotations, 3D reconstruction methods are used to generate 3D human body data.

[0030] Step 3: Extract semantic and geometric information from the structured behavioral semantic space, specifically including:

[0031] Step 3.1: Extract the semantic information of the nodes, encoding the multiple action verbs and rich textual descriptions contained in the nodes. (This involves assigning node m...) j After the included verbs and example sentences are integrated into a complete natural language sentence, the CLIP model's text encoder encodes it into a semantic information vector t.j .

[0032] Step 3.2: Extract the geometric information of the nodes and model the hierarchical relationships between nodes using hyperbolic space. After randomly initializing the geometric information vectors of all nodes in the structured behavioral semantic space, optimize the target space using Riemann stochastic gradient descent for 600 iterations to obtain the geometric information vector g of each node. j Nodes with the same parent node should be close to each other; nodes with different parent nodes should be far apart; the shortest path between sibling nodes should be close to their parent node.

[0033] Step 4: Establish a mapping model between the physical space and the structured behavioral semantic space, specifically including:

[0034] Step 4.1: Extract the physical representation of the sample: For a given sample p in physical space, obtain its physical representation v using a multimodal encoder. raw For images, an image encoder based on a convolutional neural network or a converter is used; for videos, each frame is first encoded, and then temporal encoding is performed using a temporal layer; for 3D human body data, it is first converted into a point cloud, and then PointNet++ is used as an encoder for feature extraction.

[0035] Step 4.2, Node Semantic Decoupling: The verb node labels of the sample are However, due to some labeling issues in practice, the sample labels are... By decoupling the network, the original physical representation v raw Transformation into a new representation That is, each node has a customized representation as a condition. During training, the node label (y) is determined. i (∈{0,1}) can be associated with indeterminate node labels. Decoupling specifically involves designing a decoupling function f for the i-th node. i Thus, the decoupled characterization v is obtained. i =f i (v raw ), f i It is implemented as a learnable multilayer perceptron network.

[0036] Step 4.3, Pseudo-label enhancement: Based on the structure of the structured behavioral semantic space and language priors, calculate the correlation matrix C between nodes. Calculate the labels of verb nodes with low certainty based on the labels of verb nodes with high certainty, thereby enhancing the pseudo-labeling capabilities. The label is determined to be y. i ∈[0,1], specifically: First, obtain the relevance matrix C through the co-occurrence relationships between N nodes. The relevance matrix C between nodes consists of two components: C obtained based on language priors. langAnd C obtained based on verb tree structure stru To obtain C lang The information of each node i is encoded into t by a text encoder. i Then C lang This is the cosine distance matrix of the node features. To obtain C... stru The information of each node i is represented by its features in hyperbolic geometric space, C stru This is the Poincaré distance matrix representing the node features. Subsequently, C... lang and C stru If all values ​​are normalized to the interval [0,1], then C = (C lang +C stru ) / 2. Then, for the node i that was missed being labeled, assign its pseudo-label as

[0037] Step 4.4, Semantic Alignment: By minimizing the loss function L cls (D T (v i ,t i ),y i Semantic alignment is performed. For an i-th node with a defined label, semantic alignment is performed based on the extracted node semantic information vector t. j Calculate D T (v i ,t i )=Sigmoid(γ·cos(v i ,t i )), where: cos(·,·) calculates the cosine similarity between two vectors, and γ is the scaling factor.

[0038] Step 4.5, Geometric Alignment: By minimizing the loss function L cls (D G (v i ,g i ),y i To perform geometric alignment, comparison and Each component's internal distance matrix is ​​aligned, specifically based on the distance indicator matrix. L cls (D G (v i ,g i ),y i Instantiated as This allows for geometric alignment.

[0039] Step 4.6, Mapping Model Training: Based on the loss function L = L cls (D T (vi ,t i ),y i )+L cls (D G (v i ,g i ),y i The two losses correspond to semantic alignment and geometric alignment, respectively, where: object representation Verb node tags are obtained through semantic decoupling calculation. Obtained through pseudo-label enhancement.

[0040] Step 5: Transfer learning for specific behavior understanding scenarios: Based on the mapping model in the structured behavior semantic space, this model is used for testing on existing behavior understanding dataset benchmarks. Since the behavior category definitions and training data differ, transfer learning is required. When inferring behavior, the mapping model outputs a prediction S for the behavior category in the transfer dataset. ori And the prediction of node category S node S node Transformed into S′ through a learnable network ori Ultimately, by combining S ori and S′ ori Output behavior category predictions to complete the transfer of specific behavior understanding scenarios.

[0041] Experiments were conducted using four Nvidia RTX 3090 GPUs. For image data, pre-trained CLIPViT-B / 32 model weights were imported to extract physical features, with the image resolution set to 224. For video data, the processing method for each frame was the same as for image data, while additionally using existing temporal coding layer methods. For 3D human body data, PointNet++ was used as the encoder to extract physical features. The batch size was set to 64, and the initial learning rate was linearly warmed up from 5e-7 to 1e-5. Then, a cosine learning rate was decreased, and the model was trained using the AdamW (weight decaying Adam) stochastic gradient descent optimizer. After 30 training epochs, an experimental data point of 34.46 mAP was obtained on the node recognition test set, reaching the current state-of-the-art level in behavior understanding.

[0042]

[0043] In summary, compared with existing technologies, this invention significantly improves the unified structured behavioral semantic space, supports the training and learning of a unified large model, and makes full use of existing data; at the same time, it can achieve efficient transfer and generalization of specific behavioral understanding scenarios.

[0044] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A method for behavior recognition detection based on structured behavior semantic space, characterized in that, The structured behavior semantic space is established according to a hierarchical verb tree in VerbNet in an offline stage, semantic representations and geometric representations of nodes in the semantic space are extracted from a general behavior data set and a physical-semantic mapping model is established after a unified database with a unified label system is constructed through consistency processing, semantic behavior categories of a to-be-tested sample are recognized according to the trained physical-semantic mapping model in an online stage, and a specific behavior understanding scene is specified through migration recognition. The physical-semantic mapping model comprises a node decoupling unit, a pseudo-label enhancement unit, a semantic alignment unit and a geometric alignment unit, wherein: the node decoupling unit decouples the physical representation according to the semantic information of the node to obtain a visual representation conditioned on the node; the pseudo-label enhancement unit calculates a correlation matrix between nodes by using structural and language prior information of the semantic space , and according to the verb node label and the correlation matrix with high certainty, i.e., a large number of training samples , to calculate the verb node label with low certainty, i.e., a small number of training samples, i.e., a pseudo-label; the semantic alignment unit aligns the semantic representation and the physical representation of the node in the semantic space, and outputs the semantic behavior category of the to-be-tested sample, i.e., the node of the structured behavior semantic space corresponding to the to-be-tested sample; and the geometric alignment unit aligns the geometric representation and the physical representation of the node in the semantic space, and outputs the semantic behavior category of the to-be-tested sample.

2. The method of claim 1, wherein the method further comprises: The structured behavior semantic space takes each node in the hierarchical verb tree in VerbNet as a category in the semantic space; each node contains a plurality of verbs with common semantics and provides rich textual descriptions for explaining the meaning of the node. The hierarchical organization of nodes establishes semantic connections between the nodes and forms structured knowledge.

3. The method of claim 1, wherein the method further comprises: The unified database with the unified label system refers to that, after behavior category mapping, the behavior label definitions of each existing behavior data set are represented through the structured behavior semantic space, so that the database has unified behavior definitions and a unified format. ​ The behavior category mapping refers to that the behavior categories defined in any behavior data set are mapped to corresponding verbs under any node of the structured behavior semantic space.

4. The method of claim 3, wherein the method further comprises: The behavior category mapping is realized through semi-automatic crowdsourcing annotation, and the behavior categories of each data set only need to be aligned once.

5. The behavior recognition and detection method based on structured behavioral semantic space according to claim 1, characterized in that, The semantic representation of the node in the semantic space refers to encoding a plurality of action verbs and rich text descriptions contained in the node, specifically After the verbs and example sentences contained in the node are integrated into a sentence, the text encoder of the CLIP model encodes the semantic information vector .

6. The method of claim 1, wherein the method further comprises: The geometric representation refers to modeling the hierarchical relationship between nodes in a hyperbolic space, specifically, after random initialization of the geometric information vectors of all nodes in the structured semantic space, 600 cycles of optimization of the target hyperbolic space are performed by using Riemannian stochastic gradient descent to obtain the geometric information vectors of each node .

7. The method of claim 1, wherein the method further comprises: determining a structured behavior semantic space based on the structured behavior semantic space; and determining the behavior recognition based on the structured behavior semantic space. The alignment refers to: according to the semantic representation p of the node in the semantic space, encoding into object representation v through a deep neural network, according to the verb node label of the semantic representation , a loss function is calculated , wherein N is the number of target nodes in the structured semantic space S, , respectively are the physical representation, semantic representation and geometric representation of the node, is the distance measurement function between the object representation and the semantic representation of the node, is the distance measurement function between the object representation and the geometric representation of the node, is a classification loss function.

8. The method of claim 1, wherein the method further comprises: The decoupling refers to: according to the semantic information of the node , the network is decoupled into a visual representation conditioned on the node , wherein: The node index is calculated as a condition.​ 9. The behavior recognition and detection method based on structured behavioral semantic space according to claim 1, characterized in that, The aforementioned identification of specified behavior categories through transfer learning refers to: predicting the behavior categories in the transfer dataset based on the output of the trained physical-semantic mapping model. Prediction of node categories The prediction of node categories can be further improved through a learnable network. Transform into node-based behavior category prediction Ultimately through combination and Output the predicted behavior category to complete the transfer of specific behavior understanding in the scenario.

10. A system for implementing the method of claim 1-9 for behavior recognition detection based on structured behavior semantic space, characterized in that, The method comprises the following steps: A structured behavior semantic space establishing unit, a unified database establishing unit, a node semantic information extracting unit, a node geometric information extracting unit, a physical-semantic mapping model unit and a specific behavior understanding scene migration unit, wherein: the structured behavior semantic space establishing unit establishes a structured behavior semantic space according to a hierarchical verb tree in VerbNet through corpus organization and geometric information processing; the unified database establishing unit obtains a unified database with a unified label system through consistency processing according to existing behavior understanding data sets; the semantic information extracting unit obtains a semantic information vector for each node through natural language coding according to a plurality of behavior verbs and rich textual descriptions contained in the node; the geometric information extracting unit obtains a geometric information vector for each node through hyperbolic space modeling and optimization according to the hierarchical connections between the nodes defined in VerbNet; the physical-semantic mapping model unit establishes a physical-semantic mapping model according to the unified database, the node semantic information and the node geometric information, and identifies the semantic behavior category of a given sample; and the specific behavior understanding scene migration unit outputs a recognition result of a new specified behavior category through label migration mapping according to the physical-semantic mapping model.

Images