Household behavior knowledge graph construction method and device, detection system and method

Abstract

The knowledge graph construction method, model, detection device and method of home behavior can solve the problem of complex human behavior recognition in indoor scenes, more deeply understand the potential relationship between behaviors, more accurately model the context relationship between behaviors, and improve the accuracy of behavior detection. The method comprises: (1) describing behaviors from multiple semantic levels to construct HAhb-KG; (2) enhancing the expression of node information in the graph and modeling the relationship between nodes, enhancing the expression of node information using different modal knowledge, and modeling the relationship between behaviors using co-occurrence probability; (3) using the human daily activity dataset Charades to design the prototype of HAhb-KG, and according to steps (1)-(2), obtaining a HAhb-KG with 244 nodes and 15651 edges.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and behavior detection, and in particular to a method for constructing a knowledge graph of home behavior, a behavior detection device for home behavior, a behavior detection system for human home behavior, and a behavior detection method for human home behavior. Background Technology

[0002] Detecting behavior across multiple timescales in uncropped, densely annotated video datasets is one of the most challenging tasks in human behavior analysis. This task involves segmenting and identifying behaviors within long sequences of actions and requires a higher level of understanding of human activity. Recent studies have explored behavioral details by leveraging visually relevant knowledge such as motion trajectories, optical flow, and human posture. However, human behavior is complex and diverse; therefore, relying solely on visual information neglects contextual information about the behavior, such as the location and time of the action and the environment in which it occurred.

[0003] Knowledge graphs, as a data structure composed of entities, relationships, and attributes, can express rich information in a semantically related manner. In the field of behavior detection, knowledge graphs can capture background information about people, objects, and the behaviors involved, providing semantic representations of behaviors and revealing rich contextual information through the relationships between nodes. Yuan and Gupt constructed a reasoning network based on an interaction knowledge graph using spatiotemporal information to model the relationships between people and objects, as well as between objects themselves. Ma constructed a knowledge graph based on body parts and used vision-driven semantic knowledge mining to progressively infer human behavior from body movements in videos. These methods construct prior knowledge graphs of behavior using representations such as <object, person (or body part), verb>, effectively utilizing this prior information to delve into the details of behavior and enhance visual information.

[0004] However, in real-world scenarios, the execution of behavior is influenced not only by the performer's physical movements and interactions with objects, but also by the surrounding environment. Furthermore, the occurrence of behavior may be influenced by co-occurring or prior behaviors, creating a form of context dependency. Previous graphs ignored this information, leading to insufficient contextual understanding of behavior by detection devices and limiting their performance. Moreover, previous graphs described the relationships between nodes using text, lacking a rich data perspective; therefore, behavior detection requires a more robust knowledge graph.

[0005] Human behavior is often intricately linked to specific scenarios, and the inability to distinguish scenarios can lead to serious misclassification of behavior. For example, mistaking an indoor scenario of a "white plush carpet" for an outdoor scenario of "snow" might incorrectly categorize the behavior of "tidying up things on the floor" as "playing in the snow." Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, the technical problem to be solved by this invention is to provide a knowledge graph construction method for home behavior, which can solve the problem of recognizing complex human behaviors in indoor scenes, gain a deeper understanding of the potential relationships between behaviors, more accurately model the contextual relationships between behaviors, and improve the accuracy of behavior detection.

[0007] The technical solution of this invention is: a knowledge graph construction method for home behavior, comprising the following steps: (1) The structure design of the hierarchical enhanced human behavior knowledge graph HAhb-KG is to describe the behavior from multiple semantic levels and construct HAhb-KG; (2) Enhance the expression of node information in the graph and model the relationship between nodes. Use different modal knowledge to enhance the expression of node information and use co-occurrence probability to model the relationship between behaviors; (3) Use the Charades dataset of human daily activities to design the prototype of HAhb-KG. Charades contains 9848 videos of indoor daily actions, 157 action class labels, 38 object class labels, 33 verb labels, 16 scene labels, and time annotations corresponding to the action classes. Following steps (1)-(2), HAhb-KG with 244 nodes and 15651 edges is obtained.

[0008] It also provides a constructed behavior detection apparatus, which includes a knowledge mining block and a cross-modal learning block. The knowledge mining block is used for node matching and knowledge reasoning; node matching involves detecting scene and object nodes from the input video and then matching them with nodes in HAhb-KG to construct a subgraph called possible behaviors, denoted as... Knowledge reasoning is an action importance vector used to infer the importance weight of each action category in the input video.

[0009] It also provides a behavior detection system using home behavior detection devices, which includes: The extraction module is configured to extract visual information from the input video, following the action detection network MS-TCT, which includes a visual encoder, a temporal encoder, and a temporal scale mixer. The extracted visual features are fused with knowledge embeddings from the mining module for action detection. The knowledge mining module, configured to mine knowledge related to the input video from HAhb-KG, is a multimodal knowledge understanding framework (MMKU). It extracts information from the input video from HAhb-KG and integrates the output of the knowledge mining module into a multimodal knowledge embedding through a cross-modal attention mechanism. This knowledge embedding is fused with visual features from the extraction module as knowledge enhancement of visual features.

[0010] It also provides a behavior detection method using a home behavior behavior detection system, which includes the following steps: (a) MS-TCT is used as the baseline action detection network. This network uses a visual encoder to extract features from the input video and utilizes I3D for video encoding. Each video is divided into T non-overlapping segments, each segment consisting of 8 frames. These RGB frames are fed into the I3D network as input segments, and the output produces basic visual features. , in, Indicates basic visual features. Represents an RGB segment; (b) Basic visual features With knowledge embedding To achieve fusion characteristics , Used as input to the subsequent temporal encoder module, which aims to explore global and local temporal relationships across multiple temporal resolutions, it effectively models intra-video temporal dependencies by alternately studying local and global information across N stages. It utilizes 1D temporal convolutional layers and Transformer layers to learn behavioral representations at different temporal scales. The temporal scale mixer module is responsible for creating a unified feature representation. This module aggregates tokens from different temporal scales into a representation with a consistent time length. The aggregation process involves multiple stages, including upsampling and linear projection operations, ultimately producing the representation as... The multi-scale video representation is input into the classification module for prediction; (c) The detection module uses action relevance weights to predict the temporal location and classification score of each action instance. , This represents the action relevance weight, which is the co-occurrence probability between different behavior categories. It is a multi-scale video representation; it is the output of the temporal-scale mixer module. The loss function of the device, which represents the probability of the behavior category, is designed according to MS-TCT and includes heatmap loss and cross-entropy loss.

[0011] This invention primarily addresses the problem of recognizing complex human behaviors in indoor scenes. To solve this problem, it proposes a knowledge graph for describing indoor behaviors and an action detection device based on multimodal knowledge. Specifically, the solution employs an action detection method that enhances visual features using multimodal knowledge from a hierarchical knowledge graph. To support this solution, this invention constructs a hierarchically enhanced human behavior knowledge graph (HAhb-KG), describing behaviors from different semantic levels and data perspectives, and strengthening the relationships between behavior nodes through co-occurrence probabilities. Based on this, an action detection system is proposed that utilizes knowledge related to the input video from the knowledge graph as visual feature enhancement. This system includes a baseline action detection network and a multimodal knowledge understanding framework (MMKU), which extracts multimodal knowledge from the knowledge graph as a supplement to the visual information of the baseline action detection network. Therefore, this invention can solve the problem of recognizing complex human behaviors in indoor scenes, gain a deeper understanding of the potential relationships between behaviors, more accurately model the contextual relationships between behaviors, and improve the accuracy of behavior detection. Attached Figure Description

[0012] Figure 1 A schematic diagram of a knowledge graph construction method for home behavior according to the present invention is shown.

[0013] Figure 2 A behavior detection device for home behavior according to the present invention is shown.

[0014] Figure 3 A schematic diagram of the knowledge reasoning process according to the present invention is shown.

[0015] Figure 4 A behavior detection method using a home behavior behavior detection system according to the present invention is shown.

[0016] Figure 5 A visualization of the detection results is shown on an example video along the timeline. Detailed Implementation

[0017] like Figure 1 As shown, the knowledge graph construction method for this type of home behavior includes the following steps: (1) The structure design of the hierarchical enhanced human behavior knowledge graph HAhb-KG is to describe the behavior from multiple semantic levels and construct HAhb-KG; (2) Enhance the expression of node information in the graph and model the relationship between nodes. Use different modal knowledge to enhance the expression of node information and use co-occurrence probability to model the relationship between behaviors; (3) Use the Charades dataset of human daily activities to design the prototype of HAhb-KG. Charades contains 9848 videos of indoor daily actions, 157 action class labels, 38 object class labels, 33 verb labels, 16 scene labels, and time annotations corresponding to the action classes. Following steps (1)-(2), HAhb-KG with 244 nodes and 15651 edges is obtained.

[0018] Preferably, step (1) includes the following sub-steps: (1.1) Start building a hierarchical knowledge graph from scene nodes to form a partial graph of scene-object-action. The scene provides background information, representing the actions that occur in a specific scene and the interaction with objects in that scene. By adding connections between scene and action nodes, actions related to the current scene are filtered. (1.2) Construct a partial graph consisting of object-verb-action, where the object represents the object involved in the behavioral interaction, the verb represents the action related to the behavior, and the partial graph consisting of object-verb-action depicts the subject and object of the behavior, has clear semantic information about the action, and provides a clear representation of the subject, behavior and recipient; (1.3) Combine the partial graph of scene-object-action and the partial graph of object-verb-action into a unified knowledge graph about human behavior. The nodes in the graph are divided into four categories: scene, object, verb and behavior. The relationship definition in HAhb-KG is as follows: an object and a verb are linked if and only if object-person-verb is true; when a verb and an object constitute an action, they are linked with the behavior; when an object appears in a scene, it is linked with the scene; when a behavior appears in a scene, it is linked with the scene; if there is a co-occurrence relationship between behaviors, a connection is established.

[0019] Preferably, step (2) includes the following sub-steps: (2.1) Introduce semantic descriptions to enrich the various information in the knowledge graph; for object nodes in the graph, enrich the information expression of the nodes by matching each object category with images in various encyclopedias; (2.2) Treat the co-occurrence probability of actions as a relationship between action nodes, and use conditional probability to calculate the co-occurrence relationship: , in Indicating in behavior Simultaneous behavior under the condition of occurrence The probability, Indicates behavior category and The number of times they appear at the same time Indicates behavior category The number of times it appears in the dataset Used to represent C The co-occurrence probability among different behavior categories is calculated to obtain the probability of occurrence when the behavior... The probability of co-occurrence occurring in relation to other behaviors is called the behavior relevance weight, and this co-occurrence probability is used as the edge between behavior nodes.

[0020] It also provides a constructed knowledge graph apparatus, which includes knowledge mining blocks and cross-modal learning blocks. The knowledge mining blocks are used for node matching and knowledge reasoning; node matching involves detecting scene and object nodes from the input video and then matching them with nodes in the HAhb-KG to construct a subgraph called Possible Behaviors, denoted as... Knowledge reasoning is an action importance vector used to infer the importance weight of each action category in the input video.

[0021] Preferably, the node matching includes: First, using a mature model based on the DETR architecture, information related to scene and object nodes is detected from the input video. A keyframe is extracted from the input video as input to the scene detector, and then scene labels are obtained. For object detection, all RGB frames of the video are used as input to DETR to obtain a list of objects in the entire video, represented as... , It is the index of each object. =1,2… M , M Indicates the number of objects detected by DETR; The detected results are then matched with nodes in HAhb-KG to obtain possible behavioral subgraphs. : Locating the scene detected in HAhb-KG and object For nodes with the same label, based on the direct connections in the graph, find the nodes associated with the aforementioned nodes; these two groups of nodes are then collected. The set of nodes, represented as According to the definition of derived subgraph: given a graph ,in This represents all nodes in the graph. Let represent all the edges in the graph. yes G Let G be any subset of the vertices in G, and define an induced subgraph of G such that its vertex set is... And its edge set is composed of The edges in the array are composed of two vertices that belong to the same region. Using the introduced information about Subgraphs are used to represent possible behavior subgraphs. : , in express All nodes in express All edges in the set, based on which, represent a set of behaviors associated with an object in a certain scene, are represented by the following: , in, Indicates the category of behavior. , Subgraph The number of behavior categories.

[0022] Preferably, the knowledge reasoning includes: First, define a binary vector. It means in the set From all possibilities with Behavior categories found in related behavior categories , , in Indicates possible with All related behavioral categories, of Norm representation subgraph The number of behaviors matched by scene and object in the middle. c =1, 2… C, C It represents the total number of behavior categories, and the weighted mechanism is expressed as: , , in This represents the weight of the behavior category corresponding to each scene and object. This represents the total weight of each behavior category across all combinations, forming the behavior importance vector.

[0023] Preferably, the cross-modal learning block includes: First, BERT is used to extract the semantic vector of each behavior label and its text description. , in, Represents a semantic vector. This refers to the text feature extractor BERT. Subgraph Description of behavioral nodes in the code; Subsequently, using importance vectors as guidance, the semantic representations of all behavior categories are aggregated into the semantic knowledge vector of the video, as follows: , in Represents semantic knowledge of behavior categories; To further learn semantic knowledge about instances of human behavior and provide more comprehensive semantic information, BERT is used to learn the semantic knowledge of each triple, and semantic vectors are constructed by performing average pooling on all triple-level vectors, represented as: , in Represents the formation of subgraphs All triples, by splicing and Subgraph The semantic features obtained are represented as follows: , in Indicates originating from a subgraph semantic features, It is a fully connected layer used for feature transformation. For from subgraph Image knowledge of object nodes is used to extract features from the corresponding class images using VGG: , in Representing image features, Represents the image corresponding to the class of the object node. For subgraph The adjacency relationships between nodes, as a form of knowledge, capture the semantic associations between nodes to better understand the context of actions. Graph convolution is used to extract adjacency features. , After obtaining the multimodal features, these features are merged to generate a comprehensive multimodal feature representation, thereby enhancing visual features with knowledge.

[0024] Preferably, in the cross-modal learning block, feature merging is achieved through cross-modal attention: First, calculate the semantic features. and image features Attention score between: , in , , It is a learnable weight matrix used to calculate semantic features. and adjacency features Attention score between , , Calculate semantic features and image features Attention weight distribution , in Representing the dimensions of features to obtain semantic features and adjacency features Attention weight distribution between ; Finally, attention weight distribution is used to linearly combine features from each semantic modality with features from the other two modalities to obtain cross-modal feature representations of the text modality: , Calculate cross-modal feature representation of image modalities and cross-modal feature representation of graph modes This connects the fused cross-modal feature representations together to form fused features. As a multimodal knowledge embedding, it is used to enhance visual features with knowledge: , In acquiring multimodal knowledge embedding Then, it is compared with basic visual features. Fusion to create fusion features .

[0025] It also provides a detection system for knowledge graph devices used in home life, which includes: The extraction module is configured to extract visual information from the input video, following the action detection network MS-TCT, which includes a visual encoder, a temporal encoder, and a temporal scale mixer. The extracted visual features are fused with knowledge embeddings from the mining module for action detection. The knowledge mining module, configured to mine knowledge related to the input video from HAhb-KG, is a multimodal knowledge understanding framework (MMKU). It extracts information from the input video from HAhb-KG and integrates the output of the knowledge mining module into a multimodal knowledge embedding through a cross-modal attention mechanism. This knowledge embedding is fused with visual features from the extraction module as knowledge enhancement of visual features.

[0026] It also provides a method for detecting home use behavior using a knowledge graph system, which includes the following steps: (a) MS-TCT is used as the baseline action detection network. This network uses a visual encoder to extract features from the input video and utilizes I3D for video encoding. Each video is divided into T non-overlapping segments, each segment consisting of 8 frames. These RGB frames are fed into the I3D network as input segments, and the output produces basic visual features. , in, Indicates basic visual features. Represents an RGB segment; (b) Basic visual features With knowledge embedding To achieve fusion characteristics , Used as input to the subsequent temporal encoder module, which aims to explore global and local temporal relationships across multiple temporal resolutions, it effectively models intra-video temporal dependencies by alternately studying local and global information across N stages. It utilizes 1D temporal convolutional layers and Transformer layers to learn behavioral representations at different temporal scales. The temporal scale mixer module is responsible for creating a unified feature representation. This module aggregates tokens from different temporal scales into a representation with a consistent time length. The aggregation process involves multiple stages, including upsampling and linear projection operations, ultimately producing the representation as... The multi-scale video representation is input into the classification module for prediction; (c) The detection module uses action relevance weights to predict the temporal location and classification score of each action instance. , This represents the action relevance weight, which is the co-occurrence probability between different behavior categories. It is a multi-scale video representation; it is the output of the temporal-scale mixer module. The loss function of the device, which represents the probability of the behavior category, is designed according to MS-TCT and includes heatmap loss and cross-entropy loss.

[0027] This invention primarily addresses the problem of recognizing complex human behaviors in indoor scenes. To solve this problem, it proposes a knowledge graph for describing indoor behaviors and an action detection device based on multimodal knowledge. Specifically, the solution employs an action detection method that enhances visual features using multimodal knowledge from a hierarchical knowledge graph. To support this solution, this invention constructs a hierarchically enhanced human behavior knowledge graph (HAhb-KG), describing behaviors from different semantic levels and data perspectives, and strengthening the relationships between behavior nodes through co-occurrence probabilities. Based on this, an action detection system is proposed that utilizes knowledge related to the input video from the knowledge graph as visual feature enhancement. This system includes a baseline action detection network and a multimodal knowledge understanding framework (MMKU), which extracts multimodal knowledge from the knowledge graph as a supplement to the visual information of the baseline action detection network. Therefore, this invention can solve the problem of recognizing complex human behaviors in indoor scenes, gain a deeper understanding of the potential relationships between behaviors, more accurately model the contextual relationships between behaviors, and improve the accuracy of behavior detection.

[0028] This invention compares the above method with the latest related methods and achieves significant results. In addition, ablation experiments were conducted on the proposed knowledge mining block, action relevance weight, cross-modal learning block and knowledge embedding dimension, and the selection of visual encoder.

[0029] 1. Experimental data: This invention was tested on the Charades and TSU datasets.

[0030] 2. Measurement metrics: The mean accuracy (mAP) of the overall category is used for evaluation.

[0031] 3. Experimental Setup: I3D was used to extract basic visual features, resulting in a 1024-dimensional representation, with the label count fixed at T=256. The spatial resolution of the input video fed into I3D was set to 224x224. In this study, BERT was used for text encoding to generate 768-dimensional text features, and a VGG network was used to extract 1024-dimensional image features. Training was performed using RTX 2060s with a batch size of 16 and 50 epochs.

[0032] 4. Ablation experiment:

[0033] 4.1 Knowledge Mining Block: As shown in Table 1, the role of the knowledge mining block was verified. MS-TCT* indicates the result of replicating MS-TCT; the slightly lower performance is mainly attributed to the difference in parameters used for visual feature extraction when using the backbone. First, the entire knowledge mining block was disabled, and cross-modal learning block (CMLB) was performed on the entire knowledge graph (Table 1, row 2). The results show that only a 0.22 improvement was achieved compared to the baseline, while the time cost increased by more than three times. Analysis indicates that the limited improvement is mainly due to a large amount of redundant information during training, which not only limits the improvement in performance but also leads to a considerable waste of time and resources.

[0034] Secondly, the knowledge mining block was extended to include scene and object recognition (CMLB+DETR), constructing a "possible behavior" subgraph for cross-modal learning (Table 1, row 3). Results show that compared to the baseline, this method achieves a performance improvement of 0.29 and significantly reduces time consumption. This indicates that constructing a "possible behavior" subgraph can improve device performance while reducing time complexity. Furthermore, it demonstrates that by hierarchically layering behavioral knowledge and constructing a knowledge graph, the knowledge in the graph can be better utilized, reducing a large amount of redundant information and improving the device's accuracy and time efficiency. However, some redundant knowledge may still exist, which could adversely affect the device's performance.

[0035] Finally, the knowledge reasoning component (CMLB+KMB) of the knowledge mining module was expanded (Table 1, row 4). This significantly improved the device's performance by 0.62 times compared to the baseline, with almost no increase in time cost. Analysis shows that this is mainly because the knowledge mining module adds a knowledge reasoning process, which can more effectively integrate rich common-sense knowledge from visual and textual data, thus significantly improving the device's performance. Since knowledge is reasoned from a hierarchical knowledge graph, it is clear that this graph construction method not only incorporates rich prior knowledge but also has a hierarchical structure conducive to reasoning.

[0036] 4.2 Behavioral Relevance Weights: In Table 1, in addition to the ablation analysis of the knowledge mining blocks, experiments were conducted to analyze the ablation of behavioral relationship modeling (CMLB+KMB+W). Notably, as shown in the last row of Table 1, the accuracy of the device was further improved by introducing behavioral relevance weights (an increase of 0.71 compared to the baseline device). This indicates that introducing the probability of behavioral co-occurrence effectively enhances the understanding of behavioral contextual knowledge, enabling the device to fully learn the relationships between behaviors. Therefore, this greatly enhances the device's ability to understand human behavior. This also shows that enhancing the relationships between behavioral nodes in HAhb-KG is beneficial for behavior detection tasks.

[0037] Table 1 Ablation of Knowledge Mining Block Components (Bold text indicates best results)

[0038] 4.3 Cross-modal learning blocks: The impact of different modalities on device performance was verified, and the specific results are shown in Table 2. Table 2 clearly shows that each modality has a positive impact on the performance of our device, especially when the text modality is combined with other modalities, the device's performance shows a significant improvement. This is due to the semantic context provided by the text modality, which helps to eliminate visual ambiguity and accurately identify behaviors. The introduction of textual information helps to overcome the limitations of visual information, providing a more comprehensive context for behavior detection, thereby improving the device's performance. Through ablation analysis of the impact of different modalities on device accuracy, it was verified that multimodal information can effectively complement each other, enabling the device to infer and verify behaviors from multiple perspectives, ultimately improving performance. This also precisely illustrates the necessity of enhancing node representations in the knowledge graph.

[0039] Table 2. The impact of different modal knowledge on the device

[0040] 4.4 Knowledge Embedding Dimension: The impact of the multimodal knowledge embedding dimension on device performance was comprehensively explored, and the results are shown in Table 3. Based on this, the following conclusion can be drawn: the device achieves optimal performance when the multimodal knowledge embedding dimension is set to 128. This finding indicates that a larger embedding size is not necessarily better. When the dimension is too large, the device is susceptible to noise, which may lead to overfitting. Conversely, if the dimension is too small, more information may be lost. Furthermore, the impact of different token sizes (i.e., the number of video segments) on device performance was investigated. The results show that the device generally performs better when the token count is 256 than when it is 128. This is mainly due to the more refined segmentation assistance in extracting complete visual features. These findings provide valuable guidance for optimizing device performance and selecting parameters.

[0041] Table 3. Impact of the dimensionality of multimodal knowledge embedding and the number of video segments on device performance. KE Representing the knowledge embedding dimension

[0042] 4.5 Visual Encoder: An alternative visual encoder was selected to evaluate the generalization ability of our device. In recent years, OpenAI's CLIP has demonstrated significant effectiveness in pre-training image and text encoders for various downstream tasks. Therefore, the original backbone network I3D was replaced with a ViT network pre-trained based on the CLIP architecture. As shown in Table 4, the baseline visual encoder was replaced with ViT, and the same operations were applied to the visual encoder of our device. The results show that the device can still achieve performance improvements under these conditions. This experiment not only strongly validates the effectiveness of our device but also demonstrates its remarkable generalization ability when dealing with different visual encoders. This means that our device not only performs well under specific conditions but also adapts to a wider range of visual encoding environments, providing strong support for various downstream tasks in multiple domains. This result has important practical and theoretical significance for research.

[0043] Table 4. Impact of different visual encoders on the device

[0044] 5. Comparison with State-of-the-art Methods: Table 5 shows the results of comparing our method with state-of-the-art techniques on the Charades and TSU large indoor datasets. The comparisons with leading methods on these datasets include those using TCN, Transformer, GCN, and ConvTransformer. Our method outperforms these existing methods by +0.72 (using the I3D visual encoder) and +1.45 (using the ViT visual encoder) on Charades, and by +1.32 (using the I3D visual encoder) and +2.28 (using the ViT visual encoder) on TSU. These comparative studies demonstrate the superior performance of our method in behavior detection tasks, reaffirming that HAhb-KG provides rich semantic knowledge of behavior, thereby enhancing the capabilities of human behavior detection devices. Notably, experiments were conducted on the datasets using different backbone devices, showing that the MMKU framework performs well in various scenarios. This highlights the effectiveness of knowledge-based human behavior detection methods.

[0045] Furthermore, to visually demonstrate the results of this behavior detection device, a qualitative evaluation was performed on the Charades dataset, comparing PDAN, MS-TCT, and our method, such as... Figure 5As shown, the black line represents the timeline, and different colored blocks indicate the temporal locations where action detection was performed using different methods. Here, it is observed that this method can predict action instances more accurately compared to PDAN and MS-TCT. This highlights that HAhb-KG provides rich contextual knowledge from the video, and the MMKU framework effectively utilizes this knowledge, thereby improving the performance of the action detection device.

[0046] Table 5 Comparison with state-of-the-art methods on densely labeled datasets

[0047] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for constructing a knowledge graph of home-based behaviors, characterized by: The method includes the following steps: (1) The structure design of the hierarchical enhanced human behavior knowledge graph HAhb-KG is to describe the behavior from multiple semantic levels and construct HAhb-KG; (2) Enhance the expression of node information in the graph and model the relationship between nodes. Use different modal knowledge to enhance the expression of node information and use co-occurrence probability to model the relationship between behaviors; (3) Use the Charades dataset of human daily activities to design the prototype of HAhb-KG. Charades contains 9848 videos of indoor daily actions, 157 action class labels, 38 object class labels, 33 verb labels, 16 scene labels, and time annotations corresponding to the action classes. Following steps (1)-(2), we obtain HAhb-KG with 244 nodes and 15651 edges. Step (1) includes the following sub-steps: (1.1) Start building a hierarchical knowledge graph from scene nodes to form a partial graph of scene-object-action. The scene provides background information, representing the actions that occur in a specific scene and the interaction with objects in that scene. By adding connections between scene and action nodes, actions related to the current scene are filtered. (1.2) Construct a partial graph consisting of object-verb-action, where the object represents the object involved in the behavioral interaction, the verb represents the action related to the behavior, and the partial graph consisting of object-verb-action depicts the subject and object of the behavior, has clear semantic information about the action, and provides a clear representation of the subject, behavior and recipient; (1.3) Combine the partial graphs of scene-object-action and object-verb-action into a unified knowledge graph about human behavior. The nodes in the graph are divided into four categories: scene, object, verb and behavior. The relationship definition in HAhb-KG is as follows: an object and a verb are linked if and only if object-person-verb is true; when a verb and an object constitute an action, a link is established with the behavior; when an object appears in a scene, a link is established with the scene; when a behavior appears in a scene, a link is established with the scene; if there is a co-occurrence relationship between behaviors, a link is established. Step (2) includes the following sub-steps: (2.1) Introduce semantic descriptions to enrich the various information in the knowledge graph; for object nodes in the graph, enrich the information expression of the nodes by matching each object category with images in various encyclopedias; (2.2) Treat the co-occurrence probability of actions as a relationship between action nodes, and use conditional probability to calculate the co-occurrence relationship: ， in Indicating in behavior Simultaneous behavior under the conditions of occurrence The probability, Indicates behavior category and The number of times they appear at the same time Indicates behavior category The number of times it appears in the dataset Used to represent C The co-occurrence probability among different behavior categories is calculated to obtain the probability of occurrence when the behavior... The probability of co-occurrence occurring in relation to other behaviors is called the behavior relevance weight, and this co-occurrence probability is used as the edge between behavior nodes.

2. The behavior detection device of the knowledge graph construction method for home behavior according to claim 1, characterized in that: The device includes a knowledge mining block and a cross-modal learning block. The knowledge mining block is used for node matching and knowledge reasoning. Node matching involves detecting scene and object nodes from the input video and then matching them with nodes in the HAhb-KG to construct a subgraph called Possible Behaviors, denoted as... ; Knowledge reasoning is an action importance vector used to infer the importance weight of each action category in the input video.

3. The behavior detection device according to claim 2, characterized in that: The node matching includes: First, using a mature model based on the DETR architecture, information related to scene and object nodes is detected from the input video. A keyframe is extracted from the input video as input to the scene detector, and then scene labels are obtained. For object detection, all RGB frames of the video are used as input to DETR to obtain a list of objects in the entire video, represented as... , It is the index of each object. =1,2… M , M Indicates the number of objects detected by DETR; The detected results are then matched with nodes in HAhb-KG to obtain possible behavioral subgraphs. : Locating the scene detected in HAhb-KG and object For nodes with the same label, based on the direct connections in the graph, find the nodes associated with the aforementioned nodes; these two groups of nodes are then collected. The set of nodes, represented as According to the definition of derived subgraph: given a graph ,in This represents all nodes in the graph. Let represent all the edges in the graph. yes G Let G be any subset of the vertices in G, and define an induced subgraph of G such that its vertex set is... And its edge set is composed of E The edges in the array are composed of two vertices that belong to the same region. Using the introduced information about Subgraphs are used to represent possible behavior subgraphs. : ， in express All nodes in express All edges in the set, based on which, represent a set of behaviors associated with an object in a certain scene, are represented by the following: ， in, Indicates the category of behavior. , Subgraph The number of behavior categories.

4. The behavior detection device according to claim 3, characterized in that: The knowledge reasoning includes: First, define a binary vector. It means in the set From all possibilities with Behavior categories found in related behavior categories ， ， in Indicates possible with All related behavioral categories, of Norm representation subgraph The number of behaviors matched by scene and object in the middle. c =1, 2… C, C It represents the total number of behavior categories, and the weighted mechanism is expressed as: ， ， in This represents the weight of the behavior category corresponding to each scene and object. This represents the total weight of each behavior category across all combinations, forming the behavior importance vector.

5. The behavior detection device according to claim 4, characterized in that: The cross-modal learning block includes: First, BERT is used to extract the semantic vector of each behavior label and its text description. ， in, Represents a semantic vector. This refers to the text feature extractor BERT. Subgraph Description of behavioral nodes in the code; Subsequently, using importance vectors as guidance, the semantic representations of all behavior categories are aggregated into the semantic knowledge vector of the video, as follows: ， in Represents semantic knowledge of behavior categories; To further learn semantic knowledge about instances of human behavior and provide more comprehensive semantic information, BERT is used to learn the semantic knowledge of each triple, and semantic vectors are constructed by performing average pooling on all triple-level vectors, represented as: ， in Represents the formation of subgraphs All triples, by splicing and Subgraph The semantic features obtained are represented as follows: ， in Indicates originating from a subgraph semantic features, It is a fully connected layer used for feature transformation. For from subgraph Image knowledge of object nodes is used to extract features from the corresponding class images using VGG: ， in Representing image features, Represents the image corresponding to the class of the object node. For subgraph The adjacency relationships between nodes, as a form of knowledge, capture the semantic associations between nodes to better understand the context of actions. Graph convolution is used to extract adjacency features. ， After obtaining the multimodal features, these features are merged to generate a comprehensive multimodal feature representation, thereby enhancing visual features with knowledge.

6. The behavior detection device according to claim 5, characterized in that: In the cross-modal learning block, feature merging is achieved through cross-modal attention: first Calculate semantic features and image features Attention score between: ， in , , It is a learnable weight matrix used to calculate semantic features. and adjacency features Attention score between , , Calculate semantic features and image features Attention weight distribution ， in Representing the dimensions of features to obtain semantic features and adjacency features Attention weight distribution between ; Finally, attention weight distribution is used to linearly combine features from each semantic modality with features from the other two modalities to obtain cross-modal feature representations of the text modality: ， Calculate cross-modal feature representation of image modes and cross-modal feature representation of graph modes This connects the fused cross-modal feature representations together to form fused features. As a multimodal knowledge embedding, it is used to enhance visual features with knowledge: ， In acquiring multimodal knowledge embedding Then, it is compared with basic visual features. Fusion to create fusion features .

7. A behavior detection system using the behavior detection device according to claim 6, characterized in that: It includes: The visual feature extraction module is configured to extract visual information from the input video, following the action detection network MS-TCT, which includes a visual encoder, a temporal encoder, and a temporal scale mixer. The extracted visual features are fused with knowledge embeddings from the mining module for action detection. The knowledge understanding module, configured to mine knowledge related to the input video from HAhb-KG, is a multimodal knowledge understanding MMKU framework. It extracts information from the input video from HAhb-KG and integrates the output of the knowledge mining module into a multimodal knowledge embedding through a cross-modal attention mechanism. This knowledge embedding is fused with visual features from the extraction module as knowledge enhancement of visual features.

8. A behavior detection method using the behavior detection system according to claim 7, characterized in that: It includes the following steps: (a) MS-TCT is used as the baseline action detection network. This network extracts features from the input video using a visual encoder and performs video encoding using I3D. Each video is divided into T non-overlapping segments, each segment consisting of 8 frames. These RGB frames are fed into the I3D network as input segments, and the output produces basic visual features. ， in, Indicates basic visual features. Represents an RGB segment; (b) Basic visual features With knowledge embedding To achieve fusion characteristics , Used as input to the subsequent temporal encoder module, which aims to explore global and local temporal relationships across multiple temporal resolutions, it effectively models intra-video temporal dependencies by alternately studying local and global information across N stages. It utilizes 1D temporal convolutional layers and Transformer layers to learn behavioral representations at different temporal scales. The temporal scale mixer module is responsible for creating a unified feature representation. This module aggregates tokens from different temporal scales into a representation with a consistent time length. The aggregation process involves multiple stages, including upsampling and linear projection operations, ultimately producing the representation as... The multi-scale video representation is input into the classification module for prediction; (c) The detection module uses action relevance weights to predict the temporal location and classification score of each action instance. ， This represents the action relevance weight, which is the co-occurrence probability between different behavior categories. It is a multi-scale video representation; it is the output of the temporal-scale mixer module. The loss function of the model, which represents the probability of the behavior category, is designed according to MS-TCT and includes heatmap loss and cross-entropy loss.

Images