Knowledge-based action tool object intelligent interaction detection method
Through the knowledge-based intelligent interaction detection method of action tool object, the problem that binary detection algorithm cannot express the relationship between tools and object space is solved, tool functional evaluation and fine-grained interactive recognition are realized, suitable for a variety of application scenarios, and real-time Feedback and guidance.
Patent Information
- Application Number
- CN202411448817.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-17
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing binary detection algorithm cannot express the spatial relationship between the operation tool and the operation object in the scenario where human operation tool is executed, cannot describe the ambiguity of the tool, and cannot subdivide the execution actions of different tools acting on one operation.
Using the knowledge-based action tool object intelligent interaction detection method, the visual characteristics of the input image are obtained through the backbone network, the interaction detector is activated for interaction prediction, and the instance matcher is combined to analyze the predicted HTOL instance and the target HTOL instance to achieve the matching threshold. The interactive detection result is output when the value is.
It realizes an understanding of how and context of the tool is used, evaluates the functional use of the tool, provides fine-grained interactive recognition, and has customizable and real-time feedback capabilities, suitable for scenarios where precise operation and high security are required.
Smart Images

Figure CN118968260A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image recognition technology, and in particular to a knowledge-based intelligent interaction detection method for action tool objects. Background Art
[0002] In the field of human interaction detection, especially in scenarios involving tool operation, traditional methods are often limited to binary detection algorithms, that is, they mainly focus on the relationship between human actions and actions or objects, while ignoring the complex and delicate spatial relationship between operating tools and operating objects. This limitation makes the existing technology powerless when facing ambiguous tools and different operating actions, and it is impossible to accurately subdivide and describe the specific behaviors of different tools when performing the same operation. For example, existing binary detection algorithms can usually only output action categories, but cannot describe in detail the interaction details between tools and objects, such as how tools are used, the functional attributes of tools, and how these attributes affect the execution of operations. In addition, due to the lack of in-depth understanding of tool functionality, existing technologies are often unable to accurately evaluate the correctness of tool use, which is particularly important in scenarios that require precise operation and high safety.
[0003] In summary, the existing binary detection algorithms cannot express the spatial relationship between the operating tools and the operating objects in the scenarios where people operate tools. They only express the action categories, cannot describe the ambiguity of tools, and cannot subdivide the execution actions of different tools acting on one operation.
[0004] Therefore, how to extend the binary detection algorithm of people to the ternary detection of people's actions, tools, and objects, and consider the functional attributes of tools to make judgments on action categories and tool matching needs to be solved urgently. Summary of the invention
[0005] The purpose of this application is to provide a knowledge-based method for intelligent interaction detection of action tools and objects, so as to solve the problem that the existing detection method cannot express the spatial relationship between the operating tool and the operating object in the scenario where people operate the tool, only expresses the action category, cannot describe the ambiguity of the tool, and cannot subdivide the execution actions of different tools acting on one operation.
[0006] In view of the above problems, the present application provides a knowledge-based intelligent interaction detection method for action tool objects. The method comprises: obtaining visual features of an input image through a backbone network, wherein the backbone network refers to a convolutional neural network; activating an interaction detector, wherein an interaction prediction strategy is embedded in the interaction detector, and analyzing the visual features according to the interaction prediction strategy to obtain a predicted HTOL instance; performing matching analysis on the predicted HTOL instance and the target HTOL instance through an instance matcher to obtain a predicted matching degree; when the predicted matching degree reaches a matching threshold value, using the predicted HTOL instance as the interaction detection result of the input image.
[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages: The visual features of the input image are obtained through a backbone network, wherein the backbone network refers to a convolutional neural network; an interaction detector is activated, wherein the interaction detector has an interaction prediction strategy embedded therein, and the visual features are analyzed according to the interaction prediction strategy to obtain a predicted HTOL instance; the predicted HTOL instance is matched and analyzed with a target HTOL instance through an instance matcher to obtain a predicted matching degree; when the predicted matching degree reaches a matching threshold value, the predicted HTOL instance is used as an interaction detection result of the input image.
[0008] 1. Focus on the contextual understanding of tool use: Tool Function Matcher focuses on understanding how and in what context the tool is used, which is very important for certain application scenarios such as education, industrial training, safety supervision, etc. It not only identifies people and tools, but also evaluates whether the tool is used in the right way, which is usually not involved in human interaction detection algorithms.
[0009] 2. Functional matching and evaluation: This algorithm can evaluate the functional use of tools, such as determining whether a hammer is used to hammer a nail. This functional insight is critical to ensuring that tasks are completed correctly and improving safety.
[0010] 3. Fine-grained interaction recognition: By combining image features and hand tracking data, the tool function matcher is able to identify and understand the interaction between people and tools at a finer granularity, which is especially important for scenarios that require precise operational guidance (such as surgical simulation, precision manufacturing, etc.).
[0011] 4. Customizability and scalability: By defining functional embeddings for different tools and actions, this algorithm can be customized and extended according to specific application requirements. This flexibility can adapt to a variety of different application scenarios and requirements.
[0012] 5. Potential for real-time feedback and guidance: Since algorithms can evaluate the correctness of tool usage in real time, it is possible to provide real-time feedback and guidance, which is very helpful for learning and improving tool usage skills.
[0013] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented according to the contents of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are specifically cited below. It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0015] Figure 1 A flowchart of the knowledge-based action tool object intelligent interaction detection method of this application; Figure 2 This is a flow chart of obtaining predicted HTOL instances in the knowledge-based action-tool-object intelligent interaction detection method of this application. DETAILED DESCRIPTION
[0016] The present application solves the problem of existing binary detection algorithms by providing a knowledge-based method for intelligent interaction detection of action tools and objects. In scenarios where people operate tools, the algorithm cannot express the spatial relationship between the operating tool and the operating object, and can only express the action category, but cannot describe the ambiguity of the tool and cannot subdivide the execution actions of different tools acting on one operation.
[0017] Below, the technical solutions in the present application will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments of the present application. It should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application. It should also be noted that, for the convenience of description, only the parts related to the present application are shown in the accompanying drawings, rather than all of them.
[0018] For example, please refer to the attached Figure 1The present application provides a knowledge-based method for intelligent interaction detection of action tool objects, the method specifically comprising the following steps: The visual features of the input image are obtained through a backbone network, wherein the backbone network refers to a convolutional neural network.
[0019] Specifically, a convolutional neural network (CNN) is used as the backbone network to extract visual features from the input image. First, the color image is input into the backbone network to generate a feature map of shape (H, W, C) containing high-level semantic concepts. Then, a 1×1 convolution layer is used to reduce the channel dimension from C to d. The spatial dimension is folded into one dimension using the flatten operation. This results in a feature map of shape [H×W, d], which is the flatten feature in the figure. This spatial dimension conversion is important because the following Transformer encoder requires a sequence as input, so the feature map can be interpreted as a sequence of length H×W, and the value of each time step is a vector of size d. In this method, ResNet is used as the backbone network, and the dimension of the feature conv-5 is reduced from C=2048 to d=256.
[0020] An interaction detector is activated, wherein the interaction detector has an interaction prediction strategy embedded therein, and the visual features are analyzed according to the interaction prediction strategy to obtain a predicted HTOL instance.
[0021] Further, as attached Figure 2 As shown, including: The interaction detector is composed of an encoder, a decoder and a multi-layer perceptron; the encoder performs a global context analysis on the visual features to obtain global information; the decoder performs a predictive analysis on the global information in combination with the interaction prediction strategy to obtain a predicted HTOL relationship; the multi-layer perceptron performs an embedding decoding process on the predicted HTOL relationship to obtain the predicted HTOL instance.
[0022] Further, including: The multilayer perceptron includes M one-layer MLP branches and N three-layer MLP branches, where M=7 and N=3; the predicted HTOL relationship is analyzed by the M one-layer MLP branches to obtain a layer of analysis results, and the layer of analysis results includes the confidence of people, the confidence of relationships, the confidence of tools, the confidence of objects and the confidence of interactions; the predicted HTOL relationship is analyzed by the N three-layer MLP branches to obtain a three-layer analysis result, and the three-layer analysis result includes a predicted human action box, a predicted tool box and a predicted object box; a predetermined eight-tuple strategy is read, and the one-layer analysis result and the three-layer analysis result are analyzed based on the predetermined eight-tuple strategy to obtain a HTOL eight-tuple; the HTOL eight-tuple is used as the predicted HTOL instance.
[0023] Specifically, the encoder first performs a global context analysis of the visual features of the input image. This step is crucial, allowing the model to capture global information in the image rather than just local features. As a result, the model can better understand the entire human action, the interaction scene between tools and objects. The decoder then performs a predictive analysis of the global information in combination with the interaction prediction strategy. The decoder uses the information provided by the encoder and the prediction strategy to predict the relationship between people, tools and objects. This strategy may include functional analysis of tools and dynamic interactions between tools, people and objects. Next, the predicted HTOL relationship is embedded and decoded by a multi-layer perceptron to convert the information output by the decoder into a specific prediction instance, namely the HTOL instance. In addition, the role of the multi-layer perceptron here is to refine the relationship and ensure the accuracy and reliability of the prediction results. By introducing a tool function matcher, the model can evaluate whether the tool is used in the correct way, which is crucial to ensure the correct completion of the task and improve safety. Finally, through this comprehensive analysis, it is not only possible to detect the interaction between people, tools and objects, but also possible to combine the tool's function and the current action to determine whether the correct tool is used to perform the correct action. This provides a more precise method for understanding complex scenes and improving the decision-making ability of machines.
[0024] Specifically, the encoder layer is based on the standard Transformer architecture, including a multi-head self-attention module and a feed-forward neural network (FFN). In theory, the Transformer architecture is permutation invariant. To enable it to distinguish relative positions in the sequence, position encodings are added to the input of each attention layer. The sum of the flattened feature and the position encoding is input into the Transformer encoder to summarize the global information. The output of the encoder is represented as global memory in the figure. The decoder layer is also based on the Transformer architecture, with an additional multi-head cross-attention layer added on top of the encoder layer. The decoder transforms N learned position embeddings (represented as HTOI queries in the figure) into N output embeddings. Then, it is decoded into HTOI instances by the following MLP, and the details of the MLP will be described in detail in the next section.
[0025] In general, the decoder has three inputs: one is the global memory from the encoder, one is the HTOI queries, and one is the position encoding. For the multi-head cross attention layer, the Value comes directly from the global memory. The Key is the sum of the global memory and the input position encoding. The Query is the sum of the input position encoding and the input HTOI queries. For the self-attention layer, the Query, Key, and Value all come from the HTOI queries or the output of the previous decoder layer. The output of the decoder is represented as output embeddings in the figure.
[0026] Further, each HTOI instance is defined as an octet of (human category, relation category, tool category, interaction category, object category, human action box, tool box, object box), i.e., the predetermined octet strategy. The output embedding of each HTOI query is decoded into an HTOI instance through several MLP branches. Among them, there are seven one-layer MLP branches for predicting the confidence of people, the confidence of relations, the confidence of tools, the confidence of objects, and the confidence of interactions, respectively, and three three-layer MLP branches for predicting human action boxes, tool boxes, and object boxes. All one-layer MLP branches for predicting confidence use a softmax function. For the human confidence branch, the output size is 2, representing the confidence of the foreground and background. For the object and tool confidence branches, and the relationship and interaction confidence branches, the output size is C+1, representing the confidence of all C types of objects or verbs defined in the dataset, plus a background confidence. For the human action and object box branches, the output size is 4, representing the normalized center coordinates (xc, yc), height, and width of the box, thereby obtaining a HTOL octet, which is finally used as the predicted HTOL instance.
[0027] The predicted HTOL instance and the target HTOL instance are matched and analyzed by an instance matcher to obtain a predicted matching degree.
[0028] Further, it includes: the instance matcher stores a predetermined matching rule; the predicted HTOL instance and the target HTOL instance are matched and analyzed according to the predetermined matching rule to obtain the predicted matching degree; wherein the predetermined matching rule includes position matching, relationship category matching and one-to-one matching.
[0029] Furthermore, the target HTOL instance is a HTOL instance established according to the real interaction information of the input image.
[0030] Further, it includes: sequentially acquiring the predicted octet of the predicted HTOL instance and the target octet of the target HTOL instance; performing matching analysis on the predicted octet and the target octet according to the predetermined matching rule to obtain the predicted matching degree.
[0031] Specifically, an instance matcher is used to match the predicted human-tool-object (HTOL) instance with the target HTOL instance to evaluate the accuracy of the prediction. The design of the instance matcher includes predetermined matching rules for accurately matching the predicted HTOL instance with the target HTOL instance.
[0032] Specifically, the instance matcher stores predetermined matching rules. These rules are the core of the matching process and define how to judge whether the predicted HTOL instance matches the target HTOL instance. The predetermined matching rules include position matching, relationship category matching, and one-to-one matching, which ensure the comprehensiveness and accuracy of the matching process. Then, the predicted HTOL instance and the target HTOL instance are matched and analyzed according to the predetermined matching rules, and the differences between the predicted instance and the target instance in terms of position, relationship category, etc. are compared to determine whether the two match. Next, the predicted octet of the predicted HTOL instance and the target octet of the target HTOL instance are matched and analyzed, and the accuracy of the predicted instance and the target instance in each dimension is ensured by refining the matching process. In addition, the target HTOL instance is established based on the real interaction information of the input image. This means that the target instance is an accurate representation of the real scene, providing a reliable benchmark for the matching process. Finally, the quantitative goal of the degree of matching between the predicted HTOL instance and the target HTOL instance is achieved through this matching analysis.
[0033] Further, the instance matcher refers to matching the predicted human action-tool-object interaction (HTOI) instance with the real HTOI instance to evaluate the performance of the model. The goal of the matching is to match each predicted HTOI instance with its corresponding real HOI instance to determine whether the identification and location of the human action, tool, object and interaction are correctly predicted. Among them, before matching, it is usually necessary to define some matching rules, namely the predetermined matching rules, which specifically include position matching, relationship category matching and one-to-one matching. Among them, the position matching refers to determining whether the position matches by calculating the IoU (intersection over union) between the predicted human action box and object box and the real box. If the IoU is greater than the threshold, the position matching is considered successful. The relationship category matching refers to determining whether the predicted human action category, tool, object category and relationship category, interaction category are matched by comparing them with the real category. If the category matching is successful, it is considered to be a matched instance. The one-to-one matching means that during the matching process, it is necessary to ensure that each real target HTOL instance can only match one predicted HTOL instance, and each predicted HTOL instance also only matches one real target HTOL instance. This can be achieved by selecting the best match, for example the one with the highest IoU.
[0034] Through the instance matcher and the predetermined matching rules, the accurate matching analysis between the predicted HTOL instance and the target HTOL instance is realized. This provides an important basis for evaluating the performance of the model and the accuracy of the prediction, especially in application scenarios that require precise operation and functional evaluation.
[0035] Furthermore, it also includes: Reading a matching loss function; performing a loss analysis on the predicted matching degree based on the matching loss function to obtain a matching loss degree; and optimizing the predetermined matching rule with the goal of minimizing the matching loss degree.
[0036] Specifically, the predicted matching degree is analyzed using the matching loss function, and the matching loss degree is minimized by optimizing the predetermined matching rules. First, the matching loss function is read. This loss function is the core of evaluating the predicted matching degree and defines how to calculate the difference between the predicted matching degree and the actual matching degree. The design of the loss function directly affects the model's evaluation of the matching error and the subsequent optimization process. Then, the predicted matching degree is analyzed based on the matching loss function to quantify the difference between the predicted matching degree and the target matching degree, that is, the matching loss degree is calculated. Next, the predetermined matching rules are optimized with the goal of minimizing the matching loss degree. In other words, by adjusting the matching rules, the difference between the predicted matching degree and the target matching degree is minimized. The goal of the optimization is to find a set of optimal matching rules so that the prediction results of the model are as close to the actual situation as possible. Finally, through this optimization process, the accuracy of its prediction can be improved, and the interaction between human actions, tools and objects can be better understood and simulated.
[0037] Specifically, an instance is an octet (ch, cr, ct, ci, co, bh, bt, bo), where (ch, cr, ct) represents the category confidence of human action, relation and tool, (ct, ci, co) represents the category confidence of tool, interaction and object, and (bh, bt, bo) is the bounding box of human action, tool and object. The two-stage HOI detector first predicts object proposals (ch, bh), (ct, bt), (co, bo) using an object detector, then enumerates the detected (human action, tool) pairs, predicts cr through relation classification, (tool, object) pairs through interaction classification, and predicts ci through interaction classification.
[0038] Try to approximate the following probabilities in a given dataset: p(h,r,t,i,o)=p(r,h,t)p(i,t,o) =p(h,t)p(r|h,t)p(t,o)p(i|t,o)≈p(h)p(t)p(r|h,t)p(t)p(o)p(i|t,o) The bounding box confidences of human actions, tools, and objects are denoted as p(h), p(t), and p(o), respectively. p(r|h, t) represents the probability of relation r occurring given the human action bounding box h and tool bounding box t, and p(i|t,o) represents the probability of interaction i occurring given the tool bounding box t and object bounding box o, which is usually implemented by a multi-stream interaction recognition model. In this approach, the detector and classifier are optimized separately.
[0039] Instead, HTOI detection is considered as a set prediction problem of bipartite graph matching between predicted and true data. Elements in the HTOI set are directly predicted, and the proposed HTOI matching loss is optimized in a unified way. By converting the HTOI detection problem into a set prediction problem and uniformly optimizing the HTOI matching loss, the mutual relations and interactions between HTOI instances can be better captured. The advantage of this approach is that the HTOI detection problem can be considered from a global perspective, avoiding the step-by-step processing in traditional methods, and the relations and interactions between HTOI instances can be better handled. By predicting and matching HTOI instances, the HTOI detector attempts to approximate this probability distribution to identify and localize human actions, tool objects, and interactions in the image. It is assumed that the model outputs a fixed-size prediction set, where N is chosen to be larger than the number of HTOI relations in the image. Denote the predicted HTOI set by P = pi , i = 1, 2, ..., N, and the true HTOI set by G = gi , i = 1, 2, ..., M, ∅,..., ∅, where M ≤ N. M represents the number of true HOIs in the image. The lengths of the two sets are made equal by padding ∅ into the true set.
[0040] When the predicted matching degree reaches a matching threshold, the predicted HTOL instance is used as an interaction detection result of the input image.
[0041] Further, when the predicted matching degree does not reach the matching threshold value, a prediction optimization instruction is issued, and a secondary prediction analysis is performed on the global information based on the prediction optimization instruction.
[0042] Specifically, whether to accept the predicted HTOL instance as the interactive detection result of the input image is determined based on whether the predicted matching degree reaches the matching threshold, and a prediction optimization instruction is issued for secondary prediction analysis when necessary.
[0043] First, when the predicted match reaches the matching threshold, the predicted HTOL instance is used as the interactive detection result of the input image. This threshold is a preset standard used to determine whether the prediction accuracy is high enough. When the predicted match exceeds this standard, the model's prediction can be considered reliable, so it is used as the final detection result. Then, a prediction optimization instruction is issued when the predicted match does not reach the matching threshold. This instruction is the key to the model's self-adjustment and optimization, indicating that the model needs to re-evaluate and improve the prediction process. Next, a secondary prediction analysis of the global information is performed based on the prediction optimization instruction, that is, the process of self-learning and adjustment according to the prediction optimization instruction. By re-analyzing the global information, the model tries to find better prediction results to improve the prediction match. Finally, through this prediction optimization process, the model can continuously learn and improve, thereby improving the accuracy and reliability of its predictions.
[0044] By setting matching thresholds and prediction optimization instructions, dual evaluation and optimization of the model prediction results are achieved, which provides important support for improving the prediction accuracy of the model and understanding complex interaction scenarios.
[0045] One or more technical solutions provided in this application have at least the following technical effects or advantages: The visual features of the input image are obtained through a backbone network, wherein the backbone network refers to a convolutional neural network; an interaction detector is activated, wherein the interaction detector has an interaction prediction strategy embedded therein, and the visual features are analyzed according to the interaction prediction strategy to obtain a predicted HTOL instance; the predicted HTOL instance is matched and analyzed with a target HTOL instance through an instance matcher to obtain a predicted matching degree; when the predicted matching degree reaches a matching threshold value, the predicted HTOL instance is used as an interaction detection result of the input image.
[0046] 1. Focus on the contextual understanding of tool use: Tool Function Matcher focuses on understanding how and in what context the tool is used, which is very important for certain application scenarios such as education, industrial training, safety supervision, etc. It not only identifies people and tools, but also evaluates whether the tool is used in the right way, which is usually not involved in human interaction detection algorithms.
[0047] 2. Functional matching and evaluation: This algorithm can evaluate the functional use of tools, such as determining whether a hammer is used to hammer a nail. This functional insight is critical to ensuring that tasks are completed correctly and improving safety.
[0048] 3. Fine-grained interaction recognition: By combining image features and hand tracking data, the tool function matcher is able to identify and understand the interaction between people and tools at a finer granularity, which is especially important for scenarios that require precise operational guidance (such as surgical simulation, precision manufacturing, etc.).
[0049] 4. Customizability and scalability: By defining functional embeddings for different tools and actions, this algorithm can be customized and extended according to specific application requirements. This flexibility can adapt to a variety of different application scenarios and requirements.
[0050] 5. Potential for real-time feedback and guidance: Since algorithms can evaluate the correctness of tool usage in real time, it is possible to provide real-time feedback and guidance, which is very helpful for learning and improving tool usage skills.
[0051] In summary, we propose an eight-tuple HTOI instance structure based on the existing five-tuple HOI instance structure, propose a relationship classification for human-tool interaction, and an action classification structure for tool-object interaction, and express the human-tool-object ternary action with a combination of relationships and actions. This provides additional value for application scenarios that require precise operation and functional evaluation.
[0052] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
[0053] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the present application and its equivalent technology, the present application is also intended to include these modifications and variations.
Claims
1. A knowledge-based intelligent interaction detection method for action tool objects, characterized in that: include: Acquire visual features of an input image through a backbone network, wherein the backbone network refers to a convolutional neural network; activating an interaction detector, wherein the interaction detector has an interaction prediction strategy embedded therein, and analyzing the visual features according to the interaction prediction strategy to obtain a predicted HTOL instance; Performing matching analysis on the predicted HTOL instance and the target HTOL instance through an instance matcher to obtain a predicted matching degree; When the predicted matching degree reaches a matching threshold, taking the predicted HTOL instance as an interaction detection result of the input image; Among them, include: The interaction detector is composed of an encoder, a decoder and a multi-layer perceptron; Performing a global context analysis on the visual features by the encoder to obtain global information; The decoder performs prediction analysis on the global information in combination with the interactive prediction strategy to obtain a predicted HTOL relationship; Performing embedding decoding processing on the predicted HTOL relationship through the multi-layer perceptron to obtain the predicted HTOL instance; Among them, include: The multilayer perceptron includes M one-layer MLP branches and N three-layer MLP branches, where M=7 and N=3; Analyzing the predicted HTOL relationship through the M first-layer MLP branches to obtain a layer of analysis results, wherein the first-layer analysis results include a person's confidence, a relationship's confidence, a tool's confidence, an object's confidence, and an interaction's confidence; Analyzing the predicted HTOL relationship through N three-layer MLP branches to obtain a three-layer analysis result, wherein the three-layer analysis result includes a predicted human action frame, a predicted tool frame, and a predicted object frame; Reading a predetermined eight-tuple strategy, and analyzing the first-layer analysis result and the third-layer analysis result based on the predetermined eight-tuple strategy to obtain a HTOL eight-tuple; The HTOL octet is used as the predicted HTOL instance.
2. The method for intelligent interaction detection of action tool objects based on knowledge according to claim 1, characterized in that: include: The instance matcher stores a predetermined matching rule; Performing matching analysis on the predicted HTOL instance and the target HTOL instance according to the predetermined matching rule to obtain the predicted matching degree; The predetermined matching rules include location matching, relationship category matching and one-to-one matching.
3. The method for intelligent interaction detection of action tool objects based on knowledge according to claim 2 is characterized in that: The target HTOL instance refers to a HTOL instance established according to the real interaction information of the input image.
4. The method for detecting the intelligent interaction of motion tool objects based on knowledge according to claim 2, characterized in that: Also includes: Sequentially acquiring a predicted octet of the predicted HTOL instance and a target octet of the target HTOL instance; The predicted octet is matched and analyzed with the target octet according to the predetermined matching rule to obtain the predicted matching degree.
5. According to claim 3, the method for intelligent interaction detection of action tool objects based on knowledge is characterized in that: Also includes: Read the matching loss function; Performing loss analysis on the predicted matching degree based on the matching loss function to obtain a matching loss degree; The predetermined matching rule is optimized with the goal of minimizing the matching loss.
6. The method for intelligent interaction detection of action tool objects based on knowledge according to claim 1, characterized in that: When the prediction matching degree does not reach the matching threshold value, a prediction optimization instruction is issued, and a secondary prediction analysis is performed on the global information based on the prediction optimization instruction.
Citation Information
Patent Citations
Ternary interaction action detection method, system, equipment and medium
CN117953590A
