A knowledge graph-based unpacking responsibility tracking and tracing method

Abstract

This invention discloses a knowledge graph-based method for tracking and tracing responsibility in intelligent unboxing, comprising the following steps: constructing an intelligent unboxing responsibility knowledge graph; generating a corresponding set of responsibility constraints upon receiving an intelligent unboxing request; collecting control process information during the unboxing execution process and constructing a control-level atomic behavior sequence; collecting image or video data during the unboxing process, calling an improved AnomalyGPT model for anomaly detection, and generating visual evidence nodes; locating the violation semantic path and determining the responsibility trigger point in the intelligent unboxing responsibility knowledge graph; calculating the responsibility contribution of each responsible entity using the Shapley value method based on the violation semantic path and the set of responsible entities; generating responsibility tracing results and updating the intelligent unboxing responsibility knowledge graph. This invention achieves traceable, quantifiable, and interpretable tracing of responsibility during the intelligent unboxing process, improving the accuracy of anomaly identification and the objectivity of responsibility determination.

Description

Technical Field

[0001] This invention relates to the field of intelligent equipment management technology, and in particular to an out-of-box responsibility tracking and tracing method based on knowledge graphs. Background Technology

[0002] With the widespread application of smart lockers, smart boxes, and other smart unpacking devices in warehousing and logistics, industrial manufacturing, special materials management, and confidential scenarios, the need for accountability and traceability regarding the smart unpacking process is becoming increasingly prominent. Existing smart unpacking management technologies typically record unpacking behavior through access control records, operation logs, or simple video recordings, and determine responsibility afterward based on the operator or the last unpacking record. However, in practical applications, the above-mentioned technical solutions generally suffer from the following problems: Existing technologies often use linear logs or isolated records to describe the intelligent unpacking process, lacking a unified model of the relationships between the container, key components, personnel roles, operating rules, and evidence. This makes it difficult to reconstruct the complete unpacking behavior chain when anomalies or disputes occur, and accountability relies on human experience, resulting in poor accuracy and consistency. Existing technologies mainly rely on fixed rule thresholds or manual video playback to identify abnormal behavior, which is difficult to effectively handle complex scenarios such as occlusion, reflection, and rapid operations. Abnormal behavior is highly concealed, visual evidence is insufficient, and it is difficult to form structured evidence that can be used for liability determination. In addition, in intelligent unpacking scenarios involving multiple parties, existing technologies usually use simple "last operator" or "authority attribution" methods to determine liability, lacking means to quantify the contribution of different responsible parties in the formation of abnormal results. This easily leads to disputes over liability determination and fails to meet the application needs of auditing, arbitration, and compliance management.

[0003] Therefore, how to provide an out-of-box responsibility tracking and tracing method based on knowledge graphs is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] One objective of this invention is to propose a knowledge graph-based method for tracking and tracing responsibility in unboxing. This invention comprehensively utilizes knowledge graph modeling, anomaly visual detection, and responsibility quantification analysis techniques to perform correlation modeling of personnel, rules, behaviors, and evidence during the intelligent unboxing process. It achieves anomaly identification through an improved AnomalyGPT and completes responsibility tracing and quantification assessment based on the violation semantic path and Shapley value method. It has the advantages of accurate responsibility tracking, strong interpretability of tracing results, and objective and reliable responsibility determination.

[0005] An out-of-box responsibility tracing and attribution method based on knowledge graphs according to an embodiment of the present invention includes the following steps: Step 1: Construct a knowledge graph of intelligent unboxing responsibility; Step 2: Upon receiving a smart unboxing request, generate a set of responsibility constraints corresponding to this smart unboxing based on the smart unboxing responsibility knowledge graph; Step 3: During the execution of intelligent unpacking, collect intelligent unpacking control process information, construct a control-level atomic behavior sequence, and write it into the intelligent unpacking responsibility knowledge graph; Step 4: Collect image or video data during the intelligent unpacking process, and call the improved AnomalyGPT model to perform anomaly detection on the image or video data. The improved AnomalyGPT model includes an evidence constraint prompt generation module, a component-level ROI adaptive localization module, a temporal consistency constraint module, and a structured anomaly output module. It outputs the anomaly type, anomaly region, anomaly time interval, and anomaly confidence level, and writes the output results as visual evidence nodes into the intelligent unpacking responsibility knowledge graph. Step 5: Based on visual evidence nodes, control-level atomic behavior sequences, and responsibility constraint sets, locate the violation semantic path and determine the responsibility trigger point in the intelligent unboxing responsibility knowledge graph; Step Six: Based on the violation semantic path and the set of responsible parties, calculate the responsibility contribution of each responsible party using the Shapley value method; Step 7: Generate intelligent unboxing responsibility tracing results that include a set of responsibility constraints, a sequence of control-level atomic behaviors, visual evidence nodes, violation semantic paths, and responsibility contribution, and update the intelligent unboxing responsibility knowledge graph accordingly.

[0006] Optionally, step one specifically includes: Collect the intelligent unpacking equipment ledger, box structure configuration file, personnel and role authorization list, and rule clause configuration file, and perform field normalization processing on the collected data, mapping the box number, component number, personnel number, role number, and rule number to unique identifiers; Based on the preset graph ontology definition, the node types of the intelligent unpacking responsibility knowledge graph are constructed. The node types include box entity, key component entity, personnel entity, role entity, rule entity, behavior entity, and evidence entity. In the intelligent unpacking responsibility knowledge graph, entity association relationships are established, including the inclusion relationship between the box entity and the key component entity, the possession relationship between the personnel entity and the role entity, the constraint relationship between the role entity and the rule entity, the constraint relationship between the rule entity and the behavior entity, and the pointing relationship between the evidence entity and the behavior entity. Based on the configuration file of the box structure, an observable mapping relationship between key components and behavioral entities is generated, and the observable mapping relationship is written into the intelligent unpacking responsibility knowledge graph; The rule content in the rule clause configuration file is segmented at the clause level, the segmented rule clauses are bound to the corresponding rule entities, and written into the intelligent unpacking responsibility knowledge graph.

[0007] Optionally, step two specifically includes: Upon receiving a smart box opening request, the system reads the box entity, personnel entity, and role entity corresponding to the smart box opening request from the smart box opening responsibility knowledge graph, and obtains the risk level information associated with the box entity. Based on the risk level information of the container entity, the rule entities that match the risk level are retrieved in the intelligent unpacking responsibility knowledge graph to form a set of candidate rules; Based on the relationship between personnel entities and role entities, the candidate rule set is filtered for role adaptation, and rule entities that are not applicable to the current personnel role are removed; Based on the constraint relationship between rule entities and behavior entities, the set of allowed behavior types and the order of behavior execution are determined for this smart unpacking, forming a subset of behavior constraints. Based on the evidence requirement field recorded in the rule entity, the set of evidence types that must be collected during this smart unpacking process is determined, forming an evidence constraint subset; The behavioral constraint subset and the evidence constraint subset are combined to generate a set of responsibility constraints that uniquely corresponds to this smart unboxing request, and the set of responsibility constraints is written into the smart unboxing responsibility knowledge graph.

[0008] Optionally, step three specifically includes: During the intelligent unpacking process, control process information corresponding to the intelligent unpacking request is collected in chronological order. The collected control process information is processed into events, and the control process information is mapped into control-level atomic behaviors according to preset behavior types. Each control-level atomic behavior is assigned a unique behavior identifier and time identifier. Based on the execution order of behaviors determined in the set of responsibility constraints, the control-level atomic behaviors are sequentially arranged to form a sequence of control-level atomic behaviors corresponding to this smart unpacking request; The sequence of control-level atomic behaviors is written into the intelligent unpacking responsibility knowledge graph as behavioral entities, and the association between behavioral entities and corresponding rule entities and evidence entities is established.

[0009] Optionally, the improved AnomalyGPT model includes an evidence constraint hint generation module, a component-level ROI adaptive localization module, a temporal consistency constraint module, and a structured anomaly output module. The evidence constraint prompt generation module reads the rule entities and key component entities corresponding to the responsibility constraint set from the intelligent unpacking responsibility knowledge graph, encodes the evidence requirements recorded in the rule entities and the component types of the key component entities to form an evidence constraint prompt vector, and inputs it into the text encoding layer. The evidence constraint prompt vector is then converted into semantic constraint features through a multi-layer attention mapping function. The component-level ROI adaptive localization module inputs image or video data into the visual feature extraction layer to generate a multi-scale feature map. Based on the component type information recorded in the key component entity, it generates a set of candidate regions on the multi-scale feature map and calculates the matching weight between each candidate region and the component type. The target ROI features are obtained by weighted summation. The temporal consistency constraint module performs temporal alignment on the features extracted from the same target ROI in consecutive time frames, and constructs a temporally consistent feature representation by weighted fusion of the similarity of feature vectors in adjacent time frames. The structured anomaly output module jointly calculates temporal consistency feature representation and semantic constraint feature in the cross-modal fusion layer. By weighting and combining the visual anomaly response intensity and semantic constraint matching degree, anomaly activation results are obtained. Based on the anomaly activation results, anomaly type, anomaly region, anomaly time interval, and anomaly confidence are output. The anomaly confidence is obtained by normalizing the anomaly activation results and is used as a weight attribute to form visual evidence nodes together with the corresponding anomaly type, anomaly region, and anomaly time interval. The visual evidence nodes are then written into the intelligent unpacking responsibility knowledge graph.

[0010] Optionally, step five specifically includes: In the intelligent unpacking responsibility knowledge graph, an evidence retrieval set is constructed with visual evidence nodes as the starting node. The evidence retrieval set includes behavioral entities that have a pointing relationship with the visual evidence nodes, and rule entities that have a constraint relationship with the behavioral entities. Based on the time markers in the control-level atomic behavior sequence, behavioral entities that overlap with the abnormal time intervals corresponding to the visual evidence nodes are selected from the behavioral entities to form a candidate responsibility triggering behavior set, and the candidate responsibility triggering behavior set is added to the evidence retrieval set; In the evidence retrieval set, semantic path search is performed according to the relationship between rule entities, behavior entities and evidence entities to generate a semantic path set consisting of rule entities, behavior entities and evidence entities; The semantic path set is filtered, and the semantic path with the shortest path length that simultaneously covers the visual evidence nodes and the candidate responsibility triggering behavior set is selected as the violation semantic path. The behavioral entities that are directly related to visual evidence nodes and are constrained by rule entities in the violation semantic path are identified as responsibility trigger points. After establishing the association between the responsibility trigger points and the corresponding rule entities and visual evidence nodes, they are written into the intelligent unboxing responsibility knowledge graph.

[0011] Optionally, step six specifically includes: The set of responsible entities corresponding to the violation semantic path is read from the intelligent unpacking responsibility knowledge graph, and the rule entities, behavioral entities and visual evidence nodes associated with the violation semantic path are read to form a set of elements for calculating the responsibility contribution. A feature set for calculating responsibility contribution is constructed based on the element set. The feature set includes violation identifiers corresponding to rule entities, trigger identifiers corresponding to behavior entities, and anomaly confidence levels corresponding to visual evidence nodes. The anomaly confidence level is written into the feature set as a feature weight. For each responsible entity in the set of responsible entities, a subset of feature items corresponding to the responsible entity is determined based on the relationship between the responsible entity and rule entities, behavioral entities, and visual evidence nodes in the intelligent unboxing responsibility knowledge graph; Perform combination enumeration on the feature items subset of each responsible entity in the set of responsible entities, calculate the abnormal result function value for each combination, the abnormal result function value is obtained by multiplying the weights of each feature item in the combination, and record the abnormal result function value corresponding to each combination as the marginal contribution value; The weighted average of the marginal contribution values ​​obtained by the same responsible entity under different combinations is used to obtain the Shapley value of the responsible entity, and the Shapley value is determined as the responsibility contribution degree of the responsible entity; The responsibility contribution of each responsible entity is normalized to form a responsibility contribution vector, and then the responsibility contribution vector is associated with the violation semantic path and written into the intelligent unboxing responsibility knowledge graph.

[0012] Optionally, step seven specifically includes: The data package of responsibility tracing results for this smart unboxing request is constructed by reading the set of responsibility constraints, control-level atomic behavior sequence, visual evidence nodes, violation semantic paths, responsibility trigger points, and responsibility contribution of each responsible entity from the smart unboxing responsibility knowledge graph. The data packets of the intelligent unpacking responsibility tracing results are subjected to consistency verification. The consistency verification includes verifying the correspondence between rule entities and responsibility constraint sets in the violation semantic path, verifying the temporal correspondence between responsibility trigger points and control-level atomic behavior sequences, and verifying the correspondence between abnormal time intervals of visual evidence nodes and time identifiers of control-level atomic behavior sequences. After the consistency verification is passed, the intelligent unpacking responsibility tracing result is generated. The intelligent unpacking responsibility tracing result includes a set of responsibility constraints, a sequence of control-level atomic behaviors, visual evidence nodes, violation semantic paths, responsibility trigger points, and responsibility contribution vectors. The intelligent unpacking responsibility tracing result is written into the intelligent unpacking responsibility knowledge graph. The intelligent unpacking responsibility knowledge graph is updated based on the intelligent unpacking responsibility tracing results. The update includes marking the rule entity, behavior entity and evidence entity associated with this intelligent unpacking request as an instance that has occurred, and writing the responsibility contribution vector into the historical contribution record field of the corresponding responsible subject node.

[0013] The beneficial effects of this invention are: This invention constructs a knowledge graph of responsibility for intelligent unpacking, unifying and modeling the box, key components, personnel, roles, rules, behaviors, and evidence to achieve a structured representation and traceable management of the entire intelligent unpacking process. Based on this, an improved AnomalyGPT model is introduced to detect anomalies in image or video data generated during intelligent unpacking, targeting key components and responsibility constraints. This effectively improves the accuracy and reliability of evidence in complex scenarios such as occlusion, reflection, and rapid operations. Simultaneously, by combining violation semantic path analysis and a responsibility contribution calculation method based on Shapley values, the invention quantifies and attributes abnormal results in situations involving multiple responsible parties, avoiding the bias of determining responsibility solely based on operation sequence or authority. This invention can establish a responsibility tracking and tracing mechanism in complex intelligent unpacking application scenarios that is "verifiable in evidence, explainable in path, and quantifiable in responsibility," significantly improving the objectivity, consistency, and auditability of responsibility determination. It has significant engineering application value and practical significance for ensuring the safe operation of intelligent unpacking systems, reducing management dispute risks, and improving the compliance management level of intelligent equipment. Attached Figure Description

[0014] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of an open-box responsibility tracking and tracing method based on knowledge graphs proposed in this invention; Figure 2 This is a schematic diagram of an open-box responsibility tracking and tracing method based on knowledge graphs proposed in this invention; Figure 3 This is a framework diagram of the improved AnomalyGPT model in the knowledge graph-based open-box responsibility tracing and attribution method proposed in this invention. Detailed Implementation

[0015] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0016] refer to Figures 1-3 An out-of-box responsibility tracing and attribution method based on knowledge graphs includes the following steps: Step 1: Construct an intelligent unpacking responsibility knowledge graph. The intelligent unpacking responsibility knowledge graph includes box entities, key component entities, personnel entities, role entities, rule entities, behavior entities, and evidence entities, and establishes the relationships between these entities. Step 2: Upon receiving a smart unboxing request, generate a set of responsibility constraints corresponding to this smart unboxing based on the smart unboxing responsibility knowledge graph; Step 3: During the execution of intelligent unpacking, collect intelligent unpacking control process information, construct a control-level atomic behavior sequence, and write it into the intelligent unpacking responsibility knowledge graph; Step 4: Collect image or video data during the intelligent unpacking process, and call the improved AnomalyGPT model to perform anomaly detection on the image or video data. The improved AnomalyGPT model includes an evidence constraint prompt generation module, a component-level ROI adaptive localization module, a temporal consistency constraint module, and a structured anomaly output module. It outputs the anomaly type, anomaly region, anomaly time interval, and anomaly confidence level, and writes the output results as visual evidence nodes into the intelligent unpacking responsibility knowledge graph. Step 5: Based on visual evidence nodes, control-level atomic behavior sequences, and responsibility constraint sets, locate the violation semantic path and determine the responsibility trigger point in the intelligent unboxing responsibility knowledge graph; Step Six: Based on the violation semantic path and the set of responsible parties, calculate the responsibility contribution of each responsible party using the Shapley value method; Step 7: Generate intelligent unboxing responsibility tracing results that include a set of responsibility constraints, a sequence of control-level atomic behaviors, visual evidence nodes, violation semantic paths, and responsibility contribution, and update the intelligent unboxing responsibility knowledge graph accordingly.

[0017] In this embodiment, step one specifically includes: Collect the intelligent unpacking equipment ledger, box structure configuration file, personnel and role authorization list, and rule clause configuration file, and perform field normalization processing on the collected data, mapping the box number, component number, personnel number, role number, and rule number to unique identifiers; Based on the preset graph ontology definition, the node types of the intelligent unpacking responsibility knowledge graph are constructed. The node types include box entity, key component entity, personnel entity, role entity, rule entity, behavior entity, and evidence entity. Among them, the box entity includes box number and box model attributes, the key component entity includes component number, component type, and box number attributes, the personnel entity includes personnel number attribute, the role entity includes role number attribute, the rule entity includes rule number and rule triggering condition attributes, the behavior entity includes behavior type attribute, and the evidence entity includes evidence type attribute. In the intelligent unpacking responsibility knowledge graph, entity relationships are established, including the inclusion relationship between the box entity and the key component entity, the possession relationship between the personnel entity and the role entity, the constraint relationship between the role entity and the rule entity, the constraint relationship between the rule entity and the behavior entity, and the pointing relationship between the evidence entity and the behavior entity. Based on the configuration file of the box structure, an observable mapping relationship between key components and behavioral entities is generated, and the observable mapping relationship is written into the intelligent unpacking responsibility knowledge graph; The rule content in the rule clause configuration file is segmented at the clause level, the segmented rule clauses are bound to the corresponding rule entities, and written into the intelligent unpacking responsibility knowledge graph.

[0018] In this embodiment, step two specifically includes: Upon receiving a smart box opening request, the system reads the box entity, personnel entity, and role entity corresponding to the smart box opening request from the smart box opening responsibility knowledge graph, and obtains the risk level information associated with the box entity. Based on the risk level information of the container entity, the rule entities that match the risk level are retrieved in the intelligent unpacking responsibility knowledge graph to form a set of candidate rules; Based on the relationship between personnel entities and role entities, the candidate rule set is filtered for role adaptation, and rule entities that are not applicable to the current personnel role are removed; Based on the constraint relationship between rule entities and behavior entities, the set of allowed behavior types and the order of behavior execution are determined for this smart unpacking, forming a subset of behavior constraints. Based on the evidence requirement field recorded in the rule entity, the set of evidence types that must be collected during this smart unpacking process is determined, forming an evidence constraint subset; The behavioral constraint subset and the evidence constraint subset are combined to generate a set of responsibility constraints that uniquely corresponds to this smart unboxing request, and the set of responsibility constraints is written into the smart unboxing responsibility knowledge graph.

[0019] In this embodiment, step three specifically includes: During the intelligent unpacking process, control process information corresponding to the intelligent unpacking request is collected in chronological order. The control process information includes identity authentication results, authorization verification results, unlocking control commands, lock control execution feedback, door status change information, and reset and lock feedback information. The collected control process information is processed into events, and the control process information is mapped into control-level atomic behaviors according to preset behavior types. Each control-level atomic behavior is assigned a unique behavior identifier and time identifier. Based on the execution order of behaviors determined in the set of responsibility constraints, the control-level atomic behaviors are sequentially arranged to form a sequence of control-level atomic behaviors corresponding to this smart unpacking request; The sequence of control-level atomic behaviors is written into the intelligent unpacking responsibility knowledge graph as behavioral entities, and the association between behavioral entities and corresponding rule entities and evidence entities is established.

[0020] In this embodiment, the improved AnomalyGPT model includes an evidence constraint hint generation module, a component-level ROI adaptive localization module, a temporal consistency constraint module, and a structured anomaly output module: The evidence constraint prompt generation module reads the rule entities and key component entities corresponding to the responsibility constraint set from the intelligent unpacking responsibility knowledge graph, encodes the evidence requirements recorded in the rule entities and the component types of the key component entities to form an evidence constraint prompt vector, and inputs it into the text encoding layer. The evidence constraint prompt vector is converted into semantic constraint features through a multi-layer attention mapping function. The component-level ROI adaptive localization module inputs image or video data into the visual feature extraction layer to generate a multi-scale feature map. Based on the component type information recorded in the key component entity, it generates a set of candidate regions on the multi-scale feature map and calculates the matching weight of each candidate region with the component type. The target ROI features are obtained by weighted summation, where the matching weight is used to amplify the feature response related to the target component and suppress the features of non-key regions. The temporal consistency constraint module performs temporal alignment on features extracted from the same target ROI in consecutive time frames. By weighted fusion of the similarity of feature vectors in adjacent time frames, a temporal consistency feature representation is constructed. The time weight decreases with the inter-frame interval to reduce the impact of instantaneous occlusion, jitter, or rapid action on the anomaly judgment result. The structured anomaly output module jointly calculates temporal consistency feature representation and semantic constraint feature in the cross-modal fusion layer. By weighting and combining the visual anomaly response intensity and semantic constraint matching degree, it obtains the anomaly activation result and outputs the anomaly type, anomaly region, anomaly time interval, and anomaly confidence based on the anomaly activation result. The anomaly confidence is obtained by normalizing the anomaly activation result and is used as a weight attribute to form a visual evidence node together with the corresponding anomaly type, anomaly region, and anomaly time interval. The visual evidence node is then written into the intelligent unboxing responsibility knowledge graph.

[0021] The improved AnomalyGPT model is built on the basis of the existing AnomalyGPT model. It is consistent with the original AnomalyGPT model in terms of overall technical approach, cross-modal visual-language modeling framework and basic idea of ​​anomaly detection. Both adopt a combination of visual feature extraction and text semantic encoding to perceive anomalies in the input image or video data and output anomaly-related results through cross-modal fusion mechanism. This avoids the reliance on a large number of anomaly sample annotations and has good versatility and transferability.

[0022] Compared to the original AnomalyGPT model, the improved AnomalyGPT model in this implementation has made several structural improvements for the smart unboxing scenario: By introducing an evidence constraint prompt generation module, the evidence requirements related to rule entities and key component entities in the responsibility constraint set are encoded into semantic constraint features, giving the model clear rule guidance during anomaly detection; through a component-level ROI adaptive localization module, the visual analysis scope is limited to key component areas such as the door and latch, and non-critical area interference is suppressed through a matching weighting mechanism; at the same time, a temporal consistency constraint module is set to perform weighted fusion of target ROI features in consecutive time frames, effectively reducing the impact of occlusion, jitter, and instantaneous actions on anomaly detection; finally, the structured anomaly output module transforms the anomaly results into standardized visual evidence nodes that can be directly written into the knowledge graph.

[0023] Through improvements, this implementation method can significantly enhance the targeting, stability, and interpretability of anomaly detection in intelligent unboxing scenarios: on the one hand, the model output results are highly consistent with the responsibility constraint set and knowledge graph structure, avoiding the problem of only outputting anomaly scores without being able to be used for responsibility identification; on the other hand, the anomaly results are expressed in a structured manner in the form of anomaly type, anomaly region, anomaly time interval, and anomaly confidence, providing reliable input for subsequent violation semantic path localization and responsibility contribution calculation, thereby effectively supporting the responsibility tracking and tracing needs in complex intelligent unboxing scenarios.

[0024] In this embodiment, step five specifically includes: In the intelligent unpacking responsibility knowledge graph, an evidence retrieval set is constructed with visual evidence nodes as the starting node. The evidence retrieval set includes behavioral entities that have a pointing relationship with the visual evidence nodes, and rule entities that have a constraint relationship with the behavioral entities. Based on the time markers in the control-level atomic behavior sequence, behavioral entities that overlap with the abnormal time intervals corresponding to the visual evidence nodes are selected from the behavioral entities to form a candidate responsibility triggering behavior set, and the candidate responsibility triggering behavior set is added to the evidence retrieval set; In the evidence retrieval set, semantic path search is performed according to the relationship between rule entities, behavior entities and evidence entities to generate a semantic path set consisting of rule entities, behavior entities and evidence entities; The semantic path set is filtered, and the semantic path with the shortest path length that simultaneously covers the visual evidence nodes and the candidate responsibility triggering behavior set is selected as the violation semantic path. The path length is compared based on the number of entity nodes and related relationships in the path. The behavioral entities that are directly related to visual evidence nodes and are constrained by rule entities in the violation semantic path are identified as responsibility trigger points. After establishing the association between the responsibility trigger points and the corresponding rule entities and visual evidence nodes, they are written into the intelligent unboxing responsibility knowledge graph.

[0025] In this embodiment, step six specifically includes: The set of responsible entities corresponding to the violation semantic path is read from the intelligent unpacking responsibility knowledge graph, and the rule entities, behavioral entities and visual evidence nodes associated with the violation semantic path are read to form a set of elements for calculating the responsibility contribution. Based on the set of elements, a set of feature items is constructed to calculate the responsibility contribution. The set of feature items includes violation identifiers corresponding to rule entities, trigger identifiers corresponding to behavior entities, and anomaly confidence levels corresponding to visual evidence nodes. The anomaly confidence level is written into the set of feature items as the feature item weight. For each responsible entity in the set of responsible entities, a subset of feature items corresponding to the responsible entity is determined based on the relationship between the responsible entity and rule entities, behavioral entities, and visual evidence nodes in the intelligent unboxing responsibility knowledge graph; Perform combination enumeration on the feature subsets of each responsible entity in the set of responsible entities, calculate the abnormal result function value for each combination, obtain the abnormal result function value by multiplying the weights of each feature item in the combination, and record the abnormal result function value corresponding to each combination as the marginal contribution value; The weighted average of the marginal contribution values ​​obtained by the same responsible entity under different combinations is used to obtain the Shapley value of the responsible entity, and the Shapley value is determined as the responsibility contribution degree of the responsible entity; The responsibility contribution of each responsible entity is normalized to form a responsibility contribution vector, and then the responsibility contribution vector is associated with the violation semantic path and written into the intelligent unboxing responsibility knowledge graph.

[0026] In this embodiment, step seven specifically includes: The data package of responsibility tracing results for this smart unboxing request is constructed by reading the set of responsibility constraints, control-level atomic behavior sequence, visual evidence nodes, violation semantic paths, responsibility trigger points, and responsibility contribution of each responsible entity from the smart unboxing responsibility knowledge graph. The data packets of the intelligent unpacking responsibility tracing results are subjected to consistency verification. The consistency verification includes verifying the correspondence between rule entities and responsibility constraint sets in the violation semantic path, verifying the temporal correspondence between responsibility trigger points and control-level atomic behavior sequences, and verifying the correspondence between abnormal time intervals of visual evidence nodes and time identifiers of control-level atomic behavior sequences. After the consistency verification is passed, the intelligent unpacking responsibility tracing result is generated. The intelligent unpacking responsibility tracing result includes the responsibility constraint set, the control-level atomic behavior sequence, the visual evidence node, the violation semantic path, the responsibility trigger point and the responsibility contribution vector, and the intelligent unpacking responsibility tracing result is written into the intelligent unpacking responsibility knowledge graph. The intelligent unpacking responsibility knowledge graph is updated based on the intelligent unpacking responsibility tracing results. The update includes marking the rule entity, behavior entity and evidence entity associated with this intelligent unpacking request as an instance that has occurred, and writing the responsibility contribution vector into the historical contribution record field of the corresponding responsible subject node.

[0027] Example 1: To verify the feasibility of this invention in practice, it was applied to an intelligent unpacking management scenario in a large manufacturing enterprise. This enterprise deployed intelligent unpacking equipment on its production floor for storing high-value tooling, precision measuring instruments, and critical spare parts. The boxes are equipped with electronic locks, cameras, door magnetic sensors, and a back-end control system. The unpacking process involves multiple responsible parties, including operators, reviewers, and equipment maintenance personnel. Furthermore, in actual operation, there have been multiple instances where responsibility was difficult to accurately determine after unpacking anomalies.

[0028] In this application scenario, the intelligent unpacking device triggers an average of approximately 120 unpacking operations per day, with each unpacking session lasting between 30 and 180 seconds. The system pre-classifies the risk levels of the boxes into three categories: ordinary, important, and high-risk. High-risk boxes require dual authorization and confirmation of complete opening and closing status. To adapt to this scenario, the system constructs an intelligent unpacking responsibility knowledge graph, which includes 48 box entities, 192 key component entities, 86 personnel entities, 5 types of role entities, and 32 rule entities. It establishes the relationships between boxes and key components, personnel and roles, rules and behaviors, and evidence. The rules are detailed in the system as triggering conditions and evidence requirements, such as "high-risk boxes must have visual evidence of complete door closure and latch return to position collected at the end of the unpacking process."

[0029] In actual operation, when an operator initiates a smart unlocking request, the system automatically generates a set of responsibility constraints based on a knowledge graph. This set specifies the allowed sequence of actions for this unlocking, the types of evidence to be collected, and the conditions for triggering violations. During the unlocking process, the system continuously collects control information such as identity authentication results, unlocking commands, lock control feedback, and changes in door status, forming a control-level atomic behavior sequence. Simultaneously, it collects video data from the cabinet's camera.

[0030] For the collected video data, the system calls an improved AnomalyGPT model for anomaly detection. Building upon the existing AnomalyGPT, the model is guided by an evidence constraint prompt generation module to focus on key component areas such as doors and latches; a component-level ROI adaptive localization module limits the analysis scope to key component areas and applies weighted analysis; a temporal consistency constraint module fuses features from consecutive frames to reduce occlusion and jitter interference; finally, it outputs the anomaly type, anomaly region, anomaly time interval, and anomaly confidence level, and writes the results as visual evidence nodes into the responsibility knowledge graph.

[0031] To verify the effectiveness of this invention, two existing technical solutions were compared and tested. One solution is a traditional intelligent unpacking management method, which only records unpacking logs and access control information, and relies on manual video playback for anomalies; the other solution is a video detection method based on fixed rules, which judges whether the door status is abnormal by threshold, but does not combine responsibility rules and semantic path analysis. The method of this invention and the comparative method were statistically analyzed on intelligent unpacking data for 30 consecutive days within the same production cycle, and the following results were obtained: Table 1 Comprehensive Comparison Data Table

[0032] As shown in Table 1, traditional log-based methods suffer from low anomaly detection rates due to a lack of structured visual evidence and rule-based semantic support. Furthermore, they require significant manual intervention for video playback and responsibility analysis after anomalies occur, with an average location time exceeding 18 minutes. While fixed-rule video methods improve anomaly detection rates to some extent, their limited rule expression capabilities prevent them from effectively handling complex scenarios such as occlusion and rapid operations, resulting in a significantly higher false alarm rate. Additionally, they remain difficult to accurately determine responsibility in scenarios involving multiple responsible parties.

[0033] In comparison, the method of this invention achieves an anomaly detection rate of 91.4%, significantly higher than the comparative methods. This is mainly due to the improved AnomalyGPT model's focused analysis of key components and its temporal consistency constraint mechanism, which enables the model to reliably identify fine-grained anomalies such as open doors and unreturned latches. Furthermore, because the anomaly results are written into the knowledge graph as structured visual evidence, the system can quickly locate the violation semantic path and automatically pinpoint the responsibility trigger point, reducing the average anomaly localization time to 2.6 minutes.

[0034] Regarding the determination of responsible parties, this invention combines the semantic path of violations with the Shapley value for responsibility contribution calculation to quantitatively analyze the responsibilities of operators, reviewers, and equipment. This effectively avoids the simplistic attribution method of "the last operator is the responsible party," increasing the accuracy of responsibility determination to 93.8%. Simultaneously, the system automatically generates auditable responsibility tracing results, significantly reducing the proportion of manual intervention and alleviating the workload of management personnel.

[0035] As can be seen from the above embodiments, in complex application scenarios of intelligent unpacking, the present invention can not only effectively improve the accuracy and stability of anomaly identification, but also realize automated reasoning of responsibility paths and quantitative calculation of responsibility contribution. It solves the problems of scattered evidence, unclear responsibility, and reliance on manual judgment in the prior art, and has significant engineering practical value and promotion significance.

[0036] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A knowledge graph-based method for unboxing responsibility tracing and attribution, characterized in that, Includes the following steps: Step 1: Construct a knowledge graph of intelligent unboxing responsibility; Step 2: Upon receiving a smart unboxing request, generate a set of responsibility constraints corresponding to this smart unboxing based on the smart unboxing responsibility knowledge graph; Step 3: During the execution of intelligent unpacking, collect intelligent unpacking control process information, construct a control-level atomic behavior sequence, and write it into the intelligent unpacking responsibility knowledge graph; Step 4: Collect image or video data during the intelligent unpacking process, and call the improved AnomalyGPT model to perform anomaly detection on the image or video data. The improved AnomalyGPT model includes an evidence constraint prompt generation module, a component-level ROI adaptive localization module, a temporal consistency constraint module, and a structured anomaly output module. It outputs the anomaly type, anomaly region, anomaly time interval, and anomaly confidence level, and writes the output results as visual evidence nodes into the intelligent unpacking responsibility knowledge graph. Step 5: Based on visual evidence nodes, control-level atomic behavior sequences, and responsibility constraint sets, locate the violation semantic path and determine the responsibility trigger point in the intelligent unboxing responsibility knowledge graph; Step Six: Based on the violation semantic path and the set of responsible parties, calculate the responsibility contribution of each responsible party using the Shapley value method; Step 7: Generate intelligent unboxing responsibility tracing results that include a set of responsibility constraints, a sequence of control-level atomic behaviors, visual evidence nodes, violation semantic paths, and responsibility contribution, and update the intelligent unboxing responsibility knowledge graph accordingly.

2. The knowledge graph-based method for out-of-box responsibility tracing and attribution as described in claim 1, characterized in that, Step one specifically includes: Collect the intelligent unpacking equipment ledger, box structure configuration file, personnel and role authorization list, and rule clause configuration file, and perform field normalization processing on the collected data, mapping the box number, component number, personnel number, role number, and rule number to unique identifiers; Based on the preset graph ontology definition, the node types of the intelligent unpacking responsibility knowledge graph are constructed. The node types include box entity, key component entity, personnel entity, role entity, rule entity, behavior entity, and evidence entity. In the intelligent unpacking responsibility knowledge graph, entity association relationships are established, including the inclusion relationship between the box entity and the key component entity, the possession relationship between the personnel entity and the role entity, the constraint relationship between the role entity and the rule entity, the constraint relationship between the rule entity and the behavior entity, and the pointing relationship between the evidence entity and the behavior entity. Based on the configuration file of the box structure, an observable mapping relationship between key components and behavioral entities is generated, and the observable mapping relationship is written into the intelligent unpacking responsibility knowledge graph; The rule content in the rule clause configuration file is segmented at the clause level, the segmented rule clauses are bound to the corresponding rule entities, and written into the intelligent unpacking responsibility knowledge graph.

3. The knowledge graph-based method for out-of-box responsibility tracing and attribution as described in claim 1, characterized in that, Step two specifically includes: Upon receiving a smart box opening request, the system reads the box entity, personnel entity, and role entity corresponding to the smart box opening request from the smart box opening responsibility knowledge graph, and obtains the risk level information associated with the box entity. Based on the risk level information of the container entity, the rule entities that match the risk level are retrieved in the intelligent unpacking responsibility knowledge graph to form a set of candidate rules; Based on the relationship between personnel entities and role entities, the candidate rule set is filtered for role adaptation, and rule entities that are not applicable to the current personnel role are removed; Based on the constraint relationship between rule entities and behavior entities, the set of allowed behavior types and the order of behavior execution are determined for this smart unpacking, forming a subset of behavior constraints. Based on the evidence requirement field recorded in the rule entity, the set of evidence types that must be collected during this smart unpacking process is determined, forming an evidence constraint subset; The behavioral constraint subset and the evidence constraint subset are combined to generate a set of responsibility constraints that uniquely corresponds to this smart unboxing request, and the set of responsibility constraints is written into the smart unboxing responsibility knowledge graph.

4. The knowledge graph-based method for out-of-box responsibility tracing and attribution as described in claim 1, characterized in that, Step three specifically includes: During the intelligent unpacking process, control process information corresponding to the intelligent unpacking request is collected in chronological order. The collected control process information is processed into events, and the control process information is mapped into control-level atomic behaviors according to preset behavior types. Each control-level atomic behavior is assigned a unique behavior identifier and time identifier. Based on the execution order of behaviors determined in the set of responsibility constraints, the control-level atomic behaviors are sequentially arranged to form a sequence of control-level atomic behaviors corresponding to this smart unpacking request; The sequence of control-level atomic behaviors is written into the intelligent unpacking responsibility knowledge graph as behavioral entities, and the association between behavioral entities and corresponding rule entities and evidence entities is established.

5. The knowledge graph-based method for out-of-box responsibility tracing and attribution as described in claim 1, characterized in that, The improved AnomalyGPT model includes an evidence constraint hint generation module, a component-level ROI adaptive localization module, a temporal consistency constraint module, and a structured anomaly output module. The evidence constraint prompt generation module reads the rule entities and key component entities corresponding to the responsibility constraint set from the intelligent unpacking responsibility knowledge graph, encodes the evidence requirements recorded in the rule entities and the component types of the key component entities to form an evidence constraint prompt vector, and inputs it into the text encoding layer. The evidence constraint prompt vector is then converted into semantic constraint features through a multi-layer attention mapping function. The component-level ROI adaptive localization module inputs image or video data into the visual feature extraction layer to generate a multi-scale feature map. Based on the component type information recorded in the key component entity, it generates a set of candidate regions on the multi-scale feature map and calculates the matching weight between each candidate region and the component type. The target ROI features are obtained by weighted summation. The temporal consistency constraint module performs temporal alignment on the features extracted from the same target ROI in consecutive time frames, and constructs a temporally consistent feature representation by weighted fusion of the similarity of feature vectors in adjacent time frames. The structured anomaly output module jointly calculates temporal consistency feature representation and semantic constraint feature in the cross-modal fusion layer. By weighting and combining the visual anomaly response intensity and semantic constraint matching degree, anomaly activation results are obtained. Based on the anomaly activation results, anomaly type, anomaly region, anomaly time interval, and anomaly confidence are output. The anomaly confidence is obtained by normalizing the anomaly activation results and is used as a weight attribute to form visual evidence nodes together with the corresponding anomaly type, anomaly region, and anomaly time interval. The visual evidence nodes are then written into the intelligent unpacking responsibility knowledge graph.

6. The knowledge graph-based out-of-box responsibility tracing and attribution method according to claim 1, characterized in that, Step five specifically includes: In the intelligent unpacking responsibility knowledge graph, an evidence retrieval set is constructed with visual evidence nodes as the starting node. The evidence retrieval set includes behavioral entities that have a pointing relationship with the visual evidence nodes, and rule entities that have a constraint relationship with the behavioral entities. Based on the time markers in the control-level atomic behavior sequence, behavioral entities that overlap with the abnormal time intervals corresponding to the visual evidence nodes are selected from the behavioral entities to form a candidate responsibility triggering behavior set, and the candidate responsibility triggering behavior set is added to the evidence retrieval set; In the evidence retrieval set, semantic path search is performed according to the relationship between rule entities, behavior entities and evidence entities to generate a semantic path set consisting of rule entities, behavior entities and evidence entities; The semantic path set is filtered, and the semantic path with the shortest path length that simultaneously covers the visual evidence nodes and the candidate responsibility triggering behavior set is selected as the violation semantic path. The behavioral entities that are directly related to visual evidence nodes and are constrained by rule entities in the violation semantic path are identified as responsibility trigger points. After establishing the association between the responsibility trigger points and the corresponding rule entities and visual evidence nodes, they are written into the intelligent unboxing responsibility knowledge graph.

7. The knowledge graph-based method for out-of-box responsibility tracing and attribution as described in claim 1, characterized in that, Step six specifically includes: The set of responsible entities corresponding to the violation semantic path is read from the intelligent unpacking responsibility knowledge graph, and the rule entities, behavioral entities and visual evidence nodes associated with the violation semantic path are read to form a set of elements for calculating the responsibility contribution. A feature set for calculating responsibility contribution is constructed based on the element set. The feature set includes violation identifiers corresponding to rule entities, trigger identifiers corresponding to behavior entities, and anomaly confidence levels corresponding to visual evidence nodes. The anomaly confidence level is written into the feature set as a feature weight. For each responsible entity in the set of responsible entities, a subset of feature items corresponding to the responsible entity is determined based on the relationship between the responsible entity and rule entities, behavioral entities, and visual evidence nodes in the intelligent unboxing responsibility knowledge graph; Perform combination enumeration on the feature items subset of each responsible entity in the set of responsible entities, calculate the abnormal result function value for each combination, the abnormal result function value is obtained by multiplying the weights of each feature item in the combination, and record the abnormal result function value corresponding to each combination as the marginal contribution value; The weighted average of the marginal contribution values ​​obtained by the same responsible entity under different combinations is used to obtain the Shapley value of the responsible entity, and the Shapley value is determined as the responsibility contribution degree of the responsible entity. The responsibility contribution of each responsible entity is normalized to form a responsibility contribution vector, and then the responsibility contribution vector is associated with the violation semantic path and written into the intelligent unboxing responsibility knowledge graph.

8. The knowledge graph-based out-of-box responsibility tracing and attribution method according to claim 1, characterized in that, Step seven specifically includes: The data package of responsibility tracing results for this smart unboxing request is constructed by reading the set of responsibility constraints, control-level atomic behavior sequence, visual evidence nodes, violation semantic paths, responsibility trigger points, and responsibility contribution of each responsible entity from the smart unboxing responsibility knowledge graph. The data packets of the intelligent unpacking responsibility tracing results are subjected to consistency verification. The consistency verification includes verifying the correspondence between rule entities and responsibility constraint sets in the violation semantic path, verifying the temporal correspondence between responsibility trigger points and control-level atomic behavior sequences, and verifying the correspondence between abnormal time intervals of visual evidence nodes and time identifiers of control-level atomic behavior sequences. After the consistency verification is passed, the intelligent unpacking responsibility tracing result is generated. The intelligent unpacking responsibility tracing result includes a set of responsibility constraints, a sequence of control-level atomic behaviors, visual evidence nodes, violation semantic paths, responsibility trigger points, and responsibility contribution vectors. The intelligent unpacking responsibility tracing result is written into the intelligent unpacking responsibility knowledge graph. The intelligent unpacking responsibility knowledge graph is updated based on the intelligent unpacking responsibility tracing results. The update includes marking the rule entity, behavior entity and evidence entity associated with this intelligent unpacking request as an instance that has occurred, and writing the responsibility contribution vector into the historical contribution record field of the corresponding responsible subject node.

Images