Local conflict map error correction method, device, equipment and medium

Abstract

The application discloses a local conflict graph error correction method, device, equipment and medium. The method performs initial automatic labeling on multi-modal data, and constructs a heterogeneous consistency graph by standardizing the candidate results. Then, the associated edge weight is calculated, and a constraint label propagation that retains the initial information is performed to obtain an updated distribution. Then, the conflict score is calculated by combining the distribution difference and the business constraint to screen the conflict nodes, and the local conflict subgraph is dynamically extracted accordingly. After reviewing the subgraph to obtain the revised distribution and the allowed propagation mask, the limited backhaul update is performed according to the mask and the path credibility until the final result is output when the convergence condition is met. The method significantly reduces the labeling and error correction cost and the interaction burden, and greatly improves the error correction efficiency, accuracy and global consistency in the cross-modal scene.

Description

Technical Field

[0001] This invention relates to the field of multimodal information processing technology, and in particular to a method, apparatus, device, and medium for correcting local conflict graphs. Background Technology

[0002] With the widespread application of artificial intelligence in scenarios such as video understanding, document recognition, intelligent quality inspection, and business data processing, multimodal annotation has become an important foundation for model training, data governance, and intelligent review. However, in multimodal annotation scenarios involving cross-modality and cross-temporal space, different data sources often exhibit significant differences in representation forms, long time spans, complex inter-page or inter-frame relationships, and scattered local evidence. This makes the annotation results dependent not only on the recognition accuracy of a single modality but also on the consistency verification between multiple sources of information. Existing technologies typically use independent modeling or local rule verification to correct multimodal annotation results. However, due to the lack of a unified association modeling mechanism, it is difficult to incorporate object instances, text fragments, page areas, page information, and temporal evolution relationships into the same reasoning framework. This results in the ineffective fusion of cross-modal, cross-page, and cross-temporal association evidence, leading to inaccurate conflict localization and insufficient basis for error correction.

[0003] On the other hand, existing multimodal annotation and correction methods often involve extensive manual review or global re-reasoning after anomalies are detected. They fail to dynamically compress the review scope based on the correlation strength of conflicting nodes and local evidence chains, leading to problems such as excessive review targets, high manual costs, and low processing efficiency. Especially in scenarios with multi-level referencing, cross-frame tracking, cross-page coreference, and intertwined rule constraints, the correction of a single node often affects its surrounding related nodes. Without effective control over the propagation path, scope, and direction of the correction results, the results may spread without boundaries among unrelated nodes, causing the originally localized correction operation to trigger new annotation biases and secondary errors, further reducing the reliability of the correction.

[0004] Furthermore, existing technologies typically lack a hierarchical retention mechanism for initial annotation results and subsequent correction results. This makes it easy for valid information to be overwritten during the propagation and update process, or for local errors to spread to the global annotation space. This makes it difficult for the system to maintain the stability and controllability of the error correction process while utilizing multi-source correlation information to improve consistency. Therefore, in multimodal annotation scenarios that span multiple modes and time periods, there is an urgent need for an error correction method that can perform unified correlation modeling of multi-source annotation information, accurately compress the scope of manual review based on conflict evidence, and implement restricted propagation control of correction results. This would reduce annotation error correction costs, improve error correction efficiency, and avoid the generation of secondary errors. Summary of the Invention

[0005] The embodiments of the present invention provide a method, apparatus, device and medium for local conflict graph error correction, which aims to solve the technical problems of high cost, low efficiency and easy to cause secondary errors in the existing technology in multimodal annotation scenarios across modalities and time and space.

[0006] In a first aspect, embodiments of the present invention provide a method for correcting local conflict graphs. The method includes: acquiring multimodal input data and performing initial automatic annotation to obtain candidate annotation results; performing unified node standardization processing on the candidate annotation results to construct a heterogeneous consistency graph containing multimodal object nodes and related edges; calculating the weights of the related edges based on their types and characteristics, and performing constrained label propagation while retaining the initial annotation information to obtain a node updated label distribution; calculating a node conflict score based on the difference between the node updated label distribution and the neighborhood reference distribution, as well as preset business constraints, and filtering conflicting nodes based on the node conflict score; dynamically extracting local conflict subgraphs around the conflicting nodes based on the weights of the related edges and the importance of the nodes; performing review processing on the local conflict subgraphs to obtain a corrected label distribution and an allowed propagation mask; performing a restricted backpropagation update on the corrected label distribution to the related nodes in the local conflict subgraphs based on the allowed propagation mask and path credibility; and outputting the final annotation result when the restricted backpropagation update is completed and a preset convergence condition is met.

[0007] Secondly, embodiments of the present invention also provide a local conflict graph correction device for performing the local conflict graph correction method described above.

[0008] Thirdly, embodiments of the present invention also provide a computer device, the computer device including a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program stored in the memory to perform the steps of the above-described local conflict graph error correction method.

[0009] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, can implement the steps of the aforementioned local conflict graph error correction method.

[0010] Compared with the prior art, the beneficial effects of the present invention are: In the technical solution of this invention, the local conflict graph error correction method performs initial automatic annotation on multimodal data and standardizes candidate results to construct a heterogeneous consistency graph. Then, it calculates the associated edge weights and performs constrained label propagation while retaining initial information to obtain an updated distribution. Next, it calculates a conflict score based on distribution differences and business constraints to filter conflict nodes and dynamically extracts local conflict subgraphs accordingly. After reviewing the subgraphs to obtain the corrected distribution and allowed propagation mask, it performs restricted backpropagation updates based on the mask and path credibility until the convergence condition is met and the final result is output. This method significantly reduces the annotation and error correction cost and interaction burden, and greatly improves the error correction efficiency, accuracy, and global consistency in cross-modal scenarios. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart of the local conflict graph error correction method provided by the present invention; Figure 2 A first sub-flowchart of the local conflict graph error correction method provided by the present invention; Figure 3 This is a second sub-flowchart of the local conflict graph error correction method provided by the present invention; Figure 4 This is the third sub-flowchart of the local conflict graph error correction method provided by the present invention; Figure 5 The fourth sub-flowchart of the local conflict graph error correction method provided by the present invention; Figure 6 This is the fifth sub-flowchart of the local conflict graph error correction method provided by the present invention; Figure 7 The sixth sub-flowchart of the local conflict graph error correction method provided by the present invention; Figure 8 The seventh sub-flowchart of the local conflict graph error correction method provided by the present invention; Figure 9 The eighth sub-flowchart of the local conflict graph error correction method provided by the present invention; Figure 10 A schematic block diagram of a unit of the local conflict diagram correction device provided by the present invention; Figure 11 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0015] It should also be understood that the terminology used in this specification is for the purpose of describing embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0016] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0017] To address the technical problems of high cost, low efficiency, and susceptibility to secondary errors in existing multimodal annotation and correction methods across modal and spatiotemporal scenarios, this invention discloses a local conflict graph correction method. This method can be applied to video understanding, document understanding, intelligent quality inspection, business document review, and other scenarios requiring annotation and correction of multimodal data.

[0018] Reference Figures 1 to 9 The local conflict graph correction method includes the following steps: S110. Acquire multimodal input data and perform initial automatic annotation to obtain candidate annotation results; S120. Perform unified node standardization processing on the candidate annotation results to construct a heterogeneous consistency graph containing multimodal object nodes and association relationship edges; S130. Calculate the weight of the associated edge based on the type and characteristics of the associated edge, and perform constrained label propagation while retaining the initial labeling information to obtain the node updated label distribution; S140. Calculate the node conflict score based on the difference between the node update label distribution and the neighborhood reference distribution and the preset business constraints, and filter conflicting nodes based on the node conflict score. S150. Dynamically extract local conflict subgraphs around the conflict nodes based on the associated edge weights and node importance; S160. Perform a review process on the local conflict subgraph to obtain the corrected label distribution and allowed propagation mask; S170. Based on the allowed propagation mask and path credibility, the corrected label distribution is sent to the associated nodes in the local conflict subgraph for restricted backhaul update. S180. When the restricted backhaul update is completed and the preset convergence condition is met, output the final annotation result.

[0019] In the data input stage of the local conflict graph correction method of this invention, the multimodal input data may include one or more of the following: video frame data, static image data, multi-page document data, text data, OCR result data, and structured business record data. The local conflict graph correction method first acquires the multimodal input data and performs initial automatic annotation on the multimodal input data to generate candidate annotation results. Initial automatic annotation can be performed by an object detection model, text recognition model, layout parsing model, entity recognition model, or structured field extraction model. Candidate annotation results may include object instances, text fragments, layout regions, page nodes, time nodes, and structured fields, etc. The purpose of initial automatic annotation is to form basic candidate results so that consistency correction can be performed subsequently by combining cross-modal association relationships.

[0020] After obtaining the candidate annotation results, a unified node standardization process is performed. This process transforms candidate annotation objects with different representations across different modalities into a unified graph node representation, thereby constructing a heterogeneous consistency graph containing multimodal object nodes and relational edges. Nodes in the heterogeneous consistency graph can include one or more of the following: object nodes, attribute nodes, text nodes, region nodes, page nodes, time nodes, and constraint nodes. Edges can include one or more of the following: temporally continuous edges, tracking-corresponding edges, page-to-page corresponding edges, image-text aligned edges, entity coreference edges, and knowledge constraint edges. Through this unified graph construction method, annotation objects originally scattered across different modalities can be placed into the same relational framework, enabling subsequent reasoning to simultaneously utilize temporal, spatial, textual, and rule-based relationships.

[0021] After constructing the heterogeneous consistency graph, the weights of the associated edges need to be calculated based on their types and characteristics. Constrained label propagation is then performed while preserving the initial annotation information to obtain the node updated label distribution. Specifically, the weights of the associated edges can be calculated based on information such as node feature similarity, geometric or layout overlap, time decay, layout role matching, and rule satisfaction. Different edge types can be assigned different weights to characterize the degree of influence of that type of relationship on consistency propagation. Then, using the label distribution obtained from the initial annotation as anchor information, the neighborhood propagation results are fused with the initial annotation information to form an update process with anchored terms. To prevent erroneous labels from spreading in the graph, constraint violation costs can be introduced during the propagation process to suppress or relabel candidate labels that do not conform to business rules or knowledge rules, thereby obtaining the node updated label distribution. In this way, the node updated label distribution retains the basic information of the initial automatic annotation while incorporating neighborhood consistency evidence from the graph.

[0022] Next, a node conflict score is calculated based on the difference between the node's updated label distribution and the neighborhood reference distribution, as well as preset business constraints. Conflicting nodes are then selected based on this score. The neighborhood reference distribution can be understood as a reference probability distribution weighted by the label distributions of the current node's neighboring nodes and the associated edge weights, used to describe the consistency expectation of the current node in the local graph. If the updated label distribution of a node differs significantly from the neighborhood reference distribution, or if the annotation result corresponding to that node violates attribute mutual exclusion relationships, rule constraints, or cross-modal consistency requirements, then the conflict score of that node will increase. The conflict score can be comprehensively formed by distribution deviation terms, attribute mutual exclusion terms, rule violation terms, and cross-modal inconsistency terms. After selecting conflicting nodes through the conflict score, the system can limit the focus of review to nodes with a truly significant risk of inconsistency, without requiring manual inspection of all annotation results.

[0023] After identifying conflicting nodes, a local conflict subgraph is dynamically extracted around these nodes based on their associated edge weights and node importance. The extraction process starts with the conflicting nodes, prioritizing the expansion of associated nodes with higher edge weights and greater contributions to conflict interpretation. Simultaneously, node importance is considered to determine whether candidate expansion nodes should be included in the subgraph. During expansion, further expansion stops when the contribution of candidate expansion nodes falls below a preset threshold, the expansion layer reaches a preset upper limit, or a closed-loop evidence chain has been formed, resulting in the local conflict subgraph. The local conflict subgraph typically includes the conflicting node, adjacent nodes strongly associated with the conflicting node, intermediate nodes that can explain the source of the conflict, and relevant constraint nodes. This approach compresses what could otherwise be a large-scale data correction task into a limited review task within a closed loop of local evidence, significantly reducing the scope of manual review.

[0024] After obtaining the local conflict subgraph, a review process is performed on it to obtain the corrected label distribution and the allowed propagation mask. The review process can be performed manually or by a dedicated review model. Manual review is suitable for scenarios with complex rules, long evidence chains, or high business risks; dedicated review models are suitable for scenarios with strong rule codedability and strong pattern stability. The output of the review process includes not only the corrected labels for the conflicting nodes but also the corresponding corrected label distribution, i.e., the probability or confidence distribution of reassigning different candidate labels. The allowed propagation mask is used to limit which nodes the corrected result can subsequently propagate to, along which edge types, in which direction, and at which level. The introduction of the allowed propagation mask prevents the corrected result obtained from the review from being diffused without boundaries to irrelevant nodes.

[0025] Subsequently, based on the allowed propagation mask and path confidence, a restricted backpropagation update is performed on the corrected label distribution to the associated nodes in the local conflict subgraph. Path confidence measures the reliability of the correction result as it propagates along the path in the graph, and is typically related to edge weight, path length, edge type, and the degree of consistency between nodes. During the backpropagation update, not all associated nodes participate in the update. Instead, propagable paths are first selected based on the allowed propagation mask, and then the backpropagation gain is calculated in conjunction with the path confidence. Only when the backpropagation gain meets the backpropagation condition is the label distribution and node confidence of the target node updated using a convex combination of the corrected label distribution and the original label distribution. This ensures that the correction result is propagated only within a reliable and controlled range, achieving local consistency correction while preventing the correction result from spreading unbounded to irrelevant nodes.

[0026] After completing the restricted feedback update, the system determines whether preset convergence conditions are met. Convergence conditions may include: all nodes in the local conflict subgraph having conflict scores no higher than the conflict threshold; the change in node label distribution within the local conflict subgraph not exceeding the convergence threshold; and the number of nodes awaiting review within the local conflict subgraph decreasing to below a preset number. When the preset convergence conditions are met, the final annotation results are output, and the error correction results, conflict evidence, and propagation trajectory can be written to the result storage area for subsequent auditing, backtracking, and rule updates. By recording conflict evidence and propagation trajectories, the system can further identify high-frequency conflict patterns, providing a basis for subsequent multimodal annotation model optimization and business rule correction.

[0027] From an implementation perspective, this local conflict graph error correction method first maps multimodal candidate labels into a heterogeneous consistency graph, then achieves global consistency pre-inference through constrained label propagation, subsequently locates abnormal nodes through conflict scoring, dynamically extracts local conflict subgraphs around these abnormal nodes, and finally achieves accurate error correction through review and constrained backpropagation updates. Compared with traditional direct global review or unconstrained propagation methods, this scheme can more effectively integrate cross-modal and cross-temporal correlation information, narrow the scope of manual review, suppress the spread of erroneous labels, improve the efficiency and consistency of label correction, and reduce the probability of secondary errors.

[0028] In one embodiment, step S110 includes: S111, Receive input data including at least two modalities of video frames, still images, multi-page documents, text, or structured records; S112. Call the basic recognition model corresponding to each modality to perform initial automatic labeling on the input data to obtain the initial label and initial label distribution for each modality; S113. The annotation results of each modality are uniformly represented as node tuples. The node tuples include at least node type, initial label, initial label distribution, spatial or page position, timestamp or page number order, source identifier and node confidence. S114. Summarize the node tuples of all modalities to obtain the candidate annotation results.

[0029] First, the system receives input data in at least two modalities, including video frames, still images, multi-page documents, text, or structured records. Specifically, the system's data access module can collect multimodal input data from video streams, image files, document files, optical character recognition results, and business databases to meet the data access needs of different business scenarios such as industrial inspection, invoice review, contract understanding, and intelligent quality inspection.

[0030] Upon receiving input data, the system invokes the basic recognition model corresponding to each modality to perform initial automatic labeling of the input data, obtaining the initial labels and initial label distribution for each modality. Specifically, for input data in video frame or static image modalities, the system invokes the object detection model or image segmentation model to obtain detection boxes, segmented regions, category labels, and attribute labels; for input data in multi-page document modalities, the system invokes the layout analysis model and information extraction model to obtain the layout area, field extraction results, page numbers, and inter-page references; for input data in text or optical character recognition result modalities, the system invokes the named entity recognition model to obtain named entities, attribute words, quantifiers, position words, and status words; for input data in structured record modalities, the system obtains field values, primary keys, timestamps, and business constraints through a parsing interface.

[0031] The basic recognition model outputs both initial labels and initial label distributions. These initial label distributions represent the predicted probability distributions of each candidate category for a node. After obtaining the initial labels and initial label distributions for each modality, the system uniformly represents the annotation results for each modality as node tuples.

[0032] A node tuple includes at least the node type, initial label, initial label distribution, spatial or page location, timestamp or page number order, source identifier, and node confidence score. Specifically, the node type distinguishes object nodes, attribute nodes, text nodes, region nodes, page nodes, time nodes, and constraint nodes; the spatial or page location records the node's bounding box coordinates or page location information; the timestamp or page number order records the node's time information or page sequence information; and the source identifier records the node's data source or the identifier of the underlying recognition model used.

[0033] Node confidence is defined as the maximum probability in the initial label distribution. The formula for calculating node confidence is: in, Represents a set of label categories. This indicates the category of the node corresponding to the node tuple. The initial probability. Node confidence is used to measure the reliability of the initial automatic annotation and to provide a benchmark for subsequent screening of conflicting nodes and limited backhaul updates.

[0034] Finally, the system aggregates node tuples from all modalities to obtain candidate annotation results. The system gathers all node tuples from different modalities and different basic recognition models together to form a unified candidate node set. The node tuples in the candidate annotation results will serve as the complete set of nodes for subsequent construction of a heterogeneous consistency graph.

[0035] In one embodiment, step S120 includes: S121. Map each modal object in the candidate annotation results to a unified node pattern. The node type includes at least two of the following: object node, attribute node, text node, region node, page node, time node, and constraint node. S122. Extract a unified feature vector for each node, and use a unified coding function to map the nodes with different modalities and different structures to the same vector space. S123. Generate multiple types of edges based on the association relationships between the nodes. The types of edges include at least two of the following: temporally continuous edges, tracking corresponding edges, page-to-page corresponding edges, image-text aligned edges, entity coreference edges, and knowledge constraint edges. S124. Construct the heterogeneous consistency graph using the nodes and the edges.

[0036] First, the system maps each modal object in the candidate annotation results to a unified node pattern. The node types in the unified node pattern include at least two of the following: object nodes, attribute nodes, text nodes, region nodes, page nodes, time nodes, and constraint nodes. In practice, the system retains the corresponding core attributes for the recognition results of different modalities to complete the mapping. For example, for text nodes, it retains the word literal, normalized lexical, page number, sentence index, and semantic role; for region nodes, it retains the bounding box, page number, or frame number; for object nodes, it retains the trajectory identifier, category, attribute, and spatiotemporal location; and for constraint nodes, it retains the business rule identifier and constraint template.

[0037] By using a unified node pattern, the system eliminates the differences in underlying data structure between different modalities. Subsequently, the system extracts a unified feature vector from each node. This unified feature vector, through a unified encoding function, maps nodes of different modalities and structures to the same vector space. The process of extracting the unified feature vector can be represented by the following formula: in, Represents a node The unified feature vector, This represents a unified encoding function used to transform heterogeneous node attributes into dense vector representations of the same dimension.

[0038] In practical applications, a unified coding function can be used to employ a multimodal pre-trained model or a feature fusion network, which makes nodes with similar semantics but different modalities closer together in the vector space, thus providing a unified metric basis for subsequent calculation of feature similarity between nodes.

[0039] After extracting a unified feature vector, the system generates various types of edges based on the relationships between nodes. These edges include at least two of the following types: temporally continuous edges, tracking-corresponding edges, layout-corresponding edges, image-text aligned edges, entity coreference edges, and knowledge-constrained edges. Specifically, temporally continuous edges connect candidate nodes with the same trajectory whose time difference does not exceed a preset time threshold, ensuring the continuity of object states in video or time-series data. Tracking-corresponding edges connect cross-frame object nodes identified by the same tracker, associating observations of the same physical target at different times. Layout-corresponding edges connect region nodes with preset layout roles to field nodes, such as connecting header areas, signature areas, and corresponding text fields. Image-text aligned edges connect attribute descriptors in text to candidate attribute nodes in image areas, establishing cross-modal semantic alignment. Entity coreference edges connect nodes pointing to the same entity across different pages, sentences, or records, resolving cross-page or cross-paragraph referencing issues. Knowledge-constrained edges connect a group of nodes bound by the same business rule, explicitly introducing prior business logic into the graph structure.

[0040] During the edge generation process, the system generates the corresponding edge only when the node pair meets the pre-screening conditions; otherwise, no connection is established. This effectively controls the size of the heterogeneous consistency graph and improves the efficiency of subsequent operations.

[0041] In a preferred embodiment, the preset condition for generating corresponding edges only when the pre-screening condition is met, and not establishing connections otherwise, includes: the time difference between the connection of temporally consecutive edges does not exceed a threshold. Candidate nodes with the same trajectory; tracking corresponding edges connect cross-frame object nodes identified by the same tracker; page corresponding edges connect area nodes and field nodes with fixed page roles, such as header, signature area, signature area, and amount area; image-text alignment edges connect position words, color words, and quantity words in text with candidate attributes of image areas; entity co-reference edges connect nodes pointing to the same entity in different pages, different sentences, or different records; knowledge constraint edges connect a group of nodes constrained by the same business rule.

[0042] Finally, the system constructs a heterogeneous consistency graph from the mapped nodes and generated edges, denoted as: Where V represents the set of nodes and E represents the set of edges.

[0043] In a preferred embodiment, the node set is represented as: Each node It can be represented as: in, Indicates the node type. Indicates the initial label. Indicates the initial label distribution. Indicates spatial location or page layout. Indicates timestamp or page number order. Indicates the source identifier. This represents the node confidence level.

[0044] The node types include at least object nodes, attribute nodes, text nodes, region nodes, page nodes, time nodes, and constraint nodes.

[0045] Further construct the edge set: in, It represents a set of edge types, including at least temporally continuous edges, tracking corresponding edges, page-to-page corresponding edges, image-text aligned edges, entity coreference edges, and knowledge constraint edges.

[0046] In one embodiment, step S130 includes: S131. For the associated edge, calculate at least two of the following: feature similarity, spatial or layout overlap, time decay, layout role matching degree, and rule satisfaction degree, and obtain the associated edge weight after weighting and normalization. S132. When performing label propagation, an anchoring coefficient is set, and the initial label distribution is weighted and fused with the neighborhood propagation result through the anchoring coefficient to obtain an intermediate label distribution; S133. Based on the preset constraint violation cost, the intermediate label distribution is recalibrated and normalized to suppress label categories that violate business rules. After iterating until the distribution is stable, the node updated label distribution is obtained.

[0047] For related edges, the system calculates at least two of the following: feature similarity, spatial or page overlap, time decay, page role matching, and rule satisfaction. These are then weighted and normalized to obtain the edge weight. Specifically, feature similarity measures the proximity of nodes' unified feature vectors; spatial or page overlap measures the degree of overlap between nodes in space or on a page, preferably calculated using the intersection-union ratio (IUU); time decay measures the proximity of nodes in time or page number order, preferably calculated using an exponential decay function; page role matching measures whether a node corresponds to a compatible page role; and rule satisfaction measures whether nodes satisfy the same business rule. The system then calculates a weighted score for each of these factors to obtain the basic association score between node pairs. The formula for calculating the basic association score is as follows: in, Represents the feature similarity function. A function representing spatial or page overlap. Represents the time decay function. This represents the function for matching page characters. The function representing the rule satisfaction degree, , , , and edge type The non-negative weighting coefficients of each item.

[0048] Subsequently, the system normalizes the basic association scores to obtain the association edge weights corresponding to each type of edge. The formula for calculating the association edge weights is as follows: in, Represents a node In edge type The set of adjacent nodes is normalized so that the sum of the weights of the associated edges of the same type for any node is one, thus ensuring the probabilistic nature of the label distribution during subsequent label propagation.

[0049] After calculating the associated edge weights, the system sets an anchoring coefficient during label propagation. This coefficient is used to weight and fuse the initial label distribution with the neighborhood propagation results, yielding an intermediate label distribution. The anchoring coefficient ranges from zero to one, balancing the weight of a node's initial label distribution with the neighborhood propagation results; a larger anchoring coefficient tends to retain the initial model output. In each propagation iteration, the system calculates the intermediate label distribution using the following formula: in, Indicates the anchoring coefficient. Representing edge type The propagation weight, Indicates the associated edge weight, This represents the label distribution of neighboring nodes in round t-1, i.e., the result of neighborhood propagation. This represents the initial label distribution of the nodes.

[0050] Through the aforementioned weighted fusion, the system effectively prevents the initial annotation information from being completely overwritten during the propagation process while utilizing cross-modal and cross-temporal correlation evidence.

[0051] After obtaining the intermediate label distribution, the system recalibrates and normalizes it based on a preset constraint violation cost, suppressing label categories that violate business rules. This process iterates until the distribution stabilizes, resulting in the updated label distribution for each node. In practice, the system introduces constraint strength parameters and a preset constraint violation cost to recalibrate and normalize the intermediate label distribution. The formula for calculating the updated label distribution is as follows: in, Represents the constraint strength parameter. Represents a node Take category The cost of violating the preset constraints at that time This represents the set of label categories. The preset constraint violation cost is used to measure the degree to which a node violates preset business rules when choosing a specific category. The constraint strength parameter is used to adjust the impact of the preset constraint violation cost on the propagation result. Through exponential recalibration, the system can effectively suppress label categories that violate business rules and renormalize them into a probability distribution by summing the denominators.

[0052] The system continuously executes the above label propagation iterations, stopping the iteration when the maximum number of propagation rounds is reached or the total change in the label distribution across the entire image falls below the propagation stopping threshold. The total change in the label distribution across the entire image is defined as follows: After the iteration stops, the system uses the final updated label distribution as the node updated label distribution for subsequent conflict scoring.

[0053] In one embodiment, step S140 includes: S141. Using the associated edge weights as weights, aggregate the update label distributions of adjacent nodes to generate a neighborhood consistency reference distribution. S142. Calculate the distribution deviation between the node update label distribution and the neighborhood consistency reference distribution, and perform weighted fusion by combining attribute mutual exclusion, rule violation and cross-modal inconsistency to obtain the node conflict score. S143. The node conflict score is compared with a preset conflict threshold, and the node with the score exceeding the preset conflict threshold is identified as the conflict node.

[0054] The system aggregates the updated label distributions of adjacent nodes using the weights of associated edges to generate a neighborhood-consistent reference distribution. In practice, this neighborhood-consistent reference distribution reflects the label consensus of the surrounding context of each node. The system calculates the multimodal object node using the following formula. Neighborhood consistency reference distribution: in, Represents a multimodal object node The neighborhood consistent reference distribution Indicates the type of the edge representing the association relationship. Represents a multimodal object node In edge type The set of adjacent nodes below, Represents a multimodal object node With adjacent nodes Between edge types The associated edge weights below, Indicates adjacent nodes The system updates the label distribution of nodes. Through the above aggregation operation, the system can integrate the label prediction results of adjacent nodes under different edge types, providing a contextual reference benchmark for evaluating the rationality of the labels of multimodal object nodes themselves.

[0055] After generating the neighborhood consistency reference distribution, the system calculates the distribution deviation between the node update label distribution and the neighborhood consistency reference distribution, and then performs a weighted fusion based on attribute mutual exclusivity, rule violation, and cross-modal inconsistency to obtain a node conflict score. In practice, the system first calculates the distribution deviation using the Kullback-Leibler divergence metric. Used to measure the distribution of node update labels for multimodal object nodes. Neighborhood Consistent Reference Distribution The greater the difference in distribution deviation, the more likely the multimodal object node is to conflict with the context.

[0056] At the same time, the system calculates the mutual exclusion of attributes, which is defined as: in, Represents a set of mutually exclusive property pairs. Represents a multimodal object node The set of predictable attributes, As an indicator function, attribute mutual exclusion is used to measure whether a multimodal object node simultaneously predicts mutually exclusive attributes that cannot coexist in business logic.

[0057] The system also calculates the rule violation degree, which is defined as: in, Represents multimodal object nodes The set of business rules associated by knowledge-constrained edges For business rules The non-negative weights and rule violation rates are used to measure the degree to which multimodal object nodes violate preset business constraints.

[0058] In addition, the system calculates the cross-modal inconsistency degree, which is defined as: in, Represents multimodal object nodes A set of cross-modal related nodes that are associated by text-image alignment edges or entity core-referenced edges and belong to different modalities. In order to integrate border rights, The feature similarity function is used to measure the degree of semantic inconsistency between multimodal object nodes and cross-modal associated nodes.

[0059] Finally, the system uses the following formula to weight and fuse the above items to obtain the node conflict score: in, to These are non-negative weighting coefficients used to adjust the relative contributions of conflicting factors in different business scenarios.

[0060] After obtaining the node conflict score, the system compares the node conflict score with a preset conflict threshold, and identifies nodes with scores exceeding the preset conflict threshold as conflicting nodes. In practice, the system traverses all multimodal object nodes in the heterogeneous consistency graph, determining whether the node conflict score of each multimodal object node meets the judgment criteria. ,in This indicates the preset conflict threshold.

[0061] When the node conflict score of a multimodal object node exceeds a preset conflict threshold, the system determines that the multimodal object node has a significant consistency conflict or logical error, and marks the multimodal object node as a conflict node, thereby aggregating all conflict nodes into a conflict node set. By setting a preset conflict threshold, the system can effectively filter out minor deviations with low node conflict scores, accurately focusing subsequent review resources on high-risk nodes that are indeed suspected of errors.

[0062] In one embodiment, step S150 includes: S151. Calculate the expansion contribution of the candidate adjacent node relative to the conflicting node, wherein the expansion contribution is determined by the product of the associated edge weight and the node importance. S152. Starting from the conflicting node, the adjacent nodes are included in the subgraph range in descending order of the expansion contribution. S153. When any of the following conditions are met: the current expansion contribution is lower than the preset expansion threshold, the number of expansion layers reaches the preset upper limit, or a closed-loop evidence chain has been formed, the expansion is stopped and the local conflict subgraph is output.

[0063] The system calculates the expansion contribution of candidate adjacent nodes relative to conflicting nodes. The expansion contribution is determined by the product of the associated edge weight and the node's importance. In practice, the system calculates candidate adjacent nodes using the following formula. Relative to conflict nodes Extended contribution: in, Indicates candidate adjacent nodes Relative to conflict nodes The expanded contribution, Indicates conflicting nodes With candidate adjacent nodes The associated edge weights between them Indicates candidate adjacent nodes The importance of nodes.

[0064] The node importance value is set to a range of zero to one, calculated jointly by node type weight, source reliability, and citation count. This value is used to prioritize the inclusion of more critical evidence nodes during expansion. Through the above product calculation, the system can comprehensively consider the correlation strength between candidate adjacent nodes and conflicting nodes, as well as the evidentiary value of the candidate adjacent nodes themselves in the entire graph.

[0065] After calculating the expansion contribution, the system starts from the conflicting node and sequentially includes adjacent nodes in the subgraph according to their expansion contribution from highest to lowest. In practice, the system employs a greedy expansion strategy. At each expansion level, it prioritizes selecting the candidate adjacent node with the highest expansion contribution and includes it in the local conflict subgraph. The adjacent nodes of the newly included node are then used as candidate adjacent nodes for the next level, and their expansion contributions are calculated. This method of sorting and including nodes sequentially according to their expansion contribution ensures that the nodes most closely related to the conflicting node and with the highest evidentiary value are extracted first, thereby quickly constructing the core evidence network supporting the conflict determination.

[0066] During the process of sequentially incorporating adjacent nodes into the subgraph, the system stops expanding and outputs a local conflict subgraph when any of the following conditions are met: the current expansion contribution is lower than a preset expansion threshold, the expansion layer reaches a preset upper limit, or a closed-loop evidence chain has been formed. Specifically, when the expansion contribution of the current candidate adjacent node is lower than the preset expansion threshold, it indicates that the subsequent candidate node's association with the conflict node is too weak or its evidentiary value is too low, and further expansion has minimal marginal contribution to error correction. In this case, expansion is stopped to filter weakly associated nodes. When the expansion layer reaches the preset upper limit, it indicates that the number of hops in the subgraph has reached the maximum depth allowed by the system. In this case, expansion is stopped to strictly control the size of the local conflict subgraph and prevent the scope of manual review from expanding indefinitely. When a closed-loop evidence chain has been formed, it indicates that the current subgraph already contains a complete evidence path sufficient to confirm or disprove the conflict determination, and there is no need to continue expanding outward. After stopping expansion when any of the above conditions are met, the system outputs a local conflict subgraph composed of nodes and edges obtained from the expansion. The local conflict subgraph contains at least conflicting nodes, directly influencing nodes, source nodes, and constraint nodes.

[0067] In one embodiment, step S160 includes: S161. Send the local conflict subgraph to the manual review terminal or the dedicated review model, and only display the local nodes, local edges and source evidence related to the conflict; S162. Receive the corrected labels and the corrected probability distribution returned by the review operation, and generate the corrected label distribution; S163. Based on the correction information confirmed by the review, generate the allowed propagation mask, which is used to limit the correction result to be transmitted only along a specified edge type, a specified propagation direction, or a specified hop count range.

[0068] The system sends the local conflict subgraph to the manual reviewer or a dedicated review model, displaying only the local nodes, edges, and source evidence related to the conflict. In practice, the system extracts the topology and node attribute information within the local conflict subgraph and presents it to reviewers or inputs it into the dedicated review model through a visual interface or structured data interface. During the display, the system only highlights or renders the local nodes directly related to the conflict, the local edges connecting these nodes, and the source evidence supporting the initial annotation and conflict determination, while deliberately hiding other nodes and edges in the heterogeneous consistency graph that are unrelated to the current conflict. Local nodes include conflict nodes, directly influencing nodes, source nodes, and constraint nodes; source evidence includes original image slices, text fragments, layout areas, or business records. This locally focused display strategy significantly reduces the cognitive load on reviewers, enabling them to quickly locate the root cause of errors and thus improve review efficiency.

[0069] In a preferred embodiment, the review output result can be expressed as: in, Indicates a correction label. This indicates the corrected label distribution. This indicates a propagation mask. The propagation mask is used to limit the propagation of correction information to a specified edge type, a specified direction, and a specified number of layers.

[0070] After displaying the local conflict subgraph, the system receives the corrected labels and corrected probability distribution returned by the review operation, and generates a corrected label distribution. In practice, human reviewers or a dedicated review model analyze the initial labeling of conflicting nodes based on the displayed local nodes, local edges, and source evidence, providing correct category determinations and corresponding confidence assessments. The system receives the corrected labels and corresponding corrected probability distribution returned by the review operation and encapsulates them into a corrected label distribution. This corrected label distribution not only includes the determined corrected labels but also the re-evaluation probabilities of each candidate category by the human reviewer or dedicated review model, providing a precise target distribution reference for subsequent restricted feedback updates.

[0071] While generating the distribution of correction labels, the system generates a propagation mask based on the correction information confirmed by the review. This mask restricts the propagation of correction results to only along specified edge types, propagation directions, or hop count ranges. In practice, reviewers or dedicated review models configure the propagation rules for correction information based on the specific nature of the current conflict and the business logic. For example, if the conflict originates from state transitions between video frames, reviewers can set the propagation mask to only allow correction results to be propagated along temporally continuous edges or tracking corresponding edges; if the conflict involves fields with unidirectional dependencies, such as summarizing details into totals, the propagation direction can be set to propagate unidirectionally from detail nodes to summary nodes; to prevent excessive diffusion of correction results in complex graph structures, a hop count range can also be set to limit the maximum propagation layer.

[0072] Based on the above configuration, the system generates an allowed propagation mask. At the underlying implementation level, this mask can be represented as a Boolean matrix or filtering vector associated with edge type, direction, and layer number. During subsequent backhaul, it serves as a hard constraint to filter out propagation paths that do not meet the requirements. Finally, the system encapsulates the corrected label, the corrected probability distribution (i.e., the corrected label distribution), and the allowed propagation mask together into a review output triple for use in subsequent restricted backhaul updates.

[0073] In one embodiment, step S170 includes: S171. Filter invalid propagation paths in the local conflict subgraph according to the allowed propagation mask, calculate the edge weight multiplication value of each path in the remaining allowed paths, and determine the maximum value as the backpropagation gain. S172. Compare the backhaul gain with a preset backhaul threshold, and only update the associated nodes whose backhaul gain is greater than the preset backhaul threshold. S173. Using the backhaul gain as the fusion weight, perform a convex combination calculation on the original label distribution and the modified label distribution of the target node to obtain the updated node label distribution, and update the node confidence level synchronously.

[0074] The system filters invalid propagation paths in the local conflict subgraph based on the allowable propagation mask, calculates the product of edge weights for each path among the remaining allowable paths, and determines the maximum value as the backpropagation gain. In practice, the allowable propagation mask is implemented at the lower level as filtering conditions associated with edge type, propagation direction, and propagation layer. The system traverses all paths in the local conflict subgraph and uses the allowable propagation mask to eliminate invalid propagation paths that do not conform to the specified edge type, specified propagation direction, or exceed the specified hop count range, thereby obtaining the set of remaining allowable paths.

[0075] For each allowed path in the set of allowed paths, the system calculates the product of the edge weights of all edges on the allowed path. Then, the system calculates the return gain using the following formula: in, Indicates from the modified node To the associated node The return gain Represents the set of allowed paths. This represents a permitted path in the set of permitted paths. Indicates that edges on the allowed path are allowed. Representing an edge In edge type The associated edge weights below, Indicates whether the propagation mask allows edge types. Indicator functions for propagation.

[0076] The system determines the maximum value among the combined edge weights of all allowed paths as the backhaul gain. The backhaul gain is used to measure the strongest reliable path for corrective information to reach the associated node.

[0077] After calculating the backhaul gain, the system compares the backhaul gain with a preset backhaul threshold and only updates associated nodes whose backhaul gain is greater than the preset backhaul threshold. In practice, the preset backhaul threshold is a preset positive real number used to filter out low-reliability paths. Since the backhaul gain is composed of the product of several associated edge weights with values ​​between zero and one, the longer the path or the weaker the edge weight, the smaller the backhaul gain.

[0078] When the backhaul gain of the associated node is less than or equal to the preset backhaul threshold, it indicates that the propagation path from the corrected node to the associated node is too unreliable. The system does not perform updates on the associated node, thereby preventing the correction results from spreading to weakly associated nodes. Only when the backhaul gain of the associated node is greater than the preset backhaul threshold will the system identify the associated node as the target node and perform subsequent tag distribution update operations on the target node.

[0079] After identifying the target nodes that need updating, the system uses the backhaul gain as the fusion weight to perform a convex combination calculation on the original and corrected label distributions of the target nodes, obtaining the updated node label distribution, and simultaneously updating the node confidence. In practice, the system performs the convex combination calculation using the following formula: in, This represents the updated node label distribution. This represents the backhaul gain, i.e., the fusion weight. This represents the original label distribution of the target node. This indicates the distribution of correction labels for the corrected nodes.

[0080] Since the return gain ranges from zero to one, the above calculation constitutes a strictly convex combination update. A larger return gain indicates a more reliable propagation path, and the system is more inclined to use a modified label distribution during the update; a smaller return gain tends to retain the original label distribution. After completing the update of the node label distribution, the system synchronously updates the node confidence level using the following formula: in, This represents the updated node confidence level. This represents the maximum probability value in the updated node label distribution. Through the above mechanism, manual or model corrections will not spread without boundaries, but will only be stably propagated back along a high-confidence path.

[0081] In one embodiment, step S180 includes: S181. After the restricted backhaul update is completed, recalculate the conflict score and label distribution change of each node in the local conflict subgraph. S182. Determine whether the preset convergence condition is met. The preset convergence condition includes: the conflict scores of all nodes in the subgraph are not higher than the preset conflict threshold, the total change in the node label distribution is lower than the preset convergence threshold, or the number of nodes to be reviewed is reduced to below the preset number. S183. If the preset convergence condition is not met, return to re-execute the dynamic extraction of local conflict subgraphs; S184. If the preset convergence condition is met, the final annotation result is output, and the error correction record, conflict evidence, and propagation trajectory are written into the result storage area.

[0082] After the restricted backhaul update is completed, the system recalculates the conflict score and label distribution change of each node within the local conflict subgraph. In practice, because the restricted backhaul update alters the label distribution of some nodes within the local conflict subgraph, the system needs to reassess the conflict state within the subgraph. The system recalculates the conflict score of each node within the local conflict subgraph using the aforementioned conflict score calculation method. Simultaneously, the system calculates the sum of the changes in node label distribution within the local conflict subgraph. The formula for calculating the sum of the changes in node label distribution is: in, Represents a local conflict subgraph. This represents a node within a local conflict subgraph. This indicates the label distribution after the node's restricted backhaul update. This indicates the label distribution before the node's restricted backhaul update. express Norm. The sum of changes in node label distribution is used to quantify the degree of perturbation caused by this restricted backhaul update to the overall label distribution within the locally conflicting subgraph.

[0083] After recalculating the conflict scores and label distribution changes, the system determines whether the preset convergence conditions are met. The preset convergence conditions include any one of the following three scenarios: The first scenario is that the conflict scores of all nodes within the subgraph are not higher than a preset conflict threshold, i.e., the following conditions are met: This indicates that all substantive conflicts within the locally conflicting subgraph have been eliminated; the second scenario is when the sum of the changes in node label distribution is below a preset convergence threshold, i.e., satisfying: Where xi represents the preset convergence threshold, which indicates that the label distribution within the local conflict subgraph has become stable and the marginal benefit of continuing to backtrack and update is extremely low; the third case is that the number of nodes to be reviewed has dropped below the preset number, which indicates that after multiple rounds of error correction, the number of remaining conflict nodes that still require manual or model intervention has dropped to an acceptable range for the project.

[0084] If the system determines that any of the above preset convergence conditions are not met, it means that there are still unresolved conflicts or the label distribution is still changing significantly within the local conflict subgraph. The system then returns to re-execute the step of dynamically extracting the local conflict subgraph around the conflict node based on the associated edge weight and node importance, and sequentially executes the subsequent review process and restricted backhaul update, forming an iterative closed loop of local detection, review and backhaul.

[0085] If the system determines that the preset convergence conditions are met, it outputs the final annotation result and writes the error correction record, conflict evidence, and propagation trajectory into the result storage area. In practice, the system outputs the label distribution that has reached a stable state after multiple iterations as the final annotation result for use by downstream business systems. Simultaneously, the system uniformly writes the error correction record, conflict evidence, and propagation trajectory from this error correction process into the result storage area to achieve end-to-end traceability of the error correction process. The error correction record includes a comparison of labels before and after correction, and the identification of the reviewer or review model; the conflict evidence includes the values ​​of various indicators that trigger conflict scoring and the corresponding source data; and the propagation trajectory includes the actual propagation path of the corrected information, the return gain of each path, and the allowed propagation mask configuration.

[0086] Furthermore, the system performs statistical analysis on the conflict patterns recorded in the results storage area. For conflict patterns that occur more frequently than a preset frequency threshold, the system automatically generates new rule templates and adds them to the knowledge constraint node set of the heterogeneous consistency graph so that they can directly participate in conflict scoring and constrained label propagation in subsequent batches of multimodal data annotation, thereby realizing the self-evolution and continuous optimization of system rules.

[0087] The local conflict graph error correction method of the present invention can be widely applied to professional fields with complex cross-modal and cross-temporal associations, high requirements for annotation consistency, and sensitivity to the cost of manual review. The following uses two typical complex vision task scenarios, industrial inspection and autonomous driving, as examples for illustration.

[0088] In industrial inspection scenarios, this method can be used for joint annotation, error correction, and compliance review of factory equipment monitoring videos and multi-source business data. Without requiring full manual review, the system automatically extracts multimodal objects from hours of inspection videos, sensor logs, and maintenance work orders, generating the following conflict detection and correction process: First, a heterogeneous consistency graph is constructed and constrained label propagation is performed, fusing cross-frame temporal continuous edges and graph-text aligned edges to smooth the initial annotation distribution; then, based on a unified conflict score, cross-modal conflict nodes such as "the video shows the valve is open but the sensor pressure is abnormal" are accurately located, and local conflict subgraphs containing relevant video clips, sensor readings, and work order records are dynamically extracted; finally, the final label set after restricted backhaul updates is output using an allowed propagation mask. This process not only significantly reduces the data screening and review costs for quality inspectors but also provides a basis for consistency review of multimodal annotation results, contributing to equipment fault early warning and safety production audit traceability.

[0089] In autonomous driving applications, this method can be used for long-sequence annotation and error correction, as well as scene understanding verification, of multi-sensor fusion perception data. For example, in the joint annotation of continuous video streams and high-precision map data collected during autonomous driving road tests, the system automatically extracts multimodal object nodes from perception results that have not undergone global consistency verification. It captures the visual recognition results of traffic signs, vehicle tracking trajectories in continuous frames, and structured lane line information in high-precision maps, respectively. It calculates cross-modal inconsistency and rule violation degrees to filter conflict nodes, and then generates a local conflict sub-graph containing only conflicting vehicles, related road signs, and corresponding map elements. Annotators can not only obtain the minimum evidence loop directly related to the conflict for accurate error correction, but also view the controlled propagation trajectory and node confidence scores of the correction results, enhancing the quality and iteration efficiency of training data for autonomous driving perception models.

[0090] As can be seen from the above applications, the method of the present invention supports global consistent reasoning, local accurate verification and controlled backhaul updates in complex vision task scenarios such as industrial inspection and autonomous driving, which are closely related across modalities and have extremely high requirements for annotation consistency. It has good practical value and development prospects.

[0091] Figure 10 This is a schematic block diagram of a local conflict diagram correction device 600 provided in an embodiment of the present invention. Figure 10 As shown, corresponding to the above-described local conflict graph correction method, the present invention also provides a local conflict graph correction device 600. This local conflict graph correction device 600 includes a unit for performing the above-described local conflict graph correction method, and the device can be configured in a desktop computer, tablet computer, smartphone, or other terminal.

[0092] Specifically, please refer to Figure 10 The local conflict map correction device 600 includes: The data access and initial annotation unit 610 is used to acquire multimodal input data and perform initial automatic annotation to obtain candidate annotation results; The node standardization and graph construction unit 620 is used to perform unified node standardization processing on the candidate annotation results and construct a heterogeneous consistent graph containing multimodal object nodes and association relationship edges. The edge weight calculation and constraint propagation unit 630 is used to calculate the associated edge weights based on the type and characteristics of the associated edges, and perform constrained label propagation while retaining the initial label information to obtain the node updated label distribution; The conflict scoring and filtering unit 640 is used to calculate the node conflict score based on the difference between the node update label distribution and the neighborhood reference distribution and the preset business constraints, and to filter conflicting nodes based on the node conflict score. The subgraph dynamic extraction unit 650 is used to dynamically extract local conflict subgraphs around the conflict nodes based on the associated edge weights and node importance. The review processing unit 660 is used to perform review processing on the local conflict subgraph to obtain the corrected label distribution and the allowed propagation mask. The restricted backhaul update unit 670 is used to perform restricted backhaul update on the distribution of the corrected label to the associated nodes in the local conflict subgraph based on the allowed propagation mask and path confidence. The convergence determination and output unit 680 is used to output the final annotation result when the restricted backhaul update is completed and the preset convergence condition is met.

[0093] In one embodiment, the data access and initial annotation unit 610 includes: A multimodal data receiving unit is used to receive input data in at least two modalities, including video frames, still images, multi-page documents, text, or structured records. The initial automatic labeling unit is used to call the basic recognition model corresponding to each modality to perform initial automatic labeling on the input data, and obtain the initial label and initial label distribution corresponding to each modality; The node tuple mapping unit is used to uniformly represent the annotation results of each modality as node tuples. The node tuples include at least node type, initial label, initial label distribution, spatial or page position, timestamp or page number order, source identifier, and node confidence. The candidate result aggregation unit is used to aggregate the node tuples of all modalities to obtain the candidate annotation results.

[0094] In one embodiment, the node standardization and graph construction unit 620 includes: A unified node mapping unit is used to map each modal object in the candidate annotation results to a unified node pattern. The types of the nodes include at least two of the following: object nodes, attribute nodes, text nodes, region nodes, page nodes, time nodes, and constraint nodes. A unified feature extraction unit is used to extract a unified feature vector for each node. The unified feature vector maps the nodes with different modalities and different structures to the same vector space through a unified coding function. The associated edge generation unit is used to generate multiple types of edges based on the association relationships between the nodes. The types of edges include at least two of the following: temporally continuous edges, tracking corresponding edges, page-to-page corresponding edges, image-text aligned edges, entity coreference edges, and knowledge constraint edges. A heterogeneous consistency graph construction unit is used to construct the heterogeneous consistency graph from the nodes and the edges.

[0095] In one embodiment, the edge weight calculation and constraint propagation unit 630 includes: The associated edge weight calculation unit is used to calculate at least two of the following for the associated edge: feature similarity, spatial or page overlap, time decay, page role matching degree and rule satisfaction degree, and obtain the associated edge weight after weighting and normalization. Anchoring fusion unit is used to set anchoring coefficients when performing tag propagation, and to perform weighted fusion of the initial tag distribution and the neighborhood propagation results through the anchoring coefficients to obtain an intermediate tag distribution; The constraint recalibration unit is used to recalibrate and normalize the intermediate label distribution based on the preset constraint violation cost, suppress label categories that violate business rules, and obtain the node updated label distribution after iterating until the distribution is stable.

[0096] In one embodiment, the conflict scoring and screening unit 640 includes: The neighborhood reference distribution aggregation unit is used to aggregate the update label distribution of adjacent nodes using the associated edge weights as weights, and generate a neighborhood consistent reference distribution. The conflict score calculation unit is used to calculate the distribution deviation between the node update label distribution and the neighborhood consistency reference distribution, and to perform weighted fusion by combining attribute mutual exclusion, rule violation and cross-modal inconsistency to obtain the node conflict score. The conflict node determination unit is used to compare the node conflict score with a preset conflict threshold, and determine the node whose score exceeds the preset conflict threshold as the conflict node.

[0097] In one embodiment, the subgraph dynamic extraction unit 650 includes: An extended contribution calculation unit is used to calculate the extended contribution of a candidate adjacent node relative to the conflicting node. The extended contribution is determined by the product of the associated edge weight and the node importance. The priority expansion inclusion unit is used to include adjacent nodes into the subgraph range in descending order of expansion contribution, starting from the conflicting node. The extension termination determination unit is used to stop the extension and output the local conflict subgraph when any of the following conditions are met: the current extension contribution is lower than the preset extension threshold, the extension layer reaches the preset upper limit, or a closed-loop evidence chain has been formed.

[0098] In one embodiment, the review processing unit 660 includes: The subgraph targeted sending unit is used to send the local conflict subgraph to the manual review terminal or the dedicated review model, and only displays the local nodes, local edges and source evidence related to the conflict; The modified distribution generation unit is used to receive the modified labels and the modified probability distribution returned by the review operation, and generate the modified label distribution. The propagation mask configuration unit is used to generate the allowed propagation mask based on the correction information confirmed by the review. The allowed propagation mask is used to limit the correction result to be transmitted only along a specified edge type, a specified propagation direction, or a specified hop count range.

[0099] In one embodiment, the restricted postback update unit 670 includes: The path filtering and gain calculation unit is used to filter invalid propagation paths in the local conflict subgraph according to the allowed propagation mask, calculate the edge weight multiplication value of each path in the remaining allowed paths, and determine the maximum value as the backpropagation gain. The backhaul threshold comparison unit is used to compare the backhaul gain with a preset backhaul threshold, and only update the associated nodes whose backhaul gain is greater than the preset backhaul threshold. The convex combination update unit is used to perform convex combination calculation on the original label distribution and the modified label distribution of the target node using the backhaul gain as the fusion weight, to obtain the updated node label distribution, and to update the node confidence level synchronously.

[0100] In one embodiment, the convergence determination and output unit 680 includes: The distribution change recalculation unit is used to recalculate the conflict score and label distribution change of each node in the local conflict subgraph after the restricted backhaul update is completed. The convergence condition determination unit is used to determine whether the preset convergence condition is met. The preset convergence condition includes: the conflict scores of all nodes in the subgraph are not higher than the preset conflict threshold, the total change in node label distribution is lower than the preset convergence threshold, or the number of nodes to be reviewed is reduced to below the preset number. An iterative loop control unit is used to return and re-execute the dynamic extraction of local conflict subgraphs if the preset convergence condition is not met. The result output and archiving unit is used to output the final annotation result if the preset convergence condition is met, and to write the error correction record, conflict evidence and propagation trajectory into the result storage area.

[0101] The aforementioned local conflict graph correction device 600 can be implemented as a computer program, which can, for example... Figure 11 It runs on the computer device shown.

[0102] Please see Figure 11 , Figure 11This is a schematic block diagram of a computer device 500 provided in an embodiment of this application. The computer device 500 can be a terminal or a server. The terminal can be an electronic device with communication functions, such as a desktop computer, tablet computer, or smartphone. The server can be a standalone server or a server cluster composed of multiple servers.

[0103] See Figure 11 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0104] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a local conflict graph error correction method.

[0105] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0106] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a local conflict graph error correction method.

[0107] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0108] The processor 502 is used to run a computer program 5032 stored in a memory to implement the steps of the above method.

[0109] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0110] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0111] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When executed by a processor, the program instructions cause the processor to perform the steps of the above-described method.

[0112] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0113] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0114] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0115] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0116] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0117] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0118] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for correcting local conflict graphs, characterized in that, The method includes: Acquire multimodal input data and perform initial automatic annotation to obtain candidate annotation results; The candidate annotation results are subjected to unified node standardization processing to construct a heterogeneous consistency graph containing multimodal object nodes and associated relationship edges; The weights of the associated edges are calculated based on their types and characteristics, and constrained label propagation is performed while preserving the initial labeling information to obtain the node updated label distribution. Node conflict scores are calculated based on the differences between the node update label distribution and the neighborhood reference distribution, as well as preset business constraints, and conflicting nodes are filtered based on the node conflict scores. Around the conflict nodes, a local conflict subgraph is dynamically extracted based on the associated edge weights and node importance; The local conflict subgraph is reviewed to obtain the corrected label distribution and the allowed propagation mask; Based on the allowed propagation mask and path confidence, the corrected label distribution is sent to the associated nodes in the local conflict subgraph for restricted backhaul update; Once the restricted backhaul update is completed and the preset convergence conditions are met, the final annotation result is output.

2. The local conflict graph error correction method according to claim 1, characterized in that, The steps of acquiring multimodal input data and performing initial automatic annotation to obtain candidate annotation results include: Receive input data in at least two modalities, including video frames, still images, multi-page documents, text, or structured records; The basic recognition model corresponding to each modality is invoked to perform initial automatic labeling on the input data to obtain the initial label and initial label distribution corresponding to each modality; The annotation results of each modality are uniformly represented as node tuples. Each node tuple includes at least node type, initial label, initial label distribution, spatial or page position, timestamp or page number order, source identifier, and node confidence. The candidate annotation results are obtained by summing the node tuples of all modalities.

3. The local conflict graph error correction method according to claim 1, characterized in that, The step of performing unified node standardization processing on the candidate annotation results and constructing a heterogeneous consistent graph containing multimodal object nodes and association edges includes: Each modal object in the candidate annotation results is mapped to a unified node pattern, and the node type includes at least two of the following: object node, attribute node, text node, region node, page node, time node, and constraint node. A unified feature vector is extracted from each node, and the unified feature vector maps the nodes with different modalities and structures to the same vector space through a unified encoding function. Based on the association between the nodes, various types of edges are generated. The types of edges include at least two of the following: temporally continuous edges, tracking corresponding edges, page-to-page corresponding edges, image-text aligned edges, entity coreference edges, and knowledge constraint edges. The heterogeneous consistency graph is constructed from the nodes and the edges.

4. The local conflict graph error correction method according to claim 1, characterized in that, The steps of calculating the weights of the associated edges based on their types and characteristics, and performing constrained label propagation while preserving the initial labeling information, to obtain the node updated label distribution, include: For the aforementioned association edges, at least two of the following are calculated: feature similarity, spatial or layout overlap, time decay, layout role matching degree, and rule satisfaction degree. After weighting and normalization, the weight of the association edge is obtained. When performing label propagation, an anchoring coefficient is set, and the initial label distribution is weighted and fused with the neighborhood propagation result through the anchoring coefficient to obtain an intermediate label distribution; The intermediate label distribution is recalibrated and normalized based on the cost of violating preset constraints, and label categories that violate business rules are suppressed. The node updated label distribution is obtained after iterating until the distribution is stable.

5. The local conflict graph error correction method according to claim 1, characterized in that, The step of calculating a node conflict score based on the difference between the node update label distribution and the neighborhood reference distribution, and predefined business constraints, and then filtering conflicting nodes based on the node conflict score, includes: Using the associated edge weights as weights, the updated label distributions of adjacent nodes are aggregated to generate a neighborhood consistency reference distribution; The distribution deviation between the node update label distribution and the neighborhood consistency reference distribution is calculated, and the node conflict score is obtained by weighted fusion of attribute mutual exclusion, rule violation, and cross-modal inconsistency. The node conflict score is compared with a preset conflict threshold, and the node whose score exceeds the preset conflict threshold is identified as the conflict node.

6. The local conflict graph error correction method according to claim 1, characterized in that, The step of dynamically extracting a local conflict subgraph around the conflict node based on the associated edge weights and node importance includes: Calculate the expansion contribution of the candidate adjacent node relative to the conflicting node, whereby the expansion contribution is determined by the product of the associated edge weight and the node importance; Starting from the conflicting node, adjacent nodes are included in the subgraph in descending order of their expansion contribution. When any of the following conditions are met: the current contribution of the expansion is lower than the preset expansion threshold, the number of expansion layers reaches the preset upper limit, or a closed-loop evidence chain has been formed, the expansion stops and the local conflict subgraph is output.

7. The local conflict graph error correction method according to claim 1, characterized in that, The step of performing a review process on the local conflict subgraph to obtain the corrected label distribution and the allowed propagation mask includes: The local conflict subgraph is sent to a manual review terminal or a dedicated review model, and only the local nodes, local edges and source evidence related to the conflict are displayed; Receive the corrected labels and corrected probability distribution returned by the review operation, and generate the corrected label distribution; Based on the revised information confirmed by the review, the allowed propagation mask is generated. The allowed propagation mask is used to limit the revision results to be transmitted only along a specified edge type, a specified propagation direction, or a specified hop count range.

8. The local conflict graph error correction method according to claim 1, characterized in that, The step of performing restricted backhaul updates on the distribution of the corrected labels to the associated nodes in the local conflict subgraph based on the allowed propagation mask and path reliability includes: Invalid propagation paths in the local conflict subgraph are filtered according to the allowed propagation mask. The edge weight multiplication value of each path is calculated in the remaining allowed paths, and the maximum value is determined as the backpropagation gain. The backhaul gain is compared with a preset backhaul threshold, and updates are performed only on associated nodes whose backhaul gain is greater than the preset backhaul threshold. Using the backhaul gain as the fusion weight, a convex combination calculation is performed on the original label distribution and the modified label distribution of the target node to obtain the updated node label distribution, and the node confidence is updated synchronously.

9. The local conflict graph error correction method according to claim 1, characterized in that, The step of outputting the final annotation result after the restricted backhaul update is completed and the preset convergence condition is met includes: After the restricted backhaul update is completed, the conflict score and label distribution change of each node in the local conflict subgraph are recalculated. Determine whether the preset convergence conditions are met. The preset convergence conditions include: the conflict scores of all nodes in the subgraph are not higher than the preset conflict threshold, the total change in the distribution of node labels is lower than the preset convergence threshold, or the number of nodes to be reviewed is reduced to below the preset number. If the preset convergence condition is not met, the process of dynamically extracting local conflict subgraphs will be repeated. If the preset convergence condition is met, the final annotation result is output, and the error correction record, conflict evidence, and propagation trajectory are written into the result storage area.

10. A local conflict graph error correction device, characterized in that, Used to perform the local conflict graph error correction method as described in any one of claims 1 to 9.

11. A computer device, characterized in that, The computer device includes a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions that, when executed by a processor, can implement the steps of the method as described in any one of claims 1 to 9.

Images