Target detection scene compliance reasoning system and method based on multi-modal large model and GraphRAG

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a target detection scene compliance reasoning system and method based on a multi-modal large model and GraphRAG. The method comprises the following steps: firstly inputting scene data into a multi-modal target detection and information extraction module, and outputting structured multi-modal detection information; the scene compliance rule is input into a scene rule knowledge graph construction module, and a searchable knowledge graph is output; the detection information and the knowledge graph are synchronously input into a GraphRAG retrieval reasoning module, and a reasoning result is output after entity matching, rule filtering and path sorting; a reasoning result is input into a compliance judgment and intelligent suggestion generation module, and a compliance conclusion and an intelligent suggestion are output; and when a new rule is accessed, dynamic updating of the knowledge graph is supported. The information extraction integrity is improved, accurate association of multi-modal detection information and compliance rules is ensured, cooperation of multi-modal target detection and structured compliance reasoning is realized, and the problems of detection information fragmentation, rule modeling non-structuring and opaque reasoning process in a traditional scheme are solved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a compliant inference system and method for object detection scenarios based on a multimodal large model and GraphRAG. Background Technology

[0002] Multimodal large models possess cross-modal information fusion capabilities. In object detection scenarios, the technology can accurately identify basic information such as the category and location of objects in images, and simultaneously extract visual features, textual information, and numerical parameters from the images. However, current multimodal large model technology and object detection technology have significant shortcomings when applied to "scene compliance reasoning" scenarios. They cannot associate scene rules for compliance judgment, and multimodal large models lack the ability to model structured rules, making the reasoning process opaque and difficult to adapt to dynamic rules.

[0003] To address the aforementioned issues, existing solutions mainly fall into two categories: The first is a "target detection + manual verification" model, where target information is obtained through a detection model, and then a human verifies the results against scenario rules to perform compliance analysis. This method relies on human experience, is inefficient, and prone to misjudgments due to rule omissions. The second approach involves associating target detection results with a simple knowledge graph to achieve basic rule matching. However, this model cannot integrate multimodal detection information for deep reasoning, making it difficult to meet the compliance judgment needs of complex scenarios. Summary of the Invention

[0004] The technical problem to be solved by this invention is that current multimodal large model and object detection technologies lack structured rule modeling, have opaque reasoning, and are difficult to adapt dynamic rules in scene compliance reasoning, as well as existing solutions are inefficient and unable to deeply integrate multimodal information.

[0005] To this end, the present invention provides a target detection scene compliance reasoning system and method based on multimodal large model and GraphRAG. By integrating the cross-modal detection capability of multimodal large model with the structured knowledge reasoning advantage of GraphRAG, it provides a target detection scene compliance reasoning system and method that accurately extracts multimodal target information, efficiently associates scene rules, supports dynamic rule adaptation, and has an interpretable reasoning process, thereby realizing intelligent, efficient and accurate scene compliance judgment.

[0006] The technical solution adopted by this invention to solve its technical problem is: A compliance-compliant inference system for object detection scenarios based on a multimodal large model and GraphRAG, comprising, The multimodal target detection and information extraction module is responsible for collecting scene data and simultaneously extracting target category, location, text information, numerical parameters and attribute features, and outputting structured multimodal detection information. The scenario rule knowledge graph construction module transforms scattered scenario compliance rules into a hierarchical knowledge graph, supporting structured storage and dynamic updates of the rules. The GraphRAG retrieval and reasoning module establishes a connection between multimodal detection information and knowledge graphs. Through two-stage retrieval and logical reasoning, it locates matching compliance rules and generates reasoning paths. The compliance judgment and intelligent suggestion generation module completes the compliance judgment based on the reasoning results of the GraphRAG retrieval reasoning module, and generates targeted optimization suggestions for non-compliant scenarios.

[0007] During the system's data flow, scene data is first input into the multimodal target detection and information extraction module, which outputs structured multimodal detection information. Scene compliance rules are input into the scene rule knowledge graph construction module, which outputs a searchable knowledge graph. Detection information and the knowledge graph are simultaneously input into the GraphRAG retrieval and reasoning module, which outputs reasoning results after entity matching, rule filtering, and path sorting. The reasoning results are input into the compliance judgment and intelligent suggestion generation module, which outputs compliance conclusions and intelligent suggestions. When new rules are introduced, the rules are dynamically updated through the knowledge graph, enabling system adaptation without retraining the model.

[0008] It should be noted that the model refers to the models involved in the multimodal target detection and information extraction module, the GraphRAG retrieval and reasoning module, the scene rule knowledge graph construction module, and the compliance judgment and intelligent suggestion generation module.

[0009] Furthermore, the multimodal target detection and information extraction module is built based on the BLIP-2 model, and the visual branch of the BLIP-2 model is used to extract multi-scale feature maps (visual features). ,in , , These are the feature map height, width, and number of channels, respectively. The language branch of the BLIP-2 model is used to map text descriptions to semantic embeddings (text features). , For language feature dimensions; The multimodal target detection and information extraction module also includes a dedicated numerical feature encoding layer. This encoding layer uses a combination of 1D convolution and global pooling to convert the raw numerical string extracted by OCR (Optical Character Recognition) into numerical features. , For numerical feature dimensions.

[0010] The above scheme can achieve synchronous extraction of target-related quantitative information by introducing a dedicated numerical feature encoding layer on the basis of the BLIP-2 model.

[0011] Furthermore, the multimodal target detection and information extraction module adopts an iterative cross-attention mechanism that integrates cross-modal fusion.

[0012] Specifically, the multimodal target detection and information extraction module incorporates numerical features into the fusion logic and completes the interaction of the three modal features through an iterative attention mechanism:

[0013]

[0014]

[0015] in, The learnable parameter matrix; For the attention dimension; semantic embedding for language branches Learnable parameter matrix The attention key vector obtained from the mapping; semantic embedding for language branches Learnable parameter matrix The resulting attention value vector; For module weight coefficients, satisfying Final fusion features Used for subsequent detection and information extraction.

[0016] By adopting an iterative cross-attention mechanism for cross-modal fusion, the weighted fusion of visual features, text features, and numerical features is achieved, which improves the correlation and fusion accuracy of multimodal information and makes the extracted target information more completely reflect the actual state of the scene.

[0017] Furthermore, the scenario rule knowledge graph construction module adopts a three-layer compliance hierarchy structure of "target entity - rule relationship - numerical range"; The target entity layer uses the normalized target entity as the core node; The rule relationship layer uses compliance rules as its core nodes; The numerical interval layer uses standardized numerical intervals as its core nodes.

[0018] This structure avoids retrieval redundancy caused by mixed elements in general structures.

[0019] Furthermore, the scenario rule knowledge graph construction module adopts a hybrid mode of "automatic parsing + manual verification".

[0020] This hybrid process ensures an accuracy rate of 94% for feature extraction, while reducing the time required by 70% compared to a purely manual method.

[0021] Furthermore, the scene rule knowledge graph construction module includes a graph database, which stores structured data including knowledge graph nodes and relationships generated after being processed by the scene rule knowledge graph construction module.

[0022] The graph database uses Neo4j as its storage medium.

[0023] Furthermore, the GraphRAG retrieval reasoning module includes a first-stage entity matching retrieval; The GraphRAG retrieval inference module also includes a second-stage interval / relational filtering retrieval.

[0024] By employing the aforementioned technical means, we can ensure the semantic accuracy of entity associations while filtering out invalid rules that do not conform to quantitative indicators and relationship constraints, thus preventing irrelevant rules from interfering with subsequent compliance judgments.

[0025] Furthermore, the GraphRAG retrieval reasoning module adopts a three-level association logic, including first-level direct field mapping; second-level semantic correction mapping; and third-level fuzzy adaptation mapping.

[0026] The above-mentioned solution effectively solves the adaptation problems of ambiguous detection information and missing core parameters, and avoids inference interruption caused by information mismatch.

[0027] Furthermore, the GraphRAG retrieval inference module includes an inference path weight calculation model that integrates multi-dimensional indicators.

[0028] The reasoning path weight calculation model generates interpretable reasoning paths and sorts them by priority based on the associations between entities, rules, and intervals in the knowledge graph.

[0029] The above technical solutions prioritize the display of paths with high semantic matching, high priority rules, and high numerical matching, thereby increasing the priority rate of effective paths.

[0030] Furthermore, the compliance judgment and intelligent suggestion generation module calculates a comprehensive compliance score based on the reasoning results of the GraphRAG retrieval reasoning module, and makes a compliance judgment according to the set judgment threshold.

[0031] By using the aforementioned technical means, compliance judgments are remodeled from a "subjective qualitative" problem into an "objective quantitative" problem, thereby reducing the misjudgment rate.

[0032] Furthermore, the system also includes a data storage and management module, where the input and output data of the multimodal target detection and information extraction module, the scene rule knowledge graph construction module, the GraphRAG retrieval and reasoning module, and the compliance judgment and intelligent suggestion generation module are all synchronized to the data storage and management module for archiving.

[0033] A compliance reasoning method includes the following steps: S1, Multimodal target detection and information extraction: Based on the improved BLIP-2 multimodal large model, it realizes the simultaneous extraction and structured output of three modal information: visual features, text features, and numerical features; S2, Constructing a scenario rule knowledge graph: Transforming unstructured compliance rules into a three-layer structured knowledge graph of "target entity - rule relationship - numerical range", and supporting dynamic updates; S3, GraphRAG Two-Stage Retrieval and Reasoning: Based on multimodal detection information and knowledge graph, it generates an interpretable reasoning path through two-stage retrieval and multi-dimensional ranking; S4, Compliance Judgment and Intelligent Suggestion Generation: Based on the final inference path set given by the GraphRAG retrieval inference module, the compliance status is determined by quantifying the comprehensive compliance score and logical verification, and structured suggestions are generated.

[0034] The beneficial effects of this invention are: 1. It achieves synergy between multimodal target detection and structured compliance reasoning, solving the problems of fragmented detection information, unstructured rule modeling, and opaque reasoning process in traditional solutions; 2. The multimodal target detection and information extraction module uses cross-modal feature fusion technology to simultaneously acquire the target's visual features, text features, numerical features, and attribute information, improving the completeness of information extraction by 41% compared to single-modal detection; 3. The two-stage retrieval and hierarchical knowledge graph design of the GraphRAG retrieval and reasoning module ensures the accurate association between multimodal detection information and compliance rules, with a numerical range matching accuracy of 94%. This significantly reduces the risk of misjudgment due to rule matching deviations compared to traditional simple knowledge graph matching accuracy. Attached Figure Description

[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0036] Figure 1 This is a structural block diagram of the compliance reasoning system of this invention; Figure 2 This is a flowchart of the compliance reasoning method of the present invention; Figure 3 This is an internal flowchart of the multimodal target detection and information extraction module of the present invention; Figure 4This is an internal flowchart of the GraphRAG retrieval and inference module of this invention; Figure 5 This is an internal flowchart of the compliance judgment and intelligent suggestion generation module of this invention. Detailed Implementation

[0037] The invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention and therefore showing only the components relevant to the invention. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0038] It should also be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn to actual scale. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0039] Techniques, methods, and apparatus known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0040] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, features defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0041] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0042] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0043] Example 1 Reference Figure 1 A compliance reasoning system for object detection scenarios based on a multimodal large model and GraphRAG is disclosed. The system includes a multimodal object detection and information extraction module, a scene rule knowledge graph construction module, a GraphRAG retrieval and reasoning module, a compliance judgment and intelligent suggestion generation module, and a data storage and management module. The GraphRAG retrieval and reasoning module is connected to the multimodal object detection and information extraction module and the scene rule knowledge graph construction module, while the compliance judgment and intelligent suggestion generation module is connected to the GraphRAG retrieval and reasoning module. Input and output data from these modules are synchronized to the data storage and management module for archiving.

[0044] In this embodiment, the multimodal target detection and information extraction module includes the existing BLIP-2 model. The BLIP-2 model, as a large multimodal model, adopts a visual-language dual-branch structure and is optimized for information extraction requirements. The visual branch extracts multi-scale feature maps (visual features) based on the existing model CSPDarkNet-53. ,in , , These represent the feature map height, width, and number of channels, respectively; the language branch maps text descriptions to semantic embeddings (text features) using the existing RoBERTa encoder. , For the language feature dimension, the multimodal object detection and information extraction module also extends the numerical feature-specific encoding layer on the existing BLIP-2 model, using a combination of 1D convolution and global pooling to convert the original numerical strings extracted by OCR into numerical features. , This is a numerical feature dimension. A dedicated encoding layer for numerical features enables the synchronous extraction of target association quantification information, realizing scene quantification parameters. Furthermore, the multimodal target detection and information extraction module incorporates numerical features into the fusion logic, proposing an iterative cross-attention mechanism for cross-modal fusion. This iterative attention mechanism facilitates the interaction of the three modal features.

[0045]

[0046]

[0047] in, The learnable parameter matrix; For the attention dimension; semantic embedding for language branches Learnable parameter matrix The attention key vector obtained from the mapping; semantic embedding for language branches Learnable parameter matrix The resulting attention value vector; These are the module weight coefficients, which undergo adaptive learning during training and satisfy... The values ​​were verified to be 0.4, 0.3, and 0.3; the final fused features were... Used for subsequent detection and information extraction.

[0048] In this embodiment, the multimodal target detection and information extraction module adopts a parallel synchronous extraction process and a multi-dimensional verification mechanism; Specifically, the multimodal target detection and information extraction module is based on the improved YOLOv8 detection head and fused features. The detection is performed, and the bounding box regression accuracy is optimized using the CIoU loss function. The normalized target bounding box coordinates are then output. and category confidence , The function used is the Sigmoid function; the target bounding box coordinates are used for region localization in subsequent cross-modal information extraction, meaning information extraction is only performed on the region covered by the bounding box; the class confidence score is used to filter valid targets, and low-confidence noisy targets below the threshold will be filtered out to avoid interfering with subsequent inference processes; the multimodal target detection and information extraction module calls PP-OCRv4 to perform text detection and recognition for each target bounding box coverage area, using regular expressions to parse numerical units, and employing a lightweight MobileNetV3 classifier, with the classification loss using a labeled smoothing form adapted and optimized from the existing cross-entropy loss:

[0049] in, Index the true category of the target attribute; True label of target attribute for one-hot encoding The target predicted by the model belongs to the category The probability of; The total number of categories for the target attribute; This is the smoothing factor. The final output target labels include color, status, and material.

[0050] In this embodiment, the multimodal target detection and information extraction module identifies erroneous extraction results through a cross-modal consistency verification mechanism and labels missing non-critical information as "unknown" to ensure the reliability of the output data.

[0051] In this embodiment, the multimodal target detection and information extraction module reduces the regression error of the target bounding box through CIoU loss, thereby improving the localization accuracy of the detection area; the labeled smooth cross-entropy loss reduces the risk of overfitting in the classifier's attribute classification; and the cross-modal consistency check improves the reliability of the output data and prevents erroneous information from flowing into the subsequent inference process.

[0052] In this embodiment, the output data of the multimodal target detection and information extraction module is stored in a structured JSON format, strictly adhering to the GraphRAG retrieval adaptation requirements. The fields included in each target data point are shown in Table 1. This format ensures information integrity and allows for direct use in retrieval and inference without additional data conversion.

[0053] Table 1. Example of stored field format

[0054] In this embodiment, the scenario rule knowledge graph construction module transforms unstructured scenario compliance rules such as industry standards, security specifications, and management requirements into a structured, searchable, hierarchical knowledge graph. This enables the entityification, relationalization, and simplified modeling of rules, while providing a dynamically updated structure to support rapid integration of new rules and knowledge graph iteration. The knowledge graph adopts a three-layer structure design: "target entity - rule relationship - numerical range". The target entity layer uses normalized target entities as core nodes; the rule relationship layer uses compliance rules as core nodes; and the numerical range layer uses standardized numerical ranges as core nodes.

[0055] Specifically, the scenario rules are organized using the logic of "rule decomposition, element extraction, and standard definition." The core elements include: 1. Target entities are objects that need to be inspected in the scenario and are defined according to the two-level classification system of "major category - minor category", such as: fire equipment - fire extinguishers, traffic facilities - speed limit signs.

[0056] 2. Compliance Rules: These are divided into mandatory rules and flexible rules. Mandatory rules include: "Fire extinguishers must be placed at a distance from the point of ignition." "Placement"; flexible rules such as: "Road signs should use reflective materials". The rule logic is extracted through the Qwen-2-VL-7B multimodal large model.

[0057] 3. Numerical intervals: Based on the rules for quantitative indicators, interval mathematical representation is used to define intervals, including closed intervals, open intervals, and half-open intervals. At the same time, numerical units are extracted and standardized.

[0058] 4. Relationship constraints define the association types between entities and rules, entities and entities, and rules and numerical ranges. Constraint types include "applicable rule", "contains subclass", "dependency parameter", "mutual exclusion relationship", etc.

[0059] In this embodiment, the scene rule knowledge graph construction module adopts a balanced process of "automatic parsing + manual verification" for element extraction; Specifically, the knowledge graph construction process adopts a hybrid model of "automated parsing + manual verification," which consists of four steps: The first step is rule parsing, where the rule document is structured and divided into rule text blocks using the MinerU tool. The existing multimodal large model Qwen-2-VL-7B is then used to extract core elements, outputting a candidate set of elements. The second step involves defining entities / relationships / attributes and manually verifying the candidate set of elements, with a pass rate of [missing information]. If deemed qualified, the data undergoes entity normalization, relation type standardization, and attribute format normalization to generate structured data (node ​​data and relation data) of knowledge graph nodes and relations. Entity similarity is calculated using the cosine similarity formula.

[0060] in, Entity word vector representation, similarity The first step is to determine if they are the same entity; the second step is to store the knowledge graph, import the structured data into the graph database, use Neo4j as the storage medium, store the node data according to the key-value pairs of "attribute name-attribute value", and the relation data includes directionality and weight; the third step is to design the dynamic update interface, based on the RESTful API incremental update interface, which supports the addition, modification and deletion of rules. During the update, the conflict detection algorithm avoids duplicate or contradictory rules. If a conflict is determined, the manual review process is triggered.

[0061] In this embodiment, the scene rule knowledge graph construction module adopts multi-dimensional index optimization for compliant retrieval, thereby significantly improving the retrieval speed of GraphRAG; Specifically, the storage solution is optimized for the hierarchical structure and retrieval requirements of the knowledge graph: 1. Neo4j version 5.10 was selected as the graph database.

[0062] 2. Node storage format: Target entity layer nodes additionally store an embedding field, which is generated as a 3072-dimensional vector using the text-embedding-3-large model; rule relationship layer nodes store the rule_logic field; and numerical interval layer nodes store the interval_code field.

[0063] 3. Index Design: Three core indexes are established: a full-text index for entity_name (entity name) to support fuzzy matching queries; a combined index for interval_id (unique identifier of numerical interval) and param_name (parameter name corresponding to the quantitative indicator bound to the numerical interval in the compliance rule, such as fire extinguisher pressure value, speed limit sign speed limit value, etc.) to optimize the association query of parameters (param_name mentioned in the index design) and intervals (intervals are the compliant numerical ranges defined by standardization in the knowledge graph numerical interval layer, corresponding to interval_id in the index design); and a sorted index for rule_priority (rule priority) to support filtering rules according to priority.

[0064] In this embodiment, the core function of the GraphRAG retrieval and reasoning module is to establish a precise association between multimodal detection information and compliance knowledge graph, and to generate an interpretable compliance reasoning path. Specifically, a two-stage retrieval strategy of "entity matching + interval / relationship dual constraints" is adopted for compliance scenarios. The first stage is entity matching retrieval, which uses the target category, attributes and text information in the multimodal detection information for semantic matching of the target entity layer nodes of the knowledge graph. That is, the category (target category), attributes (target attributes) and text_info (target text information) fields of the detection information are concatenated into entity description text, and a 3072-dimensional feature vector is generated by the existing text-embedding-3-large model. , and the pre-stored embedding entity vectors of entity nodes in the knowledge graph Calculate cosine similarity:

[0065] Set similarity threshold ,recall All entity nodes form a candidate entity set. Simultaneously, by leveraging the hierarchical relationships within the entity classification tree, parent / child entity nodes of candidate entities are recalled, avoiding missed detections due to entity segmentation. The second stage involves interval / relationship filtering retrieval, targeting the candidate entity set. The associated rule relationship layer nodes are subjected to dual constraint filtering: first, numerical range filtering, extracting the numeric_params values ​​from the detection information. The numerical range nodes that the nodes in the rule relationship layer depend on. The matching process is performed, and the numerical matching degree is calculated as follows:

[0066] Set numerical matching threshold ,reserve The rules, It is a standardized, dynamically quantified range extracted from compliance rules for different scenarios, such as the rule "fire extinguishers must be placed at a distance from the point of ignition". "meters", the nodes of the numerical interval are closed intervals. Second, relation constraint filtering verifies whether the relationship between the target and other entities in the scene conforms to the relation constraints in the knowledge graph. This is achieved by traversing the knowledge graph path to verify the rationality of the relationships, filtering out rules that violate the relation constraints, and ultimately forming a valid rule set. .

[0067] In this embodiment, the GraphRAG retrieval inference module adopts a three-level multimodal detection information association mechanism of "field mapping - semantic correction - fuzzy adaptation", which is a three-level association logic; Specifically, it includes: First-level fields are directly mapped to establish a fixed association between the core fields of the detection information and the attributes of the knowledge graph nodes (the core attributes corresponding to the nodes in the three-layer structure of the knowledge graph: "target entity layer", "rule relationship layer", and "numerical interval layer"). The second-level semantic correction mapping addresses the issue of ambiguous or abbreviated text information by using a pre-built thesaurus and semantic correction model (based on DeepSeek V3, inputting ambiguous text and outputting standard descriptions) for correction. The three-level fuzzy adaptation mapping is designed for scenarios where detection information is missing. It adapts the data through the transitivity of knowledge graph relationships (referring to the "relationship constraints" mentioned above, with constraint types including "applicable rules", "contains subclasses", "dependency parameters", and "mutually exclusive relationships"), while recording the missing fields to provide a basis for subsequent "to be verified" fuzzy judgments.

[0068] In this embodiment, to address the problem that existing technologies rely heavily on single semantic similarity for reasoning path ranking, without considering core dimensions such as the priority of compliance rules and numerical matching degree, resulting in ranking results that do not meet the requirements of compliance scenarios, the GraphRAG retrieval reasoning module integrates a reasoning path weight calculation model with multi-dimensional indicators. It generates interpretable reasoning paths and ranks them according to priority based on the associations between entities, rules, and intervals in the knowledge graph. Specifically, firstly, based on an effective rule set Construct the reasoning path corresponding to each rule. ,in As candidate entities, For effective rules, For nodes within the numerical range that the rules depend on; the weight calculation of the path comprehensively considers entity matching similarity, rule priority, and numerical matching degree, and the formula is:

[0069] Among them, 0.4, 0.35, and 0.25 are weighting coefficients, satisfying 0.4 + 0.35 + 0.25 = 1; The rule priority (levels 1-5) is normalized to a value in the range [0,1].

[0070] Then, all reasoning paths were analyzed according to... Sort the paths in descending order and select the top 10 paths to form the final reasoning path set. If there are cases where path weights are the same, further ranking will be based on the authority of the rule source (international > industry standards > corporate specifications) and the numerical matching degree (the numerical matching degree is reflected by the calculation result of the numerical matching degree formula). The numerical values ​​in the detection information... The closer The higher the compliance level of the center, The closer the value is to 1, the better; if it exceeds the range, the value is 0. Sort the values ​​in the high and low ranges in a second sorting process.

[0071] In this embodiment, the compliance judgment and intelligent suggestion generation module is based on the final inference path set output by the GraphRAG retrieval inference module. By quantifying compliance scores and performing multi-dimensional logical verification, the system accurately determines the compliance status of targets (compliant / non-compliant / pending verification), and generates intelligent optimization suggestions for different compliance statuses and violation types, ultimately outputting the results in the form of a structured report.

[0072] Specifically, addressing the issue that existing compliance assessments often rely on single-rule matching results, lacking quantitative evaluation and multi-dimensional verification, and resulting in subjective and easily misjudged outcomes, the compliance assessment and intelligent suggestion generation module adopts a compliance status determination mechanism of "quantitative scoring + multi-dimensional logical verification." This mechanism is first based on the final inference path set. Calculate the overall compliance score Taking into account inference path weights, rule matching completeness, and numerical fit:

[0073] in The maximum weight value in the inference path set reflects the relevance of the optimal matching rule. Its weighting coefficient; The number of rules that were successfully matched. The total number of all rules to which the target applies; Its weighting coefficient; The numerical matching degree of all matching rules is averaged. Its weighting coefficient; The range of values ​​is .

[0074] Three levels of judgment thresholds are set based on the overall compliance score: [Settings] , ,when When, it is judged as "compliant"; when When, it is judged as "non-compliant"; when Further verification of the completeness of the detection information is performed: if a core field is missing, it is judged as "to be verified" and the missing field is marked; if the information is complete but some rules are not matched, the judgment is made in combination with the rule priority: if the unmatched rule includes a 1-2 level high priority rule, it is judged as "non-compliant", otherwise it is judged as "compliant (needs optimization)".

[0075] In this embodiment, to address the problem that existing intelligent suggestions lack specificity and operability, the compliance judgment and intelligent suggestion generation module adopts a strategy of subdividing violation types and generating scenario-based structured suggestions.

[0076] Specifically, the "non-compliance" status is first further subdivided into violation types, including: 1) Value exceeding the standard type (detected value exceeds the rule range); 2) Relationship violation type (violation of inter-entity relationship constraints); 3) Attribute mismatch type (target attribute does not meet rule requirements); 4) Entity missing type (essential entities for the scene were not detected).

[0077] Secondly, for different violation types, suggestions are generated by guiding the large language model (based on the existing Qwen-72B-Instruct model) through prompt word engineering. The prompt words include "violation type + rule text + detection result + suggestion generation requirements". Furthermore, the hierarchy is clear, divided into two categories: "urgent rectification" and "optimization and improvement." For the "pending verification" status, it is recommended to focus on "supplementary information collection," clearly specifying the testing fields that need to be supplemented and the collection methods (e.g., "the pressure value of the fire extinguisher needs to be supplemented; it is recommended to use a pressure testing instrument for on-site measurement"). For the "compliant" status, the suggestion to "maintain the current status" should be output, while simultaneously marking the core compliance basis to enhance the credibility of the results.

[0078] In this embodiment, the output format is designed as a standardized JSON structured report, which takes into account both machine readability and human readability.

[0079] Specifically, for the multimodal target detection and information extraction module, the field definitions and formats are as follows: Table 2. Table 2 Example of Result Output Format

[0080] For the scenario rule knowledge graph construction module, each target is listed in turn according to its target_id, including its compliance status, comprehensive compliance score, violation type, and core matching rule. The GraphRAG retrieval inference module contains detailed information on the first three paths of the inference path set; The compliance assessment and intelligent suggestion generation module lists suggestions according to the categories of "urgent rectification", "optimization and improvement" and "pending verification and supplementation".

[0081] Example 2 Reference Figures 2-5 A compliant inference method for object detection scenarios based on a multimodal large model and GraphRAG includes the following steps: S1, Multimodal Target Detection and Information Extraction: Based on the improved BLIP-2 multimodal large model, it realizes the simultaneous extraction and structured output of three modal information: visual features, text features, and numerical features.

[0082] Specifically, it includes: S1-1 Input Data Preprocessing: Acquire the original scene image data, perform Mosaic enhancement and pixel normalization operations on the image, and generate a standardized input image. This eliminates the influence of illumination and scale differences on the detection results, and standardizes the input image as the core input for the subsequent multimodal large model vision branch.

[0083] S1-2 Trimodal Feature Encoding: For visual feature encoding in the visual branch, CSPDarkNet-53 is used as the backbone network of the visual branch, and the standardized input image is processed... Extracting multi-scale feature maps For text feature encoding in the language branch, the RoBERTa encoder is used to perform semantic mapping on the text information recognized by subsequent OCR, generating language feature embeddings. For numerical feature encoding, the dedicated numerical feature encoding layer uses a combination of 1D convolution and global pooling to convert the original numerical strings extracted by OCR into numerical features. .

[0084] S1-3 Trimodal Feature Fusion: An optimized iterative cross-attention mechanism is used to fuse visual features. Text features Numerical characteristics Perform deep interactive fusion and output the final fused features. .

[0085] S1-4 Parallelized Detection and Information Extraction: In the target detection stage, feature fusion is based on an improved YOLOv8 detection head pair. The detection is performed, and the bounding box regression accuracy is optimized by the CIoU loss function. The normalized target bounding box coordinates and class confidence are output. In the text and numerical extraction stage, PP-OCRv4 is called to perform text detection and recognition for each target bounding box region, and regular expressions are used to parse numerical units and standardize them. In the attribute classification stage, a lightweight MobileNetV3 classifier is used to classify the target attributes. The classification loss is the labeled smooth cross-entropy loss, and the target attribute label is output.

[0086] S1-5 Cross-modal validation and structured output: Error-based extraction results are eliminated through cross-modal consistency validation mechanisms (such as the matching of numerical units with target categories). Missing non-critical information is labeled as "unknown". Finally, data such as target categories, bounding boxes, text information, numerical parameters, and attribute labels are structured and stored in JSON format to directly adapt to subsequent retrieval and inference needs.

[0087] S2, Constructing a Scenario Rule Knowledge Graph: Transforming unstructured compliance rules into a three-layer structured knowledge graph of "target entity - rule relationship - numerical range", supporting dynamic updates.

[0088] Specifically, it includes: S2-1 Rule Parsing and Core Element Extraction: A hybrid mode of "automatic parsing + manual verification" is adopted. The MinerU tool is used to perform structured parsing of rule documents such as industry standards and security specifications, and divide them into rule text blocks. The multimodal large model Qwen-2-VL-7B is called to extract customized core elements from the text blocks.

[0089] S2-2 Feature Verification and Normalization: The extracted candidate feature set is manually verified through entity normalization, relation type standardization, and attribute format standardization.

[0090] S2-3 Three-layer knowledge graph construction and storage: For the construction of the knowledge graph structure, a three-layer structure of "target entity layer - rule relationship layer - numerical interval layer" is constructed. The target entity layer takes normalized entities as the core node, the rule relationship layer takes compliant rules as the core node, and the numerical interval layer takes standardized numerical intervals as the core node. For storage deployment: Import structured data into the Neo4j 5.10 graph database, and store node data as key-value pairs of "attribute name-attribute value"; For index optimization, three types of core indexes are created.

[0091] S2-4 Dynamic Update Interface Deployment: The incremental update interface is designed based on a RESTful API, supporting the addition, modification, and deletion of rules. During updates, a conflict detection algorithm is used to determine rule contradictions. Conflicting rules trigger a manual review process to ensure that the knowledge graph iterates without contradictions.

[0092] S3, GraphRAG Two-Stage Retrieval and Reasoning: Based on multimodal detection information and knowledge graphs, it generates interpretable reasoning paths through two-stage retrieval and multi-dimensional ranking.

[0093] Specifically, it includes: S3-1 Entity Matching Retrieval: Concatenate the fields from the structured detection information output in step S1-5 into entity description text, generate a 3072-dimensional feature vector using the text-embedding-3-large model, and calculate the relationship between this vector and the pre-stored embedding vector of the target entity layer node in the knowledge graph. Cosine similarity, setting a similarity threshold Recall entity nodes with similarity greater than a threshold to form a candidate entity set. At the same time, by using the hierarchical association of the entity classification tree, the parent / child entity nodes of candidate entities are recalled to avoid missed detections.

[0094] S3-2 Interval / Relation Dual Constraint Filtering: For numerical interval filtering, extract the numeric_params value from the detection information and compare it with the candidate entity set. The association rule layer matches the numerical ranges of the nodes' dependencies. For relationship constraint filtering, the knowledge graph path is traversed to verify whether the relationship between the target and other entities in the scene conforms to constraints such as "mutual exclusion" and "dependency" in the knowledge graph. Rules that violate relationship constraints are filtered out to form valid rules. .

[0095] S3-3 Three-Level Information Association Mapping: This system employs a three-level mapping rule—"field mapping - semantic correction - fuzzy adaptation"—to establish a connection between multimodal detection information and the knowledge graph. The first-level field mapping directly maps the core detection fields to fixed associations with knowledge graph node attributes. The second-level semantic correction mapping uses a pre-built thesaurus and the DeepSeek V3 model to correct ambiguous / abbreviated text to standard descriptions. The third-level fuzzy adaptation mapping marks missing fields when detection information is missing, using general rules for knowledge graph relation transitivity matching.

[0096] S3-4 Inference Path Weight Calculation and Ranking: Path Construction Based on Effective Rule Sets Construct a structured reasoning path; use a multi-dimensional weight model for weight calculation; sort and filter according to weight in descending order, take the top 10 paths to form the final path set, and when the weights are the same, sort them a second time according to the authority of the rule source and the degree of numerical matching.

[0097] S4, Compliance Judgment and Intelligent Suggestion Generation: Based on the final inference path set given by the GraphRAG retrieval inference module, the compliance status is determined by quantifying the comprehensive compliance score and logical verification, and structured suggestions are generated.

[0098] Specifically, it includes: S4-1 Compliance Overall Score Calculation: Calculating the overall compliance score based on the final path set. .

[0099] S4-2 Multi-dimensional Compliance Status Assessment: Design of Three-Level Assessment Thresholds , ,when When, it is judged as "compliant"; when When, it is judged as "non-compliant"; when When verifying the completeness of the detection information: if a core field is missing, it is judged as "to be verified" and the missing field is marked; if the information is complete but does not match any rules containing 1-2 high-priority rules, it is judged as "non-compliant", otherwise it is judged as "compliant (needs optimization)".

[0100] S4-3 Violation Type Subdivision and Suggestion Generation: For non-compliant status: Violation types are subdivided into numerical exceedance type, relationship violation type, attribute mismatch type, and entity missing type. Customized prompts (including "violation type + rule text + detection result + suggestion requirements") guide Qwen-72B-Instruct to generate structured suggestions such as "urgent rectification" (high-priority rule violation) or "optimization and improvement" (low-priority rule violation); For pending verification status: Information supplementation collection suggestions are generated, specifying the detection fields to be supplemented and the collection methods; For compliant status: "maintain current status" suggestion is output, indicating the core compliance basis.

[0101] S4-4 Structured Report Output: Integrates compliance status, reasoning path, compliance score, and intelligent suggestions into a structured report according to a preset template. It supports direct export and system integration to complete the entire compliance reasoning process.

[0102] This concludes the detailed description of a compliance inference system and method for object detection scenarios based on a multimodal large model and GraphRAG, according to this disclosure. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions disclosed herein based on the above description.

[0103] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined by the scope of the claims.

Claims

1. A compliance-compliant inference system for object detection scenarios based on a multimodal large model and GraphRAG, characterized in that, include, The multimodal target detection and information extraction module is responsible for collecting scene data and extracting visual and text features. It also converts the original numerical strings extracted by OCR into numerical features through a dedicated numerical feature encoding layer, and outputs structured multimodal detection information. The scenario rule knowledge graph construction module transforms scattered scenario compliance rules into a hierarchical knowledge graph, supporting structured storage and dynamic updates of the rules. The GraphRAG retrieval and reasoning module establishes a connection between multimodal detection information and knowledge graphs. Through two-stage retrieval and logical reasoning, it locates matching compliance rules and generates reasoning paths. The compliance judgment and intelligent suggestion generation module completes the compliance judgment based on the reasoning results of the GraphRAG retrieval reasoning module, and generates targeted optimization suggestions for non-compliant scenarios.

2. The object detection scene compliance inference system based on multimodal large model and GraphRAG as described in claim 1, characterized in that, The multimodal target detection and information extraction module completes visual feature extraction through an iterative attention mechanism. Text features Numerical characteristics Interaction: in, The learnable parameter matrix; For the attention dimension; semantic embedding for language branches Learnable parameter matrix The attention key vector obtained from the mapping; semantic embedding for language branches Learnable parameter matrix The attention value vector obtained by mapping; Let these be the module weight coefficients, and satisfy... ; This is the final fusion feature.

3. The object detection scene compliance inference system based on multimodal large model and GraphRAG as described in claim 2, characterized in that, The multimodal target detection and information extraction module employs a parallel synchronous extraction process and a multi-dimensional verification mechanism to complete: image preprocessing to generate standardized input data; and feature fusion based on an improved YOLOv8 detection head. The detection process is performed, and the bounding box regression accuracy is optimized using the CIoU loss function. The normalized target bounding box coordinates and class confidence scores are output. Text detection and recognition are performed on each target bounding box region, and the numerical units are parsed. A lightweight classifier is used, and the classification loss adopts the labeled smooth cross-entropy loss.

4. The object detection scene compliance inference system based on multimodal large model and GraphRAG as described in claim 1, characterized in that, The knowledge graph in the scenario rule knowledge graph construction module adopts a three-layer structure of target entity-rule relationship-numerical interval. The target entity layer uses the normalized target entity as the core node; the rule relationship layer uses the compliant rule as the core node; and the numerical interval layer uses the standardized numerical interval as the core node.

5. The object detection scene compliance inference system based on multimodal large model and GraphRAG according to claim 4, characterized in that, The knowledge graph construction process in the scenario rule knowledge graph construction module adopts a hybrid mode of automated parsing + manual verification. The hybrid mode specifically includes: Step 1: rule parsing, which involves structured parsing of the rule document, segmenting it into rule text blocks, calling a multimodal large model to extract core elements, and outputting a candidate set of elements; Step 2: defining entities / relationships / attributes, manually verifying the candidate set of elements, generating structured data of knowledge graph nodes and relationships, and determining the same entity based on the entity similarity calculated by the cosine similarity formula; Step 3: knowledge graph storage, which involves importing the structured data into the graph database; and Step 4: dynamic update interface design.

6. The object detection scene compliance inference system based on multimodal large model and GraphRAG according to claim 5, characterized in that, The scenario rule knowledge graph construction module optimizes the index and establishes three types of core indexes.

7. The object detection scene compliance inference system based on multimodal large model and GraphRAG as described in claim 1, characterized in that, The two-stage retrieval in the GraphRAG retrieval inference module is a two-stage retrieval strategy that combines entity matching with range / relationship constraints for compliant scenarios. The entity matching retrieval generates feature vectors, and the cosine similarity of the feature vectors is calculated. Forming a candidate entity set ; The interval / relationship filtering retrieval targets candidate entity sets. The numerical matching degree is obtained by filtering the numerical range of the associated rule relationship layer nodes. Relationship constraint filtering yields an effective rule set. .

8. The object detection scene compliance inference system based on multimodal large model and GraphRAG according to claim 7, characterized in that, The GraphRAG retrieval and reasoning module adopts a three-level mapping rule of field mapping, semantic correction, and fuzzy adaptation to establish the association between multimodal detection information and knowledge graph; The reasoning path is constructed based on an effective rule set. Construct a structured reasoning path; the weight calculation of the reasoning path adopts the calculation formula in the reasoning path weight calculation model that integrates multi-dimensional indicators: in, Rule priority; The reasoning paths are sorted in descending order of weight to form the final path set. When the weights are the same, they are sorted a second time according to the authority of the rule source and the degree of numerical matching.

9. The object detection scene compliance inference system based on multimodal large model and GraphRAG according to claim 1, characterized in that, The compliance assessment and intelligent suggestion generation module uses a compliance status determination mechanism that combines quantitative scoring with multi-dimensional logical verification to calculate the overall compliance score. in, The maximum weight value in the inference path set reflects the relevance of the optimal matching rule. Its weighting coefficient; The number of rules that were successfully matched. The total number of all rules to which the target applies; Its weighting coefficient; The numerical matching degree of all matching rules is averaged. Its weighting coefficient; The range of values ​​is .

10. A method for compliant inference in object detection scenarios based on a multimodal large model and GraphRAG according to any one of claims 1-9, characterized in that, Includes the following steps: S1, Multimodal target detection and information extraction: Input data preprocessing to generate standardized input images. Extracting visual features Text features Numerical characteristics Three-modal features; an optimized iterative cross-attention mechanism is used to fuse the three-modal features, outputting the final fused features. ; Then, parallel detection and information extraction, cross-modal verification, and structured output are performed; S2, Constructing a Scene Rule Knowledge Graph: Rule parsing and core element extraction; element verification and normalization; Constructing a three-layer structure: target entity layer - rule relationship layer - numerical interval layer; Dynamically updating interface deployment; S3, GraphRAG two-stage retrieval reasoning: entity matching retrieval; interval / relation dual constraint filtering; A three-level mapping rule of field mapping, semantic correction, and fuzzy adaptation is adopted to establish the association between multimodal detection information and knowledge graph; inference path weight calculation and ranking are performed. S4, Compliance Judgment and Intelligent Suggestion Generation: Based on the final inference path set given by the GraphRAG retrieval inference module, compliance status is judged by quantifying the comprehensive compliance score and designing a three-level judgment threshold; violation type subdivision and suggestion generation; compliance status, inference path, compliance score, and intelligent suggestions are integrated into a structured report.

Images