Multi-graph hidden danger prediction method based on retrieval enhancement and full-graph sub-graphs

Abstract

The invention relates to the technical field of internet data services, and particularly discloses a multi-graph hidden danger prediction method based on retrieval enhancement and full-graph sub-graphs. The method constructs a complete technical closed loop from data preparation, model training to intelligent application by combining external knowledge retrieval, multi-granularity visual perception and hierarchical reasoning, and specifically comprises three core stages: firstly, automatically constructing a high-quality field specific training data set by using a pre-training large model; secondly, specialized cultivation of a lightweight target multi-modal large model is carried out through supervised fine tuning and reinforcement learning alignment; and finally, introducing'full graph-sub graph 'multi-vision input and regulation knowledge retrieval enhancement in a prediction stage, performing hierarchical reasoning, and outputting a structured hidden danger report.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet data services, and more particularly, relates to a multi-graph hazard prediction method based on retrieval enhancement and full-graph sub-graphs. BACKGROUND

[0002] Industrial safety production is the cornerstone of national economic development and social stability, and its core lies in the accurate prediction and timely intervention of potential hazards. Traditional safety hazard investigation mainly relies on manual inspection, which has inherent defects such as low efficiency, inconsistent standards, and susceptibility to subjective factors. In recent years, artificial intelligence technology represented by multi-modal large models has been considered as an ideal path to realize automatic hazard intelligent prediction due to its powerful image understanding and content generation capabilities. However, when directly applying general multi-modal large models to the highly specialized field of industrial safety, the following three interrelated technical bottlenecks need to be addressed: First, the lack of domain knowledge and the coexistence of "hallucination" problems lead to low prediction result credibility. General multi-modal large models do not inject sufficient and professional safety production regulations and standard knowledge during the training process. When faced with complex industrial scenes, the model often relies on the semantic prior learned from general data to "guess" rather than rigorous reasoning based on image evidence and professional knowledge, which is prone to produce "hallucination" outputs that do not match the image content or are completely fictitious. For example, the model may ignore the intact fire extinguisher in the image, but "fantasize" about non-existent equipment damage descriptions. This factual error makes the model's prediction results lack credibility in rigorous industrial scenarios and cannot be directly used to guide safety decisions.

[0003] Second, the "coarse granularity" of visual perception leads to missed detection and misjudgment of key details. The visual attention mechanism of existing multi-modal large models usually focuses on the global saliency regions of the image, and lacks sufficient perception ability for key local details that determine the hazard state (such as specific readings of instrument panels, warning information on equipment nameplates, tiny leakage marks, and safety valve jump states, etc.). This neglect of fine-grained visual features makes the model unable to accurately judge the real-time running state of the equipment, resulting in missed detection of key hazards or misjudgment of normal working conditions, which seriously restricts its application value in fine-grained safety management.

[0004] Third, knowledge solidification and update lag, leading to the model is difficult to adapt to the dynamic changing regulatory requirements. Industrial safety regulations and standards will be updated with technological progress and practical experience. If you try to solidify the vast field of knowledge through retraining to the model parameters, not only need to spend huge computing resources and labeling costs, its iteration cycle is far from meeting the needs of the actual application for fast response. This difficulty in knowledge update makes it difficult for traditional models based on fixed parameters to continuously adapt to the latest safety specifications, and there is a systematic risk of false decisions due to outdated knowledge.

[0005] Therefore, there is an urgent need for an Internet data service technology that can realize the automatic, accurate, and traceable intelligent prediction of industrial safety hazards. SUMMARY

[0006] In order to overcome the above-mentioned defects of the prior art, the present application provides a multi-graph hazard prediction method based on retrieval enhancement and full graph subgraph, which solves the problems raised in the above background technology by constructing a high-quality domain-specific training set, fine-tuning the target multi-modal large model according to the instructions and aligning it with reinforcement learning, and introducing "full graph-subgraph" multi-vision input and external knowledge base retrieval during prediction.

[0007] To achieve the above purpose, the present application provides the following technical solutions: A multi-graph hazard prediction method based on retrieval enhancement and full graph subgraph, comprising the following steps: Step 1: High-quality training dataset construction: a pre-trained multi-modal large model with a large parameter size is used to process the original industrial site images to automatically generate initial scene labels, entity labels and target bounding boxes; based on the scene labels and entity labels, vectorized retrieval is performed from the safety regulation knowledge base to obtain relevant regulation text segments; the hazard prediction description generated by the pre-trained multi-modal large model based on the image and the retrieved regulations is compared with the standard answer annotated by humans in terms of semantic similarity to select high-quality samples; at the same time, the target region subgraph is cropped from the original image according to the target bounding box, and the subgraph is enlarged by no less than twice the original size and subjected to image enhancement processing, and after effectiveness verification, the original image, label, bounding box and standard hazard description are combined to form the final training dataset; Step 2: Fine-tuning of the target model: the dataset constructed in step 1 is used to supervise the fine-tuning of the target multi-modal large model, so that it learns to follow the pre-set structured instruction template to output standard scene labels, entity lists and target bounding boxes; then, a reinforcement learning fine-tuning method is used to optimize the model with a composite reward function composed of format specification reward and answer correctness reward; Step three, hazard prediction: using the fine-tuned target multi-modal model to analyze the to-be-tested image, obtaining scene label, entity label and target bounding box; based on the bounding box, the target region is cropped and enlarged for enhancement, forming a multi-image input of "full image + sub-image"; based on the label, the safety regulation knowledge base is retrieved; the multi-image input and the retrieved regulation text are jointly input into the fine-tuned model for hierarchical reasoning, and the scene-level and entity-level hazard prediction results are fused to output a structured report containing hazard type, location and regulation basis.

[0008] The technical effects and advantages of the multi-image hazard prediction method based on retrieval enhancement and full image sub-image of the present application are as follows: The present application uses the external knowledge base as a reliable information source through "retrieval enhancement", guides the model to reason based on facts and image evidence, and reduces false positives and fictitious outputs; The present application introduces a multi-image input mechanism of "full image + enlarged sub-image", so that the model can pay attention to the global scene context and the fine details of the local target at the same time, thereby realizing the identification and state judgment of key hazard points such as small devices, instrument readings and text identification; The present application externalizes the huge regulation knowledge base outside the model parameters, dynamically associates through vector retrieval, solves the problem of model knowledge solidification and update difficulty, does not need to retrain the model, only updates the knowledge base to adapt to the change of regulations, has low deployment cost and high flexibility; The present application forces the model to output machine-resolvable structured results through instruction fine-tuning and reinforcement learning, and clearly associates the regulation basis, so that the generated prediction report not only has accurate results, but also has clear traceability, greatly facilitating subsequent rectification and decision-making. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 The flowchart of the overall method of the present application; Figure 2 The detailed flowchart of the training data set preparation stage of the present application; Figure 3 The multi-image construction and hierarchical reasoning flowchart of the prediction stage of the present application; Figure 4 The structure diagram of hierarchical reasoning and result fusion in the embodiment of the present application; Figure 5 The structure diagram of the multi-image hazard prediction system of the chemical storage tank area in the embodiment of the present application. DETAILED DESCRIPTION

[0010] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0011] Example 1

[0012] This embodiment provides a multi-graph hazard prediction method based on retrieval enhancement and full-graph subgraphs, such as... Figure 1 As shown, the complete process of this method is clearly divided into three core stages: training dataset construction, fine-tuning training of the target multimodal large model, and retrieval-enhanced multi-graph hazard prediction. These three stages constitute a closed loop from automated data preparation and professional model training to practical intelligent application.

[0013] I. Construction of Training Dataset This embodiment uses a large-scale pre-trained multimodal model (e.g., Qwen2.5-VL-72B) as a data annotation engine to process unlabeled industrial site images. The operation process is as follows: Figure 2 As shown. This pre-trained multimodal large model performs two key tasks according to preset instructions: The first task is scene recognition. The pre-trained multimodal large model needs to determine whether the image content belongs to a preset list of scenes covering typical industrial environments. This list includes, but is not limited to, "workshop," "warehouse," "paint booth," "polishing workshop," "chemical warehouse," "printing workshop," "liquefied gas room," "battery storage area," "hotel," "dormitory," and "factory area." If the image is identified as belonging to a scene in the list, the pre-trained multimodal large model outputs the name label of the scene according to a strict format. The second task is object detection. The pre-trained multimodal large model needs to identify all entities in the image that are closely related to safe production, such as "fire extinguisher," "electrical distribution box," "safety passage," "exposed wire," "pressure gauge," and "safety helmet," and generate an accurate target bounding box represented by pixel coordinates for each successfully detected entity. Its standard format is [x min ,y min ,x max ,y max ] represents the coordinates of the top left and bottom right corners of the bounding box, respectively.

[0014] After obtaining the initial scene and entity labels, the process proceeds to the semantic-based regulatory knowledge retrieval stage. This invention constructs a regulatory knowledge base specifically for the field of production safety, whose content sources include national laws and regulations such as the "Production Safety Law" and the "Fire Protection Law," industry safety standards, and internal safety rules and regulations of enterprises. First, these original regulatory documents are preprocessed, segmenting them into semantically relatively complete text fragments. Then, a powerful text encoding model (BGE-M3 in this embodiment) is used to convert each text fragment into a high-dimensional dense vector. All these vectors and their corresponding original text fragments are systematically stored in a database optimized for vector retrieval (such as Milvus). When a retrieval is needed, the scene labels and each entity label generated in the aforementioned steps are used as independent query inputs. These query texts are converted into query vectors using the same BGE-M3 encoding model. The semantic relevance is measured by calculating the cosine similarity between each query vector and all document fragment vectors in the knowledge base. The pre-trained multimodal large model ultimately selects the top K (e.g., Top 5) document fragments with the highest similarity for each query as the most relevant regulatory knowledge basis for that scene or entity.

[0015] This method inputs the original image and the regulatory text fragments retrieved for the scene back into the pre-trained multimodal large model. It guides the pre-trained multimodal large model to combine visual information and professional knowledge to generate a hazard prediction description for the entire scene. For each detected entity, an initial sub-image of the entity is cropped from the original image based on its target bounding box. This sub-image and the regulatory text fragments retrieved for the entity are then input into the model to generate a fine-grained hazard prediction description for the entity. The initial labels generated by the pre-trained multimodal large model are manually sampled and verified to ensure the reliability of the basic labels.

[0016] To ensure the high reliability of the final dataset, this invention introduces a rigorous data cleaning mechanism. This mechanism utilizes a pre-trained language model (e.g., a model used to calculate BERTS score) to calculate the similarity at the semantic level between the hazard descriptions generated by the model and the standard answers manually annotated by safety production experts. A similarity threshold (e.g., 0.7) is set, and only samples whose semantic similarity to the standard answers exceeds this threshold are retained, while other low-quality samples are discarded. This effectively filters out high-quality, high-confidence portions from the massive, automatically generated data.

[0017] Simultaneously, this invention performs precise spatial cropping of the original image based on the coordinates of the target bounding box to obtain an initial sub-image for each entity. Each initial sub-image is then magnified to at least twice its original size. This embodiment preferably employs a bicubic interpolation algorithm that effectively preserves image edge and texture quality. The magnified sub-images undergo further image sharpening processing, such as using a 3x3 Laplacian kernel for convolution operations to enhance edge and detail contrast. For regions containing key details such as text (e.g., device nameplates), scales, and pointers, additional local contrast enhancement algorithms, such as adaptive histogram equalization, are used to make these subtle features more clearly discernible. After enhancement, these sub-images undergo validity assessment, for example, by being input into a visual model for recognition. Only sub-images that can be clearly and unambiguously identified as their corresponding entity categories are considered valid and ultimately included in the dataset.

[0018] After a series of rigorous steps, each training sample that is finally constructed contains comprehensive and structured information. Table 1 shows all the elements contained in a complete training sample.

[0019] Table 1. Examples of Training Dataset Sample Composition

[0020] II. Fine-tuning and Training of the Target Model The fine-tuning training phase of the target model uses the high-quality, multimodal dataset built in the first phase to professionally train a target multimodal large model (e.g., Qwen2.5-VL-7B) with a relatively small parameter size that is easier to deploy, so that it has accurate structured output capabilities and reliable hazard identification capabilities. The training process is divided into two progressive phases.

[0021] The first step is supervised fine-tuning (SFT). In this stage, a pre-defined, highly structured instruction template is used to perform full fine-tuning of the target multimodal large-scale model. The template explicitly requires the target multimodal large-scale model to perform two tasks sequentially: first, scene recognition, determining whether the image belongs to a pre-defined industrial scene and outputting the scene name in a formatted manner; second, object detection, listing the main entities identified in the image and outputting the entity names and corresponding bounding box coordinates in a strict format. During training, the image and this structured instruction are used as input to the model, while the corresponding standard scene labels, entity lists, and bounding box coordinates from the training dataset are used as the target outputs that the model needs to learn and fit. By minimizing the loss function between the model's predictions and the true labels on large-scale, high-quality data, the target multimodal large-scale model gradually learns to stably and accurately output structured parsing results in a pre-defined format after receiving industrial images, laying the foundation for subsequent retrieval and inference.

[0022] After supervised fine-tuning is completed and the target multimodal large model has mastered basic structured output capabilities, this invention further employs reinforcement learning fine-tuning methods to guide the target multimodal large model to improve the accuracy and reliability of its output content. This embodiment preferably uses the Group Relative Policy Optimization (GRPO) algorithm, which achieves stable and efficient model optimization by constructing a comparison mechanism between a reference model group and the policy model to be optimized.

[0023] The optimization process is executed as follows: First, the target multimodal large model that has already undergone supervised fine-tuning is used as the benchmark for both the policy model to be optimized and the reference model. During training, multiple response outputs of the policy model and the reference model group to the same input data are obtained through parallel sampling. Subsequently, the quality of these outputs is evaluated based on a pre-defined composite reward function, and the optimization direction of the policy model is determined by comparing the relative quality of the responses within the group.

[0024] The design of the composite reward function includes two key dimensions: format conformance reward and answer correctness reward. The format conformance reward uses regular expression rules to strictly validate the text structure of the target multimodal large model's output, ensuring that it fully conforms to the preset structured template requirements, including elements such as field completeness, correct order, and delimiter conformance. The target multimodal large model can only receive this reward when the output meets all format specifications.

[0025] The correctness reward evaluates output quality from two levels: semantic accuracy and spatial localization accuracy. At the semantic level, the BERTS score is used to calculate the similarity between the hazard description output by the target multimodal large model and the standard answer in the semantic space. After converting the text into semantic vectors through a pre-trained language model, similarity comparison is performed to ensure that the hazard phenomenon described by the target multimodal large model is consistent with the facts at the semantic level. At the spatial localization level, the intersection-union ratio (IUGR) metric is used to quantify the degree of overlap between the bounding boxes predicted by the target multimodal large model and the ground truth bounding boxes. A reward is only given if the value reaches a preset threshold, thereby improving the target multimodal large model's ability to locate target objects.

[0026] Finally, the rewards for format conformity and answer correctness are merged according to preset weights to form the total reward signal driving the group's relative strategy optimization training. This reinforcement learning fine-tuning stage, together with the aforementioned supervised fine-tuning stage, constitutes a complete model optimization process. It is closely linked to the training dataset construction stage and the retrieval-enhanced multi-graph hazard prediction stage, forming a complete technical closed loop from data preparation and model training to practical application, ensuring the effectiveness and practicality of the method of this invention in industrial hazard prediction tasks.

[0027] To ensure the stability and effectiveness of the reinforcement learning fine-tuning process, this embodiment specifies the key parameters and reward weights of the group relative policy optimization algorithm: The learning rate for group relative strategy optimization training is set to 1×10. -6 The AdamW optimizer was used, with a sliding baseline coefficient of 0.95, a batch size of 32, and a reference model group response sample size of 4. In the composite reward function, the format normalization reward (R0) is used. format The weighting coefficient for the correct answer is set to 0.3, and the reward for correct answer (R) is... content The weighting coefficient for ) is set to 0.7. Within the reward for correct answer, the reward for semantic consistency of the hidden danger label (R) is... semantic ) and target bounding box localization accuracy bonus (R iou The weights are assigned to 0.6 and 0.4.

[0028] The format compliance bonus is a binary bonus, meaning that R is awarded when the format requirements are fully met. format =1.0 otherwise 0; the semantic consistency reward for potential hazard labels is directly taken from the F1 score calculated by BERTScore, i.e., R semantic =BERTScoreF1; The target bounding box localization accuracy bonus is set with an intersection-union ratio (IU) threshold of 0.5. When the IU is ≥ 0.5, R... iou =IoU; when the intersection-union ratio is <0.5, R iou =0.

[0029] Based on the above configuration, the total reward (R) of the optimized strategy model is used. total Calculate R using the following formula: total =0.3×R format +0.7×(0.6×R semantic +0.4×R iou ) This parameter combination has been verified through extensive experiments. It can effectively balance semantic accuracy and localization precision while encouraging the model to output in a standardized format, thereby guiding the improvement of the performance of the target multimodal large model.

[0030] III. Multi-image Hidden Danger Prediction Based on Retrieval Enhancement This stage involves deploying the fully refined, specialized model into practical applications. The complete process is as follows: Figure 3 As shown.

[0031] First, this invention inputs the industrial site image to be detected into a finely tuned target multimodal large model. The target multimodal large model utilizes its learned capabilities during training to quickly and accurately parse the image and output a set of structured results. These results include: a scene label (e.g., "chemical warehouse"), a list of detected entities (e.g., ["fire extinguisher", "stockpiled materials"]), and the bounding box coordinates corresponding to each entity in the list.

[0032] After obtaining the structured parsing results, the prediction process triggers two key sub-processes in parallel: the knowledge retrieval sub-process uses the parsed scene and entity tags as query keys to perform concurrent retrieval in a pre-built safety regulation vector knowledge base. Its retrieval mechanism is identical to that of the training phase, quickly recalling several (e.g., Top 5) regulatory text fragments most relevant to the current scene and each entity through vector similarity calculation; the multi-graph construction sub-process accurately crops the corresponding target region from the original test image based on the bounding box coordinates of each entity output by the model, forming an initial set of sub-graphs. Then, image enhancement processing, consistent with the training phase, is performed on these initial sub-graphs, specifically a magnification of at least two times, to ensure that the subsequent model can clearly perceive local details. Finally, a multi-graph input package is formed, consisting of one "original full image" and several "enhanced sub-graphs".

[0033] After knowledge retrieval and multi-graph construction are completed, the core multimodal fusion and hierarchical reasoning stage begins. The processing flow for this stage is as follows: Figure 4 As shown, the multi-image input package (full image + all enhanced sub-images) prepared in the previous step and the retrieved regulatory text (scene-level regulations + entity-level regulations) are input into the fine-tuned target multimodal large model. Inside the model, the deep visual features of the full image and each sub-image are extracted in parallel through a shared visual encoder. Through a cross-modal attention mechanism, global visual features, local visual features, and textual features are aligned and fused in the embedding space. This effectively integrates visual information from different granularities and regions to form a unified visual representation rich in global context and local details.

[0034] Simultaneously, all retrieved regulatory texts are encoded using a language encoder to obtain their textual features. Based on this, the target multimodal large model performs hierarchical reasoning and decision-making, integrating the global visual features provided by the full image and the professional knowledge carried by the scene-level regulatory texts to generate a scene-level hazard prediction result, denoted as A1. A1 focuses on judging the overall environmental compliance, such as "the safety passage is blocked by debris, hindering passage." For each detected entity, the target multimodal large model combines the portion of the fused visual features corresponding to the entity's sub-image (i.e., local details) with the relevant regulatory text features of the entity for focused analysis, generating a fine-grained hazard prediction result for that entity, denoted as A2 (e.g., A2-1, A2-2...). For example, for the "fire extinguisher" entity, its A2 result might be "the fire extinguisher pressure gauge pointer is in the red zone, indicating insufficient pressure."

[0035] The target multimodal large model intelligently integrates the scenario-level prediction result A1 with all entity-level prediction results A2, weighting and fusing them based on the severity, urgency, and impact range of different hazards to output a structured hazard prediction report that is complete in information, clearly structured, and can directly guide action. This report typically includes the following elements: hazard level (e.g., high risk, medium risk, low risk), hazard location (precisely indicated by bounding box coordinates or text description), hazard description (clearly explaining the hazard phenomenon), legal basis (directly citing the specific legal provisions or indexes violated), and remediation recommendations (providing specific and actionable rectification or remediation measures).

[0036] To comprehensively and quantitatively demonstrate the effectiveness of this method, this embodiment conducted a rigorous performance evaluation on a standardized test set consisting of 500 real industrial hazard images. This test set covers multiple typical hazard categories such as electrical safety, fire safety, and mechanical safety. This invention compares the fine-tuned target multimodal large model with the pre-trained multimodal large model as the baseline. In addition to accuracy, recall and F1 score, two crucial metrics in classification tasks, were introduced to comprehensively reflect the overall performance of the model. The results are shown in Table 2.

[0037] Table 2 Comparison of the accuracy of model hazard identification

[0038] As can be seen from the comprehensive comparison data in Table 2, the target multimodal large model used in this invention has only about one-tenth the number of parameters of the baseline model, but it surpasses the powerful baseline model in all three core indicators: accuracy, recall, and F1 score. The improvement in accuracy demonstrates the overall predictive correctness advantage of the method of this invention; the improvement in recall proves that the method of this invention can discover more real hidden dangers missed by the baseline model, effectively reducing the risk of false negatives; the overall increase in F1 score comprehensively reflects the balanced improvement of the model in accuracy and recall.

[0039] Example 2

[0040] This embodiment takes an outdoor tank area in a chemical industrial park as the object, combined with the attached... Figure 5 This paper describes the application of a multi-map hazard prediction method based on retrieval enhancement and full-map subgraphs in real working conditions.

[0041] like Figure 5 As shown, the system in this embodiment mainly includes: a tank area 1, surveillance cameras 2 deployed around the tank area and at key locations, an edge computing server 3 equipped with a target multimodal large model, a vector library 4 storing safety regulations and hazard knowledge, and a safety management platform 5 for displaying hazard results and issuing rectification work orders. Cameras 2 send the captured industrial site images to the edge computing server 3. After retrieving relevant knowledge from the vector library 4, the server 3 completes hazard identification based on the method of this invention and pushes the results to the safety management platform 5, realizing a complete process from image acquisition and intelligent identification to rectification closed loop.

[0042] The core method of this invention can be summarized as "three stages and one closed loop", namely: high-quality training data construction, fine-tuning of the target multimodal large model, retrieval-enhanced multi-graph hidden danger prediction, and the data-model-knowledge-application closed loop formed around these three parts.

[0043] First, in the training data construction phase, this embodiment extracts original images from historical inspection videos of tank area 1, covering multiple time periods, weather conditions, and perspectives. Using a pre-trained multimodal large-scale model with a large parameter scale, scene labels (such as "outdoor tank area," "tank unloading area," etc.), entity labels (such as "tank body," "safety valve," "level gauge," "drainage ditch," "hot work point," etc.), and corresponding target bounding boxes are automatically generated for these images. Then, using these labels as queries, vectorized clauses of the Safety Production Law, fire protection regulations, chemical industry standards, and enterprise safety systems pre-stored in vector library 4 are retrieved to obtain the regulatory fragments most relevant to the current scene and entities. The pre-trained multimodal large-scale model generates an initial hazard description under the condition of "image + regulatory fragment," and then compares its semantic similarity with manually written standard hazard descriptions, retaining only samples with similarity reaching a preset threshold. Simultaneously, key entity sub-images are cropped from the original images based on the bounding boxes, and these sub-images are enlarged, sharpened, and enhanced in contrast, particularly improving the visibility of details such as instrument scales and nameplate text. Through the above processing, a high-quality, structured training dataset containing "original full image, enhanced sub-image, labels, bounding boxes, and standard hazard descriptions" is finally constructed.

[0044] Secondly, in the fine-tuning stage of the target multimodal large model, this embodiment selects a multimodal model with a moderate parameter scale as the target model and deploys it on the edge computing server 3. Using the aforementioned training data, the model is fine-tuned with supervised instructions, enabling it to stably output standardized scene labels, entity lists, and corresponding bounding box coordinates under the given structured instruction template. This invention ensures that the output is in a machine-parseable structure, such as JSON or list format, facilitating subsequent retrieval and integration with business systems. Based on this, a reinforcement learning alignment method is further adopted, using indicators such as "whether the output format is standardized," "semantic consistency between the hazard description and the standard answer," and "intersection over union ratio (IoU) between the predicted bounding box and the true bounding box" to construct a composite reward function to optimize the target multimodal large model, thereby simultaneously improving its text output quality and spatial positioning accuracy, and reducing illusions and unfounded judgments.

[0045] In the enhanced multi-image hazard prediction phase, the system enters online operation. Camera 2 captures images from tank area 1 at preset time intervals and sends them to edge computing server 3. Server 3 uses a fine-tuned target multimodal large model to first analyze the entire image, automatically identifying scene categories and multiple entity targets, and outputting entity names and bounding box coordinates. Based on these bounding boxes, entity sub-images are cropped, enlarged, and enhanced to form a multi-image input set of "full image + multiple sub-images". Subsequently, server 3 uses the current scene label and entity label as queries to retrieve the most relevant regulatory texts to the vector library 4 for topics such as "outdoor tank area, hot work, dike management, and pressure vessel safety", obtaining scene-level and entity-level regulatory fragments respectively.

[0046] This invention employs a hierarchical, multimodal fusion reasoning mechanism in the reasoning phase: On one hand, it fuses the visual features of the entire image with scene-level regulatory text features to generate scene-level hazard prediction results A1, such as "significant water accumulation and oil contamination within the tank area dike, and the presence of material stockpiles." On the other hand, it fuses the fine-grained visual features of each entity sub-image with the corresponding entity-level regulatory fragments to generate entity-level hazard prediction results A2, such as "safety valve outlet is blocked," "liquid level indicator exceeds the safety limit," "drainage ditch is blocked," and "insufficient fire extinguishers are not provided in the hot work area." Finally, based on the severity of the hazard and the model confidence level, A1 and all A2 are weighted and fused to output a structured hazard report. The report clearly lists the hazard type, associated entities, spatial location, risk level, and corresponding regulatory clause number and summary description.

[0047] like Figure 5 As shown, the generated structured hazard results are transmitted to the safety management platform 5 via the network and visualized on the central control room's large screen or client interface in the form of lists, color-coded levels, and overlay selection boxes. The safety management platform 5 can automatically generate rectification work orders based on the hazard type and level, assign them to the corresponding responsible persons, and after rectification is completed, the feedback information is recorded and can be used as a data source for subsequent model retraining, thus forming a closed-loop iteration of "on-site data - knowledge base - model - business system".

[0048] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0049] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-graph hazard prediction method based on retrieval enhancement and full-graph subgraphs, characterized in that, include: Step 1, Training Dataset Construction: Use a pre-trained multimodal large model to process industrial field images and generate initial scene labels, entity labels, and target bounding boxes; The safety regulations knowledge base is retrieved based on the tags; the hazard descriptions generated by the pre-trained multimodal large model are semantically compared with the manually labeled answers to filter samples; at the same time, the sub-images are cropped according to the target bounding boxes and enlarged and enhanced; after verification, a training dataset containing the original image, enhanced sub-images, labels, bounding boxes and standard descriptions is constructed. Step 2, Fine-tuning and Training of the Target Multimodal Large Model: The target multimodal large model is fine-tuned in a supervised manner using the dataset constructed in Step 1, so that it follows the structured instruction template to output scene labels, entity labels and target bounding boxes; then, a reinforcement learning fine-tuning method is used to train the target multimodal large model based on a composite reward function consisting of format reward and answer reward. Step 3, multi-image hazard prediction based on retrieval enhancement: The image to be tested is analyzed using a fine-tuned target multimodal large model to obtain scene labels, entity labels and target bounding boxes; the target region is cropped and enlarged based on the target bounding boxes to form a multi-image input of the full image and sub-images; Based on tag-based retrieval of security regulations knowledge base; The multi-modal large model is subjected to hierarchical reasoning by inputting multiple images and searched texts into the target, and the prediction results at the scene level and entity level are integrated to output a structured report that includes the type, location and legal basis of the hazard.

2. The multi-graph hidden danger prediction method based on retrieval enhancement and full-graph subgraphs according to claim 1, characterized in that... The pre-trained multimodal large model and the target multimodal large model are instances of the same model architecture with different parameter scales, and the parameter scale of the pre-trained multimodal large model is larger than that of the target multimodal large model.

3. The multi-graph hidden danger prediction method based on retrieval enhancement and full-graph subgraphs as described in claim 1, characterized in that... In step one, the image enhancement processing includes: using a bicubic interpolation algorithm to enlarge the sub-image to 2 to 4 times its original size; using the Laplacian operator to sharpen the image to enhance edge features; and using an adaptive histogram equalization algorithm to enhance local contrast in areas containing equipment nameplates, instrument scales, or text labels.

4. The multi-graph hidden danger prediction method based on retrieval enhancement and full-graph subgraphs according to claim 1, characterized in that, In step two, the structured instruction template is configured to guide the target multimodal large model: first, it is determined whether the image scene belongs to the preset industrial scene list, and if it is determined to belong, the scene name is output; Subsequently, the target objects in the image are identified, and for each object, the object name and the corresponding bounding box coordinate array are output sequentially.

5. The multi-graph hidden danger prediction method based on retrieval enhancement and full-graph subgraphs according to claim 1, characterized in that, In step two, the reinforcement learning fine-tuning employs a population relative policy optimization algorithm. The format reward is given by matching the output of the target multimodal large model with regular expressions to see if it meets the required structured format.

6. The multi-graph hidden danger prediction method based on retrieval enhancement and full-graph subgraphs according to claim 5, characterized in that, The semantic consistency reward for the hazard label is obtained by calculating the BERTScoreF1 value between the hazard description output by the target multimodal large model and the standard answer; the intersection-union ratio (IUGR) localization accuracy reward for the target box is obtained by calculating the IUGR between the bounding box predicted by the target multimodal large model and the ground truth bounding box. When the IUGR between the predicted box and the ground truth box is lower than a preset threshold, this reward is zero.

7. The multi-graph hidden danger prediction method based on retrieval enhancement and full-graph subgraphs according to claim 1, characterized in that, The safety regulations knowledge base used in steps one and three consists of safety production laws and regulations, industry standards, and internal enterprise rules and regulations documents. First, the original documents are segmented, and each document segment is encoded into a high-dimensional dense vector using a text encoding model. The document segments and their vectors are then stored in a vector database. When searching based on scene tags and entity tags, the scene tags and entity tags are input into the same text encoding model to obtain the query vector. By calculating the similarity between the query vector and the document segment vector, the document segments with the highest similarity are selected from the vector database as scene-level and entity-level search results.

8. The multi-graph hidden danger prediction method based on retrieval enhancement and full-graph subgraphs according to claim 1, characterized in that, In step three, the hierarchical reasoning is as follows: the target multimodal large model extracts the visual features of the whole image and each sub-image in parallel through its visual encoder, and fuses the global scene visual features, the visual features of each local sub-image and the regulatory text features encoded by the language encoder through a cross-modal attention mechanism; based on the fused features, a scene-level hazard prediction result A1 describing the overall environmental compliance is first generated, and then for each entity, an entity-level hazard prediction result A2 combining its local visual details and specific regulations is generated.

9. The multi-graph hidden danger prediction method based on retrieval enhancement and full-graph subgraphs according to claim 1, characterized in that, In step three, the fusion of scenario-level and entity-level prediction results is based on a preset weighted average of the severity of the hazards, and the A1 and all A2 results are weighted to generate the final comprehensive hazard assessment.

10. The multi-graph hidden danger prediction method based on retrieval enhancement and full-graph subgraphs according to claim 1, characterized in that, The reinforcement learning fine-tuning of the group relative policy optimization can also be implemented using proximal policy optimization or other policy gradient-based reinforcement learning algorithms. The text encoding model can be a multilingual universal encoding model or other encoders with semantic vector representation capabilities. The vector database can be any vector database system with similar vector retrieval capabilities. When different reinforcement learning algorithms, text encoding models, or vector databases are used without changing the overall process of training dataset creation, model fine-tuning training, and multi-image hazard prediction based on retrieval enhancement as described in steps one to three of claim 1, the corresponding functions can be achieved using technologies known in the art without departing from the core inventive points of this invention.

Images