Method for identifying safety hazards in hazardous chemical industry based on multi-modal large model

Abstract

The application discloses a dangerous chemical industry safety hidden danger identification method based on a multi-modal large model, and belongs to the cross field of dangerous chemical safety and artificial intelligence. The method automatically constructs a three-layer associated knowledge base of "equipment-checking item-standard clause", and supports standard hot plug update. Three-stage cascade reasoning is adopted: visual-semantic dual-channel verification is introduced in equipment identification; visual checkability evaluation and environment self-adaptive adjustment are introduced in checking list matching; and "observation-contrast-conclusion" three-stage reasoning is introduced in hidden danger research and judgment. A structured report containing hidden danger description, standard traceability and rectification suggestion is generated. The safety hidden danger identification new paradigm of "standard driving, model execution" is effectively realized.

Description

Technical Field

[0001] This invention relates to the intersection of hazardous chemical safety management and artificial intelligence technology, specifically to an intelligent identification method for safety hazards in the hazardous chemical industry based on a multimodal large model. Background Technology

[0002] The hazardous chemicals industry is a vital pillar of the national economy, but it is also a high-risk area for workplace accidents. Taking LNG receiving terminals as an example, these involve numerous critical pieces of equipment, including cryogenic storage tanks, vaporizers, BOG compressors, loading arms, and flare systems. The operating environment is complex, and safety inspection requirements are extremely stringent. Currently, safety hazard identification in the industry primarily relies on the following methods: manual inspection, traditional image recognition, and rule-based engine-based methods.

[0003] Manual inspection relies on the personal experience of safety officers, who carry paper checklists to check each item one by one. This is inefficient, highly subjective, and prone to overlooking hidden hazards. Traditional image recognition methods, based on convolutional neural network (CNN) target detection methods (such as YOLO and Faster R-CNN), can identify specific equipment defects, but require training with a large number of separately labeled samples for each type of hazard, and cannot be linked to safety standards for compliance judgment. The rule engine approach hardcodes safety standards as condition-execution rules, resulting in high maintenance costs, the need for redevelopment when standards are updated, and poor scalability.

[0004] The existing technology suffers from the following major shortcomings: First, visual recognition is disconnected from standard knowledge. Most existing solutions follow a sequential "identify first, then look up" approach. The visual model only outputs the equipment category or defect type, and then an independent module matches the standard entries. Visual features and standard semantics cannot be deeply integrated, making it difficult to accurately identify complex hazards requiring comprehensive judgment (such as "valve markings do not meet the color code requirements specified in GB / T 7512"). Second, standard knowledge updates are lagging. The hazardous chemicals industry standard system is vast and continuously updated (a three-tiered system of national, industry, and enterprise standards). Existing systems require data re-annotation, model training, or rule modification when standards are updated, making plug-and-play updates of standard knowledge impossible. Third, there is a lack of fine-grained equipment-level inspection mechanisms. The safety inspection points for different equipment types vary significantly (e.g., LNG storage tanks focus on liquid level, pressure, and insulation integrity, while loading / unloading arms focus on sealing, alignment, and emergency detachment devices). Existing general-purpose detection models cannot dynamically load differentiated inspection checklists according to equipment type. Fourth, hazard identification results lack traceability. The hazard reports output by the existing system usually only have simple textual descriptions and lack corresponding standard clauses and sources of evidence, which does not meet the compliance requirements of safety regulatory authorities for "being verifiable and having standards to follow".

[0005] Among the existing publicly available technical documents, CN121146069A discloses a hazard investigation method based on a multimodal large model; CN120804618A discloses a hazard identification, monitoring, and early warning method based on multimodality; CN121146969A discloses a real-time safety hazard identification method based on a multimodal large model; CN119851204A discloses a hazard identification method for chemical equipment; CN121170993A discloses an intelligent identification and early warning method for chemical safety hazards; and CN121167777A discloses a safety production hazard identification method based on a large model. None of these existing technologies have achieved a systematic integration of three aspects: hot-swappable dynamic updating of standard knowledge, fine-grained equipment-level inspection, and standard traceability of hazard results. Summary of the Invention

[0006] Technical problems to be solved

[0007] To address the aforementioned shortcomings of existing technologies, this invention provides a method for identifying safety hazards in the hazardous chemicals industry based on a multimodal large model. This method solves the technical problems of visual recognition being disconnected from standard knowledge, difficulty in updating standards, coarse inspection granularity, and lack of traceability of hazard results in existing technologies, thereby realizing a new paradigm of safety hazard identification that is "standard-driven and model-executed".

[0008] Technical solution

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] A method for identifying safety hazards in the hazardous chemicals industry based on a multimodal large model includes the following steps:

[0011] S1. Standard Knowledge Parsing Steps: Receive hazardous chemical industry standard documents, extract the structure of the standard documents using a document parsing engine, perform semantic analysis on the standard text using a large language model, automatically extract equipment entities and inspection elements, and construct a three-layer related knowledge base of "equipment - inspection items - standard clauses"; wherein, the three-layer related knowledge base adopts a standard hot-swap mechanism, and when a new standard document is imported, the equipment list and inspection item list are automatically and incrementally updated through the document parsing engine;

[0012] S2. Equipment subject identification steps: Input the equipment images collected on site into the multimodal large model, and use the visual-semantic dual-channel verification mechanism to identify the equipment subject and determine the type of equipment in the image;

[0013] S3, Checklist Matching Step: Based on the device type determined in step S2, retrieve the corresponding checklist from the three-layer association knowledge base, and intelligently filter the check items in the checklist through visual inspectability assessment and adaptive adjustment of environmental factors to obtain the filtered checklist.

[0014] S4. Safety Hazard Assessment Steps: Based on the multimodal large model, the screened checklist is assessed item by item. A complete reasoning chain containing observed phenomena, standard comparisons, and reasoning conclusions is generated using a thinking chain reasoning mechanism. Each check item is assessed for compliance. The assessment results are classified into safety hazards according to the comparison between confidence level and preset threshold to obtain safety hazard results. The safety hazards include confirmed hazards, suspected hazards, and undiscovered hazards.

[0015] S5. Report generation step: Generate a structured report from the safety hazard results identified in step S4. The structured report includes a hazard description field, a hazard basis field, and a rectification suggestion field, wherein the hazard basis field is associated with the clause number and the original text of the corresponding standard.

[0016] Furthermore, the automatic extraction of device entities adopts a strategy of combining prompt word engineering with entity disambiguation. It guides the large language model to identify device nouns in standard text through a dedicated prompt word template, and eliminates ambiguity through synonym merging and hyponymy modeling. The automatic extraction of inspection elements extracts inspection items containing four attribute fields (inspection item number, inspection content description, judgment standard, and standard source) from the associated standard clauses for each device entity.

[0017] Furthermore, in step S2, the visual-semantic dual-channel verification mechanism specifically includes:

[0018] By using the visual channel, the visual understanding capability based on a multimodal large model is employed to perform a coarse classification of the device's appearance features, and a coarse classification result is obtained.

[0019] By using semantic channels, device identification information in images is extracted and matched based on optical character recognition technology. The device identification information includes device nameplate text and serial number labels.

[0020] The final device type is obtained by weighted fusion of the coarse classification result and the matching result. The weighted fusion dynamically adjusts the weights of the two channels based on whether the semantic channel successfully extracts the device nameplate text information. When the image contains multiple devices, the multimodal large model automatically performs device instance segmentation, performs subsequent inspection processes independently for each device, and records the spatial relationship between devices.

[0021] Furthermore, the three-layered knowledge base includes an equipment type layer, an inspection item layer, and a standard clause layer;

[0022] The device type layer includes key attributes such as device name, device code, region, and associated standard list;

[0023] The inspection item layer includes inspection item ID, inspection content, judgment criteria, inspection method, and priority;

[0024] The standard clause layer includes the standard number, clause number, original text of the clause, effective date, and whether it is mandatory.

[0025] Furthermore, the process of dynamically adjusting the weights of the two channels based on whether the semantic channel successfully extracts the text information of the device nameplate is as follows:

[0026] When the semantic channel successfully recognizes the text information on the device nameplate, the semantic channel weight is set higher than that of the visual channel, with the nameplate text information as the primary focus and appearance features as a secondary focus.

[0027] When the semantic channel fails to extract effective text information, the visual channel recognition result takes precedence and the semantic channel weight is reduced.

[0028] When the identification results from the two channels are inconsistent, the device is marked as "pending confirmation" and the user is prompted to verify.

[0029] Furthermore, in step S3, the visual detectability assessment is automatically performed by the multimodal large model based on the image content to determine whether each inspection item can be judged by the current image. Inspection items that cannot be judged by the image are automatically marked as "cannot be inspected in this image" and skipped to avoid misjudgment. The environmental factor adaptive adjustment dynamically adjusts the confidence threshold of each inspection item based on the image acquisition environment parameters in the image. The image acquisition environment parameters include lighting conditions, weather conditions, and shooting angle.

[0030] Further, in step S4, the comparison and grading rules between the confidence level and the preset threshold are as follows: when the confidence level is greater than the first preset threshold, it is determined as a confirmed hidden danger; when the confidence level is between the second preset threshold and the first preset threshold, it is determined as a suspected hidden danger and marked "recommended for on-site verification"; when the confidence level is not greater than the second preset threshold, it is determined as no hidden danger was found; wherein, the preset threshold includes a first preset threshold and a second preset threshold, and the first preset threshold is greater than the second preset threshold; the first preset threshold and the second preset threshold are determined by statistical analysis of the historical labeled detection sample dataset, and the optimal threshold combination is determined by cross-validation with the optimization objective of maximizing the harmonic mean F1 of precision and recall, and is adjusted according to the safety level requirements of different hazardous chemical scenarios.

[0031] Furthermore, the thought chain reasoning mechanism includes the following steps:

[0032] The region of visual interest related to the current inspection item is located in the image and key visual features are extracted. These key visual features are automatically extracted through the self-attention mechanism of the multimodal large model visual encoder, including spatial features of device surface edge contours, texture distribution, and color anomalies, to generate a description of the observed phenomenon.

[0033] The extracted key visual features are compared with the judgment criteria of the inspection items one by one to generate a standard comparison analysis;

[0034] Based on the sufficiency of the extracted key visual features and the degree of conformity between the visual features and the standard judgment conditions, a compliance conclusion and a confidence score are output.

[0035] Furthermore, it also includes multiple rounds of interactive in-depth inspection steps: allowing users to follow up and conduct in-depth inspections of the security risks output by the S4 step in the following ways: zooming in on a specified area in the image for analysis and re-executing the inspection process; incorporating supplementary text information provided by the user into the reasoning context for comprehensive judgment; and performing correlation analysis on multiple images from the same device to conduct a comprehensive inspection by integrating information from multiple shooting angles.

[0036] Furthermore, in step S1, the standard document includes three levels: national standards, industry standards, and enterprise standards. The standard document is in PDF and Word formats. The document parsing engine uses optical character recognition and layout analysis technology to perform structured extraction of the standard document, including standard number and name extraction, chapter hierarchy structure recognition, extraction of the correspondence between clause content and clause number, and structured conversion of table and appendix content.

[0037] Beneficial effects

[0038] The present invention has the following beneficial effects:

[0039] (1) This invention proposes a "standard hot-swap" mechanism, which takes standard documents as external knowledge input and automatically parses them through a large language model to generate a structured knowledge base. When the standard is updated, there is no need to retrain the model, realizing a system architecture of "one-time deployment and continuous evolution", which significantly reduces the system maintenance cost and solves the problem of difficulty in updating standard knowledge in the prior art.

[0040] (2) The present invention proposes a device identification method with visual-semantic dual-channel verification, which integrates the device appearance features and nameplate text information for fusion identification. It significantly improves the accuracy and robustness of device identification in complex industrial scenarios, especially in scenarios where the devices have similar appearances or are partially obscured.

[0041] (3) This invention proposes two technologies: visual detectability assessment and adaptive adjustment of environmental factors, which enable the system to maintain stable detection performance under the changing conditions of real industrial environment, and effectively reduce false alarm rate while ensuring detection rate.

[0042] (4) This invention introduces the thinking chain reasoning mechanism into the field of safety hazard identification, enabling the multimodal large model to explicitly generate a complete reasoning chain when conducting hazard assessment, giving the identification results a high degree of interpretability and meeting the compliance requirements of safety production supervision for "judgment with verifiable evidence".

[0043] (5) In the structured report implemented by this invention, each hazard description is precisely linked to a specific standard clause, realizing a complete traceability link of "hazard - standard". In the pilot application at the LNG receiving station, the test set containing 200 on-site equipment pictures was verified, covering core equipment such as storage tanks, vaporizers, compressors, and loading arms. The hazard detection rate reached 91.2%, and the standard traceability rate reached 100%, which is about 35 percentage points higher than the detection rate of traditional manual inspection methods. Attached Figure Description

[0044] Figure 1 This is the overall flowchart of the method for identifying safety hazards in the hazardous chemicals industry based on a multimodal large model according to the present invention. S1 to S5 correspond to five steps: standard knowledge parsing, equipment identification, checklist matching, safety hazard assessment, and report generation, respectively.

[0045] Figure 2 This is a schematic diagram of the three-layer related knowledge base structure of "equipment - inspection item - standard clause" of the present invention. The first layer is the equipment type layer, the second layer is the inspection item layer, and the third layer is the standard clause layer. The layers are connected by association relationships.

[0046] Figure 3 This is a schematic diagram of the three-stage cascaded reasoning process of the present invention, namely "equipment main body identification - checklist matching - item-by-item safety assessment", which shows the complete reasoning link from equipment image input to hidden danger assessment output.

[0047] Figure 4 This is a schematic diagram of the four-layer system architecture of the present invention, which consists of, from bottom to top, the data access layer, the knowledge engine layer, the inference engine layer, and the application service layer. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. As those skilled in the art will understand, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Please see Figures 1 to 4 This invention provides a method for identifying safety hazards in the hazardous chemicals industry based on a multimodal large model, comprising the following steps:

[0050] Specifically, the S1 step—standard knowledge parsing and knowledge base construction—is as follows:

[0051] The system receives relevant standard documents from the hazardous chemicals industry as input. These standard documents cover three levels: national standards (GB / T), industry standards (NB / T, SY / T), and enterprise standards. The standard documents support PDF and Word formats.

[0052] The preprocessing stage uses a document parsing engine (based on optical character recognition and layout analysis technology) to extract the structure of standard documents, including: standard number and name extraction, chapter hierarchy structure recognition, extraction of the correspondence between clause content and clause number, and structured conversion of table and appendix content.

[0053] A large language model is used to perform semantic analysis on the preprocessed standard text, automatically extracting the following information:

[0054] Equipment Entity Extraction: Identify and extract all equipment type entities ("LNG storage tank", "BOG compressor", "high-pressure vaporizer", "loading boom", "flare system") from standard text to construct an equipment entity list. The extraction process employs a "cue word engineering + entity disambiguation" strategy. A dedicated cue word template is designed to guide the large language model to identify equipment terms in the standard text, and ambiguity is eliminated through synonym merging and hierarchical relationship modeling (e.g., "cryogenic storage container" and "LNG storage tank" are merged into the same entity).

[0055] Inspection Element Extraction: For each equipment entity, inspectable items are extracted from the associated standard clauses. Each inspection item contains four attribute fields: inspection item number (automatically generated), inspection content description (inspection requirements described in natural language), judgment criteria (specific judgment conditions for pass or fail), and standard source (standard number, clause number, and original clause content).

[0056] The extracted results are organized into a three-tiered knowledge base: "Equipment - Inspection Items - Standard Clauses," such as... Figure 2 As shown. The first layer is the equipment type layer, which records the equipment name, equipment code, region, and key attributes of the associated standard list; the second layer is the inspection item layer, which records the inspection item ID, inspection content, judgment criteria, inspection method (visual or instrumental), and priority; the third layer is the standard clause layer, which records the standard number, clause number, original text of the clause, effective date, and whether it is mandatory.

[0057] This knowledge base supports a standard hot-swap mechanism: when a new standard document is imported into the system, the standard parsing engine automatically runs the above parsing process, incrementally updating the equipment list and checklist without manual intervention or model retraining. Taking an LNG receiving terminal as an example, after automatically parsing relevant standard documents including GB / T 26978 "Safety Technical Specifications for LNG Receiving Terminals" and NB / T 10838 "Operation and Maintenance Specifications for LNG Receiving Terminals," the system generates an equipment entity list containing 32 categories of core equipment and a checklist of 586 check items.

[0058] Specifically, the process of step S2 – device entity identification – is as follows:

[0059] The equipment images collected on-site are input into a multimodal large model, and equipment entity recognition is performed, such as... Figure 3 As shown, the multimodal large model receives the original image, first performs global scene understanding, and identifies the main device objects in the image and their spatial relationships.

[0060] A dual-channel visual-semantic verification mechanism is employed for equipment classification and matching: the visual channel, based on the visual understanding capabilities of a multimodal large-scale model, performs coarse classification of equipment appearance features (shape, color, size ratio, and signage) to obtain a coarse classification result; the semantic channel, based on optical character recognition (OCR) technology, identifies visible text information (equipment nameplates, serial number labels) in the image for precise matching; the results from the two channels are then weighted and fused to determine the final equipment type. The specific weighting method is as follows: when the semantic channel successfully identifies the text information on the equipment nameplate, its weight is set higher than that of the visual channel, prioritizing the nameplate information and secondarily considering appearance features; when the semantic channel fails to extract valid text information, the visual channel's recognition result takes precedence, and the semantic channel's weight is reduced; when the recognition results from the two channels are inconsistent, the system marks the equipment as "pending confirmation" and prompts the user for manual verification. This fusion strategy improves recognition accuracy and robustness, especially showing significant advantages in scenarios where equipment appearances are similar (e.g., valves of different specifications) or partially obscured.

[0061] When an image contains multiple devices, the model automatically segments the devices into instances, performs subsequent inspection processes independently for each device, and records the spatial relationships between devices (such as "valves are located at pipe connections") to provide contextual information for identifying potential related hazards.

[0062] Taking the inspection of an LNG receiving terminal as an example, the model identified the main equipment in the image as the "high-pressure vaporizer (ORV)" with a confidence level of 0.93. It also identified the auxiliary equipment "inlet valve" and "temperature sensor".

[0063] Specifically, the S3 step—checking the list for matches—is as follows:

[0064] Based on the device type identified in step S2, the corresponding checklist is retrieved from the three-layer association knowledge base, and a context-aware check item filtering mechanism is introduced.

[0065] After precise matching based on equipment type, all inspection items corresponding to that equipment type are extracted. Then, a visual inspectability assessment is performed on the inspection items: the multimodal large model automatically determines whether each inspection item can be judged based on the image content. For inspection items that cannot be judged based on the image (such as "corrosion of the inner wall of the storage tank" which requires endoscopic inspection), they are automatically marked as "cannot be inspected in this image" and skipped to avoid misjudgment.

[0066] Meanwhile, the confidence threshold for the inspection items is adaptively adjusted based on the image acquisition environment parameters (lighting conditions, weather conditions, shooting angle) in the image. For example, for images taken at night or in foggy weather, the confidence requirement for color-related inspection items (such as pipe color mark recognition) is reduced and a "recommended daytime review" label is added.

[0067] Taking the aforementioned high-pressure vaporizer as an example, the system loads 47 corresponding inspection items from the knowledge base, and retains 28 inspection items that can be determined by images after visual inspection filtering; at the same time, corresponding inspection items are loaded for auxiliary equipment (15 inlet valves and 8 temperature sensors).

[0068] Specifically, the S4 step—the process of assessing each safety hazard—is as follows:

[0069] The multimodal large model performs a safety hazard assessment on each item of the screened checklist. For each check item, the following process is executed:

[0070] Visual evidence extraction: The model locates the Region of Interest (ROI) in the image that is relevant to the current inspection item and extracts key visual features. For example, for the inspection item "the valve handwheel should be intact and free of rust", the model locates the valve handwheel area and analyzes its surface texture and color change features.

[0071] Standard-Visual Joint Reasoning: The model employs a chain-of-thought reasoning mechanism to explicitly generate the reasoning process: first, the observed visual phenomenon is described; then, it is compared against each judgment criterion; finally, a compliance conclusion is reached. This mechanism improves the interpretability and accuracy of the reasoning.

[0072] The specific process of the thought chain reasoning mechanism is as follows: Locate the visual interest region related to the current inspection item in the image and extract key visual features. The key visual features are automatically extracted through the self-attention mechanism of the multimodal large model visual encoder, including the spatial features of the device surface edge contour, texture distribution and color anomaly, and generate a description of the observed phenomenon.

[0073] The extracted key visual features are compared with the judgment criteria of the inspection items one by one to generate a standard comparison analysis;

[0074] Based on the sufficiency of the extracted key visual features and the degree of conformity between the visual features and the standard judgment conditions, a compliance conclusion and a confidence score are output.

[0075] Confidence Assessment and Grading: The assessment results for each inspection item are evaluated for confidence level, and the results are categorized into three levels based on the confidence level: "Confirmed Hidden Danger" (confidence level greater than 0.85 in this embodiment, i.e., the first preset threshold is 0.85), "Suspected Hidden Danger" (confidence level between 0.60 and 0.85 in this embodiment, i.e., the second preset threshold is 0.60, marked "On-site verification recommended"), and "No Hidden Danger Found" (confidence level not greater than 0.60). In other application scenarios, the above thresholds can be adjusted according to the detection accuracy requirements and false alarm tolerance. For example, for high-risk equipment, the first preset threshold can be lowered to 0.80 to improve the detection rate.

[0076] Taking the inspection item "no obvious deformation or damage to the fins of the high-pressure gasifier" as an example, the reasoning process of the model is as follows: [Observation Phenomenon] The gasifier fins in the picture are neatly arranged, but there are 2 to 3 fins in the middle area of ​​the third row that are obviously bent outwards at an angle of about 15 degrees, deviating from the normal arrangement direction; [Standard Comparison] Clause 7.2.3 of NB / T 10838-2021 stipulates: "The fins of open-frame gasifiers should be neatly arranged and should not have obvious bending, damage or corrosion"; [Reasoning Conclusion] The observed fin bending phenomenon does not meet the standard requirements and is judged as a hidden danger. Bending the fins will reduce the gasification efficiency and may cause stress concentration. It is recommended to classify it as a general hidden danger for rectification. The confidence level is 0.88, which is a confirmed hidden danger.

[0077] In addition, this implementation also includes multiple rounds of interactive in-depth inspection steps: allowing users to follow up and conduct in-depth inspections of the security risk results output by step S4 in the following ways: zooming in on a specified area in the image for analysis and re-executing the inspection process; incorporating supplementary text information provided by the user into the reasoning context for comprehensive judgment; and performing correlation analysis on multiple images from the same device to conduct a comprehensive inspection by integrating information from multiple shooting angles.

[0078] Specifically, the S5 step—the process of generating structured reports—is as follows:

[0079] For each identified safety hazard, the system automatically generates a structured hazard report, which includes the following three core fields:

[0080] Hazard Description Field: Provides a clear and specific natural language description of the identified safety hazard, including the hazard location, hazard symptoms, and severity assessment. Example: "There is a bulge deformation of approximately 30cm × 20cm on the outer surface of the north insulation layer of LNG storage tank T-101. Some insulation material has fallen off, exposing the inner metal wall. Assessment level: Major hazard."

[0081] Hazard Basis Field: Precisely cites the relevant standard clause number and original text to provide complete compliance judgment basis. Example: "According to Clause 6.3.2 of GB / T 26978-2020 'Safety Technical Specification for LNG Receiving Stations': The cold insulation layer of cryogenic storage tanks should be kept intact, and there should be no cracks, bulges, or peeling defects on the surface; furthermore, according to Clause 8.4.1 of NB / T 10838-2021: The cold insulation structure of the outer surface of the storage tank should be inspected regularly, and any damage should be repaired promptly."

[0082] The rectification suggestion field provides targeted rectification measures, including rectification methods, time limits, and precautions. Example: "1) Immediately carry out temporary protective treatment on the damaged area to prevent moisture intrusion and further cold loss; 2) Arrange for a professional cold insulation construction team to carry out repairs within 7 days, and the repair materials should meet the requirements of GB / T 17794; 3) Conduct infrared thermal imaging detection after repair to confirm the cold insulation effect."

[0083] During this inspection of the LNG receiving terminal, a total of 3 confirmed potential hazards and 5 suspected potential hazards were identified. The safety officer inquired further about one of the suspected hazards ("suspected leakage at the bottom pipe interface of the vaporizer"), and after uploading close-up photos of the area, the system confirmed through magnified analysis that it was rust stains rather than a medium leak. The item was then changed to "no hazard found," effectively avoiding false alarms.

[0084] Specifically, the system architecture is implemented as follows: Figure 4 As shown:

[0085] The data access layer is responsible for accessing standard files (supporting PDF, Word, and scanned document optical character recognition processing) and collecting and uploading on-site images (supporting collection methods such as mobile phones, tablets, drones, and fixed cameras).

[0086] The knowledge engine layer includes a standard parsing engine and a three-layer associated knowledge base, which is responsible for the automatic parsing of standard documents, extraction of device entities, generation of check items, and dynamic updates and version management of the knowledge base.

[0087] The inference engine layer is based on a multimodal large model, integrating equipment identification, checklist matching, and hazard assessment modules to execute three-stage cascaded inference. The inference engine supports cue word engineering configuration, controlling the inference process and output format through structured cue word templates.

[0088] The application service layer provides user interfaces for web and mobile devices, supporting functions such as image uploading, viewing inspection results, multi-round interactive follow-up questions, exporting hazard reports, and managing historical inspection records.

[0089] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0090] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. As those skilled in the art will understand, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for identifying safety hazards in the hazardous chemicals industry based on a multimodal large model, characterized in that, Includes the following steps: S1. Standard Knowledge Parsing Steps: Receive hazardous chemical industry standard documents, extract the structure of the standard documents using a document parsing engine, perform semantic analysis on the standard documents using a large language model, automatically extract equipment entities and inspection elements, and construct a three-layer related knowledge base of "equipment - inspection items - standard clauses"; wherein, the three-layer related knowledge base adopts a standard hot-swap mechanism, and when a new standard document is imported, the equipment list and inspection item list are automatically and incrementally updated through the document parsing engine; S2. Equipment subject identification step: Input the equipment images collected on site into the multimodal large model, and use the visual-semantic dual-channel verification mechanism to identify the equipment subject and determine the type of equipment in the image; when the image contains multiple devices, the multimodal large model automatically performs equipment instance segmentation, performs subsequent inspection processes independently for each device and records the spatial relationship between devices, and uses the spatial relationship between devices as contextual information for the judgment of related hidden dangers; S3. Checklist Matching Step: Based on the device type determined in step S2, retrieve the corresponding checklist from the three-layer associated knowledge base, and intelligently filter the checklist items through visual inspectability assessment and adaptive adjustment of environmental factors to obtain a filtered checklist; the visual inspectability assessment is automatically determined by the multimodal large model based on the image content to determine whether each checklist item can be judged by the current image, and checklist items that cannot be judged by the image are automatically marked as "cannot be checked in this image" and skipped; the adaptive adjustment of environmental factors dynamically adjusts the confidence threshold of each checklist item based on the image acquisition environment parameters in the image, including lighting conditions, weather conditions, and shooting angle. S4. Safety Hazard Assessment Steps: Based on the multimodal large model, each item on the screened checklist is assessed. A thought chain reasoning mechanism is used to generate a complete reasoning chain containing observed phenomena, standard comparisons, and reasoning conclusions. The thought chain reasoning mechanism includes the following steps: Locating the visual region of interest related to the current inspection item in the image and extracting key visual features. The key visual features are automatically extracted through the self-attention mechanism of the multimodal large model visual encoder, including spatial features of device surface edge contours, texture distribution, and color anomalies, generating a description of the observed phenomena; Comparing the extracted key visual features with the judgment criteria of each inspection item to generate a standard comparison analysis; Outputting a compliance conclusion and confidence score based on the sufficiency of the extracted key visual features and the degree of conformity between the visual features and the standard judgment conditions; And performing a compliance judgment on each inspection item. The judgment results are classified into safety hazards according to the comparison results between the confidence score and the preset threshold after adaptive adjustment by the environmental factors, to obtain the safety hazard results. The safety hazards include confirmed hazards, suspected hazards, and undiscovered hazards. S5. Report generation step: Generate a structured report from the safety hazard results identified in step S4. The structured report includes a hazard description field, a hazard basis field, and a rectification suggestion field, wherein the hazard basis field is associated with the clause number and the original text of the corresponding standard.

2. The method according to claim 1, characterized in that, In step S1, the automatic extraction of equipment entities adopts a strategy of combining prompt word engineering with entity disambiguation. It guides the large language model to identify equipment names in the standard document through a dedicated prompt word template, and eliminates ambiguity through synonym merging and hierarchical relationship modeling. The automatic extraction of inspection elements extracts inspection items containing four attribute fields (inspection item number, inspection content description, judgment standard, and standard source) from the associated standard clauses for each equipment entity.

3. The method according to claim 1, characterized in that, In step S2, the visual-semantic dual-channel verification mechanism specifically includes: By using the visual channel, the visual understanding capability based on a multimodal large model is employed to perform a coarse classification of the device's appearance features, and a coarse classification result is obtained. By using semantic channels, device identification information in images is extracted and matched based on optical character recognition technology. The device identification information includes device nameplate text and serial number labels. The final device type is obtained by weighted fusion of the coarse classification result and the matching result. The weighted fusion dynamically adjusts the weights of the two channels based on whether the semantic channel successfully extracts the text information of the device nameplate.

4. The method according to claim 1, characterized in that, The three-layered knowledge base includes an equipment type layer, an inspection item layer, and a standard clause layer. The device type layer includes key attributes such as device name, device code, region, and associated standard list; The inspection item layer includes inspection item ID, inspection content, judgment criteria, inspection method, and priority; The standard clause layer includes the standard number, clause number, original text of the clause, effective date, and whether it is mandatory.

5. The method according to claim 3, characterized in that, The weighted fusion process, which dynamically adjusts the weights of the two channels based on whether the semantic channel successfully extracts the text information from the device nameplate, is as follows: When the semantic channel successfully recognizes the text information on the device nameplate, the semantic channel weight is set higher than that of the visual channel, and the recognition is based primarily on the text information on the nameplate and secondarily on the appearance features. When the semantic channel fails to extract valid text information, the visual channel recognition result takes precedence, and the weight of the semantic channel is reduced. When the identification results from the two channels are inconsistent, the device is marked as "pending confirmation" and the user is prompted to verify.

6. The method according to claim 1, characterized in that, In step S4, the process of classifying safety hazards based on the comparison between the confidence level and the preset threshold is as follows: when the confidence level is greater than the first preset threshold, it is determined to be a confirmed hazard; when the confidence level is between the second preset threshold and the first preset threshold, it is determined to be a suspected hazard and marked "recommended for on-site verification"; When the confidence level is not greater than the second preset threshold, it is determined that no hidden danger has been found; wherein, the preset threshold includes a first preset threshold and a second preset threshold, and the first preset threshold is greater than the second preset threshold; The first preset threshold and the second preset threshold are determined by statistical analysis of the historical labeled detection sample dataset. The optimal threshold combination is determined by cross-validation with the optimization objective of maximizing the harmonic mean F1 of precision and recall. The thresholds are then adjusted according to the safety level requirements of different hazardous chemical scenarios.

7. The method according to claim 1, characterized in that, It also includes multiple rounds of interactive in-depth inspection steps: allowing users to follow up and conduct in-depth inspections of the security risks output by the S4 step in the following ways: zooming in on a specified area in the image for analysis and re-executing the inspection process; incorporating supplementary text information provided by the user into the reasoning context for comprehensive judgment; and performing correlation analysis on multiple images from the same device to conduct a comprehensive inspection by integrating information from multiple shooting angles.

8. The method according to claim 1, characterized in that, In step S1, the standard documents include three levels: national standards, industry standards, and enterprise standards. The standard documents are in PDF and Word formats. The document parsing engine uses optical character recognition and layout analysis technology to extract the structure of the standard documents, including standard number and name extraction, chapter hierarchy structure recognition, extraction of the correspondence between clause content and clause number, and structured conversion of table and appendix content.

Images