Industrial hidden danger intelligent identification system based on multi-modal information fusion and RAG mechanism

Abstract

The invention discloses an industrial hidden danger intelligent identification system based on multi-modal information fusion and an RAG mechanism, which relates to the technical field of artificial intelligence and industrial safety and comprises an enterprise portrait information and real-time image acquisition module, a multi-modal semantic extraction module, a dynamic prompt generation module and a related basis recall module. According to the method, a triple information fusion mechanism of an enterprise portrait, an equipment knowledge graph and real-time image semantics is constructed, and a dynamic prompt generation and basis retrieval mechanism is combined, so that context perception recognition and basis support judgment of hidden dangers are realized, the recognition accuracy and specialty are improved, and the recognition efficiency is improved. And the method also has excellent interpretation capability and scene adaptive capability.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and industrial safety technology, and more specifically, to an intelligent industrial hazard identification system based on multimodal information fusion and RAG mechanism. Background Technology

[0002] With the increasing demands for intelligent manufacturing and safe production, traditional hazard identification methods can no longer meet the needs of complex and ever-changing industrial scenarios. While existing computer vision-based identification methods have made some progress in image target detection and surface defect recognition, they still exhibit significant shortcomings when dealing with industry-specific hazard characteristics and personalized equipment scenarios. This is mainly reflected in the following two aspects:

[0003] Insufficient ability to identify industry-specific equipment: Traditional image recognition models struggle to accurately identify specialized equipment in the industrial field (such as eyewash stations, pressure vessels, and pipelines), and are unable to understand the relationships and operating status between equipment, resulting in blind spots in the identification of potential equipment hazards.

[0004] Lack of hazard knowledge base: Even if the equipment can be identified, the system lacks professional hazard knowledge for different equipment and cannot determine whether the equipment has specific hazards such as aging, blockage, illegal modification, or missing safety devices.

[0005] Therefore, there is an urgent need for an identification system to solve the above problems. Summary of the Invention

[0006] This invention overcomes the shortcomings of existing technologies and provides an intelligent industrial hazard identification system based on multimodal information fusion and RAG mechanism to enhance generalization ability and improve accuracy.

[0007] The technical solution of the present invention is as follows:

[0008] An intelligent industrial hazard identification system based on multimodal information fusion and RAG mechanism includes the following modules:

[0009] The enterprise profile information and real-time image acquisition module is used to access and integrate enterprise profile information, which includes basic enterprise information dimensions and security feature dimensions. The basic information dimension covers industry type, geographical location, main products or services, and employee size, while the security feature dimension covers security level and characteristics of high-risk locations. At the same time, this module acquires on-site image data through real-time image acquisition equipment.

[0010] The multimodal semantic extraction module is used to process real-time image data, including image normalization, image block partitioning, image block expansion, dimensional transformation, positional encoding fusion, multi-layer feature modeling, and device sub-graph recall based on cosine similarity. The device sub-graph recall includes graph entity encoding and candidate entity screening, and the recall results include device attribute relationships and hazard description information. The multi-layer feature modeling module adopts a stacked structure of encoding sub-modules, including a multi-head self-attention sub-module, a residual connection and layer normalization structure, and a feedforward neural network sub-module, which are used to generate L2-normalized visual semantic vectors.

[0011] The dynamic prompt generation module is used to generate structured prompt words based on the enterprise profile information, recalled equipment sub-graph information and real-time image semantic information. The prompt words adopt a triple template format: [enterprise profile information] + [graph matching equipment features] + [image extraction semantics], and are input into the large language model LLM for hazard identification.

[0012] The relevant evidence recall module is used to retrieve relevant evidence from the database based on the type of hazard and contextual information after the hazard is located, and output the hazard description, rectification measures and relevant evidence standards to form a complete identification result.

[0013] Furthermore, the enterprise profile information and the security feature dimension in the real-time image acquisition module include the identification of confined spaces, high temperature, high pressure or high dust risk locations, and the image size output by the real-time image acquisition module is normalized to 224×224 pixels.

[0014] Furthermore, the multimodal semantic extraction module specifically performs the following steps:

[0015] 1.1) Image normalization step: Adjust the input image to a resolution of 224×224;

[0016] 1.2) Image block division step: Divide the image into N non-overlapping image blocks with a fixed window size P×P, where N=(H / P) × (W / P), H and W are the image height and width, respectively; P takes the value 16;

[0017] 1.3) Image patch unfolding and dimension transformation steps: flatten each image patch into a one-dimensional vector and linearly project it onto the preset embedding dimension D, where D is 1024;

[0018] 1.4) Position encoding fusion step: Add learnable position encoding vectors to preserve spatial structure information;

[0019] 1.5) Multi-layer feature modeling steps: L-layer encoding sub-modules are used for context modeling. Each encoding sub-module includes multi-head self-attention calculation, residual connection and layer normalization, and feedforward neural network, which outputs L2-normalized visual semantic vectors.

[0020] Furthermore, the cosine similarity threshold in the device subgraph recall is set to 0.8 to filter candidate entities. If the similarity is lower than 0.8, the device graph information is not passed in.

[0021] Furthermore, the dynamic prompt generation module generates prompt words that include the enterprise profile context and prior knowledge of equipment potential hazards, and inputs them into a multimodal large language model for accurate annotation and description.

[0022] Furthermore, the output format of the relevant evidence recall module is a structured result, including a description of the location of the hidden danger, rectification measures, and relevant evidence standards, to ensure that the result is supported by laws and regulations and to achieve end-to-end identification from image acquisition to evidence output.

[0023] Furthermore, the results of the hazard identification include marking the location of the hazard on the map and providing rectification suggestions.

[0024] Furthermore, the feedforward neural network submodule in the multi-layer feature modeling module adopts the GELU activation function and Dropout operation, with the intermediate dimension set to 4D, where D is the embedding dimension, to enhance the nonlinear expressive power.

[0025] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the system described above.

[0026] A computer-readable storage medium storing a computer program that, when executed, implements the system as described above.

[0027] The advantages of this invention compared to the prior art are:

[0028] This invention achieves context-aware identification and evidence-based judgment of potential hazards by constructing a triple information fusion mechanism of enterprise profiling, equipment knowledge graph, and real-time image semantics, combined with dynamic prompt generation and evidence retrieval mechanisms. This method not only improves the accuracy and professionalism of identification but also possesses excellent interpretability and scenario adaptability, making it particularly suitable for automatic identification of safety hazards in high-risk industries, complex equipment systems, and customized scenarios. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0030] Figure 2 This is a partial flowchart of the multimodal semantic extraction module of the present invention. Detailed Implementation

[0031] Embodiments of the present invention are described in detail below, wherein the same or similar reference numerals denote the same or similar elements or elements with similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as limiting the present invention.

[0032] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0033] The numbering of steps mentioned in the various embodiments is merely for descriptive convenience and does not imply a sequential relationship. Different steps in various specific embodiments can be combined in different orders to achieve the inventive objective of this invention.

[0034] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0035] like Figure 1 , 2 As shown, an intelligent industrial hazard identification system based on multimodal information fusion and RAG mechanism includes the following modules:

[0036] The enterprise profile information and real-time image acquisition module is used to access and integrate enterprise profile information, which includes basic enterprise information dimensions and security feature dimensions. The basic information dimension covers industry type, geographical location, main products or services, and employee size, while the security feature dimension covers security level and characteristics of high-risk locations. Simultaneously, this module acquires on-site image data through real-time image acquisition equipment. In other words, this module primarily constructs a semantic context tailored to specific enterprise security scenarios, providing precise contextual support for dynamic prompt generation through structured collection and integration of relevant enterprise information.

[0037] The basic information dimension of an enterprise specifically includes the enterprise type, i.e., the industry it operates in (such as chemical, power, metallurgy, etc.), geographical location (such as being located in environments with high temperature, high humidity, coastal areas, or lots of dust, where machinery is prone to corrosion), main products or services, and the size of its workforce.

[0038] Safety characteristics dimensions: covering the enterprise's safety level (such as the classification of high-risk industries) and whether it includes high-risk locations such as confined spaces, high temperatures, high pressures, and dust.

[0039] By integrating the above information, the system can construct enterprise-level security semantic prior knowledge, ensuring the comprehensiveness and timeliness of the profile information. Simultaneously, the system accesses the enterprise's profile information and acquires on-site images through a real-time image acquisition module.

[0040] The multimodal semantic extraction module processes real-time image data, including image normalization, image block segmentation, image block expansion, dimensional transformation, positional encoding fusion, multi-layer feature modeling, and equipment sub-graph recall based on cosine similarity. Equipment sub-graph recall includes graph entity encoding and candidate entity filtering, and the recall results include equipment attribute relationships and hazard description information. The multi-layer feature modeling module adopts a stacked structure of encoding sub-modules, including a multi-head self-attention sub-module, a residual connection and layer normalization structure, and a feedforward neural network sub-module, used to generate L2-normalized visual semantic vectors. In other words, by combining multimodal features from images and text, it achieves accurate identification and structured semantic understanding of on-site equipment, and efficiently recalls corresponding equipment sub-graph information based on the extraction results, supporting intelligent identification and analysis of industrial site hazards.

[0041] The multimodal semantic extraction module specifically performs the following steps:

[0042] 1.1) The image normalization step is to normalize the input real-time image data, mainly by adjusting the image size to uniformly adjust the input image to the size required by the model (224*224) to ensure that the input vector dimension is consistent.

[0043] 1.2) Image block partitioning step: Divide the image into N non-overlapping image blocks with a fixed window size P×P, where N=(H / P) × (W / P), and H and W are the image height and width, respectively. Generally, P is taken as 16.

[0044] Specifically, a sliding sampling operation is performed on the input image using a fixed window size, extracting one image region at a time. When P=16, with an input image of 224×224, 196 image blocks are obtained, where the nth image block can be denoted as... , where P is the side length of the image block and C is the number of image channels.

[0045] 1.3) Image patch unfolding and dimension transformation steps: flatten each image patch into a one-dimensional vector and linearly project it onto the preset embedding dimension D, where D is 1024.

[0046] Specifically, the original image is divided into multiple image blocks of size P×P×C. Each image block is flattened to form a vector of length L=P×P×C. That is, the expression is: .

[0047] Each image patch vector is then input into a linear projection layer, mapped to a preset embedding dimension D, typically 1024 dimensions. The formula is as follows:

[0048]

[0049] in, Let n be the vector representation of the nth image patch in the input. It is the weight matrix of the linear projection layer. It is the bias term of the linear projection layer.

[0050] All image patch embedding vectors are combined to form the following input sequence:

[0051]

[0052] 1.4) The location encoding fusion step adds learnable location encoding vectors to preserve spatial structure information. That is, since there is a lack of spatial location information between image patches from the original image, learnable location encoding vectors are introduced to preserve structural features. And add it element-wise to each embedding vector to get the following formula:

[0053]

[0054] 1.5) Multi-layer feature modeling steps: An L-layer encoding submodule is used for context modeling. Each encoding submodule includes multi-head self-attention computation, residual connections and layer normalization, and a feedforward neural network, outputting an L2-normalized visual semantic vector. This is used for context modeling and semantic enhancement of image patch sequences after image patch embedding and position encoding, achieving long-distance dependency modeling and global feature fusion between different regions in the input image.

[0055] Specifically, this module includes several encoder sub-modules (Encoder Layers) with identical structures and independent parameters, which are stacked sequentially. The encoder structure is defined to contain a total of... Each layer takes the output of the previous layer as its input. The input of the first layer is the position-encoded image patch embedding sequence, denoted as... ,in The number of image patches, For the embedded dimension.

[0056] Each coding submodule includes the following modules:

[0057] Multi-head self-attention submodule: Used to extract global dependencies between image patches in the input sequence. Based on the self-attention mechanism, it maps the input sequence into three matrices: query, key, and value, and calculates the attention-weighted output as follows:

[0058]

[0059] in , , For learnable parameter matrix, The dimension is defined for each attention head. A multi-head attention mechanism is used (number of heads is...). Multiple heads are computed in parallel and then concatenated, and merged into the original dimension through a linear transformation.

[0060] Concat( )

[0061] Concat combines the outputs of multiple attention heads (such as...) , , ..., The data is then concatenated along the last dimension, thus merging the information from each head into a unified representation.

[0062] Residual connection and layer normalization structure: This structure adds the residuals of the multi-head self-attention output and the original input, and then performs layer normalization, as follows:

[0063]

[0064] The feedforward neural network (FFN) submodule consists of two fully connected layers with a non-linear activation function (GELU) and Dropout operation in between to enhance the non-linear expressive power of the encoding model. This submodule reduces the embedding dimension from... Extend to the intermediate dimension Then project it back to the original dimension, which is represented as:

[0065]

[0066] Where Linear1 is the embedding dimension changed from Extend to the intermediate dimension Linear2 is used to add the intermediate dimension Project back to the original dimension. General settings. = 4D.

[0067] Residual connection and layer normalization structure (second layer): The output of the FFN module is also added to the input as a residual, and then subjected to layer normalization to obtain the final output of this layer:

[0068]

[0069] Ultimately, after The output sequence is obtained after processing by the layer encoder. Each image patch contains its original feature information and global context-aware information at its location.

[0070]

[0071] The feature vectors were processed Normalization, i.e., setting its Euclidean norm to 1, simplifies the similarity calculation of vectors to an inner product calculation.

[0072] Preferably, step 1.6) is also included: image data (such as device appearance images) associated with each device node in the knowledge graph are used to extract features using the same encoder described above, ensuring consistency in the feature space, and constructing a visual feature index library. .

[0073] 1.7) Cosine similarity measurement steps: The specific formula is as follows:

[0074]

[0075] in, Vector encoding of the input image. This refers to all vectors in the visual feature index library.

[0076] It uses inner product operations to accelerate calculation (features have been normalized), and the similarity range is [-1, 1]. The larger the value, the stronger the semantic association.

[0077] 1.8) Candidate Entity Screening Step: By calculating the cosine similarity between the visual semantic vector of the input image and the visual semantic vector of each device node in the device knowledge graph, semantic matching is performed between potential hazard areas in the image and entities in the graph. Entity nodes with similarity higher than a preset value are screened out by setting a threshold. These nodes are identified as potentially associated entities with hazard areas in the image and are recalled as candidate entities.

[0078] Set dynamic threshold (Take 0.8), recall satisfied. For all entities ≥ τ, output a sorted list of candidate entities: If there are no device entities exceeding the threshold, then no relevant device map information will be passed in. For the first in the knowledge base Devices (such as "eyewash stations") Let be the similarity between the i-th device and the input image. For similarity greater than a threshold The candidate entity set is sorted in descending order of similarity.

[0079] Supplementary Information and Potential Hazard Reasoning: The recall results will not only include the device entity directly related to the potentially hazardous area in the image, but will also include the following supplementary information:

[0080] Equipment attribute relationships: This includes descriptions of potential equipment hazards and information on corrective measures, in order to fully understand the contextual knowledge of potential equipment hazards.

[0081] The dynamic prompt generation module generates structured prompts based on the enterprise profile information, recalled equipment subgraph information, and real-time image semantic information. These prompts use a triplet template format: [Enterprise Profile Information] + [Graph Matching Equipment Features] + [Image Extracted Semantics], and are input into a Large Language Model (LLM) for hazard identification. This structured prompt integrates industry context, equipment knowledge graph, and image semantic information, enhancing the accuracy of hazard identification.

[0082] The relevant evidence retrieval module, after hazard identification, retrieves relevant evidence from the database based on hazard type and contextual information, outputting a hazard description, rectification measures, and relevant standards to form a complete identification result. To improve the accuracy and efficiency of hazard identification and ensure that the identification results are supported by relevant evidence, this system employs a relevant evidence retrieval mechanism. The specific implementation process is as follows:

[0083] Hazard identification: Based on the above process, the large model is fed with prompts from all equipment-related information to generate relevant hazards and mark the specific locations of the hazards.

[0084] Relevant Evidence Retrieval: Once a potential hazard is successfully located, the system will automatically trigger a retrieval mechanism from the relevant evidence database. At this time, the system will combine the specific type of the hazard with relevant contextual information to retrieve relevant evidence related to the hazard.

[0085] like Figure 2 As shown, taking an eyewash station as an example, the system accesses the company's profile information and acquires on-site images through a real-time image acquisition module.

[0086] 1.1) Image normalization step: The on-site image is a color RGB image. Image normalization is performed by normalizing and scaling the acquired image to a uniform size, reducing the resolution to 224×224 pixels, and then inputting the image into the image block generation module.

[0087] 1.2) Image block division steps: Divide the image into a fixed window size of 16×16 with a step size of 16. Divide the entire image into 196 non-overlapping image blocks, each with a size of 16×16×3, where 3 represents that each pixel has 3 channels, corresponding to the three color channels of an RGB color image.

[0088] 1.3) Image patch unfolding and dimensional transformation steps: each image patch (16×16×3) is flattened into a vector of length 768, input into the linear projection layer for dimensional transformation, and transformed into a one-dimensional vector of length 1024.

[0089] 1.4) Positional encoding fusion step: Since there is a lack of spatial positional information between image blocks from the original image, in order to preserve structural features and retain the spatial order relationship between blocks in the image, a learnable positional encoding is added to the embedding vector of each image block. The positional encoding can be a learnable positional embedding vector with the same dimension as the embedding vector (a one-dimensional vector of length 1024).

[0090] 1.5) The multi-layer feature modeling step involves embedding image patches with location encoding into a multi-layer self-attention encoding structure as a vector input. Feature interaction and information fusion are performed sequentially to uncover the relationships between image patches. After multi-layer encoding processing, an image semantic representation containing global context information is obtained. The vector dimension remains consistent with the input vector, with a length of 1024 and is a one-dimensional vector.

[0091] The system compares the image with a device library. If the similarity between the image and a device in the library is greater than 0.8, the device is considered to have been identified. If a device is identified (such as the eyewash station in this embodiment), the system will recall the knowledge graph associated with that device. The knowledge graph contains a description of the potential hazard and corrective measures.

[0092] For example, the company profile is as follows: Industry: Chemical; Employee size: 100-499 people; Safety characteristics: High-risk chemical industry, including confined space work areas; Main product: Potassium permanganate; Other characteristics: Located in a high-humidity environment, machinery is prone to corrosion.

[0093] The recalled equipment includes: Eyewash Station Information: 1. Hazard Description: Severe corrosion of the eyewash station material; Corrective Measures: Upgrade the material to stainless steel. 2. Hazard Description: Missing safety signs and operating procedures; Corrective Measures: Include operating procedures and safety signs with the entire equipment. 3. Hazard Description: Missing dust cover; Corrective Measures: Install a dust cover.

[0094] The generated dynamic prompt words are:

[0095] You are a meticulous and professional safety hazard inspection expert. Please complete the following tasks based on the provided images:

[0096] Scene background description:

[0097] According to the company profile, it currently operates in the chemical industry, with 100-499 employees, and features confined workspaces and a high-humidity, highly corrosive environment. Its main product is potassium permanganate, which has a certain degree of toxicity and strong oxidizing properties.

[0098] Preliminary analysis using an image recognition system suggests the device in the image may be an eyewash station. The system's associated device map may have the following potential risks:

[0099] 1. Hazard description: The eyewash station material is severely corroded; Corrective measures: Upgrade the material to stainless steel.

[0100] 2. Hazard description: Safety signs and operating procedures are missing; Corrective measures: The equipment should be equipped with operating procedures and safety signs.

[0101] 3. Hazard description: Dust cover missing; Corrective measures: Install a dust cover.

[0102] Please analyze the following task based on the image content:

[0103] 1. Please accurately mark all existing safety hazards in the diagram, circle the location of each hazard with a number, and provide a concise and clear description of each hazard.

[0104] 2. For each potential hazard, propose clear, specific, and actionable rectification measures, avoiding vague and general statements.

[0105] The output format requirements are as follows:

[0106]

Hazard 1

[0107] Corrective measures: Specific operational suggestions.

[0108]

Hazard 2

[0109] Corrective measures: Specific operational suggestions.

[0110] The prompts, which combine information from the recalled device sub-image and enterprise profile, are input into a multimodal large model along with real-time collected image data to achieve accurate identification.

[0111] Model Output and Assembly: The output of the large model is sent to the model assembly module, where it is integrated with the recalled dependency data to form a complete output result, as shown below:

[0112]

Hazard 1

[0113] Corrective measures: Install dedicated eyewash station dust covers to ensure the nozzles are closed when the equipment is not in use. Include this in the monthly inspection checklist, checking whether the dust covers are intact and sealed.

[0114] Relevant basis: Clause 5.4.4 of GB / T 38144.1-2019 Eye irrigation device Part 1: Technical requirements stipulates that the nozzle should be equipped with a dust cover that can open automatically during use to prevent the nozzle from being contaminated and affecting the rinsing effect.

[0115]

Hazard 2

[0116] Corrective measures: Standardize the wiring of the nozzle hose, and use anti-detachment clips or fixing clamps to secure the pipeline to avoid affecting the water flow direction or causing secondary damage during emergency operations.

[0117] Relevant basis: Clause 5.3.3 of GB / T 38144.1-2019 Eye irrigation device Part 1: Technical requirements clearly requires that the connecting pipes should be securely fixed, and the installation should avoid blockage or misuse to ensure that the equipment can be quickly activated in an emergency.

[0118]

Hazard 3

[0119] Corrective measures: Post standardized operating procedures and emergency use guidelines in a prominent location on the eyewash station, clearly specifying the usage method, inspection frequency, and responsible person.

[0120] According to the relevant standards GB 13495.1-2015 Fire Safety Signs Part 1: Signs and GB / T 2893.1-2013 Safety Colors and Safety Signs, easily identifiable safety signs should be installed near emergency facilities.

[0121] In one embodiment, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the system steps provided in the above embodiments.

[0122] In one embodiment, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the system steps provided in the above embodiments.

[0123] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0124] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] The several embodiments described in this application are quite specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and refinements should also be considered within the scope of protection of this invention. Therefore, the scope of protection of this patent application should be determined by the appended claims.

Claims

1. An intelligent industrial hazard identification system based on multimodal information fusion and RAG mechanism, characterized in that, Includes the following modules: The enterprise profile information and real-time image acquisition module is used to access and integrate enterprise profile information, which includes basic enterprise information dimensions and security feature dimensions. The basic information dimension covers industry type, geographical location, main products or services, and employee size, while the security feature dimension covers security level and characteristics of high-risk locations. At the same time, this module acquires on-site image data through real-time image acquisition equipment. The multimodal semantic extraction module is used to process real-time image data, including image normalization, image block division, image block expansion, dimensional transformation, position encoding fusion, multi-layer feature modeling, and equipment sub-graph recall based on cosine similarity. The equipment sub-graph recall includes graph entity encoding and candidate entity screening. The recall results include equipment attribute relationships and hazard description information. The multi-layer feature modeling module adopts a stacked structure of encoding sub-modules, including a multi-head self-attention sub-module, a residual connection and layer normalization structure, and a feedforward neural network sub-module, which are used to generate L2-normalized visual semantic vectors. The dynamic prompt generation module is used to generate structured prompt words based on the enterprise profile information, recalled equipment sub-graph information and real-time image semantic information. The prompt words adopt a triple template format: [enterprise profile information] + [graph matching equipment features] + [image extraction semantics], and are input into the large language model LLM for hazard identification. The relevant evidence recall module is used to retrieve relevant evidence from the database based on the type of hazard and contextual information after the hazard is located, and output the hazard description, rectification measures and relevant evidence standards to form a complete identification result.

2. The intelligent industrial hazard identification system based on multimodal information fusion and RAG mechanism according to claim 1, characterized in that: The enterprise profile information and the security feature dimension in the real-time image acquisition module include the identification of confined spaces, high temperature, high pressure or high dust risk locations, and the image size output by the real-time image acquisition module is normalized to 224×224 pixels.

3. The intelligent industrial hazard identification system based on multimodal information fusion and RAG mechanism according to claim 1, characterized in that: The multimodal semantic extraction module specifically performs the following steps: 1.1) Image normalization step: Adjust the input image to a resolution of 224×224; 1.2) Image block division step: Divide the image into N non-overlapping image blocks with a fixed window size P×P, where N = (H / P) × (W / P), H and W are the image height and width, respectively; P is 16. 1.3) Image patch unfolding and dimension transformation steps: flatten each image patch into a one-dimensional vector and linearly project it onto the preset embedding dimension D, where D is 1024; 1.4) Position encoding fusion step: Add learnable position encoding vectors to preserve spatial structure information; 1.5) Multi-layer feature modeling steps: L-layer encoding sub-modules are used for context modeling. Each encoding sub-module includes multi-head self-attention calculation, residual connection and layer normalization, and feedforward neural network, which outputs L2-normalized visual semantic vectors.

4. The intelligent industrial hazard identification system based on multimodal information fusion and RAG mechanism according to claim 3, characterized in that: The cosine similarity threshold in the device subgraph recall is set to 0.8 to filter candidate entities. If the similarity is lower than 0.8, the device graph information will not be passed on.

5. The intelligent industrial hazard identification system based on multimodal information fusion and RAG mechanism according to claim 1, characterized in that: The dynamic prompt generation module generates prompt words that include the context of the enterprise profile and prior knowledge of potential equipment hazards, and inputs them into a multimodal large language model for accurate annotation and description.

6. The intelligent industrial hazard identification system based on multimodal information fusion and RAG mechanism according to claim 1, characterized in that: The output format of the relevant evidence recall module is a structured result, including a description of the location of the hidden danger, rectification measures, and relevant evidence standards, ensuring that the result has legal support and realizing end-to-end identification from image acquisition to evidence output.

7. The intelligent industrial hazard identification system based on multimodal information fusion and RAG mechanism according to claim 1, characterized in that: The results of the hazard identification include marking the location of the hazard on the map and providing rectification suggestions.

8. The intelligent industrial hazard identification system based on multimodal information fusion and RAG mechanism according to claim 1, characterized in that: The feedforward neural network submodule in the multi-layer feature modeling module adopts the GELU activation function and Dropout operation, with the intermediate dimension set to 4D, where D is the embedding dimension, to enhance the nonlinear expressive power.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the system as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the program is executed, it implements the steps of the system as described in claims 1 to 8.

Images