Chemical safety risk grading method based on large-model multi-modal knowledge injection

Abstract

The invention provides a chemical safety risk grading method based on large model multi-modal knowledge injection. The method comprises the following steps: collecting multi-modal data in the field of chemical safety, and constructing a multi-modal knowledge graph based on the collected data; then, feature coding is carried out on the image data and the text data through a vision-language large model, and most relevant visual features are extracted and embedded through a text through an image-text feature matcher; then, through a link-guided gating fusion network, combined representation of visual and structural features of entities is realized, features of neighbor nodes are further aggregated, multi-modal knowledge graph feature embedding is realized, and alignment of image entities and corresponding text features in a knowledge graph is realized based on comparative learning; and finally, comprehensive analysis of potential risks in chemical safety related scenes is realized by updating parameters specified by the model, and risk levels of the corresponding scenes are obtained.

Description

Technical Field

[0001] This invention relates to the field of chemical safety technology, and more specifically to a chemical safety risk classification method based on large-scale model multimodal knowledge injection. Background Technology

[0002] The chemical industry is a key pillar industry of the national economy, encompassing multiple fields such as refining plants, fine chemicals, and the production and handling of hazardous chemicals. Its production processes are complex and involve a large number of hazardous chemicals. The production environment contains dangerous factors such as flammable and explosive substances, toxic and harmful substances, and high temperature and pressure, making it prone to accidents such as explosions, fires, and leaks of toxic substances. These accidents not only cause significant casualties and direct economic losses, but may also trigger continuous environmental pollution and social panic. Therefore, safety monitoring and management in the chemical industry has always been a core challenge for the industry's development.

[0003] Chemical enterprises accumulate rich multimodal data, including production monitoring data, equipment operating status data, images, and text records. However, most current analysis methods do not fully utilize the value of multimodal data, focusing only on one type of data—images or text—thus ignoring the correlations and complementary information between different modalities. Large-scale modeling technology, with its powerful feature extraction, cross-modal feature alignment, and knowledge reasoning capabilities, can uncover potential risk patterns behind multimodal data, thereby improving the accuracy and efficiency of chemical safety risk classification. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a chemical safety risk classification method based on multimodal knowledge injection from a large-scale model. This method utilizes a large-scale visual-language model to simultaneously process multimodal data such as images and text. Knowledge graphs can represent semantic features and integrate relevant information through graphs of entities and their relationships. By fusing multimodal knowledge graphs, the ability of the large-scale model to capture contextual information is further enhanced, improving semantic understanding and enabling the analysis and rating of potential risks in complex scenarios.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a chemical safety risk classification method based on large-model multimodal knowledge injection, comprising the following steps: Step 1: Collect multimodal data related to chemical safety and construct a multimodal knowledge graph; Step 2: Encode the image and text features in the multimodal data related to chemical safety collected in Step 1 based on the vision-language big model; Step 3: Extract key visual features based on the image-text feature matcher; Step 4: Embed multimodal knowledge graph features based on a link-guided gated fusion network; Step 5: Perform text alignment on image entities in the multimodal knowledge graph through contrastive learning; Step six: Optimize the model and classify the risks of problems in the input data.

[0006] As a further improvement to the present invention, the specific steps for image and text feature encoding based on the vision-language large model in step two are as follows: First, an image feature encoder composed of multiple pre-trained Vision Transformers is used to progressively extract global features from the image, thereby obtaining the visual features for chemical safety. Text features are then mapped to a fixed-dimensional vector space using a pre-trained large language model, thus obtaining the effective text encoding features. ; Then, the image features are processed using a linear layer. Mapping to text features A consistent feature space ensures that the dimensions of image feature vectors and text feature vectors are consistent, thus enabling encoding.

[0007] As a further improvement of the present invention, the specific method for extracting key visual features based on the image-text feature matcher in step three is as follows: After obtaining the feature representations of the image and text, the text features are then used... Image features as query vectors As key and value vectors, the semantic relevance of each image region feature to the current text description is quantified by calculating the similarity between the query vector and the key vector. This allows for the extraction of the most relevant visual features for text embedding, achieving semantic fusion of cross-modal representations. Where d represents the text embedding feature The learned features are then concatenated before the input text embedding of the large language model, so that the learned visual representation can be interpreted by the large language model.

[0008] As a further improvement of the present invention, the specific method for embedding multimodal knowledge graph features in step four is as follows: First, based on the input image and text description, Top-K semantic association triples are retrieved from the complete knowledge graph. ),in These represent the head entity, relationship, and tail entity, respectively, and form a context subgraph G; Then, in the constructed context subgraph G, the link-guided fusion network first achieves multimodal entity embedding fusion by calculating the importance of entities and their associated images in different link information; then, by calculating the correlation between entities and their neighbors, it assigns the structural information of the neighbors to the central entity, enriching the feature representation of the entity.

[0009] As a further improvement of the present invention, the specific method for realizing multimodal entity embedding fusion is as follows: first, initial structural embedding is performed on entities and relations, and the initial embedding of the image is projected onto the hidden embedding space through linear mapping to perform similarity matching in a unified dimension; then, the importance scores of the image and entities are calculated based on different link information.

[0010] As a further improvement of the present invention, the method for initial structural embedding of entities and relations is as follows: For a given triple, the initial structural embeddings of the head entity, relation, and tail entity are concatenated, and then the structural embedding of the triple link information is obtained through a trainable linear transformation matrix. This is then combined with the visual embedding of the image and normalized to obtain the importance score of the image to the entity, as detailed below: For the head entity in subgraph G In other words, its associated image set is : relation , This represents a splicing operation. It is a learnable weight parameter matrix. Representatives and Entities The associated k-th image embedding, For a given triplet's structural embedding, Is with entity The associated image importance score; similarly, for tail entities in the subgraph. It can be obtained .

[0011] As a further improvement of the present invention and The importance of different linking information in the head and tail entities is determined, and then a gating mechanism is used to preserve the important features of the images associated with the entities, resulting in the visual embedding of the entities. Further integrate with the entity Visual embedding and structural embedding are combined to obtain entity embeddings that integrate multimodal information: After obtaining entity embeddings that incorporate visual information, the entities are further aggregated. Improving the quality of the corresponding feature representations by incorporating the structural information of neighboring entities yields the feature embeddings of the subgraph. : in In the case of relation r, the node The neighbors of the subgraph, R, represent the set of relations contained in the subgraph.

[0012] As a further improvement to the present invention, the specific steps for text alignment in step five are as follows: For image entities in the retrieved subgraph G, contrastive learning prompts the model to accurately match them with their corresponding text entities, thereby achieving cross-modal feature alignment. The goal is to minimize the distance between positive samples and target samples, while maximizing the distance between negative samples and target samples. Specifically, the corresponding optimization objective is: Where N is the number of image entities that need to be queried. and These represent the encoded vectors of image entities and text, respectively. and These represent the entity codes for the corresponding positive and negative samples, respectively. This represents the number of entities corresponding to the negative samples. This is the temperature coefficient.

[0013] The beneficial effects of this invention are that it jointly models heterogeneous modal information such as images and text using a large visual-language model, and achieves semantic interaction of multimodal features through an image-text feature matcher. Simultaneously, by combining a link-guided gated fusion network, it embeds the topological features of the multimodal chemical safety knowledge graph into the cross-modal representation space of the large model, enhancing the model's ability to capture the contextual logic of complex scenarios. This supports real-time analysis of potential risks throughout the entire chemical production process, determining whether the risk level in the input scenario is low, medium, or high, and providing corresponding rectification analysis suggestions. Attached Figure Description

[0014] Figure 1 The flowchart shows the chemical safety risk classification method based on large model multimodal knowledge injection of the present invention. Detailed Implementation

[0015] The present invention will now be described in further detail with reference to the embodiments shown in the accompanying drawings.

[0016] Reference Figure 1 As shown in this embodiment, a chemical safety risk classification method based on large model multimodal knowledge injection is described. The method steps are as follows: Step 1: Collect multimodal data related to chemical safety and construct a multimodal knowledge graph; Image data: Images of hazardous chemical labels, safety signs and warnings, etc., further collecting images of the normal operation status of chemical equipment, as well as images of equipment in the event of malfunctions or abnormal situations, and obtaining information such as equipment number and equipment name.

[0017] Text data: This includes textual descriptions of images, such as safety issues at a certain location of equipment in the image, or violations of certain regulations. In addition, it is necessary to collect textual data related to industry standards and regulations, laws and regulations, equipment technical documents, accident reports, operation logs, maintenance records, etc. in the field of chemical safety. This data will be further integrated with data from sensors such as temperature and pressure to construct a multimodal knowledge graph. Image-text multimodal knowledge graph: Hazardous chemical symbols, safety signs and warning images are linked to corresponding industry standards, regulations and laws. Equipment-related images are linked to information in the document data, such as the equipment name, function, maintenance requirements, etc., and the changes in the corresponding sensors. Furthermore, through relation extraction, related content is stored as triples of head entity-relation-tail entity. Considering that the chemical production environment, such as changes in equipment status and sensor parameters, is constantly changing, and that accident cases and process optimization rules are constantly being updated, the knowledge graph construction possesses scalability and other characteristics.

[0018] Step 2: Image and text feature encoding based on a large vision-language model; Image and text data related to chemical safety are widely available, contain a vast amount of information, and are highly complex and specialized. This paper utilizes a pre-trained Vision-Language Model (VLM) to encode image and text features. Firstly, the image feature encoder typically consists of multiple layers of pre-trained Vision Transformer (ViT) layers, progressively extracting global features from the image to obtain the visual features related to chemical safety. Text features are then mapped to a fixed-dimensional vector space using a pre-trained large language model, thus obtaining the effective text encoding features. Then, a linear layer is used to extract image features. Mapping to text features A consistent feature space ensures that the dimensions of image feature vectors and text feature vectors are consistent.

[0019] Step 3: Extraction of key visual features based on image-text feature matcher; After obtaining the feature representations of the image and text, the text features are then used... Image features as query vectors As key and value vectors, the semantic relevance of each image region feature to the current text description is quantified by calculating the similarity between the query vector and the key vector. This allows for the extraction of the most relevant visual features for text embedding, achieving semantic fusion of cross-modal representations. Where d represents the text embedding feature. The dimension. Then the learned features... It is concatenated before the input text embedding of the large language model so that the learned visual representation can be interpreted by the large language model.

[0020] Step 4: Embed multimodal knowledge graph features based on a link-guided gated fusion network; Considering that the number of nodes and edges in a complete chemical safety multimodal knowledge graph is often quite large, its computational complexity increases rapidly with the number of nodes and edges involved. Therefore, based on the input image and text description, Top-K semantic association triples are first retrieved from the complete knowledge graph. ),in These represent the head entity, relationship, and tail entity, respectively, constructing a context subgraph G. The subgraph can be dynamically updated as needed to prevent resource waste caused by recalculating the entire graph.

[0021] In the constructed context subgraph G, the link-guided fusion network first achieves multimodal entity embedding fusion by calculating the importance of entities and their associated images in different link information; then, by calculating the correlation degree between entities and their neighbors, it assigns the structural information of the neighbors to the central entity, enriching the feature representation of the entity.

[0022] Considering that entities in a knowledge graph have different attributes in different link information (triples), the contribution of associated images to entities also varies. For the central entity and its neighboring entity set, initial structural embeddings are first performed on the entities and relations. The initial image embeddings are then projected onto the hidden embedding space using a linear mapping to perform similarity matching with a unified dimension. Then, importance scores between images and entities are calculated based on different link information. Specifically, for a given triple, the initial structural embeddings of the head entity, relation, and tail entity are concatenated, and a trainable linear transformation matrix is ​​used to obtain the structural embedding of the triple link information. This is then dot-producted with the visual embedding of the image and normalized to obtain the importance score of the image to the entity. Specifically, for the head entity in subgraph G... In other words, its associated image set is : relation , This represents a splicing operation. It is a learnable weight parameter matrix. Representatives and Entities The associated k-th image embedding, For a given triplet's structural embedding, Is with entity The associated image importance score; similarly, for tail entities in the subgraph. It can be obtained .

[0023] and The importance of different linking information in the head and tail entities is determined, and then a gating mechanism is used to preserve the important features of the images associated with the entities, resulting in the visual embedding of the entities. Further integrate with the entity Visual embedding and structural embedding are combined to obtain entity embeddings that integrate multimodal information: After obtaining entity embeddings that incorporate visual information, the entities are further aggregated. Improving the quality of the corresponding feature representations by incorporating the structural information of neighboring entities yields the feature embeddings of the subgraph. : in In the case of relation r, the node The neighbors of the subgraph are represented by R, which represents the set of relations contained in the subgraph. Subsequently, in order for the large language model to understand the multimodal knowledge node embeddings, the knowledge graph features are encoded into a vector subspace consistent with the text features through a knowledge adapter. This is then converted into text embeddings that the large language model can understand, bridging the inherent gap between multimodal knowledge and text and promoting smoother alignment.

[0024] Step 5: Perform text alignment on image entities in the multimodal knowledge graph through contrastive learning; For image entities in the retrieved subgraph G, contrastive learning prompts the model to accurately match them with their corresponding text entities, thereby achieving cross-modal feature alignment. The goal is to minimize the distance between positive samples and target samples, while maximizing the distance between negative samples and target samples. Specifically, the corresponding optimization objective is: Where N is the number of image entities that need to be queried. and These represent the encoded vectors of image entities and text, respectively. and These represent the entity codes for the corresponding positive and negative samples, respectively. This represents the number of entities corresponding to the negative samples. This is the temperature coefficient.

[0025] Step 6: Model optimization, risk classification of problems in the input data; Considering the large number of parameters involved in the text encoder and image encoder of the visual-language large model, during training, these parameters are kept frozen. Only the parameters of the linear layers used to maintain consistency between the image and text feature spaces, the parameters in the image-text feature matcher used to extract key visual features that match the text, the parameters in the link-guided gated fusion network used to encode knowledge graph features, and the parameters in the linear layers that maintain consistency between the knowledge graph and the visual-language large model feature spaces are updated. After sufficient model training, by uploading relevant images and corresponding questions, the model analyzes the risk level of various operations in the input chemical production images during the reasoning process in a chemical safety scenario. It categorizes these operations as low-risk, medium-risk, or high-risk. Simultaneously, leveraging the powerful understanding and reasoning capabilities of the large model, corresponding rectification analysis suggestions are provided.

[0026] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A chemical safety risk classification method based on large-scale model multimodal knowledge injection, characterized in that: The steps include: Step 1: Collect multimodal data related to chemical safety and construct a multimodal knowledge graph; Step 2: Encode the image and text features in the multimodal data related to chemical safety collected in Step 1 based on the vision-language big model; Step 3: Extract key visual features based on the image-text feature matcher; Step 4: Embed multimodal knowledge graph features based on a link-guided gated fusion network; Step 5: Perform text alignment on image entities in the multimodal knowledge graph through contrastive learning; Step six: Optimize the model and classify the risks of problems in the input data.

2. The chemical safety risk classification method based on large-model multimodal knowledge injection according to claim 1, characterized in that: The specific steps for image and text feature encoding based on the large vision-language model in step two are as follows: First, an image feature encoder composed of multiple pre-trained Vision Transformers is used to progressively extract global features from the image, thereby obtaining the visual features for chemical safety. Text features are then mapped to a fixed-dimensional vector space using a pre-trained large language model, thus obtaining the effective text encoding features. ; Then, the image features are processed using a linear layer. Mapping to text features A consistent feature space ensures that the dimensions of image feature vectors and text feature vectors are consistent, thus enabling encoding.

3. The chemical safety risk classification method based on large-model multimodal knowledge injection according to claim 1 or 2, characterized in that: The specific method for extracting key visual features based on the image-text feature matcher in step three is as follows: After obtaining the feature representations of the image and text, the text features are then used... Image features as query vectors As key and value vectors, the semantic relevance of each image region feature to the current text description is quantified by calculating the similarity between the query vector and the key vector. This allows for the extraction of the most relevant visual features for text embedding, achieving semantic fusion of cross-modal representations. Where d represents the text embedding feature The learned features are then concatenated before the input text embedding of the large language model, so that the learned visual representation can be interpreted by the large language model.

4. The chemical safety risk classification method based on large-model multimodal knowledge injection according to claim 1 or 2, characterized in that: The specific method for embedding multimodal knowledge graph features in step four is as follows: First, based on the input image and text description, Top-K semantic association triples are retrieved from the complete knowledge graph. ),in These represent the head entity, relationship, and tail entity, respectively, and form a context subgraph G; Then, in the constructed context subgraph G, the link-guided fusion network first achieves multimodal entity embedding fusion by calculating the importance of entities and their associated images in different link information; then, by calculating the correlation between entities and their neighbors, it assigns the structural information of the neighbors to the central entity, enriching the feature representation of the entity.

5. The chemical safety risk classification method based on large-model multimodal knowledge injection according to claim 4, characterized in that: The specific method for achieving multimodal entity embedding fusion is as follows: First, initial structural embedding is performed on entities and relationships. The initial embedding of the image is projected onto the hidden embedding space through linear mapping to perform similarity matching with a unified dimension. Then, the importance scores of the image and entities are calculated based on different link information.

6. The chemical safety risk classification method based on multimodal knowledge injection in a large model according to claim 5, characterized in that: The method for initial structural embedding of entities and relations is as follows: For a given triple, the initial structural embeddings of the head entity, relation, and tail entity are concatenated, and then the structural embedding of the triple link information is obtained through a trainable linear transformation matrix. This is then multiplied by the visual embedding of the image and normalized to obtain the importance score of the image to the entity, as detailed below: For the head entity in subgraph G In other words, its associated image set is : relation , This represents a splicing operation. It is a learnable weight parameter matrix. Representatives and Entities The associated k-th image embedding, For a given triplet's structural embedding, Is with entity The associated image importance score; similarly, for tail entities in the subgraph. It can be obtained .

7. The chemical safety risk classification method based on large model multimodal knowledge injection according to claim 6, characterized in that: and The importance of different linking information in the head and tail entities is determined, and then a gating mechanism is used to preserve the important features of the images associated with the entities, resulting in the visual embedding of the entities. Further integrate with the entity Visual embedding and structural embedding are combined to obtain entity embeddings that integrate multimodal information: After obtaining entity embeddings that incorporate visual information, the entities are further aggregated. Improving the quality of the corresponding feature representations by incorporating the structural information of neighboring entities yields the feature embeddings of the subgraph. : in In the case of relation r, the node The neighbors of the subgraph, R, represent the set of relations contained in the subgraph.

8. The chemical safety risk classification method based on large model multimodal knowledge injection according to claim 1 or 2, characterized in that: The specific steps for text alignment in step five are as follows: For image entities in the retrieved subgraph G, contrastive learning prompts the model to accurately match them with their corresponding text entities, thereby achieving cross-modal feature alignment. The goal is to minimize the distance between positive samples and target samples, while maximizing the distance between negative samples and target samples. Specifically, the corresponding optimization objective is: Where N is the number of image entities that need to be queried. and These represent the encoded vectors of image entities and text, respectively. and These represent the entity codes for the corresponding positive and negative samples, respectively. This represents the number of entities corresponding to the negative samples. This is the temperature coefficient.

Images