Port risk identification method and system based on multi-modal large model, and storage medium

Abstract

The invention discloses a port risk identification method and system based on a multi-modal large model, and a storage medium, and relates to the technical field of port safety management. The method comprises the following specific steps: acquiring multi-modal port monitoring data; carrying out preprocessing operation on the multi-modal port monitoring data to obtain normalized data; multi-modal features in the normalized data are extracted; establishing a multi-modal fusion framework, and fusing the multi-modal features to obtain multi-modal fusion features; constructing a security risk identification multi-modal large model based on the multi-modal fusion features; and inputting multi-modal data collected in real time into the trained security risk identification multi-modal large model, carrying out risk identification, and automatically triggering an early warning mechanism according to an identification result. According to the method, multi-modal data such as video monitoring, environment monitoring, equipment states and operation processes of a dangerous cargo wharf of a port are collected, and a multi-modal fusion model is constructed by using a deep learning algorithm, so that automatic identification and early warning of safety production risks are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of port safety management, and in particular to a port risk identification method, system and storage medium based on a multi-modal large model. Background Art

[0002] As an important link in logistics and transportation, the safe production of dangerous goods terminals in ports is directly related to personnel safety, environmental protection and social stability. However, traditional safety risk identification methods mostly rely on manual inspections and empirical judgments, which have problems such as low efficiency and easy omissions. With the rapid development of technologies such as big data and artificial intelligence, multimodal information processing has become an important research direction. Multimodal information refers to data from different forms, such as text, images, sounds, videos, etc., which play a vital role in real-world interactions. Therefore, for those skilled in the art, how to use multimodal data fusion and deep learning models for risk identification is an urgent problem to be solved. Summary of the invention

[0003] The purpose of the present invention is to provide a port risk identification method, system and storage medium based on a multimodal large model to solve the problems raised in the background technology. By collecting multimodal data such as video surveillance, environmental monitoring, equipment status, and operation processes of dangerous goods terminals in ports, a multimodal fusion model is constructed using a deep learning algorithm to achieve automatic identification and early warning of production safety risks.

[0004] To achieve the above object, the present invention provides the following scheme: a method for identifying safety risks of dangerous goods operations in ports based on a multimodal large model, the specific steps of which include the following:

[0005] Obtain multimodal port monitoring data;

[0006] Performing a preprocessing operation on the multimodal port monitoring data to obtain normalized data;

[0007] Extracting multimodal features from the normalized data;

[0008] Building a multimodal fusion architecture, fusing the multimodal features to obtain multimodal fusion features;

[0009] Constructing a large multimodal model for security risk identification based on the multimodal fusion features;

[0010] The multimodal data collected in real time is input into the trained multimodal large model for security risk identification to perform risk identification, and the early warning mechanism is automatically triggered based on the identification results.

[0011] Preferably, the multimodal port monitoring data includes video surveillance data, port environment monitoring data, equipment status data, operation process data, and images of various production safety risks at port dangerous goods terminals.

[0012] Preferably, the specific steps of preprocessing the multimodal port monitoring data are:

[0013] Performing data cleaning, denoising and normalization operations on the multimodal port monitoring data to obtain preprocessed data;

[0014] Performing text annotation for each picture and surveillance video in the multimodal port monitoring data, wherein the annotation content includes the type of security risk, specific location, and potential hazards;

[0015] Constructing a multimodal data set corresponding to images and texts based on the text annotation content and the preprocessed data;

[0016] The multimodal data set is cleaned again to obtain the normalized data.

[0017] Preferably, for the image and the video surveillance data, a pre-trained CLIP model of the ViT structure is used for feature extraction; for the operation process data, Transformer technology is used for feature extraction; for text annotation, NLP technology is used for text feature extraction.

[0018] Preferably, it also includes comparing the risk identification results with relevant laws and regulations, and automatically matching the corresponding legal clauses, chapters and clause numbers.

[0019] Preferably, it also includes generating a production safety risk inspection report based on the identification result, and the report is presented in the form of a checklist or check line.

[0020] On the other hand, a port dangerous goods operation safety risk identification system based on a multimodal large model is provided, including a data acquisition module, a data preprocessing module, a feature extraction module, a feature fusion module, a model building module, and a risk identification module; wherein,

[0021] The data acquisition module is used to acquire multi-modal port monitoring data;

[0022] The data preprocessing module is used to perform preprocessing operations on the multimodal port monitoring data to obtain normalized data;

[0023] The feature extraction module is used to extract multimodal features from the normalized data;

[0024] The feature fusion module is used to build a multimodal fusion architecture to fuse the multimodal features to obtain multimodal fusion features;

[0025] The model building module is used to build a large multimodal model for security risk identification based on the multimodal fusion features;

[0026] The risk identification module is used to input the multimodal data collected in real time into the trained multimodal large model for security risk identification, perform risk identification, and automatically trigger an early warning mechanism based on the identification results.

[0027] Finally, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for identifying safety risks of dangerous goods operations in ports based on a multimodal large model are implemented.

[0028] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: comprehensive analysis of multi-dimensional information such as video surveillance and document records of port dangerous goods terminals to efficiently and accurately identify and locate production safety risks and ensure the safety and compliance of port operations; by integrating multiple data sources and advanced artificial intelligence technologies, accurate and efficient identification of production risks of port dangerous goods terminal enterprises can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] Figure 1 is a flow chart of the method of the present invention;

[0031] Figure 2 It is a system structure diagram of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] The purpose of this invention is to provide a method for identifying safety risks of dangerous goods operations in ports based on a multimodal large model. Figure 1 As shown, the specific steps include the following:

[0034] S1. Obtain multimodal port monitoring data;

[0035] S2, preprocessing the multimodal port monitoring data to obtain normalized data;

[0036] S3, extracting multimodal features from normalized data;

[0037] S4. Build a multimodal fusion architecture, fuse multimodal features, and obtain multimodal fusion features;

[0038] S5. Construct a large multimodal model for security risk identification based on multimodal fusion features;

[0039] S6. Input the multimodal data collected in real time into the trained multimodal large model for security risk identification to perform risk identification and automatically trigger the early warning mechanism based on the identification results.

[0040] Furthermore, multimodal port monitoring data includes video surveillance data, port environment monitoring data, equipment status data, operation process data, and images of various safety production risks at port dangerous goods terminals, including but not limited to equipment failures, illegal operations, environmental anomalies, etc. Environmental monitoring data includes temperature, humidity, gas concentration, etc.; equipment status data includes equipment operating parameters, fault alarms, etc.; operation process data refers to the process of on-site ship berthing, cargo loading and unloading, etc.

[0041] Furthermore, the specific steps of preprocessing the multimodal port monitoring data are as follows:

[0042] S21, performing data cleaning, denoising and normalization operations on the multimodal port monitoring data to obtain preprocessed data;

[0043] S22. Annotate each image in the multimodal port monitoring data with text, including the type of safety risk, specific location, and potential hazards;

[0044] S23, constructing a multimodal dataset corresponding to images and texts based on the text annotation content and preprocessed data;

[0045] S24. Perform data cleaning again on the multimodal data set to obtain normalized data.

[0046] Furthermore, for image and video surveillance data, the pre-trained CLIP model with ViT structure is used for feature extraction; for operation process data, Transformer technology is used for feature extraction; for text annotation, NLP technology is used for text feature extraction.

[0047] Furthermore, in step S4, a multimodal fusion architecture is built to fuse the features of different modalities, and the attention mechanism technology can be used to improve the model's ability to understand complex scenes.

[0048] Furthermore, the multimodal data collected in real time is input into the trained multimodal large model for security risk identification to carry out risk identification. According to the identification results, the early warning mechanism is automatically triggered to send early warning information to relevant personnel, including risk type, location, recommended measures, etc.

[0049] Furthermore, it also includes comparing the risk identification results with relevant laws and regulations, automatically matching the corresponding legal clauses, chapters and clause numbers, providing companies with a clear legal basis and guiding them in risk rectification and compliance management.

[0050] Furthermore, it also includes generating a production safety risk inspection report based on the identification results, and the report is presented in the form of a checklist or check line.

[0051] On the other hand, a port dangerous goods operation safety risk identification system based on a multimodal large model is provided. Figure 2 As shown, it includes a data acquisition module, a data preprocessing module, a feature extraction module, a feature fusion module, a model building module, and a risk identification module; wherein,

[0052] Data acquisition module, used to acquire multimodal port monitoring data;

[0053] A data preprocessing module is used to preprocess the multimodal port monitoring data to obtain normalized data;

[0054] Feature extraction module, used to extract multimodal features from normalized data;

[0055] The feature fusion module is used to build a multimodal fusion architecture, fuse multimodal features, and obtain multimodal fusion features;

[0056] Model building module, used to build a large multimodal model for security risk identification based on multimodal fusion features;

[0057] The risk identification module is used to input the multimodal data collected in real time into the trained multimodal large model for security risk identification, conduct risk identification, and automatically trigger the early warning mechanism based on the identification results.

[0058] Furthermore, in order to better illustrate the method of the present invention, a specific embodiment is used for illustration. For example, when a small fire occurs in a port, the specific execution steps are as follows:

[0059] Through the data acquisition module, the monitoring camera is used to shoot the port site conditions in real time and obtain pictures of the port site;

[0060] Send the fire image data with a smaller scope to the preprocessing module to obtain normalized data;

[0061] Input the normalized data into the feature extraction module to obtain multimodal features;

[0062] By inputting multimodal features into a multimodal large model, the current port safety risks can be extracted and the identification results can be automatically triggered to trigger early warnings. Fires with smaller areas can be warned in a timely manner, thus avoiding the spread and expansion of fires.

[0063] Finally, a computer storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of a method for identifying safety risks of dangerous goods operations in ports based on a multimodal large model are implemented.

[0064] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for identifying safety risks of dangerous goods operations in ports based on a multimodal large model, characterized in that: The specific steps include the following: Obtain multimodal port monitoring data; Performing a preprocessing operation on the multimodal port monitoring data to obtain normalized data; Extracting multimodal features from the normalized data; Building a multimodal fusion architecture, fusing the multimodal features to obtain multimodal fusion features; Constructing a large multimodal model for security risk identification based on the multimodal fusion features; The multimodal data collected in real time is input into the trained multimodal large model for security risk identification to perform risk identification, and the early warning mechanism is automatically triggered based on the identification results.

2. According to claim 1, a method for identifying safety risks of dangerous goods operations in ports based on a multimodal large model is characterized in that: The multimodal port monitoring data includes video surveillance data, port environment monitoring data, equipment status data, operation process data, and images of various production safety risks at port dangerous goods terminals.

3. According to claim 1, a method for identifying safety risks of dangerous goods operations in ports based on a multimodal large model is characterized in that: The specific steps of preprocessing the multimodal port monitoring data are as follows: Performing data cleaning, denoising and normalization operations on the multimodal port monitoring data to obtain preprocessed data; Performing text annotation for each picture and surveillance video in the multimodal port monitoring data, wherein the annotation content includes the type of security risk, specific location, and potential hazards; Constructing a multimodal data set corresponding to images and texts based on the text annotation content and the preprocessed data; The multimodal data set is cleaned again to obtain the normalized data.

4. According to claim 2, a method for identifying safety risks of dangerous goods operations in ports based on a multimodal large model is characterized in that: For the image and the video surveillance data, the pre-trained CLIP model of the ViT structure is used for feature extraction; for the operation process data, the Transformer technology is used for feature extraction; for the text annotation, the NLP technology is used for text feature extraction.

5. According to claim 1, a method for identifying safety risks of dangerous goods operations in ports based on a multimodal large model is characterized in that: It also includes comparing the risk identification results with relevant laws and regulations, and automatically matching the corresponding legal clauses, chapters and clause numbers.

6. The method for identifying safety risks of dangerous goods operations in ports based on a multimodal large model according to claim 1 is characterized in that: It also includes generating a production safety risk inspection report based on the identification results, and the report is presented in the form of a checklist or inspection line.

7. A port dangerous goods operation safety risk identification system based on a multimodal large model, characterized in that: It includes data acquisition module, data preprocessing module, feature extraction module, feature fusion module, model building module and risk identification module; among them, The data acquisition module is used to acquire multi-modal port monitoring data; The data preprocessing module is used to perform preprocessing operations on the multimodal port monitoring data to obtain normalized data; The feature extraction module is used to extract multimodal features from the normalized data; The feature fusion module is used to build a multimodal fusion architecture to fuse the multimodal features to obtain multimodal fusion features; The model building module is used to build a large multimodal model for security risk identification based on the multimodal fusion features; The risk identification module is used to input the multimodal data collected in real time into the trained multimodal large model for security risk identification, perform risk identification, and automatically trigger an early warning mechanism based on the identification results.

8. A computer storage medium, characterized in that: The computer storage medium stores a computer program, which, when executed by a processor, implements the steps of a method for identifying safety risks of dangerous goods operations in ports based on a multimodal large model as described in any one of claims 1 to 6.

Images