Gas station risk early warning method and system based on video stream and multi-modal large model

Abstract

The invention discloses a gas station risk early warning method and system based on a video stream and a multi-modal large model, and relates to the technical field of risk early warning, and the method comprises the steps: S1, obtaining a real-time video stream of a gas station site through a monitoring camera, and carrying out the frame extraction and image preprocessing of the video stream; s2, analyzing the video frame by using a multi-modal large model and generating a corresponding text description; s3, retrieving risk information related to the text description based on a retrieval enhancement generation technology and generating security suggestions; and S4, carrying out risk assessment on an analysis result, and sending a real-time early warning notification to related personnel when an early warning threshold value is exceeded. According to the method, the risk identification precision can be improved, the response speed can be remarkably improved, real-time safety suggestions can be provided for different types of risks, and safe operation of the gas station is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of risk warning, and in particular to a gas station risk warning method and system based on video stream and multimodal large model. Background Art

[0002] As high-risk locations like gas stations expand, traditional manual monitoring and video playback methods are no longer sufficient to meet the demands for real-time early warning and rapid response. Existing monitoring systems typically rely solely on simple video recording and manual inspections, which is not only inefficient but also difficult to accurately and promptly identify and predict potential safety risks. Traditional video surveillance systems typically only record image data and lack intelligent means to conduct in-depth analysis and multi-level understanding of video streams. Even with the emergence of some deep learning-based abnormal behavior detection systems, they still lack comprehensive analysis of complex environments and are unable to provide specific corrective recommendations based on video content. Furthermore, existing early warning systems mostly remain at the simple alarm level, lacking follow-up guidance and action recommendations. Summary of the Invention

[0003] In response to the needs and shortcomings of current technological development, the present invention provides a gas station risk warning method and system based on video streaming and multimodal large models, which can realize intelligent identification and rapid warning response of potential safety hazards in gas stations.

[0004] In the first aspect, the present invention provides a gas station risk warning method based on video streams and a multimodal large model, which solves the above-mentioned technical problems using the following technical solutions:

[0005] A gas station risk warning method based on video stream and multimodal large model includes the following steps:

[0006] S1. Obtain real-time video streams from the gas station through surveillance cameras, and perform frame extraction and image preprocessing on the video streams.

[0007] S2. Analyze the video frame using a multimodal large model and generate a corresponding text description;

[0008] S3, retrieving risk information related to the text description based on retrieval enhancement generation technology and generating security recommendations;

[0009] S4. Conduct risk assessment on the analysis results and send real-time warning notifications to relevant personnel when the warning threshold is exceeded.

[0010] Optionally, the video stream is obtained through the RTSP or RTMP protocol and transmitted using H.264 or H.265 encoding; the frame rate of the frame extraction is 5-10 frames per second, and the extracted frames are pre-processed by denoising and contrast enhancement.

[0011] Optionally, step S2 specifically includes:

[0012] Use target detection algorithms to identify flames, smoke, and human targets in video frames;

[0013] Use action recognition algorithms to detect abnormal behavior of people;

[0014] The pre-trained multimodal model is used to convert the video frame content into natural language text descriptions.

[0015] Optionally, step S3 specifically includes:

[0016] Pre-vectorize and store historical risk event data in a vector database;

[0017] Based on the text description in step S2, a similarity search is performed in the vector database to obtain relevant risk case information, and the retrieval enhancement generation model combines the retrieval results to generate corresponding safety disposal suggestions.

[0018] Optionally, step S4 may include:

[0019] S4.1. Calculate the risk index of fire, explosion, or leakage incidents using the risk assessment model. The risk assessment model expression is as follows:

[0020] Risk index = L*E*C,

[0021] L represents the probability of an accident. This probability is dynamically assessed based on historical equipment failure data, routine maintenance records, and environmental parameters, combined with a machine learning algorithm. The probability is divided into five levels: "very low, low, medium, high, and very high," with corresponding scores of 1-5.

[0022] E represents the frequency of personnel exposure to hazardous environments. The video analysis system monitors the time and frequency of personnel staying in hazardous areas in real time. Combined with the access control system data, it is divided into six levels according to "several times a year, several times a month, several times a week, several times a day, and continuous exposure", with corresponding scores of 1-6;

[0023] C represents the possible consequences of an accident, which is divided into five levels of "minor, small, moderate, major, and catastrophic" based on the gas station's oil storage capacity, surrounding population density, and firefighting facilities, with corresponding scores of 1-15 points;

[0024] Multiply the scores of the three parameters L, E, and C to obtain the risk index;

[0025] S4.2. Compare the calculated risk index with the preset threshold to determine whether to trigger an early warning;

[0026] S4.3. When the risk index exceeds the preset threshold, an early warning message is immediately sent to relevant personnel at the gas station through a low-latency communication channel. The early warning content includes: warning information, risk type, location, risk level, and safety recommendations generated by the intelligent decision-making system based on the risk type.

[0027] In a second aspect, the present invention provides a gas station risk warning system based on video streams and a multimodal large model, which solves the above-mentioned technical problems using the following technical solutions:

[0028] A gas station risk warning system based on video streaming and multimodal large models, comprising:

[0029] Video stream acquisition module, used to obtain real-time video stream of the gas station through surveillance cameras;

[0030] A preprocessing module is used to extract frames and preprocess images of the video stream;

[0031] The data conversion module is used to analyze the pre-processed video frames using a multimodal large model and generate corresponding text descriptions;

[0032] The retrieval and generation module is used to retrieve risk information related to the text description and generate security recommendations based on retrieval-enhanced generation technology;

[0033] The assessment and early warning module is used to conduct risk assessment on the analysis results and send real-time early warning notifications to relevant personnel when the early warning threshold is exceeded.

[0034] Optionally, the video stream acquisition module obtains the video stream through the RTSP or RTMP protocol and transmits it using H.264 or H.265 encoding;

[0035] The preprocessing module extracts frames from the video stream at 5-10 frames per second, and performs preprocessing of denoising and contrast enhancement on the extracted frames.

[0036] Optionally, the data conversion modules involved specifically include:

[0037] A target recognition unit is used to recognize flames, smoke and human targets in video frames using a target detection algorithm;

[0038] Anomaly recognition unit, used to detect abnormal behavior of personnel using action recognition algorithms;

[0039] The text generation unit is used to convert the video frame content into a natural language text description using a pre-trained multimodal model.

[0040] Optionally, the retrieval and generation modules involved specifically include:

[0041] Vector database, used to pre-store vectorized historical risk event data;

[0042] A similarity retrieval unit is used to perform similarity retrieval in a vector database based on text descriptions to obtain relevant risk case information;

[0043] The suggestion generation unit is used to generate corresponding safety disposal suggestions by using the retrieval enhancement generation model combined with the retrieval results.

[0044] Optional assessment and early warning modules include:

[0045] The risk assessment model is used to calculate the risk index of fire, explosion or leakage events. The expression is as follows:

[0046] Risk index = L*E*C,

[0047] L represents the probability of an accident. This probability is dynamically assessed based on historical equipment failure data, routine maintenance records, and environmental parameters, combined with a machine learning algorithm. The probability is divided into five levels: "very low, low, medium, high, and very high," with corresponding scores of 1-5.

[0048] E represents the frequency of personnel exposure to hazardous environments. The video analysis system monitors the time and frequency of personnel staying in hazardous areas in real time. Combined with the access control system data, it is divided into six levels according to "several times a year, several times a month, several times a week, several times a day, and continuous exposure", with corresponding scores of 1-6;

[0049] C represents the possible consequences of an accident, which is divided into five levels of "minor, small, moderate, major, and catastrophic" based on the gas station's oil storage capacity, surrounding population density, and firefighting facilities, with corresponding scores of 1-15 points;

[0050] Multiply the scores of the three parameters L, E, and C to obtain the risk index;

[0051] An early warning judgment unit is used to compare the calculated risk index with a preset threshold to determine whether an early warning is triggered;

[0052] The early warning sending unit is used to send early warning information to relevant personnel at the gas station through a low-latency communication channel when the risk index exceeds the preset threshold. The early warning content includes: warning information, risk type, location, risk level and safety recommendations generated by the intelligent decision-making system based on the risk type.

[0053] The gas station risk warning method and system based on video stream and multimodal large model of the present invention has the following beneficial effects compared with the existing technology:

[0054] 1. This invention collects video from surveillance cameras in real time, and then goes through frame extraction and preprocessing, multimodal large model analysis and text description generation, risk information retrieval and safety recommendation generation based on retrieval enhancement generation technology, risk assessment and judgment, and finally pushes real-time warning information to relevant personnel. This fully demonstrates the entire process from video acquisition to warning push, can improve the accuracy of risk identification and significantly increase response speed, and can provide real-time safety recommendations for different types of risks, ensuring the safe operation of gas stations.

[0055] 2. Through real-time analysis of video streams and multimodal data fusion, the present invention can promptly detect safety hazards within gas stations and make rapid and accurate early warning responses to complex environmental changes. By leveraging the intelligent analysis of large models and knowledge retrieval enhanced by retrieval generation technology, it not only provides alarm functions but also provides specific emergency response suggestions to help managers make scientific decisions and avoid situations where there are only alarms but no countermeasures. The entire process of risk identification, analysis, and notification is automated without the need for human intervention, greatly improving the efficiency of safety monitoring and reducing human oversight. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Attachment Figure 1 is a flow chart of a method according to embodiment 1 of the present invention;

[0057] Attachment Figure 2 This is a module connection block diagram of the second embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the technical solution, the technical problems solved and the technical effects of the present invention more clear, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.

[0059] Example 1:

[0060] Combined with attachment Figure 1 This embodiment proposes a gas station risk warning method based on video stream and multimodal large model, which includes the following steps:

[0061] S1. Obtain the real-time video stream of the gas station through the surveillance camera, and perform frame extraction and image preprocessing on the video stream.

[0062] During specific implementation, the video stream is obtained through the RTSP or RTMP protocol and transmitted using H.264 or H.265 encoding; the frame rate of the frame extraction is 5-10 frames per second, and the extracted frames are pre-processed by denoising and contrast enhancement.

[0063] S2. Analyze the video frame using a large multimodal model and generate a corresponding text description, specifically including:

[0064] Use target detection algorithms to identify flames, smoke, and human targets in video frames;

[0065] Use action recognition algorithms to detect abnormal behavior of people;

[0066] The pre-trained multimodal model CLIP (Contrastive Language-Image Pre-training) is used to convert the video frame content into a natural language text description.

[0067] S3. Retrieving risk information related to the text description and generating security recommendations based on retrieval enhancement generation technology, specifically including:

[0068] Pre-vectorize and store historical risk event data in a vector database;

[0069] Based on the text description in step S2, a similarity search is performed in the vector database to obtain relevant risk case information, and the retrieval enhancement generation model combines the retrieval results to generate corresponding safety disposal suggestions.

[0070] S4. Conduct risk assessment on the analysis results and send real-time warning notifications to relevant personnel when the warning threshold is exceeded. Specifically, it includes:

[0071] S4.1. Calculate the risk index of fire, explosion, or leakage incidents using the risk assessment model. The risk assessment model expression is as follows:

[0072] Risk index = L*E*C,

[0073] L represents the probability of an accident. This probability is dynamically assessed based on historical equipment failure data, routine maintenance records, and environmental parameters, combined with a machine learning algorithm. The probability is divided into five levels: "very low, low, medium, high, and very high," with corresponding scores of 1-5.

[0074] E represents the frequency of personnel exposure to hazardous environments. The video analysis system monitors the time and frequency of personnel staying in hazardous areas in real time. Combined with the access control system data, it is divided into six levels according to "several times a year, several times a month, several times a week, several times a day, and continuous exposure", with corresponding scores of 1-6;

[0075] C represents the possible consequences of an accident, which is divided into five levels of "minor, small, moderate, major, and catastrophic" based on the gas station's oil storage capacity, surrounding population density, and firefighting facilities, with corresponding scores of 1-15 points;

[0076] Multiply the scores of the three parameters L, E, and C to obtain the risk index;

[0077] S4.2. Compare the calculated risk index with the preset threshold to determine whether to trigger an early warning;

[0078] S4.3. When the risk index exceeds the preset threshold, the early warning information will be immediately sent to the relevant personnel of the gas station through the low-latency communication channel. The early warning content includes: warning information (including the specific location where the risk occurs), risk type (fire, explosion, leakage), location, risk level (corresponding to the risk index, including low risk, general risk, increased risk, major risk and extremely large risk) and calling the intelligent decision-making system to generate safety recommendations based on the risk type (for example, for fire risk, immediately cut off the power supply, start the fire extinguishing device, and evacuate the surrounding personnel; for explosion, immediately cut off the power supply and sound a sharp alarm; for leakage risk, stop operations, set up a warning area, and use adsorption materials to deal with the leak).

[0079] Example 2:

[0080] Combined with attachment Figure 2 This embodiment proposes a gas station risk warning system based on video streams and a multimodal large model, which includes:

[0081] Video stream acquisition module, used to obtain real-time video stream of the gas station through surveillance cameras;

[0082] A preprocessing module is used to extract frames and preprocess images of the video stream;

[0083] The data conversion module is used to analyze the pre-processed video frames using a multimodal large model and generate corresponding text descriptions;

[0084] The retrieval and generation module is used to retrieve risk information related to the text description and generate security recommendations based on retrieval-enhanced generation technology;

[0085] The assessment and early warning module is used to conduct risk assessment on the analysis results and send real-time early warning notifications to relevant personnel when the early warning threshold is exceeded.

[0086] In this embodiment, the video stream acquisition module obtains the video stream through the RTSP or RTMP protocol and transmits it using H.264 or H.265 encoding; the preprocessing module extracts frames from the video stream at 5-10 frames per second and performs denoising and contrast enhancement preprocessing on the extracted frames.

[0087] In this embodiment, the data conversion module specifically includes:

[0088] A target recognition unit is used to recognize flames, smoke and human targets in video frames using a target detection algorithm;

[0089] Anomaly recognition unit, used to detect abnormal behavior of personnel using action recognition algorithms;

[0090] The text generation unit is used to convert the video frame content into a natural language text description using the pre-trained multimodal model CLIP (Contrastive Language-Image Pre-training).

[0091] In this embodiment, the retrieval and generation modules specifically include:

[0092] Vector database, used to pre-store vectorized historical risk event data;

[0093] A similarity retrieval unit is used to perform similarity retrieval in a vector database based on text descriptions to obtain relevant risk case information;

[0094] The suggestion generation unit is used to generate corresponding safety disposal suggestions by using the retrieval enhancement generation model combined with the retrieval results.

[0095] In this embodiment, the evaluation and warning module specifically includes:

[0096] The risk assessment model is used to calculate the risk index of fire, explosion or leakage events. The expression is as follows:

[0097] Risk index = L*E*C,

[0098] L represents the probability of an accident. This probability is dynamically assessed based on historical equipment failure data, routine maintenance records, and environmental parameters, combined with a machine learning algorithm. The probability is divided into five levels: "very low, low, medium, high, and very high," with corresponding scores of 1-5.

[0099] E represents the frequency of personnel exposure to hazardous environments. The video analysis system monitors the time and frequency of personnel staying in hazardous areas in real time. Combined with the access control system data, it is divided into six levels according to "several times a year, several times a month, several times a week, several times a day, and continuous exposure", with corresponding scores of 1-6;

[0100] C represents the possible consequences of an accident, which is divided into five levels of "minor, small, moderate, major, and catastrophic" based on the gas station's oil storage capacity, surrounding population density, and firefighting facilities, with corresponding scores of 1-15 points;

[0101] Multiply the scores of the three parameters L, E, and C to obtain the risk index;

[0102] An early warning judgment unit is used to compare the calculated risk index with a preset threshold to determine whether an early warning is triggered;

[0103] The early warning sending unit is used to send early warning information to relevant personnel of the gas station through a low-latency communication channel in real time when the risk index exceeds the preset threshold. The early warning content includes: warning information (including the specific location where the risk occurs), risk type (fire, explosion, leakage), location, risk level (corresponding to the risk index, including low risk, general risk, increased risk, major risk and extremely large risk) and calling the intelligent decision-making system to generate safety recommendations based on the risk type (for example, for fire risk, immediately cut off the power supply, start the fire extinguishing device, and evacuate the surrounding personnel; for explosion, immediately cut off the power supply and sound a sharp alarm; for leakage risk, stop operation, set up a warning area, and use adsorption materials to deal with the leak).

[0104] In summary, the gas station risk warning method and system based on video stream and multimodal large model of the present invention, through frame cutting and deep learning analysis of monitoring video, converts each frame into a text description, combines retrieval enhancement generation technology to identify potential safety hazards in gas stations in real time and generate corresponding safety disposal suggestions, which not only improves the accuracy of risk identification, but also speeds up the response speed, ensuring the safe operation of gas stations.

[0105] The above specific examples are used to illustrate the principles and implementation methods of the present invention in detail. These examples are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made by those skilled in the art without departing from the principles of the present invention should fall within the scope of patent protection of the present invention.

Claims

1. A gas station risk warning method based on video stream and multimodal large model, characterized by: The steps include: S1. Obtain real-time video streams from the gas station through surveillance cameras, and perform frame extraction and image preprocessing on the video streams. S2. Analyze the video frame using a multimodal large model and generate a corresponding text description; S3, retrieving risk information related to the text description based on retrieval enhancement generation technology and generating security recommendations; S4. Conduct risk assessment on the analysis results and send real-time warning notifications to relevant personnel when the warning threshold is exceeded.

2. The gas station risk warning method based on video stream and multimodal large model according to claim 1 is characterized in that: The video stream is obtained through the RTSP or RTMP protocol and transmitted using H.264 or H.265 encoding; the frame rate of the frame extraction is 5-10 frames per second, and the extracted frames are pre-processed by denoising and contrast enhancement.

3. The gas station risk warning method based on video stream and multimodal large model according to claim 1 is characterized in that: The step S2 specifically includes: Use target detection algorithms to identify flames, smoke, and human targets in video frames; Use action recognition algorithms to detect abnormal behavior of people; The pre-trained multimodal model is used to convert the video frame content into natural language text descriptions.

4. The gas station risk warning method based on video stream and multimodal large model according to claim 1 is characterized in that: The step S3 specifically includes: Pre-vectorize and store historical risk event data in a vector database; Based on the text description in step S2, a similarity search is performed in the vector database to obtain relevant risk case information, and the retrieval enhancement generation model combines the retrieval results to generate corresponding safety disposal suggestions.

5. The gas station risk warning method based on video stream and multimodal large model according to claim 1 is characterized in that: The step S4 specifically includes: S4.

1. Calculate the risk index of fire, explosion, or leakage incidents using the risk assessment model. The risk assessment model expression is as follows: Risk index = L*E*C, L represents the probability of an accident. This probability is dynamically assessed based on historical equipment failure data, routine maintenance records, and environmental parameters, combined with a machine learning algorithm. The probability is divided into five levels: "very low, low, medium, high, and very high," with corresponding scores of 1-5. E represents the frequency of personnel exposure to hazardous environments. A video analysis system monitors the time and frequency of personnel in hazardous areas in real time. Combined with access control system data, the system categorizes personnel exposure into six levels, with scores ranging from 1 to 6, based on "a few times a year, a few times a month, a few times a week, a few times a day, and continuous exposure." C represents the possible consequences of an accident, which is divided into five levels: "minor, minor, moderate, major, and catastrophic", with corresponding scores of 1-15 points, taking into account the gas station's fuel storage capacity, surrounding population density, and fire protection facilities. Multiply the scores of the three parameters L, E, and C to obtain the risk index; S4.

2. Compare the calculated risk index with the preset threshold to determine whether to trigger an early warning; S4.

3. When the risk index exceeds the preset threshold, an early warning message is immediately sent to relevant personnel at the gas station through a low-latency communication channel. The early warning content includes: warning information, risk type, location, risk level, and safety recommendations generated by the intelligent decision-making system based on the risk type.

6. A gas station risk warning system based on video stream and multimodal large model, characterized by: It includes: Video stream acquisition module, used to obtain real-time video stream of the gas station through surveillance cameras; A preprocessing module is used to extract frames and preprocess images of the video stream; The data conversion module is used to analyze the pre-processed video frames using a multimodal large model and generate corresponding text descriptions; The retrieval and generation module is used to retrieve risk information related to the text description and generate security recommendations based on retrieval-enhanced generation technology; The assessment and early warning module is used to conduct risk assessment on the analysis results and send real-time early warning notifications to relevant personnel when the early warning threshold is exceeded.

7. The gas station risk warning system based on video stream and multimodal large model according to claim 6 is characterized in that: The video stream acquisition module acquires the video stream through the RTSP or RTMP protocol and transmits it using H.264 or H.265 encoding; The preprocessing module extracts frames from the video stream at 5-10 frames per second, and performs preprocessing of denoising and contrast enhancement on the extracted frames.

8. The gas station risk warning system based on video stream and multimodal large model according to claim 6 is characterized in that: The data conversion module specifically includes: A target recognition unit is used to recognize flames, smoke and human targets in video frames using a target detection algorithm; Anomaly recognition unit, used to detect abnormal behavior of personnel using action recognition algorithms; The text generation unit is used to convert the video frame content into a natural language text description using a pre-trained multimodal model.

9. The gas station risk warning system based on video stream and multimodal large model according to claim 6 is characterized in that: The retrieval and generation module specifically includes: Vector database, used to pre-store vectorized historical risk event data; A similarity retrieval unit is used to perform similarity retrieval in a vector database based on text descriptions to obtain relevant risk case information; The suggestion generation unit is used to generate corresponding safety disposal suggestions by using the retrieval enhancement generation model combined with the retrieval results.

10. The gas station risk warning system based on video stream and multimodal large model according to claim 6 is characterized in that: The assessment and early warning module specifically includes: The risk assessment model is used to calculate the risk index of fire, explosion or leakage events. The expression is as follows: Risk index = L*E*C, L represents the probability of an accident. This probability is dynamically assessed based on historical equipment failure data, routine maintenance records, and environmental parameters, combined with a machine learning algorithm. The probability is divided into five levels: "very low, low, medium, high, and very high," with corresponding scores of 1-5. E represents the frequency of personnel exposure to hazardous environments. A video analysis system monitors the time and frequency of personnel in hazardous areas in real time. Combined with access control system data, the system categorizes personnel exposure into six levels, with scores ranging from 1 to 6, based on "a few times a year, a few times a month, a few times a week, a few times a day, and continuous exposure." C represents the possible consequences of an accident, which is divided into five levels: "minor, minor, moderate, major, and catastrophic", with corresponding scores of 1-15 points, taking into account the gas station's fuel storage capacity, surrounding population density, and fire protection facilities. Multiply the scores of the three parameters L, E, and C to obtain the risk index; An early warning judgment unit is used to compare the calculated risk index with a preset threshold to determine whether an early warning is triggered; The early warning sending unit is used to send early warning information to relevant personnel at the gas station through a low-latency communication channel when the risk index exceeds the preset threshold. The early warning content includes: warning information, risk type, location, risk level and safety recommendations generated by the intelligent decision-making system based on the risk type.

Images