Gas station safety intelligent monitoring method and system based on multi-mode data analysis

Abstract

This invention provides a method and system for intelligent safety monitoring of gas stations based on multimodal data analysis, relating to the fields of safety monitoring and artificial intelligence. The method includes the following steps: collecting multiple real-time video streams and IoT sensor status data; extracting visual feature information from the multiple real-time video streams; fusing the visual feature information with the IoT sensor status data using multimodal data fusion to construct a real-time status vector for the monitoring area; performing temporal logic analysis and process compliance determination on the status vector; and generating and pushing alarm events when violations or safety hazards are detected. This invention improves the intelligent perception capability and alarm response timeliness of complex operational processes through multimodal data fusion analysis; simultaneously, based on a closed-loop feedback mechanism, it continuously optimizes the artificial intelligence model, effectively solving the problems of small target recognition and high false alarm rates, achieving intelligent and precise control of gas stations around the clock and throughout the entire process.

Description

Technical Field

[0001] This invention belongs to the field of safety monitoring and artificial intelligence technology, specifically relating to a method and system for intelligent safety monitoring of gas stations based on multi-modal data analysis, which is particularly suitable for all-weather, full-process intelligent safety supervision of refueling areas, unloading areas and oil tank areas of highway service area gas stations. Background Technology

[0002] Gas stations, as retail outlets for refined oil products, are typical flammable and explosive hazardous areas, and their operational safety is crucial to public safety. With the continuous expansion of the highway network, service area gas stations are characterized by their dispersed locations, large number, and wide management radius. Currently, safety supervision of gas stations mainly relies on traditional video surveillance systems and remote centralized management platforms; however, both models have revealed insurmountable technical limitations when facing the ever-increasing demands for safety control.

[0003] The core function of traditional video surveillance systems is 24 / 7 recording, and their technical architecture dictates that they are essentially post-incident tracking tools. When safety hazards or violations occur, the system can only passively record the scene, unable to identify and warn of ongoing dangerous behavior in real time. For example, if someone uses a mobile phone or smokes in the refueling area, although the surveillance footage records the entire process, the system cannot proactively detect this violation and issue a warning. The problem can only be discovered when management reviews the recording after the incident. This post-incident verification model is far removed from the concept of proactive safety defense and cannot meet the urgent needs of modern gas stations for real-time safety management.

[0004] While remote centralized management platforms attempt to compensate for the shortcomings of traditional monitoring by aggregating video signals from various gas stations to a monitoring center for review by designated personnel, this model still fundamentally relies on human visual attention and judgment. Due to limited manpower, monitoring personnel cannot simultaneously and continuously monitor all footage from all stations, making them highly susceptible to visual fatigue and oversights. Studies have shown that human monitoring personnel experience a significant decline in concentration and a sharp increase in missed detection rates after continuously viewing multiple monitoring feeds for 30 minutes. Therefore, this manual monitoring model cannot fundamentally solve the real-time problem, and most security risks remain undetected in a timely manner.

[0005] In terms of data perception, existing technical solutions suffer from serious "perception blind spots." While some gas stations are equipped with sensor alarm systems, such as high / low liquid level alarms and combustible gas detector alarms, these sensors can only detect threshold exceedances of physical parameters, representing typical "point-based" monitoring. However, many safety hazards at gas stations involve complex and dynamic processes, such as the standardization of unloading procedures (e.g., whether the oil has been stabilized for 5 minutes, and whether the static grounding clamp is effectively connected), violations by personnel in the refueling area (e.g., refueling without wearing anti-static work clothes), and identifying early fire signs such as smoke and open flames in the monitoring footage. These complex behaviors and processes cannot be identified by simple sensors, and traditional video analytics lacks the ability to deeply understand image content, creating a significant technological gap.

[0006] Furthermore, the existing system suffers from severe data silos. Video data, sensor alarm data, and work process records are often stored separately in different subsystems, lacking effective correlation mechanisms. When a sensor triggers an alarm, the system cannot automatically correlate the alarm information with the corresponding video clip or the ongoing work process (such as an oil unloading operation). Upon receiving an alarm, managers must manually search for the corresponding video footage and compare it manually to determine the authenticity and severity of the alarm, resulting in extremely low traceability efficiency. More importantly, the system lacks a tracking and recording mechanism for the handling results of alarm events (such as whether rectification has been completed, whether it was a false alarm or a valid alarm), failing to form a complete management loop from "perception, identification, alarm, handling, and backtracking," leading to a lack of data support for safety management optimization.

[0007] In terms of data processing efficiency, existing technologies lack an effective method for integrating multi-source heterogeneous data such as video streams, equipment status, and work process timelines. For example, simply identifying "people gathering" based on image recognition as an anomaly could easily lead to numerous false alarms due to customers queuing to pay normally. Furthermore, complex AI (artificial intelligence) recognition algorithms consume enormous amounts of computing power. Processing video streams from all sites centrally on a server would result in significant network bandwidth pressure and computational latency, rendering real-time alarm notifications meaningless.

[0008] In summary, the industry urgently needs an intelligent monitoring method and system that can integrate multi-source data, possess complex event perception capabilities, and achieve localized real-time processing and cloud-based collaborative management to overcome the many shortcomings of existing technologies and realize all-weather, full-process, and intelligent safety control of gas stations. Summary of the Invention

[0009] This invention aims to overcome the shortcomings of existing technologies and provide a method and system for intelligent safety monitoring of gas stations based on multi-modal data analysis. This addresses technical problems in existing technologies, such as reliance on manual experience in supervision leading to a lack of real-time performance, single data dimensions failing to perceive complex scenarios, severe information silos lacking streamlined closed-loop management, and insufficient multi-source data processing capabilities resulting in high false alarm rates and delayed responses. For ease of description, the following terms are defined in this invention: Central platform: refers to the centralized management software system deployed at the superior management unit, used to receive, display, and manage alarm data from all subordinate gas stations, providing GIS map display, multi-dimensional data statistics, and system management functions. When it is necessary to emphasize its superior status relative to the local gas station, it is called the "superior central platform".

[0010] Central business server: refers to the physical server equipment that runs the central platform, including hardware resources such as CPU, memory, and hard disk.

[0011] Central Business System: Synonymous with "Central Platform", it refers to the overall software system contained in the central platform, emphasizing its business processing functions.

[0012] The above terms all refer to the same technical entity, namely the centralized management system deployed in the superior management unit. Different terms may be selected in different contexts as needed.

[0013] To achieve the above objectives, the present invention provides the following technical solution: A method for intelligent safety monitoring of gas stations based on multi-modal data analysis includes the following steps: Collect video stream data and IoT sensor status data from gas stations; Several AI analysis tasks are run in parallel on the video stream data to extract visual feature information; The visual feature information is fused with IoT sensor status data in a multimodal manner to construct a real-time status vector of the monitoring area. Based on a predefined business rule base and process state machine model, the real-time state vector is subjected to time-series logic analysis and process compliance determination. An alarm event is generated when violations or security risks are detected.

[0014] Furthermore, several AI analysis tasks are run in parallel on the video stream data, including: The video stream data is decoded and frames are extracted. The extracted video frames are then input into several parallel deep learning models. The deep learning models include a people detection and tracking model, a vehicle detection and tracking model, a small target detection model, a people behavior recognition model, and a scene semantic understanding model.

[0015] Furthermore, the step of performing time-series logic analysis and process compliance determination on the real-time state vector based on a predefined business rule base and process state machine model includes: Define the set of ideal states :

[0016] in, Indicates the first step in the workflow. Individual events, The total number of sub-events; For each child event Define detection function :

[0017] in, for Multimodal data input at any given time. For video frame data, For IoT sensor data; For sub-events The scoring function for training; For the child incident The threshold for judgment.

[0018] Furthermore, the process compliance determination also includes steps such as verifying timing constraints, calculating the work process integrity score, and executing anomaly determination rules, wherein: Verify timing constraints:

[0019] in, , Indicates sub-event The end time; when the timing constraint is not met, it is judged as a process timing disorder abnormality; Calculate the job process integrity score:

[0020] in, This represents the total number of steps in the ideal process. For the indicator function, when a child event occurs Its value is 1 if it is successfully detected during the entire operation, and 0 otherwise; Score the process completeness; Execution exception judgment rules: Rule A: Child events exist. Throughout the entire work cycle Inside, its detection function Hengchengli:

[0021] Rule B: Observing a sub-event Time of occurrence Earlier than the incident End time ,and :

[0022] Rule C: For sub-events with duration requirements Its duration Minimum duration requirement not met:

[0023] When any one of the conditions in rule A, rule B, or rule C is met, the workflow is determined to be abnormal and an alarm is triggered.

[0024] Furthermore, the generation of alarm events includes alarm confidence calculation:

[0025] in, For the final alarm confidence level; Confidence level based on visual detection; Confidence level based on logical violations; These are weighting coefficients, and their values ​​range from [value range missing]. .

[0026] Furthermore, the method also includes the step of associating and storing alarm events with corresponding evidence information, which includes: Generate a globally unique event ID, which includes a gas station identifier, a monitoring area identifier, an occurrence timestamp, and a sequence number; The core information of the alarm event is associated with the evidence information, which includes the on-site image encoding data at the time of the alarm, the image storage address, and the video clip storage address. For alarms related to work processes, further associate them with work process context information, including work process ID, list of missing steps, and integrity score.

[0027] Furthermore, it also includes the step of pushing alarm events to on-site broadcast terminals, the superior central platform, and / or mobile devices in real time, which includes: Set differentiated push strategies based on the severity level of the alarm event. For alarm events with a severity level of "severe", the on-site broadcast terminal will be immediately triggered to broadcast the alarm via voice. At the same time, the alarm will be pushed to the upper-level central platform through the message queue middleware and to the designated management personnel's mobile terminal via WeChat official account or SMS. For alarm events with a severity level of "general", they are recorded in the local alarm list and pushed to the upper-level central platform, without triggering real-time voice broadcast.

[0028] Furthermore, edge computing nodes are used to run several AI analysis tasks in parallel on the video stream data to extract visual feature information, which includes the step of self-monitoring the state of the edge computing nodes: The edge computing node sends a heartbeat message to the upper-level central platform every first preset time interval; The edge computing node reports its operating status information to the upper-level central platform every second preset time interval.

[0029] Furthermore, the method also includes a step of closed-loop processing of alarm events: Receive feedback on the handling of alarm events, including two types of feedback: rectified and false alarm; The feedback information is associated with and stored with the original alarm event; Alarm events marked as false alarms are treated as negative samples, while normal operating procedures that do not trigger alarms are treated as positive samples.

[0030] This invention also provides a gas station safety intelligent monitoring system based on multi-modal data analysis, which executes the monitoring method described above. The system includes an edge-side sensing layer, an edge-side processing layer, a cloud-side management layer, and a broadcast terminal layer, wherein: The edge-side perception layer includes several camera devices and IoT sensors, which are used to collect video stream data and IoT sensor status data, respectively. The edge processing layer includes a video stream access module, a multi-task AI analysis engine, a multi-modal data fusion module, a process state machine determination module, an alarm event management module, and a local storage module; The cloud-side management layer is deployed on the central business server, which includes a message queue module, a GIS map display module, a multi-dimensional data statistics module, and a system management module. The broadcast terminal layer is deployed on large screens and voice broadcast all-in-one machines to display alarm information and broadcast it via voice in real time.

[0031] Compared with the prior art, the present invention has the following significant advantages: This invention achieves a proactive safety defense model encompassing post-event investigation, pre-event warning, and in-event intervention. Addressing the technical limitations of traditional video surveillance systems mentioned in the background, which can only perform post-event investigation and cannot provide real-time warnings, this invention deploys an AI analysis all-in-one machine locally at gas stations to perform 24 / 7 uninterrupted real-time intelligent analysis of multiple video streams. When a customer makes a phone call in the refueling area, the system can complete the entire process of identification, judgment, and voice announcement within 3 seconds of the action, achieving a real-time control effect of "detection and prevention." This proactive defense model shortens the discovery time of safety hazards from "hours or even days after the event" to "seconds during the event," greatly improving the timeliness and effectiveness of safety management. It also achieves deep perception and intelligent judgment of complex operational processes. A globally unique event ID is generated for each alarm, and evidence images, evidence videos, and process context information are associated and stored, enabling automatic association between alarm information and business processes. Meanwhile, the system provides multiple feedback channels for handling, receiving confirmation information from on-site personnel, and storing the handling results in association with the original alarms, forming a complete management closed loop from "perception-identification-alarm-handling-backtracking". It achieves efficient collaboration between edge real-time processing and cloud-based collaborative management. The original video stream is stored locally, with only structured alarm data and device status information uploaded, significantly reducing network bandwidth pressure and ensuring video data security. It enables data-driven continuous learning and model self-optimization, using alarm events marked as "false alarms" as negative samples and normal operating procedures that do not trigger alarms as positive samples to build an incremental training dataset for regular optimization and updates to the AI ​​model. This improves detection reliability. Attached Figure Description

[0032] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings: Figure 1 This is a schematic diagram of the architecture of the intelligent safety monitoring system for gas stations provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the state machine model of the oil unloading operation process provided in an embodiment of the present invention; Figure 3 This is a flowchart of the multi-level alarm information push mechanism provided in the embodiments of the present invention; Figure 4 This is a time-series diagram of real-time violation monitoring in the refueling area provided in an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it.

[0034] This invention provides a method for intelligent safety monitoring of gas stations based on multi-modal data analysis, which includes the following steps: S100. Collects multiple real-time video streams and IoT sensor status data from various monitoring areas of the gas station; S200. Run multiple AI analysis tasks in parallel on the multiple real-time video stream data to extract visual feature information, including personnel targets, vehicle targets, object targets, behavioral features, and scene semantic features; Preferably, multiple AI analysis tasks are run in parallel on the multiple real-time video streams at the edge computing node; Running multiple AI analysis tasks to extract visual feature information, specifically including: Each video stream is decoded and its frames are extracted. The extracted video frames are then input into multiple parallel deep learning models, which include at least a people detection and tracking model, a vehicle detection and tracking model, a small target detection model, a people behavior recognition model, and a scene semantic understanding model. The small target detection model is designed to identify target objects such as electrostatic grounding clamps, fire extinguishers, samplers, and fuel nozzles that occupy less than 5% of the pixels in the monitoring screen.

[0035] S300. The visual feature information is fused with the IoT sensor status data in a multimodal manner to construct a real-time status vector of the monitoring area; S400. Based on a predefined business rule base and process state machine model, perform time-series logic analysis and process compliance determination on the real-time state vector. When a violation or security risk is detected, an alarm event is generated. The logic analysis and compliance determination specifically include: Taking oil unloading as an example, an ideal state set is defined for oil unloading operations. :

[0036] in, Indicates the first step in the workflow. Sub-events (or steps) The total number of sub-events during the oil unloading operation This indicates the first step in the oil unloading operation process. Sub-events (or steps); The invention appears or "Each event" represents a sub-event in the workflow, and all belong to the aforementioned set of ideal states. Subscripts such as i, i+1, j, k, etc., are used to distinguish different sub-events (or steps), or to identify their characteristics, for example... It means a key sub-event (or key step), for example... These respectively represent the steps of guiding, stabilizing oil, static electricity, fire prevention, sampling, signing, unloading oil, leaving the site, and tidying up; for example... This indicates a sub-event. Is the incident The preceding sub-event, Indicates sub-event Time of occurrence Earlier than its predecessor event End time .

[0037] For each child event Define detection function :

[0038] in, for Multimodal data input at any given time. For video frame data, For IoT sensor data; For sub-events The scoring function for training; For the child incident The threshold for judgment.

[0039] Preferably, the following timing constraints are used for verification:

[0040] in, Indicates sub-event The start time, Indicates sub-event The end time; when the timing constraint is not met, it is judged as a process timing disorder abnormality.

[0041] More preferably, the process compliance determination further includes a sub-step of calculating the process integrity score C, which is calculated using the following formula:

[0042] in, This represents the total number of steps in the ideal process. For the indicator function, when a child event occurs Its value is 1 if it is successfully detected during the entire operation, and 0 otherwise; Score the completeness of the process.

[0043] More preferably, the process compliance determination also includes the execution of anomaly determination rules: Rule A: There is a critical step. Throughout the entire work cycle Inside, its detection function Hengchengli:

[0044] Rule B: Observed event Time of occurrence Earlier than its preceding events End time ,and :

[0045] Rule C: For sub-events with duration requirements Its duration Minimum duration requirement not met:

[0046] When any of the conditions in Rule A, Rule B, or Rule C are met, the oil unloading operation process is deemed abnormal and an alarm is triggered.

[0047] Preferably, to improve alarm accuracy and prevent false alarms, a sub-step of calculating the alarm confidence level for the alarm event is included before triggering the alarm:

[0048] in, For the final alarm confidence level; Confidence level based on visual detection; Confidence level based on logical violations; These are weighting coefficients, and their values ​​range from [value range missing]. This is used to balance the importance of visual and logical evidence.

[0049] S500. The alarm event is associated with and stored with the corresponding evidence information, and pushed to the on-site broadcast terminal, the superior central platform and / or mobile device in real time.

[0050] Preferably, the associated storage specifically includes: Generate a globally unique event ID, which includes at least the gas station identifier, the monitoring area identifier, the occurrence timestamp, and the sequence number; The core information of the alarm event is associated with the evidence information and stored together. The evidence information includes at least Base64 encoded data of the scene image at the time of the alarm, the image storage URL address, and the video clip storage URL address. Taking oil unloading operation process alarms as an example, the alarms are further associated with the context information of the operation process, including the oil unloading process ID, the list of missing steps, and the integrity score.

[0051] Preferably, the real-time push to the on-site broadcast terminal, the upper-level central platform, and / or mobile devices may include the following sub-steps: Differentiated push notification strategies are set based on the severity level of alarm events. For example, the severity level includes "severe" and "normal". For alarm events with a severity level of "severe", the on-site broadcast terminal is immediately triggered to broadcast the alarm via voice. At the same time, the alarm is pushed to the upper-level central platform through the message queue middleware, and optionally pushed to the mobile terminal of the designated administrator via WeChat official account or SMS. For alarm events with a severity level of "general", they are recorded in the local alarm list and pushed to the upper-level central platform, without triggering real-time voice broadcast.

[0052] The method of the present invention further includes a sub-step of performing closed-loop processing on alarm events: Receive feedback from on-site personnel regarding the handling of alarm events, including at least two types of feedback: "rectified" and "false alarm"; The feedback information is associated with and stored with the original alarm event; Alarm events marked as "false alarms" are used as negative samples, while normal operating procedures that do not trigger alarms are used as positive samples, which are then used for incremental training and optimization of the AI ​​analysis model.

[0053] According to another embodiment of the present invention, the intelligent monitoring method of the present invention further includes a step of self-monitoring the state of edge computing nodes, as follows: Edge computing nodes send heartbeat messages to the upper-level central platform every first preset time interval:

[0054] in, For heartbeat timestamps, Code the device; Edge computing nodes report device operating status information to the upper-level central platform every second preset time interval:

[0055] in, Total hard drive capacity, in GB; The used capacity of the hard drive is in GB. Hard drive used capacity percentage; CPU usage percentage; This represents the percentage of memory usage.

[0056] like Figure 1 As shown, another embodiment of the present invention also provides a gas station safety intelligent monitoring system based on multi-modal data analysis. This system executes the above-described monitoring method during operation. The system includes: The edge sensing layer 10 includes multiple camera guns deployed in various monitoring areas of the gas station, such as camera guns in the refueling area, camera guns in the unloading area, and camera guns in the tank area. The camera guns can also be replaced with other suitable camera devices. The edge sensing layer 10 also includes several Internet of Things (IoT) sensors. The camera and sensor devices are used to collect multi-channel real-time video stream data and IoT sensor status data, such as the status data of devices and facilities like electrostatic clamps and fire extinguishers. The edge processing layer 20 is an AI analysis all-in-one machine deployed locally at the gas station. The AI ​​analysis all-in-one machine includes a video stream access module, a multi-task AI analysis engine, a multi-modal data fusion module, a process state machine determination module, an alarm event management module, and a local storage module (i.e., a local evidence storage repository). The cloud-side management layer 30 is deployed on the central business server of the superior management unit. The central business server includes a message queue module, a GIS map display module, a multi-dimensional data statistics module, and a system management module; it also includes a central database.

[0057] The broadcast terminal layer consists of a large display screen and a voice broadcasting all-in-one machine deployed at the gas station site, used to display alarm information and broadcast it via voice in real time.

[0058] The following are examples of the application of the intelligent monitoring method and system of this invention in gas station operation scenarios.

[0059] Example 1: Intelligent monitoring and compliance assessment of the entire oil unloading process This embodiment describes in detail how the method and system of the present invention achieve intelligent monitoring and compliance determination throughout a complete oil unloading operation.

[0060] Step 1: System Initialization and Parameter Configuration Before the system is put into use, the administrator completes the following initialization configuration through the configuration interface of the central business system: Ideal configuration set for oil unloading operations in AI analytics appliance According to the "Safety Specifications for Gas Station Operations" AQ3010-2022 and actual business needs, the following definitions apply:

[0061] The threshold for each sub-event Set them to: , , , , , , , , Minimum duration requirement for oil price stabilization events minute.

[0062] Meanwhile, regions of interest (ROIs) are defined in the monitoring screen of the unloading area, including the unloading vehicle area, the fire equipment placement area, and the electrostatic clamp connection point area, for subsequent target detection and event determination.

[0063] Step 2: Real-time acquisition of multimodal data Reference Figure 1 The system architecture shown illustrates how, when a tanker truck (license plate number: XA·XXXXX) enters the unloading area of ​​a gas station: The camera (Camera_ID: CAM_XYQ_01) deployed on top of the unloading area began capturing real-time video streams. With a resolution of 1920×1080 and a frame rate of 25fps, the video stream is transmitted in real time to the AI ​​analysis all-in-one machine via the ONVIF protocol.

[0064] Meanwhile, IoT sensors deployed in the unloading area began collecting status data. The electrostatic grounding clamp sensor reports its connection status every second (0 indicates not connected, 1 indicates connected); fire extinguishers with RFID tags report whether they are placed in the designated location via a location sensor (each fire extinguisher reports independently).

[0065] The video stream access module of the AI ​​analysis all-in-one machine decodes and extracts frames from multiple video streams, extracting a key image frame every 100ms and sending it to the multi-task AI analysis engine.

[0066] Step 3: Multi-task AI parallel analysis The multi-task AI analysis engine runs the following deep learning models in parallel on video frames: (1) Vehicle Detection and Tracking Model: Detects vehicle targets in the image, identifies vehicle type (tanker truck / non-tanker truck), and continuously tracks its position and motion. The model output includes vehicle bounding box coordinates. Vehicle Category Confidence level Unique Tracking .

[0067] (2) Personnel detection and behavior recognition model: Detects personnel targets in the image, identifies key points of the human body, and judges the posture and behavior of personnel. The key behaviors to be detected include: hand gestures, smoking behavior, mobile phone use, signing actions, etc.

[0068] (3) Small target detection model: This model identifies targets that occupy a small portion of the image, such as electrostatic grounding clamps, fire extinguishers, samplers, and glass measuring cylinders. To improve the detection accuracy of small targets, the model adopts a Feature Pyramid Network (FPN) and a multi-scale training strategy. For electrostatic clamp targets, the model outputs their connection status (determined by detecting the relative positional relationship between the clamp and the tanker truck).

[0069] (4) Scene semantic understanding model: Analyze the macroscopic state of the entire unloading area, including: whether the tanker truck is stationary (for oil stability determination), whether isolation cones are placed at the entrance of the unloading area, and whether there is smoke or open flame on site.

[0070] Step 4: State machine tracking and event detection for the oil unloading process When the vehicle detection model reports the event "the tanker truck has entered the unloading position and stopped", the system creates a new unloading process instance, assigns a unique process ID: UL_20241001_001, and starts the unloading process state machine.

[0071] state machine according to Figure 2 The process sequence shown waits for each sub-event to occur in turn: Sub-event 1: Guiding the vehicle into position ( ) detection During the process of the tanker truck reversing into the parking space, the personnel behavior recognition model continuously analyzes the position of the person in charge. According to safety regulations, the person in charge must not stand directly in front of or behind the tanker truck, but should give instructions from the side at a distance of more than 1.5 meters.

[0072] Detection function Calculate the azimuth and distance of the command personnel relative to the tanker truck. Let the coordinates of the tanker truck's center point be... The direction vector of the car's front is Personnel coordinates are Then the azimuth angle of the personnel relative to the tanker truck The calculation formula is:

[0073] Distance between personnel and tanker truck The calculation formula is:

[0074] if Landing on the front direction vector Within ±30°, and If the distance is less than 1 meter, it is considered an improper positioning. Output a low score; otherwise, output a high score.

[0075] In this embodiment, the commander always stands about 2 meters to the left of the tanker truck to give instructions. Continuous output above the threshold The system records sub-events in the rating. It happened, at what time , .

[0076] Sub-event 2: Stabilize oil for 5 minutes ( ) detection After the tanker truck comes to a complete stop, the vehicle detection model continuously tracks its position. Let the tanker truck be at time [time missing]. The coordinates of the center point are If continuous Within minutes, the tanker truck's position change was less than the threshold. (Set at 0.5 meters):

[0077] The system then assumes the tanker truck is in a stable oil condition. The system starts recording oil levels from the moment the tanker truck comes to a complete stop. Start timing and continuously monitor vehicle location. When the fuel stability condition is met and the duration reaches [a certain value], [the timer will continue]. At the minute mark, the system records the sub-event. occur, , .

[0078] Sub-event 3: Electrostatic clip connection ( ) detection The small target detection model continuously monitors the status of the electrostatic grounding clamp. The model first detects whether the electrostatic clamp target exists in the image, and then detects the relative positional relationship between the electrostatic clamp and the tanker truck. If the bounding boxes of the electrostatic clamp target and the tanker truck target overlap, and the electrostatic clamp is in a suspended state, then it is considered a valid connection.

[0079] In this embodiment, due to staff negligence, the electrostatic clamp was not connected until the oil unloading pipe was about to be connected. At 10:05:30, the system's state machine still hadn't received [the necessary information]. The event triggers exception judgment rule A (step missing), generating alarm event ID: 2002 (missing valid connection electrostatic clip stage).

[0080] Sub-event 4: Placement of fire extinguishers ( ) detection The small target detection model simultaneously monitors the placement of fire-fighting equipment. The system needs to detect whether there is one large fire extinguisher (35kg) and two small fire extinguishers (5kg) in the screen, and these fire extinguishers should be placed in the designated location in the unloading area (determined by ROI area).

[0081] In this embodiment, at 10:03:00, staff placed three fire extinguishers at a designated location in the southeast corner of the unloading area. The model successfully detected the three fire extinguisher targets, and their locations were within the fire equipment ROI area. The system recorded the sub-event. occur, , .

[0082] Sub-event 5: Sampling and testing ( ) detection Sampling and testing require the identification of two key actions: taking a sample using a bottom sampler and pouring the oil sample into a glass graduated cylinder for appearance testing. The system detects the continuous occurrence of these two actions through a timing action recognition model.

[0083] In this embodiment, the staff completed the sampling and testing at 10:06:00, and the system recorded the sub-event. occur, , .

[0084] Sub-event 6: Signature confirmation ( ) detection The signing confirmation action is detected using a human posture recognition model. The model detects whether the person is holding the document and making writing motions. Because the signing action may be small in the image, it is difficult to recognize; therefore, a threshold for judging a signing event is set. The value is set relatively low (0.75), while a comprehensive judgment is made in conjunction with the timing context.

[0085] In this embodiment, the staff completed the signature confirmation at 10:07:30, and the system recorded the sub-event. occur, , .

[0086] Sub-event 7: Oil unloading process ( ) detection The oil unloading process inspection includes two aspects: first, checking whether the oil unloading pipe is connected (identifying the oil unloading pipe through a small target detection model); second, checking whether there are at least two staff members supervising the unloading on site (counting the number of people through a personnel detection model).

[0087] In this embodiment, the system detected that the unloading pipe was connected at 10:08:30, and that there were always 2-3 workers on site, which meets the specifications. The system records the sub-event. occur, , (Time of completion of oil unloading).

[0088] Sub-event 8: Tanker truck leaves the scene ( ) detection When the vehicle detection model detects that the tanker truck has started to move and has left the unloading area, it is considered a departure event. The system records the sub-event. occur, , .

[0089] Sub-event 9: Cleaning up the scene ( ) detection After the tanker truck leaves the site, the system checks whether fire-fighting equipment and electrostatic clamps have been promptly retrieved. (Detection function) Monitor the location of fire extinguishers and the status of their electrostatic clamps. If more than 10 minutes have passed since the tanker truck left the site and the fire extinguisher is still left in the unloading area or the electrostatic clamp has not been retrieved, the site cleanup is deemed non-compliant.

[0090] In this embodiment, all equipment was retrieved immediately after the tanker truck left the site, and the system recorded the sub-event. occur, , .

[0091] Step 5: Comprehensive assessment of process compliance After the oil unloading operation is completed, the system calls a compliance assessment model to perform a comprehensive score and anomaly detection: (1) First, perform timing constraint verification to check whether the order of occurrence of each sub-event conforms to the specification:

[0092] It has been verified that, except Except for the missing event, the temporal relationships of all other events satisfy the constraints.

[0093] (2) Calculate the work process integrity score : % (3) Execute the exception judgment rules: Rule A (Missing Step): Exists Throughout the entire work cycle If the condition is always true, it is determined to be an anomaly due to missing steps.

[0094] Rules B and C do not apply in this example.

[0095] Step 6: Alarm Event Generation and Push Based on the above judgment results, the system generates two alarm events: Alarm 1 (Missing process step): Event ID: WD_441226_XYQ_20241001_001 Alarm Category: 2002 (Missing valid connection electrostatic clip stage) Alarm area: 2 (oil unloading area) Alarm time: 2024-10-01 10:05:30 Confidence level:

[0096] Alarm 2 (Abnormal oil unloading process): Event ID: WD_441226_XYQ_20241001_002 Alarm Category: 2010 (Verification of Oil Unloading Operation Procedure Standards) Alarm area: 2 (oil unloading area) Alarm time: 2024-10-01 10:28:00 Missing step: [" "] Integrity score: 88.9% Confidence level:

[0097] Reference Figure 3 The multi-level alarm push mechanism shown indicates that alarm 1 has a severity level of "critical." The system immediately triggers the on-site large-screen broadcast all-in-one machine to broadcast a voice announcement: "Warning! The static grounding clamp is not connected during oil unloading!" At the same time, the alarm information is pushed to the central platform via the RabbitMQ message queue, and the gas station icon on the central platform's GIS map immediately turns red and flashes.

[0098] Alarm 2 has a severity level of "moderate". The system records it to the local alarm list and pushes it to the central platform for subsequent statistical analysis, but does not trigger an instant voice broadcast.

[0099] Step 7: Evidence Association Storage The system generates associated evidence information for each alarm: Image evidence: Keyframes were extracted from the video stream at the time the alarm occurred, converted to Base64 encoding format, and stored as JPG files. The URL is: / data / evidence / 20241001 / 441226_XYQ_20241001_001.jpg Video evidence: Extract 10-second video clips before and after the alarm, save them as MP4 files, and the URL is:

[0100] / data / evidence / 20241001 / 441226_XYQ_20241001_001.mp4 Process context: Associated unloading process ID (UL_20241001_001), missing steps list ([" Completeness score (88.9%)

[0101] Step 8: Handling Feedback and Closing the Loop Upon receiving the voice prompt for Alarm 1, on-site personnel immediately corrected the violation and properly connected the electrostatic grounding clamp. After the operation was completed, on-site personnel logged into the system via a tablet computer, located the alarm record, and clicked the "rectified" button to confirm the handling. The handling feedback information is stored in association with the original alarm event, forming a complete management loop. Central management personnel can view the alarm handling rate of each gas station through statistical reports to evaluate the effectiveness of safety management.

[0102] Example 2: Rapid Response to Real-Time Violations in Gas Station Areas This embodiment describes in detail how the method and system of the present invention can quickly detect and respond to momentary violations (such as using a mobile phone or smoking) in the refueling area.

[0103] Step 1: Real-time monitoring and behavior recognition Reference Figure 4 The timing diagram shown illustrates how the AI ​​analytics system performs continuous, high-frequency inference analysis on the real-time video stream captured by the refueling area camera (Camera_ID: CAM_JYQ_01). The system runs a lightweight convolutional neural network model optimized for behaviors such as "using a mobile phone," "smoking," and "fireworks."

[0104] The logic for detecting phone use is as follows: The model first detects the person as a target, then checks if there is a phone target in the person's hand area, and simultaneously detects the relative position of the hand and head. If a phone target is detected in the hand area, and the hand is close to the head (indicating a call being answered), then it is determined to be a phone use behavior.

[0105] Let the coordinates of the center point of the hand be... The coordinates of the center point of the head are Distance between hands and head for:

[0106] if (Threshold set at 30 pixels), and the phone's detection confidence level. Then, the behavior score for using a mobile phone Output high scores.

[0107] Step 2: Determining the Violation and Calculating Confidence At a certain moment, the system detects a customer placing their phone to their ear next to the gas pump, and the model outputs: Mobile phone detection confidence level: 0.85 Distance between hand and head: 15 pixels Mobile phone usage behavior score because , The system determined this to be a violation of the rule of using a mobile phone.

[0108] To reduce false alarms, the system introduces a time smoothing mechanism: an alarm event is triggered only if three consecutive frames (approximately 0.12 seconds) are determined to be violations. In this embodiment, the system generates an alarm event only if five consecutive frames meet the condition.

[0109] Step 3: Real-time Alarms and On-site Intervention Reference Figure 3 The alarm push mechanism shown defines the severity level of making phone calls as "serious." The system immediately triggers the on-site large screen broadcasting all-in-one machine to issue a voice announcement: "Warning! Making phone calls is prohibited in the refueling area. Please hang up immediately!" At the same time, the system generates an alarm event: Event ID: WD_441226_JYQ_20241001_058 Alarm Category: 1002 (Calling a mobile phone) Alarm area: 1 (refueling area) Alarm time: 2024-10-01 14:23:45 Confidence level: 0.88 Evidence images:

[0110] / data / evidence / 20241001 / 441226_JYQ_20241001_058.jpg Evidence video:

[0111] / data / evidence / 20241001 / 441226_JYQ_20241001_058.mp4 Step 4: Alarm Reporting and Recording The system pushes alarm events to the central platform via RabbitMQ and simultaneously stores them in the local alarm list. The central platform's monitoring dashboard displays the alarm information in real time, and the gas station icon is highlighted in red on the GIS map.

[0112] Upon hearing the voice announcement, on-site staff immediately approached the customer and advised them to stop using their mobile phone, successfully averting a potential safety risk. The entire process, from the occurrence of the behavior to on-site intervention, took no more than 3 seconds, achieving a real-time control effect of "detection and prevention."

[0113] Example 3: Electronic Fence and Intrusion Detection in Tank Area This embodiment describes in detail how the method and system of the present invention achieve intelligent protection of key areas in oil tank farms.

[0114] Step 1: Electronic Fence Configuration During system initialization, the administrator manually delineates the virtual electronic fence ROI area on the tank area monitoring screen, precisely covering the prohibited area within a 5-meter radius around the tank. The ROI area is represented by a sequence of polygon vertex coordinates:

[0115] in The number of vertices of the polygon is shown in this embodiment. .

[0116] Step 2: Personnel Detection and Intrusion Determination The AI ​​analytics appliance continuously runs a personnel detection model to detect and track all personnel entering the tank area's monitoring range. For each detected personnel, the system calculates its center point coordinates. Then determine whether the point is located within the ROI region.

[0117] The determination of whether a point is inside a polygon uses the ray casting method: starting from the point... Cast a ray horizontally to the right and count the number of intersections between the ray and each side of the polygon. If the number of intersections is odd, the point is inside the polygon; if the number is even, the point is outside the polygon.

[0118] When the system detects that a person's center point is located within the ROI area, an intrusion event is triggered.

[0119] Step 3: Work uniform detection and identity recognition The system further analyzes the target area of ​​the intruder, using a workwear recognition model to determine whether they are wearing distinctive workwear. This model combines color and texture features, extracting the HSV color histogram and LBP texture features of the target area, which are then input into an SVM classifier for classification.

[0120] If the classifier outputs the confidence score for "wearing work clothes" If yes, they are considered staff members; otherwise, they are considered non-staff members.

[0121] Step 4: Differentiation Processing Scenario 1 (Staff Inspection): If an intruder is wearing work clothes, the system will not trigger an alarm, but will record an "inspection event" for subsequent statistical analysis. Event information includes: time, personnel ID, duration of stay, etc.

[0122] Scenario 2 (Unauthorized Personnel Intrusion): If the intruder is not wearing a uniform, the system immediately generates an "Unauthorized Personnel Intrusion" alarm (Event ID: 3001). Event ID: WD_441226_YGQ_20241001_012 Alarm Category: 3001 (Unauthorized Entry) Alarm area: 3 (oil tank area) Alarm time: 2024-10-01 09:15:30 Confidence level: 0.95 Evidence images: / data / evidence / 20241001 / 441226_YGQ_20241001_012.jpg Evidence video: / data / evidence / 20241001 / 441226_YGQ_20241001_012.mp4 The system triggered a voice announcement on-site: "Warning! Personnel have intruded into the oil tank area. Please leave immediately!" Simultaneously, the alarm information was reported to the central platform. Upon seeing the alarm, the central monitoring operator can review the situation via video and notify on-site security personnel to handle the situation.

[0123] Example 4: Alarm Handling Closed Loop and Model Self-Optimization This embodiment describes in detail how the method and system of the present invention achieve a closed loop in alarm handling, and how the AI ​​model is optimized using handling feedback data.

[0124] Step 1: Alarm Handling Feedback Upon receiving the alarm, on-site personnel will handle and provide feedback through the following methods: Method 1 (On-site large screen touch control): On the on-site large screen all-in-one machine, each alarm record has a "Handle" button next to it. Clicking it will bring up a handling confirmation box, providing three options: "Rectified", "False Alarm", and "Not Handled for Now", and you can fill in the remarks.

[0125] Method 2 (Mobile App / WeChat): For alarms pushed to mobile devices, administrators can directly view the alarm details on their mobile phones and click the "Confirm Handling" button to provide feedback.

[0126] Method 3 (Central Platform): When handling alarms, the central monitoring staff can perform batch processing or single processing through the alarm management interface of the central platform.

[0127] Step 2: Handling Data Association Storage Feedback information is associated with and stored alongside the original alarm event, forming a complete data loop. An example of a database record is shown below: Step 3: Sample Library Construction and Model Optimization The system periodically (e.g., weekly) extracts alarm events marked as "false alarms" from the database as negative samples and normal work processes that do not trigger alarms as positive samples to build an incremental training dataset.

[0128] For false alarm samples, the system extracts the video frame sequence before and after the alarm occurs, and manually labels the correct tags (e.g., a frame that should have been detected as "not using a mobile phone" was falsely reported as "using a mobile phone"), forming a set of difficult case samples.

[0129] The system uses these new samples to incrementally fine-tune the AI ​​model, updating the model's weight parameters and optimizing its performance in similar scenarios. For example, by adding training to differentiate between "holding a cigarette" and "holding a mobile phone," false positives caused by holding similar objects are reduced.

[0130] Step 4: Model Update and Deployment The optimized model is distributed to the AI ​​analysis all-in-one machines at each gas station through the model management module of the central platform. After receiving the new model, the AI ​​analysis all-in-one machine updates the model using a hot-loading method, without requiring a system service restart, thus ensuring the continuity of monitoring tasks.

[0131] Through this continuous "collection-identification-feedback-optimization" closed-loop mechanism, the system's identification accuracy is constantly improving, the false alarm rate is continuously decreasing, and ultimately, efficient and accurate control over the safety risks of gas stations is achieved.

[0132] In summary, compared with the prior art, the present invention has the following significant advantages: (i) It has realized a proactive security defense model that has shifted from "post-event verification" to "pre-event warning and in-event intervention". Addressing the shortcomings of traditional video surveillance systems mentioned in the background section, which can only perform post-event verification and cannot provide real-time early warnings, this invention deploys an AI analysis all-in-one machine locally at gas stations to perform uninterrupted real-time intelligent analysis of multiple video streams 24 / 7. When a customer makes a phone call in the refueling area, the system can complete the entire process of identification, judgment, and voice announcement within 3 seconds of the action occurring, achieving a real-time control effect of "detection and prevention." This proactive defense mode shortens the time for discovering safety hazards from "hours or even days after the event" to "seconds during the event," greatly improving the timeliness and effectiveness of safety management.

[0133] (ii) It has achieved deep perception and intelligent judgment of complex work processes. Addressing the shortcomings of existing technologies mentioned in the background section, such as their limited data dimensions and inability to perceive complex scenarios, this invention innovatively constructs a multimodal data fusion analysis model and a process state machine. The system can not only identify individual violations but also understand the entire oil unloading operation process comprised of multiple sub-events. Through mathematical models such as time-series constraint verification, integrity score calculation, and anomaly rule determination, it performs in-depth analysis and intelligent evaluation of the process's compliance. When a step is missing, the system can accurately locate the missing link and provide an integrity score. This ability to perceive complex processes is completely unattainable by traditional sensor alarm systems.

[0134] (iii) Automatic association and closed-loop management of alarm information and business processes have been achieved. To address the technical shortcomings mentioned in the background section, such as severe information silos and a lack of process-oriented closed-loop management, this invention establishes a comprehensive alarm event association storage and handling feedback mechanism. The system generates a globally unique event ID for each alarm and associates it with evidence images, evidence videos, and process context information, achieving automatic association between alarm information and business processes. Simultaneously, the system provides multiple handling feedback channels, receiving handling confirmation information from on-site personnel and associating the handling results with the original alarm, forming a complete management closed loop from "perception-identification-alarm-handling-backtracking".

[0135] (iv) It has achieved efficient collaboration between real-time edge processing and cloud-based collaborative management. To address the technical shortcomings mentioned in the background technology, such as insufficient multi-source data processing capabilities and high latency caused by centralized processing, this invention adopts a three-tiered collaborative architecture of "cloud-edge-device". The core AI analysis tasks are deployed on an AI analysis all-in-one machine locally at the gas station. Video streams can be analyzed in real time without being uploaded to a central location, ensuring the real-time nature of alarms. Simultaneously, the raw video stream is stored locally, with only structured alarm data and device status information uploaded, significantly reducing network bandwidth pressure and ensuring the security of video data.

[0136] (v) Data-driven continuous learning and model self-optimization have been achieved. To address the technical shortcomings mentioned in the background art, such as model rigidity and inability to adapt to complex and changing scenarios, this invention establishes a model self-optimization mechanism based on feedback. The system uses alarm events marked as "false alarms" as negative samples and normal operating procedures that do not trigger alarms as positive samples to construct an incremental training dataset, regularly optimizing and updating the AI ​​model. Through this continuous "collection-identification-feedback-optimization" closed-loop mechanism, the system's identification accuracy is continuously improved, and the accuracy of key detection items is significantly enhanced.

[0137] (vi) Achieving an effective breakthrough in the challenge of detecting small targets. Addressing the technical shortcomings mentioned in the background technology regarding the small size and low accuracy of small targets such as electrostatic grounding clamps and refueling nozzles under existing monitoring conditions, this invention achieves a significant breakthrough through collaborative innovation using multiple methods. For example, the system introduces logical evidence weights in the alarm confidence calculation, and when the visual detection confidence level... When the target is too small to be considered high enough, logical evidence is provided through the logical constraints of the process state machine. This comprehensively improves the confidence level of the final alarm and significantly enhances the reliability of detection.

[0138] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for intelligent monitoring of gas station security based on multi-modal data analysis, characterized in that, Includes the following steps: Collect video stream data and IoT sensor status data from gas stations; Several AI analysis tasks are run in parallel on the video stream data to extract visual feature information; The visual feature information is fused with the IoT sensor status data in a multimodal manner to construct a real-time status vector of the monitoring area; Based on a predefined business rule base and process state machine model, the real-time state vector is subjected to time-series logic analysis and process compliance determination. An alarm event is generated when violations or security risks are detected.

2. The method according to claim 1, characterized in that, Several AI analysis tasks are run in parallel on the video stream data, including: The video stream data is decoded and frames are extracted. The extracted video frames are then input into several parallel deep learning models. The deep learning models include a people detection and tracking model, a vehicle detection and tracking model, a small target detection model, a people behavior recognition model, and a scene semantic understanding model.

3. The method according to claim 1, characterized in that, The process of performing time-series logic analysis and process compliance determination on the real-time state vector based on a predefined business rule base and process state machine model includes: Define the set of ideal states : in, Indicates the first step in the workflow. Individual events, The total number of sub-events; For each child event Define detection function : in, for Multimodal data input at any given time. For video frame data, For IoT sensor data; For sub-events The scoring function for training; For the child incident The threshold for judgment.

4. The method according to claim 1, characterized in that, The process compliance determination also includes the steps of verifying timing constraints, calculating the work process integrity score, and executing anomaly determination rules, wherein: Verify timing constraints: in, Indicates sub-event The start time, Indicates sub-event The end time; when the timing constraint is not met, it is judged as a process timing disorder abnormality; Calculate the job process integrity score: in, This represents the total number of steps in the ideal process. For the indicator function, when a child event occurs Its value is 1 if it is successfully detected during the entire operation, and 0 otherwise; Score the process completeness; Execution exception judgment rules: Rule A: Child events exist. Throughout the entire work cycle Inside, its detection function Hengchengli: Rule B: Observing a sub-event Time of occurrence Earlier than the incident End time ,and : Rule C: For sub-events with duration requirements , Its duration Minimum duration requirement not met: When any one of the conditions in rule A, rule B, or rule C is met, the workflow is determined to be abnormal and an alarm is triggered.

5. The method according to claim 1, characterized in that, The generation of alarm events includes alarm confidence calculation: in, For the final alarm confidence level; Confidence level based on visual detection; Confidence level based on logical violations; These are weighting coefficients, and their values ​​range from [value range missing]. .

6. The method according to claim 5, characterized in that, The method also includes the step of associating and storing alarm events with corresponding evidence information, which includes: Generate a globally unique event ID, which includes a gas station identifier, a monitoring area identifier, an occurrence timestamp, and a sequence number; The core information of the alarm event is associated with the evidence information, which includes the on-site image encoding data at the time of the alarm, the image storage address, and the video clip storage address. For alarms related to work processes, further associate them with work process context information, including work process ID, list of missing steps, and integrity score.

7. The method according to claim 6, characterized in that, It also includes the step of pushing alarm events to on-site broadcast terminals, the superior central platform, and / or mobile devices in real time, which includes: Set differentiated push strategies based on the severity level of the alarm event. For alarm events with a severity level of "severe", the on-site broadcast terminal will be immediately triggered to broadcast the alarm via voice. At the same time, the alarm will be pushed to the upper-level central platform through the message queue middleware and to the designated management personnel's mobile terminal via WeChat official account or SMS. For alarm events with a severity level of "general", they are recorded in the local alarm list and pushed to the upper-level central platform, without triggering real-time voice broadcast.

8. The method according to claim 1, characterized in that, The process involves using edge computing nodes to run several AI analysis tasks in parallel on the video stream data to extract visual feature information, including the step of self-monitoring the state of the edge computing nodes. The edge computing node sends a heartbeat message to the upper-level central platform every first preset time interval; The edge computing node reports its operating status information to the upper-level central platform every second preset time interval.

9. The method according to claim 1, characterized in that, The method also includes a step of closed-loop processing for alarm events: Receive feedback on the handling of alarm events, including two types of feedback: rectified and false alarm; The feedback information is associated with and stored with the original alarm event; Alarm events marked as false alarms are treated as negative samples, while normal operating procedures that do not trigger alarms are treated as positive samples.

10. A gas station safety intelligent monitoring system based on multi-modal data analysis, characterized in that, The system performs the method as described in any one of claims 1-9, and the system includes an edge-side perception layer, an edge-side processing layer, a cloud-side management layer, and a broadcast terminal layer, wherein: The edge sensing layer includes several camera devices and IoT sensors, which are used to collect video stream data and IoT sensor status data, respectively. The edge processing layer includes a video stream access module, a multi-task AI analysis engine, a multi-modal data fusion module, a process state machine determination module, an alarm event management module, and a local storage module; The cloud-side management layer is deployed on the central business server, which includes a message queue module, a GIS map display module, a multi-dimensional data statistics module, and a system management module. The broadcast terminal layer is deployed on large screens and voice broadcast all-in-one machines to display alarm information and broadcast it via voice in real time.

Images