Multi-mode sensing gas station environment safety early warning method and system

Abstract

The invention provides a multi-mode perception gas station environment safety early warning method and system, which are applied to the technical field of safety early warning, and are used for obtaining video and audio data, extracting vehicle and environment sound information, carrying out preliminary judgment, and carrying out event confirmation, priority ranking and linkage instruction generation through a rule inference engine. Therefore, deep fusion and comprehensive analysis of the multi-modal information are achieved, the accuracy and robustness of event judgment are improved, concurrent events can be effectively processed, refined linkage control is achieved, the accuracy and robustness of event judgment are improved through multi-modal information fusion, priority ranking and linkage can be carried out according to the events, and the efficiency of event judgment is improved. The method has the advantages that the problems that single-modal information is limited, simple fusion is insufficient, and concurrent event processing is not fine are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of safety warning, in particular to a multi-modal perception gas station environment safety warning method and system. BACKGROUND

[0002] As an important energy supply facility, the safety management of the gas station area is crucial. The gas station area is usually equipped with sensing devices such as visible light cameras to monitor the activities of vehicles and personnel. The existing gas station safety monitoring system can receive real-time data streams from sensing devices and perform preliminary processing. However, in a complex gas station environment, single modal sensing information often has limitations, and relying solely on video analysis may not be able to accurately determine the nature and risk level of some sudden events.

[0003] Some existing systems attempt to simply fuse video and audio information to make judgments, for example, when video detects abnormal behavior of a vehicle, if a screeching sound or tire friction sound is detected in the same period and in the same area, the confidence of the abnormal behavior is increased; when video detects vehicle contact or position mutation, combined with the collision sound characteristics, it is determined that a vehicle collision event has occurred. However, this simple fusion method cannot achieve deep correlation and comprehensive analysis of multi-modal information, thereby affecting the accuracy and robustness of event determination.

[0004] Further, when the system determines that a vehicle-related safety event has occurred, it usually generates a warning message and triggers local response devices for linkage. However, the gas station area is usually large and vehicle activity is frequent, and there may be complex situations where multiple vehicles simultaneously exhibit abnormal behavior or collide in different areas. The simple fusion judgment method is prone to linkage conflicts or improper resource allocation when handling these concurrent events.

[0005] In view of the above problems, the prior art needs to be improved. SUMMARY

[0006] In view of the above deficiencies of the prior art, the present application provides a multi-modal perception gas station environment safety warning method and system, which has improved accuracy and robustness of event determination through multi-modal information fusion, and can prioritize and link according to events, overcoming the limitations of single modal information, the shortcomings of simple fusion, and the difficulties of handling concurrent events.

[0007] In a first aspect, a multi-modal perception gas station environment safety warning method is provided, the method comprising the steps of: S1: obtaining video data and audio data of each divided area of the gas station; S2: extracting vehicle target information, vehicle behavior information, and environmental sound information according to the video data and the audio data; S3: performing preliminary event determination according to the vehicle target information, the vehicle behavior information and the environmental sound information, to obtain a preliminary event determination result; S4: inputting the preliminary event determination result into a rule reasoning engine, the rule reasoning engine performing event confirmation on the preliminary event determination result according to a preset rule base, performing priority ranking when there are multiple events, and generating a linkage instruction matched with the event type and the occurrence position according to the event type and the occurrence position; S5: sending the linkage instruction to a corresponding local response device for execution.

[0008] The multi-modal perception gas station environment safety warning method provided in the application extracts vehicle and environmental sound information through obtaining video and audio data, performs preliminary determination, and performs event confirmation, priority ranking and linkage instruction generation through a rule reasoning engine, thereby realizing deep fusion and comprehensive analysis of multi-modal information, improving the accuracy and robustness of event determination, and effectively processing concurrent events to realize fine linkage control.

[0009] Further, step S2 includes: S21: identifying a vehicle target in the gas station area according to the video data, tracking the motion trajectory of the vehicle target, and analyzing the behavior pattern of the vehicle target; S22: identifying a preset environmental sound event according to the audio data, and determining the approximate occurrence area of the environmental sound event; S23: associating the vehicle target with the environmental sound event according to the motion trajectory, the behavior pattern and the approximate occurrence area of the environmental sound event, and determining the vehicle target information, the vehicle behavior information and the environmental sound information based on the association result.

[0010] The multi-modal perception gas station environment safety warning method provided in the application separates video data processing and audio data processing, then integrates the two types of information to realize effective extraction and association of information from multi-modal data, providing more accurate and richer basic data for subsequent preliminary event determination.

[0011] Further, step S21 includes: S211: determining the position of the vehicle target in the gas station area division according to the video data; S212: continuously tracking the vehicle target to generate a motion trajectory of the vehicle target; S213: analyzing a behavior mode of the vehicle target according to the motion trajectory of the vehicle target and a position of the vehicle target in the gas station division area.

[0012] The multi-modal perception gas station environment safety warning method provided in the application refines the process of extracting vehicle information from video data into three specific sub-steps, providing a more structured and reliable method to obtain key vehicle data for subsequent safety warning, aiming to solve the challenge of accurately extracting vehicle position, trajectory and behavior mode in a complex gas station environment.

[0013] Further, step S23 comprises: S231: obtaining a vehicle target list to be associated according to the motion trajectory and the behavior mode; S232: filtering a candidate vehicle target set located in the approximate occurrence area of the environmental sound event from the vehicle target list; S233: calculating the correlation degree of the vehicle target and the environmental sound event for each vehicle target in the candidate vehicle target set to obtain the correlation result; S234: determining the vehicle target information, the vehicle behavior information and the environmental sound information according to the correlation result.

[0014] The multi-modal perception gas station environment safety warning method provided in the application refines the step of calculating the correlation result, associates the environmental sound event with the vehicle target, thereby improving the accuracy of extracting information, and further improving the accuracy of subsequent event determination and warning.

[0015] Further, step S233 comprises: S2331: calculating the spatial correlation relationship between the vehicle target and the environmental sound event for each vehicle target in the candidate vehicle target set to obtain a spatial correlation score; S2332: calculating the temporal correlation relationship between the vehicle target and the environmental sound event to obtain a temporal correlation score; S2334: integrating the spatial correlation score and the temporal correlation score to calculate the correlation degree to obtain the correlation result.

[0016] Further, step S3 comprises: S321: determining rules for each preliminary event type according to a plurality of preset preliminary event types; S322: For each preliminary event type, evaluate whether the corresponding decision rule is satisfied according to the vehicle target information, the vehicle behavior information, and the environmental sound information; S323: When any of the decision rules is satisfied, generate the preliminary event decision result.

[0017] Further, in step S321, the decision rules include at least one decision condition based on the position of the vehicle target in the refueling station divided area, the behavior pattern of the vehicle target, or the type and occurrence area information of the environmental sound event.

[0018] Further, step S4 includes: S41: According to the preset event confirmation rule, confirm each preliminary event in the preliminary event decision result to obtain a confirmed event list; S42: When there are multiple events in the confirmed event list, prioritize the multiple events according to the preset priority rule and the type and occurrence position of the confirmed event to obtain a priority sorting result; S43: According to the priority sorting result and the preset linkage strategy rule, generate the linkage instruction for the confirmed event.

[0019] Further, step S43 includes: S44: Detect conflicts in the linkage instruction and identify the conflict types in the linkage instruction; S45: According to the conflict types, resolve the conflicts according to the preset conflict resolution rule and the priority sorting result associated with the confirmed event to obtain the final linkage instruction.

[0020] In a second aspect, a multi-modal perception refueling station environment safety warning system is applied in the steps of any of the above-mentioned methods, and the system includes: An acquisition module: acquires video data and audio data of each divided area of the refueling station; An extraction module: extracts vehicle target information, vehicle behavior information, and environmental sound information according to the video data and the audio data; A decision module: performs preliminary event decision according to the vehicle target information, the vehicle behavior information, and the environmental sound information to obtain a preliminary event decision result; An instruction generation module: inputs the preliminary event decision result into a rule reasoning engine, which performs event confirmation on the preliminary event decision result according to a preset rule base, prioritizes multiple events when they exist, and generates a linkage instruction matching the type and occurrence position of the event according to the type and occurrence position of the event. instruction sending module: send the linkage instruction to the corresponding local response device for execution.

[0021] Beneficial effects: the multi-modal perception gas station environment safety warning method and system proposed in the present application, by acquiring video and audio data, extracting vehicle and environmental sound information, making preliminary determination, and through the rule reasoning engine, event confirmation, priority sorting and linkage instruction generation, realizes the deep fusion and comprehensive analysis of multi-modal information, improves the accuracy and robustness of event determination, and can effectively handle concurrent events, realizes fine linkage control, has the advantages of multi-modal information fusion, improves the accuracy and robustness of event determination, and can sort the priority according to the event and linkage, solves the problems of single modal information limitation, simple fusion deficiency and non-fine concurrent event processing. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The flowchart of the multi-modal perception gas station environment safety warning method proposed in the present application.

[0023] Figure 2 The structural diagram of the multi-modal perception gas station environment safety warning system proposed in the present application.

[0024] Figure 3 The architecture diagram of the multi-modal perception gas station environment safety warning system proposed in the present application.

[0025] Label explanation: 201, acquisition module; 202, extraction module; 203, determination module; 204, instruction generation module; 205, instruction sending module. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and indicated in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0027] It should be noted that similar reference numerals and letters refer to like items in the accompanying drawings, and once an item is defined in one drawing, it need not be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0028] In a first aspect, a multi-modal perception-based gas station environment safety early warning method is provided. The method comprises the following steps: S1: obtaining video data and audio data of each divided area of the gas station; S2: extracting vehicle target information, vehicle behavior information, and environmental sound information according to the video data and the audio data; S3: performing preliminary event determination according to the vehicle target information, the vehicle behavior information, and the environmental sound information to obtain a preliminary event determination result; S4: inputting the preliminary event determination result into a rule reasoning engine, and performing event confirmation on the preliminary event determination result according to a preset rule base, performing priority ranking when there are multiple events, and generating a linkage instruction matched with the event type and the occurrence location according to the event type and the occurrence location; S5: sending the linkage instruction to a corresponding local response device for execution.

[0029] The video data refers to a data stream reflecting visual information in the gas station area collected by a camera device. It can be implemented by using a visible light camera, an infrared camera, or a panoramic camera. It is mainly used to obtain image information of vehicles and the environment.

[0030] The audio data refers to a data stream reflecting sound information in the gas station area collected by a sound collecting device. It can be implemented by using a microphone array or a distributed audio sensor. It is mainly used to obtain environmental sound event information.

[0031] The vehicle target information refers to identification information about vehicles in the gas station area obtained by processing the video data, such as vehicle type, vehicle identification, or vehicle position. It can be implemented by using a target detection and identification algorithm. It is mainly used to determine the object of monitoring and analysis.

[0032] The vehicle behavior information refers to a vehicle activity pattern obtained by analyzing the motion trajectory and state of the vehicle target, such as driving speed, stay duration, or motion direction. It can be implemented by using a target tracking and behavior analysis algorithm. It is mainly used to judge whether the vehicle has abnormal activities.

[0033] The environmental sound information refers to the identification information about the sound events in the gas station area obtained by processing the audio data, such as sound type, sound intensity or sound occurrence area, which can be realized by sound event detection and sound source positioning technology, and is mainly used to assist in judging the scene or identifying specific risks.

[0034] The preliminary event determination refers to the preliminary identification of potential safety events based on the extracted multi-modal information, which can be realized by rule-based matching or machine learning classification model, and is mainly used to quickly screen out possible abnormal situations.

[0035] The rule reasoning engine refers to a computing unit that makes logical judgments and decisions based on preset rules, which can be realized by expert system or logical reasoning framework, and is mainly used for in-depth processing and decision-making of the preliminary determination results.

[0036] The preset rule base refers to a set of logical rules stored in the system for guiding the rule reasoning engine to confirm events, prioritize and generate linkage instructions, which can be stored in the form of fact base and rule set, and is mainly used to provide decision-making basis for the rule reasoning engine.

[0037] Event confirmation refers to the process of further verifying the potential events determined by the preliminary determination to determine whether they are real safety events, which can be realized by combining more evidence or higher confidence threshold, and is mainly used to reduce false positives.

[0038] As a specific implementation, the scheme of the present application can be implemented as follows: High-definition network cameras and high-sensitivity audio sensors are deployed at key locations in the gas station area, such as fueling islands, access channels and parking areas.

[0039] The video stream captured by the camera and the audio stream captured by the audio sensor are transmitted in real time to the local edge computing device or server. The device runs video analysis algorithms to identify vehicle targets in the picture, continuously tracks their motion trajectories, and analyzes their behavior patterns in different areas, such as whether they stay in designated areas or their driving speed is abnormal. At the same time, the device runs audio analysis algorithms to identify preset dangerous sound events, such as screeching, collision or abnormal shouting, and estimates the approximate source area of the sound.

[0040] Then, the system correlates and matches the real-time position of the vehicle, the behavior pattern with the occurrence time and position of the sound event, such as whether the screeching sound is associated with a fast-moving vehicle in the nearby area. Based on these associated information, the system makes a preliminary event determination according to the preset rules, for example, if a vehicle moves quickly near the fueling island and screeching sound is detected at the same time, a dangerous driving event is preliminarily determined.

[0041] The preliminary determination result is sent to a rule inference engine. The engine has a built-in rule library for secondary confirmation of the preliminary event to exclude false positives. If there are multiple confirmed events at the same time, such as a minor scratch and an abnormal stay in the oil area in different areas, the engine will sort these events according to the priority defined in the rule library, for example, the oil area event has higher priority than the parking area event.

[0042] Finally, the engine consults the linkage strategy rules according to the type and location of the confirmed event to generate accurate linkage instructions, for example, if a collision occurs in the oil area, generate instructions to stop the operation of the dispenser in the area and trigger the area voice warning. These instructions are sent to the on-site execution equipment through the network, such as the area broadcast system, the dispenser controller or the area lighting system, to execute the corresponding safety response action.

[0043] Through the above scheme, the application can effectively fuse multi-modal perception information such as video and audio, and improve the accuracy of the determination of the safety event of the gas station environment. In particular, the rule inference engine confirms and prioritizes the preliminary event, solves the problem of insufficient handling of concurrent events, and ensures that high-risk events can be responded to in priority.

[0044] In addition, according to the type and location of the event, the matching linkage instruction is generated to realize the fine and regional control of the safety response, avoid unnecessary global intervention, improve the efficiency and pertinence of the safety intervention, and thus comprehensively improve the safety management level of the gas station.

[0045] Further, step S2 comprises: S21: identifying a vehicle target in the gas station area according to the video data, and tracking the motion trajectory of the vehicle target to analyze the behavior pattern of the vehicle target; S22: identifying a preset environmental sound event according to the audio data, and determining the approximate occurrence area of the environmental sound event; S23: associating the vehicle target with the environmental sound event according to the motion trajectory, the behavior pattern and the approximate occurrence area of the environmental sound event, and determining the vehicle target information, the vehicle behavior information and the environmental sound information based on the association result.

[0046] The method preliminarily processes the video data and the audio data respectively, obtains the vehicle subject and its dynamic behavior information from the video, and obtains the specific sound event in the environment and its approximate location information from the audio. Then, the vehicle motion trajectory and the behavior pattern obtained by video analysis are associated with the environmental sound event and its occurrence area obtained by audio analysis.

[0047] This association process is not simply a data superposition, but through analyzing the correspondence of vehicle behavior and sound events in time and space, to determine whether there is a causal or accompanying relationship between them. For example, if a sudden braking sound is detected in a certain area, and at the same time there is a vehicle in the area that is performing a sudden braking behavior (through video analysis), then the sudden braking sound can be associated with this vehicle. Through this cross-modal association, the sound event can be effectively "attributed" to a specific vehicle or vehicle behavior, overcoming the limitations of single-modal information that cannot determine the source of the event.

[0048] Based on this association result, the system can determine a comprehensive information set containing the vehicle target, its behavior, and the environmental sound events associated with it. This information set is more rich and accurate than the video information or audio information alone, and can provide more reliable and comprehensive input for subsequent preliminary event determination, thereby improving the accuracy and reliability of the entire safety warning system. This processing flow of separate extraction and then cross-modal integration can more effectively utilize the complementarity of multi-modal data, and solve the problem of incomplete or inaccurate information when processed independently.

[0049] Among them, identifying the vehicle target in the gas station area refers to detecting and distinguishing the vehicle object in the video image through image processing technology, which can be realized by a target detection model based on deep learning. There are many target detection models for distinguishing vehicle objects from video in the prior art, for example, a vehicle detection model based on convolutional neural network frames the vehicle in the video and uses a sliding window for target tracking. This extraction method is not the core of the present application, and will not be described here. Similarly, analyzing the behavior pattern of the vehicle target refers to judging whether the vehicle exhibits a specific driving or parking behavior according to the vehicle's motion trajectory, speed, acceleration, and residence time in a specific area, which can be realized by a rule-based or model-based behavior analysis method.

[0050] Identifying the preset environmental sound event refers to detecting a specific sound type that meets the preset characteristics from the environmental sound through audio signal processing technology, such as sudden braking sound, collision sound, and whistle sound, which can be realized by an audio event detection model.

[0051] Determining the approximate occurrence area of the environmental sound event refers to estimating the spatial location range of the sound event occurrence according to the collected audio signal, which can be realized by a sound source positioning technology.

[0052] Associating the vehicle target with the environmental sound event refers to establishing a correspondence between the vehicle identified in the video and the sound event identified in the audio, and determining whether a certain sound event is related to a specific behavior of a certain vehicle.

[0053] Further, step S21 includes: S211: determining the position of the vehicle target in the gas station area division according to the video data; S212: continuously tracking the vehicle target to generate a motion trajectory of the vehicle target; S213: analyzing the behavior pattern of the vehicle target according to the motion trajectory of the vehicle target and the position of the vehicle target in the gas station division area.

[0054] In the above steps, determining the position of the vehicle target in the gas station area division, continuously tracking the vehicle target and generating a motion trajectory, and analyzing the behavior pattern according to the motion trajectory and the position are the key technical features of the present scheme.

[0055] Specifically, determining the position of the vehicle target in the gas station area division means associating the vehicle target detected in the video with the pre-set functional areas of the gas station, such as associating the vehicle target with the refueling area, parking area, passageway, etc., so as to obtain the spatial context information of the vehicle. This can be achieved by recognizing the vehicle through target detection technology and combining spatial data of area division for judgment.

[0056] Continuously tracking the vehicle target and generating a motion trajectory means recognizing and associating the same vehicle target in consecutive video frames, recording its position information changing over time, and thus forming a data sequence reflecting the moving path, speed and direction of the vehicle. This is usually completed with the help of multi-target tracking algorithm.

[0057] Analyzing the behavior pattern of the vehicle target according to the motion trajectory of the vehicle target and the position of the vehicle target in the gas station division area means comprehensively utilizing the dynamic moving information (trajectory) of the vehicle and the static environmental information (area position) where it is located, and judging the specific action or state that the vehicle is performing or has performed through pre-set rules or models, such as whether it stays for a long time, whether it enters a no-entry area, whether it drives according to the specified route, etc.

[0058] As a specific implementation of the present scheme, the above steps can be implemented as follows: In step S211, a target detection model based on deep learning can be used to process the video frames to recognize the vehicle target in the picture and obtain its bounding box coordinates. At the same time, the polygon coordinate data of each area of the gas station is pre-stored in the system. Then, by calculating whether the center point coordinates of the vehicle bounding box fall within the polygon range of a certain area, the current area where the vehicle is located is determined.

[0059] In step S212, a multi-target tracking algorithm, such as an appearance feature and motion prediction based tracking algorithm, can be employed. The algorithm receives the vehicle target information detected in S211, and in subsequent frames, associates the detected target in the current frame with the tracked target in the previous frame through a matching algorithm, thereby maintaining the identity of each vehicle target and recording its position coordinates in each frame, thus generating the motion trajectory data of the vehicle.

[0060] In step S213, a behavior rule engine can be established. The engine receives the vehicle motion trajectory data generated in S212 and the vehicle region position information determined in S211. The rules can be defined as follows: if the length of the trajectory of a vehicle in a certain region (such as a no-parking zone) exceeds a threshold, it is determined as the "entering a no-parking zone" behavior pattern; if the speed of a vehicle in a certain region (such as a fueling position) is below a threshold for a long time and the stay time exceeds a threshold, it is determined as the "long-time occupation" behavior pattern. The system evaluates whether the pre-set rules are met based on the input trajectory and position data, thereby outputting the behavior pattern determination result of the vehicle.

[0061] Further, step S23 includes: S231: obtaining a list of vehicle targets to be associated according to the motion trajectory and the behavior pattern; S232: filtering a candidate vehicle target set located in the approximate occurrence region of the environmental sound event from the vehicle target list according to the approximate occurrence region of the environmental sound event; S233: calculating the association degree of the vehicle target and the environmental sound event for each vehicle target in the candidate vehicle target set, obtaining an association result; S234: determining the vehicle target information, the vehicle behavior information, and the environmental sound information according to the association result.

[0062] Specifically, after obtaining the video and audio data, the present scheme first identifies vehicles, tracks trajectories, and analyzes behavior patterns through video analysis, and identifies environmental sound events and determines their approximate occurrence regions through audio analysis. In order to accurately associate sound events with vehicles, the present scheme employs a step-by-step refinement association process.

[0063] First, according to the motion trajectory and the behavior pattern of the vehicle, the vehicle that may be related to the sound event is preliminarily filtered out to form a list of target vehicles to be associated. This step utilizes the dynamic characteristics of the vehicle itself, thereby narrowing the range of potential associated objects.

[0064] Then, combined with the approximate occurrence region information of the sound event, further filter out the vehicles close to the sound event in space from the to-be-associated list to form a candidate target vehicle set. This step utilizes the spatial positioning information of the sound to exclude irrelevant interference vehicles in space. Then, for each vehicle in the candidate set, calculate the association degree thereof with the sound event to quantify the correlation therebetween.

[0065] Finally, according to the calculated association degree, determine which vehicle the sound event should be associated with, and based on the association result, integrate the information extracted from the video and the audio to form more accurate target vehicle information, vehicle behavior information and environmental sound information. Through this multi-step screening and matching process from behavior trajectory to spatial region to quantitative association degree, the present scheme can effectively associate a specific environmental sound event to a specific vehicle in a complex environment with multiple vehicles and multiple sound events. This accurate association enables the subsequent preliminary event determination to be based on more reliable and complete input information, thereby improving the accuracy and reliability of the overall early warning method. Compared with the scheme of simply determining based on video or audio information or simply associating based on simple spatial co-occurrence, the present scheme significantly improves the accuracy of sound event and vehicle association by introducing preliminary screening based on behavior trajectory, spatial screening based on sound region and quantitative association degree calculation, thereby improving the accuracy of subsequent event determination.

[0066] Further, step S233 includes: S2331: For each vehicle target in the candidate vehicle target set, calculate the spatial association relationship between the vehicle target and the environmental sound to obtain a spatial association score; S2332: Calculate the temporal association relationship between the vehicle target and the environmental sound event to obtain a temporal association score; S2334: Integrate the spatial association score and the temporal association score to calculate the association degree to obtain an association result.

[0067] In one specific embodiment, when the system detects a sudden braking sound event, it can analyze the motion trajectories and behavior patterns of all vehicles currently in the gas station area, filter out vehicles whose speed change rate exceeds a preset threshold within a few seconds before or at the time of the sound occurrence, and add the identifiers of these vehicles to the target vehicle list to be associated. Assuming that the sudden braking sound event is determined by sound source positioning to be in a circular area with a center at coordinates (x, y) and a radius R, the system can iterate through each vehicle in the target vehicle list to be associated, check whether its current position is within the circular area or whether its motion trajectory in the recent period of time passes through the area, and add vehicles that meet the conditions to the candidate target vehicle set. For each vehicle in the candidate set, the spatial association score and the temporal association score with the sudden braking sound event can be calculated. The spatial association score can be calculated based on the distance between the vehicle's current position and the center of the sound area, with a shorter distance resulting in a higher score. The temporal association score can be calculated based on the time difference between the time point of the vehicle's speed change and the time point of the sound event, with a smaller time difference resulting in a higher score.

[0068] The association degree can be calculated as the weighted sum of the spatial association score and the temporal association score. After calculating the association degrees of all vehicles in the candidate set, the system can select the vehicle with the highest association degree as the vehicle most relevant to the sudden braking sound event, and integrate the vehicle's identifier, its behavior pattern, and information such as the type and occurrence area of the sound event to form a complete information unit for subsequent preliminary event determination.

[0069] Further, step S3 includes: S321: determining rules for each preliminary event type according to a plurality of preset preliminary event types; S322: for each preset preliminary event type, evaluating whether the corresponding determination rule is met according to the vehicle target information, the vehicle behavior information, and the environmental sound information; S323: generating a preliminary event determination result when any determination rule is met.

[0070] Further, in step S321, the determination rule includes at least one determination condition based on the position of the vehicle target in the divided area of the gas station, the behavior pattern of the vehicle target, or the type and occurrence area information of the environmental sound event.

[0071] The plurality of preset preliminary event types refer to predefined classifications of potential safety events that may occur in the gas station environment and need to be focused on and handled by the system, which can include but are not limited to specific event categories such as vehicle reverse, overspeed, abnormal parking, area intrusion, sudden braking, and collision.

[0072] The determination rule refers to a set of determination criteria or conditions set for identifying each preliminary event type, which can be implemented in the form of logical expressions, threshold settings, pattern matching, etc.

[0073] For example, in a specific embodiment, "vehicle reverse", "vehicle overspeed", "abnormal parking", and "sudden braking accompanied by abnormal behavior" can be preset as preliminary event types.

[0074] For the "vehicle reverse" type, the determination rule can be set as: the motion trajectory of the vehicle is opposite to the preset normal driving direction of the region.

[0075] For the "vehicle overspeed" type, the determination rule can be set as: the average speed or instantaneous speed of the vehicle in a specific region exceeds the preset threshold.

[0076] For the "abnormal parking" type, the determination rule can be set as: the vehicle is stationary for a long time in a non-designated parking area.

[0077] For the "sudden braking accompanied by abnormal behavior" type, the determination rule can be set as: the vehicle behavior pattern shows sudden deceleration or stop, and a sudden braking sound event is detected in the area near the vehicle.

[0078] In actual operation, after the system obtains the vehicle information such as the motion trajectory, speed, and position of the target vehicle, as well as the sound information such as the type and occurrence area of the environmental sound event, it will check whether these information meet the determination rule of "vehicle reverse", whether they meet the determination rule of "vehicle overspeed", whether they meet the determination rule of "abnormal parking", and whether they meet the determination rule of "sudden braking accompanied by abnormal behavior". If it is found that the motion trajectory of the vehicle is opposite to the normal direction, the determination rule of "vehicle reverse" is met, the system generates a preliminary event determination result, indicating that a vehicle reverse event may have occurred. If the speed of the vehicle is too high in a specific area, the determination rule of "vehicle overspeed" is met, the system generates a preliminary event determination result, indicating that a vehicle overspeed event may have occurred. If the vehicle is stationary for a long time in a non-designated area, the determination rule of "abnormal parking" is met, the system generates a preliminary event determination result, indicating that an abnormal parking event may have occurred. If the vehicle suddenly decelerates and there is a sudden braking sound near it, the determination rule of "sudden braking accompanied by abnormal behavior" is met, the system generates a preliminary event determination result, indicating that a sudden braking accompanied by abnormal behavior event may have occurred. If multiple determination rules are met within the same time period, for example, the vehicle both overspeeds and abnormally parks, a preliminary determination result containing multiple event types is generated.

[0079] Further, step S4 comprises: S41: According to the preset event confirmation rule, confirm each preliminary event in the preliminary event determination result to obtain a confirmed event list. S42: When multiple events exist in the confirmed event list, the multiple events are prioritized according to preset priority rules and types and occurrence locations of the confirmed events, to obtain a priority sorting result; S43: According to the priority sorting result and a preset linkage strategy rule, a linkage instruction is generated for the confirmed event.

[0080] The preset event confirmation rule refers to a standard set for verifying validity of the preliminary event determination result, which can include conditions based on an event confidence threshold, a duration, and relevance to other sensing information.

[0081] The confirmed event list refers to a set of events that are determined to be real and valid after being filtered by the event confirmation rule.

[0082] For example, an event confirmation rule can be set as follows: only when an event preliminarily determined as "vehicle collision" has associated audio information detecting "metal collision sound" or "sudden braking sound" and video analysis obtaining a vehicle speed change rate exceeding a certain threshold, the event is confirmed as a real event.

[0083] For the priority rule, the priority of "oil zone collision" can be set to be higher than that of "access channel reverse driving", and the priority of "access channel reverse driving" can be set to be higher than that of "parking area abnormal parking".

[0084] The linkage strategy rule can be set as follows: for a confirmed "oil zone collision" event, if the priority is the highest, an instruction is generated to stop all oil dispensers in the collision occurrence area, start a sound and light alarm in the area, and send a highest-level alarm to a monitoring center; for a confirmed "access channel reverse driving" event, if the priority is higher, an instruction is generated to start voice broadcast warning in the access channel and display warning information on an entrance display screen; for a confirmed "parking area abnormal parking" event, if the priority is lower, an instruction is generated to display prompt information on a display screen in the parking area. Through application of such specific rules, the system can effectively filter preliminary determination results, reasonably sort concurrent events, and generate linkage instructions with strong pertinence.

[0085] Further, step S43 includes the following steps: S44: Conflict detection is performed on the linkage instruction, and a conflict type existing in the linkage instruction is detected; S45: According to the conflict type, the conflict is resolved according to a preset conflict resolution rule and a priority sorting result associated with the confirmed event, to obtain a final linkage instruction.

[0086] The conflict detection refers to identifying whether there is a situation of mutual contradiction, resource competition or logical inconsistency in the preliminary generated linkage instruction set. The conflict type refers to the specific form of the conflict, for example, multiple instructions attempt to control the same device to perform different actions at the same time, or multiple instructions compete for the use of limited system resources. The conflict resolution rule refers to a set of pre-set strategies for handling different types of conflicts. These rules can define the processing method to be taken when a specific conflict occurs, for example, which instruction to execute first, how to modify or discard the conflicting instructions, how to allocate the competing resources, etc.

[0087] The priority ranking result refers to the result obtained by ranking the importance or urgency of multiple confirmed events according to pre-set rules. This result is used to guide decision-making in the conflict resolution process, ensuring that linkage instructions corresponding to high-priority events are given priority or retained.

[0088] For example, suppose the system detects two confirmed events at the same time: one is a vehicle collision in the oil area (high-priority event) that triggers the linkage instructions "stop all oil pumps in the oil area" and "start the oil area fire sprinkler"; the other is a vehicle scratch in the parking area (low-priority event) that triggers the linkage instructions "start the parking area area lighting" and "play a voice warning". After preliminarily generating these instructions, the system performs conflict detection.

[0089] If there is a conflict resolution rule in the pre-set rule base, for example, the rule states that "when the fire sprinkler is started, the related area lighting should be turned off", the system will detect a possible conflict between "start the oil area fire sprinkler" and "start the parking area area lighting" (for example, if the start of the fire sprinkler requires the shutdown of all non-emergency lighting).

[0090] At this time, the system will decide how to handle it according to the conflict resolution rule and the event priority (the oil area collision priority is higher than the parking area scratch). One possible resolution is that since the oil area collision has high priority and the fire sprinkler instruction is more critical, the system will retain the "start the oil area fire sprinkler" instruction and modify or cancel the "start the parking area area lighting" instruction according to the rule, or adjust its execution mode to avoid conflict and ensure that the response measures for high-priority events are effectively executed. The final output linkage instruction set will be a set of consistent instructions after conflict resolution, for example, it may include "stop all oil pumps in the oil area", "start the oil area fire sprinkler", "play a voice warning", and the "start the parking area area lighting" instruction may be modified or removed.

[0091] Please refer to Figure 2 , Figure 3 A multi-modal perception gas station environment safety warning system is applied in the steps of any of the above methods, and the system comprises: The acquisition module 201 acquires video data and audio data of each divided area of the gas station. The extraction module 202 extracts vehicle target information, vehicle behavior information, and environmental sound information from the video data and the audio data. The determination module 203 performs preliminary event determination based on the vehicle target information, the vehicle behavior information, and the environmental sound information, and obtains a preliminary event determination result. The instruction generation module 204 inputs the preliminary event determination result into a rule reasoning engine, and the rule reasoning engine performs event confirmation on the preliminary event determination result based on a pre-set rule base, performs priority sorting when there are multiple events, and generates a linkage instruction matching the event type and the occurrence location according to the event type and the occurrence location. The instruction sending module 205 sends the linkage instruction to a corresponding local response device for execution.

[0092] The acquisition module 201 is a hardware or software unit for collecting raw perception data, which can be realized by using a video acquisition device (such as a camera), an audio acquisition device (such as a microphone), and a data interface or a driver connected with the devices.

[0093] The extraction module 202 is a processing unit for identifying and analyzing specific information from raw perception data, which can be realized by using image processing algorithms, computer vision techniques, audio signal processing algorithms, acoustic event detection techniques, and processors (such as CPU, GPU, NPU) running these algorithms.

[0094] The determination module 203 is a logic processing unit for preliminary risk assessment or event classification based on the extracted information, which can be realized by using a rule-based expert system, a machine learning model (such as a classifier), or a statistical analysis method.

[0095] The instruction generation module 204 is a complex processing unit for further intelligent decision-making and generating control signals based on the preliminary determination result, which can be realized by using a rule reasoning engine, a priority sorting algorithm, a linkage strategy configuration module, and a computing platform running these logics.

[0096] The instruction sending module 205 is a communication or control interface for transmitting the generated control signals to the execution device, which can be realized by using a network communication protocol (such as TCP / IP), an industrial control bus (such as Modbus), a wireless communication technology, or a relay control circuit.

[0097] The working principle of the system is as follows: the acquisition module 201 continuously collects video data and audio data of each area of the gas station as the original input of the system. These original data are transmitted to the extraction module, and the extraction module 202 performs vehicle recognition, tracking and behavior analysis on the video data in parallel or series, and performs environmental sound event recognition and positioning on the audio data. The extraction module 202 outputs the processed structured information, including target vehicle information, vehicle behavior information and environmental sound information, to the judgment module. The judgment module 203 uses these multi-modal information to make a fusion judgment according to the preset preliminary event judgment rule, quickly identifies potential safety events and generates a preliminary event judgment result. The preliminary event judgment result is then sent to the instruction generation module 204. The core of the instruction generation module 204 is the rule reasoning engine, which verifies and confirms the preliminary judgment result according to the preset rule base, and excludes false positives. When there are multiple confirmed events, the rule reasoning engine prioritizes these events according to the preset priority rules, taking into account factors such as event type and location. Finally, according to the type, location and priority of the confirmed events, the instruction generation module calls the preset linkage strategy rule to generate a linkage instruction that accurately matches the specific event scenario. The generated linkage instruction is transmitted to the instruction sending module, and the instruction sending module 205 is responsible for sending these instructions to the corresponding local response device to trigger the corresponding safety response measures.

[0098] Through the cooperative work of the above modules, the system can efficiently execute the above method, integrating multi-modal perception, intelligent judgment, priority management and linkage control in a unified architecture, providing comprehensive early warning and response capabilities for gas station environmental safety. This system architecture provides a reliable carrier for the effective landing of the above method, enabling real-time and accurate implementation of complex multi-modal processing and intelligent decision-making processes, solving the problem of difficulty in effectively landing the method or difficulty in efficiently implementing the method by existing systems.

[0099] In this document, the term "example" is used only to refer to some implementations, and does not necessarily refer to the best implementation. Relational terms such as first and second and the like can only be used to distinguish one entity or action from another, and do not necessarily require or imply any actual such relationship or order between or among the entities or actions.

[0100] The above only describes the embodiments of the present application and does not limit the protection scope of the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-modal perception based forewarning method for safe environment of a fueling station, characterized in that, The method comprises steps of: S1: acquiring video data and audio data of each divided area of a gas station; S2: extracting vehicle target information, vehicle behavior information and environmental sound information according to the video data and the audio data; S3: performing preliminary event determination according to the vehicle target information, the vehicle behavior information and the environmental sound information to obtain a preliminary event determination result; S4: inputting the preliminary event determination result into a rule reasoning engine, the rule reasoning engine performing event confirmation on the preliminary event determination result according to a preset rule base, performing priority ranking when there are multiple events, and generating a linkage instruction matched with an event type and an occurrence position according to the event type and the occurrence position; S5: sending the linkage instruction to a corresponding local response device for execution.

2. A multi-modal perception based forecourt environment safety warning method as claimed in claim 1, wherein, Step S2 comprises: S21: identifying a vehicle target in the gas station area according to the video data, tracking a motion trajectory of the vehicle target, and analyzing a behavior mode of the vehicle target; S22: identifying a preset environmental sound event and determining an approximate occurrence area of the environmental sound event according to the audio data; S23: associating the vehicle target with the environmental sound event according to the motion trajectory, the behavior mode and the approximate occurrence area of the environmental sound event, and determining the vehicle target information, the vehicle behavior information and the environmental sound information based on an association result.

3. A multi-modal perception based forecourt environment safety warning method as claimed in claim 2, wherein, Step S21 comprises: S211: determining a position of the vehicle target in gas station area division according to the video data; S212: continuously tracking the vehicle target to generate a motion trajectory of the vehicle target; S213: analyzing a behavior mode of the vehicle target according to the motion trajectory of the vehicle target and the position of the vehicle target in the gas station division area.

4. A multi-modal perception based forecourt environment safety warning method as claimed in claim 2, wherein, Step S23 comprises: S231: obtaining a vehicle target list to be associated according to the motion trajectory and the behavior mode; S232: screening a candidate vehicle target set located in the area from the vehicle target list according to the approximate occurrence area of the environmental sound event; S233: calculating an association degree of the vehicle target and the environmental sound event for each vehicle target in the candidate vehicle target set to obtain an association result; S234: determining the vehicle target information, the vehicle behavior information and the environmental sound information according to the association result.

5. A multi-modal perception based forecourt environment safety warning method as claimed in claim 4, wherein, Step S233 comprises: S2331: calculating a spatial association relationship of the vehicle target and the environmental sound event for each vehicle target in the candidate vehicle target set to obtain a spatial association score; S2332: calculating a temporal association relationship of the vehicle target and the environmental sound event to obtain a temporal association score; S2334: synthesizing the spatial association score and the temporal association score to calculate the association degree to obtain the association result.

6. A multi-modal perception based forecourt environment safety warning method as claimed in claim 1, wherein, Step S3 comprises: S321: determining rules for each preliminary event type according to a plurality of preset preliminary event types; S322: For each preset preliminary event type, whether a corresponding decision rule is satisfied is evaluated according to the vehicle target information, the vehicle behavior information and the environmental sound information; S323: When any of the decision rules is satisfied, the preliminary event decision result is generated.

7. A multi-modal perception based forecourt environment safety warning method as claimed in claim 6, wherein, In step S321, the decision rules include at least one decision condition based on the position of the vehicle target in the divided area of the gas station, the behavior mode of the vehicle target or the type and occurrence area information of the environmental sound event.

8. A multi-modal perception based forecourt environment safety warning method as claimed in claim 1, wherein, Step S4 includes: S41: According to a preset event confirmation rule, each of the preliminary events in the preliminary event decision result is confirmed to obtain a confirmed event list; S42: When there are multiple events in the confirmed event list, the multiple events are prioritized according to a preset priority rule and the type and occurrence position of the confirmed events to obtain a priority sorting result; S43: According to the priority sorting result and a preset linkage strategy rule, the linkage instruction is generated for the confirmed events.

9. A multi-modal perception based forecourt environment safety warning method as claimed in claim 8, wherein, After step S43, it includes: S44: The linkage instruction is conflict detected to identify the conflict type existing in the linkage instruction; S45: According to the conflict type, the conflict is resolved according to a preset conflict resolution rule and the priority sorting result associated with the confirmed events to obtain the final linkage instruction.

10. A multi-modal perception based forecourt environment safety warning system, characterized in that, The system is applied to the steps of the method of any one of claims 1-9, and the system includes: An acquisition module that acquires video data and audio data of each divided area of the gas station; An extraction module that extracts vehicle target information, vehicle behavior information and environmental sound information according to the video data and the audio data; A decision module that performs preliminary event decision according to the vehicle target information, the vehicle behavior information and the environmental sound information to obtain a preliminary event decision result; An instruction generation module that inputs the preliminary event decision result into a rule reasoning engine, the rule reasoning engine performs event confirmation on the preliminary event decision result according to a preset rule base, prioritizes when there are multiple events, and generates a linkage instruction matched with the type and occurrence position of the events according to the type and occurrence position of the events; An instruction sending module that sends the linkage instruction to a corresponding local coping device for execution.

Images