A method for detecting and identifying hidden dangers in the operation process of loading and unloading vehicles at a gas station

Abstract

The present application relates to the field of process detection technology, and discloses a method for detecting and identifying hidden dangers in the operation process of loading and unloading vehicles at a gas station, wherein the method includes: obtaining the loading and unloading configuration data of the vehicle to be loaded and unloaded in the gas station; determining the working condition cluster of the vehicle to be loaded and unloaded according to the loading and unloading configuration data, and obtaining the local image acquisition area represented by the working condition cluster; obtaining the reference video frame of the vehicle to be loaded and unloaded, and based on the local image acquisition area, cropping a local image from the reference video frame; identifying whether there is a specific scene mark in the local image, and selecting a detection model that matches the recognition result; using the selected detection model to detect the local image to determine the hidden dangers of the operation process in the local image, and generating alarm information that matches the hidden dangers of the operation process. The technical solution provided by one or more embodiments of the present application can improve the detection efficiency and accuracy of the operation process.

Description

Technical Field

[0001] The present application relates to the field of process detection technology, and in particular to a method, device and system for detecting and identifying hidden dangers in the operation process of loading and unloading vehicles at a gas station. Background Art

[0002] The operation process of loading and unloading vehicles in gas stations is cumbersome and complicated. Once the operation process is abnormal, it will cause serious consequences. In view of this, the operation process in gas stations is currently managed through manual supervision. For example, it can be judged whether the operation process in the gas station is compliant by manually watching the monitoring video.

[0003] However, the existing manual supervision method is usually time-consuming and labor-intensive, and the environment in the gas station is relatively complex. It is difficult to find hidden dangers in the operation process only through manual subjective judgment. Therefore, a more efficient and accurate operation process detection method is urgently needed. Summary of the invention

[0004] The present application provides a method, device and system for detecting and identifying hidden dangers in the operation process of loading and unloading trucks at a gas station, which can improve the detection efficiency and accuracy of the operation process.

[0005] In a first aspect, the present application provides a method for detecting and identifying hidden dangers in the operation process of loading and unloading vehicles at a gas station, the method comprising: obtaining loading and unloading configuration data of the vehicle to be loaded and unloaded in the gas station, the loading and unloading configuration data being used to at least characterize the gas interface parameters of the vehicle to be loaded and unloaded and the target hose parameters for connecting the gas interface; rewriting the loading and unloading configuration data according to a timestamp to generate time series data, the time series data comprising a plurality of rows of data entries, wherein different data entries correspond to different timestamps, and in any data entry, data parameters arranged according to preset field identifiers are included; dividing the time series data into a plurality of different sequence subsets according to field identifiers, wherein each sequence subset comprises at least one data parameter under a field identifier; determining an index subset and an attribute subset in the divided sequence subsets, and generating index features of the index subset and attribute features of the attribute subset respectively, and assigning different weight values ​​to the index features and the attribute features, and assigning different weight values ​​to the index features and the attribute features, and assigning different weight values ​​to the index features and the attribute features based on the assigned weight values. The attribute features are weighted to obtain weighted features corresponding to the time series data; the weighted features are matched with representative features of each working condition cluster, and the working condition cluster with the highest matching degree is used as the working condition cluster where the vehicle to be loaded and unloaded is located, and the local image acquisition area represented by the working condition cluster is obtained; when the vehicle to be loaded and unloaded arrives at the specified position, a reference video frame of the vehicle to be loaded and unloaded is obtained, and a local image is cropped from the reference video frame based on the local image acquisition area; it is identified whether there is a specific scene mark in the local image, and a detection model matching the recognition result is selected, wherein the detection model is an RGB detection model or an IR detection model; wherein, if after the IR detection model is selected, it is identified that the specific scene mark in the local image disappears, the IR detection model is switched back to the RGB detection model; the local image is detected using the selected detection model to determine the operation process hidden dangers in the local image, and an alarm information adapted to the operation process hidden dangers is generated.

[0006] The technical solution provided by this embodiment of the present application can determine the area with a high probability of process hidden dangers in the monitoring video screen according to the working condition cluster of the vehicle to be loaded and unloaded, so as to perform process detection on the area. During the detection process, in order to cope with the complex loading and unloading environment in the gas station, the specific scene identifier can be identified to select the detection model that matches the current loading and unloading environment. Through the dual screening of the detection screen and the detection model, the detection efficiency and accuracy of the process hidden dangers can be greatly improved, and then the alarm information adapted to the operating process hidden dangers can be provided to the on-site staff. Among them, after the time series data is divided according to the field identifier, multiple sequence subsets with different importance can be obtained. In these sequence subsets, an index subset that can reflect the current loading and unloading characteristics and an attribute subset that can reflect the historical loading and unloading characteristics can be determined. Subsequently, by weighting the features of these two subsets, the weighted features of the time series data can be obtained. Using the weighted features for feature matching, the working condition cluster of the vehicle to be loaded and unloaded can be accurately determined.

[0007] In one embodiment, the type of vehicle to be loaded and unloaded is a tank truck, and the loading and unloading configuration data is retrieved through a unique identifier of the tank truck.

[0008] In one embodiment, respectively generating the index features of the index subset and the attribute features of the attribute subset includes: inputting the data of each timestamp in the index subset into a long short-term memory network to extract the index features of the index subset through the long short-term memory network; and inputting the data of each timestamp in the attribute subset into a long short-term memory network to extract the attribute features of the attribute subset through the long short-term memory network.

[0009] The technical solution provided in this embodiment of the present application generates index features and attribute features respectively through long short-term memory networks, which can ensure the timing characteristics of index features and attribute features, and further ensure that the weighted features finally generated can accurately represent time series data.

[0010] In one embodiment, the local image acquisition area represented by the working condition cluster is generated in the following manner: obtaining an operation process hidden danger set corresponding to the working condition cluster, the operation process hidden danger set including characteristic positions where operation process hidden dangers appear in the video frame; identifying the confidence of each characteristic position in the operation process hidden danger set, and screening out multiple candidate characteristic positions with confidence higher than a specified threshold; determining a minimum bounding rectangle in the video frame for covering all candidate characteristic positions, and using the area where the minimum bounding rectangle is located in the video frame as the local image acquisition area represented by the working condition cluster.

[0011] The technical solution provided in this embodiment of the present application uses confidence levels to screen the characteristic positions of each hidden danger, and can obtain a relatively accurate minimum enclosing rectangle, thereby identifying local image acquisition areas in video frames where hidden dangers are likely to occur, reducing the subsequent data processing volume and improving the efficiency of hidden danger detection.

[0012] In one embodiment, identifying whether there is a specific scene marker in the local image includes: detecting whether there is a smoke area in the local image, and if so, detecting the proportion of the smoke area in the local image; if the proportion of the smoke area in the local image is higher than a specified ratio threshold, determining that there is a specific scene marker in the local image.

[0013] In one embodiment, selecting a detection model that matches the recognition result includes: if the recognition result indicates that a specific scene identifier exists in the local image, selecting an IR detection model; if the recognition result indicates that no specific scene identifier exists in the local image, selecting an RGB detection model.

[0014] The technical solution provided in this embodiment of the present application can identify the smoke area and select the IR detection model for infrared detection when smoke appears in the video frame. In this way, when the accuracy of visible light detection cannot be guaranteed, the infrared detection model can be used for detection, thereby ensuring the accuracy of hidden danger detection.

[0015] In one embodiment, the detection model is trained in the following manner: an RGB data set and an IR data set are obtained, wherein the RGB data set includes visible light image samples and the IR data set includes infrared image samples; training labels are set for the visible light image samples and the infrared image samples respectively, and the training labels are used at least to characterize the position of the vehicle to be loaded and unloaded, the connection status of the hose on the vehicle to be loaded and unloaded, and the status information of the staff, wherein the visible light image samples also have additional labels for characterizing smoke scenes; under the supervision of the training labels, the RGB data set and the IR data set are trained respectively using an improved YOLOv8 network, and after the training is completed, the first network weight for the RGB data set and the second network weight for the IR data set are saved; wherein, in the improved YOLOv8 network, the Backbone part adopts a DenseNet structure; based on the first network weight, an RGB detection model is generated, and based on the second network weight, an IR detection model is generated.

[0016] The technical solution provided in this embodiment of the present application modifies and replaces the Backbone part of YOLOv8 with the DenseNet structure, which reduces the computational overhead and increases the model's detection capability for small target objects without losing the original characteristics of YOLOv8. During the training process, by setting additional smoke scene labels for infrared image samples, the RGB detection model and the IR detection model can be accurately distinguished, thereby improving the accuracy of hidden danger detection.

[0017] In one embodiment, generating alarm information that is compatible with the hidden dangers of the operation process includes: if the hidden danger of the operation process is represented by the hose switching from a connected state to a disconnected state, generating a first alarm message; if the hidden danger of the operation process is represented by the number of workers being less than a specified number threshold, or the distance between the workers and the vehicle to be loaded and unloaded is greater than a specified distance, generating a second alarm message.

[0018] The technical solution provided in this implementation mode of the present application generates different alarm information for different hidden dangers, which can effectively improve the efficiency of hidden danger detection and correction and ensure the safe progress of loading and unloading work.

[0019] In a second aspect, the present application further provides a device for detecting and identifying hidden dangers in the operation process of loading and unloading vehicles in a gas station, the device comprising:

[0020] A configuration data acquisition unit, used to acquire loading and unloading configuration data of a vehicle to be loaded and unloaded in a gas station, wherein the loading and unloading configuration data is used to characterize at least gas interface parameters of the vehicle to be loaded and unloaded and target hose parameters for connecting the gas interface;

[0021] The acquisition area acquisition unit is used to rewrite the loading and unloading configuration data according to the timestamp to generate time series data, wherein the time series data includes a plurality of rows of data entries, wherein different data entries correspond to different timestamps, and in any data entry, data parameters arranged according to preset field identifiers are included; and the time series data is divided into a plurality of different sequence subsets according to the field identifiers, wherein each sequence subset includes at least one data parameter under a field identifier;

[0022] The acquisition area acquisition unit is further used to determine an index subset and an attribute subset in the divided sequence subset, and respectively generate index features of the index subset and attribute features of the attribute subset, and assign different weight values ​​to the index features and the attribute features, and perform feature weighting on the index features and the attribute features based on the assigned weight values ​​to obtain weighted features corresponding to the time series data;

[0023] The acquisition area acquisition unit is further used to match the weighted features with representative features of each operating condition cluster, and use the operating condition cluster with the highest matching degree as the operating condition cluster where the vehicle to be loaded and unloaded is located, and acquire a local image acquisition area represented by the operating condition cluster;

[0024] A local image determination unit, configured to obtain a reference video frame of the vehicle to be loaded or unloaded when the vehicle to be loaded or unloaded arrives at a designated position, and to crop a local image from the reference video frame based on the local image acquisition area;

[0025] A model selection unit, used to identify whether a specific scene identifier exists in the partial image, and select a detection model that matches the recognition result, wherein the detection model is an RGB detection model or an IR detection model;

[0026] The hidden danger alarm unit is used to detect the local image using the selected detection model to determine the operation process hidden dangers existing in the local image and generate alarm information adapted to the operation process hidden dangers.

[0027] On the third aspect, the present application also provides a system for detecting and identifying hidden dangers in the operating process of loading and unloading vehicles at a gas station. The system includes a memory and a processor. The memory is used to store a computer program. When the computer program is executed by the processor, the above-mentioned method for detecting and identifying hidden dangers in the operating process of loading and unloading vehicles at a gas station is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 A step diagram of a method for detecting and identifying hidden dangers in the operation process of a gas station loading and unloading vehicle provided in an embodiment of the present application;

[0030] Figure 2 This is a schematic diagram of the structure of a detection system in one embodiment of the present application;

[0031] Figure 3 A flow chart of a detection method in an application scenario for this application;

[0032] Figure 4 A schematic diagram of functional modules of a system for detecting and identifying hidden dangers in the operation process of a gas station loading and unloading vehicle provided in one embodiment of the present application;

[0033] Figure 5 A schematic structural diagram of a system for detecting and identifying hidden dangers in the operation process of loading and unloading vehicles at a gas station provided in one embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0035] In addition, the descriptions involving "first", "second", etc. in this application are for descriptive purposes only and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more. In addition, the use of "based on" or "according to" means openness and inclusiveness, because the process, steps, calculations or other actions "based on" or "according to" one or more of the conditions or values ​​can be based on additional conditions or beyond the described values ​​in practice.

[0036] The operation procedures for loading and unloading trucks at gas stations are relatively complicated and numerous. Among them, factors such as whether the hoses are connected properly and whether the personnel are on duty are important causes of safety accidents.

[0037] At present, the management of the operation process in the gas production station is mainly carried out by manually watching video monitoring, but this method is time-consuming and labor-intensive and has low effect. At the same time, a large amount of smoke will be generated during the loading and unloading process of the vehicle to be loaded and unloaded, which has a great impact on human judgment and cannot detect the operation problems of the staff in the first time. Therefore, the present invention designs and proposes a process detection method for loading and unloading vehicles in a gas production station, which can detect whether the personnel are on duty, whether the hose connection is compliant, and whether there is smoke.

[0038] See also Figure 1 An embodiment of the present application provides a method for detecting and identifying hidden dangers in the operation process of loading and unloading vehicles at a gas station, and the method includes the following steps.

[0039] S1: Acquire loading and unloading configuration data of a vehicle to be loaded and unloaded in a gas station, wherein the loading and unloading configuration data is at least used to characterize gas interface parameters of the vehicle to be loaded and unloaded and target hose parameters for connecting the gas interface.

[0040] In this embodiment, the type of vehicle to be loaded and unloaded may specifically be a tank truck. For the tank truck to be loaded and unloaded, the loading and unloading configuration data of the tank truck to be loaded and unloaded may be obtained, and the loading and unloading configuration data may be retrieved through the unique identification of the tank truck. Among them, the unique identification of the tank truck may be the license plate number of the tank truck, or it may be the unique identification information such as the factory number of the tank truck. During the use of the vehicle to be loaded and unloaded, the loading and unloading configuration data may be regularly reported on the ground, and the loading and unloading configuration data may include the reported timestamp, the current loading and unloading personnel, the gas interface parameters, the target hose parameters for connecting the gas interface, the loading and unloading location and other information related to loading and unloading. Among them, the gas interface parameters may characterize the parameters such as the size of the gas interface, the length of use, the number of loading and unloading times, etc. The target hose parameters may characterize the parameters such as the length, inner and outer diameters, materials, and length of use of the hose. The server will record the loading and unloading configuration data reported by the vehicle to be loaded and unloaded each time, and these loading and unloading configuration data may be combined to detect hidden dangers in the operation process later.

[0041] S2: Determine the working condition cluster where the vehicle to be loaded and unloaded is located according to the loading and unloading configuration data, and obtain a local image acquisition area represented by the working condition cluster.

[0042] In this embodiment, the loading and unloading configuration data can be rewritten according to the reporting timestamp in the loading and unloading configuration data to generate time series data. The time series data can include several rows of data entries, where different data entries correspond to different timestamps, and in any data entry, data parameters arranged according to preset field identifiers are included. Among them, the field identifier can be the current loading and unloading personnel (stuff), gas interface parameters (parameter1), target hose parameters for connecting the gas interface (parameter2), loading and unloading location (location) and other identifiers in step S1. The data parameters under these field identifiers can be summarized according to the timestamp, thereby forming data entries sorted according to the timestamp.

[0043] In this embodiment, the time series data can be divided according to the field identifier to obtain multiple different sequence subsets. In each sequence subset, data parameters with different timestamps can be included. In practical applications, different field identifiers usually have different importance for hidden danger detection. For example, gas interface parameters and target hose parameters are relatively important. Based on the importance of the field identifier, an index subset and an attribute subset can be determined from the sequence subset, among which the index subset has a relatively high importance and can better characterize the current loading and unloading characteristics, and the attribute subset has a relatively low importance, but can characterize the historical loading and unloading characteristics.

[0044] In this embodiment, in order to digitally process the index subset and the attribute subset, the long short-term memory (LSTM) network can be used to generate index features and attribute features respectively. Specifically, the data of each time stamp in the index subset can be input into the long short-term memory network to extract the index features of the index subset through the long short-term memory network. Similarly, the data of each time stamp in the attribute subset can be input into the long short-term memory network to extract the attribute features of the attribute subset through the long short-term memory network.

[0045] In this embodiment, the importance of the index subset and the attribute subset is usually different. By quantifying the importance, different weight values ​​can be assigned to the index features and attribute features. Specifically, the weight value can be a value greater than 0 and less than 1. The higher the importance, the larger the corresponding weight value. The specific value of the weight value can be flexibly set according to the actual application scenario. After the weight value is assigned, the index features and attribute features can be weighted based on the assigned weight value to obtain the weighted feature corresponding to the time series data. The weighted feature combines the current loading and unloading environment and the historical loading and unloading environment, and sets the respective proportions more accurately through the weight value, so it can accurately characterize the characteristics of the time series data.

[0046] The technical solution provided in this embodiment of the present application generates index features and attribute features respectively through long short-term memory networks, which can ensure the timing characteristics of index features and attribute features, and further ensure that the weighted features finally generated can accurately represent time series data.

[0047] In this embodiment, after obtaining the weighted features, the weighted features can be matched with the representative features of each operating condition cluster, and the operating condition cluster with the highest matching degree can be used as the operating condition cluster where the vehicle to be loaded and unloaded is located. In practical applications, different tank trucks can be divided into different operating condition clusters as the working time accumulates. In the same operating condition cluster, the operating process hazards that occur in the tank truck during the loading and unloading process may also be relatively similar. In view of this, by matching the operating conditions of the vehicle to be loaded and unloaded, it is possible to query the operating condition cluster that is relatively close to the operating condition of the vehicle to be loaded and unloaded. Specifically, each operating condition cluster has its own representative feature, which can be the average of the weighted features of each tank truck in the operating condition cluster. By matching the weighted features of the vehicle to be loaded and unloaded with the representative features of each operating condition cluster, the representative feature with the highest matching degree can be queried, and the operating condition cluster corresponding to the representative feature with the highest matching degree can be used as the operating condition cluster where the vehicle to be loaded and unloaded is located.

[0048] The technical solution provided in this embodiment of the present application can obtain multiple sequence subsets of different importance after dividing the time series data according to the field identifier. In these sequence subsets, an index subset that can reflect the current loading and unloading characteristics and an attribute subset that can reflect the historical loading and unloading characteristics can be determined. Subsequently, by weighting the features of these two subsets, the weighted features of the time series data can be obtained. By using the weighted features for feature matching, the working condition cluster of the vehicle to be loaded and unloaded can be accurately determined.

[0049] In this embodiment, the tank trucks in the working condition cluster often have similar operational process risks during loading and unloading, and the operational process risks are usually located in a certain local position in the monitoring video frame. Subsequently, for tank trucks with similar working conditions, only the image at the local position can be detected, thereby greatly reducing the amount of data for image detection.

[0050] In practical applications, the local positions of the operating process hidden dangers of each tank truck in the working condition cluster can be counted, and the local image acquisition area represented by the working condition cluster can be obtained. Specifically, the operating process hidden danger set corresponding to the working condition cluster can be obtained, and the operating process hidden danger set can include the characteristic position where the operating process hidden danger appears in the video frame. For example, in the case of a hose connection hidden danger, the characteristic position can be a position near the gas interface. For another example, in the case of a staff number hidden danger, the characteristic position can be the position where the staff is located. In the operating process hidden danger set, each different hidden danger can correspond to a different number of occurrences. The higher the number, the higher the confidence of the corresponding characteristic position. In view of this, the confidence of each characteristic position in the operating process hidden danger set can be identified, and multiple candidate characteristic positions with confidence higher than the specified threshold can be screened out. The purpose of screening is to eliminate individual abnormal characteristic positions in the operating process hidden danger set. After the candidate characteristic positions are screened out, the minimum bounding rectangle for covering all candidate characteristic positions can be determined in the video frame, and the area where the minimum bounding rectangle is located in the video frame is used as the local image acquisition area represented by the working condition cluster. Generally speaking, the local image acquisition area can be determined by the pixel positions of the four vertices of the minimum circumscribed rectangle.

[0051] The technical solution provided in this embodiment of the present application uses confidence levels to screen the characteristic positions of each hidden danger, and can obtain a relatively accurate minimum enclosing rectangle, thereby identifying local image acquisition areas in video frames where hidden dangers are likely to occur, reducing the subsequent data processing volume and improving the efficiency of hidden danger detection.

[0052] S3: When the vehicle to be loaded or unloaded arrives at the designated position, a reference video frame of the vehicle to be loaded or unloaded is obtained, and a local image is cropped from the reference video frame based on the local image acquisition area.

[0053] In this embodiment, when the vehicle to be loaded and unloaded arrives at the designated loading and unloading location, the camera can collect the video stream data of the vehicle to be loaded and unloaded, and by sampling the video stream data, the reference video frame of the vehicle to be loaded and unloaded can be obtained from the video stream data. When performing hidden danger detection, it is not necessary to detect the entire screen of the reference video frame, but it is possible to crop a local image from the reference video frame based on the local image acquisition area represented by the working condition cluster, and perform hidden danger detection on the local image.

[0054] S4: Identify whether there is a specific scene marker in the local image, and select a detection model that matches the identification result, wherein the detection model is an RGB detection model or an IR detection model; wherein, if after selecting the IR detection model, it is identified that the specific scene marker in the local image disappears, switch back to the RGB detection model from the IR detection model.

[0055] In this embodiment, since a large amount of smoke is generated during the loading and unloading process of the tank truck, in order to accurately detect the operation process, when smoke appears, it is necessary to switch from the visible light image to the infrared image for detection. Of course, in actual applications, the specific scene identification is not limited to the smoke scene, but can also include other scenes with obstructed vision. For example, in extreme weather conditions, when the visible light image is blurry, it can also be switched to the infrared image for detection.

[0056] In a specific application scenario, when identifying a specific scene marker, it is possible to detect whether there is a smoke area in the local image. If so, the proportion of the smoke area in the local image is detected. If the proportion of the smoke area in the local image is higher than the specified ratio threshold, it can be determined that there is a specific scene marker in the local image. At this time, since the smoke area is large, many details in the visible light image will be obscured by the smoke, and it is possible to switch to the infrared image for detection. Specifically, if the recognition result indicates that there is a specific scene marker in the local image, the IR (Infrared Ray) detection model can be selected; if the recognition result indicates that there is no specific scene marker in the local image, the RGB (Red, Green, Blue) detection model can be selected.

[0057] The technical solution provided in this embodiment of the present application can identify the smoke area and select the IR detection model for infrared detection when smoke appears in the video frame. In this way, when the accuracy of visible light detection cannot be guaranteed, the infrared detection model can be used for detection, thereby ensuring the accuracy of hidden danger detection.

[0058] In this embodiment, if after selecting the IR detection model, it is recognized that the specific scene mark in the local image disappears, then the IR detection model can be switched back to the RGB detection model, thereby saving algorithm pressure on the server.

[0059] In practical applications, the selected RGB detection model and IR detection model can be obtained by pre-training with data sets. Specifically, an RGB data set and an IR data set can be obtained, wherein the RGB data set includes visible light image samples, and the IR data set includes infrared image samples. After obtaining the data set, training labels can be set for the visible light image samples and the infrared image samples, respectively. The training labels are used to characterize at least the position of the tank truck, the connection status of the hose on the tank truck, and the status information of the staff. Compared with the infrared image samples, the visible light image samples also have additional labels for characterizing the smoke scene. Among them, the connection status of the hose can be indicated by pole_on to indicate connection, and by pole_off to indicate disconnection. The status information of the staff can be represented by the number of people and the distance from the tank truck.

[0060] Under the supervision of the training labels, the improved YOLOv8 network can be used to train the RGB data set and the IR data set respectively, and after the training is completed, the first network weight for the RGB data set and the second network weight for the IR data set are saved. Among them, in the improved YOLOv8 network, the Backbone part adopts the DenseNet structure. Finally, based on the first network weight, an RGB detection model can be generated, and based on the second network weight, an IR detection model can be generated.

[0061] The technical solution provided in this embodiment of the present application modifies and replaces the Backbone part of YOLOv8 with the DenseNet structure, which reduces the computational overhead and increases the model's detection capability for small target objects without losing the original characteristics of YOLOv8. During the training process, by setting additional smoke scene labels for infrared image samples, the RGB detection model and the IR detection model can be accurately distinguished, thereby improving the accuracy of hidden danger detection.

[0062] S5: Detect the local image using the selected detection model to determine the operation process hidden dangers in the local image, and generate alarm information adapted to the operation process hidden dangers.

[0063] In this embodiment, by detecting the local image with the selected detection model, the operation process hidden dangers existing in the local image can be output. Different types of hidden dangers can be represented by different status codes. In this way, based on the status code output by the model, the type of hidden danger can be quickly located, so that the hidden danger can be checked and corrected.

[0064] In practical applications, if the operation process hidden danger characterization hose is switched from a connected state to a disconnected state, a first alarm message may be generated, and the first alarm message may include a first status code. If the number of the operation process hidden danger characterization workers is less than a specified number threshold, or the distance between the workers and the tank truck is greater than a specified distance, a second alarm message may be generated, and the second alarm message may include a second status code.

[0065] The technical solution provided in this implementation mode of the present application generates different alarm information for different hidden dangers, which can effectively improve the efficiency of hidden danger detection and correction and ensure the safe progress of loading and unloading work.

[0066] In a specific application scenario, the technical solution of the present application can be implemented in the following manner.

[0067] The technical solution of this application mainly detects objects including people, tank trucks, connection hose status and smoke. The main detection models are visible light detection and infrared detection, which assist each other in determining the safety hazard monitoring of the operation process of loading and unloading vehicles at gas stations. At the model level, the Backbone part of YOLOv8 is modified and replaced with the DenseNet structure. Without losing the original characteristics of YOLOv8, it reduces computational overhead and increases the model's detection capability for small target objects. At the logical level, smoke detection is used to determine whether to switch to the infrared detection model, and the relative position of people and vehicles is used to estimate the on-the-job status of staff. The specific monitoring process is as follows:

[0068] Step 1:

[0069] First, the image data of the gas station loading and unloading trucks are collected. The collected images include hoses, loading and unloading trucks, and staff. Then, the data set is enhanced, including blur, brightness change, and mosaic enhancement. Named tankers_RGB and tankers_IR datasets, tankers_RGB is visible light data, and tankers_IR is infrared data. Finally, the dataset is labeled, and the label pole_on represents the hose connection state, pole_off represents the hose disconnection state, tank represents the tank truck, person represents the staff, and smoke represents smoke, where smoke is a unique label for tankers_RGB.

[0070] Step 2:

[0071] The annotated tankers_RGB and tankers_IR datasets are respectively input into the Dense-YOLOv8 network for training, and the model weights obtained after the training are saved. The obtained model weights are transformed to finally obtain tankers_RGB_model (RGB detection model) and tankers_IR_model (IR detection model). These two models are used to monitor the operation process of gas station loading and unloading vehicles.

[0072] Step 3:

[0073] Get the real-time RTSP (Real Time Streaming Protocol) video stream and input it into tankers_RGB_model. First, when the model detects a tanker, submit the time of loading and unloading gas to the background management system, and remind the staff to monitor the status of the tanker. Second, when the hose connection status changes from pole_off to pole_on, remind the staff in the monitoring room to observe and obtain the specific situation of the gas station loading and unloading truck in time, and respond to emergencies in time. Then, when the loading and unloading truck is working stably, if a large amount of smoke is detected, the background will automatically switch to tankers_IR_model for supervision. When the smoke disappears, turn off the supervision of tankers_IR_model to save the pressure on the algorithm server. Finally, during the operation of the tanker, the status of the staff is detected. During the loading and unloading process, it is usually necessary to ensure that there are two people in one car (the number of people may be slightly different in different scenarios), so the ratio of the number of people to the number of tankers should be greater than 2, and the position of the person should not exceed a certain distance from the position of the tanker. This distance is flexibly set according to the site and the location of the surveillance camera. If there is any operational problem when loading or unloading the vehicle, an alarm will be issued in time and relevant personnel will be notified to handle it in time.

[0074] See also Figure 2 and Figure 3 In another specific application scenario, deep learning and image algorithms can be combined to detect whether the hoses in the gas station are connected normally and whether the staff are on duty, so as to monitor the safety hazards of the gas station loading and unloading operation process. The detection system includes RGB camera, infrared camera (IR camera), AI algorithm server, and host computer. The specific detection steps are as follows:

[0075] Step 1: Use two cameras to collect image data of the gas station and mark it.

[0076] Step 2: Input the dataset into Dense-YOLOv8 for training and obtain two models, named tankers_RGB_model and tankers_IR_model.

[0077] Step 3, get the real-time RTSP video stream and input it into tankers_RGB_model, determine whether there is a tanker in the frame, the status of the tanker hose, whether there is a lot of smoke, and the number of staff. If the pipe connection status changes from pole_off to pole_on, notify the staff to observe; if a lot of smoke is detected, switch to the tankers_IR_model model in the background to detect infrared photos. Detect personnel during the operation of the tanker to ensure that there are two people in one car (the number of people may be slightly different in different scenarios).

[0078] Step 4: If there is any operational problem with the loading and unloading vehicle, an alarm will be issued in time and relevant personnel will be notified to handle it in time.

[0079] In a specific application example, the specific steps of step 3 are as follows: If it is detected that there is no tank truck in the video frame, continue to read the video frame. If a tank truck appears in the video screen, it indicates that the station starts to load and unload gas. At this time, it is necessary to submit the time of loading and unloading gas by the loading and unloading truck to the background management system and record it. If the tank truck exists, continue to monitor whether a hose appears. If a hose appears and the hose connection state changes from pole_off to pole_on, remind the staff in the monitoring room to observe and obtain the specific situation of the gas station loading and unloading truck in time to deal with emergencies. At the same time, continue to detect whether the on-site staff of the loading and unloading truck is present, and determine whether they are in the working area. If there is a lot of smoke, the background automatically switches the screen to an infrared image, and the model is replaced with tankers_IR_model to detect hidden dangers until the smoke disappears before closing the tankers_IR_model. If no smoke is detected, the tankers_RGB_model is used to detect the RGB video frame. If no staff is detected during the operation of the tank truck, remind the staff to monitor the status of the tank truck. If a staff member is detected, it is determined whether the ratio of the number of persons to the number of tankers is greater than 2, and whether the position of the person exceeds a certain distance from the position of the tanker. If the above conditions are not met, the staff member is reminded to monitor the status of the tanker.

[0080] The technical solution provided in this embodiment of the present application can determine the area in the monitoring video screen where process hidden dangers are likely to occur based on the working condition cluster of the vehicle to be loaded and unloaded, so as to perform process detection on this area. During the detection process, in order to cope with the complex loading and unloading environment in the gas station, specific scene identifiers can be identified to select a detection model that matches the current loading and unloading environment. Through the dual screening of the detection image and the detection model, the detection efficiency and accuracy of process hidden dangers can be greatly improved, thereby providing on-site staff with alarm information that is compatible with the hidden dangers of the operating process.

[0081] See also Figure 4 The present application also provides a device for detecting and identifying hidden dangers in the operation process of loading and unloading vehicles in a gas station, the device comprising:

[0082] A configuration data acquisition unit, used to acquire loading and unloading configuration data of a vehicle to be loaded and unloaded in a gas station, wherein the loading and unloading configuration data is used to characterize at least gas interface parameters of the vehicle to be loaded and unloaded and target hose parameters for connecting the gas interface;

[0083] The acquisition area acquisition unit is used to rewrite the loading and unloading configuration data according to the timestamp to generate time series data, wherein the time series data includes a plurality of rows of data entries, wherein different data entries correspond to different timestamps, and in any data entry, data parameters arranged according to preset field identifiers are included; and the time series data is divided into a plurality of different sequence subsets according to the field identifiers, wherein each sequence subset includes at least one data parameter under a field identifier;

[0084] The acquisition area acquisition unit is further used to determine an index subset and an attribute subset in the divided sequence subset, and respectively generate index features of the index subset and attribute features of the attribute subset, and assign different weight values ​​to the index features and the attribute features, and perform feature weighting on the index features and the attribute features based on the assigned weight values ​​to obtain weighted features corresponding to the time series data;

[0085] The acquisition area acquisition unit is further used to match the weighted features with representative features of each operating condition cluster, and use the operating condition cluster with the highest matching degree as the operating condition cluster where the vehicle to be loaded and unloaded is located, and acquire a local image acquisition area represented by the operating condition cluster;

[0086] A local image determination unit, configured to obtain a reference video frame of the vehicle to be loaded or unloaded when the vehicle to be loaded or unloaded arrives at a designated position, and to crop a local image from the reference video frame based on the local image acquisition area;

[0087] A model selection unit, used to identify whether a specific scene identifier exists in the partial image, and select a detection model that matches the recognition result, wherein the detection model is an RGB detection model or an IR detection model;

[0088] The hidden danger alarm unit is used to detect the local image using the selected detection model to determine the operation process hidden dangers existing in the local image and generate alarm information adapted to the operation process hidden dangers.

[0089] The specific implementation of each unit module can be found in the description of the aforementioned method implementation method, which will not be repeated here.

[0090] See also Figure 5 The present application also provides a system for detecting and identifying hidden dangers in the operation process of loading and unloading vehicles at a gas station. The system includes a memory and a processor. The memory is used to store a computer program. When the computer program is executed by the processor, the above-mentioned method for detecting and identifying hidden dangers in the operation process of loading and unloading vehicles at a gas station is implemented.

[0091] like Figure 5 As shown, the system includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0092] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0093] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0094] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0095] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0096] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0097] The embodiment of the present application also provides a computer-readable storage medium. The above method according to the embodiment of the present application can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0098] The embodiment of the present application provides a computer program product, which includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of any embodiment of the present application.

[0099] Although the embodiments of the present application are described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations are all within the scope defined by the appended claims.

[0100] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0101] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0102] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0103] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0104] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0105] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0106] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

[0107] Although the embodiments of the present application have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for detecting and identifying hidden dangers in the operation process of loading and unloading vehicles at a gas station, characterized in that: The method comprises: Acquire loading and unloading configuration data of a vehicle to be loaded and unloaded in a gas station, wherein the loading and unloading configuration data is used to characterize at least gas interface parameters of the vehicle to be loaded and unloaded and target hose parameters for connecting the gas interface; Rewriting the loading and unloading configuration data according to the timestamp to generate time series data, wherein the time series data includes a plurality of rows of data entries, wherein different data entries correspond to different timestamps, and in any data entry, data parameters arranged according to preset field identifiers are included; Dividing the time series data into a plurality of different sequence subsets according to the field identifier, wherein each sequence subset contains at least one data parameter under the field identifier; Determine an index subset and an attribute subset in the sequence subset obtained by the division, and respectively generate index features of the index subset and attribute features of the attribute subset, and assign different weight values ​​to the index features and the attribute features, and perform feature weighting on the index features and the attribute features based on the assigned weight values ​​to obtain weighted features corresponding to the time series data; Matching the weighted features with representative features of each operating condition cluster, taking the operating condition cluster with the highest matching degree as the operating condition cluster where the vehicle to be loaded and unloaded is located, and obtaining a local image acquisition area represented by the operating condition cluster; When the vehicle to be loaded or unloaded arrives at the designated location, a reference video frame of the vehicle to be loaded or unloaded is acquired, and a partial image is cropped from the reference video frame based on the partial image acquisition area; Identify whether there is a specific scene identifier in the partial image, and select a detection model that matches the identification result, wherein the detection model is an RGB detection model or an IR detection model; wherein, if after selecting the IR detection model, it is identified that the specific scene identifier in the partial image disappears, switch from the IR detection model back to the RGB detection model; The local image is detected using the selected detection model to determine the operation process hidden dangers present in the local image, and alarm information adapted to the operation process hidden dangers is generated.

2. The method according to claim 1, characterized in that The type of the vehicle to be loaded and unloaded is a tank truck, and the loading and unloading configuration data is retrieved through the unique identification of the tank truck.

3. The method according to claim 1, characterized in that Generating the index features of the index subset and the attribute features of the attribute subset respectively includes: The data of each timestamp in the index subset is input into the long short-term memory network to extract the index features of the index subset through the long short-term memory network; and the data of each timestamp in the attribute subset is input into the long short-term memory network to extract the attribute features of the attribute subset through the long short-term memory network.

4. The method according to claim 1, characterized in that: The local image acquisition area represented by the working condition cluster is generated in the following manner: Acquire an operation process hidden danger set corresponding to the working condition cluster, wherein the operation process hidden danger set includes characteristic positions where the operation process hidden danger appears in the video frame; Identify the confidence of each feature position in the set of hidden dangers in the operation process, and select multiple candidate feature positions whose confidence is higher than a specified threshold; A minimum bounding rectangle covering all candidate feature positions is determined in the video frame, and the area where the minimum bounding rectangle is located in the video frame is used as the local image acquisition area for representing the working condition cluster.

5. The method according to claim 1, characterized in that Identifying whether a specific scene identifier exists in the partial image includes: Detecting whether there is a smoke area in the partial image, and if so, detecting the proportion of the smoke area in the partial image; If the proportion of the smoke area in the local image is higher than a specified proportion threshold, it is determined that a specific scene identifier exists in the local image.

6. The method according to claim 1 or 5, characterized in that: The detection models selected to match the recognition results include: If the recognition result indicates that a specific scene identifier exists in the local image, selecting an IR detection model; If the recognition result indicates that there is no specific scene identifier in the partial image, the RGB detection model is selected.

7. The method according to claim 1, characterized in that The detection model is trained in the following way: Acquire an RGB data set and an IR data set, wherein the RGB data set includes visible light image samples, and the IR data set includes infrared image samples; Setting training labels for the visible light image samples and the infrared image samples respectively, wherein the training labels are used to characterize at least the location of the vehicle to be loaded and unloaded, the connection status of the hose on the vehicle to be loaded and unloaded, and the status information of the staff, wherein the visible light image samples also have labels that characterize the smoke scene; Under the supervision of the training labels, the RGB data set and the IR data set are trained respectively using an improved YOLOv8 network, and after the training is completed, a first network weight for the RGB data set and a second network weight for the IR data set are saved; wherein, in the improved YOLOv8 network, the Backbone part adopts a DenseNet structure; Based on the first network weights, an RGB detection model is generated, and based on the second network weights, an IR detection model is generated.

8. The method according to claim 1 or 7, characterized in that: Generating alarm information adapted to the hidden dangers of the operation process includes: If the operation process hidden danger characterization hose is switched from a connected state to a disconnected state, a first alarm message is generated; If the number of workers representing hidden dangers in the operation process is less than a specified number threshold, or the distance between the workers and the vehicle to be loaded and unloaded is greater than a specified distance, a second alarm message is generated.

9. A device for detecting and identifying hidden dangers in the operation process of loading and unloading vehicles at a gas station, characterized in that: The device comprises: A configuration data acquisition unit, used to acquire loading and unloading configuration data of a vehicle to be loaded and unloaded in a gas station, wherein the loading and unloading configuration data is used to characterize at least gas interface parameters of the vehicle to be loaded and unloaded and target hose parameters for connecting the gas interface; A collection area acquisition unit, used for rewriting the loading and unloading configuration data according to the timestamp to generate time series data, wherein the time series data includes a plurality of rows of data entries, wherein different data entries correspond to different timestamps, and in any data entry, data parameters arranged according to preset field identifiers are included; and dividing the time series data into a plurality of different sequence subsets according to the field identifier, wherein each sequence subset contains at least one data parameter under the field identifier; The acquisition area acquisition unit is further used to determine an index subset and an attribute subset in the divided sequence subset, and respectively generate index features of the index subset and attribute features of the attribute subset, and assign different weight values ​​to the index features and the attribute features, and perform feature weighting on the index features and the attribute features based on the assigned weight values ​​to obtain weighted features corresponding to the time series data; The acquisition area acquisition unit is further used to match the weighted features with representative features of each operating condition cluster, and use the operating condition cluster with the highest matching degree as the operating condition cluster where the vehicle to be loaded and unloaded is located, and acquire a local image acquisition area represented by the operating condition cluster; A local image determination unit, configured to obtain a reference video frame of the vehicle to be loaded or unloaded when the vehicle to be loaded or unloaded arrives at a designated position, and to crop a local image from the reference video frame based on the local image acquisition area; A model selection unit, used to identify whether a specific scene identifier exists in the partial image, and select a detection model that matches the recognition result, wherein the detection model is an RGB detection model or an IR detection model; The hidden danger alarm unit is used to detect the local image using the selected detection model to determine the operation process hidden dangers existing in the local image and generate alarm information adapted to the operation process hidden dangers.

10. A gas station loading and unloading vehicle operation process hidden danger detection and identification system, characterized in that: The system comprises a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 8 is implemented.

Images