A refueling process monitoring method and device, electronic equipment and storage medium

Abstract

The application relates to a refueling process monitoring method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring to-be-processed videos for multiple refueling processes, wherein the to-be-processed videos comprise refueling personnel, refueling vehicles and refueling equipment; for each refueling process in a vehicle leading-in process, a bowing greeting process, an oil meter zeroing process and a vehicle leading-out process, a target classification result of a current refueling process in the to-be-processed videos is identified; and for each refueling process in a driving position going process, an oil cover opening process, a gun refueling process and an oil cover closing process, a target detection result of the current refueling process in the to-be-processed videos is identified. Through the method, based on eight standard refueling processes corresponding to the eight-step refueling method, whether each refueling process in the to-be-processed videos meets the requirements of the standard refueling process is identified, so that the service process of the refueling vehicle is monitored, and the refueling process can be monitored automatically and effectively.

Description

Technical Field

[0001] This invention relates to the fields of machine learning and refueling management technology. Specifically, this invention relates to a refueling process monitoring method, device, electronic device, and storage medium. Background Technology

[0002] When gas stations were first established, security cameras were installed in various areas, such as the cashier area, unloading area, and refueling islands, to monitor the station's safe operations. However, this traditional video surveillance technology cannot intelligently identify the content of the monitored footage; all information obtained from the video must be manually reviewed by staff. Faced with massive sequences of video images, staff cannot maintain prolonged and highly focused observation to handle abnormal events. Furthermore, investing significant manpower in monitoring low-probability events is not a desirable practice for any organization. Therefore, achieving intelligent video surveillance has become an inevitable trend in the internet age.

[0003] With the development of artificial intelligence, intelligent video surveillance technology has seen many successful applications. Through deep learning-based image processing algorithms, it can quickly identify various objects in surveillance images to determine anomalies and issue alarms or trigger other actions in the fastest and most efficient way. This enables a fully automated, 24 / 7, real-time intelligent monitoring system that effectively provides pre-event warnings, in-event handling, and timely post-event evidence collection. Many key security measures at gas stations can now be automated and monitored using intelligent video surveillance technology.

[0004] In the current technology, customers are increasingly valuing the service quality they receive at gas stations, and gas stations also need to continuously improve the service level of their employees. However, there is currently a lack of effective automated monitoring and service evaluation mechanisms for standardized refueling procedures. As a result, employees often fail to complete all the refueling steps stipulated by the gas station, significantly compromising service quality and posing a risk of operational violations. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method, device, electronic device and storage medium for monitoring the refueling process, in order to solve at least one of the above-mentioned technical problems.

[0006] Firstly, the technical solution of the present invention to solve the above-mentioned technical problems is as follows: a refueling process monitoring method, the method comprising:

[0007] Acquire videos for multiple refueling processes, including refueling personnel, refueling vehicles, and refueling equipment. The multiple refueling processes are executed sequentially according to the refueling time sequence. The multiple refueling processes include the process of leading the vehicle into the station, the process of going to the driver's seat, the process of bowing and greeting, the process of resetting the fuel gauge, the process of opening the fuel tank cap, the process of picking up the refueling nozzle, the process of closing the fuel tank cap, and the process of leading the vehicle out of the station.

[0008] For each refueling process in the vehicle-in-station process, the bowing and greeting process, the fuel gauge resetting process, and the vehicle-out-of-station process, identify the target classification result of the current refueling process in the video to be processed. The target classification result is either that the current refueling process is the corresponding standard refueling process, or that the current refueling process is not the corresponding standard refueling process. Multiple standard refueling processes include the vehicle-in-station standard process, the process of proceeding to the driver's seat standard process, the bowing and greeting standard process, the fuel gauge resetting standard process, the process of opening the fuel tank cap standard process, the process of picking up the refueling nozzle standard process, the process of closing the fuel tank cap standard process, and the vehicle-out-of-station standard process.

[0009] For each refueling process in the process of going to the driver's seat, opening the fuel tank cap, picking up the refueling nozzle, and closing the fuel tank cap, identify the target detection result of the current refueling process in the video to be processed. The target detection result is either the current refueling process matches the corresponding standard refueling process or the current refueling process does not match the corresponding standard refueling process.

[0010] The beneficial effects of this invention are: In this application, based on the eight standardized refueling processes corresponding to the eight-step refueling method, by identifying whether each refueling process in the video to be processed meets the requirements of the standardized refueling process, the service process of refueling personnel on refueling vehicles can be monitored automatically and effectively.

[0011] Based on the above technical solution, the present invention can be further improved as follows.

[0012] Furthermore, for each refueling process among the gestures of guiding the vehicle into the station, bowing in greeting, resetting the fuel gauge, and guiding the vehicle out of the station, the target classification result of the current refueling process identified in the video to be processed includes:

[0013] Identify refueling personnel in the video to be processed;

[0014] Identify the key points of the refueling personnel in the video to be processed;

[0015] Based on the key point location of the refueling personnel, the first classification result of the current refueling action of the refueling personnel is determined by the preset first classification model;

[0016] Based on the key point location of the refueling personnel, the second classification result of the current refueling action of the refueling personnel is determined by the preset second classification model;

[0017] Based on the first and second classification results, determine the target classification result for the current refueling process corresponding to the refueling personnel.

[0018] The beneficial effect of adopting the above-mentioned further scheme is that, by using the two classification results obtained from two different classification models, the target classification result of the current refueling process can be determined more accurately.

[0019] Furthermore, for the process of heading to the driver's seat, the target detection results of the current refueling process in the video to be processed are identified, including:

[0020] For the target image in the video to be processed, identify the position of the rearview mirror of the refueling vehicle stopped in the parking area and the position of the refueling staff. The target image is the image corresponding to when the refueling vehicle enters the parking area.

[0021] Determine the distance between the rearview mirror and the refueling personnel in the target image based on the position of the rearview mirror and the position of the employee;

[0022] If the distance in consecutive frames following the target image in the video to be processed meets the first condition, then it is determined that the current refueling process matches the standard process for going to the driving position. If the distance does not meet the first condition, then it is determined that the current refueling process does not match the standard process for going to the driving position. The first condition is that the distance is constantly decreasing and the distance is less than a set threshold when the distance does not change.

[0023] For each refueling process within the opening and closing of the fuel tank, identify the target detection result of the current refueling process in the video to be processed, including:

[0024] Identify the open / closed state of the fuel tank cap of the refueling vehicle in the video to be processed, indicating whether the open / closed state changes from closed to open or from open to closed.

[0025] Based on the opening and closing status and the execution order of the current refueling process in multiple standardized refueling processes, determine the target detection result of the current refueling process in the video to be processed;

[0026] Refueling equipment includes a refueling nozzle and a refueling hose. For the refueling process, the target detection results in the current refueling process within the video to be processed are identified, including:

[0027] Identify whether the video to be processed includes actions of holding a refueling nozzle and actions of holding a refueling hose;

[0028] If the video to be processed includes actions of holding the refueling nozzle and holding the refueling hose, then it is determined that the current refueling process in the video to be processed matches the standard refueling process.

[0029] If the video to be processed does not simultaneously include the actions of holding the refueling nozzle and holding the refueling hose, then it is determined that the current refueling process in the video to be processed does not match the standard refueling process.

[0030] The advantage of adopting the above-mentioned further solution is that different methods can be used to monitor different refueling processes, so as to ensure that the monitoring results of each refueling process are relatively accurate.

[0031] Furthermore, if there is a first target refueling process in each target classification result that is not a standard refueling process, and / or, there is a second target refueling process in each target detection result that does not match the standard refueling process;

[0032] Generate alarm information and send it to the terminal devices of relevant personnel;

[0033] If the second refueling process includes a fuel tank cap closing procedure, an alarm message is generated and sent to the terminal devices of relevant personnel, including:

[0034] If a change in the location of a refueling vehicle in the video to be processed is detected, an alarm message is generated and sent to the terminal device of the relevant personnel.

[0035] The beneficial effect of adopting the above-mentioned further solution is that when there are processes in each refueling process that do not conform to multiple standard refueling processes (first target refueling process and second target refueling process), alarm information can be generated in a timely manner and sent to the terminal devices of relevant personnel. In addition, during the monitoring of the fuel tank cap closing process, it is necessary to consider not only whether the refueling process conforms to the standard refueling process, but also the positional changes of the refueling vehicle, so as to more accurately monitor the fuel tank cap closing process.

[0036] Furthermore, the method also includes:

[0037] Based on the target detection results and target classification results of each refueling process, the service score of the refueling staff after completing a refueling is determined.

[0038] The beneficial effect of adopting the above-mentioned further solutions is that service ratings can be used to standardize the management of refueling personnel.

[0039] Furthermore, the method also includes:

[0040] Determine the number of customers served by refueling personnel in the pending video.

[0041] The beneficial effect of adopting the above-mentioned further solution is that, based on the number of customers served by refueling personnel in the pending video, the attendance management of refueling personnel can be further strengthened.

[0042] Furthermore, the first classification model mentioned above is a model trained based on a long short-term memory network, and the second classification model mentioned above is a model trained based on an improved shufflenetv2 classification network. The improved shufflenetv2 classification network is formed by replacing the 3x3DWConv in the original shufflenetv2 classification network with 5x5DWConv, changing the stride from 1 to 2, and removing DWConv to obtain 1x1Conv.

[0043] The beneficial effect of adopting the above-mentioned further scheme is that replacing the 3x3DWConv in the original shufflenetv2 classification with 5x5DWConv and changing the step size from 1 to 2 can ensure that the accuracy is improved while maintaining the same resolution as before, and the number of parameters will not increase significantly. In addition, removing DWConv and then adding 1x1Conv can further reduce the amount of computation.

[0044] Secondly, to solve the above-mentioned technical problems, the present invention also provides a refueling process monitoring device, the device comprising:

[0045] The video acquisition module is used to acquire videos to be processed for multiple refueling processes. The videos to be processed include refueling personnel, refueling vehicles and refueling equipment. The multiple refueling processes are executed sequentially according to the refueling time sequence. The multiple refueling processes include the process of leading the vehicle into the station, the process of going to the driver's seat, the process of bowing and greeting, the process of resetting the fuel gauge, the process of opening the fuel tank cap, the process of picking up the refueling nozzle, the process of closing the fuel tank cap and the process of leading the vehicle out of the station.

[0046] The first monitoring module is used to identify the target classification result of the current refueling process in the video to be processed for each refueling process in the process of guiding the vehicle into the station, the process of bowing and greeting, the process of resetting the fuel gauge, and the process of guiding the vehicle out of the station. The target classification result is either that the current refueling process is the corresponding standard refueling process, or that the current refueling process is not the corresponding standard refueling process. Multiple standard refueling processes include the standard process of guiding the vehicle into the station, the standard process of going to the driver's position, the standard process of bowing and greeting, the standard process of resetting the fuel gauge, the standard process of opening the fuel tank cap, the standard process of picking up the refueling nozzle, the standard process of closing the fuel tank cap, and the standard process of guiding the vehicle out of the station.

[0047] The second monitoring module is used to identify the target detection result of the current refueling process in the video to be processed for each refueling process in the process of going to the driver's position, opening the fuel tank cap, picking up the refueling nozzle, and closing the fuel tank cap. The target detection result is whether the current refueling process matches the corresponding standard refueling process or does not match the corresponding standard refueling process.

[0048] Thirdly, in order to solve the above-mentioned technical problems, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the refueling process monitoring method of the present application.

[0049] Fourthly, in order to solve the above-mentioned technical problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the refueling process monitoring method of the present application.

[0050] Additional aspects and advantages of this application will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of this application. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below.

[0052] Figure 1 A flowchart illustrating a refueling process monitoring method according to an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of a human posture for guiding a vehicle into a station, provided as an embodiment of the present invention.

[0054] Figure 3 A schematic diagram of a human posture for bowing in greeting, provided as an embodiment of the present invention;

[0055] Figure 4 A schematic diagram of a human posture for a zeroing gesture provided in one embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram of a human posture for guiding a vehicle out of a station, provided as an embodiment of the present invention.

[0057] Figure 6 A schematic diagram of the structure of a primitive shufflenetv2 network provided in one embodiment of the present invention;

[0058] Figure 7 A schematic diagram of an improved shufflenetv2 network provided in one embodiment of the present invention;

[0059] Figure 8 This is a schematic diagram of a Yolov5 network structure provided in one embodiment of the present invention;

[0060] Figure 9 A flowchart illustrating another refueling process monitoring method provided in an embodiment of the present invention;

[0061] Figure 10 This is a schematic diagram of a refueling process monitoring device provided in one embodiment of the present invention;

[0062] Figure 11 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0063] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0064] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0065] The solution provided in this invention can be applied to any application scenario that requires monitoring the refueling process. The solution provided in this invention can be executed by any electronic device, such as a user's terminal device. This terminal device can be any device that can install applications and monitor the refueling process through those applications, including at least one of the following: smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart TVs, and smart in-vehicle devices.

[0066] This invention provides a possible implementation, such as... Figure 1 As shown, a flowchart of a refueling process monitoring method is provided. This method can be executed by any electronic device, such as a terminal device, or jointly executed by a terminal device and a server. For ease of description, the method provided in this embodiment will be described below using a terminal device as the execution subject as an example. Figure 1 The flowchart shown indicates that the method may include the following steps:

[0067] Step S110: Obtain videos to be processed for multiple refueling processes. The videos to be processed include refueling personnel, refueling vehicles and refueling equipment. Multiple refueling processes are executed sequentially according to the refueling time sequence. Multiple refueling processes include the process of leading the vehicle into the station, the process of going to the driver's seat, the process of bowing and greeting, the process of resetting the fuel gauge, the process of opening the fuel tank cap, the process of picking up the nozzle to refuel, the process of closing the fuel tank cap and the process of leading the vehicle out of the station.

[0068] Step S120: For each refueling process in the vehicle entry process, the bowing and greeting process, the fuel gauge reset process, and the vehicle exit process, identify the target classification result of the current refueling process in the video to be processed. The target classification result is either that the current refueling process is the corresponding standard refueling process, or that the current refueling process is not the corresponding standard refueling process. Multiple standard refueling processes include the vehicle entry standard process, the process of proceeding to the driver's seat standard process, the bowing and greeting standard process, the fuel gauge reset standard process, the process of opening the fuel tank cap standard process, the process of picking up the refueling nozzle standard process, the process of closing the fuel tank cap standard process, and the vehicle exit process.

[0069] Step S130: For each refueling process in the process of going to the driver's seat, opening the fuel tank cap, picking up the refueling nozzle, and closing the fuel tank cap, identify the target detection result of the current refueling process in the video to be processed. The target detection result is whether the current refueling process matches the corresponding standard refueling process or does not match the corresponding standard refueling process.

[0070] The method of this invention, based on the eight standardized refueling procedures corresponding to the eight-step refueling method, identifies whether each refueling procedure in the video to be processed meets the requirements of the standardized refueling procedure, thereby enabling refueling personnel to monitor the service process of refueling vehicles. This allows for automated and effective monitoring of the refueling process.

[0071] The following specific embodiments further illustrate the solution of the present invention. In these embodiments, the refueling process monitoring method may include the following steps:

[0072] Step S110: Obtain videos to be processed for multiple refueling processes. The videos to be processed include refueling personnel, refueling vehicles, and refueling equipment. The multiple refueling processes are executed sequentially according to the refueling time sequence. The multiple refueling processes include the process of leading the vehicle into the station, the process of going to the driver's seat, the process of bowing and greeting, the process of resetting the fuel gauge, the process of opening the fuel tank cap, the process of picking up the refueling nozzle, the process of closing the fuel tank cap, and the process of leading the vehicle out of the station.

[0073] A complete refueling process can include eight steps: leading the vehicle into the station, proceeding to the driver's seat, bowing and greeting the driver, resetting the fuel gauge, opening the fuel tank cap, picking up the refueling nozzle, closing the fuel tank cap, and leading the vehicle out of the station.

[0074] The video to be processed can be real-time video captured by cameras at the gas station, or video footage captured after a vehicle is detected entering the refueling area. "Refueling personnel" refers to gas station staff, "refueling vehicles" refers to vehicles preparing for refueling, and "refueling equipment" refers to the equipment used for refueling.

[0075] Step S120: For each refueling process in the vehicle entry process, the bowing and greeting process, the fuel gauge reset process, and the vehicle exit process, identify the target classification result of the current refueling process in the video to be processed. The target classification result is either that the current refueling process is the corresponding standard refueling process, or that the current refueling process is not the corresponding standard refueling process. Multiple standard refueling processes include the vehicle entry standard process, the process of proceeding to the driver's seat standard process, the bowing and greeting standard process, the fuel gauge reset standard process, the process of opening the fuel tank cap standard process, the process of lifting the nozzle for refueling standard process, the process of closing the fuel tank cap standard process, and the vehicle exit process.

[0076] Among these, each of the multiple standardized refueling procedures refers to the standardized operating procedure for refueling. See [link / reference]. Figures 2 to 5 The standardized procedures for guiding a vehicle into the station can be categorized into four types: guiding the vehicle into the station (using standardized hand gestures), bowing and greeting (using standardized bowing and greeting postures), resetting the fuel gauge (using standardized resetting hand gestures), and guiding the vehicle out of the station (using standardized hand gestures). Each refueling process is monitored based on the target classification results. The current refueling process can be any one of the following: guiding the vehicle into the station, bowing and greeting, resetting the fuel gauge, or guiding the vehicle out of the station. Figures 2 to 5 Each number in the table represents a different key point, for a total of 16 nodes.

[0077] The current refueling process is the corresponding standard refueling process, meaning that the current refueling process matches the corresponding standard refueling process, that is, the current refueling process is a standard process. The current refueling process is not the corresponding standard refueling process, meaning that the current refueling process does not match the corresponding standard refueling process, that is, the current refueling process is not a standard process.

[0078] Optionally, for each refueling process among the gestures of guiding the vehicle into the station, bowing in greeting, resetting the fuel gauge, and guiding the vehicle out of the station, identify the target classification result of the current refueling process in the video to be processed, including:

[0079] Identify refueling personnel in the video to be processed;

[0080] Identify the key points of the refueling personnel in the video to be processed;

[0081] Based on the key point location of the refueling personnel, the first classification result of the current refueling action of the refueling personnel is determined by the preset first classification model;

[0082] Based on the key point location of the refueling personnel, the second classification result of the current refueling action of the refueling personnel is determined by the preset second classification model;

[0083] Based on the first and second classification results, determine the target classification result for the current refueling process corresponding to the refueling personnel.

[0084] The key points of the refueling personnel in the video to be processed can be identified by methods in the prior art, which will not be elaborated in the present invention.

[0085] Optionally, one possible way to determine the target classification result for the current refueling process corresponding to the refueling personnel based on the first classification result and the second classification result is to perform weighted processing based on the first classification result, the first weight, the second classification result, and the second weight to obtain the target classification result. The first weight and the second weight can be preset. For different refueling processes, the first weight and the second weight can be different or the same.

[0086] Optionally, the aforementioned first and second classification results may include two parts: an action recognition result and a confidence score. The action recognition result indicates that the action category corresponding to the current refueling process is the same as the action category corresponding to the standard refueling process. The confidence score represents the probability that the current action is a standard action corresponding to the standard refueling process. The first classification result includes a first confidence score and a second confidence score. When the action category corresponding to the current refueling process is the same as the action category corresponding to the standard refueling process, and the first confidence score is greater than a first preset value, and the second confidence score is greater than a second preset value, it indicates that the current refueling process matches the corresponding standard refueling process. The first and second preset values ​​can be the same or different; for example, both can be 0.7.

[0087] Optionally, the first classification model is a model trained based on a long short-term memory network, and the second classification model is a model trained based on an improved shufflenetv2 classification network. The improved shufflenetv2 classification network is formed by replacing the 3x3DWConv in the original shufflenetv2 classification network with 5x5DWConv, changing the stride from 1 to 2, and removing DWConv to obtain 1x1Conv.

[0088] The first classification model described above mainly performs text processing on the input image data and uses the advantages of long short-term memory networks to classify the text. This invention is used to identify which step in the refueling process an employee's action in the monitoring screen belongs to, that is, to identify the key point position of the refueling personnel in the video to be processed.

[0089] The first classification model described above can be trained through the following steps:

[0090] S1, Preparation of the keypoint dataset

[0091] Based on the manually labeled results of 20,000 training images and 2,000 test images (marking the positions of each joint point under different postures) of the proposed posture estimation algorithm, the manually labeled results are normalized. Then, the image name, the normalized label result, and the true category are written into a CSV file. The true categories are: guiding the vehicle into the station, bowing to greet the driver, resetting the fuel gauge, and guiding the vehicle out of the station. For each image, the image name indicates which refueling process the image corresponds to.

[0092] S2, using a Long Short-Term Memory (LSTM) network for classification.

[0093] The Keras library is used to call the LSTM network to classify the key text data. That is, the training data obtained in S1 is input into the LSTM network to classify each image and obtain the predicted category of each image.

[0094] S3, Network Training

[0095] The loss function value is determined based on the predicted category and the true category of each image. When the loss function value meets the preset training termination condition, the LSTM network that meets the training termination condition is used as the first classification model.

[0096] During the training of the first classification model, the model parameters can be continuously adjusted based on the loss function value and test data to obtain the best model, that is, the model with the highest accuracy.

[0097] The aforementioned second classification model is also used to identify which step in the refueling process an employee's action in the monitoring footage belongs to, that is, to identify the key points of the refueling personnel in the video to be processed.

[0098] The above-mentioned second classification model can be trained through the following steps:

[0099] S1, Preparation of different action datasets.

[0100] The training data is the same as the data required by the attitude estimation algorithm (the training data corresponding to the first classification model mentioned above). The data for guiding the vehicle into the station, bowing and greeting, resetting the fuel gauge, guiding the vehicle out of the station, and other actions are stored in five separate folders.

[0101] S2, a neural network algorithm for multi-class image classification.

[0102] The basic network architecture uses shufflenetv2, i.e., the original shufflenetv2 network (its network structure is as follows). Figure 6As shown in the diagram, since most of the computational load in this architecture is in the 1x1 Conv operation, this application replaces all 3x3 DWConv operations with 5x5 DWConv operations, and changes the padding from 1 to 2. This ensures improved accuracy while maintaining the same resolution as before, and the number of parameters does not increase significantly. Furthermore, to further reduce computational load, the 1x1 Conv operation after DWConv can be removed. This does not affect the fusion of channel information and can reduce the number of parameters by approximately 20%-30%, with little impact on accuracy. Overall, the number of parameters is reduced by 15%-25%, and the accuracy is improved by 1%-2%. The specific network architecture diagram is shown below. Figure 7 As shown, the improved ShuffleNetv2 network is used to classify each image in the training data to obtain the predicted category of each image.

[0103] S3, Network Training.

[0104] The loss function value is determined based on the predicted category and the true category of each image. When the loss function value meets the preset training termination condition, the improved ShuffleNetv2 model that meets the training termination condition is used as the second classification model.

[0105] During the training of the second classification model described above, before inputting each image into the improved ShuffleNetv2 network, operations such as image synthesis, mirroring, and Gaussian noise addition can be performed on the training data to enrich it. Normalization of the training data can also be performed first to accelerate network convergence.

[0106] During the training of the second classification model, the model parameters can be continuously adjusted based on the loss function value and test data to obtain the best model, that is, the model with the highest accuracy.

[0107] Step S130: For each refueling process in the process of going to the driver's seat, opening the fuel tank cap, picking up the refueling nozzle, and closing the fuel tank cap, identify the target detection result of the current refueling process in the video to be processed. The target detection result is whether the current refueling process matches the corresponding standard refueling process or does not match the corresponding standard refueling process.

[0108] It should be noted that the execution order of steps S120 and S130 is not limited. Step S120 can be executed first, or step S130 can be executed first.

[0109] After obtaining the classification results and detection results of each target, multiple refueling processes can be monitored and managed based on these results. For example, the refueling process of refueling staff can be scored, safety warnings can be issued, and the refueling process can be standardized.

[0110] Specifically, for the process of moving to the driver's seat, the target detection results of the current refueling process in the video to be processed are identified, including:

[0111] For the target image in the video to be processed, identify the position of the rearview mirror of the refueling vehicle stopped in the parking area and the position of the refueling staff. The target image is the image corresponding to when the refueling vehicle enters the parking area. Starting the subsequent recognition and processing from the target image can save computation.

[0112] Based on the position of the rearview mirror and the employee's position, determine the distance between the rearview mirror and the refueling personnel in the target image; this distance can be the center distance between the rearview mirror and the refueling personnel.

[0113] If the distance in consecutive frames following the target image in the video to be processed meets the first condition, then the current refueling process is determined to match the standard process for proceeding to the driving position. If the distance does not meet the first condition, then the current refueling process is determined to not match the standard process for proceeding to the driving position. The first condition is that the distance is continuously decreasing, and the distance when it does not change is less than a set threshold. When the first condition is met, it indicates that the refueling personnel have arrived at the designated location in the refueling area, where they can use the refueling equipment to refuel the vehicle.

[0114] For each refueling process within the opening and closing of the fuel tank, identify the target detection result of the current refueling process in the video to be processed, including:

[0115] Identify the opening and closing status of the fuel tank cap of the refueling vehicle in the video to be processed, which is either changing from closed to open or from open to closed; determine the target detection result of the current refueling process in the video to be processed based on the opening and closing status and the execution order of the current refueling process in multiple standard refueling processes;

[0116] Specifically, for the fuel tank opening process within the refueling procedure, the corresponding opening / closing state should change from closed to open; conversely, for the fuel tank closing process, the corresponding opening / closing state should change from open to closed. Therefore, determining the target detection result requires considering the execution order of the current refueling procedure within multiple standard refueling procedures—that is, whether the current refueling procedure is a fuel tank opening or closing process. The target detection result indicates whether the fuel tank opening process conforms to the standard fuel tank closing process, and vice versa.

[0117] The aforementioned refueling equipment includes a refueling nozzle and a refueling hose. For the nozzle-lifting refueling process, the target detection results in the current refueling process within the video to be processed are identified, including:

[0118] The system identifies whether the video to be processed includes actions of holding the refueling nozzle and holding the refueling hose. If the video includes both actions, it determines that the current refueling process in the video matches the standard refueling procedure. If the video does not include both actions, it determines that the current refueling process in the video does not match the standard refueling procedure.

[0119] The ability to identify whether the video being processed includes actions such as holding a refueling gun or a refueling hose can be achieved through a pre-defined multi-class object detection model, which will be described later.

[0120] Optionally, the method further includes:

[0121] If any of the target classification results contains a first target refueling process that is not part of the standard refueling process, and / or if any of the target detection results contain a second target refueling process that does not match the standard refueling process, an alarm message is generated and sent to the terminal devices of relevant personnel.

[0122] If the refueling process for the second target includes the process of closing the fuel tank cap, then the terminal device that generates alarm information and sends it to relevant personnel includes: if it is detected that the position of the refueling vehicle in the video to be processed has changed, then it generates alarm information and sends it to the terminal device of relevant personnel.

[0123] Optionally, the method further includes:

[0124] Determine the number of customers served by refueling personnel in the video to be processed. The number of customers served can be a statistical measure over a period of time, such as one day or one week.

[0125] Optionally, the method further includes:

[0126] Based on the target detection results and target classification results of each refueling process, the service score of the refueling staff after completing a refueling is determined.

[0127] In this system, gas station employees can rate and score the service after each refueling session, indicating any steps that were not completed. The rating can be based on the number of customers served and the number of actions performed by each customer.

[0128] Optionally, the license plates of refueling vehicles in the video to be processed can also be identified and recorded.

[0129] It should be noted that the above multi-class object detection model is used to identify refueling staff, refueling vehicles, refueling equipment, car rearview mirrors, license plates, refueling nozzles in different postures, fuel tank cap open state, fuel tank cap closed state, handheld refueling nozzles, handheld refueling hoses, etc. in the video to be processed.

[0130] The above multi-class object detection model can be trained in the following way:

[0131] S1, Preparation of datasets for nine categories of objects.

[0132] We prepared approximately 20,000 training images and 2,000 test images by collecting copyright-free online materials and manually taking photos. We then expanded the dataset using data augmentation techniques. The dataset includes images of fuel nozzles, fuel tank caps (open and closed), employees, car rearview mirrors, vehicles, license plates, handheld fuel nozzles, and handheld fuel hoses in different scenes, sizes, and poses. These images were manually labeled using a first bounding box (ground truth box) to categorize the different objects. During the labeling process, the LabelMe tool was used to mark the approximate rectangular outlines of the different objects.

[0133] S2, Build a neural network for a multi-class object detection algorithm. The network architecture uses Yolov5, such as... Figure 8 As shown.

[0134] S3, Define the loss function.

[0135] In Yolov5, GIOU Loss is used as the loss function for the bounding box.

[0136] The prediction results for each image obtained from the Yolov5 network can be marked in the image using the second bounding box (prediction box).

[0137] Then calculate the areas A and B of the minimum closure regions of the two boxes, and then calculate the IoU (simply put: the area of ​​the smallest box that contains both the predicted box and the ground truth box), as shown in the following formula:

[0138]

[0139] The intersection-over-union (IoU) ratio is used to characterize the difference between the predicted bounding box and the ground truth bounding box.

[0140] Next, calculate the proportion of the region in the closure region that does not belong to either of the two boxes to the closure region (A∪B), and finally subtract this proportion from the IoU to obtain the GIoU.

[0141]

[0142] GIoU Loss = 1 - GIoU

[0143] Where C represents the region in the closure area that does not belong to either of the two boxes, and GIoU Loss is the loss function value of the Yolov5 network.

[0144] S4, Network Training.

[0145] When the loss function value meets the preset training termination condition, the Yolov5 network that meets the training termination condition is used as the multi-class object detection model.

[0146] During the training of the multi-class object detection model described above, before inputting each image into the Yolov5 network, operations such as image synthesis, mirroring, and Gaussian noise addition can be performed on the training data to enrich it. Normalization of the training data can also be performed first to accelerate network convergence.

[0147] During the training process of a multi-class object detection model, the model parameters can be continuously adjusted based on the loss function value and test data to obtain the optimal model, i.e., the model with the best accuracy.

[0148] It should be noted that, Figures 6 to 8 All network architectures in this paper are existing technology architectures. The layers involved in each network architecture are as follows: conv: convolutional layer, BN: batch normalization layer, ReLU: ReLU function, concat: concatenation layer, channel split: channel splitting layer, DWconv: grouped convolutional layer, padding: padding layer, channel shuffle: channel shuffle layer, BottleneckCSP: bottleneck CSP layer, SPP: spatial pyramid pooling layer.

[0149] To better illustrate and understand the principle of the method provided by this invention, the following description uses an optional specific embodiment to illustrate the solution of this invention. It should be noted that the specific implementation of each step in this specific embodiment should not be construed as a limitation of the solution of this invention. Other implementations that can be conceived by those skilled in the art based on the principle of the solution provided by this invention should also be considered within the scope of protection of this invention.

[0150] See Figure 9 The diagram illustrating the refueling process monitoring method includes the following parts:

[0151] 1) Refueling begins. Vehicle detection algorithms are used to detect refueling vehicles. When a vehicle appears in the refueling area, intelligent monitoring of the refueling process is activated, and refueling step 1 is executed. The vehicle-into-station gesture detection algorithm determines whether the employee has completed the vehicle-into-station gesture action.

[0152] 2) Refueling preparation. Refueling steps 2, 3, and 4 are executed in sequence. The three actions of the staff—going to the driver's seat, bowing and greeting, and raising their hand to indicate that the fuel gauge is zero—are detected by the algorithm of moving to the driver's seat, the algorithm of bowing and greeting, and the algorithm of raising their hand to indicate that the fuel gauge is zero. The analysis is then performed to determine whether refueling steps 2, 3, and 4 have been completed.

[0153] 3) Refueling process. Execute refueling steps 5 and 6, and use a specific object detection algorithm (a pre-established multi-class object detection model) to detect the open fuel tank cap, handheld refueling nozzle, and handheld refueling hose to analyze whether the refueling process is standardized.

[0154] 4) Refueling complete. Execute refueling step 7. Use an object detection algorithm (a pre-established multi-class object detection model) to detect the closed fuel tank cap. If the closed fuel tank cap is detected after the open fuel tank cap, then refueling is complete.

[0155] 5) Refueling Completion. Execute refueling step 8, using a vehicle departure gesture detection algorithm to determine if the employee has completed the vehicle departure gesture. Finally, a vehicle detection algorithm detects the refueling vehicle; when the vehicle leaves the refueling area, the refueling service is considered complete.

[0156] 6) End. The video surveillance of the refueling process is intelligently captured, and the employee's service is scored and evaluated based on the completion of the above 8 steps, informing them of any steps that were not completed.

[0157] The specific identification process for the above eight refueling procedures is as follows:

[0158] S1, gesture detection for guiding vehicles into the station.

[0159] 1) Use a multi-class object detection model to locate employees (gas station employees) and bind the gas station number to the employee to determine employee information, which is convenient for inclusion in the assessment, that is, to determine how many customers the gas station employee served and how many vehicles were refueled;

[0160] 2) Using a pre-defined human pose estimation algorithm, the positions of the key points of the refueling staff are located, and the human pose of the staff guiding the vehicle into the station is as follows: Figure 2 As shown;

[0161] 3) The first classification module based on Long Short-Term Memory (LSTM) network is used to classify the detected key points into multiple categories and output the categories and confidence scores (first classification results);

[0162] 4) The location of the key points detected in (2) is directly classified by a second classification model based on the improved ShuffleNetv2 classification network, and the category and confidence (second classification result) are output.

[0163] 5) Judge the classification results of (3) and (4). If the two results are the same and the classification result is the same, and both are the vehicle entry gesture, and both confidence scores are greater than 0.7, then the vehicle entry gesture is detected. Otherwise, the vehicle entry gesture is not detected.

[0164] S2, proceed to the driver's seat for inspection.

[0165] 1) First, the refueling and parking area is deployed. Multi-class object detection algorithm is used to identify vehicles and license plates. When a vehicle enters the parking area, all subsequent algorithms are executed and the license plate information is recorded.

[0166] 2) Use a multi-class object detection model to locate the employee and the rearview mirror, and record the coordinates of the center points of the detected employee and the rearview mirror;

[0167] 3) Determine whether the distance between two center points in consecutive frames has decreased;

[0168] 4) By continuously judging the distance to decrease until the coordinates of the employee's center point no longer change, it is determined whether the distance between the employee and the rearview mirror has reached the threshold to determine whether the standard has been met, that is, whether the employee has reached the designated position.

[0169] S3, bowing to greet the test.

[0170] 1) Use a multi-class object detection model to locate employees and output target images;

[0171] 2) Based on the target image, the position of the employee's key points is located using a pose estimation algorithm. The employee's posture during a bow greeting is as follows: Figure 3 As shown;

[0172] 3) Use the first classification model to perform multi-classification on the detected key point locations, and output the category and confidence score (first classification result);

[0173] 4) Use the second classification model to classify the detected key points into multiple categories, and output the category and confidence score (second classification result);

[0174] 5) Judge the classification results of (3) and (4). If the two results are the same and the classification result is a bow greeting, and both confidence scores are greater than 0.7, then a bow greeting is detected; otherwise, a bow greeting is not detected.

[0175] S4, fuel gauge zeroing gesture detection.

[0176] 1) Use a multi-class object detection model to locate employees and output target images;

[0177] 2) Based on the target image, the position of the employee's key points is located using a pose estimation algorithm, and the zeroing gesture human posture is as follows: Figure 4 As shown;

[0178] 3) Use the first classification model to perform multi-classification on the detected key point locations, and output the category and confidence score;

[0179] 4) Use the second classification model to directly perform multi-classification on the detected key point positions, and output the category and confidence score;

[0180] 5) Judge the classification results of (3) and (4). If the two results are the same and the classification result is the fuel gauge zeroing gesture, and both confidence scores are greater than 0.7, then the fuel gauge zeroing gesture is detected. Otherwise, the fuel gauge zeroing gesture is not detected.

[0181] S5, open the fuel tank cap for inspection.

[0182] 1) Use a multi-class object detection model to identify the opening and closing status of the fuel tank cap in the previous and next frames. When the fuel tank cap changes from closed to open in the current frame, it meets the standard. Meeting the standard means that the current fuel tank cap opening process conforms to the standard fuel tank cap opening process.

[0183] S6, refueling inspection.

[0184] 1) The handheld refueling nozzle and handheld refueling hose are identified using a multi-class object detection model. If both are detected at the same time, the standard is met. Meeting the standard means that the current nozzle-lifting refueling process conforms to the standard nozzle-lifting refueling process.

[0185] S7, check the fuel tank cap.

[0186] The opening and closing status of the fuel tank cap in the previous and next frames is identified by a multi-class object detection algorithm. When the fuel tank cap changes from open to closed in the current frame, the standard is met. Meeting the standard means that the current fuel tank cap closing process conforms to the standard fuel tank cap closing process.

[0187] In addition, if the multi-class object detection algorithm detects that the vehicle position changes in consecutive frames but the fuel tank cap is not closed, an alarm will be triggered.

[0188] S8, gesture detection for train departure from station.

[0189] 1) Use a multi-class object detection model to locate employees and output target images;

[0190] 2) Based on the target image, the position of the employee's key points is located using a pose estimation algorithm, and the posture of the human body leading the vehicle out of the station is as follows: Figure 5 As shown;

[0191] 3) Use the first classification model to perform multi-classification on the detected key point locations, and output the category and confidence score (first classification result);

[0192] 4) Use the second classification model to directly perform multi-classification on the detected key point locations, and output the category and confidence score (second classification result);

[0193] 5) Judge the classification results of (3) and (4). If the two results are the same and the classification result is the same, and both are the vehicle departure gesture, and both confidence scores are greater than 0.7, then the vehicle departure gesture is detected. Otherwise, the vehicle departure gesture is not detected.

[0194] 6) When the vehicle leaves the protected area, the algorithm stops and the employee scores and evaluates the service provided by the vehicle.

[0195] Based on and Figure 1 Based on the same principle as the method shown, this embodiment of the invention also provides a refueling process monitoring device 20, such as... Figure 10 As shown, the refueling process monitoring device 20 may include a video acquisition module 210, a first monitoring module 220, and a second monitoring module 230, wherein:

[0196] The video acquisition module 210 is used to acquire videos to be processed for multiple refueling processes. The videos to be processed include refueling personnel, refueling vehicles and refueling equipment. The multiple refueling processes are executed sequentially according to the refueling time sequence. The multiple refueling processes include the process of leading the vehicle into the station, the process of going to the driver's seat, the process of bowing and greeting, the process of resetting the fuel gauge, the process of opening the fuel tank cap, the process of picking up the refueling nozzle, the process of closing the fuel tank cap and the process of leading the vehicle out of the station.

[0197] The first monitoring module 220 is used to identify the target classification result of the current refueling process in the video to be processed for each refueling process in the process of guiding the vehicle into the station, the process of bowing and greeting, the process of resetting the fuel gauge, and the process of guiding the vehicle out of the station. The target classification result is that the current refueling process is the corresponding standard refueling process, or that the current refueling process is not the corresponding standard refueling process. Multiple standard refueling processes include the standard process of guiding the vehicle into the station, the standard process of going to the driver's position, the standard process of bowing and greeting, the standard process of resetting the fuel gauge, the standard process of opening the fuel tank cap, the standard process of picking up the refueling nozzle, the standard process of closing the fuel tank cap, and the standard process of guiding the vehicle out of the station.

[0198] The second monitoring module 230 is used to identify the target detection result of the current refueling process in the video to be processed for each refueling process in the process of going to the driver's position, opening the fuel tank cap, picking up the refueling nozzle, and closing the fuel tank cap. The target detection result is whether the current refueling process matches the corresponding standard refueling process or does not match the corresponding standard refueling process.

[0199] Optionally, for each refueling process among the gestures of guiding the vehicle into the station, bowing in greeting, resetting the fuel gauge, and guiding the vehicle out of the station, the first monitoring module 220, when identifying the target classification result of the current refueling process in the video to be processed, is specifically used for:

[0200] Identify refueling personnel in the video to be processed;

[0201] Identify the key points of the refueling personnel in the video to be processed;

[0202] Based on the key point location of the refueling personnel, the first classification result of the current refueling action of the refueling personnel is determined by the preset first classification model;

[0203] Based on the key point location of the refueling personnel, the second classification result of the current refueling action of the refueling personnel is determined by the preset second classification model;

[0204] Based on the first and second classification results, determine the target classification result for the current refueling process corresponding to the refueling personnel.

[0205] Optionally, for the process of heading to the driver's position, when the second monitoring module 230 identifies the target detection result of the current refueling process in the video to be processed, it is specifically used for:

[0206] For the target image in the video to be processed, identify the position of the rearview mirror of the refueling vehicle stopped in the parking area and the position of the refueling staff. The target image is the image corresponding to when the refueling vehicle enters the parking area.

[0207] Determine the distance between the rearview mirror and the refueling personnel in the target image based on the position of the rearview mirror and the position of the employee;

[0208] If the distance in consecutive frames following the target image in the video to be processed meets the first condition, then it is determined that the current refueling process matches the standard process for going to the driving position. If the distance does not meet the first condition, then it is determined that the current refueling process does not match the standard process for going to the driving position. The first condition is that the distance is constantly decreasing and the distance is less than a set threshold when the distance does not change.

[0209] For each refueling process in the process of opening and closing the fuel tank cap, the second monitoring module 230, when identifying the target detection result of the current refueling process in the video to be processed, is specifically used to: identify the opening and closing state of the fuel tank cap of the refueling vehicle in the video to be processed, wherein the opening and closing state is from closed to open, or from open to closed; and determine the target detection result of the current refueling process in the video to be processed based on the opening and closing state and the execution order of the current refueling process in multiple standard refueling processes.

[0210] The aforementioned refueling equipment includes a refueling nozzle and a refueling hose. For the refueling process of lifting the nozzle, when the second monitoring module 230 identifies the target detection result of the current refueling process in the video to be processed, it is specifically used to: identify whether the video to be processed includes the action of holding the refueling nozzle and the action of holding the refueling hose; if the video to be processed includes the action of holding the refueling nozzle and the action of holding the refueling hose, it is determined that the current refueling process in the video to be processed matches the standard refueling process of lifting the nozzle; if the video to be processed does not include the action of holding the refueling nozzle and the action of holding the refueling hose at the same time, it is determined that the current refueling process in the video to be processed does not match the standard refueling process of lifting the nozzle.

[0211] Optionally, the device may also include:

[0212] The early warning module is used to generate alarm information and send it to the terminal devices of relevant personnel when there is a first target refueling process that is not a standard refueling process in the classification results of each target, and / or a second target refueling process that does not match the standard refueling process in the detection results of each target.

[0213] If the second refueling process includes a fuel tank cap closing procedure, the aforementioned early warning module, when generating alarm information and sending it to the relevant personnel's terminal devices, is specifically used for:

[0214] If a change in the location of a refueling vehicle in the video to be processed is detected, an alarm message is generated and sent to the terminal device of the relevant personnel.

[0215] Optionally, the device may also include:

[0216] The service rating module is used to determine the service rating of the refueling staff after completing a refueling operation, based on the target detection results and target classification results of each refueling process.

[0217] Optionally, the device may also include:

[0218] The quantity determination module is used to determine the number of customers served by refueling personnel in the video to be processed.

[0219] Optionally, the first classification model mentioned above is a model trained based on a long short-term memory network, and the second classification model mentioned above is a model trained based on an improved shufflenetv2 classification network. The improved shufflenetv2 classification network is a 1x1 Conv network obtained by replacing the 3x3 DWConv in the original shufflenetv2 classification network with 5x5 DWConv, changing the stride from 1 to 2, and removing DWConv.

[0220] The refueling process monitoring device of this invention can execute the refueling process monitoring method provided in this invention. The implementation principle is similar. The actions performed by each module and unit in the refueling process monitoring device in each embodiment of this invention correspond to the steps in the refueling process monitoring method in each embodiment of this invention. For detailed functional descriptions of each module of the refueling process monitoring device, please refer to the descriptions in the corresponding refueling process monitoring methods shown above, which will not be repeated here.

[0221] The aforementioned refueling process monitoring device can be a computer program (including program code) running on a computer device, such as an application software; the device can be used to execute the corresponding steps in the method provided in the embodiments of the present invention.

[0222] In some embodiments, the refueling process monitoring device provided in this invention can be implemented using a combination of hardware and software. As an example, the refueling process monitoring device provided in this invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the refueling process monitoring method provided in this invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0223] In other embodiments, the refueling process monitoring device provided in this invention can be implemented in software. Figure 10 A refueling process monitoring device stored in a memory is shown. It can be software in the form of programs and plug-ins, and includes a series of modules, including a video acquisition module 210, a first monitoring module 220 and a second monitoring module 230, for implementing the refueling process monitoring method provided in the embodiments of the present invention.

[0224] The modules described in the embodiments of the present invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0225] Based on the same principles as the methods shown in the embodiments of the present invention, the embodiments of the present invention also provide an electronic device, which may include, but is not limited to: a processor and a memory; the memory for storing computer programs; and the processor for executing the methods shown in any embodiment of the present invention by invoking the computer programs.

[0226] In one alternative embodiment, an electronic device is provided, such as Figure 11 As shown, Figure 11 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0227] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0228] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 11 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0229] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0230] The memory 4003 stores the application code (computer program) for executing the present invention, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.

[0231] Among these, electronic devices can also be terminal devices. Figure 11 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0232] This invention provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments.

[0233] According to another aspect of the present invention, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various embodiments described above.

[0234] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0235] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0236] The computer-readable storage medium provided in this invention can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0237] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0238] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A method for monitoring the refueling process, characterized in that, include: Acquire videos for multiple refueling processes, including refueling personnel, refueling vehicles, and refueling equipment. The multiple refueling processes are executed sequentially according to the refueling time sequence. The multiple refueling processes include the process of leading the vehicle into the station, the process of going to the driver's seat, the process of bowing and greeting, the process of resetting the fuel gauge, the process of opening the fuel tank cap, the process of picking up the refueling nozzle, the process of closing the fuel tank cap, and the process of leading the vehicle out of the station. For each of the refueling procedures in the vehicle entry process, the bowing and greeting process, the fuel gauge reset process, and the vehicle exit process, the target classification result of the current refueling procedure in the video to be processed is identified. The target classification result is that the current refueling procedure is the corresponding standard refueling procedure, or that the current refueling procedure is not the corresponding standard refueling procedure. Multiple standard refueling procedures include the vehicle entry standard procedure, the heading to the driver's seat standard procedure, the bowing and greeting standard procedure, the fuel gauge reset standard procedure, the opening of the fuel tank cap standard procedure, the lifting of the refueling nozzle standard procedure, the closing of the fuel tank cap standard procedure, and the vehicle exit process. For the target image in the video to be processed corresponding to the process of going to the driver's position, identify the position of the rearview mirror of the refueling vehicle stopped in the parking area in the target image, and the position of the refueling staff. The target image is the image corresponding to when the refueling vehicle enters the parking area. Based on the position of the rearview mirror and the position of the staff, determine the distance between the rearview mirror and the refueling staff in the target image. If the distance meets the first condition in consecutive frames following the target image in the video to be processed, then it is determined that the current refueling process matches the standard process for going to the driving position. If the distance does not meet the first condition, then it is determined that the current refueling process does not match the standard process for going to the driving position. The first condition is that the distance is constantly decreasing and the distance when the distance does not change is less than a set threshold. For each refueling process in the process of opening the fuel tank cap and the process of closing the fuel tank cap, identify the opening and closing state of the fuel tank cap of the refueling vehicle in the video to be processed, wherein the opening and closing state is from closed to open or from open to closed; based on the opening and closing state and the execution order of the current refueling process in multiple standard refueling processes, determine the target detection result of the current refueling process in the video to be processed. For the refueling process, it is determined whether the video to be processed includes the actions of holding the refueling nozzle and holding the refueling tube. If the video to be processed includes the actions of holding the refueling nozzle and holding the refueling tube, it is determined that the current refueling process in the video to be processed matches the standard refueling process. If the video to be processed does not include the actions of holding the refueling nozzle and holding the refueling tube at the same time, it is determined that the current refueling process in the video to be processed does not match the standard refueling process.

2. The method according to claim 1, characterized in that, For each of the refueling procedures, including the gestures for guiding the vehicle into the station, bowing in greeting, resetting the fuel gauge, and guiding the vehicle out of the station, the target classification result for identifying the current refueling procedure in the video to be processed includes: Identify the refueling personnel in the video to be processed; Identify the key points of the refueling personnel in the video to be processed; Based on the key point position of the refueling personnel, the first classification result of the current refueling action corresponding to the refueling personnel is determined by a preset first classification model; Based on the key point position of the refueling personnel, the second classification result of the current refueling action corresponding to the refueling personnel is determined by a preset second classification model; Based on the first classification result and the second classification result, the target classification result of the current refueling process corresponding to the refueling personnel is determined.

3. The method according to claim 1, characterized in that, The method further includes: If any of the target classification results contains a first target refueling process that is not a standard refueling process, and / or if any of the target detection results contains a second target refueling process that does not match the standard refueling process; Generate alarm information and send it to the terminal devices of relevant personnel; If the second target refueling process includes the process of closing the fuel tank cap, the terminal device that generates alarm information and sends it to relevant personnel includes: If a change in the location of the refueling vehicle in the video to be processed is detected, an alarm message is generated and sent to the terminal device of the relevant personnel.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Based on the target detection results and target classification results corresponding to each refueling process, the service score of the refueling personnel after completing a refueling is determined.

5. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Determine the number of customers served by the refueling personnel in the video to be processed.

6. The method according to claim 2, characterized in that, The first classification model is a model trained based on a long short-term memory network, and the second classification model is a model trained based on an improved shufflenetv2 classification network. The improved shufflenetv2 classification network is a 1x1 Conv network obtained by replacing the 3x3 DWConv in the original shufflenetv2 classification network with 5x5 DWConv, changing the stride from 1 to 2, and removing DWConv.

7. A refueling process monitoring device, characterized in that, The device includes: The video acquisition module is used to acquire videos to be processed for multiple refueling processes. The videos to be processed include refueling personnel, refueling vehicles and refueling equipment. The multiple refueling processes are executed sequentially according to the refueling time sequence. The multiple refueling processes include the process of leading the vehicle into the station, the process of going to the driver's seat, the process of bowing and greeting, the process of resetting the fuel gauge, the process of opening the fuel tank cap, the process of picking up the refueling nozzle, the process of closing the fuel tank cap and the process of leading the vehicle out of the station. The first monitoring module is used to identify the target classification result of the current refueling process in the video to be processed for each of the refueling processes in the vehicle entry process, the bowing and greeting process, the fuel gauge resetting process, and the vehicle exit process. The target classification result is that the current refueling process is a corresponding standard refueling process, or that the current refueling process is not a corresponding standard refueling process. Multiple standard refueling processes include the vehicle entry standard process, the process of proceeding to the driver's seat standard process, the bowing and greeting standard process, the fuel gauge resetting standard process, the process of opening the fuel tank cap standard process, the process of picking up the refueling nozzle standard process, the process of closing the fuel tank cap standard process, and the vehicle exit process. The second monitoring module is used to identify the position of the rearview mirror of a refueling vehicle stopped in the parking area and the position of the refueling personnel in the target image of the video to be processed corresponding to the process of proceeding to the driving position. The target image is the image corresponding to when the refueling vehicle enters the parking area. Based on the position of the rearview mirror and the position of the personnel, the module determines the distance between the rearview mirror and the refueling personnel in the target image. If the distance satisfies a first condition in consecutive frames after the target image in the video to be processed, the current refueling process is determined to match the standard process of proceeding to the driving position. If the distance does not satisfy the first condition, the current refueling process is determined to not match the standard process of proceeding to the driving position. The first condition is that the distance is continuously decreasing and the distance when the distance does not change is less than a set threshold. For the process of opening the fuel tank cap... For each refueling process in the process of closing the fuel tank cap, the open / closed state of the fuel tank cap of the refueling vehicle in the video to be processed is identified, wherein the open / closed state is from closed to open, or from open to closed; based on the open / closed state and the execution order of the current refueling process in multiple standard refueling processes, the target detection result of the current refueling process in the video to be processed is determined; for the refueling process of lifting the nozzle, it is identified whether the video to be processed includes the actions of holding the fuel nozzle and holding the fuel hose; if the video to be processed includes the actions of holding the fuel nozzle and holding the fuel hose, it is determined that the current refueling process in the video to be processed matches the standard refueling process of lifting the nozzle; if the video to be processed does not include the actions of holding the fuel nozzle and holding the fuel hose simultaneously, it is determined that the current refueling process in the video to be processed does not match the standard refueling process of lifting the nozzle.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-6.

Images