Self-service fueling monitoring system

Abstract

To enable effective collection of learning data for re-learning a learning model used to monitor self-fueling.SOLUTION: A self-fueling monitor system comprises: a monitoring device 110 that detects a predetermined action from video data by using a learning model which has learned a predetermined action of a fueling person, and makes a determination related to permission / non-permission of fueling; a terminal device 120 that accepts an operation that allows fueling; and a learning device 210 that tags a normal action to the video data when a determination result by the monitoring device 110 related to the permission / non-permission of fueling and an operation content on the terminal device 120 do not match.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a system for monitoring self-service refueling at a gas station. [Background technology]

[0002] At conventional gas stations, a supervisor (employee) inside the station building would monitor the behavior of customers (fuel dispensers) doing self-service gas pumping either visually or on a surveillance monitor, and then operate a control device to issue permission to refuel. In recent years, there has been progress in the development of systems that use AI (Artificial Intelligence) to monitor surveillance camera footage, and these systems are beginning to be applied to the monitoring of self-service gas pumps.

[0003] Here, the following are examples of prior art in the technical field related to the present invention: For example, Patent Document 1 discloses an invention in which a monitoring device determines abnormal behavior of a user by analyzing camera footage and controls refueling of a vehicle according to the user's operation. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2021 / 015256 Summary of the Invention [Problem to be solved by the invention]

[0005] While progress is being made in the development of systems that analyze camera footage based on a predetermined learning model to determine whether or not to permit refueling, there is a problem in that the accuracy of the determination deteriorates due to changes in the surrounding environment and over time. Therefore, in order to maintain the accuracy of the determination, it is necessary to manually verify the success or failure of the determination, collect camera footage that fails to make the determination as learning data, and periodically re-run machine learning to update the learning model. However, verifying the success or failure of the determination for each gas station and selecting and collecting footage that fails to make the determination as learning data places a heavy burden on the person performing the work.

[0006] The present invention was made in consideration of the above-described conventional circumstances, and aims to enable the efficient collection of learning data for re-learning a learning model used to monitor self-service refueling. [Means for solving the problem]

[0007] In order to achieve the above object, the present invention is configured as follows. In other words, a self-service refueling monitoring system according to one embodiment of the present invention is characterized by comprising a monitoring device that uses a learning model that has learned the specific behavior of the refueler to detect specific behavior from video data and make a judgment as to whether or not to allow refueling, a terminal device that accepts operations to allow refueling, and a learning device that tags the video data as normal behavior if the judgment result by the monitoring device as to whether or not to allow refueling does not match the operation content on the terminal device.

[0008] Here, the terminal device may be configured to accept an operation to specify an area to be cut out from the video data, and the learning device may be configured to tag the area cut out from the video data as normal behavior.

[0009] In addition, a self-service refueling monitoring system according to another aspect of the present invention is characterized by comprising a monitoring device that detects specified behavior from video data using a learning model that has learned the specified behavior of refuelers, a terminal device that accepts operations to specify the type of abnormal behavior, and a learning device that tags the video data with the abnormal behavior corresponding to the operation content if the detection result by the monitoring device does not match the abnormal behavior corresponding to the operation content accepted by the terminal device.

[0010] Here, the terminal device may be configured to accept an operation to specify an area to be cut out from the video data, and the learning device may be configured to tag the area cut out from the video data with abnormal behavior. [Effects of the Invention]

[0011] According to the present invention, it becomes possible to efficiently collect learning data for retraining a learning model used to monitor self-service refueling. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a diagram showing an example of the configuration of a self-service refueling permission system included in a self-service refueling monitoring system according to an embodiment of the present invention. [Figure 2] 1 is a diagram showing an example of the configuration of a determination accuracy maintenance system included in a self-service refueling monitoring system according to one embodiment of the present invention. [Figure 3] 2 is a diagram showing an example of a display on a terminal device of the self-service refueling permission system of FIG. 1. FIG. [Figure 4] FIG. 10 is a diagram illustrating an example of a processing flow for collecting learning data according to the first embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of a processing flow for collecting learning data according to the second embodiment. [Figure 6A] FIG. 11 is a diagram illustrating an example of a processing flow for collecting learning data according to the third embodiment. [Figure 6B] FIG. 11 is a diagram illustrating an example of a processing flow for collecting learning data according to the third embodiment. [Figure 7] FIG. 13 is a diagram illustrating an example of a processing flow for collecting learning data according to the fourth embodiment. [Figure 8] FIG. 13 is a diagram illustrating an example of a processing flow for collecting learning data according to the fifth embodiment. [Figure 9] FIG. 3 is a diagram showing an example of a processing flow by the determination accuracy maintenance system of FIG. 2. DETAILED DESCRIPTION OF THE INVENTION

[0013] A self-service refueling monitoring system according to one embodiment of the present invention will be described below with reference to the drawings. The self-service refueling monitoring system according to one embodiment of the present invention has two subsystems: a self-service refueling permission system 100 configured as shown in Fig. 1, and a determination accuracy maintenance system 200 configured as shown in Fig. 2.

[0014] The self-service refueling permission system 100 is a system that determines whether or not to permit self-service refueling based on camera footage, and is installed at each gas station. The self-service refueling permission system 100 includes a monitoring device 110, a terminal device 120, a control device 130, a monitoring camera 140, a sensor 150, and a server 160.

[0015] The surveillance camera 140 captures images of the self-service refueling performed by the refueler (user) and outputs the captured image data (camera footage) to the monitoring device 110. The surveillance camera 140 may be installed so as to capture images of the actions of the refueler when performing self-service refueling, and may be installed, for example, in a position that overlooks an area including the area between the meter and the vehicle from above.

[0016] The sensor 150 detects sound, heat, odor, light, oil leakage, or the like generated by the actions of the person filling up the tank with fuel at the self-service facility, and outputs the detection results to the monitoring device 110. The sensor 150 also includes a sensor that detects when the fuel nozzle is removed from the fuel dispenser. The sensor 150 is installed at or near the fuel dispenser (for example, in a parking area where the vehicle to be filled up is parked).

[0017] The monitoring device 110 is installed in a location (for example, inside a gas station building) separate from the metering machine. The monitoring device 110 has an image analysis device 111, an interface 112, and a wireless device 113. The image analysis device 111 analyzes camera footage using an analysis program that uses a preset learning model, detects predetermined actions that the fuel dispenser may take, and determines whether or not to permit self-service fueling (permit / deny). The wireless device 113 communicates wirelessly with the terminal device 120. The interface 112 is interposed between the monitoring camera 140, the sensor 150, the control device 130, the image analysis device 111, and the wireless device 113. The monitoring device 110 is, for example, a computer equipped with hardware resources such as a processor and memory, and is configured so that the processor executes programs related to each function of the present invention.

[0018] The terminal device 120 is a device that receives operations from a monitor (employee) regarding whether or not to permit self-service refueling during self-service refueling. The terminal device 120 has a control unit 121, a communication unit 122, a display unit 123, and an operation unit 124. The control unit 121 comprehensively controls the operation of each unit of the terminal device 120. The communication unit 122 performs wireless communication with the monitoring device 110. The display unit 123 displays various information including the determination results by the monitoring device 110. The operation unit 124 receives various operations including operations regarding whether or not to permit self-service refueling. The display unit 123 and the operation unit 124 may be integrated into a touch panel. The terminal device 120 is, for example, a computer equipped with hardware resources such as a processor and memory, and is configured so that the processor executes programs related to each function of the present invention.

[0019] Terminal device 120 receives and displays the detection results of a predetermined behavior and the determination results of whether or not refueling is permitted from monitoring device 110. The monitor refers to the detection results of a predetermined behavior and the determination results of whether or not refueling is permitted displayed on terminal device 120, and determines whether or not to permit or prohibit (disallow) refueling, and inputs an operation regarding whether or not to permit self-refueling. Here, for example, when an operation to permit refueling is input, a refueling permission signal is transmitted from terminal device 120 to the metering machine, which then activates the pump inside the metering machine, allowing refueling. Here, an example is described in which terminal device 120 is a portable terminal such as a tablet that can be carried by the monitor (employee), but it may also be a stationary terminal installed (fixed) in a predetermined location.

[0020] Fig. 3 shows an example of a monitoring screen displayed on the terminal device 120. The monitoring screen 300 in Fig. 3 has a lane status display section 310, a live video display section 320, a detection history display section 330, a detected video display section 340, abnormal behavior confirmation buttons 351 to 354, and a refueling permission button 360.

[0021] The lane status display unit 310 displays the status of each fueling lane at the gas station (waiting for fuel, being fueled, etc.), the type of fuel being filled, the amount of fuel filled, other order details, etc. The live video display unit 320 displays live (real-time) camera footage of the active fueling lane the fuel dispenser is visiting (if there are multiple active fueling lanes, a fueling lane selected from among them). The detection history display unit 330 displays the history of the fuel dispenser's actions detected from the camera footage of the active fueling lane. The detection video display unit 340 displays the camera footage at the time the fuel dispenser's actions were detected (i.e., the camera footage capturing the fuel dispenser's actions).

[0022] The abnormal behavior confirmation buttons 351 to 354 are buttons that the monitor operates when he / she confirms, through camera footage or visual observation, abnormal behavior that should not be performed by a fuel dispenser, and are provided for each type of abnormal behavior. In this example, the abnormal behavior detection buttons include a cigarette possession confirmation button 351 that is operated when it is confirmed that the fuel dispenser is carrying cigarettes, a portable can confirmation button 352 that is operated when it is confirmed that the fuel dispenser is carrying a portable can, a risky behavior confirmation button 353 that is operated when it is confirmed that the fuel dispenser is engaging in risky behavior (e.g., filling up with gas with multiple people), and a nozzle half-insertion confirmation button 354 that is operated when it is confirmed that the fuel dispenser is not fully inserted into the fuel filler opening. In this example, the fuel dispenser nozzle half-insertion indicates that the fuel dispenser nozzle is not inserted properly, and indicates behavior that should be determined to be abnormal as the behavior of the fuel dispenser. The fuel dispenser permission button 360 is a button that the monitor operates when he / she confirms, through camera footage or visual observation, that it is acceptable to permit self-service fuel dispenser operation.

[0023] The control device 130 is installed, for example, near the weighing machine, and controls the operation of the weighing machine based on a control signal transmitted from the monitoring device 110 in response to the operation of the terminal device 120. For example, when the terminal device 120 receives an operation from the monitor to permit refueling, the control device 130 controls the weighing machine so that refueling is carried out by the operation of the refueler. Also, when the terminal device 120 receives an operation from the monitor to prohibit (not permit) refueling, the control device 130 controls the weighing machine so that refueling is prohibited (suspended). Note that the control signal in response to the operation of the terminal device 120 is not limited to a configuration in which it is transmitted from the monitoring device 110 to the control device 130, but may be configured to be transmitted from the terminal device 120 to the control device 130.

[0024] Like the monitoring device 110, the server 160 is installed in a location remote from the weighing machine (for example, inside a gas station building or on the cloud). The server 160 has an image database 161, a learning model management unit 162, and a communication unit 163. The image database 161 accumulates learning data for retraining a learning model that detects predetermined behaviors from camera footage. The learning data accumulated in the image database 161 is transmitted to the judgment accuracy maintenance system 200. The learning model management unit 162 updates the analysis program used in the image analysis device 111 of the monitoring device 110 based on the learning model distributed from the judgment accuracy maintenance system 200. The communication unit 163 communicates with the judgment accuracy maintenance system 200. The server 160 is realized by a computer equipped with hardware resources such as a processor and memory, and is configured so that the processor executes programs for realizing each function according to the present invention.

[0025] The determination accuracy maintenance system 200 is for maintaining the determination accuracy of the self-service refueling permission system 100, and is installed at a base (for example, a management center) connected to each gas station via a network line. The determination accuracy maintenance system 200 may be installed inside the gas station building instead of at a remote base. The determination accuracy maintenance system 200 includes a learning device 210 that retrains the learning model used in the self-service refueling permission system 100 at each gas station.

[0026] The learning device 210 has a communication unit 211, an image database 212, a machine learning unit 213, and a control unit 214. The communication unit 211 communicates with the self-service refueling permission system 100 of each gas station. The image database 212 accumulates learning data collected from the self-service refueling permission system 100 of each gas station. The machine learning unit 213 performs machine learning (relearning) based on the learning data accumulated in the image database 212, and regenerates a learning model to be used in the self-service refueling permission system 100 of each gas station. The control unit 214 controls the re-learning by the machine learning unit 213. The learning model generated by the machine learning unit 213 is distributed to the self-service refueling permission system 100 of each gas station. The learning device 210 is realized by, for example, a computer equipped with hardware resources such as a processor and a memory, and is configured so that the processor executes programs for realizing each function according to the present invention.

[0027] The learning model may be a general-purpose learning model commonly used among multiple gas stations, or a dedicated learning model used only at a specific gas station. The general-purpose learning model can be generated based on learning data collected from the self-service refueling permission systems 100 of the multiple gas stations that use it. The dedicated learning model can be generated based on learning data collected from the self-service refueling permission systems 100 of the specific gas station that uses it. These learning models may be provided separately for each type of behavior to be detected, or one learning model may be configured to be able to detect multiple types of behavior.

[0028] The main features of the self-service refueling monitoring system in this example are that it includes a monitoring device 110 that detects predetermined actions from camera footage of the self-service refueling process using a learning model that has learned predetermined actions that a refueler may perform when self-service refueling; a terminal device 120 that accepts operations from the monitor regarding whether or not to allow self-service refueling when self-service refueling is performed; and a learning device 210 that has an image database 212 that stores camera footage as learning data for re-learning the learning model for the predetermined actions when the predetermined action is determined to have been missed or falsely detected by comparing the detection results of the predetermined action by the monitoring device 110 with the operations performed on the terminal device 120.

[0029] In other words, the self-service refueling monitoring system of this example is configured to automatically recognize missed or false detections of a predetermined behavior by comparing the detection results of the predetermined behavior by the monitoring device 110 with the operation details on the terminal device 120, and to store camera images of cases where a missed or false detection of a predetermined behavior is determined as a missed or false detection in the image database 212 as learning data for re-learning. In this way, camera images of cases where a missed or false detection of a predetermined behavior occurs can be collected as learning data for re-learning based on the operations (operations related to the permission or non-permission of self-service refueling) performed by the monitor as part of their normal duties. As a result, the learning model used to monitor self-service refueling can be efficiently re-trained. Furthermore, because the judgment of the monitor at each gas station is reflected in the collection of learning data, even if a learning model is prepared for each gas station, a learning model appropriate for that gas station can be easily re-trained.

[0030] The operation of collecting learning data will be described below with reference to several examples. [First example of learning data collection] 4 shows an example of a processing flow for collecting learning data according to the first embodiment. A user (fuel dispenser) intending to perform self-service refueling gets into a vehicle and arrives at a gas station, parks the vehicle in a parking area near a meter, gets out of the vehicle, and then begins to refuel. At this time, a surveillance camera 140 captures the actions of the fuel dispenser when performing self-service refueling and transmits the camera footage to a monitoring device 110. The monitoring device 110 analyzes the camera footage using an analysis program based on a preset learning model and detects predetermined actions that the fuel dispenser may perform.

[0031] For example, the monitoring device 110 performs an analysis of camera footage using a learning model for detecting whether a fuel dispenser is carrying cigarettes (step S101). It also performs image analysis using a learning model for detecting whether a portable fuel can has been brought in (step S102). It also performs image analysis using a learning model for detecting risky behavior by fuel dispensers (e.g., filling up with fuel together with multiple people) (step S103). It also performs image analysis using a learning model for detecting the insertion state (fully inserted / partially inserted) of the fuel nozzle into the fuel filler opening (step S104).

[0032] Next, the monitoring device 110 determines whether or not to permit refueling (permit / deny) based on the results of the above detection process (steps S101 to S104) (step S105). For example, if possession of cigarettes, carrying a portable can, or risky behavior is not detected and full insertion of the fuel nozzle is detected, it is determined that refueling may be permitted. Also, if possession of cigarettes, carrying a portable can, risky behavior, or partial insertion of the fuel nozzle is detected, or if full insertion of the fuel nozzle is not detected, it is determined that refueling is not permitted. Data on the results of the above detection process and determination process are transmitted to the terminal device 120.

[0033] The terminal device 120 reflects the data received from the monitoring device 110 on the display and accepts an operation from the monitor (employee) regarding whether or not to permit refueling. For example, if the monitor decides to permit refueling, the monitor operates the refueling permission button 360 on the monitoring screen 300 in Fig. 3. On the other hand, if the monitor confirms any abnormal behavior and decides not to permit refueling, the monitor operates one of the abnormal behavior confirmation buttons 351 to 354 on the monitoring screen 300 in Fig. 3.

[0034] Next, monitoring device 110 compares the result of its own refueling permission determination with the result of the refueling permission determination by the monitor (operation related to refueling permission), and determines whether they match (step S106). If the result of the refueling permission determination by monitoring device 110 and the result of the refueling permission determination by the monitor match (step S106: Yes), monitoring device 110 transmits comparison result data recording that the detection was successful to server 160 (step S107).

[0035] On the other hand, if the result of the refueling permission determination by the monitoring device 110 does not match the result of the refueling permission determination by the monitor (step S106: No), the monitoring device 110 identifies the image analysis result of the incorrect determination and classifies it as a ``missed detection'' or ``false detection'' (step S108), and sends the comparison result data recording the detection failure to the server 160 together with the video of the incorrect determination and the classification result (step S109).

[0036] The server 160 stores the data received from the monitoring device 110 in an image database 161, and transmits the data as needed or periodically to the learning device 210 of the judgment accuracy maintenance system 200. The learning device 210 stores the data received from the server 160 in an image database 212, and executes re-learning of the learning model when predetermined conditions are met.

[0037] As described above, in the first embodiment, the detection result of the predetermined behavior is compared with the operation content of the terminal device 120 to automatically recognize that a predetermined behavior has been missed or mistakenly detected, and the camera video when the predetermined behavior is determined to have been missed or mistakenly detected is identified. This makes it possible to efficiently collect camera video when a predetermined behavior has been missed or mistakenly detected and re-learn it. Note that, in the above description, behavior detection using video analysis is performed as a step prior to granting permission for refueling, but behavior detection may also be performed after refueling is permitted, and refueling may be stopped upon detection of abnormal behavior such as possession of cigarettes, carrying a portable can, or dangerous behavior.

[0038] [Second example of learning data collection] 5 shows an example of a processing flow for collecting learning data according to the second embodiment. In the second embodiment, the explanation focuses on a learning model for detecting the insertion state (fully inserted / partially inserted) of a fuel nozzle relative to a fuel filler neck. The monitoring device 110 waits until the sensor 150 detects that the fuel filler neck has been removed from the metering machine (step S201). When it detects that the fuel filler neck has been removed from the metering machine, the monitoring device 110 analyzes the camera image based on the learning model and determines the insertion state of the fuel filler neck relative to the fuel filler neck (step S202).

[0039] If a certain time has passed since detecting that the fuel filler nozzle has been removed, and the fuel filler nozzle is not detected as being fully or partially inserted into the fuel filler opening (step S202: not detected), monitoring device 110 notifies the monitor by displaying a message to that effect on terminal device 120, and has the monitor confirm the insertion status of the fuel filler nozzle (step S203). If terminal device 120 subsequently receives an operation indicating that the fuel filler nozzle has been fully inserted (for example, pressing of refueling permission button 360) (step S204: OK), monitoring device 110 determines that full nozzle insertion was not detected, and stores the camera image at that time as positive learning data for full nozzle insertion (step S205). On the other hand, if the terminal device 120 receives an operation indicating that the fuel nozzle has been partially inserted (for example, pressing the nozzle partially inserted confirmation button 354) (step S204: NG), the monitoring device 110 determines that the nozzle has not been partially inserted and stores the camera image at that time as positive learning data for fully inserted nozzle (step S205).

[0040] Furthermore, if the monitoring device 110 detects that the fuel filler nozzle is fully inserted into the fuel filler opening within a certain time after detecting that the fuel filler nozzle has been removed (step S202: nozzle fully inserted), the monitoring device 110 notifies the monitor by displaying a message to that effect on the terminal device 120, and has the monitor confirm the insertion status of the fuel filler nozzle (step S207). If the terminal device 120 subsequently receives an operation indicating that the fuel filler nozzle has been fully inserted (e.g., pressing the fuel fill permission button 360) (step S207: OK), the monitoring device 110 determines that the full insertion of the nozzle has been accurately detected. On the other hand, if the terminal device 120 receives an operation indicating that the fuel filler nozzle has been partially inserted (e.g., pressing the nozzle partial insertion confirmation button 354) (step S204: NG (partial insertion)), the monitoring device 110 determines that this is a false detection of nozzle full insertion and a missed detection of nozzle partial insertion, and stores the camera image at that time as positive learning data for nozzle partial insertion (step S205). Furthermore, if the terminal device 120 receives an operation indicating that a different state (for example, the fuel nozzle is not inserted) has been confirmed (step S207: NG (different)), the monitoring device 110 determines that this is a false detection of the nozzle being fully inserted or partially inserted, and stores the camera images at that time as negative learning data for the nozzle being fully inserted and negative learning data for the nozzle being partially inserted (step S208).

[0041] Furthermore, if the monitoring device 110 detects that the fuel filler nozzle is partially inserted into the fuel filler opening within a certain time period after detecting that the fuel filler nozzle has been removed (step S202: nozzle partially inserted), the monitoring device 110 notifies the monitor by displaying a message to that effect on the terminal device 120, and has the monitor confirm the insertion state of the fuel filler nozzle (step S209). If the terminal device 120 subsequently receives an operation indicating that the fuel filler nozzle has been partially inserted (e.g., pressing the nozzle partially inserted confirmation button 354) (step S209: OK), the monitoring device 110 determines that the nozzle partially inserted was correctly detected. On the other hand, if the terminal device 120 receives an operation indicating that the fuel filler nozzle has been fully inserted (e.g., pressing the fuel fill permission button 360) (step S209: NG (fully inserted)), the monitoring device 110 determines that this is a false detection of nozzle partially inserted and a missed detection of nozzle fully inserted, and stores the camera image at that time as positive learning data for nozzle fully inserted (step S210). Furthermore, if the terminal device 120 receives an operation indicating that a different state (for example, the fuel nozzle is not inserted) has been confirmed (step S209: NG (different)), the monitoring device 110 determines that this is a false detection of the nozzle being fully inserted or partially inserted, and stores the camera images at that time as negative learning data for the nozzle being fully inserted and negative learning data for the nozzle being partially inserted (step S210).

[0042] The positive learning data and negative learning data on the insertion state of the fuel filler nozzle collected by the monitoring device 110 as described above are stored in the server 160 and are transmitted as needed or periodically to the learning device 210 of the judgment accuracy maintenance system 200. The learning device 210 stores the positive learning data and negative learning data on the insertion state of the fuel filler nozzle received from the server 160 in the image database 212, and performs re-learning of the learning model for detecting the insertion state of the fuel filler nozzle when predetermined conditions are met.

[0043] In this way, the second embodiment is configured to collect camera footage in the event of a detection miss and camera footage in the event of a false positive separately. That is, camera footage in the event of a predetermined action (in this example, the fuel nozzle being fully inserted or partially inserted) being determined to be a detection miss is stored as positive learning data showing the state of the predetermined action when it is correct. In addition, camera footage in the event of a predetermined action being determined to be a false positive is stored as negative learning data showing the state of the predetermined action when it is incorrect. In this way, by storing camera footage in the event of a detection miss and camera footage in the event of a false positive separately, it becomes possible to retrain a learning model that can detect predetermined actions with higher accuracy.

[0044] [Third example of learning data collection] 6A and 6B show example processing flows for collecting learning data according to the third embodiment. The processing flow example of FIG. 6A relates to the collection of learning data for a learning model for detecting abnormal behavior that should not be performed by a fuel pump operator. The processing flow example of FIG. 6B relates to the collection of learning data for a learning model for detecting normal behavior that should be performed by a fuel pump operator. Examples of abnormal behavior include carrying cigarettes, carrying a portable can, risky behavior, and partially inserting the fuel nozzle. Also, examples of normal behavior include fully inserting the fuel nozzle.

[0045] First, with reference to FIG. 6A, the collection of learning data for a learning model for detecting abnormal behavior will be described. The monitoring device 110 analyzes the camera footage based on the learning model to determine whether or not abnormal behavior is present (step S301). If abnormal behavior is not detected (step S301: No), the monitoring device 110 prompts the monitor to take an action regarding whether or not to permit refueling. If the terminal device 120 subsequently accepts an operation to permit refueling (e.g., pressing the refueling permission button 360) (step S302: OK), the monitoring device 110 determines that not detecting abnormal behavior was appropriate. On the other hand, if the terminal device 120 does not accept the operation to permit refueling but instead accepts another operation (e.g., pressing one of the abnormal behavior confirmation buttons 351 to 354) (step S302: NG), the monitoring device 110 determines that abnormal behavior was not detected and stores the camera footage at that time as positive learning data for abnormal behavior (step S303).

[0046] Furthermore, if abnormal behavior is detected by image analysis based on the learning model (step S301: Yes), the monitoring device 110 notifies the monitor by displaying a message to that effect on the terminal device 120 and prompts the monitor to take an action regarding whether or not to permit refueling (step S304). If the terminal device 120 subsequently accepts an operation to permit refueling (e.g., pressing the refueling permission button 360) (step S305: OK), the monitoring device 110 determines that the abnormal behavior was detected erroneously and stores the camera image at that time as negative learning data for the abnormal behavior (step S306). On the other hand, if the terminal device 120 does not accept the operation to permit refueling but accepts another operation (e.g., pressing one of the abnormal behavior confirmation buttons 351-354) (step S305: NG), the monitoring device 110 determines that the detection of the abnormal behavior was appropriate.

[0047] Next, learning data collection for a learning model for detecting normal behavior will be described with reference to FIG. 6B. The monitoring device 110 analyzes the camera footage based on the learning model to determine whether normal behavior is present (step S401). If normal behavior is detected (step S401: Yes), the monitoring device 110 prompts the monitor to take an action regarding whether to permit refueling. If the terminal device 120 subsequently accepts an action to permit refueling (e.g., pressing the refueling permission button 360) (step S402: OK), the monitoring device 110 determines that the detection of normal behavior was appropriate. On the other hand, if the terminal device 120 does not accept the action to permit refueling but accepts another action (e.g., pressing one of the abnormal behavior confirmation buttons 351 to 354) (step S402: NG), the monitoring device 110 determines that normal behavior is falsely detected and stores the camera footage at that time as negative learning data for normal behavior (step S403).

[0048] Furthermore, if normal behavior is not detected by image analysis based on the learning model (step S401: No), the monitoring device 110 notifies the monitor by displaying a message to that effect on the terminal device 120 and prompts the monitor to take an action regarding whether or not to allow refueling (step S404). If the terminal device 120 subsequently receives an operation to allow refueling (e.g., pressing the refueling permission button 360) (step S405: OK), the monitoring device 110 determines that normal behavior has been overlooked and stores the camera image at that time as positive learning data for normal behavior (step S406). On the other hand, if the terminal device 120 does not receive an operation to allow refueling but instead receives another operation (e.g., pressing one of the abnormal behavior confirmation buttons 351-354) (step S405: NG), the monitoring device 110 determines that not detecting normal behavior was appropriate.

[0049] The learning data for each of normal behavior and abnormal behavior collected by the monitoring device 110 as described above is stored in the server 160 and is transmitted as needed or periodically to the learning device 210 of the judgment accuracy maintenance system 200. The learning device 210 stores the learning data for each of normal behavior and abnormal behavior received from the server 160 in the image database 212, and executes re-learning of the learning model for detecting normal behavior and the learning model for detecting abnormal behavior when predetermined conditions are met.

[0050] In this way, in the third embodiment, learning data for normal behavior and learning data for abnormal behavior are collected according to different standards, which makes it possible to retrain a learning model that detects normal behavior with higher accuracy and a learning model that detects abnormal behavior with higher accuracy.

[0051] [Fourth Example of Learning Data Collection] Fig. 7 shows an example of a process flow for collecting learning data according to the fourth embodiment. The process flow example in Fig. 7 relates to the collection of learning data for a learning model that has learned multiple types of abnormal behavior that should not be performed by fuel pump drivers. The multiple types of abnormal behavior include, for example, carrying cigarettes, carrying a portable can, risky behavior, and partially inserting the fuel nozzle.

[0052] The monitoring device 110 analyzes the camera footage based on a learning model and determines whether or not multiple types of abnormal behavior are present (step S501). If none of the multiple types of abnormal behavior are detected (step S501: No), the monitoring device 110 prompts the monitor to take an action regarding whether or not to permit refueling. If the terminal device 120 subsequently accepts an action to permit refueling (e.g., pressing the refueling permission button 360) (step S502: OK), the monitoring device 110 determines that not detecting abnormal behavior was appropriate. On the other hand, if the monitor does not permit refueling because he or she has observed any of the multiple types of abnormal behavior (step S502: NG), the terminal device 120 accepts an action to specify the type of abnormal behavior that has been observed (e.g., pressing one of the abnormal behavior confirmation buttons 351 to 354) (step S503). At this time, the monitoring device 110 determines that the type of abnormal behavior designated by the monitor has been missed, and stores the camera image at that time as positive learning data for that type of abnormal behavior (step S504).

[0053] Furthermore, if any of multiple types of abnormal behavior is detected by image analysis based on the learning model (step S501: Yes), the type of detected abnormal behavior is displayed on the terminal device 120 to notify the monitor and prompt the monitor to take action regarding whether or not to allow refueling (step S505). If the terminal device 120 subsequently receives an action to allow refueling (for example, pressing the refueling permission button 360) (step S506: OK), the monitoring device 110 determines that the type of detected abnormal behavior is a false positive and stores the camera image at that time as negative learning data for that type of abnormal behavior (step S507).

[0054] On the other hand, if the monitor does not permit refueling because he / she has confirmed one of multiple types of abnormal behavior (step S506: NG), the terminal device 120 accepts an operation to specify the type of abnormal behavior confirmed (for example, pressing one of the abnormal behavior confirmation buttons 351-354) (step S508). As a result, if the type of detected abnormal behavior does not match the type of specified abnormal behavior (step S509: No), the monitoring device 110 determines that there was a detection miss for the specified type of abnormal behavior and stores the camera video at that time as positive learning data for that type of abnormal behavior (step S510). On the other hand, if the type of detected abnormal behavior matches the type of specified abnormal behavior (step S509: Yes), the monitoring device 110 determines that the type of detected abnormal behavior was appropriate.

[0055] The learning data for each of the multiple types of abnormal behavior collected by the monitoring device 110 as described above is stored in the server 160 and is transmitted as needed or periodically to the learning device 210 of the judgment accuracy maintenance system 200. The learning device 210 stores the learning data for each of the multiple types of abnormal behavior received from the server 160 in the image database 212, and executes re-learning of the learning model for detecting each of the multiple types of abnormal behavior when a predetermined condition is satisfied.

[0056] In this way, the fourth embodiment is configured to collect learning data by distinguishing between multiple types of abnormal behaviors, which makes it possible to retrain a learning model that can detect each of the multiple types of abnormal behaviors with higher accuracy.

[0057] In the fourth embodiment, one learning model is configured to detect multiple abnormal behaviors, but multiple learning models may be configured to detect different abnormal behaviors. In this case, in S504, the abnormal behavior specified by the monitor in S503 is stored as positive learning data for the learning model. In S507, the abnormal behavior detected in S505 is stored as negative learning data for the learning model. In S510, since the abnormal behavior detected in S505 was a false positive, the abnormal behavior detected in S505 is stored as negative learning data for the learning model. Furthermore, since the abnormal behavior specified by the monitor in S508 was also missed, the abnormal behavior specified by the monitor in S508 is stored as positive learning data for the learning model.

[0058] [Fifth Example of Learning Data Collection] Fig. 8 shows an example of a processing flow for collecting learning data according to the fifth embodiment. The processing flow example in Fig. 8 relates to the collection of learning data for a learning model that has learned normal behavior that a fuel dispenser should perform and abnormal behavior that a fuel dispenser should not perform. An example of such a learning model is a learning model that detects fully inserted nozzles as normal behavior and partially inserted nozzles as abnormal behavior.

[0059] The monitoring device 110 analyzes the camera footage based on the learning model to determine whether normal or abnormal behavior is present (step S601). If neither normal nor abnormal behavior is detected (step S601: not detected), the monitoring device 110 prompts the monitor to perform an operation regarding whether to permit refueling (step S602). If the terminal device 120 subsequently does not accept an operation to permit refueling (e.g., pressing the refueling permission button 360) but instead accepts another operation (e.g., pressing one of the abnormal behavior confirmation buttons 351-354) (step S603: NG), the monitoring device 110 determines that an abnormal behavior has been missed and stores the camera footage at that time as positive image data for abnormal behavior (step S604). On the other hand, if the terminal device 120 accepts an operation to permit refueling (step S603: NG), the monitoring device 110 determines that a normal behavior has been missed and stores the camera footage at that time as positive image data for normal behavior (step S605).

[0060] Furthermore, if abnormal behavior is detected by image analysis based on the learning model (step S601: Abnormal Behavior), the monitoring device 110 notifies the monitor by displaying a message to that effect on the terminal device 120 and prompts the monitor to take an action regarding whether or not to allow refueling (step S606). If the terminal device 120 subsequently receives an operation to allow refueling (e.g., pressing the refueling permission button 360) (step S607: OK), the monitoring device 110 determines that this is a missed detection of normal behavior and a false detection of abnormal behavior, and stores the camera image at that time as positive image data for normal behavior (step S608). On the other hand, if the terminal device 120 does not receive an operation to allow refueling but instead receives another operation (e.g., pressing one of the abnormal behavior confirmation buttons 351-354) (step S607: NG), the monitoring device 110 determines that the detection of abnormal behavior was appropriate.

[0061] Furthermore, if normal behavior is detected by image analysis based on the learning model (step S601: normal behavior), the monitoring device 110 prompts the monitor to take an action regarding whether or not to allow refueling. If the terminal device 120 subsequently accepts an action to allow refueling (e.g., pressing the refueling permission button 360) (step S609: OK), the monitoring device 110 determines that the detection of normal behavior was appropriate. On the other hand, if the terminal device 120 does not accept the action to allow refueling and instead accepts another action (e.g., pressing one of the abnormal behavior confirmation buttons 351-354) (step S609: NG), the monitoring device 110 determines that this is a false positive detection of normal behavior and a missed detection of abnormal behavior, and stores the camera video at that time as positive image data for abnormal behavior (step S610).

[0062] The learning data for each of normal behavior and abnormal behavior collected by the monitoring device 110 as described above is stored in the server 160 and is transmitted as needed or periodically to the learning device 210 of the judgment accuracy maintenance system 200. The learning device 210 stores the learning data for each of normal behavior and abnormal behavior received from the server 160 in the image database 212, and executes re-learning of the learning model for detecting each of normal behavior and abnormal behavior when predetermined conditions are met.

[0063] In this way, in the fifth embodiment, the learning data for normal behavior and the learning data for abnormal behavior are collected according to different standards, which makes it possible to retrain a learning model that detects normal behavior and abnormal behavior with higher accuracy.

[0064] Fig. 9 shows an example of a processing flow by the learning device 210 of the determination accuracy maintenance system 200 of Fig. 2. The processing flow example of Fig. 9 focuses on the execution of re-learning of the learning model, and is executed after the learning data has been collected by the method shown in Figs. 4 to 8. The following describes an example of processing subsequent to the processing flow shown in Fig. 4 (first embodiment).

[0065] The learning device 210 accumulates the comparison result data (detection success / detection failure), the video of the judgment error, and the classification results (detection failure / false positive detection) received from the server 160 of each gas station in the image database 212 (step S701). The control unit 214 of the learning device 210 aggregates the data accumulated in the image database 212 and determines whether the conditions for starting re-learning are met. In this example, the conditions for starting re-learning are: [Condition 1] the detection failure rate (= number of detection failures / total number of comparison results) is equal to or greater than a first threshold (step S702); [Condition 2] the false positive rate (= number of false positives / total number of comparison results) is equal to or greater than a second threshold (step S703); [Condition 3] an operation to start re-learning has been received from the administrator (step S704); and [Condition 4] a predetermined time has elapsed since the previous learning (step S705). If any one of [Condition 1] to [Condition 4] is satisfied, the control unit 214 causes the machine learning unit 213 to start relearning of the learning model (step S706). On the other hand, if none of [Condition 1] to [Condition 4] is satisfied, the relearning of the learning model is not started.

[0066] The machine learning unit 213 performs machine learning (re-learning) based on the learning data stored in the image database 212, and regenerates a learning model to be used in the self-service refueling permission system 100 of each gas station. The learning data is created by an operator visually checking and tagging the video data stored as positive learning data and the video data stored as negative learning data. Alternatively, an observer may generate learning data by pressing the abnormal behavior confirmation buttons 351 to 354 or the refueling permission button 360 in FIG. 3 to tag the corresponding abnormal behavior or normal behavior. This reduces the time required for annotation. The learning model generated by the machine learning unit 213 is distributed to the self-service refueling permission system 100 of each gas station via the communication unit 211. The server 160 of the self-service refueling permission system 100 of each gas station provides the learning model received from the learning device 210 of the judgment accuracy maintenance system 200 to the learning model management unit 162. Based on the distributed learning model, the learning model management unit 162 updates the analysis program of the image analysis device 111 of the monitoring device 110. By updating the learning model used to monitor self-service refueling in this way, it is possible to prevent the accuracy of judgment from deteriorating due to changes in the surrounding environment or over time.

[0067] In the explanation so far, the monitoring device 110 performs a process of identifying camera footage when a detection miss or false detection of a behavior is determined by comparing the detection result of a predetermined behavior with the operation content on the terminal device 120, but that process may also be performed by another device (for example, the terminal device 120, the server 160, etc.), and the entity that performs the process does not matter. However, if the above process is performed by a device located separately from the internal network of each gas station (for example, a device on the side of the determination accuracy maintenance system 200), there is a concern that the network load will increase, so it is preferable that the above process be performed by a device located on the internal network of each gas station.

[0068] The learning model may be common to all gas stations, or a dedicated learning model may be prepared for each gas station. For example, a common learning model may be used for gas stations with a standard layout and camera arrangement, and a dedicated learning model may be used for gas stations with a unique layout and camera arrangement.

[0069] The image data to be tagged may be specified by the observer via the monitoring screen of Fig. 3. For example, the observer specifies an area to be cut out from live video 320, and presses one of abnormal behavior confirmation buttons 351 to 354 or refueling permission button 360. This observer's operation causes the image data of the cut-out area to be tagged with the abnormal behavior or normal behavior corresponding to the pressed button.

[0070] Although the embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take on various other embodiments, and various modifications such as omissions and substitutions can be made without departing from the spirit of the present invention. These embodiments and modifications thereof are included in the scope and spirit of the invention described in this specification, etc., and are included in the invention described in the claims and their equivalents.

[0071] Furthermore, the present invention can be provided not only as devices such as those described above or as systems composed of these devices, but also as methods executed by these devices, programs for realizing the functions of these devices using a processor, and storage media for storing such programs in a computer-readable manner. [Industrial Applicability]

[0072] The present invention can be used in a system for monitoring self-service refueling at a gas station. [Explanation of symbols]

[0073] 100: Self-service refueling permission system, 110: Monitoring device, 111: Image analysis device, 112: Interface, 113: Wireless device, 120: Terminal device, 121: Control unit, 122: Communication unit, 123: Display unit, 124: Operation unit, 130: Control device, 140: Monitoring camera, 150: Sensor, 160: Server, 161: Image database, 162: Learning model management unit, 163: Communication unit, 200: Judgment accuracy maintenance system, 210: Learning device, 211: Communication unit, 212: Image database, 213: Machine learning unit, 214: Control unit, 300: Monitoring screen, 310: Lane status display unit, 320: Live video display unit, 330: Detection history display unit, 340: Detection video display unit, 351~354: Abnormal behavior confirmation button, 360: Refueling permission button

Claims

1. a monitoring device that uses a learning model that has learned about the action of inserting a fuel nozzle into a fuel filler opening as a predetermined action of a fuel dispenser, detects the predetermined action from video data after it has been detected that the fuel nozzle has been removed from the metering machine, and determines whether or not to permit fuel dispense; a terminal device that accepts a refueling permission operation that permits refueling; a control device that sets the metering machine to a state where fuel can be supplied in response to the fuel supply permission operation received by the terminal device; A self-service refueling monitoring system characterized by comprising a learning device that, if the judgment result by the monitoring device regarding whether refueling is permitted does not match the operation content on the terminal device, judges that the full insertion of the refueling nozzle into the refueling opening has been missed, and tags normal behavior with the video data and stores it as learning data for re-learning the learning model of the specified behavior.

2. the terminal device accepts an operation to designate an area to be cut out from the video data; The self-service refueling monitoring system according to claim 1 , wherein the learning device tags the region extracted from the video data as normal behavior.

Images