Self-service fueling management system

Abstract

To provide a self-service oil feeding management system capable of easily confirming presence of an abnormal behavior of a user and reducing checking work by an attendant, a control method for the self-service oil feeding management system, and a self-service oil feeding station.SOLUTION: A self-service oil feeding management system 100 of this invention comprises: means for picking up an image of a user MW who feeds oil; means for determining abnormality, normality, or unknown regarding a behavior of the user based on the picked-up image; means for terminating or prohibiting the oil supply when determination of abnormality or unknown is made for each oil feeding process by the user; and a management terminal X that displays the image serving as evidence of the determination.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a self-service refueling management system, a control method for a self-service refueling management system, and a self-service refueling station. [Background technology]

[0002] At conventional self-service gas stations, staff in the control room of the station building would monitor customers' gasoline filling behavior, safety, and dangerous behavior visually or on a surveillance monitor, and issue gasoline filling permission using a control device (see, for example, Patent Documents 1 and 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-269018 [Patent Document 2] Japanese Patent Application Publication No. 10-264995 Summary of the Invention [Problem to be solved by the invention]

[0004] At the conventional self-service gas stations mentioned above, staff in the control room would visually or via a monitoring monitor monitor the customer's refueling behavior, safety, and risky behavior, and would then issue permission to refuel using a control device.However, if it was difficult for the staff to check visually or via a monitoring monitor, they would have to go to the site to check, which was a time-consuming task.

[0005] In such cases, it is possible to use a monitoring device (AI robot) to determine whether or not the user is behaving abnormally from the captured images, but in cases where there is little so-called training data or the user accidentally behaves irregularly, the monitoring device alone may not be able to make an accurate determination even if the behavior is not abnormal.

[0006] Therefore, the present invention has been made in consideration of the above problems, and aims to provide a self-service refueling management system, a control method for a self-service refueling management system, and a self-service refueling station that can easily determine whether a user is behaving abnormally by having at least a monitoring device (AI robot) determine whether a user is behaving abnormally from images captured, and by having store staff check the information (images, etc.) that served as the basis for that determination, thereby reducing the effort required of store staff to check. [Means for solving the problem]

[0007] A self-service fueling system according to one aspect of the present invention comprises: means for capturing an image of a user refueling; A means for determining whether the behavior of the user is abnormal, normal, or unknown based on the captured image; A means for stopping or prohibiting refueling when the user's behavior in each refueling process is determined to be abnormal or unknown; a management terminal that displays an image that serves as the basis for the determination; The present invention is characterized by comprising:

[0008] In the self-service refueling management system, The management terminal may have a means for displaying, in addition to the image on which the determination is based, a live image of the area surrounding the image on which the determination is based.

[0009] In the self-service refueling management system, The control terminal can be operated to stop or prohibit fuel supply.

[0010] In the self-service refueling management system, When the refueling process is stopped or prohibited, a means for canceling the stop or prohibition by operating the management terminal may be provided.

[0011] In the self-service refueling management system, The system may have a means for determining whether the content of a user's actions is abnormal, normal, or unknown for each refueling procedure, and displaying a history of the determined content of the user's actions on the management terminal.

[0012] In the self-service refueling management system, The means for making the judgment may use training data from an image database created by machine learning to make a judgment as to whether the image is abnormal, normal, or unknown, and once the judgment is complete, update the image database by machine learning using the images that have been judged to be abnormal.

[0013] A gas station according to one aspect of the present invention includes: means for capturing an image of a user refueling; A means for determining whether the behavior of the user is abnormal, normal, or unknown based on the captured image; A means for stopping or prohibiting refueling when the user's behavior in each refueling process is determined to be abnormal or unknown; a management terminal that displays an image of the basis for the determination; The present invention is characterized by comprising: [Effects of the Invention]

[0014] According to the self-service refueling management system of the present invention, the monitoring device determines abnormal behavior of the user, and the store staff can check the results of the determination and easily determine whether or not there is an abnormality, thereby reducing the effort required for the staff to check inside the gas station. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a self-service refueling management system 100 in a self-service refueling station according to this embodiment. [Figure 2] FIG. 2 is a diagram showing an example of the configuration of the display unit H of the management terminal (mobile terminal) X of the self-service refueling management system 100 shown in FIG. [Figure 3] FIG. 3 is a conceptual diagram showing an example of the overall flow of a control method for self-service refueling in the self-service refueling management system 100 shown in FIG. [Figure 4] FIG. 4 is a conceptual diagram showing another example of the overall flow of the control method for self-service refueling by the self-service refueling management system 100 shown in FIG. [Figure 5A] FIG. 5A is a flow diagram showing an example of a specific flow at the time of refueling in the self-service refueling management system 100 shown in FIG. [Figure 5B] FIG. 5B is a flowchart following FIG. 5A, showing an example of a specific flow of the self-service refueling management system 100 shown in FIG. 1 during refueling. [Figure 5C] FIG. 5C is a flowchart following FIG. 5B, showing an example of a specific flow at the time of refueling in the self-service refueling management system 100 shown in FIG.

[0016] [Self-service fueling management system] Here, Fig. 1 is a diagram showing an example of the configuration of a self-service refueling management system 100 in a self-service refueling station according to this embodiment. Also, Fig. 2 is a diagram showing an example of the configuration of a display unit H of a management terminal (mobile terminal) X of the self-service refueling management system 100 shown in Fig. 1.

[0017] In this invention, self-service refueling refers to refueling performed by a customer themselves through their own operations, such as stopping the vehicle in a parking area, selecting the type and amount of fuel to be refueled, removing static electricity, removing the oil gun (nozzle) from the dispenser, inserting the oil gun into the fuel filler opening, starting refueling, finishing refueling, returning the oil gun to the dispenser, and leaving the store, all of which are performed solely by the customer themselves after confirmation by a store clerk or other person.

[0018] 1 is configured to, for example, permit a user (fueler) MW to operate a dispenser D to refuel a vehicle W, and then perform refueling. The types of fuel dispensed at the self-service refueling station include gasoline, diesel, LPG, and the like.

[0019] The self-service fuel supply management system 100 includes, for example, a monitoring device Y( judgement means and display means), a management terminal (mobile terminal) X, and a control device SSC (fuel stop The system is equipped with a monitoring camera C (imaging means), a sensor S, a dispenser D, and a server Z. In the drawings and the following description, an example of a mobile terminal X such as a tablet carried by a store clerk MX is described as one form of the management terminal, but the same explanation applies when the management terminal is a terminal installed (fixed) at a predetermined location.

[0020] [Surveillance Camera] In addition, the surveillance camera C is a means for capturing images of the user refueling, and for example, as shown in Figure 1, it captures at least the refueling action performed by the user MW, and outputs the image data obtained by capturing the image to the monitoring device Y.

[0021] This surveillance camera C includes, for example, a camera for dispenser D that captures at least the area around dispenser D and acquires image data, and a camera for stopping area that captures at least the stopping area where vehicle W being refueled by dispenser D stops and acquires image data.

[0022] Here, the "periphery of the dispenser" refers to the range within which the dispenser body can be monitored, and the range around the image that served as the basis for the determination.

[0023] Furthermore, a camera for capturing an image of the fuel filler opening of the vehicle W may be included.

[0024] [sensor] In addition, the sensor S is configured to detect at least one of sound, temperature, odor, light, or oil leakage that occurs in conjunction with the refueling or other actions of the user MW, for example, as shown in Figure 1, and output the detection results to the monitoring device Y.

[0025] The sensor S is installed, for example, around the dispenser D and / or in a stopping area where the vehicle W to be supplied with fuel by the dispenser D stops.

[0026] It should be noted that the "stopping area" here refers to the area where the vehicle stops to refuel, and the image capture includes the entire vehicle and the area where the user normally moves to refuel.

[0027] [Monitoring device] Furthermore, the monitoring device Y judges whether the user's behavior is abnormal, normal, or unknown based on the captured image. judgement The image processing unit 100 also has a means for displaying on the management terminal X an image that is the basis for the judgment.

[0028] For example, as shown in Fig. 1, the dispenser D and the control device SSC are installed apart from each other. In particular, the monitoring device Y is installed in a building within the self-service gas station, for example.

[0029] The monitoring device Y monitors the refueling of the vehicle W using the dispenser D operated by the user MW, and determines whether the user MW is behaving abnormally, as described below, or normally, or whether it is impossible to determine the behavior of the user MW, based on the analysis results of the image data obtained by capturing at least the behavior of the user MW.

[0030] Furthermore, the monitoring device Y is configured to determine whether the current user MW is behaving abnormally by using a determination model that has been machine-learned using, as training data, behavioral image data relating to images of past user operations of fueling a vehicle W using a dispenser D. The behavioral image data is used to determine whether the past user's behavior is abnormal or normal, and the classified data is used as training data. After the determination is completed, the image on which the determination was made is registered in the image database Z1 and becomes part of the training data.

[0031] In particular, the monitoring device Y calculates the degree of similarity between, for example, the behavioral image data relating to the behavioral image of the user MW operating the dispenser D to refuel the vehicle W obtained this time and the behavioral image data determined to be abnormal behavior from the previously described classified data, and determines whether the user MW is behaving abnormally based on the calculation results.

[0032] The "approximation" can be calculated using any known method. For example, it can be performed as described in Japanese Patent Publication No. 4701100. Specifically, still portions are removed from the video, and a feature vector is calculated from the moving portions of the video using cubic higher-order local autocorrelation features. Alternatively, the feature vector can be calculated using optical flow. These feature vectors are statistically processed and displayed as features. One method involves comparing these features with features of normal behavior, and determining that the closer the values ​​are, the higher the degree of approximation. Several thresholds can then be set for the difference in values, and a range within this range can be determined as normal, a range within this range as unknown, and a range outside this range as abnormal. Alternatively, a multilayer neural network, as described in Japanese Patent Publication No. 2019-96179, can be used to determine user behavior.

[0033] Furthermore, the monitoring device Y is configured to determine whether the user MW is behaving abnormally based on the analysis results of the image data and the detection results detected by the sensor S, for example, as shown in Figure 1.

[0034] This determination is generally made for each refueling process. Here, the refueling process refers to the divided actions a customer performs at a self-service gas station to refuel, such as parking the vehicle in a parking area, selecting the type and amount of fuel to refuel, discharging static electricity, removing the oil gun from the dispenser, inserting the oil gun's pipe into the fuel filler opening, starting refueling, completing refueling, removing the oil gun's pipe from the fuel filler opening, returning the oil gun to the dispenser, and leaving the station. Specifically, refueling begins when oil is dispensed from the oil gun's pipe by operating the oil gun's lever.

[0035] Furthermore, regarding the aforementioned determination of normal, abnormal, or unknown by the monitoring device Y, for example, if more than half of the oil gun pipe is inserted into the fuel filler neck, it is determined to be "normal," if the tip of the oil gun pipe is removed from the fuel filler neck, it is determined to be "abnormal," and if only the tip of the oil gun pipe is inserted into the fuel filler neck, it is determined to be "unknown."

[0036] It should be noted that abnormal behavior may be judged not only for each step but also for the overall risk of smoking, etc.

[0037] The monitoring device Y, for example, uses signal data (big data) previously acquired by the sensor S to determine whether the current user MW is exhibiting abnormal behavior based on a comparison with data on abnormal behavior of past users.

[0038] The image used to determine the behavior of the user MW at this time is transmitted from the monitoring device Y to the management terminal X and displayed. A live image of the user MW may also be displayed on the management terminal X, but this control may be performed by the monitoring device Y.

[0039] As shown in FIG. 1, the monitoring device Y includes, for example, an image analysis device Y1, an interface Y2, and a wireless device Y3.

[0040] The image analysis device Y1 analyzes image information of the user MW's behavior at least before and / or after the start of refueling, acquired by a surveillance camera C, as shown in Figure 1, and determines whether the user MW is behaving abnormally.

[0041] More specifically, the image analysis device Y1 is equipped with an AI robot (artificial intelligence), and uses behavioral image data relating to images of past user operations of fueling a vehicle W using a dispenser D to determine whether the current user MW is behaving abnormally or not, using a machine-learned determination model as training data, which is classified by determining whether the past user's behavior is abnormal or normal, based on the behavioral image data. Once the determination is complete, the image on which the determination was made is registered in the image database Z1 and becomes part of the training data.

[0042] 1, the wireless device Y3 is configured to perform wireless communication with at least the mobile terminal X.

[0043] Also, the interface Y2 is interposed between, for example, the surveillance camera, the sensor S, the control device SSC, the image analysis device Y1, and the wireless device Y3, as shown in FIG.

[0044] [server] 1, the server Z is configured to perform machine learning on the behavioral image data relating to an image of a past user's operation of refueling a vehicle W using a dispenser D, which has been acquired in the past by the monitoring device Y, and classifies the behavioral image data as abnormal or normal behavior of the past user, and uses the classified data as training data. The server Z is also configured to update the program of the above-mentioned determination model.

[0045] As shown in FIG. 1, the server Z includes an image database Z1 and an image recognition and machine learning unit Z2.

[0046] The image database Z1 stores behavior image data relating to images of the behavior of refueling the vehicle W using the dispenser D by a user's operation in the past, which has been acquired by the monitoring device Y in the past.

[0047] In addition, the image recognition and machine learning unit Z2 determines and classifies the previously described behavioral image data as abnormal or normal behavior of the user in the past, and uses the classified data as training data through machine learning, thereby updating the program of the previously described judgment model.

[0048] The classification data of the image database may not only be the registration of multiple images themselves, but also the user's behavior quantified and classified into abnormal, normal, and unknown categories.

[0049] [Administrative device] As mentioned above, in this embodiment, as one form of management terminal, an example of a mobile terminal X such as a tablet carried by a store clerk MX is described, but the same description applies when the management terminal is a terminal installed (fixed) at a predetermined location.

[0050] The mobile terminal X, which is the management terminal, displays the images that served as the basis for determining whether the behavior was abnormal, unknown, or normal. It may also display live images. Here, the live image means an image around the image that is the basis for the determination.

[0051] For example, as shown in FIG. 1, the information acquired by the monitoring device Y (such as image data acquired from a monitoring camera C and the detection results of a sensor S), the judgment results of the monitoring device Y, and the images on which the judgment is based are acquired from the monitoring device Y, output, and displayed, and are operated by a store attendant MX who monitors the refueling of a vehicle W.

[0052] This mobile terminal X is configured to receive information including the determination result from the monitoring device Y, for example, by wireless communication, and transmit information (instructions) according to the operation of the store clerk MX to the monitoring device Y. Note that this mobile terminal X may also be configured to receive information including the determination result from the monitoring device Y, and transmit information (instructions) according to the operation of the store clerk MX to the monitoring device Y, by wired communication.

[0053] Furthermore, the mobile terminal X may transmit information (instructions) according to the operation of the store clerk MX to the control device SSC as necessary.

[0054] In particular, this mobile terminal X is provided separately from the monitoring device Y and the control device SSC, for example, as shown in Fig. 1. In particular, this mobile terminal X is operated by a store clerk MX at a remote location connected to the monitoring device Y via wireless communication or radio communication.

[0055] Here, as shown in FIG. 1, the mobile terminal X includes, for example, a display unit H, an operation unit A, a communication unit T, a notification unit E, and a control unit CNT.

[0056] The display unit H of the mobile terminal X is configured to display predetermined information such as the information acquired by the monitoring device Y and the determination results of the monitoring device Y. The display unit H is, for example, a touch panel. Note that the display unit H of the mobile terminal X may be configured to display the same content as that displayed on the display unit G1 of the control device SSC, which will be described later.

[0057] The display unit H of this mobile terminal X includes, for example, a lane control status display unit CV, a live image display unit LV, a judgment image (image that served as the basis for the judgment) display unit JV, a history display unit LG, and a button display unit SW, as shown in FIG.

[0058] The lane control status display section CV displays a dispenser mark CV1 indicating each dispenser D and a fuel supply position CV2 of each lane.

[0059] The display of each refueling position CV2 includes the control status of the dispenser D of each lane (whether it is waiting for refueling or on standby, the type of fuel, the amount of fuel to be refueled, and other order details, etc.).

[0060] The live image display unit LV is also configured to display live images based on image data obtained by capturing images of the user's MW's actions in order to relay the status of fueling of the vehicle W.

[0061] Furthermore, the judgment image display unit JV is configured to display judgments and judgment images based on image data obtained by capturing images of the behavior of the user MW, which are the basis for the judgment results of the monitoring device Y.

[0062] The history display unit LG is configured to display the history (determination time, event log) of the contents of the actions of the user MW for each refueling process determined by the monitoring device Y.

[0063] The button display unit SW also displays buttons such as a "Confirm" button, a "Allow" button, a "Stop" button, and a "Cancel Stop" button, which correspond to the operation unit A used by the store clerk MX to input operations.

[0064] In this way, for example, the mobile terminal X acquires the information acquired by the monitoring device Y and the judgment results of the monitoring device Y from the monitoring device Y and displays them on the history display unit LG, and also displays a live image (around the image data) based on the image data obtained by capturing the actions of the user MW in order to relay the refueling status of the vehicle W on the live image display unit LV.

[0065] Furthermore, the mobile terminal X is configured to display a judgment image based on image data obtained by capturing an image of the user MW's behavior, which is the basis for the judgment result of the monitoring device Y, on a judgment image display unit JV.

[0066] In addition, the mobile terminal X acquires from the monitoring device Y the history (event log) of the contents of the behavior of the user MW determined by the monitoring device Y, and displays the determined image on the determination image display unit JV and also on the history display unit LG.

[0067] In addition, the mobile terminal X is configured to display on the history display unit LG the details of the behavior of the user MW newly determined by the monitoring device Y so as to add them to the history (event log such as determination time LG1, determination content LG2, determination result LG3, etc.).

[0068] In addition, the mobile terminal X acquires from the monitoring device Y the time at which the monitoring device Y determines whether the user MW is behaving abnormally, and displays this on the history display unit LG together with the history (event log) of the content of the user MW's behavior determined by the monitoring device Y.

[0069] In this way, the store clerk MX can check the determination image, history, etc. displayed on the display unit H of the mobile terminal X to more accurately determine whether or not there is an abnormality.

[0070] The operation unit A of the mobile terminal X is used by the store clerk MX to input operations, for example, as shown in Fig. 1. This operation unit A is, for example, a touch panel portion (button display unit SW) that displays buttons for various operations and selections, such as a "Confirm" button, an "Allow" button, and a "Stop" button, for operation by the store clerk MX, and in this case, the touch panel also functions as the display unit H, as described above.

[0071] Furthermore, the communication unit T of the mobile terminal X is adapted to transmit and receive data such as predetermined information and commands by wireless communication with the monitoring device Y and / or the control device SSC.

[0072] Furthermore, the notification unit E of the mobile terminal X is configured to notify the store clerk MX of predetermined information. The notification unit E is configured to output a predetermined buzzer sound. Alternatively, the notification unit E may correspond to a portion of the display unit H that outputs predetermined information.

[0073] In addition, the control unit CNT of the mobile terminal X is configured, for example, by a computer such as a CPU, and controls the operation of the display unit H, operation unit A, notification unit E, and communication unit T to perform the necessary processing of the mobile terminal X.

[0074] [Control device] In addition, the control device SSC is a means for stopping or prohibiting refueling when the monitoring device Y determines that the user's behavior is abnormal or unknown or when a stop signal or prohibition signal is sent by operation from the management terminal X, and is a means for canceling the stop or prohibition by operation from the management terminal X.

[0075] Here, stopping refueling means that the supply of fuel that is currently being refueled is prevented when a stop signal is issued during refueling, and prohibition means that the supply of fuel is prevented before refueling.

[0076] For example, as shown in FIG. 1, the fuel supply to a vehicle W using a dispenser D operated by a user MW is controlled in accordance with at least the determination result of a monitoring device Y and the operation of a mobile terminal X by a store clerk MX.

[0077] For example, the control device SSC may execute refueling of the vehicle W using the dispenser D in response to an operation input by the store clerk MX to the mobile terminal X instructing permission for refueling by the user MW.

[0078] On the other hand, the control device SSC is configured to suspend (prohibit) refueling of the vehicle W using the dispenser D in accordance with the judgment result of the monitoring device Y and the operational input of the store clerk MX to the mobile terminal X, which instructs the user MW to prohibit or stop refueling.

[0079] When the monitoring device Y determines that the user MW is behaving normally, the control device SSC allows the user MW to refuel the vehicle W using the dispenser D. For safety reasons, a system may be added in which a store clerk issues permission.

[0080] The "normal actions" of the user MW mentioned above include, for example, the user MW ordering refueling, the user MW removing static electricity, or inserting the nozzle of the dispenser D into the fuel tank opening of the user MW's vehicle W, or other normal actions required for refueling.

[0081] The control device SSC is configured to stop or prohibit refueling of the vehicle W using the dispenser D if the monitoring device Y determines that the user MW is behaving abnormally or in an undeterminable manner during each refueling process.

[0082] The judgment results of the monitoring device Y, information on the operation input of the mobile terminal X, and other information (such as image data captured by the monitoring camera C and the detection results of the sensor S) are transmitted to this control device SSC via the interface Y2 of the monitoring device Y, but for example, the necessary information may be transmitted from the mobile terminal X to this control device SSC.

[0083] Here, as shown in FIG. 1, the control device SSC includes, for example, a display unit G1, an operation unit G2, a control unit G3, and a notification unit G4.

[0084] The display unit G1 of the control device SSC is configured to display predetermined information such as the information acquired by the monitoring device Y and the determination results of the monitoring device Y. This display unit G1 is, for example, a touch panel. As described above, the display unit G1 of the control device SSC may be configured to display the same information (image) as the display unit H of the mobile terminal X.

[0085] The operation unit G2 of the control device SSC is used by the store clerk MX to input operations, for example, as shown in Fig. 1. This operation unit G2 is a touch panel that displays buttons for various operations and selections, such as a "Confirm" button, an "Allow" button, and a "Stop" button, which are operated by the store clerk MX, and in this case, the touch panel also functions as the display unit G2 of the control device SSC, as described above.

[0086] The notification unit G4 of the control device SSC is configured to notify the store clerk MX of predetermined information. This notification unit G4 is configured to output a predetermined buzzer sound. Alternatively, this notification unit G4 may correspond to a portion of the display unit G1 that outputs predetermined information.

[0087] In addition, the control unit G3 of the control device SSC is configured, for example, by a computer such as a CPU, and controls the operations of the display unit G1, operation unit G2, and notification unit G4 to perform the necessary processing of the control device SSC.

[0088] [Overall flow concept of the self-service refueling management system] Here, an example of the concept of the flow of the overall control method for self-service refueling in the self-service refueling management system 100 having the above-described configuration and functions will be described.

[0089] FIG. 3 is a conceptual diagram showing an example of the overall flow of a control method for self-service refueling in the self-service refueling management system 100 shown in FIG.

[0090] First, for example, as shown in Figure 3, a user MW arrives at the self-service refueling station in a vehicle W, stops the vehicle W in a stopping area adjacent to a dispenser D, gets out of the vehicle, and then begins refueling (step S1).

[0091] At this time, the monitoring camera C starts capturing images of at least the refueling action performed by the user MW, and outputs the captured image data to the monitoring device Y. Furthermore, the sensor S starts detecting at least one of sound, temperature, odor, light, or oil leakage that occurs in conjunction with the refueling or action performed by the user MW, and outputs the detection results to the monitoring device Y.

[0092] Then, the monitoring device Y determines whether the user MW is behaving abnormally based on the analysis results of the image data obtained by the monitoring camera C capturing the user MW's behavior and the detection results of the sensor S (step S2).

[0093] Then, if the monitoring device Y determines in step S2 that the user MW is taking normal actions for refueling, the control device SSC permits the user MW to refuel based on this determination result (step S3), and the refueling by the user MW is completed (step S4).

[0094] On the other hand, if the monitoring device Y determines in step S2 that the user MW is behaving abnormally, the control device SSC prohibits the user MW from refueling based on this determination result (step S5). At this time, an alarm is displayed on the mobile terminal X of the store clerk MX, along with an image of the basis for the determination, which is then displayed on the management terminal. The store clerk MX then checks the image and visually confirms whether or not the behavior is abnormal. If the behavior is abnormal, the control device SSC again prohibits the user MW from refueling the vehicle W using the dispenser D based on an operation input to the mobile terminal X (or the control device SSC) by the store clerk MX who has checked the image, and if the visual inspection reveals no problems, refueling is resumed (step S6).

[0095] Furthermore, if the monitoring device Y cannot determine the behavior of the user MW in step S2, the control device SSC prohibits the user MW from refueling based on this determination result (step S7), and the store clerk confirms the image that was the basis for the determination. Thereafter, based on the operation input by the store clerk MX to the mobile terminal X (or the control device SSC), the control device SSC again prohibits or permits refueling of the vehicle W using the dispenser D (step S8), and refueling by the user MW is completed.

[0096] Moreover, Fig. 4 is a conceptual diagram showing another example of the concept of the flow of the overall control method for self-service refueling in the self-service refueling management system 100 shown in Fig. 1. The flow of steps S1 to S6 shown in Fig. 4 is similar to the flow of steps S1 to S6 shown in Fig. 3, but steps S11 to S13 are different from the steps shown in Fig. 3.

[0097] That is, for example, as shown in Figure 4, a user MW arrives at the self-service refueling station in a vehicle W, stops the vehicle W in a stopping area adjacent to a dispenser D, gets out of the vehicle, and then begins refueling (step S1).

[0098] At this time, the monitoring camera C starts capturing images of at least the refueling action performed by the user MW, and outputs the captured image data to the monitoring device Y. Furthermore, the sensor S starts detecting at least one of sound, temperature, odor, light, or oil leakage that occurs in conjunction with the refueling action performed by the user MW, and outputs the detection results to the monitoring device Y.

[0099] Then, the monitoring device Y determines whether the user MW is behaving abnormally based on the analysis results of the image data obtained by the monitoring camera C capturing the user MW's behavior and the detection results of the sensor S (step S2).

[0100] Then, if the monitoring device Y determines in step S2 that the user MW is taking normal actions for refueling, the control device SSC permits the user MW to refuel based on this determination result (step S3), and the refueling by the user MW is completed (step S4).

[0101] On the other hand, if the monitoring device Y determines in step S2 that the user MW is behaving abnormally, the control device SSC stops the user MW from refueling based on this determination result (step S5), and the image displayed on the mobile terminal X as the basis for the determination is displayed on the display unit H. Thereafter, based on the operational input by the store clerk MX to the mobile terminal X (or the control device SSC), the control device SSC again stops or allows refueling of the vehicle W using the dispenser D (step S6).

[0102] Furthermore, if the monitoring device Y is unable to determine the behavior of the user MW in step S2, the store clerk MX determines whether the user MW is behaving abnormally based on image data and visual inspection of the refueling behavior performed by the user MW displayed on the display H of the mobile terminal X (step S11), and based on the operation input by the store clerk MX to the mobile terminal X (or the control device SSC), the control device SSC allows the refueling of the vehicle W using the dispenser D (step S12), and the refueling by the user MW is completed (step S13).

[0103] 3 and 4, the monitoring device (AI robot) Y determines whether the user MW is behaving abnormally, and the store clerk can easily determine whether the user is behaving abnormally by checking the information (images, etc.) that served as the basis for the determination, thereby reducing the effort required of the store clerk to check. Furthermore, while the monitoring device (AI robot) performs control, the store clerk MX can determine the user MW's abnormal behavior as needed, allowing for more accurate decisions regarding refueling.

[0104] In particular, if the monitoring device Y determines that the user MW is behaving normally for refueling, the control device SSC allows the user MW to refuel based on this determination result, and the clerk MX can complete the refueling by the user MW (steps S2 to S4).

[0105] In particular, by having the monitoring device (AI robot) Y determine abnormal behavior of the user MW, risk factors that go unnoticed by the store staff are reduced, enabling safer operation of the self-service gas station.

[0106] In addition, this has the effect of expanding the range of allowable staff placement, while satisfying the Fire Service Act regulations that require staff to be on standby in a location where they can visually monitor the refueling point.

[0107] [Specific flow of refueling using the self-service refueling management system 100] Next, an example of a specific flow of the self-service refueling management system 100 during refueling will be described, which specifically implements the concept of the control flow of the self-service refueling management system 100 shown in FIGS. 3 and 4 described above.

[0108] Here, Fig. 5A is a flow diagram showing an example of a specific flow when refueling in the self-service refueling management system 100 shown in Fig. 1. Fig. 5B is a flow diagram following Fig. 5A, showing an example of a specific flow when refueling in the self-service refueling management system 100 shown in Fig. 1. Fig. 5C is a flow diagram following Fig. 5B, showing an example of a specific flow when refueling in the self-service refueling management system 100 shown in Fig. 1. Note that the following description will mainly focus on the operation of the mobile terminal X operated by the store clerk MX in the self-service refueling management system 100, but the operation of the control device SSC will also be described in a similar manner.

[0109] For example, a user MW arrives at the self-service gas station in a vehicle W, stops the vehicle W in a stopping area near a dispenser D, gets out of the vehicle, and starts refueling. At this time, the monitoring camera C starts capturing images of at least the user MW's actions of refueling, and outputs the captured image data to the monitoring device Y.

[0110] Then, for example, as shown in FIG. 5A, the mobile terminal X displays "Waiting" on the lane control status display section CV2 of the display section H (step S101 in FIG. 5A).

[0111] Thereafter, the mobile terminal X switches the live image display unit LV of the display unit H to display the live image of each lane in sequence at predetermined time intervals (step S102 in FIG. 5A).

[0112] Thereafter, the monitoring device Y determines whether or not the fuel supply setting for the dispenser D of the lane by the user MX has been completed (step S103 in FIG. 5A).

[0113] In this step S103, if the monitoring device Y determines that the user MX has not completed the fuel supply setting for the dispenser D of the lane in question, the process returns to step S102, and the mobile terminal X causes the live image display unit LV of the display unit H to sequentially switch and display the live images of each lane at predetermined intervals.

[0114] On the other hand, in step S103, if the monitoring device Y determines that the user MX has completed the fuel supply setting for the dispenser D in the lane, the mobile terminal X determines whether or not it has acquired from the monitoring device Y the determination result and information regarding whether or not the user MX has touched the static electricity removal sheet of the dispenser D and completed static electricity removal (step S104 in Figure 5A).

[0115] Then, after obtaining the determination result and information regarding whether the static electricity removal has been completed in step S104 from the monitoring device Y, the mobile terminal X obtains the determination result and information regarding whether the user MX has removed the fuel nozzle from the dispenser D of the lane (step S105 in Figure 5A).

[0116] Then, when the mobile terminal X acquires from the monitoring device Y the determination result and information that the user MX has removed the fuel nozzle from the dispenser D in the lane, the mobile terminal X displays the determination result and information (determination image, determination time, etc.) on the history display section LG and the determination image display section JV of the display section H (step S106 in Figure 5A).

[0117] Then, the mobile terminal X switches the display of the refueling position CV2 corresponding to the lane on the lane control status display unit CV from “Waiting” to “Waiting for refueling,” and displays a live image of the lane on the live image display unit LV (step S107 in FIG. 5A).

[0118] Then, the mobile terminal X causes the notification unit E to notify the store clerk MX (step S108 in FIG. 5A).

[0119] In response to this notification from notification unit E, store clerk MX can easily determine whether or not there is an abnormality by checking the judgment image and history, etc. displayed on history display unit LG, judgment image display unit JV, and live image display unit LV of display unit H of mobile terminal X, thereby reducing the effort required of store clerk.

[0120] Thereafter, the mobile terminal X determines whether the refueling position CV2 corresponding to the lane in the lane control status display section CV of the display section H has been selected by the operation of the user MX, for example, as shown in FIG. 5B (step S109 in FIG. 5B).

[0121] Then, when the mobile terminal X determines that the refueling position CV2 corresponding to the lane in the lane control status display section CV of the display section H has been selected by the operation of the user MX, it stops the notification section E from notifying the store clerk MX (step S110 of Figure 5B).

[0122] Thereafter, the mobile terminal X selectively displays (for example, by displaying with a thick black frame) the fuel filling position CV2 corresponding to the lane selected by the operation of the user MX in the lane control status display section CV of the display section H (step S111 in FIG. 5B).

[0123] Thereafter, when the user MX starts to insert the nozzle of the dispenser D into the vehicle W, the monitoring device Y determines whether or not the insertion of the nozzle of the dispenser D into the vehicle W by the user MX is an abnormal behavior.

[0124] Then, the mobile terminal X determines whether or not it has acquired from the monitoring device Y the determination result and information regarding whether or not the user MX has touched the static electricity removal sheet of the dispenser D and completed static electricity removal (step S112 in FIG. 5B).

[0125] Then, when the mobile terminal X acquires the determination result and information regarding whether or not static electricity removal has been completed from the monitoring device Y in step S112, it displays the determination image information from the determination result and information regarding whether or not static electricity removal has been completed so that it overwrites it on the determination image display unit LV, and also displays the determination time, determination content, etc. on the history display unit LG (step S113 in Figure 5B).

[0126] This allows the store clerk MX to easily determine whether or not there is an abnormality by checking the judgment image and history displayed on the history display section LG and judgment image display section JV of the display section H of the mobile terminal X, thereby reducing the amount of work required by the store clerk.

[0127] Then, the store clerk MX operates the permission button displayed on the button display unit SW of the display unit H of the mobile terminal X (step S114 in FIG. 5B), which causes the control device SSC to drive the pump of the dispenser D (step S115 in FIG. 5B). This allows the user MW to start refueling the vehicle W using the dispenser D. Then, the monitoring device Y sequentially determines whether or not the user MW is behaving abnormally after refueling has started.

[0128] Thereafter, the mobile terminal X cancels the selection and display (for example, display with a thick black frame) of the fuel filling position CV2 corresponding to the lane in the lane control status display section CV of the display section H (step S116 in FIG. 5B).

[0129] Thereafter, the mobile terminal X sequentially switches and displays the live images of each lane on the live image display section LV of the display section H (step S117 in FIG. 5B).

[0130] 5C, for example, if the monitoring device Y does not detect any abnormal behavior of the user MW after the start of refueling (step S119 in FIG. 5C), the mobile terminal X confirms that the nozzle of the dispenser D is hung on the nozzle hanger (step S120 in FIG. 5C), which causes the control device SSC to stop driving the pump of the dispenser D (step S121 in FIG. 5C). This ends the refueling of the vehicle W by the user MW using the dispenser D.

[0131] Thereafter, the mobile terminal X erases the judgment image, judgment result, and information corresponding to the lane from the display unit H (step S122).

[0132] On the other hand, if the monitoring device Y determines in the aforementioned step S119 that the user MW is behaving abnormally after refueling has begun, and the abnormal behavior is that the nozzle has become detached from the fuel filler cap (step S123 in Figure 5), the control device SSC stops the operation of the pump of the dispenser D (step S124 in Figure 5C).

[0133] Thereafter, the mobile terminal X displays a live image of the lane on the live image display unit LV, overwrites the information on the judgment image on the judgment image display unit LV, and additionally displays the judgment time, judgment content, etc. on the history display unit LG (step S125 in FIG. 5C).

[0134] This allows the store clerk MX to easily determine whether or not there is an abnormality by checking the judgment image and history displayed on the history display section LG and judgment image display section JV of the display section H of the mobile terminal X, thereby reducing the amount of work required by the store clerk.

[0135] After that, the store clerk MX, having confirmed that there is no problem with the behavior of the user MX, operates the confirmation button displayed on the button display unit SW of the display unit H of the mobile terminal X (step S126 in FIG. 5C), which causes the display of "Confirmed" to be added to the history display unit LG, indicating that the store clerk MX has confirmed the behavior (step S127 in FIG. 5C).

[0136] After that, the store clerk MX, having confirmed that the user MX's behavior is normal, operates the stop release button displayed on the button display unit SW of the display unit H of the mobile terminal X (step S128 in FIG. 5C), which causes the control device SSC to restart the pump of the dispenser D (step S129 in FIG. 5C). This allows the user MW to resume refueling the vehicle W using the dispenser D, and the process returns to the above-mentioned step S119.

[0137] On the other hand, in the above-mentioned step S123, if the monitoring device Y determines that the abnormal behavior of the user MW after the start of refueling is other than the nozzle being detached from the fuel filler cap (for example, user MX smoking), the mobile terminal X displays a live image of the lane on the live image display unit LV, overwrites the information of the judgment image on the judgment image display unit LV, and additionally displays the judgment time, judgment content, etc. on the history display unit LG (step S130 in Figure 5C).

[0138] This allows the store clerk MX to easily determine whether or not there is an abnormality by checking the judgment image and history displayed on the history display section LG and judgment image display section JV of the display section H of the mobile terminal X, thereby reducing the amount of work required by the store clerk.

[0139] After that, the store clerk MX, who has confirmed that there is no problem with the behavior of the user MX, operates the confirmation button displayed on the button display unit SW of the display unit H of the mobile terminal X (step S131 in FIG. 5C), and the display of "Confirmed" indicating that the store clerk MX has confirmed is additionally displayed on the history display unit LG (step S132 in FIG. 5C). Then, the process returns to the above-mentioned step S119, and the user MW continues to refuel the vehicle W using the dispenser D.

[0140] As described above, according to one embodiment of the self-service refueling management system of the present invention, the display unit of the mobile terminal displays images captured by a surveillance camera, information on the judgment made by the monitoring device (AI robot), and the judgment results.This allows the store clerk to determine whether or not there is an abnormality by checking not only the captured images but also the judgment made by the monitoring device (AI robot), thereby reducing the amount of checking required by the store clerk and further increasing safety within the gas station.

[0141] Furthermore, the store clerk can compare the live image (current) taken by the surveillance camera on the display of the mobile terminal X with the judgment image (past) used for the judgment by the monitoring device, and can easily determine whether or not there is an abnormality.

[0142] Furthermore, as mentioned above, the monitoring device (AI robot) Y determines and controls abnormal behavior of the user MW, and if necessary, the store clerk MX determines and controls abnormal behavior of the user MW, thereby enabling labor savings and effective use of personnel.

[0143] In particular, by having the monitoring device (AI robot) Y determine abnormal behavior of the user MW, risk factors that go unnoticed by the store staff are reduced, enabling safer operation of the self-service gas station.

[0144] In addition, this has the effect of expanding the range of allowable staff placement, while satisfying the Fire Service Act regulations that require staff to be on standby in a location where they can visually monitor the refueling point.

[0145] Furthermore, in the examples of Figures 5A and 5B above, if the judgment result is that the behavior is abnormal and the nozzle is detached from the fuel filler opening, the pump is stopped, but this is not limited to this. If the judgment result is unclear, the store clerk may be notified with an alarm or image, etc., and the pump may be stopped if the store clerk does not press the confirmation button, or of course the pump may be stopped if the store clerk does not press the confirmation button after a predetermined time has passed since the warning.

[0146] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention described in the claims and their equivalents. [Explanation of symbols]

[0147] 100 Self-service fuel management system Y Monitoring device Y1 Image Analysis Device Y2 interface Y3 radio device X Mobile Device MX clerk H display A Control section T Communications Department E. Notification Department CNT control unit C. Surveillance camera S sensor D Dispenser Z Server SSC control unit G1 display G2 control panel G3 control unit G4 Notification Department W vehicle MW users (fuelers) CV lane control status display LV live image display section JV judgment image display section LG History display section SW button display

Claims

1. an imaging means for imaging a user filling up with fuel in a fueling lane; a determining means for determining whether the behavior of the user is abnormal, normal, or unknown based on the image obtained by the imaging means; prohibition means for stopping or prohibiting refueling when the determination means determines that the user's behavior for each refueling process is abnormal or unclear; a portable management terminal that displays not only the image of the basis of the judgment but also a live image of the area surrounding the image that is the basis of the judgment; Including, The self-service refueling management system is characterized in that the portable management terminal displays the judgment results of the judgment means based on the image obtained by the imaging means, indicating whether the user's behavior is abnormal, normal, or unknown.

2. The self-service refueling management system according to claim 1, characterized in that the mobile management terminal displays an operation section that can stop or prohibit refueling on the screen that displays the live image and the judgment result.

3. 3. The self-service refueling management system according to claim 1, further comprising a means for canceling the stop or prohibition by operating the mobile management terminal when the refueling process is stopped or prohibited.

4. A self-service refueling management system as described in any one of claims 1 to 3, characterized in that it has a means for determining whether the content of a user's behavior is abnormal, normal, or unknown for each refueling procedure, and displaying a history of the determined content of the user's behavior on the mobile management terminal.

5. A self-service refueling management system as described in any one of claims 1 to 4, characterized in that the judgment means uses a judgment model that has been machine-learned using data from an image database classified as abnormal or normal as training data to judge whether the image is abnormal, normal, or unknown, and registers the image used for the judgment in the image database.

6. an imaging means for imaging a user filling up with fuel in a fueling lane; a determining means for determining whether the behavior of the user is abnormal, normal, or unknown based on the image obtained by the imaging means; prohibition means for stopping or prohibiting refueling when the determination means determines that the user's behavior for each refueling process is abnormal or unclear; a portable management terminal that displays not only the image of the basis of the judgment but also a live image of the area surrounding the image that is the basis of the judgment; Including, the mobile management terminal displays the result of the determination made by the determination means based on the image obtained by the imaging means, whether the behavior of the user is abnormal, normal, or unknown; A gas station using a system characterized in that refueling can be stopped or prohibited by operating the mobile management terminal.

Images