Self-service fuel supply permission system, control method of self-service fuel supply permission system, and self-service fuel supply station

Abstract

To achieve labor saving, more accurate determination of fuel supply permission, and effective use of personnel by allowing a monitoring device to determine abnormal behavior of a user and perform control.SOLUTION: A self-fueling permission system includes a monitoring device that monitors fueling to a vehicle using a dispenser by an operation of a user and determines whether or not the user behaves abnormally based on an analysis result obtained by analyzing image data obtained by imaging at least behavior of the user, and the self-fueling permission system acquires information acquired by the monitoring device and a determination result of the monitoring device from the monitoring device and outputs the information and the determination result. A fuel supply system includes a management terminal operated by a monitoring person who monitors fuel supply to a vehicle, and a control device which controls fuel supply to the vehicle using a dispenser operated by a user according to at least a determination result of a monitoring device and an operation of the management terminal by the monitoring person.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a self-service refueling permission system, a control method for a self-service refueling permission system, and a self-service refueling station. [Background technology]

[0002] FIG. 8 is a diagram showing an example of the configuration of a fuel supply system 100A in a conventional self-service fuel supply station.

[0003] In conventional self-service gas stations, for example, as shown in Figure 8, a supervisor in the control room of the gas station building would visually observe customers refueling or monitor their refueling behavior, safety, and risky behavior, and then issue permission for refueling using a control device.

[0004] Additionally, if surveillance camera footage revealed any abnormalities with the dispenser or a suspicious user while refueling, a monitor would visit the self-service refueling station to recheck the situation.

[0005] For example, Patent Document 1 discloses a control technique in which a monitoring camera is provided in a fuel tanker and image information displayed on a monitoring monitor is used to permit or prohibit fueling.

[0006] In Patent Document 1, images are captured by a surveillance camera and monitored by a self-service gas station monitor on an office monitor. This monitoring prevents users from making mistakes when filling up their gas tank and also makes it possible to detect abnormal behavior. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2000-238899 Summary of the Invention [Problem to be solved by the invention]

[0008] As mentioned above, at conventional self-service gas stations, a supervisor in a control room would visually observe customers refueling or monitor their refueling behavior, safety, and risky behavior on a surveillance monitor, and then issue permission for refueling using a control device. And the watchman When monitoring with a surveillance monitor, it is possible that abnormal user behavior that is difficult for humans to detect may be overlooked.

[0009] Therefore, the present invention has been made in consideration of the above-mentioned problems, and at least a monitoring device (AI robot) determines abnormal behavior of a user and controls it. By doing so, Accurate fuel permit Judgment The present invention aims to provide a self-service refueling permission system, a control method for the self-service refueling permission system, and a self-service refueling station that enable the self-service refueling permission system. [Means for solving the problem]

[0010] A self-service refueling permission system according to one aspect of the present invention includes: A self-service fueling permission system permits a user to refuel a vehicle using a dispenser operated by the user, a monitoring device that determines whether the user is behaving abnormally based on an analysis result of image data obtained by capturing at least the user's behavior; a control device that controls the fuel supply to the vehicle using the dispenser in response to the determination result of the monitoring device; a monitoring camera that captures an image of the refueling action performed by the user and outputs the captured image data to the monitoring device; a thermal camera capable of detecting the temperature of a heat source as a thermal image, acquiring heat source coordinates of an area having a temperature equal to or higher than a preset threshold value as the coordinates of the heat source in a thermal image based on the temperature of an area overlapping an area photographed by the monitoring camera, and outputting heat source coordinate data corresponding to the acquired heat source coordinates to the monitoring device; The monitoring device compares the discrimination result of the behavioral image data with the heat source coordinate data output by the thermal camera, and determines that the user is behaving abnormally if the judgment coordinates of an area of ​​an image suspected of being a cigarette in the behavioral image corresponding to the behavioral image data overlap with the heat source coordinates of the thermal image corresponding to the heat source coordinate data; When the monitoring device determines that the user is behaving abnormally, the control device prohibits the vehicle from being refueled using the dispenser based on the determination result of the monitoring device. It is characterized by: If the monitoring device determines that the user is carrying a cigarette based on the behavioral image data, it determines that the user is behaving abnormally. It is characterized by: When the monitoring device cannot determine that the user is carrying a cigarette based on the behavioral image data, the monitoring device compares the determination result of the behavioral image data with the heat source coordinate data output by the thermal camera. It is characterized by:

[0011] A control method for a self-service refueling permission system according to one aspect of the present invention includes: A self-service refueling permission system that permits a user to refuel a vehicle using a dispenser operated by the user includes: a monitoring device that determines whether the user is behaving abnormally based on an analysis result of image data obtained by capturing at least the user's actions; a control device that controls the user's refueling of the vehicle using the dispenser operated by the user in accordance with the determination result of the monitoring device; a monitoring camera that captures images of the user's refueling actions and outputs the captured image data to the monitoring device; and a thermal camera that is capable of detecting the temperature of a heat source as a thermal image, and that acquires heat source coordinates by using an area with a temperature equal to or higher than a preset threshold as the coordinates of the heat source in a thermal image based on the temperature of an area overlapping an area captured by the monitoring camera, and outputs heat source coordinate data corresponding to the acquired heat source coordinates to the monitoring device. A control method for a self-service refueling permission system comprising: The monitoring device compares the discrimination result of the behavioral image data with the heat source coordinate data output by the thermal camera, and determines that the user is behaving abnormally if the judgment coordinates of an area of ​​an image suspected of being a cigarette in the behavioral image corresponding to the behavioral image data overlap with the heat source coordinates of the thermal image corresponding to the heat source coordinate data; When the monitoring device determines that the user is behaving abnormally, the control device prohibits the vehicle from being refueled using the dispenser based on the determination result of the monitoring device. It is characterized by:

[0012] A self-service refueling station according to one aspect of the present invention includes: At a self-service fueling station that uses a self-service fueling permission system that allows users to refuel vehicles using a dispenser operated by the user, The self-service refueling permission system includes: a monitoring device that determines whether the user is behaving abnormally based on an analysis result of image data obtained by capturing at least the user's behavior; a control device that controls the fuel supply to the vehicle using the dispenser in response to the determination result of the monitoring device; a monitoring camera that captures an image of the refueling action performed by the user and outputs the captured image data to the monitoring device; a thermal camera capable of detecting the temperature of a heat source as a thermal image, acquiring heat source coordinates of an area having a temperature equal to or higher than a preset threshold value as the coordinates of the heat source in a thermal image based on the temperature of an area overlapping an area photographed by the monitoring camera, and outputting heat source coordinate data corresponding to the acquired heat source coordinates to the monitoring device; The monitoring device compares the discrimination result of the behavioral image data with the heat source coordinate data output by the thermal camera, and determines that the user is behaving abnormally if the judgment coordinates of an area of ​​an image suspected of being a cigarette in the behavioral image corresponding to the behavioral image data overlap with the heat source coordinates of the thermal image corresponding to the heat source coordinate data; When the monitoring device determines that the user is behaving abnormally, the control device prohibits the vehicle from being refueled using the dispenser based on the determination result of the monitoring device. It is characterized by: [Effects of the Invention]

[0013] According to the self-service refueling permission system, the control method for the self-service refueling permission system, and the self-service refueling station of the present invention, the monitoring device determines abnormal behavior of the user and controls the user. This allows for more accurate fueling permission decisions. It can be made into. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a self-service refueling permission system 100 in a self-service refueling station according to this embodiment. [Figure 2] FIG. 2 is a conceptual diagram showing an example of the overall flow of a control method for self-service refueling by the self-service refueling permission system 100 shown in FIG. [Figure 3] FIG. 3 is a conceptual diagram showing another example of the concept of the flow of the overall control method for self-service refueling by the self-service refueling permission system 100 shown in FIG. [Figure 4] FIG. 4 is a flow chart showing an example of a specific flow at the time of refueling in the self-service refueling permission system 100 shown in FIG. [Figure 5] FIG. 5 is a flow chart showing a first modified example of the specific flow of the self-service refueling permission system 100 shown in FIG. 1 during refueling. [Figure 6]FIG. 6 is a flow chart showing a second modified example of the specific flow of the self-service refueling permission system 100 shown in FIG. 1 during refueling. [Figure 7] FIG. 7 is a flow chart showing a third modified example of the specific flow of the self-service refueling permission system 100 shown in FIG. 1 during refueling. [Figure 8] FIG. 8 is a diagram showing an example of the configuration of a fuel supply system 100A in a conventional self-service fuel supply station. [Figure 9] Figure 9 is a flow diagram showing an example of a flow for determining whether a user is behaving abnormally by determining whether the user is carrying cigarettes in the self-service refueling permission system 100 shown in Figure 1. [Figure 10] Figure 10 is a flow diagram showing an example of a flow for determining whether a user is behaving abnormally by determining whether or not the user has a portable fuel can that is permitted for refueling by law in the self-service refueling permission system 100 shown in Figure 1. [Figure 11] FIG. 11 is a flow chart showing an example of a flow for detecting the presence or absence of a nozzle of a dispenser in the self-service fuel supply permission system 100 shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0015] A self-service refueling permission system, a control method for a self-service refueling permission system, and a self-service refueling station according to embodiments of the present disclosure will be described in detail below with reference to the drawings. Note that the embodiments described below are examples of embodiments of the present disclosure, and the present disclosure should not be construed as being limited to these embodiments. Furthermore, terms specifying shapes, geometric conditions, and their degrees, as well as values ​​of lengths and angles, used in this specification, are not limited to their strict meanings. In addition, parts having similar functions or similar functions in the drawings referenced in this embodiment are denoted by the same or similar symbols, and repeated explanations thereof may be omitted. Furthermore, the dimensional ratios in the drawings may differ from the actual ratios for convenience of explanation, and some components may be omitted from the drawings.

[0016] [Self-service refueling permission system] FIG. 1 is a diagram showing an example of the configuration of a self-service refueling permission system 100 in a self-service refueling station according to this embodiment.

[0017] 1 is configured to permit a user (fueler) MW to operate a dispenser D to refuel a vehicle W, and then refuel the vehicle W. The types of fuel dispensed at the self-service refueling station include gasoline, diesel, kerosene, and the like.

[0018] 1, the self-service refueling permission system 100 includes a monitoring device Y, a control terminal (mobile terminal) X, a control device SSC, a monitoring camera C, a sensor S, a dispenser D, and a server Z. Note that in the drawings and the following description, an example of a mobile terminal X such as a tablet carried by a monitor MX will be described as one form of the control terminal, but the same explanation applies when the control terminal is a terminal installed (fixed) at a predetermined location.

[0019] [Surveillance Camera] Furthermore, the monitoring camera C is configured to capture at least an image of the refueling action performed by the user MW, and output the image data obtained by capturing the image to the monitoring device Y, as shown in FIG.

[0020] This surveillance camera C includes, for example, a camera for dispenser D that captures at least the area around dispenser D and acquires image data, and a camera for stopping area that captures at least the stopping area where vehicle W being refueled by dispenser D stops and acquires image data.

[0021] Here, the "periphery of the dispenser" means the range in which the dispenser body can be monitored.

[0022] [sensor] In addition, the sensor S is configured to detect at least one of sound, temperature, odor, light, or oil leakage that occurs in conjunction with the refueling or other actions of the user MW, for example, as shown in Figure 1, and output the detection results to the monitoring device Y. The sensor S is installed, for example, around the dispenser D and / or in a stopping area where the vehicle W to be supplied with fuel by the dispenser D stops.

[0023] It should be noted that the "stopping area" here refers to the area where the vehicle stops to refuel, and the image capture includes the entire vehicle and the area where the user normally moves to refuel.

[0024] [Monitoring device] Furthermore, the monitoring device Y is installed apart from the dispenser D and the control device SSC, for example, as shown in Fig. 1. In particular, the monitoring device Y is installed in a building within the self-service gas station, for example.

[0025] The monitoring device Y monitors the refueling of the vehicle W using the dispenser D operated by the user MW, and determines whether the user MW is engaging in abnormal behavior as described below based on the results of analysis of image data obtained by capturing at least the user MW's behavior.

[0026] Furthermore, this monitoring device Y is configured to determine whether the current user MW is behaving abnormally by using a machine-learned judgment model as training data, for example, based on behavioral image data relating to images of the behavior of refueling a vehicle W using a dispenser D operated by a past user, which has been previously acquired and classified as abnormal or normal behavior of the past user.

[0027] In particular, the monitoring device Y calculates the degree of similarity between, for example, the behavioral image data relating to the behavioral image of the user MW operating the dispenser D to refuel the vehicle W obtained this time and the behavioral image data determined to be abnormal behavior from the previously described classified data, and determines whether the user MW is behaving abnormally based on the calculation results.

[0028] The "approximation" can be calculated using any known method. For example, it can be performed as described in Japanese Patent Publication No. 4701100. Specifically, still portions are removed from the video, and a feature vector is calculated from the moving portions of the video using cubic higher-order local autocorrelation features. Alternatively, the feature vector can be calculated using optical flow. These feature vectors are statistically processed and displayed as features. One method involves comparing these features with features of normal behavior, and determining that the closer the values ​​are, the higher the degree of approximation. Several thresholds can then be set for the difference in values, and a range within this range can be determined as normal, a range within this range as unknown, and a range outside this range as abnormal. Alternatively, a multilayer neural network, as described in Japanese Patent Publication No. 2019-96179, can be used to determine user behavior.

[0029] Furthermore, the monitoring device Y is configured to determine whether the user MW is behaving abnormally based on the analysis results of the image data and the detection results detected by the sensor S, for example, as shown in Figure 1.

[0030] The monitoring device Y, for example, uses signal data (big data) previously acquired by the sensor S to determine whether the current user MW is exhibiting abnormal behavior based on a comparison with data on abnormal behavior of past users.

[0031] As shown in FIG. 1, the monitoring device Y includes, for example, an image analysis device Y1, an interface Y2, and a wireless device Y3.

[0032] The image analysis device Y1 analyzes image information of the user MW's behavior at least before and / or after the start of refueling, acquired by a surveillance camera C, as shown in Figure 1, and determines whether the user MW is behaving abnormally.

[0033] More specifically, this image analysis device Y1 is equipped with an AI robot (artificial intelligence), and uses behavioral image data relating to images of the behavior of refueling a vehicle W using a dispenser D operated by a past user, which has been previously acquired, to determine whether the behavior of the past user is abnormal or normal, and classifies the classified data using a machine-learned judgment model as training data to determine whether the current user MW is behaving abnormally.

[0034] 1, the wireless device Y3 is configured to perform wireless communication with at least the mobile terminal X.

[0035] Also, as shown in FIG. 1, the interface Y2 is interposed between the surveillance camera, the sensor S, the control device SSC, the image analysis device Y1, and the wireless device Y3, for example.

[0036] [server] 1, the server Z is configured to perform machine learning on the behavioral image data relating to an image of a past user's operation of refueling a vehicle W using a dispenser D, which has been acquired in the past by the monitoring device Y, and classifies the behavioral image data as abnormal or normal behavior of the past user, and uses the classified data as training data. The server Z is also configured to update the program of the above-mentioned determination model.

[0037] As shown in FIG. 1, the server Z includes an image database Z1 and an image recognition and machine learning unit Z2.

[0038] The image database Z1 stores behavior image data relating to images of the behavior of refueling the vehicle W using the dispenser D by a user's operation in the past, which has been acquired by the monitoring device Y in the past.

[0039] In addition, the image recognition and machine learning unit Z2 determines and classifies the previously described behavioral image data as abnormal or normal behavior of the user in the past, and uses the classified data as training data through machine learning, thereby updating the program of the previously described judgment model. [Management device (mobile device)] As mentioned above, in this embodiment, as one form of management terminal, an example of a mobile terminal X such as a tablet carried by a monitor MX is described, but the same description applies when the management terminal is a terminal installed (fixed) at a predetermined location.

[0040] As shown in FIG. 1, the mobile terminal X, which is the management terminal, acquires and outputs information acquired by the monitoring device Y (such as image data acquired from the monitoring camera C and the detection results of the sensor S) and the judgment results of the monitoring device Y, and is operated by an observer MX who monitors the refueling of the vehicle W.

[0041] This mobile terminal X is configured to receive information including the determination result from the monitoring device Y, for example, by wireless communication, and transmit information (instructions) according to the operation of the monitor MX to the monitoring device Y. Note that this mobile terminal X may also be configured to receive information including the determination result from the monitoring device Y, and transmit information (instructions) according to the operation of the monitor MX to the monitoring device Y, by wired communication.

[0042] The mobile terminal X may transmit information (instructions) according to the operation of the monitor MX to the control device SSC as necessary.

[0043] In particular, this mobile terminal X is provided separately from the monitoring device Y and the control device SSC, for example, as shown in Fig. 1. In particular, this mobile terminal X is operated by a monitor MX at a remote location connected to the monitoring device Y via wireless communication or radio communication.

[0044] Here, as shown in FIG. 1, the mobile terminal X includes, for example, a display unit H, an operation unit A, a communication unit T, and a control unit CNT.

[0045] The display unit H is configured to display predetermined information such as information acquired by the monitoring device Y and the determination results of the monitoring device Y. The display unit H is, for example, a touch panel. Note that the display unit H of the mobile terminal X may be configured to display the same content as that displayed on the display unit G1 of the control device SSC, which will be described later.

[0046] The operation unit A is used by the observer MX to input operations. This operation unit A is, for example, a touch panel that displays buttons for various operations and selections, such as a "confirm button," "permission button," and "stop button," for the observer MX to operate. In this case, the touch panel also functions as the display unit H, as described above.

[0047] The communication unit T is also configured to transmit and receive data such as predetermined information and commands via wireless communication with the monitoring device Y and / or the control device SSC.

[0048] The control unit CNT is configured by a computer such as a CPU, and controls the operations of the display unit H, operation unit A, and communication unit T to execute the necessary processing of the mobile terminal X.

[0049] [Control device] Furthermore, as shown in FIG. 1, the control device SSC is configured to control the refueling of a vehicle W using a dispenser D operated by a user MW in accordance with at least the determination result of the monitoring device Y and the operation of a mobile terminal X by a monitor MX.

[0050] For example, the control device SSC is configured to execute refueling of the vehicle W using the dispenser D in response to an operation input by the monitor MX to the mobile terminal X instructing permission for refueling by the user MW.

[0051] On the other hand, the control device SSC is configured to suspend (prohibit) refueling of the vehicle W using the dispenser D in response to the judgment result of the monitoring device Y and the operational input of the monitor MX to the mobile terminal X instructing the user MW to prohibit refueling.

[0052] In particular, if the monitoring device Y does not determine that the user MW is behaving abnormally, the control device SSC is configured to prohibit or permit refueling of the vehicle W using the dispenser D based on operational input by the monitor MX to the mobile terminal X.

[0053] Here, a case in which the monitoring device Y does not determine that the user MW is behaving abnormally means at least one of a case in which the monitoring device Y is unable to determine that the user MW is behaving abnormally, or a case in which the monitoring device Y determines that the user MW is behaving normally.

[0054] The "abnormal behavior" of user MW mentioned above refers to, for example, user MW's possession of a firearm, or user MW's smoking, or other abnormal (dangerous) behavior when refueling (such as refueling without removing static electricity).

[0055] Furthermore, the "normal behavior" of the user MW mentioned above refers to, for example, the user MW ordering refueling, the user MW removing static electricity, or inserting the nozzle of the dispenser D into the fuel tank opening of the user MW's vehicle W, or other normal behavior required for refueling.

[0056] The judgment results of the monitoring device Y, information on the operation input of the mobile terminal X, and other information (such as image data captured by the monitoring camera C and the detection results of the sensor S) are transmitted to this control device SSC via the interface Y2 of the monitoring device Y, but for example, the necessary information may be transmitted from the mobile terminal X to this control device SSC.

[0057] Here, as shown in FIG. 1, the control device SSC includes, for example, a display unit G1, an operation unit G2, and a control unit G3.

[0058] The display unit G1 is configured to display predetermined information such as information acquired by the monitoring device Y and the determination results of the monitoring device Y. The display unit G1 is, for example, a touch panel.

[0059] The operation unit G2 is used by the observer MX to input operations. This operation unit G2 is a touch panel that displays buttons for various operations and selections, such as a "confirm button," "permission button," and "stop button," for the observer MX to operate. In this case, the touch panel also functions as the display unit G1, as described above.

[0060] The control unit G3 is configured by a computer such as a CPU, and controls the operations of the display unit G1 and the operation unit G2 to execute the necessary processing of the control device SSC.

[0061] [Overall flow concept of self-service refueling control method for the self-service refueling permission system] Here, an example of the concept of the flow of the overall control method for self-service refueling, which is the control method for the self-service refueling permission system 100 having the above-described configuration and functions, will be described.

[0062] FIG. 2 is a conceptual diagram showing an example of the overall flow of a control method for self-service refueling by the self-service refueling permission system 100 shown in FIG.

[0063] First, for example, as shown in Figure 2, a user MW arrives at the self-service refueling station in a vehicle W, stops the vehicle W in a stopping area adjacent to a dispenser D, gets out of the vehicle, and then begins refueling (step S1).

[0064] At this time, the monitoring camera C starts capturing images of at least the refueling action performed by the user MW, and outputs the captured image data to the monitoring device Y. Furthermore, the sensor S starts detecting at least one of sound, temperature, odor, light, or oil leakage that occurs in conjunction with the refueling or action performed by the user MW, and outputs the detection results to the monitoring device Y.

[0065] Then, the monitoring device Y determines whether the user MW is behaving abnormally based on the analysis results of the image data obtained by the monitoring camera C capturing the user MW's behavior and the detection results of the sensor S (step S2).

[0066] Then, if the monitoring device Y determines in step S2 that the user MW is taking normal actions for refueling, the control device SSC permits the user MW to refuel based on this determination result (step S3), and the refueling by the user MW is completed (step S4).

[0067] On the other hand, if the monitoring device Y determines in step S2 that the user MW is behaving abnormally, the control device SSC prohibits the user MW from refueling based on this determination result (step S5), and then, based on the operation input by the monitor MX to the mobile terminal X (or the control device SSC), the control device SSC again prohibits or allows the vehicle W to be refueled using the dispenser D (step S6).

[0068] Furthermore, if the monitoring device Y is unable to determine the behavior of the user MW in step S2, the control device SSC prohibits the user MW from refueling based on this determination result (step S7), and then, based on the operational input by the monitor MX to the mobile terminal X (or the control device SSC), the control device SSC again prohibits or permits refueling of the vehicle W using the dispenser D (step S8), and refueling by the user MW is completed.

[0069] Moreover, Fig. 3 is a conceptual diagram showing another example of the concept of the flow of the overall control method for self-service refueling by the self-service refueling permission system 100 shown in Fig. 1. The flow of steps S1 to S6 shown in Fig. 3 is similar to the flow of steps S1 to S6 shown in Fig. 2, but steps S11 to S13 are different from the steps shown in Fig. 2.

[0070] That is, for example, as shown in Figure 3, a user MW arrives at the self-service refueling station in a vehicle W, stops the vehicle W in a stopping area adjacent to a dispenser D, gets out of the vehicle, and then begins refueling (step S1).

[0071] At this time, the monitoring camera C starts capturing images of at least the refueling action performed by the user MW, and outputs the captured image data to the monitoring device Y. Furthermore, the sensor S starts detecting at least one of sound, temperature, odor, light, or oil leakage that occurs in conjunction with the refueling action performed by the user MW, and outputs the detection results to the monitoring device Y.

[0072] Then, the monitoring device Y determines whether the user MW is behaving abnormally based on the analysis results of the image data obtained by the monitoring camera C capturing the user MW's behavior and the detection results of the sensor S (step S2).

[0073] Then, if the monitoring device Y determines in step S2 that the user MW is taking normal actions for refueling, the control device SSC permits the user MW to refuel based on this determination result (step S3), and the refueling by the user MW is completed (step S4).

[0074] On the other hand, if the monitoring device Y determines in step S2 that the user MW is behaving abnormally, the control device SSC prohibits the user MW from refueling based on this determination result (step S5), and then, based on the operation input by the monitor MX to the mobile terminal X (or the control device SSC), the control device SSC again prohibits or allows the vehicle W to be refueled using the dispenser D (step S6).

[0075] Furthermore, if the monitoring device Y is unable to determine the behavior of the user MW in step S2, the monitor MX determines whether the user MW is behaving abnormally based on image data and visual inspection of the refueling behavior performed by the user MW displayed on the display H of the mobile terminal X (step S11), and based on the operation input by the monitor MX to the mobile terminal X (or the control device SSC), the control device SSC allows the vehicle W to be refueled using the dispenser D (step S12), and the refueling by the user MW is completed (step S13).

[0076] Thus, according to the control flow of the self-service refueling permission system 100 shown in Figures 2 and 3 above, the monitoring device (AI robot) Y determines and controls the abnormal behavior of the user MW, and if necessary, the monitor MX determines and controls the abnormal behavior of the user MW, thereby enabling labor savings and effective use of personnel.

[0077] In particular, if the monitoring device Y determines that the user MW is behaving normally for refueling, the control device SSC allows the user MW to refuel based on this determination result, and the monitor MX can complete the refueling by the user MW (steps S2 to S4).

[0078] In particular, by having the monitoring device (AI robot) Y determine abnormal behavior of user MW, risk factors that are not noticed by monitors can be reduced, enabling safer operation of self-service gas stations.

[0079] In addition, this has the effect of expanding the range of allowable deployment of inspectors, which satisfies the regulations under the Fire Service Act, which require inspectors to be on standby in a location where they can visually observe the filling area.

[0080] [Specific flow of refueling using the self-service refueling permission system 100] Next, an example of a specific flow of the self-service refueling permission system 100 during refueling will be described, which specifically implements the concept of the control flow of the self-service refueling permission system 100 shown in FIGS. 2 and 3. FIG.

[0081] FIG. 4 is a flow chart showing an example of a specific flow at the time of refueling in the self-service refueling permission system 100 shown in FIG.

[0082] First, for example, as shown in FIG. 4, the monitoring device Y determines whether or not the user MW is behaving abnormally before starting refueling (step S101).

[0083] Then, if the monitoring device Y determines in step S101 that the user MW is behaving abnormally before refueling begins, the mobile terminal X and the control device SSC display an image of the user MW's behavior determined to be abnormal, and a confirmation button for the monitor MX to operate, on their respective display units H and G1 (touch panels) (step S102).

[0084] Thereafter, the monitor MX checks the behavior of the user MW visually or from the image on the display H of the mobile terminal X (step S103).

[0085] Thereafter, the monitor MX determines whether or not the user MW is behaving abnormally before the start of refueling (step S104).

[0086] Then, if the observer MX determines in step S104 that the user MW is behaving abnormally before the start of refueling, the observer MX warns the user MW by audio guidance or by using the observer MX (step S105).

[0087] Thereafter, the user MW responds to the warning and stops the abnormal behavior (step S106).

[0088] Here, after the monitor MX determines in step S104 that the user MW has not behaved abnormally before the start of refueling, or after determining in step S106 that the user MW has stopped behaving abnormally, the monitor MX presses (operates) the confirmation button displayed on the display H, G1 of the mobile terminal X or the control device SSC (step S107).

[0089] Then, after the monitoring device Y determines in the aforementioned step S101 that the user MW is not behaving abnormally before the start of refueling, or in step S107, the monitor MX presses (operates) the confirmation button displayed on the display H, G1 of the mobile terminal X or the control device SSC before the start of refueling, the monitor MX presses (operates) the permission button displayed on the display H, G1 of the mobile terminal X or the control device SSC (step S108).

[0090] Then, when the permission button is pressed (operated), the control device SSC drives the pump of the dispenser D (step S109).

[0091] Thereafter, the user MW starts refueling the vehicle W using the dispenser D (step S110).

[0092] Then, the monitoring device Y determines whether or not the user MW is behaving abnormally after the start of refueling (step S111).

[0093] Then, if the monitoring device Y determines in step S111 that the user MW has not behaved abnormally after the start of refueling, it allows the user MW to continue refueling the vehicle W using the dispenser D and completes the refueling (step S117).

[0094] On the other hand, if the monitoring device Y determines in step S111 that the user MW is behaving abnormally after refueling has begun, the mobile terminal X and the control device SSC display on their respective display units H and G1 an image of the user MW's behavior determined to be abnormal, as well as a confirmation button and a stop button for the monitor MX to operate (step S112).

[0095] Thereafter, the monitor MX checks the behavior of the user MW visually or from the image on the display H of the mobile terminal X (step S113).

[0096] Thereafter, the monitor MX determines whether or not the user MW has behaved abnormally after the start of refueling (step S114).

[0097] Then, if the monitor MX determines in step S114 that the user MW is behaving abnormally after refueling has begun, the monitor MX presses (operates) the stop button displayed on the display H, G1 of the mobile terminal X or the control device SSC (step S115).

[0098] Then, in step S115, the monitor MX presses (operates) the stop button displayed on the display H, G1 of the mobile terminal X or the control device SSC, causing the control device SSC to stop the pump of the dispenser D, and the refueling is completed (ended) (step S116).

[0099] On the other hand, in the aforementioned step S114, if the monitor MX does not determine that the user MW is behaving abnormally after refueling has begun, the monitor MX presses (operates) the confirmation button displayed on the display unit H, G1 of the mobile terminal X or the control device SSC (step S118).

[0100] Then, in step S118, the monitor MX presses (operates) the confirmation button displayed on the display H, G1 of the mobile terminal X or the control device SSC, allowing the user MW to continue refueling the vehicle W using the dispenser D and completing the refueling (step S119).

[0101] In addition, in the above-mentioned step S114, if the monitor MX does not determine that the user MW is behaving abnormally after refueling has begun, the monitoring device Y may operate the stop button displayed on the display unit G1 of the control device SSC (the monitoring device Y may output a command to stop refueling to the control device SSC to activate the stop button) (step S120).

[0102] Then, in this step S114, the monitoring device Y may operate the stop button displayed on the display unit G1 of the control device SSC (activate the stop button), causing the control device SSC to stop the pump of the dispenser D and complete (end) the refueling (step S121).

[0103] In this way, the monitoring device (AI robot) Y determines and controls abnormal behavior of the user MW, and if necessary, the monitor MX determines and controls abnormal behavior of the user MW, thereby enabling labor savings and effective use of personnel.

[0104] In particular, by having the monitoring device (AI robot) Y determine abnormal behavior of user MW, risk factors that are not noticed by monitors can be reduced, enabling safer operation of self-service gas stations.

[0105] In addition, this has the effect of expanding the range of allowable deployment of inspectors, which satisfies the regulations under the Fire Service Act, which require inspectors to be on standby in a location where they can visually observe the filling area.

[0106] [First Modification] Next, a first modified example of the specific flow of the self-service refueling permission system 100 shown in FIG. 1 during refueling will be described.

[0107] FIG. 5 is a flow chart showing a first modified example of the specific flow of the self-service refueling permission system 100 shown in FIG. 1 during refueling.

[0108] First, as shown in FIG. 5, the monitoring device Y determines whether the user MW is behaving abnormally before starting refueling. (Step S201) .

[0109] If the monitoring device Y determines in step S201 that the user MW is behaving abnormally before refueling begins, the mobile terminal X and the control device SSC notify the user of the abnormality and display an image of the user MW's behavior determined to be abnormal, as well as a confirmation button for the monitor MX to operate, on their respective display units H and G1 (step S202).

[0110] Thereafter, before starting refueling, it is confirmed whether or not the confirmation button displayed on the display H, G1 of the mobile terminal X or the control device SSC has been pressed (operated) by the monitor MX (step S203).

[0111] Then, after it is confirmed in step S203 that the confirmation button has not been pressed before the start of refueling, if a predetermined period of time has not elapsed since the abnormality was notified (step S204), it is confirmed whether the confirmation button has been pressed (operated). Step S203 On the other hand, if the confirmation button has not been pressed (operated) for a predetermined period of time, the process returns to step S201 in which the monitoring device Y determines whether the user MW has engaged in abnormal behavior before the start of refueling.

[0112] On the other hand, in the above-mentioned step S201, if the monitoring device Y does not determine that the user MW is behaving abnormally before refueling begins, or in step S203, if the monitor MX presses (operates) the confirmation button displayed on the display H, G1 of the mobile terminal X or the control device SSC, it is determined whether the monitor MX has pressed (operated) the permission button displayed on the display H, G1 of the mobile terminal X or the control device SSC (step S205).

[0113] Then, in step S205, if the monitor MX presses (operates) the permission button displayed on the display unit H, G1 of the mobile terminal X or the control device SSC before refueling begins, the control device SSC will activate the pump of the dispenser D (step S206).

[0114] Then, the control device SSC drives the pump of the dispenser D in step S206 to start fuel supply, and then checks whether or not fuel supply has been completed by hanging the nozzle of the dispenser D on the nozzle hanger (step S207).

[0115] Then, when it is confirmed that the nozzle of the dispenser D is hung on the nozzle hanger, the fuel supply is completed.

[0116] On the other hand, in step S207, if the control device SSC cannot confirm that the nozzle of the dispenser D is hung on the nozzle hanger after refueling has started, the monitoring device Y determines whether the user MW has behaved abnormally after refueling has started (step S208).

[0117] Then, in step S208, if the monitoring device Y determines that the user MW has been behaving abnormally after the start of refueling, the control device SSC stops the pump of the dispenser D (step S209).

[0118] After that, the control device SSC stops the pump of the dispenser D, and then the mobile terminal X and the control device SSC report the abnormality and display an image of the user MW's abnormal behavior and a confirmation button for the monitor MX to operate on their respective display units H and G1 (step S210).

[0119] Then, after the control device SSC stops the pump of the dispenser D, the monitoring device Y checks whether the monitor MX has pressed (operated) the confirmation button displayed on the display unit H, G1 of the mobile terminal X or the control device SSC (step S211).

[0120] Then, if the monitoring device Y confirms in step S211 that the monitor MX has pressed (operated) the confirmation button displayed on the display H, G1 of the mobile terminal X or the control device SSC, it returns to step S205, and if it determines that the monitor MX has pressed (operated) the permission button displayed on the display H, G1 of the mobile terminal X or the control device SSC, then in step S206, the pump is driven and refueling is resumed.

[0121] On the other hand, the monitoring device Y checks whether the confirmation button has been pressed (operated) if the predetermined period of time has not elapsed since the abnormality was reported after checking that the confirmation button has not been pressed after the start of refueling. do Returning to step S211, if the confirmation button has not been pressed (operated) for a predetermined period of time, refueling ends (step S212).

[0122] In this way, First Modification Even in this case, the monitoring device (AI robot) Y can detect abnormal behavior of the user MW while refueling, reducing risk factors that the monitors are unaware of, enabling safer operation of the self-service refueling station.

[0123] In addition, this has the effect of expanding the range of allowable deployment of inspectors, which satisfies the regulations under the Fire Service Act, which require inspectors to be on standby in a location where they can visually observe the filling area.

[0124] [Second Modification] Next, a second modified example of the specific flow of the self-service refueling permission system 100 shown in FIG. 1 during refueling will be described.

[0125] FIG. 6 is a flow chart showing a second modified example of the specific flow of the self-service refueling permission system 100 shown in FIG. 1 during refueling.

[0126] First, as shown in FIG. 6, the monitoring device Y determines whether the user MW is behaving abnormally before starting refueling. (Step S301) .

[0127] If the monitoring device Y determines that the user MW is behaving abnormally before refueling begins, the mobile terminal X and the control device SSC will report the abnormality and display an image of the user MW's behavior that has been determined to be abnormal, as well as a confirmation button for the monitor MX to operate, on their respective display units H and G1 (step S302).

[0128] Then, the monitoring device Y checks whether or not the monitor MX has pressed (operated) the confirmation button displayed on the display H, G1 of the mobile terminal X or the control device SSC before starting refueling (step S303).

[0129] Then, in step S303, the monitoring device Y confirms that the confirmation button has not been pressed before the start of refueling, and if a predetermined period of time has not elapsed since the abnormality was notified, it checks whether the confirmation button has been pressed (operated). do Returning to step S302, if the confirmation button has not been pressed (operated) for a predetermined period of time, the monitoring device Y returns to step S301 where it determines whether the user MW has engaged in abnormal behavior before refueling begins (step S304).

[0130] On the other hand, in step S301, if the monitoring device Y does not determine that the user MW is behaving abnormally before refueling begins, or if the monitor MX presses (operates) the confirmation button displayed on the display H, G1 of the mobile terminal X or the control device SSC in step S303, the monitoring device Y determines whether the monitor MX has pressed (operated) the permission button displayed on the display H, G1 of the mobile terminal X or the control device SSC (step S305).

[0131] Then, in step S305, if the monitor MX presses (operates) the permission button displayed on the display unit H, G1 of the mobile terminal X or the control device SSC before refueling begins, the control device SSC will activate the pump of the dispenser D (step S306).

[0132] Then, the control device SSC drives the pump of the dispenser D to start fuel supply, and then checks whether or not fuel supply has been completed by hanging the nozzle of the dispenser D on the nozzle hanger (step S307).

[0133] Then, when the control device SSC can confirm that the nozzle of the dispenser D is hung on the nozzle hanger, the fuel supply is completed.

[0134] On the other hand, in step S307, if the control device SSC cannot confirm that the nozzle of the dispenser D is hung on the nozzle hanger after refueling has started, the monitoring device Y determines whether the user MW has behaved abnormally after refueling has started (step S308).

[0135] Then, in step S308, if the monitoring device Y determines that the user MW has been behaving abnormally after refueling has begun, the mobile terminal X and the control device SSC will report the abnormality and display on their respective display units H and G1 an image of the user MW's behavior that has been determined to be abnormal, as well as a confirmation button for the monitor MX to operate (step S309).

[0136] Then, the monitoring device Y checks whether the confirmation button displayed on the display H, G1 of the mobile terminal X or the control device SSC has been pressed (operated) by the monitor MX after refueling has started, and if the confirmation button displayed on the display H, G1 of the mobile terminal X or the control device SSC has been pressed (operated) by the monitor MX, the monitoring device Y returns to step S305, where it determines whether the permission button displayed on the display H, G1 of the mobile terminal X or the control device SSC has been pressed (operated) by the monitor MX (step S310).

[0137] Then, in step S310, if the confirmation button displayed on the display unit H, G1 of the mobile terminal X or the control device SSC is not pressed by the monitor MX for a predetermined period of time after the start of refueling, the control device SSC stops the pump of the dispenser D and completes refueling (step S311).

[0138] In this way, Second Variant Even in this case, the monitoring device (AI robot) Y can detect abnormal behavior of the user MW while refueling, reducing risk factors that the monitors are unaware of, enabling safer operation of the self-service refueling station.

[0139] In addition, this has the effect of expanding the range of allowable deployment of inspectors, which satisfies the regulations under the Fire Service Act, which require inspectors to be on standby in a location where they can visually observe the filling area.

[0140] [Third Modification] Next, a third modified example of the specific flow of the self-service refueling permission system 100 shown in FIG. 1 during refueling will be described.

[0141] FIG. 7 is a flow chart showing a third modified example of the specific flow of the self-service refueling permission system 100 shown in FIG. 1 during refueling.

[0142] First, as shown in FIG. 7, the monitoring device Y determines whether or not the user MW is behaving abnormally before starting refueling (step S401).

[0143] Then, in step S401, if the monitoring device Y determines that the user MW is behaving abnormally before refueling begins, the mobile terminal X and the control device SSC will report the abnormality and display an image of the user MW's behavior determined to be abnormal, as well as a confirmation button for the monitor MX to operate, on their respective display units H and G1 (step S402).

[0144] Thereafter, before starting refueling, it is confirmed whether or not the confirmation button displayed on the display unit H, G1 of the mobile terminal X or the control device SSC has been pressed (operated) by the monitor MX (step S403).

[0145] Then, after confirming that the confirmation button has not been pressed before refueling begins, if the predetermined period of time has not elapsed since the abnormality was reported, the monitoring device Y returns to step S403, where it confirms whether the confirmation button has been pressed (operated) or not; on the other hand, if the confirmation button has not been pressed (operated) for the predetermined period of time, the monitoring device Y returns to step S401, where it determines whether the user MW has been behaving abnormally before refueling begins (step S404).

[0146] On the other hand, in step S403, if the monitor MX presses (operates) the confirmation button displayed on the display unit H, G1 of the mobile terminal X or the control device SSC before starting refueling, the monitoring device Y determines whether the abnormal behavior of the user MW has been resolved (is not behaving abnormally) (step S405).

[0147] Then, in step S405, if the monitoring device Y determines that the abnormal behavior of the user MW has not been resolved (the abnormal behavior continues), the mobile terminal X and the control device SSC again notify the user MW to confirm the refueling, and display an image of the user MW's behavior that has been determined to be abnormal, and display a confirmation button for the monitor MX to operate on each display unit H, G1 (step S406).

[0148] On the other hand, if in step S405 the monitoring device Y determines that the abnormal behavior of the user MW has been resolved (the user is no longer engaging in abnormal behavior), the mobile terminal X and the control device SSC notify the user MW that the abnormal behavior of the user MW has been resolved (step S407).

[0149] Then, in the aforementioned step S401, if the monitoring device Y does not determine that the user MW is behaving abnormally before refueling begins, or in step S407, if the mobile terminal X and the control device SSC report that the user MW's abnormal behavior has been resolved, it is determined whether the monitor MX has pressed (operated) the permission button displayed on the display unit H, G1 of the mobile terminal X or the control device SSC (step S408).

[0150] Then, in step S408, if the monitor MX presses (operates) the permission button displayed on the display unit H, G1 of the mobile terminal X or the control device SSC before refueling begins, the control device SSC will activate the pump of the dispenser D (step S409).

[0151] Then, the monitoring device Y drives the pump of the dispenser D in step S409 to start fuel supply, and then checks whether or not fuel supply has been completed by hanging the nozzle of the dispenser D on the nozzle hanger (step S410).

[0152] Then, when the monitoring device Y can confirm that the nozzle of the dispenser D is hung on the nozzle hanger, the fuel supply is completed.

[0153] Then, if the monitoring device Y cannot confirm that the nozzle of the dispenser D is hung on the nozzle hanger after the start of refueling, the monitoring device Y determines whether the user MW has behaved abnormally after the start of refueling (step S411).

[0154] Then, in step S411, if the monitoring device Y determines that the user MW has been behaving abnormally after the start of refueling, the control device SSC stops the pump of the dispenser D (step S412).

[0155] Then, in step S412, after the control device SSC stops the pump of the dispenser D, the mobile terminal X and the control device SSC notify the abnormality, display an image of the user MW's behavior that has been determined to be abnormal, and display a confirmation button for the monitor MX to operate on their respective display units H, G1 (touch panels) (step S413).

[0156] Then, after the control device SSC stops the pump of the dispenser D, the monitoring device Y checks whether the monitor MX has pressed (operated) the confirmation button displayed on the display unit H, G1 of the mobile terminal X or the control device SSC (step S414).

[0157] Then, in step S414, if the monitoring device Y confirms that the monitor MX has pressed (operated) the confirmation button displayed on the display H, G1 of the mobile terminal X or the control device SSC, the process returns to step S409, the pump is driven, and refueling resumes.

[0158] On the other hand, after confirming in step S414 that the confirmation button has not been pressed after the start of refueling, if a predetermined period of time has not elapsed since the abnormality was notified, the monitoring device Y checks whether the confirmation button has been pressed (operated). do The process returns to step S414. If the confirmation button has not been pressed (operated) for a predetermined period of time, the monitoring device Y ends the refueling (step S415).

[0159] In this way, Third Variant Even in this case, the monitoring device (AI robot) Y can detect abnormal behavior of the user MW while refueling, reducing risk factors that the monitors are unaware of, enabling safer operation of the self-service refueling station.

[0160] In addition, this has the effect of expanding the range of allowable deployment of inspectors, which satisfies the regulations under the Fire Service Act, which require inspectors to be on standby in a location where they can visually observe the filling area.

[0161] As described above, a self-service refueling permission system according to one aspect of the present invention comprises a monitoring device that monitors the refueling of a vehicle using a dispenser operated by a user and determines whether the user is behaving abnormally based on the results of an analysis of image data obtained by capturing at least the user's behavior; a mobile terminal that acquires and outputs the information acquired by the monitoring device and the judgment results of the monitoring device from the monitoring device and is operated by an monitor who monitors the refueling of the vehicle; and a control device that controls the refueling of a vehicle using a dispenser operated by a user in accordance with at least the judgment results of the monitoring device and the operation of the mobile terminal by the monitor.

[0162] The mobile terminal is provided separately from the monitoring device and the control device, and if the monitoring device does not determine that the user is behaving abnormally, the control device will prohibit or allow the vehicle to be refueled using the dispenser based on the monitor's operational input to the mobile terminal.

[0163] As a result, according to one embodiment of the self-service refueling permission system of the present invention, the monitoring device (AI robot) determines and controls abnormal behavior of the user MW, and if necessary, a monitor determines and controls abnormal behavior of the user, thereby enabling labor savings, more accurate refueling permission decisions, and effective use of personnel.

[0164] In particular, by having the monitoring device (AI robot) detect abnormal behavior by users, risk factors that are not noticed by monitors can be reduced, enabling safer operation of self-service gas stations.

[0165] In addition, this has the effect of expanding the range of allowable deployment of inspectors, which satisfies the regulations under the Fire Service Act, which require inspectors to be on standby in a location where they can visually observe the filling area.

[0166] [Fourth Variation] Next, a fourth modified example of the specific flow when refueling in the self-service refueling permission system 100 shown in Fig. 1 will be described. In particular, an example of the flow for determining whether a user is behaving abnormally in the self-service refueling permission system 100 shown in Fig. 1 by determining whether the user is carrying cigarettes will be described.

[0167] Here, Figure 9 is a flow diagram showing an example of a flow for determining whether a user is behaving abnormally by determining whether the user is carrying cigarettes in the self-service refueling permission system 100 shown in Figure 1.

[0168] In the example shown in Figure 9, sensor S (corresponding to sensor S in Figure 1) is equipped with a thermal camera S that can detect the temperature of a heat source such as a lit cigarette as a thermal image (including detecting the brightness of the heat source).

[0169] First, as shown in FIG. 9, a monitoring camera C (corresponding to the monitoring camera C in FIG. 1) captures an image of the refueling action performed by the user MX, and outputs the captured action image data to the monitoring device Y shown in FIG. 1 (step S901).

[0170] Then, the image analysis device Y1 of the monitoring device Y acquires behavior image data from the monitoring camera C via the interface Y2.

[0171] Then, the image analysis device Y1 of the monitoring device Y analyzes the behavior image data output by the monitoring camera C (step S902).

[0172] Then, if the image analysis device Y1 of the monitoring device Y can determine based on the behavioral image data that the user MW is carrying a cigarette with a reliability level equal to or higher than a predetermined level, it determines that the user MW is behaving abnormally (step S903 YES).

[0173] Meanwhile, the thermal camera S acquires a thermal image based on the temperature of the area that overlaps with the area photographed by the monitoring camera C (step S904).

[0174] Then, the thermal camera S acquires heat source coordinates by defining the area in the acquired thermal image where the temperature is above a predetermined threshold as the coordinates of the heat source (step S905), and outputs heat source coordinate data corresponding to the acquired heat source coordinates to the monitoring device Y.

[0175] Then, the image analysis device Y1 of the monitoring device Y acquires heat source coordinate data from the thermal camera S via the interface Y2.

[0176] Here, in the above-mentioned step S903, if the image analysis device Y1 of the monitoring device Y cannot determine that the user MW is carrying a cigarette based on the behavioral image data with a reliability level equal to or higher than the above-mentioned predetermined reliability level (step S903 NO), it compares the determination result of the behavioral image data with the heat source coordinate data output by the thermal camera S (step S906).

[0177] Then, the image analysis device Y1 of the monitoring device Y determines that the user MW is behaving abnormally if the judgment coordinates of the area of ​​the image suspected of being a cigarette in the behavioral image corresponding to the behavioral image data and the heat source coordinates of the thermal image corresponding to the heat source coordinate data overlap at a predetermined standard rate or more (step S907 YES).

[0178] On the other hand, if the judgment coordinates and the heat source coordinates do not overlap by more than the aforementioned standard ratio (step S907 NO), the image analysis device Y1 of the monitoring device Y determines that the user MW is behaving normally (not behaving abnormally).

[0179] Here, for example, the image analysis device Y1 of the monitoring device Y may use the heat source coordinate data to determine whether the acquired behavioral image data is abnormal or normal behavior of the current user MW, and use the classified data, which is then machine-learned as training data, to determine whether the next user MW will be behaving abnormally.

[0180] In this way, in the self-service refueling permission system 100 shown in Figure 1, by determining whether or not a user is carrying cigarettes, it is possible to determine whether or not the user is behaving abnormally.

[0181] [Fifth Variation] Next, a fifth modified example of the specific flow when refueling in the self-service refueling permission system 100 shown in Fig. 1 will be described. In particular, an example of the flow in the self-service refueling permission system 100 shown in Fig. 1 for determining whether or not a user is behaving abnormally by determining whether or not a portable fuel can is present, the use of which is permitted under the Fire Service Act.

[0182] Figure 10 is a flow diagram showing an example of a flow for determining whether a user is behaving abnormally by determining whether or not the user has a portable fuel can that is permitted for refueling by law in the self-service refueling permission system 100 shown in Figure 1.

[0183] First, as shown in FIG. 10, a surveillance camera C (corresponding to the surveillance camera C in FIG. 1) captures an image of the refueling action performed by a person who is a user MW or a monitor MX, and outputs the captured action image data to a surveillance device Y (step S1001).

[0184] Then, the image analysis device Y1 of the monitoring device Y acquires behavior image data from the monitoring camera C via the interface Y2.

[0185] Then, the image analysis device Y1 of the monitoring device Y analyzes the behavior image data output by the monitoring camera C (step S1002).

[0186] Then, if the image analysis device Y1 of the monitoring device Y determines that the behavior image corresponding to the behavior image data does not contain a portable can (container) that is legally permitted for refueling (for example, the person is not carrying a portable can), it determines that the person is not engaging in abnormal behavior (step S1003 YES).

[0187] On the other hand, the image analysis device Y1 of the monitoring device Y detects a person who is the user MW or the monitor MX from the behavior image corresponding to the behavior image data (step S1004). Then, when the image analysis device Y1 of the monitoring device Y determines that a portable can is present in the behavior image corresponding to the behavior image data, (Step S1003: YES)First, it is determined whether the person is an observer MX (step S1005).

[0188] If the image analyzing device Y1 of the monitoring device Y determines that the person is a monitor MX (YES in step S1005), it determines that the person is behaving normally (not behaving abnormally).

[0189] On the other hand, if the image analysis device Y1 of the monitoring device Y determines that the person is not a monitor MX (is a user MW) (step S1005 NO), it determines that the person (i.e., the user MW) is behaving abnormally.

[0190] For example, the image analysis device Y1 of the monitoring device Y detects a person from a behavior image corresponding to the behavior image data, and if the detected person has a predetermined appearance (wearing a uniform, etc.), it determines that the person is a monitor MX.

[0191] In addition, the image analysis device Y1 of the monitoring device Y may detect a person from a behavioral image corresponding to the behavioral image data, and if the detected person is carrying a management terminal X, determine that the person is a monitor MX.

[0192] In addition, the image analysis device Y1 of the monitoring device Y may detect a person from a behavioral image corresponding to the behavioral image data, and when the detected person operates the management terminal X and receives a predetermined signal from the management terminal X, determine that the person is a monitor MX.

[0193] Here, for example, the image analysis device Y1 of the monitoring device Y may determine whether the next user MW will be behaving abnormally by using a machine-learned judgment model as training data, based on the classified data that has been obtained by determining whether the current user MW is behaving abnormally or normally, based on the classified data.

[0194] In this way, the self-service refueling permission system 100 shown in Figure 1 can determine whether or not a user is behaving abnormally by determining whether or not the user has a portable fuel can that is permitted for refueling by law.

[0195] [Sixth Modification] Next, a sixth modified example of the specific flow during refueling in the self-service refueling permission system 100 shown in Fig. 1 will be described. In particular, an example of the flow for detecting the presence or absence of a dispenser nozzle in the self-service refueling permission system 100 shown in Fig. 1 will be described.

[0196] FIG. 11 is a flow chart showing an example of a flow for detecting the presence or absence of a nozzle of a dispenser in the self-service fuel supply permission system 100 shown in FIG.

[0197] In the example shown in FIG. 11, it is assumed that a predetermined discrimination mark is added to the surface of the nozzle of the dispenser D in order to discriminate the nozzle of the dispenser D by image processing.

[0198] First, as shown in FIG. 11, monitoring camera C (corresponding to monitoring camera C in FIG. 1) captures the refueling action performed by user MW and outputs the captured action image data to monitoring device Y (step S1101).

[0199] Then, the image analysis device Y1 of the monitoring device Y acquires behavior image data from the monitoring camera C via the interface Y2.

[0200] Then, the image analysis device Y1 of the monitoring device Y analyzes the behavior image data output by the monitoring camera (step S1102).

[0201] Then, the image analysis device Y1 of the monitoring device Y determines whether or not the nozzle of the dispenser D is present in the behavior image corresponding to the behavior image data (for example, whether the user MW is holding the nozzle of the dispenser D) (step S1103).

[0202] Then, if the image analysis device Y1 of the monitoring device Y can determine with a predetermined reliability or higher that the nozzle of the dispenser D is present in the behavior image corresponding to the behavior image data (step S1103 YES), it determines that the nozzle has been detected.

[0203] Meanwhile, the image analysis device Y1 of the monitoring device Y discriminates the discrimination mark attached to the surface of the nozzle of the dispenser D in the behavior image corresponding to the behavior image data, and obtains the coordinates of the discrimination mark in the behavior image as mark coordinates (step S1104).

[0204] Then, if the image analysis device Y1 of the monitoring device Y cannot determine that the nozzle of the dispenser D is present in the behavioral image with a reliability equal to or higher than the previously described predetermined reliability (step S1103 NO), it compares the determination result of the behavioral image data with the mark coordinates of the determination mark added to the surface of the nozzle in the behavioral image (step S1105).

[0205] Then, the image analysis device Y1 of the monitoring device Y determines that the nozzle has been detected if the judgment coordinates of an image suspected of being a nozzle in the behavior image corresponding to the behavior image data overlap with the mark coordinates of the discrimination mark added to the surface of the nozzle (step S1106).

[0206] On the other hand, if the determination coordinates and the mark coordinates do not overlap (NO in step S1106), the image analyzing device Y1 of the monitoring device Y determines that the nozzle has not been detected.

[0207] As described above, in the example shown in FIG. 11, a preset identification mark is added to the surface of the nozzle of the dispenser D.

[0208] In this case, the image analysis device Y1 of the monitoring device Y discriminates the discrimination mark attached to the surface of the nozzle of the dispenser D in the behavior image corresponding to the behavior image data, and obtains the coordinates of the discrimination mark in the behavior image as mark coordinates.

[0209] However, the sensor S may be configured to detect a mark signal output from the nozzle of the dispenser and output position information data relating to the position of the nozzle of the dispenser D to the monitoring device Y.

[0210] In this case, the image analysis device Y1 of the monitoring device Y acquires position information data from the sensor C via the interface Y2. Then, based on the position information data output from the sensor S, the image analysis device Y1 of the monitoring device Y may acquire the position where the mark signal is detected as mark coordinates in the behavior image.

[0211] Here, for example, the monitoring device Y may use the classification data determined and classified using mark coordinates for the determination of the nozzle of the dispenser D during the current refueling operation, and use a machine-learned determination model as training data to determine the nozzle of the dispenser D during the next refueling operation.

[0212] In this way, the self-service refueling permission system 100 shown in Fig. 1 can detect whether or not the nozzle of the dispenser is present (whether or not the user is holding the nozzle). That is, for example, if the nozzle is removed from the fuel filler opening of the vehicle while refueling (while fuel is being discharged from the nozzle), it can be determined that the user is behaving abnormally. In the above embodiment, it is possible to detect whether the nozzle is present or absent, but it is also possible to detect whether the nozzle is inserted into the fuel filler opening. Specifically, after refueling is permitted (permission button is operated), the nozzle position detected by the overlap of the judgment coordinates and the mark coordinates is at a predetermined position (fuel filler opening), or the nozzle is detected using the judgment coordinates but the mark coordinates are not detected due to the cover of the fuel filler opening of the vehicle or the nozzle being inserted into the fuel filler opening, etc., and the insertion of the nozzle into the fuel filler opening may be detected in such cases.

[0213] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention described in the claims and their equivalents. [Explanation of symbols]

[0214] 100 Self-service fuel permit system Y Monitoring device Y1 Image Analysis Device Y2 interface Y3 radio device X Mobile Device MX Warden H display A Control section T Communications Department CNT control unit C. Surveillance camera S sensor D Dispenser Z Server SSC control unit G1 display G2 control panel G3 control unit W vehicle MW users (fuelers)

Claims

1. A self-service fueling permission system permits a user to refuel a vehicle using a dispenser operated by the user, a monitoring device that monitors the refueling of the vehicle using the dispenser by the user and determines whether the user is behaving abnormally based on an analysis result of image data obtained by capturing at least the user's behavior; a control terminal that acquires and outputs information acquired by the monitoring device and a determination result of the monitoring device from the monitoring device and is operated by an observer who monitors the fuel supply to the vehicle; and a control device that controls the fuel supply to the vehicle using the dispenser operated by the user in accordance with at least the determination result of the monitoring device and the operation of the control terminal by the observer, the management terminal is provided separately from the monitoring device and the control device, If the monitoring device does not determine that the user is behaving abnormally, the control device prohibits or permits the use of the dispenser to refuel the vehicle based on an operation input to the management terminal by the monitor. A self-service fueling permission system.

2. The case where the monitoring device does not determine that the user is behaving abnormally refers to at least one of the cases where the monitoring device is unable to determine that the user is behaving abnormally or the case where the monitoring device determines that the user is behaving normally.

2. The self-service refueling permission system according to claim 1.

3. The control device refueling the vehicle using the dispenser in response to an operation input by the monitor to the management terminal instructing permission for refueling by the user; On the other hand, in response to an operation input by the monitor to the management terminal instructing prohibition of fuel supply by the user, fuel supply to the vehicle using the dispenser is prohibited.

3. The self-service refueling permission system according to claim 1 or 2.

4. The monitoring device The behavioral image data relating to the image of the behavior of refueling a vehicle using the dispenser by the operation of the past user, which has been acquired in the past, is determined and classified as abnormal behavior or normal behavior of the past user, and the classified data is used as training data to determine whether the current user is behaving abnormally, using a machine-learned determination model.

4. The self-service fueling permission system according to claim 1, wherein the self-service fueling permission system is a system for permitting fuel supply to a vehicle.

5. The monitoring device A degree of similarity is calculated between the behavioral image data relating to the behavioral image of the current user's operation of filling the vehicle with fuel using the dispenser and the behavioral image data determined to be abnormal behavior from the classified data, and based on the calculation result, it is determined whether the user is engaging in abnormal behavior.

5. The self-service refueling permission system according to claim 4.

6. When the monitoring device determines that the user is taking normal actions for refueling, the control device allows the user to refuel instead of the monitor; When the monitoring device determines that the user is behaving abnormally, the control device prohibits the user from refueling, and thereafter, based on an operation input to the management terminal by the monitor, the control device again prohibits or permits refueling of the vehicle using the dispenser; If the monitoring device cannot determine the behavior of the user, the control device prohibits the user from refueling, and thereafter, based on an operation input to the management terminal by the monitor, the control device again prohibits or permits refueling of the vehicle using the dispenser.

6. The self-service fueling permission system according to claim 1, wherein the self-service fueling permission system is a system for permitting fuel supply to a vehicle.

7. When the monitoring device determines that the user is taking normal actions for refueling, the control device allows the user to refuel instead of the monitor; When the monitoring device determines that the user is behaving abnormally, the control device prohibits the user from refueling, and thereafter, based on an operation input to the management terminal by the monitor, the control device again prohibits or permits refueling of the vehicle using the dispenser; If the monitoring device cannot determine the behavior of the user, the control device permits the vehicle to be refueled using the dispenser based on an operation input to the management terminal by the monitor.

6. The self-service fueling permission system according to claim 1, wherein the self-service fueling permission system is a system for permitting fuel supply to a vehicle.

8. The vehicle further includes a monitoring camera that captures at least the user's refueling operation and outputs the captured image data to the monitoring device.

8. The self-service fueling permission system according to claim 1, wherein the self-service fueling permission system is a system for permitting fuel supply to a vehicle.

9. The surveillance camera is a dispenser camera that captures an image of at least the periphery of the dispenser to acquire image data; a stopping area camera for capturing an image of at least a stopping area where a vehicle to be refueled by the dispenser stops, and acquiring image data.

9. The self-service refueling permission system according to claim 8.

10. The fuel supply system further includes a sensor that detects at least one of sound, temperature, odor, light, or oil leakage that occurs in association with the fuel supply operation by the user and outputs the detection result to the monitoring device; The monitoring device determines whether the user is behaving abnormally based on the analysis result of the image data and the detection result detected by the sensor. The self-service refueling permission system according to claim 9.

11. The sensors are installed around the dispenser and / or in a stopping area where a vehicle to be refueled by the dispenser stops. The self-service fueling permission system according to claim 10.

12. The self-service refueling permission system of claim 11, characterized in that the monitoring device uses signal data previously acquired by the sensor to determine whether the current user is exhibiting abnormal behavior based on a comparison with data on abnormal behavior of past users.

13. The management terminal receives information including the determination result from the monitoring device by wireless or wired communication, and transmits information according to the operation of the monitor to the monitoring device.

13. The self-service fueling permission system of claim 12.

14. The monitoring device is disposed in a building within the self-service gas station, 14. The self-service refueling permission system according to claim 13, wherein the management terminal is operated by the monitor at a remote location connected to the monitoring device by wireless or wired communication.

15. The self-service fueling permission system according to claim 14, wherein the monitoring device is installed remotely from the dispenser and the control device.

16. The monitoring device an image analysis device that analyzes information on images of the user's behavior at least before and / or after the start of refueling acquired by the surveillance camera and determines whether the user is behaving abnormally; a wireless device for wirelessly communicating with at least the management terminal; an interface interposed between the surveillance camera, the sensor, the control device, the image analysis device, and the wireless device; 16. The self-service fueling authorization system of claim 15.

17. The management terminal an operation unit for the observer to input operations; a display unit that displays predetermined information; a communication unit for wirelessly communicating with the monitoring device and / or the control device; a control unit configured by a computer and executing necessary processing of the management terminal; 17. The self-service fueling permission system according to any one of claims 1 to 16.

18. 18. The self-service refueling permission system according to claim 1, wherein the management terminal is a portable terminal carried by the monitor.

19. The self-service refueling permission system according to claim 2, wherein the abnormal behavior is the user's possession of a firearm or the user's smoking of a cigarette.

20. A self-service refueling permission system that permits a user to refuel a vehicle using a dispenser operated by the user, the self-service refueling permission system comprising: a monitoring device that monitors the refueling of the vehicle using the dispenser operated by the user and determines whether the user is behaving abnormally based on an analysis result of at least image data obtained by capturing an image of the user's behavior; a management terminal that acquires and outputs information acquired by the monitoring device and the determination result of the monitoring device from the monitoring device and is operated by an observer who monitors the refueling of the vehicle; and a control device that controls the refueling of the vehicle using the dispenser operated by the user in accordance with at least the determination result of the monitoring device and the operation of the management terminal by the observer, the management terminal is provided separately from the monitoring device and the control device, If the monitoring device does not determine that the user is behaving abnormally, the control device prohibits or permits the use of the dispenser to refuel the vehicle based on an operation input to the management terminal by the monitor. A control method for a self-service refueling permission system.

21. At a self-service fueling station that uses a self-service fueling permission system that allows users to refuel vehicles using a dispenser operated by the user, The self-service refueling permission system includes: a monitoring device that monitors the refueling of the vehicle using the dispenser by the user and determines whether the user is behaving abnormally based on an analysis result of image data obtained by capturing at least the user's behavior; a control terminal that acquires and outputs information acquired by the monitoring device and a determination result of the monitoring device from the monitoring device and is operated by an observer who monitors the fuel supply to the vehicle; and a control device that controls the fuel supply to the vehicle using the dispenser operated by the user in accordance with at least the determination result of the monitoring device and the operation of the control terminal by the observer, the management terminal is provided separately from the monitoring device and the control device, If the monitoring device does not determine that the user is behaving abnormally, the control device prohibits or permits the use of the dispenser to refuel the vehicle based on an operation input to the management terminal by the monitor. A self-service fuel station.

22. The case where the monitoring device does not determine that the user is behaving abnormally refers to the case where the monitoring device is unable to determine that the user is behaving abnormally.

2. The self-service refueling permission system according to claim 1.

23. The case where the monitoring device does not determine that the user is behaving abnormally is the case where the monitoring device determines that the user is behaving normally.

2. The self-service refueling permission system according to claim 1.

24. The cases where the monitoring device does not determine that the user is behaving abnormally include a case where the monitoring device is unable to determine that the user is behaving abnormally and a case where the monitoring device determines that the user is behaving normally.

2. The self-service refueling permission system according to claim 1.

25. the management terminal or the control device each includes a display unit that displays predetermined information, When the monitoring device determines that the user is behaving abnormally after refueling has started, the management terminal or the control device displays an image of the user's behavior determined to be abnormal, as well as a confirmation button and a stop button for the monitor to operate, on the display unit of the management terminal or the control device.

2. The self-service refueling permission system according to claim 1.

26. If the monitor operates a confirmation button displayed on the display unit of the management terminal or the control device before starting refueling, the monitoring device determines whether the abnormal behavior of the user has been resolved.

26. The self-service fueling authorization system of claim 25.

27. When the monitor operates a confirmation button displayed on the display unit of the management terminal or the control device before starting refueling, if the monitor determines that the abnormal behavior of the user has been resolved, the management terminal and the control device notify the user that the abnormal behavior of the user has been resolved.

26. The self-service fueling authorization system of claim 25.

28. When the monitoring device determines that the user is behaving abnormally before refueling begins, the management terminal or the control device displays an image of the user's behavior determined to be abnormal and a confirmation button for the monitor to operate on the display unit of the management terminal or the control device, When the monitor operates the confirmation button displayed on the display unit of the management terminal or the control device, the control device drives the pump of the dispenser.

27. The self-service fueling authorization system of claim 26.

29. the sensor includes a thermal camera capable of detecting the temperature of a heat source as a thermal image; The thermal camera Acquire a thermal image based on the temperature of an area overlapping with an area photographed by the monitoring camera, acquire heat source coordinates by defining an area in the acquired thermal image where the temperature is equal to or higher than a predetermined threshold as the coordinates of the heat source, and output heat source coordinate data corresponding to the acquired heat source coordinates to the monitoring device, The surveillance camera is capturing an image of the refueling action performed by the user and outputting the captured image data to the monitoring device; The monitoring device Analyzing the behavioral image data output by the surveillance camera; If it is determined based on the behavioral image data that the user is carrying a cigarette, it is determined that the user is behaving abnormally; On the other hand, if it is not possible to determine that the user is carrying a cigarette based on the behavioral image data, the determination result of the behavioral image data is compared with the heat source coordinate data output by the thermal camera, If the determined coordinates of an area of ​​an image suspected of being a cigarette in the behavior image corresponding to the behavior image data overlap with the heat source coordinates of the thermal image corresponding to the heat source coordinate data, it is determined that the user is behaving abnormally; On the other hand, if the judgment coordinates and the heat source coordinates do not overlap, it is determined that the user is behaving normally. The self-service fueling permission system according to claim 10.

30. The monitoring device The acquired behavior image data is determined as abnormal or normal behavior of the user using the heat source coordinate data, and the classified data is used as training data to determine whether the next user is behaving abnormally using a machine-learned determination model.

30. The self-service fueling authorization system of claim 29.

31. The surveillance camera is capturing an image of a refueling action performed by the user or the monitor, and outputting the captured image data to the monitoring device; The monitoring device Analyzing the behavioral image data output by the surveillance camera; If it is determined that the portable can is not present in the behavior image corresponding to the behavior image data, it is determined that the person is not behaving abnormally; On the other hand, when it is determined that the portable can is present in the behavior image corresponding to the behavior image data, it is determined whether the person is the monitor; If the person is determined to be the monitor, the person is determined to be behaving normally; On the other hand, if it is determined that the person is not the monitor, it is determined that the person is behaving abnormally. The self-service fueling permission system according to claim 10.

32. The monitoring device Detecting the person from a behavior image corresponding to the behavior image data; If the detected person has a predetermined appearance, the person is determined to be the monitor.

32. The self-service fueling authorization system of claim 31.

33. The monitoring device Detecting the person from a behavior image corresponding to the behavior image data; If the detected person is carrying the management terminal, the person is determined to be the monitor.

32. The self-service fueling authorization system of claim 31.

34. The monitoring device Detecting the person from a behavior image corresponding to the behavior image data; When the detected person operates the management terminal and receives a predetermined signal from the management terminal, the person is determined to be the monitor.

32. The self-service fueling authorization system of claim 31.

35. The monitoring device The acquired behavioral image data is determined and classified as abnormal or normal behavior of the user, and the classified data is used as training data to determine whether the next user is behaving abnormally using a machine-learned determination model.

32. The self-service fueling authorization system of claim 31.

36. The surveillance camera is capturing an image of the refueling action performed by the user and outputting the captured image data to the monitoring device; The monitoring device Analyzing the behavioral image data output by the surveillance camera; If it is determined that a nozzle of the dispenser is present in the behavior image corresponding to the behavior image data, it is determined that the nozzle is detected; On the other hand, when it is not possible to determine that the nozzle of the dispenser is present in the behavior image, the determination result of the behavior image data is compared with the mark coordinates of the nozzle in the behavior image; If the determined coordinates of an image suspected of being the nozzle in the behavior image corresponding to the behavior image data overlap with the mark coordinates of the nozzle, it is determined that the nozzle has been detected; On the other hand, if the determination coordinates and the mark coordinates do not overlap, it is determined that the nozzle has not been detected. The self-service fueling permission system according to claim 10.

37. A predetermined identification mark is added to the surface of the nozzle of the dispenser, The monitoring device The discrimination mark is discriminated in a behavior image corresponding to the behavior image data, and the coordinates of the discrimination mark in the behavior image are acquired as the mark coordinates.

37. The self-service fueling authorization system of claim 36.

38. the sensor detects a mark signal output from the nozzle of the dispenser and outputs position information data relating to the position of the nozzle of the dispenser to the monitoring device; The monitoring device Based on the position information data output from the sensor, the position where the mark signal is detected is acquired as the mark coordinate in the behavior image.

37. The self-service fueling authorization system of claim 36.

39. The monitoring device The nozzle of the dispenser is identified at the next refueling time using a machine-learned determination model based on the classified data determined and classified using the mark coordinates for the current refueling time.

37. The self-service fueling authorization system of claim 36.

Images