Fueling system, information processing device, fueling control method and fueling control program

Abstract

To provide a fueling system capable of executing a process in correspondence to a fueling scene.SOLUTION: The fueling system comprises: first and second fueling machines; a plurality of cameras for taking images of an area including the first and second fueling machines; and an information processing device. The information processing device includes: a first determination unit that determines a fueling permission / rejection of the first fueling machine based on a recognition result, if recognition processing is executed when image data taken of the first fueling machine and a fueling action by a fueling person is input into a first learned model; and a second determination unit that determines the fueling permission / rejection of the second fueling machine based on the recognition result, if the recognition processing is executed when the image data taken of the second fueling machine and the fueling action of the fueling person is input in a second learned model. If the first determination unit determines that the fueling is not permitted, the fueling by the first fueling machine is prohibited or stopped, and if the second determination unit determines that the fueling is not permitted, the fueling by the second fueling machine is prohibited or stopped.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present disclosure relates to a fueling system, an information processing apparatus, a fueling control method, and a fueling control program.

Background Art

[0002] As a fueling system for service stations, the development of a fueling system that monitors the fueling behavior of customers who are fuelers and determines whether to allow fueling by utilizing AI or the like is underway.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] On the other hand, the fueling scenes at service stations are various, and in the development of a fueling system, it is desirable to adopt a configuration in which the method for determining whether to allow fueling, the method for outputting the determination result, etc. are changed according to the fueling scene.

[0005] In one aspect, an object is to provide a fueling system capable of executing processing according to a fueling scene.

Means for Solving the Problems

[0006] According to one aspect, a fueling system includes: a first fuel dispenser that dispenses a first type of fuel and a second fuel dispenser that dispenses a second type of fuel; a plurality of cameras that respectively photograph an area including the first fuel dispenser and an area including the second fuel dispenser; an information processing apparatus, and is a fueling system having: the information processing apparatus includes: When the image data in which the first fuel dispenser and the fueling operation by the fueler are photographed is input to the first pre-trained model and recognition processing is performed, a first determination unit that determines whether or not to permit fueling of the first fuel dispenser based on the recognition result; When the image data in which the second fuel dispenser and the fueling operation by the fueler are photographed is input to the second pre-trained model and recognition processing is performed, a second determination unit that determines whether or not to permit fueling of the second fuel dispenser based on the recognition result, and has: When it is determined by the first determination unit that fueling is not permitted, fueling of the first fuel type is prohibited or stopped, and when it is determined by the second determination unit that fueling is not permitted, fueling of the second fuel type is prohibited or stopped.

Advantages of the Invention

[0007] It is possible to provide a fueling system capable of executing processing according to the fueling scene.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2A

Figure 2B

Figure 3A

Figure 3B

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14A

Figure 14B

Figure 15A

Figure 15B

Figure 16A

Figure 16B

Figure 17A

Figure 17B

Figure 17C

Figure 17D

Figure 18A

Figure 18B

Figure 19

Figure 20A

Figure 20B

Figure 21A

Figure 21B

Figure 21C

Figure 21D

Figure 22A

Figure 22B

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28A

Figure 28B

Embodiments for Carrying Out the Invention

[0009] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and the drawings, for components having substantially the same functional configuration, the same reference numerals are given and redundant explanations are omitted.

[0010] [First Embodiment] <Flow of Learning> First, the flow of learning for constructing the fuel supply system according to the first embodiment will be described. FIG. 1 is a diagram showing the flow of learning for constructing the fuel supply system according to the first embodiment. As shown in FIG. 1, in constructing the fuel supply system according to the first embodiment, image data collected at a plurality of service stations (for example, service stations 110, 111,...) is used. Also, at one service station, when different types of fuel can be supplied, image data collected for each type of fuel is used. Here, the different types of fuel referred to here mean fuel oil for vehicles such as gasoline and light oil (hereinafter referred to as "fuel oil of the first fuel type"), and fuel oil for uses other than vehicles such as kerosene (hereinafter referred to as "fuel oil of the second fuel type").

[0011] The example of FIG. 1 shows the case where service station 110 can supply both the fuel oil of the first fuel type and the fuel oil of the second fuel type. For this reason, in the example of FIG. 1, the image data obtained by photographing the fuel supply scene 110_1 for the first fuel type and the image data obtained by photographing the fuel supply scene 110_2 for the second fuel type are collected.

[0012] The refueling scene 110_1 for the first type of fuel includes a refueling operation by a refueler (not shown). A camera 130_1 that captures the area including the first fuel dispenser 120_1 as a video captures the first fuel dispenser 120_1 and the refueling operation by the refueler, and image data with a predetermined frame period is collected by the image data collection device 140.

[0013] As shown in FIG. 1, a fuel nozzle 121_1 is connected to the first fuel dispenser 120_1, and the fuel oil of the first type of fuel stored in the tank 126_1 is configured to be supplied to the fuel nozzle 121_1 through a fuel supply pipe by a fuel supply pump 125_1. Also, as shown in FIG. 1, a fuel supply valve 124_1 is installed in the fuel supply pipe. Further, in the first fuel dispenser 120_1, as a notification device, a voice output device 122_1 and a fueling data display 123_1 (a device that displays fueling data and messages from the information processing device 160) are installed.

[0014] Although not shown in FIG. 1, similar image data for the refueling scene 110_2 for the second type of fuel is also collected by the image data collection device 140.

[0015] When collecting the image data, the image data collection device 140 also collects a signal indicating the removal of the fuel nozzle 121_1 from the first fuel dispenser 120_1 and the return of the fuel nozzle 121_1 to the first fuel dispenser 120_1 (a signal indicating the state of the fuel nozzle).

[0016] The image data collected by the image data collection device 140 and the signal indicating the state of the fuel nozzle are transmitted to the learning device 150 via the network 180, and learning data is generated in the learning device 150. Note that the learning data is generated based on the image data collected at a plurality of service stations (for example, service stations 110, 111,...) and the signal indicating the state of the fuel nozzle. However, the learning data is generated separately for the first type of fuel and the second type of fuel.

[0017] The learning device 150 is installed at a different location (e.g., a server on the cloud) from a plurality of service stations (e.g., service stations 110, 111, ···). The learning device 150 uses the learning data for the first refueling type (referred to as the first learning data) to perform learning of the first model and generate the first learned model. Further, the learning device 150 uses the learning data for the second refueling type (referred to as the second learning data) to perform learning of the second model and generate the second learned model.

[0018] The first learned model and the second learned model generated by the learning device 150 are transmitted to a plurality of service stations (e.g., service stations 110', 111', ···) via the network 180. Thereby, they are stored in the information processing device of each service station (e.g., the information processing device 160 of the service station 110').

[0019] As shown in FIG. 1, the fueling system 110S' according to the first embodiment applied to the service station 110' includes, for the first refueling type, · a camera 130_1, · a first fuel dispenser 120_1, · a thermal sensor 170_1. The first fuel dispenser 120_1 is connected to a fueling nozzle 121_1 to which the fuel oil of the first refueling type stored in the tank 126_1 is supplied via a fueling pipe by a fueling pump 125_1. A fueling valve 124_1 is installed in the fueling pipe. Further, the first fuel dispenser 120_1 is provided with a voice output device 122_1 and a fueling data display 123_1 as notification devices.

[0020] In addition, the fueling system 110S' according to the first embodiment applied to the service station 110' has a similar configuration for the second fuel type. However, when the measurement area of the thermal sensor 170_1 includes both the fueling scene 110_1 of the first fuel type and the fueling scene 110_2 of the second fuel type, the thermal sensor 170_1 may be used for both the first fuel type and the second fuel type.

[0021] Furthermore, as shown in FIG. 1, the fueling system 110S' according to the first embodiment applied to the service station 110' includes an information processing device 160. The information processing device 160 · Cameras for the first fuel type and the second fuel type (cameras 130_1, 130_2), · Fuel dispensers (first fuel dispenser 120_1, second fuel dispenser 120_2), · Thermal sensors (thermal sensors 170_1, 170_2), · Fuel pumps (fuel pumps 125_1, 125_2), are communicably connected.

[0022] In the example of FIG. 1, the case where the fueling system 110S' according to the first embodiment is applied to the service station 110' is shown. However, the same configuration is obtained when it is applied to other service stations (for example, service stations 111',...).

[0023] <Description of the fueling scene> Next, a specific example of the fueling scene at the service station 110 will be described. FIGS. 2A and 2B are the first and second figures showing an example of the fueling scene at the service station.

[0024] Among these, FIG. 2A shows an example of the fueling scene for the first fuel type at the service station 110. As shown in FIG. 2A, the service station 110 includes a first area 210 where the vehicle 211 stops and a second area 220 where the first fuel dispenser 120_1 for fueling the first fuel type is installed and where the fueling operation by the fueler 221 is performed.

[0025] In the case of the refueling scene shown in Fig. 2A, with the vehicle 211 stopped in the first area 210, the refueler 221 removes the fuel nozzle 121_1 from the nozzle hanger of the first fuel dispenser 120_1 and inserts the fuel nozzle 121_1 into the fuel inlet of the vehicle 211. Further, the refueler 221 starts refueling by operating the lever of the fuel nozzle 121_1 and continues refueling until the desired fuel quantity is reached. Then, when the desired fuel quantity is reached, the refueler 221 stops refueling by returning the lever of the fuel nozzle 121_1, removes the fuel nozzle 121_1 from the fuel inlet, and then returns it to the nozzle hanger of the first fuel dispenser 120_1 to end the refueling operation.

[0026] On the other hand, Fig. 2B shows an example of a refueling scene for the second type of fuel at the service station 110. As shown in Fig. 2B, the service station 110 includes an area where the container 231 (poly tank) is placed and an area where the second fuel dispenser 120_2 for refueling the second type of fuel is installed, and an area where the refueling operation by the refueler 241 is performed.

[0027] In the case of the refueling scene shown in Fig. 2B, with the container 231 placed, the refueler 241 removes the fuel nozzle 121_2 from the nozzle hanger of the second fuel dispenser 120_2 and inserts the fuel nozzle 121_2 into the fuel inlet of the container 231. Further, the refueler 241 starts refueling by operating the lever of the fuel nozzle 121_2 and continues refueling until the desired fuel quantity is reached. Then, when the desired fuel quantity is reached, the refueler 241 stops refueling by returning the lever of the fuel nozzle 121_2, removes the fuel nozzle 121_2 from the fuel inlet, and then returns it to the nozzle hanger of the second fuel dispenser 120_2 to end the refueling operation.

[0028] <Process for generating learning data> Next, an overview of the process from when camera 130_1 captures a video of the area including the first area 210 and the second area 220 to when the learning device 150 generates first learning data based on the video will be described. FIG. 3A is a diagram showing the flow of the process for generating the first learning data.

[0029] As shown in FIG. 3A, the image data collection device 140 collects image data of a predetermined frame period, which is captured as a video by the camera 130_1. Also, as shown in FIG. 3A, the image data collection device 140 collects a signal indicating the state of the fuel nozzle 121_1 from the first fuel dispenser 120_1.

[0030] The learning device 150 extracts image data during the period from when the fuel nozzle 121_1 is removed from the nozzle mount of the first fuel dispenser 120_1 to when it is returned to the nozzle mount, among the image data collected by the image data collection device 140.

[0031] Also, the learning device 150 performs image recognition processing on the extracted image data to recognize each recognition target (such as a vehicle, a container, a fueler, a fuel nozzle, etc.) included in the extracted image data. Furthermore, the learning device 150 assigns annotations to the recognized recognition targets (such as a vehicle, a container, a fueler, a fuel nozzle, etc.).

[0032] Specifically, when a vehicle is recognized in the image data and is recognized within the first area 210, which is a part of the image data, an annotation indicating that it is a vehicle is assigned to the bounding box of the vehicle.

[0033] Also, when a container is recognized in the image data, an annotation indicating that it is a container is assigned to the bounding box of the container.

[0034] Also, when a fueler is recognized in the image data, · when the fueler is recognized within the second area 220 and, · The fuel nozzle was held in an appropriate posture, In this case, an annotation indicating that the fueler is the one holding the fuel nozzle in an appropriate posture is added to the circumscribed rectangle (bounding box) of the fueler.

[0035] Also, when the fueler is recognized in the image data, · The fueler is recognized within the second area 220, and · The held fuel nozzle is not recognized, In this case, an annotation (fuel nozzle unrecognizable) indicating that the fueler is not holding the fuel nozzle in an appropriate posture is added to the circumscribed rectangle (bounding box) of the fueler.

[0036] Also, when the fueler is not recognized within the second area 220 in the image data, · For example, when the fuel nozzle in the state of being inserted into the fuel inlet is recognized, In this case, an annotation (fueler unrecognizable) indicating that the fueler is not holding the fuel nozzle in an appropriate posture is added to the circumscribed rectangle (bounding box) of the recognized fuel nozzle.

[0037] The learning device 150 generates first learning data using the image data with any annotation among the extracted image data.

[0038] Subsequently, an overview of the processing from when the camera 130_2 that captures the area including the second fuel dispenser 120_2 as a video captures the second fuel dispenser 120_2 and the placed container until the learning device 150 generates second learning data based on the video will be described. FIG. 3B is a diagram showing the flow of the processing for generating the second learning data.

[0039] As shown in FIG. 3B, the image data collection device 140 collects image data with a predetermined frame period, which is captured as a video by the camera 130_2. Also, as shown in FIG. 3B, the image data collection device 140 collects a signal indicating the state of the fuel nozzle 121_2 from the second fuel dispenser 120_2.

[0040] The learning device 150 extracts, from the image data collected by the image data collection device 140, the image data during the period from when the fuel nozzle 121_2 is removed from the nozzle hanging of the second fuel dispenser 120_2 until it is returned to the nozzle hanging.

[0041] Also, the learning device 150 performs image recognition processing on the extracted image data, and recognizes the recognition targets (such as containers, fuelers, and fuel nozzles) included in the extracted image data. Further, the learning device 150 assigns annotations to the recognized recognition targets (such as containers, fuelers, and fuel nozzles).

[0042] Specifically, when a container is recognized in the image data, an annotation indicating that it is a container is assigned to the circumscribed rectangle (bounding box) of the recognized container.

[0043] Also, when a fueler is recognized in the image data, · if the fueler is holding the fuel nozzle in an appropriate posture, an annotation indicating that the fueler is holding the fuel nozzle in an appropriate posture is assigned to the circumscribed rectangle (bounding box) of the fueler.

[0044] Also, when a fueler is recognized in the image data, · if the fuel nozzle held by the fueler cannot be recognized, an annotation (fuel nozzle recognition impossible) indicating that the fueler is not holding the fuel nozzle in an appropriate posture is assigned to the circumscribed rectangle (bounding box) of the fueler.

[0045] Also, in the case where the fueler is not recognized in the image data, ·For example, when the fuel nozzle in the state of being inserted into the fuel filler opening is recognized, in this case, an annotation (fueler recognition impossible) indicating that the fueler is not holding the fuel nozzle in an appropriate posture is given to the circumscribed rectangle (bounding box) of the fuel nozzle.

[0046] The learning device 150 generates second learning data using the image data with any annotation among the extracted image data.

[0047] <Hardware Configuration of the Learning Device> Next, the hardware configuration of the learning device 150 will be described. FIG. 4 is a diagram showing an example of the hardware configuration of the learning device.

[0048] As shown in FIG. 4, the learning device 150 includes a processor 401, a memory 402, an auxiliary storage device 403, an I / F (Interface) device 404, a communication device 405, and a drive device 406. Note that the processor 401, the memory 402, the auxiliary storage device 403, the I / F device 404, the communication device 405, and the drive device 406 of the learning device 150 are interconnected via a bus 407.

[0049] The processor 401 includes various arithmetic devices such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 401 reads and executes various programs (for example, a learning program, etc.) on the memory 402.

[0050] The memory 402 includes main memory devices such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The processor 401 and the memory 402 form a so-called computer, and by the processor 401 executing various programs read on the memory 402, the computer realizes various functions.

[0051] The auxiliary storage device 403 stores various programs and various information used when the various programs are executed by the processor 401.

[0052] The I / F device 404 is a connection device for connecting the operation device 411, which is an example of an external device, the display device 412, and the learning device 150.

[0053] The communication device 405 is a communication device for communicating with the image data collection device 140 or the information processing device 160 via a network.

[0054] The drive device 406 is a device for setting the recording medium 413. The recording medium 413 here includes media that optically, electrically, or magnetically record information, such as CD-ROMs, flexible disks, and magneto-optical disks. The recording medium 413 may also include semiconductor memories that electrically record information, such as ROMs and flash memories.

[0055] Note that the various programs installed in the auxiliary storage device 403 are installed, for example, when the distributed recording medium 413 is set in the drive device 406 and the various programs recorded on the recording medium 413 are read out. Alternatively, the various programs installed in the auxiliary storage device 403 may be installed by being downloaded from a network via the communication device 405.

[0056] <Functional Configuration of the Learning Device> Next, the functional configuration of the learning device 150 will be described. FIG. 5 is a diagram showing an example of the functional configuration of the learning device. As described above, a learning program is installed in the learning device 150, and when the program is executed, the learning device 150 functions as a learning data generation unit 500A and a learning unit 500B.

[0057] As shown in FIG. 5, the learning data generation unit 500A further includes a collection data acquisition unit 510, an image recognition unit 520, an annotation addition unit 530, and a learning data generation unit 540.

[0058] The collection data acquisition unit 510 acquires, from the image data collection device 140, the image data captured by the camera 130_1 and the signal indicating the state of the fuel nozzle 121_1. Further, the collection data acquisition unit 510 acquires, from the image data collection device 140, the image data captured by the camera 130_2 and the signal indicating the state of the fuel nozzle 121_2.

[0059] The image recognition unit 520 extracts, from the image data captured by the camera 130_1, the image data during the period from when the fuel nozzle 121_1 is removed from the nozzle holder of the first fuel dispenser 120_1 until it is returned to the nozzle holder of the first fuel dispenser 120_1. Further, the image recognition unit 520 performs image recognition processing on the extracted image data. Thereby, the image recognition unit 520 recognizes the recognition targets (such as vehicles, containers, fuelers, fuel nozzles, etc.).

[0060] Also, the image recognition unit 520 extracts, from the image data captured by the camera 130_2, the image data during the period from when the fuel nozzle 121_2 is removed from the nozzle holder of the second fuel dispenser 120_2 until it is returned to the nozzle holder of the second fuel dispenser 120_2. Further, the image recognition unit 520 performs image recognition processing on the extracted image data. Thereby, the image recognition unit 520 recognizes the recognition targets (such as containers, fuelers, fuel nozzles, etc.).

[0061] The annotation addition unit 530 adds an annotation to the bounding box of the recognition target recognized by the image recognition unit 520. Since the method of adding the annotation has been described with reference to FIGS. 3A and 3B, the description is omitted here.

[0062] The learning data generation unit 540 generates first learning data and second learning data, which use the image data annotated by the annotation unit 530 as input data and the given annotation as correct answer data. The learning data generation unit 540 stores the generated first learning data and second learning data in the learning data storage unit 500C.

[0063] Also, as shown in FIG. 5, the learning unit 500B further includes a learning data reading unit 550, a first model 560_1, a comparison / modification unit 570_1, a second model 560_2, and a comparison / modification unit 570_2.

[0064] The learning data reading unit 550 reads the first learning data from the learning data storage unit 500C, inputs the input data included in the first learning data into the first model 560_1, and inputs the correct answer data into the comparison / modification unit 570_1.

[0065] Also, the learning data reading unit 550 reads the second learning data from the learning data storage unit 500C, inputs the input data included in the second learning data into the second model 560_2, and inputs the correct answer data into the comparison / modification unit 570_2.

[0066] The first model 560_1 is a machine learning model that performs image recognition processing. When the input data included in the first learning data is input, it performs image recognition processing and outputs a recognition result.

[0067] The comparison / modification unit 570_1 compares the recognition result output from the first model 560_1 with the correct answer data input by the learning data reading unit 550. Also, the comparison / modification unit 570_1 updates the model parameters of the first model 560_1 so that the recognition result output from the first model 560_1 matches the correct answer data. In this way, the learning process for the first model 560_1 is performed, and the first learned model is generated.

[0068] The second model 560_2 is a machine learning model that performs image recognition processing. When the input data included in the second learning data is input, it performs image recognition processing and outputs a recognition result.

[0069] The comparison / modification unit 570_2 compares the recognition result output from the second model 560_2 with the correct data input by the learning data reading unit 550. Further, the comparison / modification unit 570_2 updates the model parameters of the second model 560_2 so that the recognition result output from the second model 560_2 matches the correct data. In this way, the learning process for the second model 560_2 is performed, and the second learned model is generated.

[0070] Note that the first learned model and the second learned model generated by the learning unit 500B are transmitted to the information processing device 160 and the like.

[0071] <Specific example of learning data> Next, a specific example of the learning data generated by the learning data generation unit 500A of the learning device 150 will be described.

[0072] (1) Specific example of the first learning data FIG. 6 is a diagram showing a specific example of the first learning data. As shown in FIG. 6, the first learning data 600 includes "input data" and "correct data" as information items. Further, the "input data" further includes "image data ID".

[0073] The "image data ID" stores an identifier for identifying the image data annotated by the annotation adding unit 530. In the example of FIG. 6, the image data 610 is the image data with the image data ID = ID001, and the image data 620 is the image data with the image data ID = ID00L. Further, the image data 630 is the image data with the image data ID = ID00M, and the image data 640 is the image data with the image data ID = ID00N.

[0074] "Correct data" further includes "regions" and "annotation data". The "regions" store the first region 210 and the second region 220, and the "annotation data" stores annotation data assigned to the bounding box of the recognition target recognized in the corresponding region.

[0075] In the example of FIG. 6, for the bounding box 611 of the vehicle recognized in the first region 210 which is a part of the image data 610, "Annotation data indicating that it is a vehicle" is shown as being assigned. Also, in the example of FIG. 6, for the bounding box 612 of the fueler recognized in the second region 220 which is a part of the image data 610, "Annotation indicating that it is a fueler holding the fuel nozzle in an appropriate posture" is shown as being assigned.

[0076] Similarly, in the example of FIG. 6, for the bounding box 621 of the vehicle recognized in the first region 210 which is a part of the image data 620, "Annotation data indicating that it is a vehicle" is shown as being assigned. Also, in the example of FIG. 6, for the bounding box 622 of the fueler recognized in the second region 220 which is a part of the image data 620, "Annotation indicating that it is a fueler not holding the fuel nozzle in an appropriate posture (fuel nozzle recognition impossible)" is shown as being assigned.

[0077] Similarly, in the example of FIG. 6, for the bounding box 631 of the vehicle recognized in the first region 210 which is a part of the image data 630, "Annotation data indicating that it is a vehicle" is shown as being assigned. Also, in the example of FIG. 6, for the bounding box 632 of the fuel nozzle recognized in the second region 220 which is a part of the image data 630, "Annotation indicating that the fueler is not holding the fuel nozzle in an appropriate posture (fueler unrecognizable)" indicates that it has been assigned.

[0078] Similarly, in the example of FIG. 6, with respect to the bounding box 641 of the container recognized in the first region 210 which is a part of the image data 640, "Annotation data indicating that it is a container" indicates that it has been assigned. Also, in the example of FIG. 6, with respect to the bounding box 642 of the fueler recognized in the second region 220 which is a part of the image data 640, "Annotation indicating that the fueler is holding the fuel nozzle in an appropriate posture" indicates that it has been assigned.

[0079] (2) Specific examples of the second learning data FIG. 7 is a diagram showing a specific example of the second learning data. As shown in FIG. 7, the second learning data 700 includes, as information items, "input data" and "correct answer data". Further, the "input data" includes "image data ID".

[0080] The "image data ID" stores an identifier for identifying the image data to which an annotation has been given by the annotation adding unit 530. In the example of FIG. 7, the image data 710 is the image data with the image data ID = ID101, the image data 720 is the image data with the image data ID = ID10L, the image data 730 is the image data with the image data ID = ID10M, and the image data 740 is the image data with the image data ID = ID10N.

[0081] The "correct answer data" further includes "annotation data". The "annotation data" stores the annotation data given to the bounding box of the recognized recognition target.

[0082] The example in Fig. 7 shows that for the bounding box 711 of the container recognized in the image data 710, "annotation data indicating that it is a container" has been assigned. Also, the example in Fig. 7 shows that for the bounding box 712 of the fueler recognized in the image data 710, "annotation indicating that it is a fueler holding the fuel nozzle in an appropriate posture" has been assigned.

[0083] Similarly, the example in Fig. 7 shows that for the bounding box 721 of the container recognized in the image data 720, "annotation data indicating that it is a container" has been assigned. Also, the example in Fig. 7 shows that for the bounding box 722 of the fueler recognized in the image data 720, "annotation indicating that it is a fueler not holding the fuel nozzle in an appropriate posture (fuel nozzle not recognized)" has been assigned.

[0084] Similarly, the example in Fig. 7 shows that for the bounding box 731 of the container recognized in the image data 730, "annotation data indicating that it is a container" has been assigned. Also, the example in Fig. 7 shows that for the bounding box 732 of the fuel nozzle recognized in the image data 730, "annotation indicating that it is a fueler not holding the fuel nozzle in an appropriate posture (fueler not recognized)" has been assigned.

[0085] Similarly, the example in Fig. 7 shows that for the bounding box 742 of the fueler recognized in the image data 740, "annotation indicating that it is a fueler holding the fuel nozzle in an appropriate posture" has been assigned.

[0086] <Flow of Learning Process by Learning Device> Next, the flow of the learning process by the learning device 150 will be described. FIG. 8 is an example of a flowchart showing the flow of the learning process.

[0087] In step S801, the learning device 150 acquires image data and a signal indicating the state of the fuel nozzle from the image data collection device 140.

[0088] In step S802, the learning device 150 extracts the image data during the period from when the fuel nozzle is removed from the nozzle holder until it is returned to the nozzle holder, and performs image recognition processing on the extracted image data.

[0089] In step S803, the learning device 150 specifies the first region and the second region in the image data used for generating the first learning data among the image data on which the image recognition processing has been performed.

[0090] In step S804, the learning device 150 assigns annotations to the image data used for generating the first learning data. Specifically, the learning device 150 assigns annotations to the bounding boxes of the recognition targets included in the specified first region, and also assigns annotations to the bounding boxes of the recognition targets included in the specified second region.

[0091] Also, the learning device 150 assigns annotations to the image data used for generating the second learning data. Specifically, the learning device 150 assigns annotations to the bounding boxes of the recognition targets.

[0092] In step S805, the learning device 150 generates first learning data for the first fuel type and second learning data for the second fuel type based on the image data with annotations.

[0093] In step S806, the learning device 150 performs learning processing on the first model using the first learning data to generate a first learned model. Also, the learning device 150 performs learning processing on the second model using the second learning data to generate a second learned model.

[0094] In step S807, the learning device 150 transmits the generated first learned model and second learned model to the information processing device 160 etc. and stores them in the information processing device 160 etc.

[0095] <Hardware Configuration of Information Processing Device> Next, the hardware configuration of the information processing device 160 will be described. FIG. 9 is a diagram showing an example of the hardware configuration of the information processing device.

[0096] As shown in FIG. 9, the information processing device 160 includes a processor 901, a memory 902, an auxiliary storage device 903, a connection device 904, an I / F device 905, a communication device 906, and a drive device 907. Note that the processor 901, memory 902, auxiliary storage device 903, connection device 904, I / F device 905, communication device 906, and drive device 907 of the information processing device 160 are interconnected via a bus 908.

[0097] The processor 901 includes various arithmetic devices such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 901 reads out and executes various programs (for example, a fuel supply control program etc.) on the memory 902.

[0098] The memory 902 includes main memory devices such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The processor 901 and the memory 902 form a so-called computer, and the computer realizes various functions by the processor 901 executing various programs read out on the memory 902.

[0099] The auxiliary storage device 903 stores various programs and various information used when the various programs are executed by the processor 901.

[0100] The connection device 904 is a connection device for connecting the first fuel dispenser 120_1, the second fuel dispenser 120_2, and the information processing device 160. The connection device 904 receives signals indicating the state of the fuel nozzle, signals indicating the fuel quantity, etc. from the first fuel dispenser 120_1 and the second fuel dispenser 120_2. Signals indicating the state of the fuel nozzle, signals indicating the fuel quantity, etc. are measured by the first fuel sensor 921_1 and the second fuel sensor 921_2. Further, the connection device 904 transmits a message for voice output to the first fuel dispenser 120_1 and the second fuel dispenser 120_2 via the voice output devices 122_1 and 122_2. Also, the connection device 904 transmits display information for display to the first fuel dispenser 120_1 and the second fuel dispenser 120_2 via the fuel data displays 123_1 and 123_2. In addition, the connection device 904 transmits a signal for instructing the opening / closing operation of the fuel valves 124_1 and 124_2 to the first fuel dispenser 120_1 and the second fuel dispenser 120_2. Furthermore, the connection device 904 transmits a signal for instructing the start / stop of driving to the fuel pumps 125_1 and 125_2.

[0101] Also, the connection device 904 receives thermal sensor data from the thermal sensors 170_1 and 170_2.

[0102] The I / F device 905 is a connection device for connecting an operation device 911, which is an example of an external device, a display device 912, and the information processing device 160. Note that the operation device 911 is operated by, for example, the staff of the service station 110', and the display device 912 is displayed for, for example, the staff of the service station 110'.

[0103] The communication device 906 communicates with the cameras 130_1 and 130_2 via a network and receives image data. Also, the communication device 906 communicates with the learning device 150 and receives the first learned model and the second learned model.

[0104] The drive device 907 is a device for setting the recording medium 913. The recording medium 913 here includes media that record information optically, electrically, or magnetically, such as CD-ROMs, flexible disks, magneto-optical disks, etc. Further, the recording medium 913 may include semiconductor memories that record information electrically, such as ROMs, flash memories, etc.

[0105] Note that various programs installed in the auxiliary storage device 903 are installed, for example, when the distributed recording medium 913 is set in the drive device 907 and various programs recorded on the recording medium 913 are read out. Alternatively, various programs installed in the auxiliary storage device 903 may be installed by being downloaded from a network via the communication device 906.

[0106] <Functional Configuration of Information Processing Device> Next, the functional configuration of the information processing device 160 will be described. FIG. 10 is a diagram showing an example of the functional configuration of the information processing device. As described above, a fuel supply control program is installed in the information processing device 160, and when the program is executed, the information processing device 160 · First fuel supply type image data acquisition unit 1001, second fuel supply type image data acquisition unit 1002, · First fuel supply sensor data acquisition unit 1003, second fuel supply sensor data acquisition unit 1004, · Image recognition unit 1005, · Section determination units 1006, 1007, · Thermal sensor data acquisition unit 1008, · First fuel supply monitoring units 1009_1, 1009_2, · Output control unit 1010, has.

[0107] The first fuel supply type image data acquisition unit 1001 acquires the image data captured by the camera 130_1 and notifies the image recognition unit 1005.

[0108] The second fuel supply type image data acquisition unit 1002 acquires the image data captured by the camera 130_2 and notifies the image recognition unit 1005.

[0109] The first fuel supply sensor data acquisition unit 1003 acquires the signal from the first fuel supply sensor 921_1 and notifies the section determination unit 1006.

[0110] The second fuel supply sensor data acquisition unit 1004 acquires the signal from the second fuel supply sensor 921_2 and notifies the section determination unit 1007.

[0111] Based on the signal from the first fuel supply sensor 921_1, the section determination unit 1006 classifies the section from when the fuel supply nozzle 121_1 is removed from the nozzle holder of the first fuel dispenser 120_1 until it is returned to the nozzle holder into three sections. Among the three sections, the section before fuel supply start is referred to as the first section, the section during fuel supply is referred to as the second section, and the section after fuel supply end is referred to as the third section.

[0112] Based on the signal from the second fuel supply sensor 921_2, the section determination unit 1007 classifies the section from when the fuel supply nozzle 121_2 is removed from the nozzle holder of the second fuel dispenser 120_2 until it is returned to the nozzle holder into three sections. For the second fuel supply type as well, similar to the first fuel supply type, among the three sections, the section before fuel supply start is referred to as the first section, the section during fuel supply is referred to as the second section, and the section after fuel supply end is referred to as the third section.

[0113] The image recognition unit 1005 extracts the image data included in any one of the first to third sections from the image data notified by the first fuel supply type image data acquisition unit 1001. Further, the image recognition unit 1005 performs image recognition processing on the extracted image data using the first learned model and notifies the recognition result to the first fuel supply monitoring unit 1009_1.

[0114] Further, the image recognition unit 1005 extracts the image data included in any one of the first to third intervals from the image data notified by the second fuel type image data acquisition unit 1002. Further, the image recognition unit 1005 performs image recognition processing on the extracted image data using the second learned model, and notifies the recognition result to the second fuel supply monitoring unit 1009_2.

[0115] The thermal sensor data acquisition unit 1008 acquires the thermal sensor data measured by the thermal sensor 170_1 and notifies it to the first fuel supply monitoring unit 1009_1. Further, the thermal sensor data acquisition unit 1008 acquires the thermal sensor data measured by the thermal sensor 170_2 and notifies it to the second fuel supply monitoring unit 1009_2.

[0116] The first fuel supply monitoring unit 1009_1 · The recognition result notified by the image recognition unit 1005, · The section determination result notified by the section determination unit 1006, · The thermal sensor data of the thermal sensor 170_1 notified by the thermal sensor data acquisition unit 1008, are used to determine whether fuel supply for the first fuel type is permitted. Further, the first fuel supply monitoring unit 1009_1 controls the fuel supply for the first fuel type based on the determination result of whether fuel supply is permitted. Specifically, the first fuel supply monitoring unit 1009_1 controls the fuel supply for the first fuel type by controlling the fuel valve 124_1 or the fuel pump 125_1.

[0117] Further, the first fuel supply monitoring unit 1009_1 notifies the output control unit 1010 of the determination result of whether fuel supply is permitted. In addition, when the first fuel supply monitoring unit 1009_1 determines that fuel supply is not permitted, it also notifies the output control unit 1010 of the information indicating the cause of non-permission of fuel supply.

[0118] The second fuel supply monitoring unit 1009_2 · The recognition result notified by the image recognition unit 1005, · The section determination result notified by the section determination unit 1007, · The thermal sensor data of the thermal sensor 170_2 notified by the thermal sensor data acquisition unit 1008, and is used to determine whether refueling is permitted for the second refueling type. Also, the second refueling monitoring unit 1009_2 controls the refueling for the second refueling type based on the determination result of whether refueling is permitted. Specifically, the second refueling monitoring unit 1009_2 controls the refueling for the second refueling type by controlling the refueling valve 124_2 or the refueling pump 125_2.

[0119] In addition, the second refueling monitoring unit 1009_2 notifies the output control unit 1010 of the determination result of whether refueling is permitted. Note that when the second refueling monitoring unit 1009_2 determines that refueling is not permitted, it also notifies the output control unit 1010 of the information indicating the cause of the non-permission of refueling.

[0120] When the output control unit 1010 is notified of the information indicating the cause of the non-permission of refueling from the first refueling monitoring unit 1009_1, · Controls so that a message corresponding to the information indicating the cause is output as voice via the voice output device 122_1, · Controls so that the display information corresponding to the information indicating the cause is displayed and output on the refueling data display 123_1 of the first refueling machine 120_1, · Controls so that the information indicating the cause is displayed and output on the display device 912 of the information processing device 160.

[0121] Also, when the output control unit 1010 is notified of the information indicating the cause of the non-permission of refueling from the second refueling monitoring unit 1009_2, · Controls so that a message corresponding to the information indicating the cause is output as voice via the voice output device 122_2, · Controls so that the display information corresponding to the information indicating the cause is displayed and output on the refueling data display 123_2 of the second refueling machine 120_2, · Controls so that the information indicating the cause is displayed and output on the display device 912 of the information processing device 160.

[0122] <Details of each part of the information processing device> Next, details of each part of the information processing apparatus 160 (here, the section determination units 1006 and 1007, the image recognition unit 1005, the first fuel supply monitoring unit 1009_1, the second fuel supply monitoring unit 1009_2, and the output control unit 1010) will be described.

[0123] (1) Details of the section determination units 1006 and 1007 First, as details of the section determination units 1006 and 1007, the functional configurations of the section determination units 1006 and 1007 and specific examples of the processing of the section determination units 1006 and 1007 will be described.

[0124] (1-1) Details of the functional configurations of the section determination units 1006 and 1007 FIG. 11 is a diagram showing an example of the functional configuration of the section determination unit of the information processing apparatus. Since the functional configurations of the section determination unit 1006 and the section determination unit 1007 are the same, they will be described together using FIG. 11.

[0125] As shown in FIG. 11, the section determination units 1006 and 1007 · nozzle state determination unit 1101, · fuel supply amount determination unit 1102, · first section start determination unit 1103, second section transition determination unit 1104, third section transition determination unit 1105, third section end determination unit 1106, · section determination result output unit 1107, have.

[0126] The nozzle state determination unit 1101 acquires a signal indicating the state of the fuel supply nozzle 121_1 (or 121_2) from the first fuel supply sensor data acquisition unit 1003 (or the second fuel supply sensor data acquisition unit 1004).

[0127] When the nozzle state determination unit 1101 receives a signal indicating that the fuel supply nozzle 121_1 (or 121_2) has been removed from the nozzle mount of the first fuel dispenser 120_1 (or the second fuel dispenser 120_2), it notifies the first section start determination unit 1103.

[0128] Further, when the nozzle state determination unit 1101 receives a signal indicating that the fuel supply nozzle 121_1 (or 121_2) has been returned to the nozzle hook of the first fuel dispenser 120_1 (or the second fuel dispenser 120_2), it notifies the third section end determination unit 1106.

[0129] The fuel supply amount determination unit 1102 acquires a signal indicating the fuel supply amount from the first fuel supply sensor data acquisition unit 1003 (or the second fuel supply sensor data acquisition unit 1004).

[0130] When the signal indicating the fuel supply amount acquired by the fuel supply amount determination unit 1102 from the first fuel supply sensor data acquisition unit 1003 (or the second fuel supply sensor data acquisition unit 1004) changes from zero to a predetermined threshold value or more, the fuel supply amount determination unit 1102 notifies the second section transition determination unit 1104.

[0131] Further, when the signal indicating the fuel supply amount acquired by the fuel supply amount determination unit 1102 from the first fuel supply sensor data acquisition unit 1003 (or the second fuel supply sensor data acquisition unit 1004) changes from a predetermined threshold value or more to zero, the fuel supply amount determination unit 1102 notifies the third section transition determination unit 1105.

[0132] When the first section start determination unit 1103 is notified from the nozzle state determination unit 1101 of a signal indicating that the fuel supply nozzle 121_1 (or 121_2) has been removed, it determines that the first section has started. Further, the first section start determination unit 1103 notifies the second section transition determination unit 1104 and the section determination result output unit 1107 that the first section has started.

[0133] After being notified that the first section has started, when the second section transition determination unit 1104 is notified from the fuel supply amount determination unit 1102 that the fuel supply amount has changed from zero to a predetermined threshold value or more, it determines that the transition has been made from the first section to the second section. Further, the second section transition determination unit 1104 notifies the third section transition determination unit 1105 and the section determination result output unit 1107 that the transition has been made from the first section to the second section.

[0134] After being notified that the transition to the second section has occurred, the third-section transition determination unit 1105 determines that the transition from the second section to the third section has occurred when it is notified from the fuel supply amount determination unit 1102 that the fuel supply amount has changed from equal to or greater than a predetermined threshold value to zero. Further, the third-section transition determination unit 1105 notifies the third-section end determination unit 1106 and the section determination result output unit 1107 that the transition from the second section to the third section has occurred.

[0135] After being notified that the transition to the third section has occurred, the third-section end determination unit 1106 determines that the third section has ended when it is notified from the nozzle state determination unit 1101 that a signal indicating that the fuel supply nozzle 121_1 (or 121_2) has been returned is notified. Further, the third-section end determination unit 1106 notifies the section determination result output unit 1107 that the third section has ended.

[0136] When the section determination result output unit 1107 is notified from the first-section start determination unit 1103 that the first section has started, the section determination result output unit 1107 starts outputting information indicating that it is in the first section. Further, when the section determination result output unit 1107 is notified from the second-section transition determination unit 1104 that the transition to the second section has occurred, the section determination result output unit 1107 stops outputting information indicating that it is in the first section and starts outputting information indicating that it is in the second section. Further, when the section determination result output unit 1107 is notified from the third-section transition determination unit 1105 that the transition to the third section has occurred, the section determination result output unit 1107 stops outputting information indicating that it is in the second section and starts outputting information indicating that it is in the third section. Further, when the section determination result output unit 1107 is notified from the third-section end determination unit 1106 that the third section has ended, the section determination result output unit 1107 stops outputting information indicating that it is in the third section.

[0137] (1-2) Specific examples of the processing of the section determination units 1006 and 1007 FIG. 12 is a diagram showing a specific example of the processing by the section determination unit of the information processing apparatus. As shown in FIG. 12(a), when a signal indicating that the fueling nozzle 121_1 has been removed from the nozzle holder is acquired as a signal indicating the state of the fueling nozzle 121_1, the section determination unit 1006 starts outputting information indicating that it is in the first section. Also, when a signal indicating that the signal indicating the fueling amount of the first fuel dispenser 120_1 has changed from zero to a predetermined threshold value or more is acquired, the section determination unit 1006 starts outputting information indicating that it is in the second section. Further, when a signal indicating that the signal indicating the fueling amount of the first fuel dispenser 120_1 has changed from a predetermined threshold value or more to zero is acquired, the section determination unit 1006 starts outputting information indicating that it is in the third section. Furthermore, when a signal indicating that the fueling nozzle 121_1 has been returned to the nozzle holder is acquired as a signal indicating the state of the fueling nozzle 121_1, the section determination unit 1006 stops outputting information indicating that it is in the third section.

[0138] Similarly, as shown in FIG. 12(b), when a signal indicating that the fueling nozzle 121_2 has been removed from the nozzle holder is acquired as a signal indicating the state of the fueling nozzle 121_2, the section determination unit 1007 starts outputting information indicating that it is in the first section. Also, when a signal indicating that the signal indicating the fueling amount of the second fuel dispenser 120_2 has changed from zero to a predetermined threshold value or more is acquired, the section determination unit 1007 starts outputting information indicating that it is in the second section. Further, when a signal indicating that the signal indicating the fueling amount of the second fuel dispenser 120_2 has changed from a predetermined threshold value or more to zero is acquired, the section determination unit 1007 starts outputting information indicating that it is in the third section. Furthermore, when a signal indicating that the fueling nozzle 121_2 has been returned to the nozzle holder is acquired as a signal indicating the state of the fueling nozzle 121_2, the section determination unit 1007 stops outputting information indicating that it is in the third section.

[0139] (2) Details of the image recognition unit 1005 Next, as details of the image recognition unit 1005, the functional configuration of the image recognition unit 1005 and a specific example of the processing of the image recognition unit 1005 will be described.

[0140] (2-1) Details of the functional configuration of the image recognition unit 1005 FIG. 13 is a diagram showing an example of the functional configuration of the image recognition unit of the information processing apparatus. As shown in FIG. 13, the image recognition unit 1005 includes a recognition section acquisition unit 1301, a first learned model 1302, and a region determination unit 1303. Further, the image recognition unit 1005 includes a recognition section acquisition unit 1311 and a second learned model 1312.

[0141] The recognition section acquisition unit 1301 extracts the image data included in any one of the first to third sections notified from the section determination unit 1006 from the image data notified from the first fuel type image data acquisition unit 1001, and inputs the extracted image data to the first learned model 1302.

[0142] The first learned model 1302 is a first learned model generated in the learning device 150. When image data is input, it recognizes the recognition target included in the image data and outputs the recognition result.

[0143] The region determination unit 1303 determines whether the recognition result output by the first learned model 1302 is included in the first region 210 or the second region 220. Further, the region determination unit 1303 outputs the recognition result determined to be included in the first region 210 or the second region 220 to the first fuel monitoring unit 1009_1.

[0144] The recognition section acquisition unit 1311 extracts the image data included in any one of the first to third sections notified from the section determination unit 1007 from the image data notified from the second fuel type image data acquisition unit 1002, and inputs the extracted image data to the second learned model 1312.

[0145] The second learned model 1312 is a second learned model generated in the learning device 150. When image data is input, it recognizes the recognition target included in the image data and outputs the recognition result to the second fuel monitoring unit 1009_2.

[0146] (2-2) Specific Example of the Processing of the Image Recognition Unit 1005 (in the Case of the First Fueling Type) FIG. 14A is a first diagram showing a specific example of the processing by the image recognition unit of the information processing apparatus. As shown in FIG. 14A, the image recognition unit 1005 extracts the image data included in the section (the first to third sections) from when the fueling nozzle 121_1 is removed from the nozzle holder until it is returned, among the image data captured by the camera 130_1.

[0147] Also, as shown in FIG. 14A, the image recognition unit 1005 performs image recognition processing on the extracted image data using the first pre-trained model 1302, and by recognizing the recognition targets (such as vehicles, containers, fuelers, fueling nozzles, etc.), · Vehicles, · Containers, · Fuelers who hold the fueling nozzle in an appropriate posture, · Fuelers who do not hold the fueling nozzle in an appropriate posture (fueling nozzle unrecognizable), · Fuelers who do not hold the fueling nozzle in an appropriate posture (fueler unrecognizable), are output as recognition results.

[0148] Also, the image recognition unit 1005 outputs the recognition results determined to be included in the first area 210 or the second area 220. The recognition results 1401 to 1404 in FIG. 14A show output examples of the recognition results.

[0149] In the example of FIG. 14A, the recognition result 1401 output by the image recognition unit 1005 is · In the first area 210, the bounding box in which a vehicle is recognized, · In the second area 220, the bounding box in which a fueler who holds the fueling nozzle in an appropriate posture is recognized, as shown.

[0150] Also, in the example of FIG. 14A, the recognition result 1402 output by the image recognition unit 1005 is · In the first area 210, the bounding box in which a vehicle is recognized, · In the second area 220, a bounding box (fuel nozzle unrecognizable) in which a fueler not holding the fuel nozzle in an appropriate posture is recognized is shown.

[0151] Also, in the example of FIG. 14A, the recognition result 1403 output by the image recognition unit 1005 is · A bounding box in which a vehicle is recognized in the first area 210, · In the second area 220, a bounding box (fueler unrecognizable) in which a fueler not holding the fuel nozzle in an appropriate posture is recognized is shown.

[0152] Furthermore, in the example of FIG. 14A, the recognition result 1404 output by the image recognition unit 1005 is · A bounding box in which a container is recognized in the first area 210, · In the second area 220, a bounding box in which a fueler holding the fuel nozzle in an appropriate posture is recognized is shown.

[0153] (2-3) Specific example of the process of the image recognition unit 1005 (in the case of the second refueling type) FIG. 14B is a second diagram showing a specific example of the process by the image recognition unit of the information processing apparatus. As shown in FIG. 14B, the image recognition unit 1005 extracts the image data included in the section (first to third sections) from when the fuel nozzle 121_2 is removed from the nozzle holder until it is returned, among the image data captured by the camera 130_2.

[0154] Also, as shown in FIG. 14B, the image recognition unit 1005 performs image recognition processing on the extracted image data using the second learned model 1312, and by recognizing the recognition targets (container, fueler, fuel nozzle, etc.), · Container, · A fueler holding the fuel nozzle in an appropriate posture, · A fueler not holding the fuel nozzle in an appropriate posture (fuel nozzle unrecognizable), · A refueler who is not holding the fuel nozzle in an appropriate posture (refueler cannot be recognized), is output as the recognition result. The recognition results 1411 to 1414 in FIG. 14B show output examples of the recognition result.

[0155] In the example of FIG. 14B, the recognition result 1411 output by the image recognition unit 1005 is · The bounding box in which the container is recognized, · The bounding box in which a refueler holding the fuel nozzle in an appropriate posture is recognized. is shown.

[0156] Also, in the example of FIG. 14B, the recognition result 1412 output by the image recognition unit 1005 is · The bounding box in which the container is recognized, · The bounding box in which a refueler who is not holding the fuel nozzle in an appropriate posture is recognized (fuel nozzle cannot be recognized). is shown.

[0157] Also, in the example of FIG. 14B, the recognition result 1413 output by the image recognition unit 1005 is · The bounding box in which the container is recognized, · The bounding box in which a refueler who is not holding the fuel nozzle in an appropriate posture is recognized (refueler cannot be recognized). is shown.

[0158] Furthermore, in the example of FIG. 14B, the recognition result 1414 output by the image recognition unit 1005 is · The bounding box in which a refueler holding the fuel nozzle in an appropriate posture is recognized. is shown.

[0159] (3) Details of the first fueling monitoring unit 1009_1 and the second fueling monitoring unit 1009_2 Next, as details of the first fuel supply monitoring unit 1009_1 and the second fuel supply monitoring unit 1009_2, the functional configuration and processing flow of the first fuel supply monitoring unit 1009_1 and the functional configuration and processing flow of the second fuel supply monitoring unit 1009_2 will be described.

[0160] (3-1) Details of the Functional Configuration of the First Fuel Supply Monitoring Unit 1009_1 and the Second Fuel Supply Monitoring Unit 1009_2 FIG. 15A is a diagram showing an example of the functional configuration of the first fuel supply monitoring unit of the information processing apparatus. As shown in FIG. 15A, the first fuel supply monitoring unit 1009_1 includes a recognition result acquisition unit 1501, a determination logic switching unit 1502, a poly-tank determination unit 1506, a first determination logic 1503, a second determination logic 1504, a third determination logic 1505, and a fire determination unit 1507. Further, the first fuel supply monitoring unit 1009_1 includes a fuel supply control unit 1508 and a cause information acquisition unit 1509. Note that the first determination logic 1503 to the third determination logic 1505, the poly-tank determination unit 1506, and the fire determination unit 1507 are collectively referred to as the first determination unit.

[0161] The recognition result acquisition unit 1501 acquires the recognition results for the image data included in the first section to the third section from the image recognition unit 1005. The recognition result acquisition unit 1501 notifies the acquired recognition results to the determination logic switching unit 1502 and the poly-tank determination unit 1506.

[0162] The determination logic switching unit 1502 inputs the recognition result notified from the recognition result acquisition unit 1501 into any one of the first determination logic 1503 to the third determination logic 1505. Which one of the first determination logic 1503 to the third determination logic 1505 to input is determined by the information indicating in which of the first to third sections is notified from the section determination unit 1006. For example, while the information indicating that it is in the first section is notified, the determination logic switching unit 1502 inputs the recognition result into the first determination logic 1503. Also, while the information indicating that it is in the second section is notified, the determination logic switching unit 1502 inputs the recognition result into the second determination logic 1504. Also, while the information indicating that it is in the third section is notified, the determination logic switching unit 1502 inputs the recognition result into the third determination logic 1505.

[0163] Also, the determination logic switching unit 1502 notifies the poly tank determination unit 1506 and the firearm determination unit 1507 of the information indicating that it is in any one of the first to third sections.

[0164] When the poly tank determination unit 1506 is notified of the information indicating that it is in any one of the first to third sections and the recognition result is notified from the recognition result acquisition unit 1501, the poly tank determination unit 1506 determines whether a container is included in the notified recognition result. When the poly tank determination unit 1506 determines that a container is included, it notifies the fuel supply control unit 1508 of fuel supply non - permission.

[0165] The first determination logic 1503 determines whether fuel supply is permitted or not based on the recognition result in the first section. Also, the first determination logic 1503 notifies the fuel supply control unit 1508 of the determination result of whether fuel supply is permitted or not in the first section. When the first determination logic 1503 determines fuel supply non - permission, when notifying the determination result to the fuel supply control unit 1508, it also notifies the information indicating the reason for determining fuel supply non - permission.

[0166] The second determination logic 1504 determines whether refueling is permitted or not based on the recognition results during the second period. Further, the second determination logic 1504 notifies the refueling control unit 1508 of the determination result of whether refueling is permitted or not during the second period. Note that when the second determination logic 1504 determines that refueling is not permitted, it notifies the refueling control unit 1508 of the determination result and, at the same time, notifies information indicating the reason for determining that refueling is not permitted.

[0167] The third determination logic 1505 determines whether refueling is permitted or not based on the recognition results during the third period. Further, the third determination logic 1505 notifies the refueling control unit 1508 of the determination result of whether refueling is permitted or not during the third period. Note that when the third determination logic 1505 determines that refueling is not permitted, it notifies the refueling control unit 1508 of the determination result and, at the same time, notifies information indicating the reason for determining that refueling is not permitted.

[0168] The fire determination unit 1507 acquires the thermal sensor data measured by the thermal sensor 170_1 from the thermal sensor data acquisition unit 1008. Further, when information indicating that it is in any one of the first to third periods is notified, the fire determination unit 1507 determines whether or not a temperature region equal to or higher than a predetermined threshold is included. When the fire determination unit 1507 determines that a temperature region equal to or higher than the predetermined threshold is included, it notifies the refueling control unit 1508 that refueling is not permitted.

[0169] The refueling control unit 1508 controls the fueling valve 124_1 or the fuel pump 125_1 based on whether refueling is permitted or not, thereby controlling the fueling for the first fuel type. For example, when it is notified that refueling is permitted, the refueling control unit 1508 drives the fuel pump 125_1 to open the fueling valve 124_1. Further, the refueling control unit 1508 · When notified of refueling permission from the poly-tank determination unit 1506, drives the fuel pump 125_1 to open the fueling valve 124_1. Furthermore, · When notified of refueling non-permission from any one of the first determination logic 1503 to the third determination logic 1505, or · When a notification of refueling prohibition is received from the ignition determination unit 1507, the refueling is stopped by stopping the drive of the fuel pump 125_1, or the refueling is prohibited by stopping the drive of the fuel pump 125_1 and closing the fuel valve 124_1. Note that the definitions of refueling stop and refueling prohibition here are only examples, and the refueling stop and refueling prohibition may be defined by other definitions. Alternatively, without distinguishing between refueling stop and refueling prohibition, the entire state where refueling cannot be performed may be referred to as refueling stop or prohibition.

[0170] In addition, the fuel supply control unit 1508 notifies the cause information acquisition unit 1509 of fuel supply permission and fuel supply non - permission. When the fuel supply control unit 1508 notifies fuel supply non - permission, it also notifies the cause information acquisition unit 1509 of information indicating the cause of the determination of fuel supply non - permission.

[0171] The cause information acquisition unit 1509 outputs fuel supply permission and fuel supply non - permission to the output control unit 1010. When the cause information acquisition unit 1509 outputs fuel supply non - permission, it also outputs information indicating the cause of fuel supply non - permission (such as the first to fourth information, warning information, confirmation request information) notified from the fuel supply control unit 1508 to the output control unit 1010.

[0172] Figure 15B is a diagram showing an example of the functional configuration of the second fuel supply monitoring unit of the information processing device. The difference from the functional configuration of the first fuel supply monitoring unit 1009_1 shown in Figure 15A is that the functional configuration of the second fuel supply monitoring unit 1009_2 does not include a functional unit corresponding to the poly - tank determination unit 1506. Since the other functional units of the second fuel supply monitoring unit 1009_2 are the same as those of the first fuel supply monitoring unit 1009_1, the description is omitted here.

[0173] Note that what was referred to as the first determination unit in the first fuel supply monitoring unit 1009_1 is collectively referred to as the second determination unit in the second fuel supply monitoring unit 1009_2, including the first determination logic 1513 to the third determination logic 1515 and the ignition determination unit 1517.

[0174] (3-2) Flow of processing using the first determination logic 1503 of the first fueling monitoring unit 1009_1 Next, the flow of processing using the first determination logic 1503 of the first fueling monitoring unit 1009_1 will be described with reference to FIGS. 16A and 16B. FIGS. 16A and 16B are examples of the first and second flowcharts showing the flow of processing using the first determination logic of the first fueling monitoring unit.

[0175] In step S1601, the first determination logic 1503 determines whether the currently input recognition result is a recognition result determined to be fueling permitted (whether a predetermined permission condition is satisfied). Specifically, it is determined whether a vehicle is recognized in the first area 210 and whether a fueler holding the fueling nozzle in an appropriate posture is recognized in the second area.

[0176] If it is determined in step S1601 that the recognition result is a recognition result determined to be fueling permitted (YES in step S1601), the process proceeds to step S1602.

[0177] In step S1602, the first determination logic 1503 determines whether the recognition result determined to be fueling permitted has continued for a predetermined time T1 (whether a predetermined permission condition has been satisfied for a certain time). If it is determined in step S1602 that it has continued for the predetermined time T1 (YES in step S1602), the process proceeds to step S1603.

[0178] In step S1603, the first determination logic 1503 determines that fueling is permitted and proceeds to step S1614 in FIG. 16B.

[0179] On the other hand, if it is determined in step S1601 that the recognition result is not a recognition result determined to be fueling permitted (NO in step S1601), the process proceeds to step S1604. Or, if it is determined in step S1602 that it has not continued for the predetermined time T1 (NO in step S1602), the process proceeds to step S1604.

[0180] In step S1604, the first determination logic 1503 determines whether a state where the recognition result is not determined to be refueling permission continues for a predetermined time T2 (where T2 > T1).

[0181] In step S1604, if it is determined that it has not continued for the predetermined time T2 (if NO in step S1604), the process returns to step S1601. On the other hand, in step S1604, if it is determined that it has continued for the predetermined time T2 (if YES in step S1604), the process proceeds to step S1605.

[0182] In step S1605, the first determination logic 1503 determines whether the currently input recognition result is a recognition result determined to be refueling not permitted (whether a predetermined non - permission condition is satisfied). Specifically, it is determined whether a vehicle is recognized in the first area 210 and whether a refueler who is not holding the fuel nozzle in an appropriate posture is recognized (whether the fuel nozzle cannot be recognized) in the second area.

[0183] In step S1605, if it is determined that the recognition result is a recognition result determined to be refueling not permitted (if YES in step S1605), it is determined that the predetermined non - permission condition has been satisfied for a certain time, and the process proceeds to step S1606.

[0184] In step S1606, the first determination logic 1503 determines that refueling is not permitted, acquires the first information, and then proceeds to step S1611 in FIG. 16B.

[0185] On the other hand, in step S1605, if it is determined that the recognition result is another recognition result determined to be refueling not permitted (if NO in step S1605), the process proceeds to step S1617 in FIG. 16B.

[0186] In step S1611, the first determination logic 1503 determines whether the currently input recognition result is a recognition result determined as fueling permission (whether it satisfies a predetermined permission condition). Specifically, it determines whether a vehicle is recognized in the first area 210 and whether a fueler who holds the fuel nozzle in an appropriate posture is recognized in the second area.

[0187] In step S1611, if it is determined that the recognition result is not a recognition result determined as fueling permission (if NO in step S1611), the first fueling monitoring unit 1009_1 ends the process using the first determination logic 1503.

[0188] On the other hand, in step S1611, if it is determined that the recognition result is a recognition result determined as fueling permission (if YES in step S1611), the process proceeds to step S1612.

[0189] In step S1612, the first determination logic 1503 determines whether the recognition result determined as fueling permission has continued for a predetermined time T1 (whether the predetermined permission condition has been satisfied for a certain time). In step S1612, if it is determined that it has continued for the predetermined time T1 (if YES in step S1612), the process proceeds to step S1613.

[0190] In step S1613, the first determination logic 1503 determines fueling permission and proceeds to step S1614.

[0191] In step S1614, the determination logic switching unit 1502 determines whether it is within the first section. In step S1614, if it is determined that it is within the first section, it is determined that the process by the first determination logic 1503 is not ended (determined as NO in step S1614), and the process returns to step S1601 in FIG. 16A.

[0192] On the other hand, in step S1614, when it is determined that it is not within the first interval, it is determined to end the process by the first determination logic 1503 (determined as YES in step S1614). Then, the first fueling monitoring unit 1009_1 ends the process using the first determination logic 1503.

[0193] On the other hand, in step S1612, when it is determined that it has not continued for the predetermined time T1 (when NO in step S1612), the process proceeds to step S1615.

[0194] In step S1615, the first determination logic 1503 determines whether a state where the recognition result is not a fueling permission has continued for the predetermined time T3 (however, T3 > T1).

[0195] In step S1615, when it is determined that it has not continued for the predetermined time T3 (when NO in step S1615), the process returns to step S1611. On the other hand, in step S1615, when it is determined that it has continued for the predetermined time T3 (when YES in step S1615), it is determined that a predetermined non - permission condition has been satisfied for a certain period of time, and the process proceeds to step S1616.

[0196] In step S1616, the first determination logic 1503 determines non - fueling permission and acquires the first information. After that, the first fueling monitoring unit 1009_1 ends the process using the first determination logic 1503.

[0197] On the other hand, when proceeding from step S1605 in FIG. 16A to step S1617 in FIG. 16B, the first determination logic 1503 determines that a predetermined non - permission condition has been satisfied for a certain period of time. Specifically, the first determination logic 1503 determines that the vehicle has not been recognized in the first area 210. Alternatively, the first determination logic 1503 determines that a fueler who is not holding the fueling nozzle in an appropriate posture has been recognized (fueler recognition impossible) in the second area 220.

[0198] In step S1618, the first determination logic 1503 determines that refueling is not permitted. After acquiring the second information, the first refueling monitoring unit 1009_1 ends the process using the first determination logic 1503.

[0199] (3-3) Flow of the process using the second determination logic 1504 of the first refueling monitoring unit 1009_1 Next, the flow of the process using the second determination logic 1504 of the first refueling monitoring unit 1009_1 will be described with reference to FIGS. 17A to 17D. FIGS. 17A to 17D are examples of the first to fourth flowcharts showing the flow of the process using the second determination logic of the first refueling monitoring unit.

[0200] In step S1701, the second determination logic 1504 determines whether the currently input recognition result is a recognition result determined to permit refueling (whether a predetermined permission condition is satisfied). Specifically, it is determined whether a vehicle is recognized in the first area 210 and whether a refueler holding the fuel nozzle in an appropriate posture is recognized in the second area.

[0201] If it is determined in step 1701 that the recognition result is a recognition result determined to permit refueling (YES in step S1701), it is determined that the predetermined permission condition is satisfied, and the process proceeds to step S1702.

[0202] In step S1702, the second determination logic 1504 determines to permit refueling and proceeds to step S1734 in FIG. 17D.

[0203] On the other hand, if it is determined in step S1701 that the recognition result is not a recognition result determined to permit refueling (NO in step S1701), the process proceeds to step S1703.

[0204] In step S1703, the second determination logic 1504 determines whether the state where the recognition result is not a recognition result determined to permit refueling has continued for a predetermined time T4.

[0205] In step S1703, if it is determined that the predetermined time T4 has not continued (if NO in step S1703), the process returns to step S1701. On the other hand, in step S1703, if it is determined that the predetermined time T4 has continued (if YES in step S1703), the process proceeds to step S1704.

[0206] In step S1704, the second determination logic 1504 determines whether the currently input recognition result is a recognition result determined as refueling not permitted (whether a predetermined non - permission condition is satisfied). Specifically, it determines whether a vehicle is recognized in the first area 210 and whether a refueler who is not holding the fuel nozzle in an appropriate posture is recognized in the second area (whether the fuel nozzle cannot be recognized).

[0207] In step S1704, if it is determined that it is another recognition result determined as refueling not permitted (if NO in step S1704), the process proceeds to step S1721 in FIG. 17C.

[0208] On the other hand, in step S1704, if it is determined that the recognition result is a recognition result determined as refueling not permitted (if YES in step S1704), it is determined that the predetermined non - permission condition has been satisfied for a certain time, and the process proceeds to step S1705.

[0209] In step S1705, the second determination logic 1504 acquires warning information.

[0210] In step S1706, the second determination logic 1504 determines whether the currently input recognition result is a recognition result determined as refueling permitted (whether a predetermined permission condition is satisfied). Specifically, it determines whether a vehicle is recognized in the first area 210 and whether a refueler who is holding the fuel nozzle in an appropriate posture is recognized in the second area.

[0211] If it is determined in step S1706 that the recognition result is a determination of fueling permission (if YES in step S1706), the process proceeds to step S1707.

[0212] In step S1707, the second determination logic 1504 determines whether the recognition result determined as fueling permission has continued for a predetermined time T1. If it is determined in step S1707 that it has continued for the predetermined time T1 (if YES in step S1707), it is determined that the predetermined permission conditions have been satisfied for a certain period of time, and the process proceeds to step S1708.

[0213] In step S1708, the second determination logic 1504 determines fueling permission and proceeds to step S1734 in FIG. 17D.

[0214] On the other hand, if it is determined in step S1706 that the recognition result is not a determination of fueling permission (if NO in step S1706), the process proceeds to step S1709. Or, if it is determined in step S1707 that it has not continued for the predetermined time T1 (if NO in step S1707), the process proceeds to step S1709.

[0215] In step S1709, the second determination logic 1504 determines whether the state where the recognition result is not a determination of fueling permission has continued for a predetermined time T2 (where T2 > T1).

[0216] If it is determined in step S1709 that it has not continued for the predetermined time T2 (if NO in step S1709), the process returns to step S1706. On the other hand, if it is determined in step S1709 that it has continued for the predetermined time T2 (if YES in step S1709), it is determined that the predetermined non - permission conditions have been satisfied for a certain period of time, and the process proceeds to step S1711 in FIG. 17B.

[0217] In step S1711, the second determination logic 1504 determines fueling non - permission, acquires the first information, and then proceeds to step S1712.

[0218] In step S1712, the second determination logic 1504 determines whether, within a predetermined time T2, the currently input recognition result has become a recognition result determined to be refueling permitted (whether a predetermined permission condition is satisfied). Specifically, it is determined whether, within the predetermined time T2, a vehicle is recognized in the first area 210 and a refueler who holds the fuel nozzle in an appropriate posture is recognized in the second area.

[0219] In step S1712, if it is determined that the currently input recognition result has become a recognition result determined to be refueling permitted within the predetermined time T2 (if YES in step S1712), it is determined that the predetermined permission condition is satisfied, and the process proceeds to step S1713.

[0220] In step S1713, the second determination logic 1504 determines that refueling is permitted and proceeds to step S1734 in FIG. 17D.

[0221] On the other hand, in step S1712, if it is determined that the recognition result has not become a recognition result determined to be refueling permitted within the predetermined time T2 (if NO in step S1712), the first refueling monitoring unit 1009_1 ends the process using the second determination logic 1504.

[0222] On the other hand, when the process proceeds from step S1704 to step S1721 in FIG. 17C, the second determination logic 1504 determines that a predetermined non - permission condition has been satisfied for a certain period of time. Specifically, the second determination logic 1504 determines that a vehicle is not recognized in the first area 210. Alternatively, the second determination logic 1504 determines that a refueler who does not hold the fuel nozzle in an appropriate posture (refueler recognition impossible) is recognized in the second area 220.

[0223] In step S1722, the second determination logic 1504 acquires warning information.

[0224] In step S1723, the second determination logic 1504 determines whether the currently input recognition result is a recognition result determined to be fueling permission (whether it satisfies a predetermined permission condition). Specifically, it determines whether a vehicle is recognized in the first area 210 and whether a fueler who holds the fuel nozzle in an appropriate posture is recognized in the second area.

[0225] In step S1723, if it is determined that the recognition result is a recognition result determined to be fueling permission (if YES in step S1723), the process proceeds to step S1724.

[0226] In step S1724, the second determination logic 1504 determines whether the recognition result determined to be fueling permission has continued for a predetermined time T1. In step S1724, if it is determined that it has continued for the predetermined time T1 (if YES in step S1724), it is determined that the predetermined permission condition has been satisfied for a certain period of time, and the process proceeds to step S1725.

[0227] In step S1725, the second determination logic 1504 determines that fueling is permitted and proceeds to step S1734 in FIG. 17D.

[0228] On the other hand, in step S1723, if it is determined that the recognition result is not a recognition result determined to be fueling permission (if NO in step S1723), the process proceeds to step S1726. Or, in step S1724, if it is determined that it has not continued for the predetermined time T1 (if NO in step S1724), the process proceeds to step S1726.

[0229] In step S1726, the second determination logic 1504 determines whether the state where the recognition result is not a recognition result determined to be fueling permission has continued for a predetermined time T2 (where T2 > T1).

[0230] In step S1726, if it is determined that the predetermined time T2 has not elapsed continuously (if NO in step S1726), the process returns to step S1723. On the other hand, in step S1726, if it is determined that the predetermined time T2 has elapsed continuously (if YES in step S1726), it is determined that a predetermined non - permission condition has been satisfied for a certain period of time, and the process proceeds to step S1731 in FIG. 17D.

[0231] In step S1731, the second determination logic 1504 determines non - fueling permission, and after acquiring the second information, proceeds to step S1732.

[0232] In step S1732, the second determination logic 1504 determines whether the currently input recognition result has become a recognition result for which fueling is permitted (whether a predetermined permission condition has been satisfied) within the predetermined time T2. Specifically, it determines whether, within the predetermined time T2, the vehicle is recognized in the first area 210 and a fueler who holds the fuel nozzle in an appropriate posture is recognized in the second area.

[0233] In step S1732, if it is determined that the currently input recognition result has become a recognition result for which fueling is permitted within the predetermined time T2 (if YES in step S1732), it is determined that the predetermined permission condition has been satisfied, and the process proceeds to step S1733.

[0234] In step S1733, the second determination logic 1504 determines fueling permission and proceeds to step S1734.

[0235] In step S1734, the determination logic switching unit 1502 determines whether it is during the second section. In step S1734, if it is determined that it is during the second section, it is determined that the process by the second determination logic 1504 is not terminated (if determined NO in step S1734), and the process returns to step S1701 in FIG. 17A.

[0236] On the other hand, in step S1734, if it is determined that it is not within the second interval, it is determined to end the process by the second determination logic 1504 (determined as YES in step S1734). Then, the first fuel supply monitoring unit 1009_1 ends the process using the second determination logic 1504.

[0237] Also, in step S1732, if it is not determined that the recognition result is determined to be fuel supply permitted within the predetermined time T2 (if NO in step S1732), the first fuel supply monitoring unit 1009_1 ends the process using the second determination logic 1504.

[0238] (3-4) Flow of the process using the third determination logic 1505 of the first fuel supply monitoring unit 1009_1 Next, the flow of the process using the third determination logic 1505 of the first fuel supply monitoring unit 1009_1 will be described with reference to FIGS. 18A and 18B. FIGS. 18A and 18B are examples of the first and second flowcharts showing the flow of the process using the third determination logic of the first fuel supply monitoring unit.

[0239] In step S1801, the third determination logic 1505 determines whether the currently input recognition result is a recognition result determined to be fuel supply permitted (whether it satisfies a predetermined permission condition). Specifically, it determines whether a vehicle is recognized in the first area 210 and whether a fueler who holds the fuel nozzle in an appropriate posture is recognized in the second area.

[0240] In step 1801, if it is determined that the recognition result is a recognition result determined to be fuel supply permitted (if YES in step S1801), it is determined that the predetermined permission condition is satisfied, and the process proceeds to step S1802.

[0241] In step S1802, the third determination logic 1505 determines that fuel supply is permitted and proceeds to step S1813 in FIG. 18B.

[0242] On the other hand, if it is determined in step S1801 that the recognition result is not the one determined as fueling permission (if NO in step S1801), the process proceeds to step S1803.

[0243] In step S1803, the third determination logic 1505 determines whether or not the state where the recognition result is not the one determined as fueling permission has continued for a predetermined time T5.

[0244] In step S1803, if it is determined that it has not continued for the predetermined time T5 (if NO in step S1803), the process returns to step S1801. On the other hand, in step S1803, if it is determined that it has continued for the predetermined time T5 (if YES in step S1803), the process proceeds to step S1804.

[0245] In step S1804, the third determination logic 1505 determines whether the currently input recognition result is a recognition result determined as non - fueling permission (whether or not a predetermined non - permission condition is satisfied). Specifically, it is determined whether a vehicle is recognized in the first area 210 and whether a fueler who is not holding the fuel nozzle in an appropriate posture is recognized (unrecognizable fuel nozzle) in the second area.

[0246] In step S1804, if it is determined that it is another recognition result determined as non - fueling permission (if NO in step S1804), the process proceeds to step S1806.

[0247] In step S1806, the third determination logic 1505 determines that a predetermined non - permission condition has been satisfied for a certain period of time. Specifically, the third determination logic 1505 determines that no vehicle is recognized in the first area 210. Alternatively, the third determination logic 1505 determines that a fueler who is not holding the fuel nozzle in an appropriate posture is recognized (unrecognizable fueler) in the second area 220.

[0248] In step S1807, the third determination logic 1505 determines that refueling is not permitted. After acquiring the second information, it proceeds to step S1811 in FIG. 18B.

[0249] On the other hand, if it is determined in step S1804 that the recognition result is a recognition result indicating that refueling is not permitted (if YES in step S1804), it is determined that a predetermined non - permission condition has been satisfied for a certain period of time, and it proceeds to step S1805.

[0250] In step S1805, the third determination logic 1505 determines that refueling is not permitted. After acquiring the first information, it proceeds to step S1811 in FIG. 18B.

[0251] In step S1811, the third determination logic 1505 determines whether, within a predetermined time T2, the currently input recognition result has become a recognition result indicating that refueling is permitted (whether a predetermined permission condition has been satisfied). Specifically, it determines whether, within the predetermined time T2, a vehicle has been recognized in the first area 210 and a refueler who holds the fuel nozzle in an appropriate posture has been recognized in the second area.

[0252] If it is determined in step S1811 that, within the predetermined time T2, the currently input recognition result has become a recognition result indicating that refueling is permitted (if YES in step S1811), it is determined that a predetermined permission condition has been satisfied, and it proceeds to step S1812.

[0253] In step S1812, the third determination logic 1505 determines that refueling is permitted and proceeds to step S1813.

[0254] In step S1813, the determination logic switching unit 1502 determines whether it is during the third section. If it is determined in step S1813 that it is during the third section, it is determined that the processing by the third determination logic 1505 is not terminated (if determined NO in step S1813), and it returns to step S1801 in FIG. 18A.

[0255] On the other hand, in step S1813, if it is determined that it is not during the third period, it is determined to end the process by the third determination logic 1505 (determined as YES in step S1813). Then, the first fuel supply monitoring unit 1009_1 ends the process using the third determination logic 1505.

[0256] On the other hand, in step S1811, if it is determined that the currently input recognition result has not become a recognition result determined as fuel supply permission within the predetermined time T2 (if NO in step S1811), the process proceeds to step S1814.

[0257] In step S1814, the third determination logic 1505 determines whether the state of not being a recognition result determined as fuel supply permission has continued for the predetermined time T6. In step S1814, if it is determined that the state of not being a recognition result determined as fuel supply permission has not continued for the predetermined time T6 (if NO in step S1814), it waits until it continues for the predetermined time T6.

[0258] On the other hand, in step S1814, if it is determined that the state of not being a recognition result determined as fuel supply permission has continued for the predetermined time T6 (if YES in step S1814), the process proceeds to step S1815.

[0259] In step S1815, the third determination logic 1505 acquires confirmation request information, and the first fuel supply monitoring unit 1009_1 ends the process using the third determination logic 1505.

[0260] (3-5) Flow of processing using other determination logics of the first fuel supply monitoring unit 1009_1 Next, the flow of processing using other determination logics of the first fuel supply monitoring unit 1009_1 (processing using the poly-tank determination unit 1506 and processing using the fire determination unit 1507) will be described with reference to FIG. 19. FIG. 19 is an example of a flowchart showing the flow of processing using the poly-tank determination unit of the first fuel supply monitoring unit and the flow of processing using the fire determination unit.

[0261] First, the processing flow using the poly-tank determination unit 1506 of the first fuel supply monitoring unit 1009_1 will be described.

[0262] In step S1901 of FIG. 19(a), the poly-tank determination unit 1506 determines whether a container is included (whether a predetermined non-permission condition is satisfied) in the currently input recognition result. If it is determined in step S1901 that a container is included (if YES in step S1901), it is determined that the predetermined non-permission condition is satisfied, and the process proceeds to step S1902.

[0263] In step S1902, the poly-tank determination unit 1506 determines that fuel supply is not permitted, acquires the third information, and then proceeds to step S1903.

[0264] On the other hand, if it is determined in step S1901 that a container is not included (if NO in step S1901), the process directly proceeds to step S1903.

[0265] In step S1903, the poly-tank determination unit 1506 determines whether to end the process. If it corresponds to any of the first section to the third section, in step S1903, the poly-tank determination unit 1506 determines to continue the process (determines NO in step S1903) and returns to step S1901.

[0266] On the other hand, if it does not correspond to any of the first section to the third section, in step S1903, the first fuel supply monitoring unit 1009_1 ends the process using the poly-tank determination unit 1506.

[0267] Subsequently, the processing flow using the poly-tank determination unit 1506 of the second fuel supply monitoring unit 1009_2 will be described.

[0268] In step S1911 of FIG. 19(b), the fire determination unit 1507 determines whether a temperature equal to or higher than a predetermined threshold has been measured in a region other than the first region 210 in the currently input thermal sensor data (whether a predetermined non-permissible condition is satisfied). Note that the region other than the first region 210 is set to avoid misjudging fire detection because the hood of the vehicle included in the first region 210 may be at a high temperature.

[0269] If it is determined in step S1911 that a temperature equal to or higher than the predetermined threshold has been measured (if YES in step S1911), it is determined that the predetermined non-permissible condition is satisfied, and the process proceeds to step S1912.

[0270] In step S1912, the fire determination unit 1507 determines that refueling is not permitted, acquires the fourth information, and then proceeds to step S1913.

[0271] On the other hand, if it is determined in step S1911 that a temperature equal to or higher than the predetermined threshold has not been measured (if NO in step S1911), the process directly proceeds to step S1913.

[0272] In step S1913, the fire determination unit 1507 determines whether to end the process. If it corresponds to any of the first to third intervals, in step S1913, the fire determination unit 1507 determines to continue the process (determines NO in step S1913) and returns to step S1911.

[0273] On the other hand, if it does not correspond to any of the first to third intervals, in step S1913, the first fuel supply monitoring unit 1009_1 ends the process using the fire determination unit 1507.

[0274] (3-6) Flow of processing using the first determination logic 1513 of the second fuel supply monitoring unit 1009_2 Next, the processing flow using the first determination logic 1513 of the second fuel supply monitoring unit 1009_2 will be described with reference to FIGS. 20A and 20B. FIGS. 20A and 20B are examples of first and second flowcharts showing the processing flow using the first determination logic of the second fuel supply monitoring unit.

[0275] In step S2001, the first determination logic 1513 determines whether the currently input recognition result is a recognition result determined to be fuel supply permitted (whether a predetermined permission condition is satisfied). Specifically, it is determined whether a container is recognized and whether a fueler holding the fuel nozzle in an appropriate posture is recognized.

[0276] If it is determined in step S2001 that the recognition result is a recognition result determined to be fuel supply permitted (YES in step S2001), the process proceeds to step S2002.

[0277] In step S2002, the first determination logic 1513 determines whether the recognition result determined to be fuel supply permitted has continued for a predetermined time T1 (whether a predetermined permission condition has been satisfied for a certain time). If it is determined in step S2002 that it has continued for the predetermined time T1 (YES in step S2002), the process proceeds to step S2003.

[0278] In step S2003, the first determination logic 1513 determines that fuel supply is permitted and proceeds to step S2014 in FIG. 20B.

[0279] On the other hand, if it is determined in step S2001 that the recognition result is not a recognition result determined to be fuel supply permitted (NO in step S2001), the process proceeds to step S2004. Or, if it is determined in step S2002 that it has not continued for the predetermined time T1 (NO in step S2002), the process proceeds to step S2004.

[0280] In step S2004, the first determination logic 1513 determines whether a state where the recognition result is not determined as refueling permission has continued for a predetermined time T2 (however, T2 > T1).

[0281] In step S2004, if it is determined that the continuation has not occurred for the predetermined time T2 (if NO in step S2004), the process returns to step S2001. On the other hand, in step S2004, if it is determined that the continuation has occurred for the predetermined time T2 (if YES in step S2004), the process proceeds to step S2005.

[0282] In step S2005, the first determination logic 1513 determines whether the currently input recognition result is a recognition result determined as refueling not permitted (whether a predetermined non - permission condition is satisfied). Specifically, it determines whether a container is recognized and whether a refueler who is not holding the refueling nozzle in an appropriate posture is recognized (whether the refueling nozzle cannot be recognized).

[0283] In step S2005, if it is determined that the recognition result is a recognition result determined as refueling not permitted (if YES in step S2005), it is determined that the predetermined non - permission condition has been satisfied for a certain time, and the process proceeds to step S2006.

[0284] In step S2006, the first determination logic 1513 determines that refueling is not permitted, acquires the first information, and then proceeds to step S2011 in FIG. 20B.

[0285] On the other hand, in step S2005, if it is determined that the recognition result is another recognition result determined as refueling not permitted (if NO in step S2005), the process proceeds to step S2017 in FIG. 20B.

[0286] In step S2011, the first determination logic 1513 determines whether the currently input recognition result is a recognition result determined to be fueling permission (whether it satisfies a predetermined permission condition). Specifically, it determines whether a container is recognized and whether a fueler who holds the fuel nozzle in an appropriate posture is recognized.

[0287] If it is determined in step S2011 that the recognition result is not a recognition result determined to be fueling permission (if NO in step S2011), the process proceeds to step S2017.

[0288] On the other hand, if it is determined in step S2011 that the recognition result is a recognition result determined to be fueling permission (if YES in step S2011), the second fueling monitoring unit 1009_2 ends the process using the first determination logic 1513.

[0289] In step S2012, the first determination logic 1513 determines whether the recognition result determined to be fueling permission has continued for a predetermined time T1 (whether the predetermined permission condition has been satisfied for a certain time). If it is determined in step S2012 that it has continued for the predetermined time T1 (if YES in step S2012), the process proceeds to step S2013.

[0290] In step S2013, the first determination logic 1513 determines fueling permission and proceeds to step S2014.

[0291] In step S2014, the determination logic switching unit 1512 determines whether it is within the first section. If it is determined in step S2014 that it is within the first section, it is determined that the process by the first determination logic 1513 is not ended (determined as NO in step S2014), and the process returns to step S2001 in FIG. 20A.

[0292] On the other hand, in step S2014, when it is determined that it is not during the first period, it is determined to end the process by the first determination logic 1513 (determined as YES in step S2014). Then, the second fuel supply monitoring unit 1009_2 ends the process using the first determination logic 1513.

[0293] On the other hand, in step S2012, when it is determined that the predetermined time T1 has not continued (when NO in step S2012), the process proceeds to step S2015.

[0294] In step S2015, the first determination logic 1513 determines whether a state where the recognition result is not a determination of fuel supply permission has continued for a predetermined time T3 (where T3 > T1).

[0295] In step S2015, when it is determined that it has not continued for the predetermined time T3 (when NO in step S2015), the process returns to step S2011. On the other hand, in step S2015, when it is determined that it has continued for the predetermined time T3 (when YES in step S2015), it is determined that a predetermined non - permission condition has been satisfied for a certain period. Then, the second fuel supply monitoring unit 1009_2 ends the process using the first determination logic 1513.

[0296] On the other hand, when the process proceeds from step S2005 in FIG. 20A to step S2017 in FIG. 20B, the first determination logic 1513 determines that a predetermined non - permission condition has been satisfied for a certain period. Specifically, the first determination logic 1513 determines that the container has not been recognized. Alternatively, the first determination logic 1513 determines that a fueler who is not holding the fuel nozzle in an appropriate posture has been recognized (fueler recognition impossible).

[0297] In step S2018, the first determination logic 1513 determines non - permission for fuel supply, acquires second information, and the second fuel supply monitoring unit 1009_2 ends the process using the first determination logic 1513.

[0298] (3-7) Flow of Processing Using the Second Determination Logic 1514 of the Second Fuel Supply Monitoring Unit 1009_2 Next, the flow of processing using the second determination logic 1514 of the second fuel supply monitoring unit 1009_2 will be described with reference to FIGS. 21A to 21D. FIGS. 21A to 21D are examples of the first to fourth flowcharts showing the flow of processing using the second determination logic of the second fuel supply monitoring unit.

[0299] In step S2101, the second determination logic 1514 determines whether the currently input recognition result is a recognition result determined to be fuel supply permission (whether it satisfies a predetermined permission condition). Specifically, it determines whether a container is recognized and whether a fueler holding the fuel nozzle in an appropriate posture is recognized.

[0300] If it is determined in step 2101 that the recognition result is a recognition result determined to be fuel supply permission (YES in step S2101), it is determined that the predetermined permission condition is satisfied, and the process proceeds to step S2102.

[0301] In step S2102, the second determination logic 1514 determines fuel supply permission and proceeds to step S2134 in FIG. 21D.

[0302] On the other hand, if it is determined in step S2101 that the recognition result is not a recognition result determined to be fuel supply permission (NO in step S2101), the process proceeds to step S2103.

[0303] In step S2103, the second determination logic 1514 determines whether the state where the recognition result is not a recognition result determined to be fuel supply permission has continued for a predetermined time T4.

[0304] If it is determined in step S2103 that it has not continued for the predetermined time T4 (NO in step S2103), the process returns to step S2101. On the other hand, if it is determined in step S2103 that it has continued for the predetermined time T4 (YES in step S2103), the process proceeds to step S2104.

[0305] In step S2104, the second determination logic 1514 determines whether the currently input recognition result is a recognition result determined as refueling not permitted (whether a predetermined non-permission condition is satisfied). Specifically, it is determined whether a refueler who has a recognized container and does not hold the refueling nozzle in an appropriate posture is recognized (whether the refueling nozzle cannot be recognized).

[0306] In step S2104, if it is determined that the recognition result is another recognition result determined as refueling not permitted (if NO in step S2104), the process proceeds to step S2121 in FIG. 21C.

[0307] On the other hand, in step S2104, if it is determined that the recognition result is a recognition result determined as refueling not permitted (if YES in step S2104), it is determined that a predetermined non-permission condition has been satisfied for a certain period of time, and the process proceeds to step S2105.

[0308] In step S2105, the second determination logic 1514 acquires warning information.

[0309] In step S2106, the second determination logic 1514 determines whether the currently input recognition result is a recognition result determined as refueling permitted (whether a predetermined permission condition is satisfied). Specifically, it is determined whether a recognized container and a refueler who holds the refueling nozzle in an appropriate posture are recognized.

[0310] In step S2106, if it is determined that the recognition result is a recognition result determined as refueling permitted (if YES in step S2106), the process proceeds to step S2107.

[0311] In step S2107, the second determination logic 1514 determines whether the recognition result determined as fueling permission has continued for a predetermined time T1. In step S2107, if it is determined that the continuation has been for the predetermined time T1 (if YES in step S2107), it is determined that the predetermined permission conditions have been satisfied for a certain period of time, and the process proceeds to step S2108.

[0312] In step S2108, the second determination logic 1514 determines that fueling is permitted and proceeds to step S2134 in FIG. 21D.

[0313] On the other hand, in step S2106, if it is determined that the recognition result is not the one determined as fueling permission (if NO in step S2106), the process proceeds to step S2109. Or, in step S2107, if it is determined that the continuation has not been for the predetermined time T1 (if NO in step S2107), the process proceeds to step S2109.

[0314] In step S2109, the second determination logic 1514 determines whether the state where the recognition result is not the one determined as fueling permission has continued for a predetermined time T2 (where T2 > T1).

[0315] In step S2109, if it is determined that the continuation has not been for the predetermined time T2 (if NO in step S2109), the process returns to step S2106. On the other hand, in step S2109, if it is determined that the continuation has been for the predetermined time T2 (if YES in step S2109), it is determined that the predetermined non - permission conditions have been satisfied for a certain period of time, and the process proceeds to step S2111 in FIG. 21B.

[0316] In step S2111, the second determination logic 1514 determines that fueling is not permitted, acquires the first information, and then proceeds to step S2112.

[0317] In step S2112, the second determination logic 1514 determines whether the currently input recognition result has become a recognition result determined as fueling permission (whether the predetermined permission conditions are satisfied) within a predetermined time T2. Specifically, it determines whether, within the predetermined time T2, the container has been recognized and a fueler who holds the fueling nozzle in an appropriate posture has been recognized.

[0318] In step S2112, if it is determined that the currently input recognition result has become a recognition result determined as fueling permission within the predetermined time T2 (if YES in step S2112), it is determined that the predetermined permission conditions are satisfied, and the process proceeds to step S2113.

[0319] In step S2113, the second determination logic 1514 determines fueling permission and proceeds to step S2134 in FIG. 21D.

[0320] On the other hand, in step S2112, if it is determined that the recognition result has not become a recognition result determined as fueling permission within the predetermined time T2 (if NO in step S2112), the second fueling monitoring unit 1009_2 ends the process using the second determination logic 1514.

[0321] On the other hand, when proceeding from step S2104 to step S2121 in FIG. 21C, the second determination logic 1514 determines that the predetermined non - permission conditions have been satisfied for a certain period of time. Specifically, the second determination logic 1514 determines that the container has not been recognized. Alternatively, the second determination logic 1514 determines that a fueler who does not hold the fueling nozzle in an appropriate posture has been recognized (fueler recognition impossible).

[0322] In step S2122, the second determination logic 1504 acquires warning information.

[0323] In step S2123, the second determination logic 1514 determines whether the currently input recognition result is a recognition result determined as fueling permission (whether it satisfies a predetermined permission condition). Specifically, it determines whether a container is recognized and whether a fueler who holds the fuel nozzle in an appropriate posture is recognized.

[0324] In step S2123, if it is determined that the recognition result is a recognition result determined as fueling permission (if YES in step S2123), the process proceeds to step S2124.

[0325] In step S2124, the second determination logic 1514 determines whether the recognition result determined as fueling permission has continued for a predetermined time T1. In step S2124, if it is determined that it has continued for the predetermined time T1 (if YES in step S2124), it is determined that the predetermined permission condition has been satisfied for a certain period of time, and the process proceeds to step S2125.

[0326] In step S2125, the second determination logic 1514 determines fueling permission and proceeds to step S2134 in FIG. 21D.

[0327] On the other hand, in step S2123, if it is determined that the recognition result is not a recognition result determined as fueling permission (if NO in step S2123), the process proceeds to step S2126. Or, in step S2124, if it is determined that it has not continued for the predetermined time T1 (if NO in step S2124), the process proceeds to step S2126.

[0328] In step S2126, the second determination logic 1514 determines whether the state where the recognition result is not a recognition result determined as fueling permission has continued for a predetermined time T2 (where T2 > T1).

[0329] In step S2126, if it is determined that the predetermined time T2 has not elapsed continuously (if NO in step S2126), the process returns to step S2123. On the other hand, in step S2126, if it is determined that the predetermined time T2 has elapsed continuously (if YES in step S2126), it is determined that the predetermined non - permission condition has been satisfied for a certain period of time, and the process proceeds to step S2131 in FIG. 21D.

[0330] In step S2131, the second determination logic 1514 determines non - fueling permission, and after acquiring the second information, proceeds to step S2132.

[0331] In step S2132, the second determination logic 1514 determines whether the currently input recognition result has become a recognition result for which fueling is permitted (whether the predetermined permission condition has been satisfied) within the predetermined time T2. Specifically, it is determined whether, within the predetermined time T2, a container has been recognized and a fueler who holds the fueling nozzle in an appropriate posture has been recognized.

[0332] In step S2132, if it is determined that the currently input recognition result has become a recognition result for which fueling is permitted within the predetermined time T2 (if YES in step S2132), it is determined that the predetermined permission condition has been satisfied, and the process proceeds to step S2133.

[0333] In step S2133, the second determination logic 1514 determines fueling permission and proceeds to step S2134.

[0334] In step S2134, the determination logic switching unit 1512 determines whether it is within the second section. In step S2134, if it is determined that it is within the second section, it is determined that the process by the second determination logic 1514 is not terminated (if determined as NO in step S2134), and the process returns to step S2101 in FIG. 21A.

[0335] On the other hand, in step S2134, if it is determined that it is not during the second interval, it is determined to end the process by the second determination logic 1514 (determined as YES in step S2134). Then, the second fuel supply monitoring unit 1009_2 ends the process using the second determination logic 1514.

[0336] Also, in step S2132, if it is not determined that the recognition result is a fuel supply permission determination within the predetermined time T2 (NO in step S2132), the second fuel supply monitoring unit 1009_2 ends the process using the second determination logic 1514.

[0337] (3-8) Flow of the process using the third determination logic 1515 of the second fuel supply monitoring unit 1009_2 Next, the flow of the process using the third determination logic 1515 of the second fuel supply monitoring unit 1009_2 will be described with reference to FIGS. 22A and 22B. FIGS. 22A and 22B are examples of the first and second flowcharts showing the flow of the process using the third determination logic of the second fuel supply monitoring unit.

[0338] In step S2201, the third determination logic 1515 determines whether the currently input recognition result is a recognition result for which fuel supply is permitted (whether a predetermined permission condition is satisfied). Specifically, it is determined whether a container is recognized and whether a fueler holding the fuel nozzle in an appropriate posture is recognized.

[0339] In step 2201, if it is determined that the recognition result is a recognition result for which fuel supply is permitted (YES in step S2201), it is determined that the predetermined permission condition is satisfied, and the process proceeds to step S2202.

[0340] In step S2202, the third determination logic 1515 determines that fuel supply is permitted and proceeds to step S2213 in FIG. 22B.

[0341] On the other hand, if it is determined in step S2201 that the recognition result is not the one determined as fueling permission (if NO in step S2201), the process proceeds to step S2203.

[0342] In step S2203, the third determination logic 1515 determines whether the state where the recognition result is not the one determined as fueling permission has continued for a predetermined time T5.

[0343] In step S2203, if it is determined that it has not continued for the predetermined time T5 (if NO in step S2203), the process returns to step S2201. On the other hand, in step S2203, if it is determined that it has continued for the predetermined time T5 (if YES in step S2203), the process proceeds to step S2204.

[0344] In step S2204, the third determination logic 1515 determines whether the currently input recognition result is the one determined as non - fueling permission (whether a predetermined non - permission condition is satisfied). Specifically, it determines whether a container is recognized and whether a fueler who is not holding the fuel nozzle in an appropriate posture is recognized (whether the fuel nozzle is unrecognizable).

[0345] In step S2204, if it is determined that it is another recognition result determined as non - fueling permission (if NO in step S2204), the process proceeds to step S2206.

[0346] In step S2206, the third determination logic 1515 determines that a predetermined non - permission condition has been satisfied for a certain period of time. Specifically, the third determination logic 1515 determines that the container has not been recognized. Alternatively, the third determination logic 1515 determines that a fueler who is not holding the fuel nozzle in an appropriate posture has been recognized (the fueler is unrecognizable).

[0347] In step S2207, the third determination logic 1515 determines non - fueling permission, acquires the second information, and then proceeds to step S2211 in FIG. 22B.

[0348] On the other hand, in step S2204, when it is determined that the recognition result is a refueling non - permission determination (when YES in step S2204), it is determined that a predetermined non - permission condition has been satisfied for a certain period of time, and the process proceeds to step S2205.

[0349] In step S2205, the third determination logic 1515 determines refueling non - permission, acquires the first information, and then proceeds to step S2211 in FIG. 22B.

[0350] In step S2211, the third determination logic 1515 determines whether the currently input recognition result has become a recognition result for which refueling is permitted (whether a predetermined permission condition has been satisfied) within a predetermined time T2. Specifically, it is determined whether, within the predetermined time T2, the container has been recognized and a refueler who holds the fuel nozzle in an appropriate posture has been recognized.

[0351] In step S2211, when it is determined that the currently input recognition result has become a recognition result for which refueling is permitted within the predetermined time T2 (when YES in step S2211), it is determined that a predetermined permission condition has been satisfied, and the process proceeds to step S2212.

[0352] In step S2212, the third determination logic 1515 determines refueling permission and proceeds to step S2213.

[0353] In step S2213, the determination logic switching unit 1512 determines whether it is within the third section. In step S2213, when it is determined that it is within the third section, it is determined that the process by the third determination logic 1515 is not terminated (when determined as NO in step S2213), and the process returns to step S2201 in FIG. 22A.

[0354] On the other hand, in step S2213, if it is determined that it is not within the third period, it is determined to end the process by the third determination logic 1515 (determined as YES in step S2213). Then, the second fuel supply monitoring unit 1009_2 ends the process using the third determination logic 1515.

[0355] On the other hand, in step S2211, if it is determined that the currently input recognition result has not become a recognition result determined as fuel supply permission within the predetermined time T2 (if NO in step S2211), the process proceeds to step S2214.

[0356] In step S2214, the third determination logic 1515 determines whether the state of not being a recognition result determined as fuel supply permission has continued for the predetermined time T6. In step S2214, if it is determined that the state of not being a recognition result determined as fuel supply permission has not continued for the predetermined time T6 (if NO in step S2214), it waits until it continues for the predetermined time T6.

[0357] On the other hand, in step S2214, if it is determined that the state of not being a recognition result determined as fuel supply permission has continued for the predetermined time T6 (if YES in step S2214), the process proceeds to step S2215.

[0358] In step S2215, the third determination logic 1515 acquires confirmation request information, and the second fuel supply monitoring unit 1009_2 ends the process using the third determination logic 1515.

[0359] (3-9) Flow of processing using other determination logic of the second fuel supply monitoring unit 1009_2 Next, the flow of processing using other determination logic of the second fuel supply monitoring unit 1009_2 (processing using the fire determination unit 1517) will be described with reference to FIG. 23. FIG. 23 is an example of a flowchart showing the flow of processing using the fire determination unit of the second fuel supply monitoring unit.

[0360] In step S2301, the fire determination unit 1517 determines whether a temperature equal to or higher than a predetermined threshold has been measured for the currently input thermal sensor data (whether a predetermined non - permission condition is satisfied).

[0361] In step S2301, if it is determined that a temperature equal to or higher than the predetermined threshold has been measured (if YES in step S2301), it is determined that the predetermined non - permission condition is satisfied, and the process proceeds to step S2302.

[0362] In step S2302, the fire determination unit 1517 determines non - fueling permission, acquires the fourth information, and then proceeds to step S2303.

[0363] On the other hand, in step S2301, if it is determined that a temperature equal to or higher than the predetermined threshold has not been measured (if NO in step S2301), the process directly proceeds to step S2303.

[0364] In step S2303, the fire determination unit 1517 determines whether to end the process. If it corresponds to any of the first to third intervals, in step S2303, the fire determination unit 1517 determines to continue the process (determines NO in step S2303) and returns to step S2301.

[0365] On the other hand, if it does not correspond to any of the first to third intervals, in step S2303, the second fuel supply monitoring unit 1009_2 ends the process using the fire determination unit 1517.

[0366] (4) Details of the output control unit 1010 Next, as details of the output control unit 1010, the functional configuration and specific examples of the processes of the output control unit 1010 will be described.

[0367] (4 - 1) Details of the functional configuration of the output control unit 1010 FIG. 24 is a diagram showing an example of the functional configuration of the output control unit of the information processing apparatus. As shown in FIG. 24, the output control unit 1010 includes an output information acquisition unit 2401 and a transmission unit 2402.

[0368] The output information acquisition unit 2401 obtains from the first fueling monitoring unit 1009_1, · information indicating fueling permission, · information indicating non - fueling permission and the first to fourth information, · warning information, · confirmation request information, and acquires them.

[0369] Also, the output information acquisition unit 2401 obtains from the second fueling monitoring unit 1009_2, · information indicating fueling permission, · information indicating non - fueling permission and the first, second, and fourth information, · warning information, · confirmation request information, and acquires them.

[0370] Also, the output information acquisition unit 2401 notifies the information acquired from the first fueling monitoring unit 1009_1 to the transmission unit 2402. At this time, the output information acquisition unit 2401 refers to the output information storage unit 2403, · cause information (information indicating the cause of non - fueling permission) corresponding to the first to fourth information, · messages corresponding to the first to fourth information, · display information corresponding to the first to fourth information, · messages corresponding to the warning information, · display information corresponding to the warning information, · cause information corresponding to the confirmation request information, · messages corresponding to the confirmation request information, · display information corresponding to the confirmation request information, and acquires them. Then, the output information acquisition unit 2401 converts the first to fourth information, warning information, and confirmation request information into the corresponding cause information, messages, and display information, and then notifies the transmission unit 2402.

[0371] Similarly, the output information acquisition unit 2401 notifies the transmission unit 2402 of the information acquired from the second fuel supply monitoring unit 1009_2. At this time, the output information acquisition unit 2401 refers to the output information storage unit 2403 and · Cause information corresponding to the first information, the second information, and the fourth information (information indicating the cause of fuel supply non - permission), and · Messages corresponding to the first information, the second information, and the fourth information, and · Display information corresponding to the first information, the second information, and the fourth information, and · Messages corresponding to the warning information, and · Display information corresponding to the warning information, and · Cause information corresponding to the confirmation request information, and · Messages corresponding to the confirmation request information, and · Display information corresponding to the confirmation request information, and acquires them. Then, the output information acquisition unit 2401 converts the first information, the second information, the fourth information, the warning information, and the confirmation request information into the corresponding cause information, messages, and display information, and then notifies the transmission unit 2402.

[0372] The transmission unit 2402 transmits the information notified from the output information acquisition unit 2401 to · The display device 912, which is a display device for the staff of the service station 110', · The voice output device 122_1 of the first fuel dispenser 120_1 or the voice output device 122_2 of the second fuel dispenser 120_2, · The fueling data display 123_1 of the first fuel dispenser 120_1 or the fueling data display 123_2 of the second fuel dispenser 120_2. Specifically, the transmission unit 2402 controls the information acquired from the first fuel supply monitoring unit 1009_1,

[0373] · Information indicating fuel supply permission, information indicating fuel supply non - permission, cause information, and confirmation request information to be displayed on the display device 912, · Controls the message to be output as voice by the voice output device 122_1 of the first fuel dispenser 120_1, · Controls the message to be output as voice by the voice output device 122_1 of the first fuel dispenser 120_1, ·Control so that the indication information is displayed on the fueling data display 123_1 of the first fuel dispenser 120_1.

[0374] Similarly, the transmission unit 2402 controls so that the information acquired from the second fuel monitoring unit 1009_2, ·information indicating fueling permission, information indicating non - permission for fueling, cause information, and confirmation request information are displayed on the display device 912, ·a message is output as voice by the voice output device 122_2 of the second fuel dispenser 120_2, ·control so that the display information is displayed on the fueling data display 123_2 of the second fuel dispenser 120_2.

[0375] (4 - 2) Specific example of the process of the output control unit 1010 Next, as a specific example of the process of the output control unit 1010, specific examples of the cause information, message, and display information converted by the output control unit 1010 will be described. FIG. 25 is a diagram showing an example of output information.

[0376] In FIG. 25, the output information 2500 includes cause information, a message, and display information corresponding to the first to fourth information, warning information, and confirmation request information acquired from the first fuel monitoring unit 1009_1. Also, the output information 2510 includes cause information, a message, and display information corresponding to the first information, second information, fourth information, warning information, and confirmation request information acquired from the second fuel monitoring unit 1009_2.

[0377] As shown in FIG. 25, the output information 2500 and 2510 include, as information items, "information type", "cause information", "message output to the customer", and "display information on the fueling data display of the fuel dispenser".

[0378] For "information type", in the case of the output information 2500, any one of the first to fourth information is stored, and in the case of the output information 2510, any one of the first information, second information, and fourth information is stored.

[0379] "Cause information" stores either "nozzle indeterminate", "judgment impossible", "refueling from outside the vehicle", or "fire detection" in the case of output information 2500, and stores either "nozzle indeterminate", "judgment impossible", or "fire detection" in the case of output information 2510.

[0380] "Messages output to the customer" stores messages corresponding to the first to fourth information respectively, which are output as audio from the audio output device 122_1 in the case of output information 2500. Also, in the case of output information 2510, it stores messages corresponding to the first, second, and fourth information respectively, which are output as audio from the audio output device 122_2.

[0381] "Display information on the fueling data display of the fuel dispenser" stores display information corresponding to the first to fourth information respectively, which is displayed on the fueling data display 123_1 of the first fuel dispenser 120_1 in the case of output information 2500. Also, in the case of output information 2510, it stores display information corresponding to the first, second, and fourth information respectively, which is displayed on the fueling data display 123_2 of the second fuel dispenser 120_2.

[0382] <Flow of fueling control processing by the information processing device> Next, the flow of fueling control processing by the fueling system 110S' according to the first embodiment will be described. FIG. 26 is an example of a flowchart showing the flow of fueling control processing by the fueling system according to the first embodiment.

[0383] In step S2601, the information processing device 160 acquires the image data captured by the cameras 130_1 and 130_2.

[0384] In step S2602, the information processing device 160 acquires the signals from the first fueling sensor 921_1 and the second fueling sensor 921_2.

[0385] In step S2603, the information processing device 160 acquires the thermal sensor data measured by the thermal sensors 170_1 and 170_2.

[0386] In step S2604, the information processing apparatus 160 determines which of the first section to the third section it corresponds to.

[0387] In step S2605, the information processing apparatus 160 determines whether to determine the fueling permission or not. In step S2605, if it is determined that it does not correspond to any of the first section to the third section, it is determined not to determine the fueling permission (determined as NO in step S2605), and the process proceeds to step S2610.

[0388] On the other hand, in step 2605, if it is determined that it corresponds to any of the first section to the third section, it is determined to determine the fueling permission (determined as YES in step S2605), and the process proceeds to step S2606.

[0389] In step S2606, the information processing apparatus 160 performs image recognition processing on the image data using the first learned model or the second learned model.

[0390] In step S2607, the information processing apparatus 160 determines the fueling permission or not from the recognition result using the determination logic corresponding to the determined section.

[0391] In step S2608, the information processing apparatus 160 controls the fueling of the first fuel dispenser 120_1 and the second fuel dispenser 120_2 based on the determination result of the fueling permission or not. Note that when the determination result is no fueling permission, the information processing apparatus 160 controls the prohibition or stop of fueling by the first fuel dispenser 120_1 or the second fuel dispenser 120_2 according to the cause information of the no fueling permission.

[0392] In step S2609, the information processing apparatus 160 outputs the determination result of the fueling permission or not. Further, when the determination result is no fueling permission, the information processing apparatus 160 outputs the cause information of the no fueling permission and the like.

[0393] In step S2610, the information processing apparatus 160 determines whether to end the fuel supply control process. If it is determined in step S2610 to continue the fuel supply control process (if NO in step S2610), the process returns to step S2601. On the other hand, if it is determined in step S2610 to end the fuel supply control process (if YES in step S2610), the fuel supply control process ends.

[0394] <Summary> As is clear from the above description, the fuel supply system 110S' according to the first embodiment divides the period from when the fuel supply nozzle is removed from the nozzle holder until it is returned to the nozzle holder into three sections, and determines whether fuel supply is permitted using determination logic corresponding to each section. Thus, according to the first embodiment, it is possible to provide a fuel supply system capable of executing processing according to the fuel supply scene.

[0395] Also, the fuel supply system 110S' according to the first embodiment is configured to acquire cause information for non-permission of fuel supply and output a message corresponding to the cause information as voice when determining non-permission of fuel supply based on the recognition result regarding the recognition target. Thus, according to the first embodiment, it is possible to provide a fuel supply system capable of executing processing according to the fuel supply scene.

[0396] Also, the fuel supply system 110S' according to the first embodiment is configured to use a learned model corresponding to the fuel supply type when performing image recognition processing regarding the recognition target. Thus, according to the first embodiment, it is possible to provide a fuel supply system capable of executing processing according to the fuel supply scene.

[0397] [Second Embodiment] In the above first embodiment, the information processing apparatus 160 has been described as performing fueling control for the first fuel dispenser 120_1, the second fuel dispenser 120_2, and the like. However, the fueling control for the first fuel dispenser 120_1, the second fuel dispenser 120_2, and the like may be configured to be performed by a device separate from the information processing apparatus 160 (for example, a fueling controller such as a self-service console provided in a service station). Hereinafter, the second embodiment will be described centering on the differences from the above first embodiment.

[0398] <Functional Configuration of Information Processing Apparatus> FIG. 27 is a diagram showing another example of the functional configuration of the information processing apparatus. The difference from the functional configuration of the information processing apparatus 160 shown in FIG. 10 is that in the case of the information processing apparatus 160' shown in FIG. 27, the functions of the first fueling monitoring unit 1009_1' and the second fueling monitoring unit 1009_2' are different from the functions of the first fueling monitoring unit 1009_1 and the second fueling monitoring unit 1009_2. Further, the difference from the functional configuration of the information processing apparatus 160 shown in FIG. 10 is that in the case of the information processing apparatus 160' shown in FIG. 27, the first fueling monitoring unit 1009_1' and the second fueling monitoring unit 1009_2' are communicably connected to the fueling controller 2710.

[0399] The fueling controller 2710 is a self-service console (SSC: Self Service Console), and controls the fueling of the first fuel dispenser 120_1 based on the determination result of whether fueling is permitted transmitted from the first fueling monitoring unit 1009_1'. Further, the fueling controller 2710 controls the fueling of the second fuel dispenser 120_2 based on the determination result of whether fueling is permitted transmitted from the second fueling monitoring unit 1009_2'.

[0400] <Details of First Fueling Monitoring Unit and Second Fueling Monitoring Unit> Next, the details of the first fuel supply monitoring unit 1009_1' and the second fuel supply monitoring unit 1009_2' will be described. FIG. 28A is a diagram showing another example of the functional configuration of the first fuel supply monitoring unit of the information processing device. In the above first embodiment, the difference from the functional configuration of the first fuel supply monitoring unit 1009_1 of the information processing device 160 described with reference to FIG. 15A is that in the case of the first fuel supply monitoring unit 1009_1' of the information processing device 160' in FIG. 28A, it has a transmission unit 2801. Further, in the case of the first fuel supply monitoring unit 1009_1' of the information processing device 160' in FIG. 28A, the transmission unit 2801 is communicably connected to the fuel supply controller 2710.

[0401] The transmission unit 2801 transmits the determination result of fuel supply permission or fuel supply non - permission to the fuel supply controller 2710. Thereby, the fuel supply controller 2710 controls the fuel supply valve 124_1 or the fuel supply pump 125_1 to control the fuel supply for the first fuel supply type.

[0402] For example, when the fuel supply permission is transmitted from the transmission unit 2801, the fuel supply controller 2710 drives the fuel supply pump 125_1 to open the fuel supply valve 124_1. Also, the fuel supply controller 2710 · When the fuel supply permission from the poly - tank determination unit 1506 is transmitted from the transmission unit 2801, the fuel supply controller 2710 drives the fuel supply pump 125_1 to open the fuel supply valve 124_1. Further, · When the fuel supply non - permission from any of the first determination logic 1503 to the third determination logic 1505 is transmitted from the transmission unit 2801, or · When the fuel supply non - permission from the fire determination unit 1507 is transmitted from the transmission unit 2801, it stops the fuel supply by stopping the drive of the fuel supply pump 125_1, or prohibits the fuel supply by stopping the drive of the fuel supply pump 125_1 and closing the fuel supply valve 124_1. Note that the definitions of fuel supply stop and fuel supply prohibition here are only examples, and the fuel supply stop and fuel supply prohibition may be defined by other definitions. Alternatively, without distinguishing between fuel supply stop and fuel supply prohibition, the entire state where fuel supply cannot be performed may be referred to as fuel supply stop or prohibition.

[0403] Further, the transmission unit 2801 notifies the fueling permission or non - permission determination result to the cause information acquisition unit 1509. When the transmission unit 2801 notifies non - permission for fueling, it also notifies the cause information acquisition unit 1509 of the information indicating the cause for which non - permission for fueling was determined.

[0404] Note that when any of the first determination logic 1503 to the third determination logic 1505 is unable to perform the determination of fueling permission or non - permission, the transmission unit 2801 may be configured to output "determination impossible" to the cause information acquisition unit 1509. When "determination impossible" is output from the transmission unit 2801, the cause information acquisition unit 1509 may display "determination impossible" on, for example, the display device 912 which is a display device for the staff of the service station 110'. According to such a configuration, even when the determination of fueling permission or non - permission cannot be performed, the staff of the service station 110' can operate the fuel controller 2710 to control fueling.

[0405] Also, the transmission unit 2801 may be configured to monitor the communication status with the fuel controller 2710 and output "communication abnormality" to the cause information acquisition unit 1509 when a communication failure occurs. When "communication abnormality" is output from the transmission unit 2801, the cause information acquisition unit 1509 may display "communication abnormality" on, for example, the display device 912 which is a display device for the staff of the service station 110'. According to such a configuration, even when the determination result of fueling permission or non - permission cannot be transmitted to the fuel controller 2710, the staff of the service station 110' can operate the fuel controller 2710 to control fueling.

[0406] FIG. 28B is a diagram showing an example of the functional configuration of the second fueling monitoring unit of the information processing device. Since the functional configuration of the second fueling monitoring unit 1009_2' shown in FIG. 28B is the same as the functional configuration of the first fueling monitoring unit 1009_1' shown in FIG. 28A, detailed description is omitted here.

[0407] <Summary> As is clear from the above description, in the second embodiment, the functions of the information processing apparatus 160 in the first embodiment are realized by the information processing apparatus 160' and the fuel supply controller 2710. Thus, even when the information processing apparatus 160' and the fuel supply controller 2710 are configured separately, according to the second embodiment, the same effects as those in the first embodiment can be achieved.

[0408] [Third Embodiment] In the first embodiment, the case where the period from when the fuel supply nozzle is removed from the nozzle holder until it is returned to the nozzle holder is divided into three sections was described. However, the number of sections to be divided is not limited to three, and may be four or more.

[0409] Also, in the first embodiment, it was described that one determination logic is associated with each section. However, the relationship between the number of sections and the number of determination logics is not limited to this. For example, one determination logic may be associated with two consecutive sections.

[0410] Also, in the first embodiment, the case where there are two types of fuel supply was described, but the number of types of fuel supply may be three or more. Note that when there are three or more types of fuel supply, three or more learned models are also generated.

[0411] Also, in the first embodiment, for the first fuel supply type, the recognition targets were a vehicle, a container, a fuel supplier, a fuel supply nozzle, etc. However, the recognition targets for the first fuel supply type are not limited to these.

[0412] Similarly, in the first embodiment, for the second fuel supply type, the recognition targets were a container, a fuel supplier, a fuel supply nozzle, etc. However, the recognition targets for the second fuel supply type are not limited to these.

[0413] In the first embodiment described above, the first to third determination logics for the first fuel supply type were described with reference to FIGS. 16A to 18B. However, the processing contents of the first to third determination logics are not limited to these.

[0414] Similarly, in the first embodiment described above, the first to third determination logics for the second fuel supply type were described with reference to FIGS. 20A to 22B. However, the processing contents of the first to third determination logics are not limited to these.

[0415] In the first embodiment described above, the processing after displaying the cause information, for example, on the display device 912 for the staff of the service station 110' was not mentioned. However, after displaying the cause information, it may be switched to manual operation by the staff.

[0416] In the first embodiment described above, the number of executions of the learning process was not mentioned. However, the learning process may be executed multiple times. For example, in the image recognition process using the first learned model and the second learned model generated by the first learning process, if sufficient recognition accuracy cannot be obtained, or if the recognition accuracy decreases, additional learning processes may be configured to be performed.

[0417] For example, the learning device 150 may be configured to perform additional learning processes by collecting image data and signals indicating the states of the fuel nozzles again from a plurality of service stations and generating learning data.

[0418] Note that the learning device 150 may be configured to be able to collect image data and signals indicating the states of the fuel nozzles at any timing from a plurality of service stations. Also, a device that monitors the recognition accuracy, analyzes the cause when the recognition accuracy decreases, and notifies the learning device 150 may be connected to the network 180.

[0419] In addition, in the above-described first embodiment, the learning device 150 and the information processing device 160 are configured as separate bodies, but the learning device 150 and the information processing device 160 may be configured as an integrated body.

[0420] Note that the configurations and the like described in the above embodiments are not limited to the configurations shown herein, such as combinations with other elements. Regarding these points, it is possible to make changes without departing from the spirit of the present invention, and they can be appropriately determined according to the application form.

Explanation of Reference Numerals

[0421] 110, 111: Service Station 110S': Fueling System 120_1: First Fueling Machine 120_2: Second Fueling Machine 121_1, 121_2: Fueling Nozzle 122_1, 122_2: Voice Output Device 123_1, 123_3: Fueling Data Display 124_1, 124_2: Fueling Valve 125_1, 125_2: Fueling Pump 130_1, 130_2: Camera 170_1, 170_2: Thermal Sensor 150: Learning Device 160: Information Processing Device 160': Information Processing Device 210: First Region 220: Second Region 211: Vehicle 221: Fueler 231: Container 241: Fueler 500A: Learning Data Generation Unit 500B: Learning Unit 600: First Learning Data 700: Second Learning Data 912: Display Device 921_1: First Fueling Sensor 921_2: Second Fueling Sensor 1001: First fuel type image data acquisition unit 1002: Second fuel type image data acquisition unit 1003: First fueling sensor data acquisition unit 1004: Second fueling sensor data acquisition unit 1005: Image recognition unit 1006, 1007: Interval determination unit 1009_1: First fueling monitoring unit 1009_1': First fueling monitoring unit 1009_2: Second fueling monitoring unit 1009_2': Second fueling monitoring unit 1010: Output control unit 1302: First trained model 1312: Second trained model 1503, 1513: First determination logic 1504, 1514: Second determination logic 1505, 1515: Third determination logic 2710: Fueling controller 2801, 2811: Transmitter

Claims

1. A first fuel dispenser for dispensing a first type of fuel and a second fuel dispenser for dispensing a second type of fuel, a plurality of cameras for respectively photographing the area including the first fuel dispenser and the area including the second fuel dispenser, and an information processing device, a fueling system having, wherein the information processing device, when image data in which the first fuel dispenser and a fueling action by a fueler are photographed is input to a first pre-trained model and recognition processing is performed, based on the recognition result, a first determination unit that determines whether fueling of the first fuel dispenser is permitted; when image data in which the second fuel dispenser and a fueling action by a fueler are photographed is input to a second pre-trained model and recognition processing is performed, based on the recognition result, a second determination unit that determines whether fueling of the second fuel dispenser is permitted, and having, when it is determined by the first determination unit that fueling is not permitted, prohibiting or stopping the fueling of the first type of fuel, and when it is determined by the second determination unit that fueling is not permitted, prohibiting or stopping the fueling of the second type of fuel, a fueling system.

2. The first determination unit, when a recognition result is output that a vehicle is recognized but a fueler who is not holding the fuel nozzle in an appropriate posture is recognized because the fuel nozzle is not recognized, determines that fueling is not permitted, when a recognition result that a vehicle is recognized is not output, or when a recognition result is output that a fueler who is not holding the fuel nozzle in an appropriate posture is recognized because the fueler is not recognized, determines that fueling is not permitted, The fueling system according to claim 1.

3. The first determination unit, when a recognition result is output that a predetermined container is recognized, determines that fueling is not permitted, the fueling system according to claim 2.

4. The second determination unit, when a recognition result is output that a predetermined container is recognized but a fueler who is not holding the fuel nozzle in an appropriate posture is recognized because the fuel nozzle is not recognized, determines that fueling is not permitted, when a recognition result that a predetermined container is recognized is not output, or when a recognition result is output that a fueler who is not holding the fuel nozzle in an appropriate posture is recognized because the fueler is not recognized, determines that fueling is not permitted, The fueling system according to claim 2.

5. The first learned model is generated by training a first model using training data including image data in which a first fuel dispenser and a fueling operation by a fueler are photographed, and annotation data assigned to a recognition target. The fueling system according to claim 2.

6. The second learned model is generated by training a second model using training data including image data in which a second fuel dispenser and a fueling operation by a fueler are photographed, and annotation data assigned to a recognition target. The fueling system according to claim 4.

7. The first determination unit determines whether fueling of the first fuel dispenser is permitted or not based on a recognition result that a vehicle is recognized in a first area which is a part of the image data in which the first fuel dispenser and a fueling operation by a fueler are photographed, and determines whether fueling of the first fuel dispenser is permitted or not based on a recognition result that a fueler holding a fuel nozzle in an appropriate posture is recognized in a second area which is a part of the image data in which the first fuel dispenser and a fueling operation by a fueler are photographed and is different from the first area. The fueling system according to claim 5.

8. The first determination unit determines whether fueling is permitted or not from when the fuel nozzle of the first fuel dispenser is removed from the nozzle hanger until it is returned to the nozzle hanger, and the second determination unit determines whether fueling is permitted or not from when the fuel nozzle of the second fuel dispenser is removed from the nozzle hanger until it is returned to the nozzle hanger. The fueling system according to claim 2.

9. further includes a fueling controller that communicates with the information processing device, and the fueling controller forbids or stops fueling of the first fuel type when it is determined by the first determination unit that fueling is not permitted, and forbids or stops fueling of the second fuel type when it is determined by the second determination unit that fueling is not permitted. The fueling system according to any one of claims 1 to 8.

10. When image data in which a first fuel dispenser for fueling a first fuel type and a fueling operation by a fueler are photographed is input to a first learned model and recognition processing is performed, a first determination unit that determines whether fueling is permitted or not based on the recognition result, When image data in which a second fuel dispenser for fueling a second fuel type and a fueling operation by a fueler are photographed is input to a second learned model and recognition processing is performed, a second determination unit that determines whether fueling is permitted or not based on the recognition result, When it is determined by the first determination unit that refueling is not permitted, the fuel supply control unit prohibits or stops the refueling of the first fuel type, and when it is determined by the second determination unit that refueling is not permitted, prohibits or stops the refueling of the second fuel type An information processing apparatus having the same

11. A first determination step of determining whether refueling is permitted based on a recognition result when recognition processing is performed by inputting image data in which a first fuel dispenser that supplies a first fuel type and a refueling act by a refueler are photographed into a first pre-trained model A second determination step of determining whether refueling is permitted based on a recognition result when recognition processing is performed by inputting image data in which a second fuel dispenser that supplies a second fuel type and a refueling act by a refueler are photographed into a second pre-trained model A fuel supply control step of prohibiting or stopping the refueling of the first fuel type when it is determined by the first determination step that refueling is not permitted, and prohibiting or stopping the refueling of the second fuel type when it is determined by the second determination step that refueling is not permitted A fuel supply control method executed by a computer

12. A first determination step of determining whether refueling is permitted based on a recognition result when recognition processing is performed by inputting image data in which a first fuel dispenser that supplies a first fuel type and a refueling act by a refueler are photographed into a first pre-trained model A second determination step of determining whether refueling is permitted based on a recognition result when recognition processing is performed by inputting image data in which a second fuel dispenser that supplies a second fuel type and a refueling act by a refueler are photographed into a second pre-trained model A fuel supply control step of prohibiting or stopping the refueling of the first fuel type when it is determined by the first determination step that refueling is not permitted, and prohibiting or stopping the refueling of the second fuel type when it is determined by the second determination step that refueling is not permitted A fuel supply control program for causing a computer to execute the same

Images