Oil feeding system, information processing apparatus, oil feeding control method, and oil feeding control program

Abstract

To provide an oil feeding system configured to execute processing according to an oil feeding scene.SOLUTION: An oil feeding system includes a fuel dispenser, a camera which images a region including the fuel dispenser, and an information processing apparatus. The information processing apparatus includes: an image recognition unit which recognizes, from image data captured by the camera, a feeding person who holds a fuel nozzle in an appropriate attitude, or a feeding person who does not hold the fuel nozzle in an appropriate attitude; and a determination unit which determines whether to permit fueling on the basis of the result of the recognition. The image recognition unit or the determination unit eases, when the feeding person starts fueling, the conditions for outputting a permission to fuel, or tightens the conditions for outputting a refusal to fuel.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present disclosure relates to a fueling system, an information processing device, a fueling control method, and a fueling control program.

Background Art

[0002] As a fueling system for service stations, the development of a fueling system that monitors the fueling behavior of customers as fuelers and determines whether to allow fueling by utilizing AI or the like is underway.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] On the other hand, fueling scenes at service stations are diverse, and in the development of a fueling system, it is desirable to adopt a configuration in which the method for determining whether to allow fueling, the method for outputting the determination result, etc. are changed according to the fueling scene.

[0005] In one aspect, an object is to provide a fueling system capable of executing processing according to a fueling scene.

Means for Solving the Problems

[0006] According to one aspect, the fueling system includes: a fuel dispenser; a camera that photographs the area including the fuel dispenser; an information processing device, and is a fueling system having: the information processing device includes: an image recognition unit that recognizes a fueler who holds the fuel nozzle in an appropriate posture or a fueler who does not hold the fuel nozzle in an appropriate posture from the image data photographed by the camera; It has a determination unit that determines whether refueling is permitted based on the recognition result. When refueling by the refueler is started, the image recognition unit or the determination unit relaxes the conditions for outputting a determination result of refueling permission, or strengthens the conditions for outputting a determination result of refueling non - permission.

Advantages of the Invention

[0007] It is possible to provide a refueling system capable of executing processing according to the refueling scene.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2A

Figure 2B

Figure 3A

Figure 3B

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14A

Figure 14B

Figure 15A

Figure 15B

Figure 16A

Figure 16B

Figure 17A

Figure 17B

Figure 17C

Figure 17D

Figure 18A

Figure 18B

Figure 19

Figure 20A

Figure 20B

Figure 21A

Figure 21B

Figure 21C

Figure 21D

Figure 22A

Figure 22B

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28A

Figure 28B

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33A

Figure 33B

Figure 34A

Figure 34B

Figure 34C

Figure 34D

Figure 35

Figure 36

Figure 37

Embodiments for Carrying Out the Invention

[0009] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions are omitted.

[0010] [First Embodiment] <Learning Process>[ First, the learning process for constructing the fueling system according to the first embodiment will be described. FIG. 1 is a diagram showing the learning process for constructing the fueling system according to the first embodiment. As shown in FIG. 1, when constructing the fueling system according to the first embodiment, image data collected at a plurality of service stations (for example, service stations 110, 111,...) is used. Also, at one service station, when different types of fueling are possible, image data collected for each type of fueling is used. Here, the different types of fueling refer to fuel oil for vehicles (hereinafter referred to as "fuel oil of the first fueling type") such as gasoline and light oil, and fuel oil for uses other than vehicles (hereinafter referred to as "fuel oil of the second fueling type") such as kerosene.

[0011] The example of FIG. 1 shows a case where service station 110 can fuel both the fuel oil of the first fueling type and the fuel oil of the second fueling type. Therefore, in the example of FIG. 1, image data obtained by photographing the fueling scene 110_1 for the first fueling type and image data obtained by photographing the fueling scene 110_2 for the second fueling type are collected.

[0012] The fueling scene 110_1 for the first fueling type includes a fueling act by a fueler (not shown). A camera 130_1 that captures the area including the first fueling machine 120_1 as a video captures the first fueling machine 120_1 and the fueling act by the fueler, and image data with a predetermined frame period is collected by the image data collection device 140.

[0013] As shown in Fig. 1, a fueling nozzle 121_1 is connected to the first fuel dispenser 120_1, and the fuel oil of the first fuel type stored in the tank 126_1 is configured to be supplied to the fueling nozzle 121_1 through a fueling pipe by a fueling pump 125_1. Also, as shown in Fig. 1, a fueling valve 124_1 is installed in the fueling pipe. Further, in the first fuel dispenser 120_1, as a notification device, a voice output device 122_1 and a fueling data display 123_1 (a device for displaying fueling data and messages from the information processing device 160) are installed.

[0014] Although not shown in Fig. 1, similar image data for the fueling scene 110_2 of the second fuel type is also collected by the image data collection device 140.

[0015] When collecting image data, the image data collection device 140 also collects a signal indicating the removal of the fueling nozzle 121_1 from the first fuel dispenser 120_1 and the return of the fueling nozzle 121_1 to the first fuel dispenser 120_1 (a signal indicating the state of the fueling nozzle).

[0016] The image data collected by the image data collection device 140 and the signal indicating the state of the fueling nozzle are transmitted to the learning device 150 via the network 180, and learning data is generated in the learning device 150. Note that the learning data is generated based on the image data collected at a plurality of service stations (e.g., service stations 110, 111,...) and the signal indicating the state of the fueling nozzle. However, the learning data is generated separately for the first fuel type and the second fuel type.

[0017] The learning device 150 is installed at a location different from a plurality of service stations (for example, service stations 110, 111, ···) (for example, a server on the cloud). The learning device 150 uses the learning data for the first fuel type (referred to as the first learning data) to perform learning of the first model and generate a first learned model. Further, the learning device 150 uses the learning data for the second fuel type (referred to as the second learning data) to perform learning of the second model and generate a second learned model.

[0018] The first learned model and the second learned model generated by the learning device 150 are transmitted to a plurality of service stations (for example, service stations 110', 111', ···) via the network 180. Thereby, they are stored in the information processing device of each service station (for example, the information processing device 160 of the service station 110').

[0019] As shown in FIG. 1, the fuel supply system 110S' according to the first embodiment applied to the service station 110' includes, for the first fuel type, · a camera 130_1, · a first fuel dispenser 120_1, · a thermal sensor 170_1. The first fuel dispenser 120_1 is connected to a fuel nozzle 121_1 to which the fuel oil of the first fuel type stored in the tank 126_1 is supplied via a fuel supply pipe by a fuel supply pump 125_1. A fuel supply valve 124_1 is installed in the fuel supply pipe. Further, the first fuel dispenser 120_1 is provided with a voice output device 122_1 and a fuel supply data display 123_1 as notification devices.

[0020] Also, the fuel supply system 110S' according to the first embodiment applied to the service station 110' has a similar configuration for the second fuel type. However, when the measurement area of the thermal sensor 170_1 includes both the fuel supply scene 110_1 of the first fuel type and the fuel supply scene 110_2 of the second fuel type, the thermal sensor 170_1 may be used for both the first fuel type and the second fuel type.

[0021] Furthermore, as shown in FIG. 1, the fueling system 110S' according to the first embodiment applied to the service station 110' has an information processing device 160. The information processing device 160 is · Cameras for the first fuel type and the second fuel type (cameras 130_1, 130_2), · Fuel dispensers (first fuel dispenser 120_1, second fuel dispenser 120_2), · Thermal sensors (thermal sensors 170_1, 170_2), · Fuel pumps (fuel pumps 125_1, 125_2), and is communicably connected.

[0022] In the example of FIG. 1, the case where the fueling system 110S' according to the first embodiment is applied to the service station 110' is shown. However, when it is applied to other service stations (for example, service stations 111', ···), the configuration is the same.

[0023] <Description of the fueling scene> Next, a specific example of the fueling scene at the service station 110 will be described. FIGS. 2A and 2B are the first and second figures showing an example of the fueling scene at the service station.

[0024] Among these, FIG. 2A shows an example of the fueling scene for the first fuel type at the service station 110. As shown in FIG. 2A, the service station 110 includes a first area 210 where the vehicle 211 stops and a second area 220 where the first fuel dispenser 120_1 for fueling the first fuel type is installed and where the fueling operation by the fueler 221 is performed.

[0025] In the case of the refueling scene shown in Fig. 2A, with the vehicle 211 stopped in the first area 210, the refueler 221 removes the fuel nozzle 121_1 from the nozzle hanger of the first fuel dispenser 120_1 and inserts the fuel nozzle 121_1 into the fuel inlet of the vehicle 211. Also, the refueler 221 starts refueling by operating the lever of the fuel nozzle 121_1 and continues refueling until the desired fuel quantity is reached. Then, when the desired fuel quantity is reached, the refueler 221 stops refueling by returning the lever of the fuel nozzle 121_1, removes the fuel nozzle 121_1 from the fuel inlet, and then returns it to the nozzle hanger of the first fuel dispenser 120_1 to end the refueling operation.

[0026] On the other hand, Fig. 2B shows an example of a refueling scene for the second fuel type at the service station 110. As shown in Fig. 2B, the service station 110 includes an area where the container 231 (poly tank) is placed and an area where the second fuel dispenser 120_2 for refueling the second fuel type is installed, and is an area where the refueling operation by the refueler 241 is performed.

[0027] In the case of the refueling scene shown in Fig. 2B, with the container 231 placed, the refueler 241 removes the fuel nozzle 121_2 from the nozzle hanger of the second fuel dispenser 120_2 and inserts the fuel nozzle 121_2 into the fuel inlet of the container 231. Also, the refueler 241 starts refueling by operating the lever of the fuel nozzle 121_2 and continues refueling until the desired fuel quantity is reached. Then, when the desired fuel quantity is reached, the refueler 241 stops refueling by returning the lever of the fuel nozzle 121_2, removes the fuel nozzle 121_2 from the fuel inlet, and then returns it to the nozzle hanger of the second fuel dispenser 120_2 to end the refueling operation.

[0028] <Process for generating learning data> Next, an overview of the process from when the camera 130_1 captures the area including the first area 210 and the second area 220 as a video until the learning device 150 generates the first learning data based on the video will be described. Fig. 3A is a diagram showing the flow of the process for generating the first learning data.

[0029] As shown in FIG. 3A, the image data collection device 140 collects image data of a predetermined frame period, which is captured as a video by the camera 130_1. Further, as shown in FIG. 3A, the image data collection device 140 collects a signal indicating the state of the fuel nozzle 121_1 from the first fuel dispenser 120_1.

[0030] The learning device 150 extracts, from the image data collected by the image data collection device 140, the image data during the period from when the fuel nozzle 121_1 is removed from the nozzle mounting of the first fuel dispenser 120_1 until it is returned to the nozzle mounting.

[0031] Further, the learning device 150 performs image recognition processing on the extracted image data, and recognizes each recognition target (such as a vehicle, a container, a fueler, a fuel nozzle, etc.) included in the extracted image data. Furthermore, the learning device 150 assigns an annotation to the recognized recognition target (such as a vehicle, a container, a fueler, a fuel nozzle, etc.).

[0032] Specifically, when a vehicle is recognized in the image data and the vehicle is recognized within the first region 210 which is a part of the image data, an annotation indicating that it is a vehicle is assigned to the circumscribed rectangle (bounding box) of the vehicle.

[0033] When a container is recognized in the image data, an annotation indicating that it is a container is assigned to the circumscribed rectangle (bounding box) of the container.

[0034] When a fueler is recognized in the image data, · the fueler is recognized within the second region 220, and · the fuel nozzle is held in an appropriate posture, an annotation indicating that it is a fueler holding the fuel nozzle in an appropriate posture is assigned to the circumscribed rectangle (bounding box) of the fueler.

[0035] Also, in the case where the fueler is recognized in the image data, · the fueler is recognized within the second region 220, and · the fuel nozzle being held is not recognized, in this case, an annotation (fuel nozzle recognition impossible) indicating that the fueler is not holding the fuel nozzle in an appropriate posture is given to the circumscribed rectangle (bounding box) of the fueler.

[0036] Also, in the case where the fueler is not recognized within the second region 220 in the image data, · for example, the fuel nozzle in a state of being inserted into the fuel filling port is recognized, in this case, an annotation (fueler recognition impossible) indicating that the fueler is not holding the fuel nozzle in an appropriate posture is given to the circumscribed rectangle (bounding box) of the recognized fuel nozzle.

[0037] The learning device 150 generates first learning data using the image data with any annotation among the extracted image data.

[0038] Subsequently, an outline of the process from when the camera 130_2 that captures the region including the second fuel dispenser 120_2 as a video captures the second fuel dispenser 120_2 and the placed container until the learning device 150 generates second learning data based on the video will be described. FIG. 3B is a diagram showing the flow of the process for generating the second learning data.

[0039] As shown in FIG. 3B, the image data collection device 140 collects the image data of a predetermined frame period captured as a video by the camera 130_2. Also, as shown in FIG. 3B, the image data collection device 140 collects a signal indicating the state of the fuel nozzle 121_2 from the second fuel dispenser 120_2.

[0040] The learning device 150 extracts, from the image data collected by the image data collection device 140, the image data during the period from when the fueling nozzle 121_2 is removed from the nozzle mount of the second fuel dispenser 120_2 until it is returned to the nozzle mount.

[0041] Further, the learning device 150 performs image recognition processing on the extracted image data, and recognizes each recognition target (such as a container, a fueler, a fueling nozzle, etc.) included in the extracted image data. Furthermore, the learning device 150 assigns an annotation to the recognized recognition target (such as a container, a fueler, a fueling nozzle, etc.).

[0042] Specifically, when a container is recognized in the image data, an annotation indicating that it is a container is assigned to the circumscribed rectangle (bounding box) of the recognized container.

[0043] Also, when a fueler is recognized in the image data, · if the fueler is holding the fueling nozzle in an appropriate posture, then an annotation indicating that it is a fueler holding the fueling nozzle in an appropriate posture is assigned to the circumscribed rectangle (bounding box) of the fueler.

[0044] Also, when a fueler is recognized in the image data, · if the fueling nozzle held by the fueler cannot be recognized, then an annotation (fueling nozzle recognition impossible) indicating that it is a fueler not holding the fueling nozzle in an appropriate posture is assigned to the circumscribed rectangle (bounding box) of the fueler.

[0045] Also, when a fueler is not recognized in the image data, · for example, if a fueling nozzle inserted into the fueling port is recognized, In this case, an annotation (fueler unrecognizable) indicating that the fueler is not holding the fuel nozzle in an appropriate posture is attached to the circumscribed rectangle (bounding box) of the fuel nozzle.

[0046] The learning device 150 generates second learning data using the image data with any annotation among the extracted image data.

[0047] <Hardware Configuration of Learning Device> Next, the hardware configuration of the learning device 150 will be described. FIG. 4 is a diagram showing an example of the hardware configuration of the learning device.

[0048] As shown in FIG. 4, the learning device 150 includes a processor 401, a memory 402, an auxiliary storage device 403, an I / F (Interface) device 404, a communication device 405, and a drive device 406. Note that the processor 401, the memory 402, the auxiliary storage device 403, the I / F device 404, the communication device 405, and the drive device 406 of the learning device 150 are interconnected via a bus 407.

[0049] The processor 401 includes various arithmetic devices such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 401 reads and executes various programs (for example, a learning program, etc.) on the memory 402.

[0050] The memory 402 includes main storage devices such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The processor 401 and the memory 402 form a so-called computer, and the computer realizes various functions by the processor 401 executing various programs read on the memory 402.

[0051] The auxiliary storage device 403 stores various programs and various information used when the various programs are executed by the processor 401.

[0052] The I / F device 404 is a connection device for connecting an operation device 411, which is an example of an external device, a display device 412, and a learning device 150.

[0053] The communication device 405 is a communication device for communicating with, for example, an image data collection device 140 or an information processing device 160 via a network.

[0054] The drive device 406 is a device for setting a recording medium 413. The recording medium 413 here includes media that optically, electrically, or magnetically record information, such as CD-ROMs, flexible disks, magneto-optical disks, etc. The recording medium 413 may also include semiconductor memories that electrically record information, such as ROMs, flash memories, etc.

[0055] Note that various programs installed in the auxiliary storage device 403 are installed, for example, when a distributed recording medium 413 is set in the drive device 406 and various programs recorded on the recording medium 413 are read out. Alternatively, various programs installed in the auxiliary storage device 403 may be installed by being downloaded from a network via the communication device 405.

[0056] <Functional Configuration of the Learning Device> Next, the functional configuration of the learning device 150 will be described. FIG. 5 is a diagram showing an example of the functional configuration of the learning device. As described above, a learning program is installed in the learning device 150, and when the program is executed, the learning device 150 functions as a learning data generation unit 500A and a learning unit 500B.

[0057] As shown in FIG. 5, the learning data generation unit 500A further includes a collected data acquisition unit 510, an image recognition unit 520, an annotation adding unit 530, and a learning data generation unit 540.

[0058] The data collection unit 510 acquires, from the image data collection device 140, the image data captured by the camera 130_1 and the signal indicating the state of the fuel nozzle 121_1. Further, the data collection unit 510 acquires, from the image data collection device 140, the image data captured by the camera 130_2 and the signal indicating the state of the fuel nozzle 121_2.

[0059] The image recognition unit 520 extracts, from the image data captured by the camera 130_1, the image data during the period from when the fuel nozzle 121_1 is removed from the nozzle holder of the first fuel dispenser 120_1 until it is returned to the nozzle holder of the first fuel dispenser 120_1. Further, the image recognition unit 520 performs image recognition processing on the extracted image data. Thereby, the image recognition unit 520 recognizes the recognition targets (such as vehicles, containers, fuelers, fuel nozzles, etc.).

[0060] Also, the image recognition unit 520 extracts, from the image data captured by the camera 130_2, the image data during the period from when the fuel nozzle 121_2 is removed from the nozzle holder of the second fuel dispenser 120_2 until it is returned to the nozzle holder of the second fuel dispenser 120_2. Further, the image recognition unit 520 performs image recognition processing on the extracted image data. Thereby, the image recognition unit 520 recognizes the recognition targets (such as containers, fuelers, fuel nozzles, etc.).

[0061] The annotation adding unit 530 adds an annotation to the bounding box of the recognition target recognized by the image recognition unit 520. Since the method of adding the annotation has been described with reference to FIGS. 3A and 3B, the description is omitted here.

[0062] The learning data generation unit 540 generates first learning data and second learning data using the image data annotated by the annotation adding unit 530 as input data and the added annotation as correct data. The learning data generation unit 540 stores the generated first learning data and second learning data in the learning data storage unit 500C.

[0063] Also, as shown in FIG. 5, the learning unit 500B further includes a learning data reading unit 550, a first model 560_1, a comparison / modification unit 570_1, a second model 560_2, and a comparison / modification unit 570_2.

[0064] The learning data reading unit 550 reads the first learning data from the learning data storage unit 500C, inputs the input data included in the first learning data into the first model 560_1, and inputs the correct answer data into the comparison / modification unit 570_1.

[0065] Also, the learning data reading unit 550 reads the second learning data from the learning data storage unit 500C, inputs the input data included in the second learning data into the second model 560_2, and inputs the correct answer data into the comparison / modification unit 570_2.

[0066] The first model 560_1 is a machine learning model that performs image recognition processing. When the input data included in the first learning data is input, the image recognition processing is performed, and the recognition result is output.

[0067] The comparison / modification unit 570_1 compares the recognition result output from the first model 560_1 with the correct answer data input by the learning data reading unit 550. Also, the comparison / modification unit 570_1 updates the model parameters of the first model 560_1 so that the recognition result output from the first model 560_1 matches the correct answer data. In this way, the learning process for the first model 560_1 is performed, and the first learned model is generated.

[0068] The second model 560_2 is a machine learning model that performs image recognition processing. When the input data included in the second learning data is input, the image recognition processing is performed, and the recognition result is output.

[0069] The comparison / modification unit 570_2 compares the recognition result output from the second model 560_2 with the correct data input by the learning data reading unit 550. Further, the comparison / modification unit 570_2 updates the model parameters of the second model 560_2 so that the recognition result output from the second model 560_2 matches the correct data. In this way, the learning process for the second model 560_2 is performed, and the second learned model is generated.

[0070] Note that the first learned model and the second learned model generated by the learning unit 500B are transmitted to the information processing apparatus 160 or the like.

[0071] <Specific Example of Learning Data> Next, a specific example of the learning data generated by the learning data generation unit 500A of the learning apparatus 150 will be described.

[0072] (1) Specific Example of the First Learning Data FIG. 6 is a diagram showing a specific example of the first learning data. As shown in FIG. 6, the first learning data 600 includes "input data" and "correct data" as information items. Further, the "input data" further includes an "image data ID".

[0073] The "image data ID" stores an identifier for identifying the image data annotated by the annotation adding unit 530. In the example of FIG. 6, the image data 610 is the image data with the image data ID = ID001, the image data 620 is the image data with the image data ID = ID00L, the image data 630 is the image data with the image data ID = ID00M, and the image data 640 is the image data with the image data ID = ID00N.

[0074] "Correct data" further includes "regions" and "annotation data". The "regions" store the first region 210 and the second region 220, and the "annotation data" stores the annotation data assigned to the bounding box of the recognition target recognized in the corresponding region.

[0075] In the example of FIG. 6, for the bounding box 611 of the vehicle recognized in the first region 210 which is a part of the image data 610, "Annotation data indicating that it is a vehicle" is shown to be assigned. Also, in the example of FIG. 6, for the bounding box 612 of the fueler recognized in the second region 220 which is a part of the image data 610, "Annotation data indicating that it is a fueler holding the fuel nozzle in an appropriate posture" is shown to be assigned.

[0076] Similarly, in the example of FIG. 6, for the bounding box 621 of the vehicle recognized in the first region 210 which is a part of the image data 620, "Annotation data indicating that it is a vehicle" is shown to be assigned. Also, in the example of FIG. 6, for the bounding box 622 of the fueler recognized in the second region 220 which is a part of the image data 620, "Annotation data indicating that it is a fueler not holding the fuel nozzle in an appropriate posture (fuel nozzle recognition impossible)" is shown to be assigned.

[0077] Similarly, in the example of FIG. 6, for the bounding box 631 of the vehicle recognized in the first region 210 which is a part of the image data 630, "Annotation data indicating that it is a vehicle" is shown to be assigned. Also, in the example of FIG. 6, for the bounding box 632 of the fuel nozzle recognized in the second region 220 which is a part of the image data 630, "Annotation data indicating that the fueler is not holding the fuel nozzle in an appropriate posture (fueler unrecognizable)" indicates that it has been added.

[0078] Similarly, in the example of FIG. 6, with respect to the bounding box 641 of the container recognized in the first region 210 which is a part of the image data 640, "Annotation data indicating that it is a container" indicates that it has been added. Also, in the example of FIG. 6, with respect to the bounding box 642 of the fueler recognized in the second region 220 which is a part of the image data 640, "Annotation data indicating that the fueler is holding the fuel nozzle in an appropriate posture" indicates that it has been added.

[0079] (2) Specific examples of the second learning data FIG. 7 is a diagram showing a specific example of the second learning data. As shown in FIG. 7, the second learning data 700 includes, as information items, "input data" and "correct answer data". Further, the "input data" includes "image data ID".

[0080] The "image data ID" stores an identifier for identifying the image data to which an annotation has been added by the annotation adding unit 530. In the example of FIG. 7, the image data 710 is the image data with the image data ID = ID101, the image data 720 is the image data with the image data ID = ID10L, the image data 730 is the image data with the image data ID = ID10M, and the image data 740 is the image data with the image data ID = ID10N.

[0081] The "correct answer data" further includes "annotation data". The "annotation data" stores the annotation data attached to the recognized bounding box of the recognition target.

[0082] The example in FIG. 7 shows that for the bounding box 711 of the container recognized in the image data 710, "annotation data indicating that it is a container" has been assigned. Also, the example in FIG. 7 shows that for the bounding box 712 of the fueler recognized in the image data 710, "annotation data indicating that it is a fueler holding the fuel nozzle in an appropriate posture" has been assigned.

[0083] Similarly, the example in FIG. 7 shows that for the bounding box 721 of the container recognized in the image data 720, "annotation data indicating that it is a container" has been assigned. Also, the example in FIG. 7 shows that for the bounding box 722 of the fueler recognized in the image data 720, "annotation data indicating that it is a fueler not holding the fuel nozzle in an appropriate posture (fuel nozzle not recognized)" has been assigned.

[0084] Similarly, the example in FIG. 7 shows that for the bounding box 731 of the container recognized in the image data 730, "annotation data indicating that it is a container" has been assigned. Also, the example in FIG. 7 shows that for the bounding box 732 of the fuel nozzle recognized in the image data 730, "annotation data indicating that it is a fueler not holding the fuel nozzle in an appropriate posture (fueler not recognized)" has been assigned.

[0085] Similarly, the example in FIG. 7 shows that for the bounding box 742 of the fueler recognized in the image data 740, "annotation data indicating that it is a fueler holding the fuel nozzle in an appropriate posture" has been assigned.

[0086] <Flow of Learning Process by Learning Device> Next, the flow of the learning process by the learning device 150 will be described. FIG. 8 is an example of a flowchart showing the flow of the learning process.

[0087] In step S801, the learning device 150 acquires image data and a signal indicating the state of the fuel nozzle from the image data collection device 140.

[0088] In step S802, the learning device 150 extracts the image data from when the fuel nozzle is removed from the nozzle holder until it is returned to the nozzle holder, and performs image recognition processing on the extracted image data.

[0089] In step S803, the learning device 150 identifies a first region and a second region in the image data used for generating the first learning data among the image data on which the image recognition processing has been performed.

[0090] In step S804, the learning device 150 assigns an annotation to the image data used for generating the first learning data. Specifically, the learning device 150 assigns an annotation to the bounding box of the recognition target included in the identified first region, and also assigns an annotation to the bounding box of the recognition target included in the identified second region.

[0091] Also, the learning device 150 assigns an annotation to the image data used for generating the second learning data. Specifically, the learning device 150 assigns an annotation to the bounding box of the recognition target.

[0092] In step S805, the learning device 150 generates first learning data for the first fuel type and second learning data for the second fuel type based on the image data to which the annotation has been assigned.

[0093] In step S806, the learning device 150 performs learning processing on the first model using the first learning data to generate a first learned model. Also, the learning device 150 performs learning processing on the second model using the second learning data to generate a second learned model.

[0094] In step S807, the learning device 150 transmits the generated first learned model and second learned model to the information processing device 160 etc. and stores them in the information processing device 160 etc.

[0095] <Hardware Configuration of Information Processing Device> Next, the hardware configuration of the information processing device 160 will be described. FIG. 9 is a diagram showing an example of the hardware configuration of the information processing device.

[0096] As shown in FIG. 9, the information processing device 160 includes a processor 901, a memory 902, an auxiliary storage device 903, a connection device 904, an I / F device 905, a communication device 906, and a drive device 907. Note that the processor 901, memory 902, auxiliary storage device 903, connection device 904, I / F device 905, communication device 906, and drive device 907 of the information processing device 160 are interconnected via a bus 908.

[0097] The processor 901 includes various arithmetic devices such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 901 reads out and executes various programs (for example, a fuel supply control program etc.) on the memory 902.

[0098] The memory 902 includes main storage devices such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The processor 901 and the memory 902 form a so-called computer, and the computer realizes various functions by the processor 901 executing various programs read out on the memory 902.

[0099] The auxiliary storage device 903 stores various programs and various information used when the various programs are executed by the processor 901.

[0100] The connection device 904 is a connection device for connecting the first fuel dispenser 120_1, the second fuel dispenser 120_2, and the information processing device 160. The connection device 904 receives signals indicating the state of the fuel nozzle, signals indicating the fuel quantity, etc. from the first fuel dispenser 120_1 and the second fuel dispenser 120_2. Signals indicating the state of the fuel nozzle, signals indicating the fuel quantity, etc. are measured by the first fuel sensor 921_1 and the second fuel sensor 921_2. Further, the connection device 904 transmits messages for voice output to the first fuel dispenser 120_1 and the second fuel dispenser 120_2 via the voice output devices 122_1 and 122_2. Also, the connection device 904 transmits display information for display to the first fuel dispenser 120_1 and the second fuel dispenser 120_2 via the fuel data displays 123_1 and 123_2. In addition, the connection device 904 transmits signals for instructing the opening and closing operations of the fuel valves 124_1 and 124_2 to the first fuel dispenser 120_1 and the second fuel dispenser 120_2. Furthermore, the connection device 904 transmits signals for instructing the start / stop of driving to the fuel pumps 125_1 and 125_2.

[0101] Also, the connection device 904 receives thermal sensor data from the thermal sensors 170_1 and 170_2.

[0102] The I / F device 905 is a connection device for connecting an operation device 911 and a display device 912, which are examples of external devices, and the information processing device 160. Note that the operation device 911 is operated by, for example, the staff of the service station 110', and the display device 912 is displayed for, for example, the staff of the service station 110'.

[0103] The communication device 906 communicates with the cameras 130_1 and 130_2 via a network and receives image data. Also, the communication device 906 communicates with the learning device 150 and receives the first learned model and the second learned model.

[0104] The drive device 907 is a device for setting the recording medium 913. The recording medium 913 here includes media that record information optically, electrically, or magnetically, such as CD-ROMs, flexible disks, magneto-optical disks, etc. Further, the recording medium 913 may include semiconductor memories that record information electrically, such as ROMs and flash memories.

[0105] Note that various programs installed in the auxiliary storage device 903 are installed, for example, when the distributed recording medium 913 is set in the drive device 907 and various programs recorded on the recording medium 913 are read out. Alternatively, various programs installed in the auxiliary storage device 903 may be installed by being downloaded from a network via the communication device 906.

[0106] <Functional Configuration of Information Processing Device> Next, the functional configuration of the information processing device 160 will be described. FIG. 10 is a diagram showing an example of the functional configuration of the information processing device. As described above, a fuel supply control program is installed in the information processing device 160, and when the program is executed, the information processing device 160 · First fuel supply type image data acquisition unit 1001, second fuel supply type image data acquisition unit 1002, · First fuel supply sensor data acquisition unit 1003, second fuel supply sensor data acquisition unit 1004, · Image recognition unit 1005, · Section determination units 1006, 1007, · Thermal sensor data acquisition unit 1008, · First fuel supply monitoring units 1009_1, 1009_2, · Output control unit 1010, has.

[0107] The first fuel supply type image data acquisition unit 1001 acquires the image data captured by the camera 130_1 and notifies the image recognition unit 1005.

[0108] The second fuel supply type image data acquisition unit 1002 acquires the image data captured by the camera 130_2 and notifies the image recognition unit 1005.

[0109] The first fuel supply sensor data acquisition unit 1003 acquires the signal from the first fuel supply sensor 921_1 and notifies the section determination unit 1006.

[0110] The second fuel supply sensor data acquisition unit 1004 acquires the signal from the second fuel supply sensor 921_2 and notifies the section determination unit 1007.

[0111] Based on the signal from the first fuel supply sensor 921_1, the section determination unit 1006 classifies the section from when the fuel supply nozzle 121_1 is removed from the nozzle hanger of the first fuel dispenser 120_1 until it is returned to the nozzle hanger into three sections. Among the three sections, the section before fuel supply start is called the first section, the section during fuel supply is called the second section, and the section after fuel supply end is called the third section.

[0112] Based on the signal from the second fuel supply sensor 921_2, the section determination unit 1007 classifies the section from when the fuel supply nozzle 121_2 is removed from the nozzle hanger of the second fuel dispenser 120_2 until it is returned to the nozzle hanger into three sections. For the second fuel supply type as well, similar to the first fuel supply type, among the three sections, the section before fuel supply start is called the first section, the section during fuel supply is called the second section, and the section after fuel supply end is called the third section.

[0113] The image recognition unit 1005 extracts the image data included in any of the first to third sections from the image data notified by the first fuel supply type image data acquisition unit 1001. Further, the image recognition unit 1005 performs image recognition processing on the extracted image data using the first learned model and notifies the recognition result to the first fuel supply monitoring unit 1009_1.

[0114] Further, the image recognition unit 1005 extracts the image data included in any one of the first to third intervals from the image data notified by the second fuel type image data acquisition unit 1002. Further, the image recognition unit 1005 performs image recognition processing on the extracted image data using the second learned model, and notifies the recognition result to the second fuel supply monitoring unit 1009_2.

[0115] The thermal sensor data acquisition unit 1008 acquires the thermal sensor data measured by the thermal sensor 170_1 and notifies the first fuel supply monitoring unit 1009_1. Further, the thermal sensor data acquisition unit 1008 acquires the thermal sensor data measured by the thermal sensor 170_2 and notifies the second fuel supply monitoring unit 1009_2.

[0116] The first fuel supply monitoring unit 1009_1 · the recognition result notified by the image recognition unit 1005, · the section determination result notified by the section determination unit 1006, · the thermal sensor data of the thermal sensor 170_1 notified by the thermal sensor data acquisition unit 1008, and uses them to determine whether fuel supply for the first fuel type is permitted. Further, the first fuel supply monitoring unit 1009_1 controls the fuel supply for the first fuel type based on the determination result of whether fuel supply is permitted. Specifically, the first fuel supply monitoring unit 1009_1 controls the fuel supply for the first fuel type by controlling the fuel valve 124_1 or the fuel pump 125_1.

[0117] Further, the first fuel supply monitoring unit 1009_1 notifies the output control unit 1010 of the determination result of whether fuel supply is permitted. In addition, when the first fuel supply monitoring unit 1009_1 determines that fuel supply is not permitted, it also notifies the output control unit 1010 of the information indicating the cause of non-permission of fuel supply.

[0118] The second fuel supply monitoring unit 1009_2 · the recognition result notified by the image recognition unit 1005, · the section determination result notified by the section determination unit 1007, · The thermal sensor data of the thermal sensor 170_2 notified by the thermal sensor data acquisition unit 1008, and using this, determine whether refueling is permitted for the second fuel type. Also, the second refueling monitoring unit 1009_2 controls the refueling for the second fuel type based on the determination result of whether refueling is permitted. Specifically, the second refueling monitoring unit 1009_2 controls the refueling for the second fuel type by controlling the refueling valve 124_2 or the fuel pump 125_2.

[0119] Also, the second refueling monitoring unit 1009_2 notifies the output control unit 1010 of the determination result of whether refueling is permitted. Note that when the second refueling monitoring unit 1009_2 determines that refueling is not permitted, it also notifies the output control unit 1010 of the information indicating the cause of the non - permission of refueling.

[0120] When the output control unit 1010 is notified of the information indicating the cause of non - permission of refueling from the first refueling monitoring unit 1009_1, · Control so that a message corresponding to the information indicating the cause is output as voice via the voice output device 122_1, · Control so that the display information corresponding to the information indicating the cause is displayed and output on the fueling data display 123_1 of the first fuel dispenser 120_1, · Control so that the information indicating the cause is displayed and output on the display device 912 of the information processing device 160.

[0121] Also, when the output control unit 1010 is notified of the information indicating the cause of non - permission of refueling from the second refueling monitoring unit 1009_2, · Control so that a message corresponding to the information indicating the cause is output as voice via the voice output device 122_2, · Control so that the display information corresponding to the information indicating the cause is displayed and output on the fueling data display 123_2 of the second fuel dispenser 120_2, · Control so that the information indicating the cause is displayed and output on the display device 912 of the information processing device 160.

[0122] <Details of each part of the information processing device> Next, details of each part of the information processing apparatus 160 (here, the section determination units 1006 and 1007, the image recognition unit 1005, the first fuel supply monitoring unit 1009_1, the second fuel supply monitoring unit 1009_2, and the output control unit 1010) will be described.

[0123] (1) Details of the section determination units 1006 and 1007 First, as details of the section determination units 1006 and 1007, the functional configurations of the section determination units 1006 and 1007 and specific examples of the processing of the section determination units 1006 and 1007 will be described.

[0124] (1-1) Details of the functional configurations of the section determination units 1006 and 1007 FIG. 11 is a diagram showing an example of the functional configuration of the section determination unit of the information processing apparatus. Since the functional configurations of the section determination unit 1006 and the section determination unit 1007 are the same, they will be described together using FIG. 11.

[0125] As shown in FIG. 11, the section determination units 1006 and 1007 · nozzle state determination unit 1101, · fuel supply amount determination unit 1102, · first section start determination unit 1103, second section transition determination unit 1104, third section transition determination unit 1105, third section end determination unit 1106, · section determination result output unit 1107, have.

[0126] The nozzle state determination unit 1101 acquires a signal indicating the state of the fuel supply nozzle 121_1 (or 121_2) from the first fuel supply sensor data acquisition unit 1003 (or the second fuel supply sensor data acquisition unit 1004).

[0127] When the nozzle state determination unit 1101 receives a signal indicating that the fuel supply nozzle 121_1 (or 121_2) has been removed from the nozzle mount of the first fuel dispenser 120_1 (or the second fuel dispenser 120_2), it notifies the first section start determination unit 1103.

[0128] Further, when the nozzle state determination unit 1101 receives a signal indicating that the fuel supply nozzle 121_1 (or 121_2) has been returned to the nozzle hook of the first fuel dispenser 120_1 (or the second fuel dispenser 120_2), it notifies the third section end determination unit 1106.

[0129] The fuel supply amount determination unit 1102 acquires a signal indicating the fuel supply amount from the first fuel supply sensor data acquisition unit 1003 (or the second fuel supply sensor data acquisition unit 1004).

[0130] When the signal indicating the fuel supply amount acquired by the fuel supply amount determination unit 1102 from the first fuel supply sensor data acquisition unit 1003 (or the second fuel supply sensor data acquisition unit 1004) changes from zero to a predetermined threshold value or more, the fuel supply amount determination unit 1102 notifies the second section transition determination unit 1104.

[0131] Also, when the signal indicating the fuel supply amount acquired by the fuel supply amount determination unit 1102 from the first fuel supply sensor data acquisition unit 1003 (or the second fuel supply sensor data acquisition unit 1004) changes from a predetermined threshold value or more to zero, the fuel supply amount determination unit 1102 notifies the third section transition determination unit 1105.

[0132] When the first section start determination unit 1103 is notified from the nozzle state determination unit 1101 of a signal indicating that the fuel supply nozzle 121_1 (or 121_2) has been removed, it determines that the first section has started. Also, the first section start determination unit 1103 notifies the second section transition determination unit 1104 and the section determination result output unit 1107 that the first section has started.

[0133] After being notified that the first section has started, when the second section transition determination unit 1104 is notified from the fuel supply amount determination unit 1102 that the fuel supply amount has changed from zero to a predetermined threshold value or more, it determines that the transition has been made from the first section to the second section. Also, the second section transition determination unit 1104 notifies the third section transition determination unit 1105 and the section determination result output unit 1107 that the transition has been made from the first section to the second section.

[0134] After being notified of the transition to the second section, the third-section transition determination unit 1105 determines that the transition to the third section has occurred when it is notified from the fuel supply amount determination unit 1102 that the fuel supply amount has changed from equal to or greater than a predetermined threshold value to zero. Further, the third-section transition determination unit 1105 notifies the third-section end determination unit 1106 and the section determination result output unit 1107 of the transition from the second section to the third section.

[0135] After being notified of the transition to the third section, the third-section end determination unit 1106 determines that the third section has ended when it is notified from the nozzle state determination unit 1101 of a signal indicating that the fuel supply nozzle 121_1 (or 121_2) has been retracted. Further, the third-section end determination unit 1106 notifies the section determination result output unit 1107 of the end of the third section.

[0136] When the section determination result output unit 1107 is notified from the first-section start determination unit 1103 that the first section has started, it starts outputting information indicating that it is during the first section. Further, when the section determination result output unit 1107 is notified from the second-section transition determination unit 1104 of the transition to the second section, it stops outputting information indicating that it is during the first section and starts outputting information indicating that it is during the second section. Further, when the section determination result output unit 1107 is notified from the third-section transition determination unit 1105 of the transition to the third section, it stops outputting information indicating that it is during the second section and starts outputting information indicating that it is during the third section. Further, when the section determination result output unit 1107 is notified from the third-section end determination unit 1106 of the end of the third section, it stops outputting information indicating that it is during the third section.

[0137] (1-2) Specific examples of the processing of the section determination units 1006 and 1007 FIG. 12 is a diagram showing a specific example of the processing by the section determination unit of the information processing apparatus. As shown in FIG. 12(a), when a signal indicating that the fueling nozzle 121_1 has been removed from the nozzle mount is acquired as a signal indicating the state of the fueling nozzle 121_1, the section determination unit 1006 starts outputting information indicating that it is during the first section. Also, when a signal indicating that the signal indicating the fueling amount of the first fuel dispenser 120_1 has changed from zero to a predetermined threshold value or more is acquired, the section determination unit 1006 starts outputting information indicating that it is during the second section. Further, when a signal indicating that the signal indicating the fueling amount of the first fuel dispenser 120_1 has changed from a predetermined threshold value or more to zero is acquired, the section determination unit 1006 starts outputting information indicating that it is during the third section. Furthermore, when a signal indicating that the fueling nozzle 121_1 has been returned to the nozzle mount is acquired as a signal indicating the state of the fueling nozzle 121_1, the section determination unit 1006 stops outputting information indicating that it is during the third section.

[0138] Similarly, as shown in FIG. 12(b), when a signal indicating that the fueling nozzle 121_2 has been removed from the nozzle mount is acquired as a signal indicating the state of the fueling nozzle 121_2, the section determination unit 1007 starts outputting information indicating that it is during the first section. Also, when a signal indicating that the signal indicating the fueling amount of the second fuel dispenser 120_2 has changed from zero to a predetermined threshold value or more is acquired, the section determination unit 1007 starts outputting information indicating that it is during the second section. Further, when a signal indicating that the signal indicating the fueling amount of the second fuel dispenser 120_2 has changed from a predetermined threshold value or more to zero is acquired, the section determination unit 1007 starts outputting information indicating that it is during the third section. Furthermore, when a signal indicating that the fueling nozzle 121_2 has been returned to the nozzle mount is acquired as a signal indicating the state of the fueling nozzle 121_2, the section determination unit 1007 stops outputting information indicating that it is during the third section.

[0139] (2) Details of the Image Recognition Unit 1005 Next, as details of the image recognition unit 1005, the functional configuration of the image recognition unit 1005 and a specific example of the processing of the image recognition unit 1005 will be described.

[0140] (2-1) Details of the Functional Configuration of the Image Recognition Unit 1005 FIG. 13 is a diagram showing an example of the functional configuration of the image recognition unit of the information processing apparatus. As shown in FIG. 13, the image recognition unit 1005 includes a recognition section acquisition unit 1301, a first learned model 1302, and a region determination unit 1303. The image recognition unit 1005 also includes a recognition section acquisition unit 1311 and a second learned model 1312.

[0141] The recognition section acquisition unit 1301 extracts the image data included in any one of the first to third sections notified by the section determination unit 1006 from the image data notified by the first fuel type image data acquisition unit 1001, and inputs the extracted image data into the first learned model 1302.

[0142] The first learned model 1302 is the first learned model generated in the learning device 150. When image data is input, it recognizes the recognition target included in the image data and outputs the recognition result.

[0143] The region determination unit 1303 determines whether the recognition result output by the first learned model 1302 is included in the first region 210 or the second region 220. The region determination unit 1303 also outputs the recognition result determined to be included in the first region 210 or the second region 220 to the first fuel monitoring unit 1009_1.

[0144] The recognition section acquisition unit 1311 extracts the image data included in any one of the first to third sections notified by the section determination unit 1007 from the image data notified by the second fuel type image data acquisition unit 1002, and inputs the extracted image data into the second learned model 1312.

[0145] The second learned model 1312 is the second learned model generated in the learning device 150. When image data is input, it recognizes the recognition target included in the image data and outputs the recognition result to the second fuel monitoring unit 1009_2.

[0146] (2-2) Specific Example of the Processing of the Image Recognition Unit 1005 (in the Case of the First Fueling Type) FIG. 14A is a first diagram showing a specific example of the processing by the image recognition unit of the information processing apparatus. As shown in FIG. 14A, the image recognition unit 1005 extracts the image data included in the section (the first to third sections) from when the fueling nozzle 121_1 is removed from the nozzle holder until it is returned, among the image data captured by the camera 130_1.

[0147] Also, as shown in FIG. 14A, the image recognition unit 1005 performs image recognition processing on the extracted image data using the first pre-trained model 1302, and by recognizing the recognition targets (such as vehicles, containers, fuelers, fueling nozzles, etc.), · Vehicles, · Containers, · Fuelers who hold the fueling nozzle in an appropriate posture, · Fuelers who do not hold the fueling nozzle in an appropriate posture (fueling nozzle unrecognizable), · Fuelers who do not hold the fueling nozzle in an appropriate posture (fueler unrecognizable), are output as recognition results.

[0148] Also, the image recognition unit 1005 outputs the recognition results determined to be included in the first area 210 or the second area 220. The recognition results 1401 to 1404 in FIG. 14A show examples of the output of the recognition results.

[0149] In the example of FIG. 14A, the recognition result 1401 output by the image recognition unit 1005 is · In the first area 210, a bounding box in which a vehicle is recognized, · In the second area 220, a bounding box in which a fueler who holds the fueling nozzle in an appropriate posture is recognized, and is shown.

[0150] In the example of FIG. 14A, the recognition result 1402 output by the image recognition unit 1005 is · In the first area 210, a bounding box in which a vehicle is recognized, · In the second area 220, a bounding box (refueling nozzle unrecognizable) in which a refueler not holding the refueling nozzle in an appropriate posture is recognized. is shown.

[0151] Also, in the example of FIG. 14A, the recognition result 1403 output by the image recognition unit 1005 is · A bounding box in which a vehicle is recognized in the first area 210, · In the second area 220, a bounding box (refueler unrecognizable) in which a refueler not holding the refueling nozzle in an appropriate posture is recognized. is shown.

[0152] Furthermore, in the example of FIG. 14A, the recognition result 1404 output by the image recognition unit 1005 is · A bounding box in which a container is recognized in the first area 210, · In the second area 220, a bounding box in which a refueler holding the refueling nozzle in an appropriate posture is recognized. is shown.

[0153] (2-3) Specific example of the process of the image recognition unit 1005 (in the case of the second refueling type) FIG. 14B is a second diagram showing a specific example of the process by the image recognition unit of the information processing apparatus. As shown in FIG. 14B, the image recognition unit 1005 extracts the image data included in the section (the first to the third sections) from when the refueling nozzle 121_2 is removed from the nozzle holder until it is returned, among the image data captured by the camera 130_2.

[0154] Also, as shown in FIG. 14B, the image recognition unit 1005 performs image recognition processing on the extracted image data using the second learned model 1312, and recognizes the recognition targets (container, refueler, refueling nozzle, etc.), thereby · Container, · A refueler holding the refueling nozzle in an appropriate posture, · A refueler not holding the refueling nozzle in an appropriate posture (refueling nozzle unrecognizable), · A fueler who is not holding the fuel nozzle in an appropriate posture (fueler unrecognizable), is output as a recognition result. The recognition results 1411 to 1414 in Fig. 14B show examples of the output of the recognition results.

[0155] In the example of Fig. 14B, the recognition result 1411 output by the image recognition unit 1005 is · The bounding box in which the container is recognized, · The bounding box in which a fueler holding the fuel nozzle in an appropriate posture is recognized. is shown.

[0156] Also, in the example of Fig. 14B, the recognition result 1412 output by the image recognition unit 1005 is · The bounding box in which the container is recognized, · The bounding box in which a fueler not holding the fuel nozzle in an appropriate posture is recognized (fuel nozzle unrecognizable). is shown.

[0157] Also, in the example of Fig. 14B, the recognition result 1413 output by the image recognition unit 1005 is · The bounding box in which the container is recognized, · The bounding box in which a fueler not holding the fuel nozzle in an appropriate posture is recognized (fueler unrecognizable). is shown.

[0158] Furthermore, in the example of Fig. 14B, the recognition result 1414 output by the image recognition unit 1005 is · The bounding box in which a fueler holding the fuel nozzle in an appropriate posture is recognized. is shown.

[0159] (3) Details of the first fueling monitoring unit 1009_1 and the second fueling monitoring unit 1009_2 Next, as details of the first fuel supply monitoring unit 1009_1 and the second fuel supply monitoring unit 1009_2, the functional configuration and processing flow of the first fuel supply monitoring unit 1009_1 and the functional configuration and processing flow of the second fuel supply monitoring unit 1009_2 will be described.

[0160] (3-1) Details of the Functional Configuration of the First Fuel Supply Monitoring Unit 1009_1 and the Second Fuel Supply Monitoring Unit 1009_2 FIG. 15A is a diagram showing an example of the functional configuration of the first fuel supply monitoring unit of the information processing apparatus. As shown in FIG. 15A, the first fuel supply monitoring unit 1009_1 includes a recognition result acquisition unit 1501, a determination logic switching unit 1502, a poly-tank determination unit 1506, a first determination logic 1503, a second determination logic 1504, a third determination logic 1505, and a fire determination unit 1507. Further, the first fuel supply monitoring unit 1009_1 includes a fuel supply control unit 1508 and a cause information acquisition unit 1509. The first determination logic 1503 to the third determination logic 1505, the poly-tank determination unit 1506, and the fire determination unit 1507 are collectively referred to as the first determination unit.

[0161] The recognition result acquisition unit 1501 acquires recognition results for the image data included in the first section to the third section from the image recognition unit 1005. The recognition result acquisition unit 1501 notifies the acquired recognition results to the determination logic switching unit 1502 and the poly-tank determination unit 1506.

[0162] The determination logic switching unit 1502 inputs the recognition result notified by the recognition result acquisition unit 1501 into any one of the first determination logic 1503 to the third determination logic 1505. Which of the first determination logic 1503 to the third determination logic 1505 to input is determined by the information indicating in which of the first to third intervals is notified from the interval determination unit 1006. For example, while the information indicating that it is in the first interval is notified, the determination logic switching unit 1502 inputs the recognition result into the first determination logic 1503. Also, while the information indicating that it is in the second interval is notified, the determination logic switching unit 1502 inputs the recognition result into the second determination logic 1504. Also, while the information indicating that it is in the third interval is notified, the determination logic switching unit 1502 inputs the recognition result into the third determination logic 1505.

[0163] Also, the determination logic switching unit 1502 notifies the poly tank determination unit 1506 and the firearm determination unit 1507 of the information indicating that it is in any one of the first to third intervals.

[0164] When the poly tank determination unit 1506 is notified of the information indicating that it is in any one of the first to third intervals and the recognition result is notified from the recognition result acquisition unit 1501, the poly tank determination unit 1506 determines whether a container is included in the notified recognition result. When the poly tank determination unit 1506 determines that a container is included, it notifies the fuel supply control unit 1508 of fuel supply non - permission.

[0165] The first determination logic 1503 determines whether fuel supply is permitted based on the recognition result in the first interval. Also, the first determination logic 1503 notifies the fuel supply control unit 1508 of the determination result of whether fuel supply is permitted in the first interval. When the first determination logic 1503 determines fuel supply non - permission, when notifying the determination result to the fuel supply control unit 1508, it also notifies the information indicating the reason for determining fuel supply non - permission.

[0166] The second determination logic 1504 determines whether refueling is permitted based on the recognition result during the second period. Further, the second determination logic 1504 notifies the refueling control unit 1508 of the determination result of whether refueling is permitted during the second period. When the second determination logic 1504 determines that refueling is not permitted, it notifies the refueling control unit 1508 of the determination result and also notifies information indicating the reason for the determination that refueling is not permitted.

[0167] The third determination logic 1505 determines whether refueling is permitted based on the recognition result during the third period. Further, the third determination logic 1505 notifies the refueling control unit 1508 of the determination result of whether refueling is permitted during the third period. When the third determination logic 1505 determines that refueling is not permitted, it notifies the refueling control unit 1508 of the determination result and also notifies information indicating the reason for the determination that refueling is not permitted.

[0168] The fire determination unit 1507 acquires the thermal sensor data measured by the thermal sensor 170_1 from the thermal sensor data acquisition unit 1008. Further, when information indicating that it is in any one of the first to third periods is notified, the fire determination unit 1507 determines whether a temperature region equal to or higher than a predetermined threshold value is included. When the fire determination unit 1507 determines that a temperature region equal to or higher than the predetermined threshold value is included, it notifies the refueling control unit 1508 that refueling is not permitted.

[0169] The refueling control unit 1508 controls the refueling valve 124_1 or the fuel pump 125_1 based on whether refueling is permitted or not permitted, thereby controlling the refueling for the first fuel type. For example, when it is notified that refueling is permitted, the refueling control unit 1508 drives the fuel pump 125_1 to open the refueling valve 124_1. Further, the refueling control unit 1508 · When notified of refueling permission from the poly-tank determination unit 1506, drives the fuel pump 125_1 to open the refueling valve 124_1. Furthermore, · When notified of refueling non-permission from any one of the first determination logic 1503 to the third determination logic 1505, or · When a notification of refueling prohibition is received from the fire determination unit 1507, refueling is stopped by stopping the drive of the fuel pump 125_1, or refueling is prohibited by stopping the drive of the fuel pump 125_1 and closing the fuel valve 124_1. Note that the definitions of refueling stop and refueling prohibition here are only examples, and the refueling stop and refueling prohibition may be defined by other definitions. Alternatively, without distinguishing between refueling stop and refueling prohibition, the entire state where refueling cannot be performed may be referred to as refueling stop or prohibition.

[0170] In addition, the fuel supply control unit 1508 notifies the cause information acquisition unit 1509 of fuel supply permission and fuel supply non - permission. When the fuel supply control unit 1508 notifies fuel supply non - permission, it also notifies the cause information acquisition unit 1509 of information indicating the cause of the determination of fuel supply non - permission.

[0171] The cause information acquisition unit 1509 outputs fuel supply permission and fuel supply non - permission to the output control unit 1010. When the cause information acquisition unit 1509 outputs fuel supply non - permission, it also outputs information indicating the cause of fuel supply non - permission (the first to fourth information, warning information, confirmation request information) notified from the fuel supply control unit 1508 to the output control unit 1010.

[0172] Figure 15B is a diagram showing an example of the functional configuration of the second fuel supply monitoring unit of the information processing device. The difference from the functional configuration of the first fuel supply monitoring unit 1009_1 shown in Figure 15A is that the functional configuration of the second fuel supply monitoring unit 1009_2 does not include a functional unit corresponding to the poly - tank determination unit 1506. Since the other functional units of the second fuel supply monitoring unit 1009_2 are the same as those of the first fuel supply monitoring unit 1009_1, the description is omitted here.

[0173] Note that what was called the first determination unit in the first fuel supply monitoring unit 1009_1 is collectively referred to as the second determination unit in the second fuel supply monitoring unit 1009_2, including the first determination logic 1513 to the third determination logic 1515 and the fire determination unit 1517.

[0174] (3-2) Flow of Processing Using the First Determination Logic 1503 of the First Fueling Monitoring Unit 1009_1 Next, the flow of processing using the first determination logic 1503 of the first fueling monitoring unit 1009_1 will be described with reference to FIGS. 16A and 16B. FIGS. 16A and 16B are examples of first and second flowcharts showing the flow of processing using the first determination logic of the first fueling monitoring unit.

[0175] In step S1601, the first determination logic 1503 determines whether the currently input recognition result is a recognition result determined to be fueling permitted (whether it satisfies a predetermined permission condition). Specifically, it is determined whether a vehicle is recognized in the first area 210 and whether a fueler who holds the fueling nozzle in an appropriate posture is recognized in the second area.

[0176] If it is determined in step S1601 that the recognition result is a recognition result determined to be fueling permitted (YES in step S1601), the process proceeds to step S1602.

[0177] In step S1602, the first determination logic 1503 determines whether the recognition result determined to be fueling permitted has continued for a predetermined time T1 (whether the predetermined permission condition has been satisfied for a certain period of time). If it is determined in step S1602 that it has continued for the predetermined time T1 (YES in step S1602), the process proceeds to step S1603.

[0178] In step S1603, the first determination logic 1503 determines that fueling is permitted and proceeds to step S1614 in FIG. 16B.

[0179] On the other hand, if it is determined in step S1601 that the recognition result is not a recognition result determined to be fueling permitted (NO in step S1601), the process proceeds to step S1604. Or, if it is determined in step S1602 that it has not continued for the predetermined time T1 (NO in step S1602), the process proceeds to step S1604.

[0180] In step S1604, the first determination logic 1503 determines whether a state where the recognition result is not determined as fueling permission has continued for a predetermined time T2 (where T2 > T1).

[0181] In step S1604, if it is determined that it has not continued for the predetermined time T2 (if NO in step S1604), the process returns to step S1601. On the other hand, in step S1604, if it is determined that it has continued for the predetermined time T2 (if YES in step S1604), the process proceeds to step S1605.

[0182] In step S1605, the first determination logic 1503 determines whether the currently input recognition result is a recognition result determined as fueling not permitted (whether a predetermined non - permission condition is satisfied). Specifically, it determines whether a vehicle is recognized in the first area 210 and whether a fueler who is not holding the fuel nozzle in an appropriate posture (unrecognizable fuel nozzle) is recognized in the second area.

[0183] In step S1605, if it is determined that the recognition result is a recognition result determined as fueling not permitted (if YES in step S1605), it is determined that the predetermined non - permission condition has been satisfied for a certain time, and the process proceeds to step S1606.

[0184] In step S1606, the first determination logic 1503 determines that fueling is not permitted, acquires the first information, and then proceeds to step S1611 in FIG. 16B.

[0185] On the other hand, in step S1605, if it is determined that the recognition result is another recognition result determined as fueling not permitted (if NO in step S1605), the process proceeds to step S1617 in FIG. 16B.

[0186] In step S1611, the first determination logic 1503 determines whether the currently input recognition result is a recognition result determined to be fueling permission (whether it satisfies a predetermined permission condition). Specifically, it determines whether a vehicle is recognized in the first area 210 and whether a fueler who holds the fuel nozzle in an appropriate posture is recognized in the second area.

[0187] In step S1611, if it is determined that the recognition result is not a recognition result determined to be fueling permission (if NO in step S1611), the first fueling monitoring unit 1009_1 ends the process using the first determination logic 1503.

[0188] On the other hand, in step S1611, if it is determined that the recognition result is a recognition result determined to be fueling permission (if YES in step S1611), the process proceeds to step S1612.

[0189] In step S1612, the first determination logic 1503 determines whether the recognition result determined to be fueling permission has continued for a predetermined time T1 (whether the predetermined permission condition has been satisfied for a certain period of time). In step S1612, if it is determined that it has continued for the predetermined time T1 (if YES in step S1612), the process proceeds to step S1613.

[0190] In step S1613, the first determination logic 1503 determines fueling permission and proceeds to step S1614.

[0191] In step S1614, the determination logic switching unit 1502 determines whether it is within the first section. In step S1614, if it is determined that it is within the first section, it is determined that the process by the first determination logic 1503 is not ended (determined as NO in step S1614), and the process returns to step S1601 in FIG. 16A.

[0192] On the other hand, in step S1614, when it is determined that it is not within the first period, it is determined to end the process by the first determination logic 1503 (determined as YES in step S1614). Then, the first fueling monitoring unit 1009_1 ends the process using the first determination logic 1503.

[0193] On the other hand, in step S1612, when it is determined that the predetermined time T1 has not continued (when NO in step S1612), the process proceeds to step S1615.

[0194] In step S1615, the first determination logic 1503 determines whether a state where the recognition result is not a fueling permission has continued for a predetermined time T3 (however, T3 > T1).

[0195] In step S1615, when it is determined that the predetermined time T3 has not continued (when NO in step S1615), the process returns to step S1611. On the other hand, in step S1615, when it is determined that the predetermined time T3 has continued (when YES in step S1615), it is determined that a predetermined non - permission condition has been satisfied for a certain period of time, and the process proceeds to step S1616.

[0196] In step S1616, the first determination logic 1503 determines fueling non - permission and acquires the first information. After that, the first fueling monitoring unit 1009_1 ends the process using the first determination logic 1503.

[0197] On the other hand, when the process proceeds from step S1605 in FIG. 16A to step S1617 in FIG. 16B, the first determination logic 1503 determines that a predetermined non - permission condition has been satisfied for a certain period of time. Specifically, the first determination logic 1503 determines that the vehicle has not been recognized in the first area 210. Alternatively, the first determination logic 1503 determines that a fueler who is not holding the fuel nozzle in an appropriate posture has been recognized (fueler recognition impossible) in the second area 220.

[0198] In step S1618, the first determination logic 1503 determines no refueling permission and after acquiring the second information, the first refueling monitoring unit 1009_1 ends the process using the first determination logic 1503.

[0199] (3-3) Flow of the process using the second determination logic 1504 of the first refueling monitoring unit 1009_1 Next, the flow of the process using the second determination logic 1504 of the first refueling monitoring unit 1009_1 will be described with reference to FIGS. 17A to 17D. FIGS. 17A to 17D are examples of the first to fourth flowcharts showing the flow of the process using the second determination logic of the first refueling monitoring unit.

[0200] In step S1701, the second determination logic 1504 determines whether the currently input recognition result is a recognition result determined as refueling permission (whether a predetermined permission condition is satisfied). Specifically, it is determined whether a vehicle is recognized in the first area 210 and whether a refueler holding the fuel nozzle in an appropriate posture is recognized in the second area.

[0201] In step 1701, if it is determined that the recognition result is a recognition result determined as refueling permission (YES in step S1701), it is determined that the predetermined permission condition is satisfied, and the process proceeds to step S1702.

[0202] In step S1702, the second determination logic 1504 determines refueling permission and proceeds to step S1734 in FIG. 17D.

[0203] On the other hand, in step S1701, if it is determined that the recognition result is not a recognition result determined as refueling permission (NO in step S1701), the process proceeds to step S1703.

[0204] In step S1703, the second determination logic 1504 determines whether the state where the recognition result is not a recognition result determined as refueling permission has continued for a predetermined time T4.

[0205] In step S1703, if it is determined that the predetermined time T4 has not continued (if NO in step S1703), the process returns to step S1701. On the other hand, in step S1703, if it is determined that the predetermined time T4 has continued (if YES in step S1703), the process proceeds to step S1704.

[0206] In step S1704, the second determination logic 1504 determines whether the currently input recognition result is a recognition result determined as refueling not permitted (whether a predetermined non-permission condition is satisfied). Specifically, it is determined whether a vehicle is recognized in the first area 210 and whether a refueler who is not gripping the fuel nozzle in an appropriate posture is recognized (whether the fuel nozzle cannot be recognized) in the second area.

[0207] In step S1704, if it is determined that it is another recognition result determined as refueling not permitted (if NO in step S1704), the process proceeds to step S1721 in FIG. 17C.

[0208] On the other hand, in step S1704, if it is determined that the recognition result is a recognition result determined as refueling not permitted (if YES in step S1704), it is determined that the predetermined non-permission condition has been satisfied for a certain time, and the process proceeds to step S1705.

[0209] In step S1705, the second determination logic 1504 acquires warning information.

[0210] In step S1706, the second determination logic 1504 determines whether the currently input recognition result is a recognition result determined as refueling permitted (whether a predetermined permission condition is satisfied). Specifically, it is determined whether a vehicle is recognized in the first area 210 and whether a refueler who is gripping the fuel nozzle in an appropriate posture is recognized in the second area.

[0211] If it is determined that the recognition result is the one for which refueling is permitted in step S1706 (if YES in step S1706), the process proceeds to step S1707.

[0212] In step S1707, the second determination logic 1504 determines whether the recognition result for which refueling is permitted has continued for a predetermined time T1. If it is determined in step S1707 that it has continued for the predetermined time T1 (if YES in step S1707), it is determined that the predetermined permission conditions have been satisfied for a certain period of time, and the process proceeds to step S1708.

[0213] In step S1708, the second determination logic 1504 determines that refueling is permitted and proceeds to step S1734 in FIG. 17D.

[0214] On the other hand, if it is determined in step S1706 that the recognition result is not the one for which refueling is permitted (if NO in step S1706), the process proceeds to step S1709. Or, if it is determined in step S1707 that it has not continued for the predetermined time T1 (if NO in step S1707), the process proceeds to step S1709.

[0215] In step S1709, the second determination logic 1504 determines whether the state where the recognition result is not the one for which refueling is permitted has continued for a predetermined time T2 (where T2 > T1).

[0216] If it is determined in step S1709 that it has not continued for the predetermined time T2 (if NO in step S1709), the process returns to step S1706. On the other hand, if it is determined in step S1709 that it has continued for the predetermined time T2 (if YES in step S1709), it is determined that the predetermined non - permission conditions have been satisfied for a certain period of time, and the process proceeds to step S1711 in FIG. 17B.

[0217] In step S1711, the second determination logic 1504 determines that refueling is not permitted, acquires the first information, and then proceeds to step S1712.

[0218] In step S1712, the second determination logic 1504 determines whether, within a predetermined time T2, the currently input recognition result has become a recognition result determined to be fueling permission (whether a predetermined permission condition is satisfied). Specifically, it is determined whether, within the predetermined time T2, a vehicle is recognized in the first area 210 and a fueler who holds the fuel nozzle in an appropriate posture is recognized in the second area.

[0219] In step S1712, if it is determined that the currently input recognition result has become a recognition result determined to be fueling permission within the predetermined time T2 (if YES in step S1712), it is determined that the predetermined permission condition is satisfied, and the process proceeds to step S1713.

[0220] In step S1713, the second determination logic 1504 determines fueling permission and proceeds to step S1734 in FIG. 17D.

[0221] On the other hand, in step S1712, if it is determined that the recognition result has not become a recognition result determined to be fueling permission within the predetermined time T2 (if NO in step S1712), the first fueling monitoring unit 1009_1 ends the process using the second determination logic 1504.

[0222] On the other hand, when proceeding from step S1704 to step S1721 in FIG. 17C, the second determination logic 1504 determines that a predetermined non - permission condition has been satisfied for a certain period of time. Specifically, the second determination logic 1504 determines that a vehicle is not recognized in the first area 210. Alternatively, the second determination logic 1504 determines that a fueler who does not hold the fuel nozzle in an appropriate posture (fueler recognition impossible) is recognized in the second area 220.

[0223] In step S1722, the second determination logic 1504 acquires warning information.

[0224] In step S1723, the second determination logic 1504 determines whether the currently input recognition result is a recognition result determined as refueling permission (whether it satisfies a predetermined permission condition). Specifically, it determines whether a vehicle is recognized in the first area 210 and whether a refueler who holds the fuel nozzle in an appropriate posture is recognized in the second area.

[0225] In step S1723, if it is determined that the recognition result is a recognition result determined as refueling permission (if YES in step S1723), the process proceeds to step S1724.

[0226] In step S1724, the second determination logic 1504 determines whether the recognition result determined as refueling permission has continued for a predetermined time T1. In step S1724, if it is determined that it has continued for the predetermined time T1 (if YES in step S1724), it is determined that the predetermined permission condition has been satisfied for a certain period of time, and the process proceeds to step S1725.

[0227] In step S1725, the second determination logic 1504 determines that refueling is permitted and proceeds to step S1734 in FIG. 17D.

[0228] On the other hand, in step S1723, if it is determined that the recognition result is not a recognition result determined as refueling permission (if NO in step S1723), the process proceeds to step S1726. Or, in step S1724, if it is determined that it has not continued for the predetermined time T1 (if NO in step S1724), the process proceeds to step S1726.

[0229] In step S1726, the second determination logic 1504 determines whether the state where the recognition result is not a recognition result determined as refueling permission has continued for a predetermined time T2 (where T2>T1).

[0230] In step S1726, if it is determined that the predetermined time T2 has not elapsed continuously (if NO in step S1726), the process returns to step S1723. On the other hand, in step S1726, if it is determined that the predetermined time T2 has elapsed continuously (if YES in step S1726), it is determined that a predetermined non - permission condition has been satisfied for a certain period of time, and the process proceeds to step S1731 in FIG. 17D.

[0231] In step S1731, the second determination logic 1504 determines non - fueling permission. After acquiring the second information, the process proceeds to step S1732.

[0232] In step S1732, the second determination logic 1504 determines whether the currently input recognition result has become a recognition result determined as fueling - permitted within the predetermined time T2 (whether a predetermined permission condition has been satisfied). Specifically, it is determined whether, within the predetermined time T2, a vehicle is recognized in the first area 210 and a fueler who holds the fuel nozzle in an appropriate posture is recognized in the second area.

[0233] In step S1732, if it is determined that the currently input recognition result has become a recognition result determined as fueling - permitted within the predetermined time T2 (if YES in step S1732), it is determined that a predetermined permission condition has been satisfied, and the process proceeds to step S1733.

[0234] In step S1733, the second determination logic 1504 determines fueling permission and proceeds to step S1734.

[0235] In step S1734, the determination logic switching unit 1502 determines whether it is during the second section. In step S1734, if it is determined that it is during the second section, it is determined that the process by the second determination logic 1504 is not terminated (if determined as NO in step S1734), and the process returns to step S1701 in FIG. 17A.

[0236] On the other hand, in step S1734, if it is determined that it is not during the second period, it is determined to end the process by the second determination logic 1504 (determine YES in step S1734). Then, the first fueling monitoring unit 1009_1 ends the process using the second determination logic 1504.

[0237] Also, in step S1732, if it is not determined that the recognition result is a fueling permission determination within the predetermined time T2 (if NO in step S1732), the first fueling monitoring unit 1009_1 ends the process using the second determination logic 1504.

[0238] (3-4) Flow of the process using the third determination logic 1505 of the first fueling monitoring unit 1009_1 Next, the flow of the process using the third determination logic 1505 of the first fueling monitoring unit 1009_1 will be described with reference to FIGS. 18A and 18B. FIGS. 18A and 18B are examples of the first and second flowcharts showing the flow of the process using the third determination logic of the first fueling monitoring unit.

[0239] In step S1801, the third determination logic 1505 determines whether the currently input recognition result is a recognition result determined to be fueling permission (whether a predetermined permission condition is satisfied). Specifically, it is determined whether a vehicle is recognized in the first area 210 and whether a fueler holding the fuel nozzle in an appropriate posture is recognized in the second area.

[0240] In step 1801, if it is determined that the recognition result is a recognition result determined to be fueling permission (if YES in step S1801), it is determined that the predetermined permission condition is satisfied, and the process proceeds to step S1802.

[0241] In step S1802, the third determination logic 1505 determines fueling permission and proceeds to step S1813 in FIG. 18B.

[0242] On the other hand, if it is determined in step S1801 that the recognition result is not a refueling permission determination (if NO in step S1801), the process proceeds to step S1803.

[0243] In step S1803, the third determination logic 1505 determines whether the state where the recognition result is not a refueling permission determination has continued for a predetermined time T5.

[0244] In step S1803, if it is determined that it has not continued for the predetermined time T5 (if NO in step S1803), the process returns to step S1801. On the other hand, in step S1803, if it is determined that it has continued for the predetermined time T5 (if YES in step S1803), the process proceeds to step S1804.

[0245] In step S1804, the third determination logic 1505 determines whether the currently input recognition result is a recognition result determined as refueling not permitted (whether a predetermined non - permission condition is satisfied). Specifically, it determines whether a vehicle is recognized in the first area 210 and whether a refueler who is not holding the fuel nozzle in an appropriate posture is recognized (whether the fuel nozzle is unrecognizable) in the second area.

[0246] In step S1804, if it is determined that the recognition result is another recognition result determined as refueling not permitted (if NO in step S1804), the process proceeds to step S1806.

[0247] In step S1806, the third determination logic 1505 determines that a predetermined non - permission condition has been satisfied for a certain period of time. Specifically, the third determination logic 1505 determines that no vehicle is recognized in the first area 210. Alternatively, the third determination logic 1505 determines that a refueler who is not holding the fuel nozzle in an appropriate posture is recognized (refueler unrecognizable) in the second area 220.

[0248] In step S1807, the third determination logic 1505 determines no refueling permission. After acquiring the second information, it proceeds to step S1811 in FIG. 18B.

[0249] On the other hand, in step S1804, if it is determined that the recognition result is a recognition result of no refueling permission (if YES in step S1804), it is determined that a predetermined non - permission condition has been satisfied for a certain period of time, and it proceeds to step S1805.

[0250] In step S1805, the third determination logic 1505 determines no refueling permission. After acquiring the first information, it proceeds to step S1811 in FIG. 18B.

[0251] In step S1811, the third determination logic 1505 determines whether the currently input recognition result has become a recognition result of refueling permission (whether a predetermined permission condition has been satisfied) within a predetermined time T2. Specifically, it determines whether within the predetermined time T2, a vehicle is recognized in the first area 210 and a fueler who holds the fuel nozzle in an appropriate posture is recognized in the second area.

[0252] In step S1811, if it is determined that the currently input recognition result has become a recognition result of refueling permission within the predetermined time T2 (if YES in step S1811), it is determined that a predetermined permission condition has been satisfied, and it proceeds to step S1812.

[0253] In step S1812, the third determination logic 1505 determines refueling permission and proceeds to step S1813.

[0254] In step S1813, the determination logic switching unit 1502 determines whether it is during the third section. In step S1813, if it is determined that it is during the third section, it is determined that the process by the third determination logic 1505 is not terminated (if determined NO in step S1813), and it returns to step S1801 in FIG. 18A.

[0255] On the other hand, in step S1813, when it is determined that it is not during the third period, it is determined to end the process by the third determination logic 1505 (determined as YES in step S1813). Then, the first fuel supply monitoring unit 1009_1 ends the process using the third determination logic 1505.

[0256] On the other hand, in step S1811, when it is determined that the currently input recognition result has not become a recognition result determined as fuel supply permission within the predetermined time T2 (when it is NO in step S1811), the process proceeds to step S1814.

[0257] In step S1814, the third determination logic 1505 determines whether a state where the recognition result is not determined as fuel supply permission has continued for the predetermined time T6. In step S1814, when it is determined that the state where the recognition result is not determined as fuel supply permission has not continued for the predetermined time T6 (when it is NO in step S1814), it waits until it continues for the predetermined time T6.

[0258] On the other hand, in step S1814, when it is determined that a state where the recognition result is not determined as fuel supply permission has continued for the predetermined time T6 (when it is YES in step S1814), the process proceeds to step S1815.

[0259] In step S1815, the third determination logic 1505 acquires confirmation request information, and the first fuel supply monitoring unit 1009_1 ends the process using the third determination logic 1505.

[0260] (3-5) Flow of processing using other determination logics of the first fuel supply monitoring unit 1009_1 Next, the flow of processing using other determination logics of the first fuel supply monitoring unit 1009_1 (processing using the poly-tank determination unit 1506 and processing using the fire determination unit 1507) will be described with reference to FIG. 19. FIG. 19 is an example of a flowchart showing the flow of processing using the poly-tank determination unit of the first fuel supply monitoring unit and the flow of processing using the fire determination unit.

[0261] First, the processing flow using the poly-tank determination unit 1506 of the first fuel supply monitoring unit 1009_1 will be described.

[0262] In step S1901 of FIG. 19(a), the poly-tank determination unit 1506 determines whether a container is included (whether a predetermined non-permission condition is satisfied) in the currently input recognition result. If it is determined in step S1901 that a container is included (if YES in step S1901), it is determined that the predetermined non-permission condition is satisfied, and the process proceeds to step S1902.

[0263] In step S1902, the poly-tank determination unit 1506 determines non-permission of fuel supply. After acquiring the third information, the process proceeds to step S1903.

[0264] On the other hand, if it is determined in step S1901 that a container is not included (if NO in step S1901), the process directly proceeds to step S1903.

[0265] In step S1903, the poly-tank determination unit 1506 determines whether to end the process. If it corresponds to any of the first section to the third section, in step S1903, the poly-tank determination unit 1506 determines to continue the process (determines NO in step S1903), and returns to step S1901.

[0266] On the other hand, if it does not correspond to any of the first section to the third section, in step S1903, the first fuel supply monitoring unit 1009_1 ends the process using the poly-tank determination unit 1506.

[0267] Subsequently, the processing flow using the poly-tank determination unit 1506 of the second fuel supply monitoring unit 1009_2 will be described.

[0268] In step S1911 of FIG. 19(b), the fire determination unit 1507 determines whether a temperature equal to or higher than a predetermined threshold has been measured in a region other than the first region 210 in the currently input thermal sensor data (whether a predetermined non-permissible condition is satisfied). Note that the reason for setting the region other than the first region 210 is to avoid misjudging fire detection because the bonnet of the vehicle included in the first region 210 may be at a high temperature.

[0269] If it is determined in step S1911 that a temperature equal to or higher than the predetermined threshold has been measured (if YES in step S1911), it is determined that the predetermined non-permissible condition is satisfied, and the process proceeds to step S1912.

[0270] In step S1912, the fire determination unit 1507 determines that refueling is not permitted, acquires the fourth information, and then proceeds to step S1913.

[0271] On the other hand, if it is determined in step S1911 that a temperature equal to or higher than the predetermined threshold has not been measured (if NO in step S1911), the process directly proceeds to step S1913.

[0272] In step S1913, the fire determination unit 1507 determines whether to end the process. If it corresponds to any of the first to third intervals, in step S1913, the fire determination unit 1507 determines to continue the process (determines NO in step S1913) and returns to step S1911.

[0273] On the other hand, if it does not correspond to any of the first to third intervals, in step S1913, the first fuel supply monitoring unit 1009_1 ends the process using the fire determination unit 1507.

[0274] (3-6) Flow of processing using the first determination logic 1513 of the second fuel supply monitoring unit 1009_2 Next, the flow of processing using the first determination logic 1513 of the second fuel supply monitoring unit 1009_2 will be described with reference to FIGS. 20A and 20B. FIGS. 20A and 20B are examples of first and second flowcharts showing the flow of processing using the first determination logic of the second fuel supply monitoring unit.

[0275] In step S2001, the first determination logic 1513 determines whether the currently input recognition result is a recognition result determined to be fuel supply permission (whether a predetermined permission condition is satisfied). Specifically, it is determined whether a container is recognized and whether a fueler who holds the fuel nozzle in an appropriate posture is recognized.

[0276] If it is determined in step S2001 that the recognition result is a recognition result determined to be fuel supply permission (if YES in step S2001), the process proceeds to step S2002.

[0277] In step S2002, the first determination logic 1513 determines whether the recognition result determined to be fuel supply permission has continued for a predetermined time T1 (whether a predetermined permission condition has been satisfied for a certain time). If it is determined in step S2002 that it has continued for the predetermined time T1 (if YES in step S2002), the process proceeds to step S2003.

[0278] In step S2003, the first determination logic 1513 determines fuel supply permission and proceeds to step S2014 in FIG. 20B.

[0279] On the other hand, if it is determined in step S2001 that the recognition result is not a recognition result determined to be fuel supply permission (if NO in step S2001), the process proceeds to step S2004. Or, if it is determined in step S2002 that it has not continued for the predetermined time T1 (if NO in step S2002), the process proceeds to step S2004.

[0280] In step S2004, the first determination logic 1513 determines whether a state where the recognition result is not determined to be fueling permission has continued for a predetermined time T2 (where T2 > T1).

[0281] In step S2004, if it is determined that the predetermined time T2 has not continued (if NO in step S2004), the process returns to step S2001. On the other hand, in step S2004, if it is determined that the predetermined time T2 has continued (if YES in step S2004), the process proceeds to step S2005.

[0282] In step S2005, the first determination logic 1513 determines whether the currently input recognition result is a recognition result determined to be fueling not permitted (whether a predetermined non - permission condition is satisfied). Specifically, it determines whether a fueler who has a recognized container and is not holding the fuel nozzle in an appropriate posture (is the fuel nozzle unrecognizable) is recognized.

[0283] In step S2005, if it is determined that the recognition result is a recognition result determined to be fueling not permitted (if YES in step S2005), it is determined that the predetermined non - permission condition has been satisfied for a certain period of time, and the process proceeds to step S2006.

[0284] In step S2006, the first determination logic 1513 determines that fueling is not permitted, acquires the first information, and then proceeds to step S2011 in FIG. 20B.

[0285] On the other hand, in step S2005, if it is determined that the recognition result is another recognition result determined to be fueling not permitted (if NO in step S2005), the process proceeds to step S2017 in FIG. 20B.

[0286] In step S2011, the first determination logic 1513 determines whether the currently input recognition result is a recognition result determined as fueling permission (whether it satisfies a predetermined permission condition). Specifically, it determines whether a container is recognized and whether a fueler who holds the fueling nozzle in an appropriate posture is recognized.

[0287] If it is determined in step S2011 that the recognition result is not a recognition result determined as fueling permission (if NO in step S2011), the process proceeds to step S2017.

[0288] On the other hand, if it is determined in step S2011 that the recognition result is a recognition result determined as fueling permission (if YES in step S2011), the second fueling monitoring unit 1009_2 ends the process using the first determination logic 1513.

[0289] In step S2012, the first determination logic 1513 determines whether the recognition result determined as fueling permission has continued for a predetermined time T1 (whether the predetermined permission condition has been satisfied for a certain time). If it is determined in step S2012 that it has continued for the predetermined time T1 (if YES in step S2012), the process proceeds to step S2013.

[0290] In step S2013, the first determination logic 1513 determines fueling permission and proceeds to step S2014.

[0291] In step S2014, the determination logic switching unit 1512 determines whether it is within the first section. If it is determined in step S2014 that it is within the first section, it is determined that the process by the first determination logic 1513 is not ended (determined as NO in step S2014), and the process returns to step S2001 in FIG. 20A.

[0292] On the other hand, in step S2014, when it is determined that it is not during the first period, it is determined to end the process by the first determination logic 1513 (determined as YES in step S2014). Then, the second fuel supply monitoring unit 1009_2 ends the process using the first determination logic 1513.

[0293] On the other hand, in step S2012, when it is determined that the predetermined time T1 has not continued (when NO in step S2012), the process proceeds to step S2015.

[0294] In step S2015, the first determination logic 1513 determines whether a state where the recognition result is not a fuel supply permission has continued for a predetermined time T3 (however, T3 > T1).

[0295] In step S2015, when it is determined that the continuation has not occurred for the predetermined time T3 (when NO in step S2015), the process returns to step S2011. On the other hand, in step S2015, when it is determined that the continuation has occurred for the predetermined time T3 (when YES in step S2015), it is determined that a predetermined non - permission condition has been satisfied for a certain period. Then, the second fuel supply monitoring unit 1009_2 ends the process using the first determination logic 1513.

[0296] On the other hand, when the process proceeds from step S2005 in FIG. 20A to step S2017 in FIG. 20B, the first determination logic 1513 determines that a predetermined non - permission condition has been satisfied for a certain period. Specifically, the first determination logic 1513 determines that the container has not been recognized. Alternatively, the first determination logic 1513 determines that a fueler who is not holding the fuel nozzle in an appropriate posture has been recognized (fueler recognition impossible).

[0297] In step S2018, the first determination logic 1513 determines non - fuel supply permission, acquires second information, and the second fuel supply monitoring unit 1009_2 ends the process using the first determination logic 1513.

[0298] (3-7) Flow of processing using the second determination logic 1514 of the second fuel supply monitoring unit 1009_2 Next, the flow of processing using the second determination logic 1514 of the second fuel supply monitoring unit 1009_2 will be described with reference to FIGS. 21A to 21D. FIGS. 21A to 21D are examples of the first to fourth flowcharts showing the flow of processing using the second determination logic of the second fuel supply monitoring unit.

[0299] In step S2101, the second determination logic 1514 determines whether the currently input recognition result is a recognition result determined to be fuel supply permitted (whether a predetermined permission condition is satisfied). Specifically, it is determined whether a container is recognized and whether a fueler holding the fuel nozzle in an appropriate posture is recognized.

[0300] If it is determined in step 2101 that the recognition result is a recognition result determined to be fuel supply permitted (YES in step S2101), it is determined that the predetermined permission condition is satisfied, and the process proceeds to step S2102.

[0301] In step S2102, the second determination logic 1514 determines that fuel supply is permitted and proceeds to step S2134 in FIG. 21D.

[0302] On the other hand, if it is determined in step S2101 that the recognition result is not a recognition result determined to be fuel supply permitted (NO in step S2101), the process proceeds to step S2103.

[0303] In step S2103, the second determination logic 1514 determines whether the state where the recognition result is not determined to be fuel supply permitted has continued for a predetermined time T4.

[0304] If it is determined in step S2103 that it has not continued for the predetermined time T4 (NO in step S2103), the process returns to step S2101. On the other hand, if it is determined in step S2103 that it has continued for the predetermined time T4 (YES in step S2103), the process proceeds to step S2104.

[0305] In step S2104, the second determination logic 1514 determines whether the currently input recognition result is a recognition result determined as refueling not permitted (whether a predetermined non-permission condition is satisfied). Specifically, it is determined whether a refueler who has a recognized container and does not hold the fuel nozzle in an appropriate posture is recognized (whether the fuel nozzle cannot be recognized).

[0306] In step S2104, if it is determined that the recognition result is another recognition result determined as refueling not permitted (if NO in step S2104), the process proceeds to step S2121 in FIG. 21C.

[0307] On the other hand, in step S2104, if it is determined that the recognition result is a recognition result determined as refueling not permitted (if YES in step S2104), it is determined that the predetermined non-permission condition has been satisfied for a certain period of time, and the process proceeds to step S2105.

[0308] In step S2105, the second determination logic 1514 acquires warning information.

[0309] In step S2106, the second determination logic 1514 determines whether the currently input recognition result is a recognition result determined as refueling permitted (whether a predetermined permission condition is satisfied). Specifically, it is determined whether a refueler who has a recognized container and holds the fuel nozzle in an appropriate posture is recognized.

[0310] In step S2106, if it is determined that the recognition result is a recognition result determined as refueling permitted (if YES in step S2106), the process proceeds to step S2107.

[0311] In step S2107, the second determination logic 1514 determines whether the recognition result determined as fueling permission has continued for a predetermined time T1. In step S2107, if it is determined that the predetermined time T1 has continued (if YES in step S2107), it is determined that the predetermined permission conditions have been satisfied for a certain period of time, and the process proceeds to step S2108.

[0312] In step S2108, the second determination logic 1514 determines that fueling is permitted and proceeds to step S2134 in FIG. 21D.

[0313] On the other hand, in step S2106, if it is determined that the recognition result is not the one determined as fueling permission (if NO in step S2106), the process proceeds to step S2109. Or, in step S2107, if it is determined that the predetermined time T1 has not continued (if NO in step S2107), the process proceeds to step S2109.

[0314] In step S2109, the second determination logic 1514 determines whether the state where the recognition result is not the one determined as fueling permission has continued for a predetermined time T2 (where T2 > T1).

[0315] In step S2109, if it is determined that the predetermined time T2 has not continued (if NO in step S2109), the process returns to step S2106. On the other hand, in step S2109, if it is determined that the predetermined time T2 has continued (if YES in step S2109), it is determined that the predetermined non - permission conditions have been satisfied for a certain period of time, and the process proceeds to step S2111 in FIG. 21B.

[0316] In step S2111, the second determination logic 1514 determines that fueling is not permitted, acquires the first information, and then proceeds to step S2112.

[0317] In step S2112, the second determination logic 1514 determines whether the currently input recognition result has become a recognition result determined as fueling permission within a predetermined time T2 (whether a predetermined permission condition is satisfied). Specifically, it determines whether, within the predetermined time T2, a container has been recognized and a fueler who holds the fuel nozzle in an appropriate posture has been recognized.

[0318] In step S2112, if it is determined that the currently input recognition result has become a recognition result determined as fueling permission within the predetermined time T2 (if YES in step S2112), it is determined that the predetermined permission condition is satisfied, and the process proceeds to step S2113.

[0319] In step S2113, the second determination logic 1514 determines fueling permission and proceeds to step S2134 in FIG. 21D.

[0320] On the other hand, in step S2112, if it is determined that the recognition result has not become a recognition result determined as fueling permission within the predetermined time T2 (if NO in step S2112), the second fueling monitoring unit 1009_2 ends the process using the second determination logic 1514.

[0321] On the other hand, when the process proceeds from step S2104 to step S2121 in FIG. 21C, the second determination logic 1514 determines that a predetermined non - permission condition has been satisfied for a certain period of time. Specifically, the second determination logic 1514 determines that the container has not been recognized. Alternatively, the second determination logic 1514 determines that a fueler who does not hold the fuel nozzle in an appropriate posture has been recognized (fueler recognition impossible).

[0322] In step S2122, the second determination logic 1504 acquires warning information.

[0323] In step S2123, the second determination logic 1514 determines whether the currently input recognition result is a recognition result determined as refueling permission (whether it satisfies a predetermined permission condition). Specifically, it determines whether a container has been recognized and whether a refueler who holds the fuel nozzle in an appropriate posture has been recognized.

[0324] In step S2123, if it is determined that the recognition result is a recognition result determined as refueling permission (if YES in step S2123), the process proceeds to step S2124.

[0325] In step S2124, the second determination logic 1514 determines whether the recognition result determined as refueling permission has continued for a predetermined time T1. In step S2124, if it is determined that it has continued for the predetermined time T1 (if YES in step S2124), it is determined that the predetermined permission condition has been satisfied for a certain period of time, and the process proceeds to step S2125.

[0326] In step S2125, the second determination logic 1514 determines that refueling is permitted and proceeds to step S2134 in FIG. 21D.

[0327] On the other hand, in step S2123, if it is determined that the recognition result is not a recognition result determined as refueling permission (if NO in step S2123), the process proceeds to step S2126. Or, in step S2124, if it is determined that it has not continued for the predetermined time T1 (if NO in step S2124), the process proceeds to step S2126.

[0328] In step S2126, the second determination logic 1514 determines whether the state where the recognition result is not a recognition result determined as refueling permission has continued for a predetermined time T2 (where T2 > T1).

[0329] In step S2126, if it is determined that the predetermined time T2 has not elapsed continuously (if NO in step S2126), the process returns to step S2123. On the other hand, in step S2126, if it is determined that the predetermined time T2 has elapsed continuously (if YES in step S2126), it is determined that the predetermined non - permission condition has been satisfied for a certain period of time, and the process proceeds to step S2131 in FIG. 21D.

[0330] In step S2131, the second determination logic 1514 determines non - fueling permission, and after acquiring the second information, proceeds to step S2132.

[0331] In step S2132, the second determination logic 1514 determines whether the currently input recognition result has become a recognition result determined as fueling - permitted within the predetermined time T2 (whether the predetermined permission condition has been satisfied). Specifically, it determines whether, within the predetermined time T2, the container has been recognized and a fueler who holds the fuel nozzle in an appropriate posture has been recognized.

[0332] In step S2132, if it is determined that the currently input recognition result has become a recognition result determined as fueling - permitted within the predetermined time T2 (if YES in step S2132), it is determined that the predetermined permission condition has been satisfied, and the process proceeds to step S2133.

[0333] In step S2133, the second determination logic 1514 determines fueling permission and proceeds to step S2134.

[0334] In step S2134, the determination logic switching unit 1512 determines whether it is within the second section. In step S2134, if it is determined that it is within the second section, it is determined that the process by the second determination logic 1514 is not terminated (if determined as NO in step S2134), and the process returns to step S2101 in FIG. 21A.

[0335] On the other hand, in step S2134, if it is determined that it is not during the second period, it is determined to end the process by the second determination logic 1514 (determined as YES in step S2134). Then, the second fuel supply monitoring unit 1009_2 ends the process using the second determination logic 1514.

[0336] Also, in step S2132, if it is not determined that the recognition result is a fuel supply permission determination within the predetermined time T2 (NO in step S2132), the second fuel supply monitoring unit 1009_2 ends the process using the second determination logic 1514.

[0337] (3-8) Flow of the process using the third determination logic 1515 of the second fuel supply monitoring unit 1009_2 Next, the flow of the process using the third determination logic 1515 of the second fuel supply monitoring unit 1009_2 will be described with reference to FIGS. 22A and 22B. FIGS. 22A and 22B are examples of the first and second flowcharts showing the flow of the process using the third determination logic of the second fuel supply monitoring unit.

[0338] In step S2201, the third determination logic 1515 determines whether the currently input recognition result is a recognition result for which fuel supply is permitted (whether a predetermined permission condition is satisfied). Specifically, it is determined whether a container is recognized and whether a fueler holding the fuel nozzle in an appropriate posture is recognized.

[0339] In step 2201, if it is determined that the recognition result is a recognition result for which fuel supply is permitted (YES in step S2201), it is determined that the predetermined permission condition is satisfied, and the process proceeds to step S2202.

[0340] In step S2202, the third determination logic 1515 determines that fuel supply is permitted and proceeds to step S2213 in FIG. 22B.

[0341] On the other hand, if it is determined in step S2201 that the recognition result is not a refueling permission determination (if NO in step S2201), the process proceeds to step S2203.

[0342] In step S2203, the third determination logic 1515 determines whether the state where the recognition result is not a refueling permission determination has continued for a predetermined time T5.

[0343] In step S2203, if it is determined that the continuation has not occurred for the predetermined time T5 (if NO in step S2203), the process returns to step S2201. On the other hand, in step S2203, if it is determined that the continuation has occurred for the predetermined time T5 (if YES in step S2203), the process proceeds to step S2204.

[0344] In step S2204, the third determination logic 1515 determines whether the currently input recognition result is a recognition result determined as refueling not permitted (whether a predetermined non - permission condition is satisfied). Specifically, it determines whether a container is recognized and whether a refueler who is not holding the fueling nozzle in an appropriate posture is recognized (whether the fueling nozzle is unrecognizable).

[0345] In step S2204, if it is determined that the recognition result is another recognition result determined as refueling not permitted (if NO in step S2204), the process proceeds to step S2206.

[0346] In step S2206, the third determination logic 1515 determines that a predetermined non - permission condition has been satisfied for a certain period of time. Specifically, the third determination logic 1515 determines that the container has not been recognized. Alternatively, the third determination logic 1515 determines that a refueler who is not holding the fueling nozzle in an appropriate posture has been recognized (the refueler is unrecognizable).

[0347] In step S2207, the third determination logic 1515 determines refueling not permitted, acquires the second information, and then proceeds to step S2211 in FIG. 22B.

[0348] On the other hand, in step S2204, when it is determined that the recognition result is a refueling non - permission determination (when YES in step S2204), it is determined that a predetermined non - permission condition has been satisfied for a certain period of time, and the process proceeds to step S2205.

[0349] In step S2205, the third determination logic 1515 determines refueling non - permission, acquires the first information, and then proceeds to step S2211 in FIG. 22B.

[0350] In step S2211, the third determination logic 1515 determines whether the currently input recognition result has become a recognition result for which refueling is permitted (whether a predetermined permission condition has been satisfied) within a predetermined time T2. Specifically, it is determined whether, within the predetermined time T2, a container has been recognized and a refueler who holds the fuel nozzle in an appropriate posture has been recognized.

[0351] In step S2211, when it is determined that the currently input recognition result has become a recognition result for which refueling is permitted within the predetermined time T2 (when YES in step S2211), it is determined that a predetermined permission condition has been satisfied, and the process proceeds to step S2212.

[0352] In step S2212, the third determination logic 1515 determines refueling permission and proceeds to step S2213.

[0353] In step S2213, the determination logic switching unit 1512 determines whether it is during the third period. In step S2213, when it is determined that it is during the third period, it is determined that the process by the third determination logic 1515 is not terminated (when determined as NO in step S2213), and the process returns to step S2201 in FIG. 22A.

[0354] On the other hand, in step S2213, if it is determined that it is not within the third period, it is determined to end the process by the third determination logic 1515 (determined as YES in step S2213). Then, the second fuel supply monitoring unit 1009_2 ends the process using the third determination logic 1515.

[0355] On the other hand, in step S2211, if it is determined that the currently input recognition result has not become a recognition result determined as fuel supply permission within the predetermined time T2 (if NO in step S2211), the process proceeds to step S2214.

[0356] In step S2214, the third determination logic 1515 determines whether the state of not being a recognition result determined as fuel supply permission has continued for the predetermined time T6. In step S2214, if it is determined that the state of not being a recognition result determined as fuel supply permission has not continued for the predetermined time T6 (if NO in step S2214), it waits until it continues for the predetermined time T6.

[0357] On the other hand, in step S2214, if it is determined that the state of not being a recognition result determined as fuel supply permission has continued for the predetermined time T6 (if YES in step S2214), the process proceeds to step S2215.

[0358] In step S2215, the third determination logic 1515 acquires confirmation request information, and the second fuel supply monitoring unit 1009_2 ends the process using the third determination logic 1515.

[0359] (3-9) Flow of processing using other determination logic of the second fuel supply monitoring unit 1009_2 Next, the flow of processing using other determination logic of the second fuel supply monitoring unit 1009_2 (processing using the fire determination unit 1517) will be described with reference to FIG. 23. FIG. 23 is an example of a flowchart showing the flow of processing using the fire determination unit of the second fuel supply monitoring unit.

[0360] In step S2301, the fire determination unit 1517 determines whether a temperature equal to or higher than a predetermined threshold has been measured for the currently input thermal sensor data (whether a predetermined non-permissible condition is satisfied).

[0361] In step S2301, if it is determined that a temperature equal to or higher than the predetermined threshold has been measured (if YES in step S2301), it is determined that the predetermined non-permissible condition is satisfied, and the process proceeds to step S2302.

[0362] In step S2302, the fire determination unit 1517 determines no fuel supply permission, and after acquiring the fourth information, the process proceeds to step S2303.

[0363] On the other hand, in step S2301, if it is determined that a temperature equal to or higher than the predetermined threshold has not been measured (if NO in step S2301), the process directly proceeds to step S2303.

[0364] In step S2303, the fire determination unit 1517 determines whether to end the process. If it corresponds to any of the first to third intervals, in step S2303, the fire determination unit 1517 determines to continue the process (determines NO in step S2303), and returns to step S2301.

[0365] On the other hand, if it does not correspond to any of the first to third intervals, in step S2303, the second fuel supply monitoring unit 1009_2 ends the process using the fire determination unit 1517.

[0366] (4) Details of the output control unit 1010 Next, as details of the output control unit 1010, the functional configuration and specific examples of the processing of the output control unit 1010 will be described.

[0367] (4-1) Details of the functional configuration of the output control unit 1010 FIG. 24 is a diagram showing an example of the functional configuration of the output control unit of the information processing apparatus. As shown in FIG. 24, the output control unit 1010 includes an output information acquisition unit 2401 and a transmission unit 2402.

[0368] The output information acquisition unit 2401 obtains from the first fueling monitoring unit 1009_1, · information indicating fueling permission, · information indicating fueling non - permission and the first to fourth information, · warning information, · confirmation request information, and acquires them.

[0369] Also, the output information acquisition unit 2401 obtains from the second fueling monitoring unit 1009_2, · information indicating fueling permission, · information indicating fueling non - permission and the first, second, and fourth information, · warning information, · confirmation request information, and acquires them.

[0370] Also, the output information acquisition unit 2401 notifies the information acquired from the first fueling monitoring unit 1009_1 to the transmission unit 2402. At this time, the output information acquisition unit 2401 refers to the output information storage unit 2403, · cause information corresponding to the first to fourth information (information indicating the cause of fueling non - permission), · messages corresponding to the first to fourth information, · display information corresponding to the first to fourth information, · messages corresponding to the warning information, · display information corresponding to the warning information, · cause information corresponding to the confirmation request information, · messages corresponding to the confirmation request information, · display information corresponding to the confirmation request information, and acquires them. Then, the output information acquisition unit 2401 converts the first to fourth information, warning information, and confirmation request information into the corresponding cause information, messages, and display information, and then notifies the transmission unit 2402.

[0371] Similarly, the output information acquisition unit 2401 notifies the information obtained from the second fueling monitoring unit 1009_2 to the transmission unit 2402. At this time, the output information acquisition unit 2401 refers to the output information storage unit 2403 and · Cause information corresponding to the first information, the second information, and the fourth information (information indicating the cause of fueling not permitted), and · Messages corresponding to the first information, the second information, and the fourth information, and · Display information corresponding to the first information, the second information, and the fourth information, and · Messages corresponding to the warning information, and · Display information corresponding to the warning information, and · Cause information corresponding to the confirmation request information, and · Messages corresponding to the confirmation request information, and · Display information corresponding to the confirmation request information, and acquires them. Then, the output information acquisition unit 2401 converts the first information, the second information, the fourth information, the warning information, and the confirmation request information into the corresponding cause information, messages, and display information, and then notifies the transmission unit 2402.

[0372] The transmission unit 2402 transmits the information notified from the output information acquisition unit 2401 to · The display device 912, which is a display device for the staff of the service station 110', · The voice output device 122_1 of the first fuel dispenser 120_1 or the voice output device 122_2 of the second fuel dispenser 120_2, · The fueling data display 123_1 of the first fuel dispenser 120_1 or the fueling data display 123_2 of the second fuel dispenser 120_2. Specifically, the transmission unit 2402 controls the information obtained from the first fueling monitoring unit 1009_1,

[0373] · Information indicating fueling permission, information indicating fueling not permitted, cause information, and confirmation request information to be displayed on the display device 912, · Messages to be voice-output by the voice output device 122_1 of the first fuel dispenser 120_1, · Messages to be voice-output by the voice output device 122_2 of the second fuel dispenser 120_2, · Control so that the indication information is displayed on the fueling data display 123_1 of the first fuel dispenser 120_1.

[0374] Similarly, the transmission unit 2402 controls so that the information acquired from the second fueling monitoring unit 1009_2, · Information indicating fueling permission, information indicating non - permission for fueling, cause information, and confirmation request information are controlled to be displayed on the display device 912, · Controls so that a message is output as voice by the voice output device 122_2 of the second fuel dispenser 120_2, · Controls so that the display information is displayed on the fueling data display 123_2 of the second fuel dispenser 120_2.

[0375] (4 - 2) Specific example of the processing of the output control unit 1010 Next, as a specific example of the processing of the output control unit 1010, specific examples of the cause information, message, and display information converted by the output control unit 1010 will be described. FIG. 25 is a diagram showing an example of the output information.

[0376] In FIG. 25, the output information 2500 includes cause information, a message, and display information corresponding to the first to fourth information, warning information, and confirmation request information acquired from the first fueling monitoring unit 1009_1. Also, the output information 2510 includes cause information, a message, and display information corresponding to the first information, second information, fourth information, warning information, and confirmation request information acquired from the second fueling monitoring unit 1009_2.

[0377] As shown in FIG. 25, the output information 2500 and 2510 include, as information items, "information type", "cause information", "message output to the customer", and "display information on the fueling data display of the fuel dispenser".

[0378] In the case of the output information 2500, any one of the first to fourth information is stored in the "information type", and in the case of the output information 2510, any one of the first information, second information, and fourth information is stored in the "information type".

[0379] In the case of output information 2500, any one of "nozzle indeterminate", "judgment impossible", "off-vehicle fueling", and "fire detection" is stored in the "cause information", and in the case of output information 2510, any one of "nozzle indeterminate", "judgment impossible", and "fire detection" is stored.

[0380] In the case of output information 2500, messages corresponding to the first to fourth information respectively, which are output as voice from the voice output device 122_1, are stored in the "message output to the customer". Also, in the case of output information 2510, messages corresponding to the first information, the second information, and the fourth information respectively, which are output as voice from the voice output device 122_2, are stored.

[0381] In the case of output information 2500, display information corresponding to the first to fourth information respectively, which is displayed on the fueling data display 123_1 of the first fuel dispenser 120_1, is stored in the "display information of the fueling data display of the fuel dispenser". Also, in the case of output information 2510, display information corresponding to the first information, the second information, and the fourth information respectively, which is displayed on the fueling data display 123_2 of the second fuel dispenser 120_2, is stored. Note that the display information displayed on the fueling data display 123_1 (also referred to as a simple monitor) of the first fuel dispenser 120_1 and the fueling data display 123_2 (also referred to as a simple monitor) of the second fuel dispenser 120_2 is not limited to that shown in the output information 2500 of FIG. 25. For example, image data captured by the camera 130_1 or the camera 130_2 may be displayed on the fueling data display 123_1 of the first fuel dispenser 120_1 and the fueling data display 123_2 of the second fuel dispenser 120_2.

[0382] <Flow of fueling control process by information processing device> Next, the flow of the fueling control process by the fueling system 110S' according to the first embodiment will be described. FIG. 26 is an example of a flowchart showing the flow of the fueling control process by the fueling system according to the first embodiment.

[0383] In step S2601, the information processing device 160 acquires the image data captured by the camera 130_1 and the camera 130_2.

[0384] In step S2602, the information processing apparatus 160 acquires signals from the first fuel supply sensor 921_1 and the second fuel supply sensor 921_2.

[0385] In step S2603, the information processing apparatus 160 acquires thermal sensor data measured by the thermal sensor 170_1 and the thermal sensor 170_2.

[0386] In step S2604, the information processing apparatus 160 determines which of the first section to the third section it corresponds to.

[0387] In step S2605, the information processing apparatus 160 determines whether to determine the fuel supply permission or not. In step S2605, if it is determined that it does not correspond to any of the first section to the third section, it is determined not to determine the fuel supply permission (determined as NO in step S2605), and the process proceeds to step S2610.

[0388] On the other hand, in step 2605, if it is determined that it corresponds to any of the first section to the third section, it is determined to determine the fuel supply permission (determined as YES in step S2605), and the process proceeds to step S2606.

[0389] In step S2606, the information processing apparatus 160 performs image recognition processing on the image data using the first learned model or the second learned model.

[0390] In step S2607, the information processing apparatus 160 determines the fuel supply permission or not from the recognition result using the determination logic corresponding to the determined section.

[0391] In step S2608, the information processing device 160 controls the fueling of the first fuel dispenser 120_1 and the second fuel dispenser 120_2 based on the determination result of whether fueling is permitted. Note that when the determination result is that fueling is not permitted, the information processing device 160 controls the prohibition or stop of fueling by the first fuel dispenser 120_1 or the second fuel dispenser 120_2 according to the cause information of the non-permission of fueling.

[0392] In step S2609, the information processing device 160 outputs the determination result of whether fueling is permitted. Further, when the determination result is that fueling is not permitted, the information processing device 160 outputs the cause information of the non-permission of fueling and the like.

[0393] In step S2610, the information processing device 160 determines whether to end the fueling control process. If it is determined in step S2610 that the fueling control process is to be continued (if NO in step S2610), the process returns to step S2601. On the other hand, if it is determined in step S2610 that the fueling control process is to be ended (if YES in step S2610), the fueling control process is ended.

[0394] <Summary> As is clear from the above description, the fueling system 110S' according to the first embodiment divides the period from when the fueling nozzle is removed from the nozzle holder until it is returned to the nozzle holder into three sections, and determines whether fueling is permitted using the determination logic corresponding to each section. Thus, according to the first embodiment, it is possible to provide a fueling system capable of executing processing according to the fueling scene.

[0395] Further, the fueling system 110S' according to the first embodiment is configured to acquire the cause information of the non-permission of fueling and output a message corresponding to the cause information as voice when determining non-permission of fueling based on the recognition result regarding the recognition target. Thus, according to the first embodiment, it is possible to provide a fueling system capable of executing processing according to the fueling scene.

[0396] In addition, the fuel supply system 110S' according to the first embodiment is configured to use a learned model according to the type of fuel supply when performing image recognition processing on the recognition target. Thus, according to the first embodiment, it is possible to provide a fuel supply system capable of executing processing according to the fuel supply scene.

[0397] [Second Embodiment] In the above first embodiment, the information processing apparatus 160 has been described as performing fuel supply control for the first fuel dispenser 120_1, the second fuel dispenser 120_2, and the like. However, the fuel supply control for the first fuel dispenser 120_1, the second fuel dispenser 120_2, and the like may be configured to be performed by a device separate from the information processing apparatus 160 (for example, a fuel supply controller such as a self-service console provided in a service station). Hereinafter, the second embodiment will be described centering on the differences from the first embodiment.

[0398] <Functional Configuration of Information Processing Apparatus> FIG. 27 is a diagram showing another example of the functional configuration of the information processing apparatus. The difference from the functional configuration of the information processing apparatus 160 shown in FIG. 10 is that, in the case of FIG. 27, the functions of the first fuel supply monitoring unit 1009_1' and the second fuel supply monitoring unit 1009_2' are different from the functions of the first fuel supply monitoring unit 1009_1 and the second fuel supply monitoring unit 1009_2. Further, the difference from the functional configuration of the information processing apparatus 160 shown in FIG. 10 is that, in the case of FIG. 27, the first fuel supply monitoring unit 1009_1' and the second fuel supply monitoring unit 1009_2' are communicably connected to the fuel supply controller 2710.

[0399] The fuel supply controller 2710 is a self-service console (SSC), and controls the fuel supply of the first fuel dispenser 120_1 based on the determination result of whether fuel supply is permitted transmitted from the first fuel supply monitoring unit 1009_1'. Further, the fuel supply controller 2710 controls the fuel supply of the second fuel dispenser 120_2 based on the determination result of whether fuel supply is permitted transmitted from the second fuel supply monitoring unit 1009_2'.

[0400] <Details of First Fuel Supply Monitoring Unit and Second Fuel Supply Monitoring Unit> Next, the details of the first fuel supply monitoring unit 1009_1' and the second fuel supply monitoring unit 1009_2' will be described. FIG. 28A is a diagram showing another example of the functional configuration of the first fuel supply monitoring unit of the information processing apparatus. In the above first embodiment, the difference from the functional configuration of the first fuel supply monitoring unit 1009_1 of the information processing apparatus 160 described with reference to FIG. 15A is that the first fuel supply monitoring unit 1009_1' of the information processing apparatus 160' in FIG. 28A has a transmission unit 2801. Further, in the case of the first fuel supply monitoring unit 1009_1' of the information processing apparatus 160' in FIG. 28A, the transmission unit 2801 is communicably connected to the fuel supply controller 2710.

[0401] The transmission unit 2801 transmits the determination result of fuel supply permission or fuel supply non - permission to the fuel supply controller 2710. Thereby, the fuel supply controller 2710 controls the fuel supply for the first fuel supply type by controlling the fuel supply valve 124_1 or the fuel supply pump 125_1.

[0402] For example, when the fuel supply permission is transmitted from the transmission unit 2801, the fuel supply controller 2710 drives the fuel supply pump 125_1 to open the fuel supply valve 124_1. Also, the fuel supply controller 2710 · When the fuel supply permission from the poly - tank determination unit 1506 is transmitted from the transmission unit 2801, the fuel supply controller 2710 drives the fuel supply pump 125_1 to open the fuel supply valve 124_1. Further, · When the fuel supply non - permission from any of the first determination logic 1503 to the third determination logic 1505 is transmitted from the transmission unit 2801, or · When the fuel supply non - permission from the fire determination unit 1507 is transmitted from the transmission unit 2801, it stops the fuel supply by stopping the drive of the fuel supply pump 125_1, or prohibits the fuel supply by stopping the drive of the fuel supply pump 125_1 and closing the fuel supply valve 124_1. Note that the definitions of fuel supply stop and fuel supply prohibition here are only examples, and the fuel supply stop and fuel supply prohibition may be defined by other definitions. Alternatively, without distinguishing between fuel supply stop and fuel supply prohibition, the entire state where fuel supply cannot be performed may be referred to as fuel supply stop or prohibition.

[0403] Further, the transmission unit 2801 notifies the fueling permission determination result and the fueling non - permission determination result to the cause information acquisition unit 1509. When notifying fueling non - permission, the transmission unit 2801 also notifies the cause information acquisition unit 1509 of the information indicating the cause determined as fueling non - permission.

[0404] In addition, when any one of the first determination logic 1503 to the third determination logic 1505 is unable to perform the determination of fueling permission or non - permission, the transmission unit 2801 may be configured to output "determination impossible" to the cause information acquisition unit 1509. When "determination impossible" is output from the transmission unit 2801, the cause information acquisition unit 1509 may display "determination impossible" on, for example, the display device 912 which is a display device for the staff of the service station 110'. According to such a configuration, even when the determination of fueling permission or non - permission cannot be performed, the staff of the service station 110' can operate the fueling controller 2710 to control fueling.

[0405] Also, the transmission unit 2801 may be configured to monitor the communication status with the fueling controller 2710 and output "communication abnormality" to the cause information acquisition unit 1509 when a communication failure occurs. When "communication abnormality" is output from the transmission unit 2801, the cause information acquisition unit 1509 may display "communication abnormality" on, for example, the display device 912 which is a display device for the staff of the service station 110'. According to such a configuration, even when the determination result of fueling permission or non - permission cannot be transmitted to the fueling controller 2710, the staff of the service station 110' can operate the fueling controller 2710 to control fueling.

[0406] FIG. 28B is a diagram showing an example of the functional configuration of the second fueling monitoring unit of the information processing device. Since the functional configuration of the second fueling monitoring unit 1009_2' shown in FIG. 28B is the same as the functional configuration of the first fueling monitoring unit 1009_1' shown in FIG. 28A, detailed description is omitted here.

[0407] <Summary> As is clear from the above description, in the second embodiment, the functions of the information processing apparatus 160 in the first embodiment are realized by the information processing apparatus 160' and the fuel supply controller 2710. Thus, even when the information processing apparatus 160' and the fuel supply controller 2710 are configured separately, according to the second embodiment, the same effects as those in the first embodiment can be achieved.

[0408] [Third Embodiment] In the first embodiment, the case where the period from when the fuel supply nozzle is removed from the nozzle holder until it is returned to the nozzle holder is divided into three sections was described. However, the number of sections to be divided is not limited to three, and may be four or more.

[0409] Also, in the first embodiment, it was described that one determination logic is associated with each section. However, the relationship between the number of sections and the number of determination logics is not limited to this. For example, one determination logic may be associated with two consecutive sections.

[0410] Also, in the first embodiment, the case where there are two types of fuel supply was described, but the number of types of fuel supply may be three or more. Note that when there are three or more types of fuel supply, three or more learned models are also generated.

[0411] Also, in the first embodiment, for the first fuel supply type, the recognition targets were a vehicle, a container, a fuel supplier, a fuel supply nozzle, etc. However, the recognition targets for the first fuel supply type are not limited to these.

[0412] Similarly, in the first embodiment, for the second fuel supply type, the recognition targets were a container, a fuel supplier, a fuel supply nozzle, etc. However, the recognition targets for the second fuel supply type are not limited to these.

[0413] In the first embodiment described above, the first to third determination logics for the first fuel supply type were described with reference to FIGS. 16A to 18B. However, the processing contents of the first to third determination logics are not limited to these.

[0414] Similarly, in the first embodiment described above, the first to third determination logics for the second fuel supply type were described with reference to FIGS. 20A to 22B. However, the processing contents of the first to third determination logics are not limited to these.

[0415] In the first embodiment described above, the processing after displaying the cause information, for example, on the display device 912, which is a display device for the staff of the service station 110', was not mentioned. However, after displaying the cause information, it may be switched to a manual operation by the staff.

[0416] In the first embodiment described above, the number of executions of the learning process was not mentioned. However, the learning process may be executed multiple times. For example, in the image recognition process using the first learned model and the second learned model generated by the first learning process, if sufficient recognition accuracy cannot be obtained, or if the recognition accuracy decreases, additional learning processes may be configured to be performed.

[0417] For example, the learning device 150 may be configured to perform additional learning processes by collecting image data and signals indicating the states of the fuel nozzles again from a plurality of service stations and generating learning data.

[0418] Note that the learning device 150 may be configured to be able to collect image data and signals indicating the states of the fuel nozzles at any timing from a plurality of service stations. Also, a device that monitors the recognition accuracy, analyzes the cause when the recognition accuracy decreases, and notifies the learning device 150 may be connected to the network 180.

[0419] In addition, in the above-described first embodiment, the learning device 150 and the information processing device 160 are configured as separate bodies, but the learning device 150 and the information processing device 160 may be configured as one body.

[0420] [Fourth Embodiment] In the above-described first embodiment, it has been described that there is no difference in the ease or difficulty of determination when it is determined that refueling is permitted between the first section to the third section. Further, in the above-described first embodiment, it has been described that there is no difference in the ease or difficulty of determination when it is determined that refueling is not permitted between the first section to the third section. However, a difference may be provided in the ease or difficulty of each determination between the first section to the third section. In this way, by executing processing according to the refueling scene, it becomes possible to reduce the frequency with which the refueler calls the staff of the service station 110'.

[0421] In the fourth embodiment, among these, a case will be described in which a difference is provided in the ease of determination when it is determined that refueling is permitted, or the difficulty of determination when it is determined that refueling is not permitted, between the first section and the second and third sections. As a method for making it easier to determine that refueling is permitted, or a method for making it more difficult to determine that refueling is not permitted, for example, the following methods can be mentioned. · When the image recognition unit 1005 performs image recognition processing, a method of lowering the likelihood threshold when outputting a recognition result determined to permit refueling, or a method of raising the likelihood threshold when outputting a recognition result determined not to permit refueling, · As the model parameters of the first learned model used when the image recognition unit 1005 performs image recognition processing, a method of using model parameters that easily output a recognition result determined to permit refueling, or a method of using model parameters that hardly output a recognition result determined not to permit refueling, · When the second determination logic 1504 determines whether refueling is permitted or not, A method of setting a short time from when a recognition result determined as fueling permission is obtained until fueling permission is determined, or A method of setting a long time from when a recognition result determined as fueling non - permission is obtained until fueling non - permission is determined, · When the second determination logic 1504 determines fueling permission or non - permission, A method of identifying the cause of a recognition result determined as fueling non - permission and configuring not to determine fueling non - permission for a predetermined cause.

[0422] In the fourth embodiment, for each method, the differences from the first embodiment will be mainly described.

[0423] <Method for Realizing in Image Recognition Processing Part 1> First, a method of providing a difference in the ease of determination when fueling permission is determined or the difficulty of determination when fueling non - permission is determined by switching the likelihood threshold between the first section (before fueling start) and the second and third sections (after fueling start) will be described. FIG. 29 is a diagram showing another example of the functional configuration of the image recognition unit of the information processing device. The difference from FIG. 13 is that the functions of the first learned model 2902 and the second learned model 2912 are different from the functions of the first learned model 1302 and the second learned model 1312 in FIG. 13.

[0424] Specifically, when the output layer of the first learned model 2902 outputs a recognition result determined as fueling permission (a fueler holding the fuel nozzle in an appropriate posture), · Threshold value 1a is used as the first threshold value of likelihood during the first section, · Threshold value 1b is used as the first threshold value of likelihood during the second or third section, respectively (when the likelihood is greater than or equal to the first threshold value, an output is made that a fueler holding the fuel nozzle in an appropriate posture has been recognized). However, it is assumed that the first threshold values (threshold value 1a and threshold value 1b) have a relationship of threshold value 1a > threshold value 1b.

[0425] In addition, when the output layer of the first learned model 2902 outputs a recognition result (a fueler who is not holding the fuel nozzle in an appropriate posture) determined as fueling not permitted, · If it is within the first section, threshold value 2a is used as the second threshold value of the likelihood, · If it is within the second or third section, threshold value 2b is used as the second threshold value of the likelihood, and each is used (when the likelihood is equal to or greater than the second threshold value, an output is made of a recognition result that a fueler who is not holding the fuel nozzle in an appropriate posture has been recognized). However, it is assumed that the second threshold values (threshold value 2a and threshold value 2b) are in the relationship of threshold value 2a < threshold value 2b.

[0426] In addition, in the fourth embodiment, recognizing a fueler who holds the fuel nozzle in an appropriate posture includes recognizing a fueler who holds the fuel nozzle inserted into the fuel filling port in an appropriate posture. Also, recognizing a fueler who does not hold the fuel nozzle in an appropriate posture includes · Recognizing a fueler who holds a fuel nozzle not inserted into the fuel filling port, · Recognizing a fueler whose fuel nozzle is hidden behind the fueler and thus it cannot be determined that the fuel nozzle is being held, · Recognizing a fueler who holds the fuel nozzle inserted into the fuel filling port in an inappropriate posture, and any of these is included.

[0427] Similarly, when the output layer of the second learned model 2912 outputs a recognition result (a fueler who holds the fuel nozzle in an appropriate posture) determined as fueling permitted, · If it is within the first section, threshold value 1a is used as the first threshold value of the likelihood, · If it is within the second or third section, threshold value 1b is used as the first threshold value of the likelihood, and each is used (when the likelihood is equal to or greater than the first threshold value, an output is made of a recognition result that a fueler who holds the fuel nozzle in an appropriate posture has been recognized). However, it is assumed that the first threshold values (threshold value 1a and threshold value 1b) are in the relationship of threshold value 1a > threshold value 1b.

[0428] In addition, when the output layer of the second pre-trained model 2912 outputs a recognition result (a fueler who is not holding the fuel nozzle in an appropriate posture) determined as fueling not permitted, · If it is within the first section, threshold value 2a is used as the second threshold value of the likelihood, · If it is within the second or third section, threshold value 2b is used as the second threshold value of the likelihood, and each is used (when the likelihood is equal to or greater than the second threshold value, an output is made of a recognition result that a fueler who is not holding the fuel nozzle in an appropriate posture has been recognized). However, it is assumed that the second threshold values (threshold value 2a and threshold value 2b) are in the relationship of threshold value 2a < threshold value 2b.

[0429] In addition, in the fourth embodiment, recognizing a fueler who holds the fuel nozzle in an appropriate posture includes recognizing a fueler who holds the fuel nozzle inserted into the filling port of the poly tank in an appropriate posture. Also, recognizing a fueler who does not hold the fuel nozzle in an appropriate posture includes · Recognizing a fueler who holds a fuel nozzle in a state where it is not inserted into the filling port of the poly tank, · Recognizing a fueler for whom it cannot be determined that the fuel nozzle is being held because the fuel nozzle is hidden behind the fueler, · Recognizing a fueler who holds the fuel nozzle inserted into the filling port of the poly tank in an inappropriate posture, and any one of them is included.

[0430] As described above, according to the fourth embodiment, by relaxing the conditions for outputting the determination result of fueling permission, it becomes easier to determine fueling permission in the second and third sections (after fueling starts) than in the first section (before fueling starts). As a result, for example, even if it becomes difficult to recognize that the fuel nozzle has been inserted into the fueling port (or the filling port of the poly tank) after fueling starts, it is possible to determine fueling permission.

[0431] Also, in the fourth embodiment, by strengthening the conditions for outputting the determination result of refueling non-permission, it becomes less likely to be determined as refueling non-permission in the second and third intervals (after refueling starts) than in the first interval (before refueling starts). As a result, for example, even if the fueling nozzle is hidden behind the fueler after refueling starts, it will not be determined as refueling non-permission.

[0432] As a result, according to the fourth embodiment, after refueling starts, in a scene where there is no urgency to stop refueling, the frequency of refueling stop can be suppressed, and it becomes possible to reduce the frequency of the fueler calling the staff of the service station 110'.

[0433] A specific example will be described. FIG. 30 is a diagram showing a specific example of processing when switching the likelihood threshold in the image recognition process. Here, as learning data, two types of image data shown by reference numeral 3010 are respectively · "annotation data indicating that the fueler is holding the fueling nozzle in an appropriate posture" and, · "annotation data indicating that the fueler is not holding the fueling nozzle in an appropriate posture" are assumed to be attached.

[0434] When a first learned model is generated by performing learning processing using the learning data, · For the target image data shown by reference numeral 3011, when recognized as a fueler holding the fueling nozzle in an appropriate posture, the likelihood is calculated as "60%" (see reference numeral 3012). · For the target image data shown by reference numeral 3021, when recognized as a fueler not holding the fueling nozzle in an appropriate posture, the likelihood is calculated as "75%" (see reference numeral 3022).

[0435] Here, it is assumed that "70%" is set in the first interval (before refueling starts) as the threshold for outputting the recognition result that the fueler is recognized as holding the fueling nozzle in an appropriate posture. Also, it is assumed that "50%" is set in the second interval (after refueling starts) (see reference numerals 3013_1 and 3013_2).

[0436] In this case, even if the target image data shown by reference numeral 3011 is extracted during the first period, a recognition result indicating that a fueler who holds the fuel nozzle in an appropriate posture has been recognized is not output. On the other hand, when the target image data shown by reference numeral 3011 is extracted during the second period, a recognition result indicating that a fueler who holds the fuel nozzle in an appropriate posture has been recognized is output, and it is determined that fueling is permitted.

[0437] Also, assume that "70%" is set for the first period (before fueling starts) as a threshold value for outputting a recognition result indicating that a fueler who does not hold the fuel nozzle in an appropriate posture has been recognized. Also, assume that "80%" is set for the second period (after fueling starts) (see reference numerals 3023_1 and 3023_2).

[0438] In this case, when the target image data shown by reference numeral 3021 is extracted during the first period, a recognition result indicating that a fueler who does not hold the fuel nozzle in an appropriate posture has been recognized is output, and it is determined that fueling is not permitted. On the other hand, even if the target image data shown by reference numeral 3021 is extracted during the second period, a recognition result indicating that a fueler who does not hold the fuel nozzle in an appropriate posture has been recognized is not output.

[0439] <Method for Realizing in Image Recognition Processing Part 2> Next, a method for providing a difference in the ease of determination when it is determined that fueling is permitted or the difficulty of determination when it is determined that fueling is not permitted by switching model parameters between the first period (before fueling starts) and the second and third periods (after fueling starts) will be described.

[0440] As described in the first embodiment above, in generating the first learned model, first learning data is used. The first learning data includes image data to which annotation data indicating that the fueler is a fueler who holds the fuel nozzle in an appropriate posture is attached. The first learning data also includes image data to which annotation data indicating that the fueler is a fueler who does not hold the fuel nozzle in an appropriate posture is attached.

[0441] In the fourth embodiment, the image recognition unit 1005 has the model parameters of each of the first learned models obtained using different first learning data.

[0442] (1) In the case of refueling permission The first learned model used when the image recognition unit 1005 performs image recognition processing in the first section (before refueling start) is generated using the first learning data for the first section. The first learned model used when the image recognition unit 1005 performs image recognition processing in the second and third sections (after refueling start) is generated using the first learning data for the second and third sections.

[0443] The first learning data for the second and third sections is composed of image data with more variations than the first learning data for the first section, as annotation data indicating that the person is a fueler holding the fuel nozzle in an appropriate posture is attached to the image data.

[0444] In this way, by relaxing the conditions for outputting the determination result of refueling permission, it becomes easier to determine that refueling is permitted in the second section (after refueling start) than in the first section (before refueling start).

[0445] Here, the first learned model has been described, but the same applies to the second learned model.

[0446] (2) In the case of refueling not permitted The first learned model used when the image recognition unit 1005 performs image recognition processing in the first section (before refueling start) is generated using the first learning data for the first section. The first learned model used when the image recognition unit 1005 performs image recognition processing in the second and third sections (after refueling start) is generated using the first learning data for the second and third sections.

[0447] The first learning data for the second and third intervals is composed of image data with less variation than the first learning data for the first interval, as image data to which annotation data indicating that the fueler is not holding the fuel nozzle in an appropriate posture is added.

[0448] In this way, by strengthening the conditions for outputting the determination result of non-permission to refuel, it becomes less likely to be determined as non-permission to refuel in the second interval (after the start of refueling) than in the first interval (before the start of refueling).

[0449] Here, although the first learned model has been described, the same applies to the second learned model.

[0450] (3) Specific example FIG. 31 is a diagram showing a specific example of the process when switching model parameters in the image recognition process. Here, as the learning data for the second interval, to the two types of image data shown by reference numeral 3110, respectively, · "Annotation data indicating that the fueler is holding the fuel nozzle in an appropriate posture" and, · "Annotation data indicating that the fueler is not holding the fuel nozzle in an appropriate posture" are assumed to be added.

[0451] When the learning process is performed using the learning data for the second interval and the first learned model for the second interval is generated, · For the target image data shown by reference numeral 3111, when recognized as a fueler holding the fuel nozzle in an appropriate posture, the likelihood is calculated as "100%" (see reference numeral 3112). · For the target image data shown by reference numeral 3121, when recognized as a fueler not holding the fuel nozzle in an appropriate posture, the likelihood is calculated as "60%" (see reference numeral 3122).

[0452] Here, as a threshold value for outputting a recognition result that a refueler who holds the fuel nozzle in an appropriate posture is recognized, it is assumed that "70%" is set in the second section (after the start of refueling) (see reference numeral 3113).

[0453] In this case, when the target image data shown by reference numeral 3111 is extracted during the second section, a recognition result that a refueler who holds the fuel nozzle in an appropriate posture is recognized is output, and it is determined that refueling is permitted.

[0454] Also, as a threshold value for outputting a recognition result that a refueler who does not hold the fuel nozzle in an appropriate posture is recognized, it is assumed that "70%" is set in the second section (before the start of refueling) (see reference numeral 3114).

[0455] In this case, even if the target image data shown by reference numeral 3121 is extracted during the second section, a recognition result that a refueler who does not hold the fuel nozzle in an appropriate posture is recognized is not output.

[0456] <Method for realizing in the determination logic, Part 1> Next, a method will be described in which a difference is provided in the ease of determination when it is determined that refueling is permitted or the difficulty of determination when it is determined that refueling is not permitted by switching the time required for determination between the first section (before the start of refueling) and the second and third sections (after the start of refueling).

[0457] (1) In the case of permitting refueling In the above-described first embodiment, as described with reference to FIGS. 16A and 16B, when determining that refueling is permitted in step S1603 or step S1612 of the first determination logic, which is the determination logic of the first section (before the start of refueling), · The vehicle is recognized in the first area, and · A state in which a refueler who holds the fuel nozzle in an appropriate posture is recognized in the second area, are conditions that continue for a predetermined time T1 (equal to or greater than the third threshold value).

[0458] Similarly, in the above-described first embodiment, when determining that refueling is permitted in step S1708 of the second determination logic, which is the determination logic for the second period (after the start of refueling), as described with reference to FIGS. 17A to 17D, · The vehicle is recognized in the first area, and · A refueler who holds the fuel nozzle in an appropriate posture is recognized in the second area, are required to continue for a predetermined time T1 (equal to or greater than the third threshold).

[0459] In contrast, in the fourth embodiment, for example, the predetermined time T1, which is a condition for determining that refueling is permitted in step S1708 of the second determination logic, is switched to T1'. However, it is assumed that T1 (the third threshold before switching) > T1' (the third threshold after switching).

[0460] By relaxing the conditions for outputting the determination result of refueling permission in this way, it becomes easier to determine that refueling is permitted in the second period (after the start of refueling) than in the first period (before the start of refueling).

[0461] Although the first determination unit has been described here, the same applies to the second determination unit.

[0462] (2) In the case of refueling not permitted In the above-described first embodiment, when determining that refueling is not permitted in step S1606 of the first determination logic, which is the determination logic for the first period (before the start of refueling), as described with reference to FIGS. 16A and 16B, · The vehicle is recognized in the first area, and · A refueler who does not hold the fuel nozzle in an appropriate posture is recognized in the second area, are required to continue for a predetermined time T2 (equal to or greater than the fourth threshold).

[0463] Similarly, in the first embodiment, as described with reference to FIGS. 17A to 17D, when determining no refueling permission in step S1711 or step S1731 of the second determination logic, which is the determination logic for the second period (after refueling starts), · Recognize the vehicle in the first area and, · In a state where a refueler who does not hold the fuel nozzle in an appropriate posture in the second area is recognized, it is conditioned that this has continued for a predetermined time T2 (equal to or greater than the fourth threshold).

[0464] In contrast, in the fourth embodiment, for example, the predetermined time T2, which is a condition for determining no refueling permission in step S1711 or step S1731 of the second determination logic, is switched to T2'. However, it is assumed that T2 (the fourth threshold before switching) < T2' (the fourth threshold after switching).

[0465] In this way, by strengthening the conditions for outputting the determination result of no refueling permission, it becomes less likely to determine no refueling permission in the second period (after refueling starts) than in the first period (before refueling starts).

[0466] Here, the first determination unit has been described, but the same applies to the second determination unit.

[0467] <Method for realizing in the determination logic, part 2> Next, a method will be described in which the second determination logic 1504 identifies the cause of the recognition result for which no refueling permission is determined, and for a predetermined cause, is configured not to determine no refueling permission.

[0468] As described above, to recognize a refueler who does not hold the fuel nozzle in an appropriate posture, · Recognize a refueler who holds the fuel nozzle in a state where it is not inserted into the fuel filler opening, · Recognize a refueler for whom it cannot be determined that the fuel nozzle is being held because the fuel nozzle is hidden behind the refueler, Recognizing a fueler who holds the fuel nozzle in an inappropriate posture while the fuel nozzle is inserted into the fuel inlet includes any of the following

[0469] In the first determination logic 1503 executed before the start of fueling by the fueler, when the image recognition unit 1005 outputs a recognition result indicating that it has recognized a fueler who is not holding the fuel nozzle in an appropriate posture, it is determined not to permit fueling based on the recognition result.

[0470] On the other hand, in the second determination logic 1504 executed after the start of fueling by the fueler, when the image recognition unit 1005 outputs a recognition result indicating that it has recognized a fueler who is not holding the fuel nozzle in an appropriate posture, the cause of the recognition result is specified. Also, in the second determination logic 1504, when the cause is that the fuel nozzle cannot be determined to be held because it is hidden behind the fueler, it is not determined not to permit fueling based on the recognition result.

[0471] In this way, the cause of the recognition result determined not to permit fueling is specified, and for a predetermined cause, it is configured not to determine not to permit fueling (that is, the conditions for outputting a determination result not to permit fueling are strengthened). As a result, it becomes less likely to be determined not to permit fueling in the second section (after the start of fueling) than in the first section (before the start of fueling).

[0472] Here, the first determination unit has been described, but the same applies to the second determination unit.

[0473] Subsequently, a specific example will be described. FIG. 32 is a diagram showing a specific example of the process when switching the determination method based on the cause of the recognition result.

[0474] Here, as learning data, to the two types of image data shown by reference numeral 3210, respectively · "Annotation data indicating that the fueler is holding the fuel nozzle in an appropriate posture" and · "Annotation data indicating that the fueler is not holding the fuel nozzle in an appropriate posture" and shall be provided with.

[0475] When a first learned model is generated by performing learning processing using the learning data, · For the target image data indicated by reference sign 3211, when it is recognized as a fueler not holding the fuel nozzle in an appropriate posture, the likelihood is calculated as "100%" (see reference sign 3222).

[0476] Also, assume that "70%" is set in the first section (before fueling starts) as a threshold value for outputting a recognition result indicating that a fueler not holding the fuel nozzle in an appropriate posture has been recognized. Also, assume that "70%" is set in the second section (after fueling starts) (see reference signs 3213 and 3223).

[0477] In this case, when the target image data indicated by reference sign 3211 is extracted during the first section, a recognition result indicating that a fueler not holding the fuel nozzle in an appropriate posture has been recognized is output. Similarly, when the target image data indicated by reference sign 3211 is extracted during the second section, a recognition result indicating that a fueler not holding the fuel nozzle in an appropriate posture has been recognized is output (see reference signs 3214 and 3224).

[0478] Here, assume that the first determination logic 1503 is configured to determine no fueling permission when a recognition result indicating that a fueler not holding the fuel nozzle in an appropriate posture has been recognized is output. In this case, in the first section, a determination result of no fueling permission will be output (see reference sign 3215).

[0479] On the other hand, in the case of the fourth embodiment, the second determination logic 1504 · Even when a recognition result indicating that a fueler not holding the fuel nozzle in an appropriate posture has been recognized, · When it is a recognition result indicating that a fueler whose fuel nozzle cannot be determined to be held because the fuel nozzle is hidden behind the fueler has been recognized, It is configured not to determine that refueling is not permitted. Therefore, the second determination logic 1504 does not output a determination result of not permitting refueling (reference numeral 3225).

[0480] <Summary> As is clear from the above description, the refueling system 110S' according to the fourth embodiment has · an image recognition unit that recognizes a fueler who holds the fuel nozzle in an appropriate posture or a fueler who does not hold the fuel nozzle in an appropriate posture from the image data captured by the camera. · a determination unit that determines whether refueling is permitted based on the recognition result. · When refueling by the fueler is started, the image recognition unit or the determination unit relaxes the conditions for outputting a determination result of permitting refueling, or strengthens the conditions for outputting a determination result of not permitting refueling.

[0481] Accordingly, according to the fourth embodiment, it is possible to execute processing according to the refueling scene, and it is possible to reduce the frequency with which the fueler calls the staff of the service station 110'.

[0482] [Fifth Embodiment] In the first embodiment described above, the first determination unit and the second determination unit are configured to notify a message prompting improvement in the fueling behavior of the fueler before determining that refueling is not permitted when the fueling behavior of the fueler is included in the second section. However, the configuration of notifying a message prompting improvement in the fueling behavior of the fueler before determining that refueling is not permitted is not limited to the case where the fueling behavior of the fueler is included in the second section. For example, even when the fueling behavior of the fueler is included in the first section, it may be configured to notify a message prompting improvement in the fueling behavior of the fueler before determining that refueling is not permitted. This makes it possible to reduce the frequency with which the fueler calls the staff of the service station 110'. Hereinafter, the fifth embodiment will be described centering on the differences from the first embodiment.

[0483] <Flow of processing using the first determination logic 1503 of the first fuel supply monitoring unit 1009_1> FIGS. 33A and 33B are another example of the first and second flowcharts showing the flow of processing using the first determination logic of the first fuel supply monitoring unit. In the above-described first embodiment, the differences from the flowcharts described with reference to FIGS. 16A and 16B are steps S3301 and S3311.

[0484] In step S3301, the first determination logic 1503 acquires warning information. Also, in step S3311, the first determination logic 1503 determines whether the recognition result is a determination of fuel supply permission, and if it is determined that the recognition result is not a determination of fuel supply permission (if NO in step S3311), the process proceeds to step S1615.

[0485] In step S1615, the first determination logic 1503 determines whether the state where the recognition result is not a determination of fuel supply permission has continued for a predetermined time T3 (however, T3 > T1).

[0486] In step S1615, if it is determined that the continuation has not occurred for the predetermined time T3 (if NO in step S1615), the process returns to step S1611. On the other hand, in step S1615, if it is determined that the continuation has occurred for the predetermined time T3 (if YES in step S1615), it is determined that a predetermined non-permission condition has been satisfied for a certain period of time, and the process proceeds to step S3312.

[0487] In step S3312, the first determination logic 1503 determines non-permission of fuel supply and acquires second information. Thereafter, the first fuel supply monitoring unit 1009_1 ends the processing using the first determination logic 1503.

[0488] As a result, the first fuel supply monitoring unit 1009_1 in the fifth embodiment · The state where the recognition result determined as fuel supply permission is not output continues for a predetermined time T2 (NO in step S1601, YES in step S1604), and · When it is determined that the reason why the recognition result determined as refueling permission is not output is due to recognizing a refueler who does not hold the fuel nozzle in an appropriate posture in the second area (YES in step S1605), immediately do not set it as refueling not permitted, and can notify the warning information (a message prompting improvement regarding the refueling behavior of the refueler) to the notification device (the voice output device 122_1 and the fueling data display 123_1). Also, the first fueling monitoring unit 1009_1 in the fifth embodiment, · Even after the warning information is notified, when the state where the recognition result determined as refueling permission is not output continues for a predetermined time T3 (NO in step S3311, YES in step S1615), determine that refueling is not permitted (step S3312). In this case, the refueler waits until the staff responds.

[0489] Here, the first determination unit has been described, but the same applies to the second determination unit.

[0490] <Summary> As is clear from the above description, the fueling system 110S' according to the fifth embodiment notifies a message prompting improvement regarding the fueling behavior of the refueler even when the fueling behavior of the refueler is included in the first section, before determining that refueling is not permitted. Thus, according to the fifth embodiment, it becomes possible to execute processing according to the fueling scene, and it also becomes possible to reduce the frequency with which the refueler calls the staff of the service station 110'.

[0491] [Sixth Embodiment] In the above-described fourth and fifth embodiments, a configuration for reducing the frequency with which a fueler calls the staff at the service station 110' by executing processing according to the fueling scene has been described. On the other hand, even if the frequency of calling the staff at the service station 110' is reduced, there is still an assumed case where the staff at the service station 110' cannot immediately respond to the fueler's call. In the sixth embodiment, processing for cases where the staff at the service station 110' cannot immediately respond to the fueler's call (referred to as processing 1 and 2 for non-corresponding cases) will be described.

[0492] <Processing 1 for Non-Corresponding Cases> First, processing 1 for non-corresponding cases will be described. In processing 1 for non-corresponding cases, in the second determination logic 1504, · It is determined that fueling is not permitted because the fueling nozzle is hidden behind the fueler, · The staff at the service station 110' is called, · Before the staff at the service station 110' responds, a recognition result that determines fueling permission is output, In this case, the second determination logic 1504 re-determines that fueling is permitted.

[0493] Figs. 34A to 34D are another example of the first to fourth flowcharts showing the flow of processing using the second determination logic of the first fueling monitoring unit. In the above-described first embodiment, the differences from the flowcharts shown in Figs. 17A to 17D are steps S3401 to S3402, steps S3411 to S3414, step S3415, steps S3421 to S3422, and step S3431.

[0494] Among these, steps S3401 to S3402, step S3415, steps S3421 to S3422, and step S3431 are · Those indicating that the predetermined time T1, which is the condition for determining fueling permission described in the above-described fourth embodiment, is switched to T1', or ·Indicates that the predetermined time T2, which is the condition for determining no refueling permission described in the above fourth embodiment, is switched to T2'. Therefore, the description is omitted here.

[0495] In step S3411, after the second determination logic 1504 determines no refueling permission and acquires the second information, it proceeds to step S3412. In this case, the refueler waits until the staff responds.

[0496] In step S3412, the second determination logic 1504 determines whether the reason for determining no refueling permission in step S3411 is due to the fueling nozzle being hidden behind the refueler.

[0497] In step S3412, if it is determined that the reason for determining no refueling permission in step S3411 is due to the fueling nozzle being hidden behind the refueler (YES in step S3412), it proceeds to step S3413. On the other hand, in step S3412, if it is determined that the reason for determining no refueling permission in step S3411 is not due to the fueling nozzle being hidden behind the refueler (NO in step S3412), it proceeds to step S3415.

[0498] In step S3413, the second determination logic 1504 determines whether it has recognized a refueler who holds the fueling nozzle in an appropriate posture in the second area before the staff responds.

[0499] In step S3413, if it is determined that it has not recognized a refueler who holds the fueling nozzle in an appropriate posture in the second area (NO in step S3413), the process using the second determination logic ends.

[0500] On the other hand, in step S3413, if it is determined that the refueler who holds the fuel nozzle in an appropriate posture in the second area is recognized (if YES in step S3413), the process proceeds to step S3414.

[0501] In step S3414, the second determination logic 1504 determines that refueling is permitted and proceeds to step S1734 in FIG. 34D.

[0502] <Processing 2 for non - applicable cases> Next, processing 2 for non - applicable cases will be described. In the fueling control process by the information processing device 160 in processing 2 for non - applicable cases, · Since it is determined that refueling is not permitted, fueling is stopped or prohibited, · Information corresponding to the cause information is output, · The staff of the service station 110' is called, · It is determined that the staff of the service station 110' could not respond within a predetermined time, In this case, the information processing device 160 stops the fueling control process (that is, stops the process of automatically determining fueling permission and non - permission).

[0503] FIG. 35 is an example of a flowchart showing the flow of the fueling control process by the fueling system according to the sixth embodiment. The difference from the flowchart shown in FIG. 26 is step S3501.

[0504] In step S3501, the information processing device 160 determines whether the staff of the service station 110' has responded within a predetermined time. In step S3501, if it is determined that the staff of the service station 110' has responded within a predetermined time (if YES in step S3501), the process proceeds to step S2610.

[0505] On the other hand, in step S3501, if it is determined that the staff of the service station 110' cannot respond within a predetermined time (if NO in step S3501), the fueling control process is terminated. As a result, hereinafter, the information processing device 160 stops using the determination result by the first determination unit or the second determination unit.

[0506] <Summary> As is clear from the above description, the fueling system 110S' according to the sixth embodiment · When recognizing a fueler who cannot be determined to be holding the fueling nozzle because the fueling nozzle is hidden behind the fueler, and · When recognizing a fueler who holds the fueling nozzle in an appropriate posture before the staff of the service station 110' responds after being notified that the fueling is not permitted Determine that fueling is permitted.

[0507] Also, the fueling system 110S' according to the sixth embodiment · When no response is made by the staff of the service station 110' within a predetermined time after being notified that the fueling is not permitted, stop using the determination result by the first determination unit or the second determination unit.

[0508] Thereby, according to the sixth embodiment, it becomes possible to handle a case where the staff of the service station 110' cannot immediately respond to the call of the fueler.

[0509] [Seventh Embodiment] In the first embodiment described above, various recognition results are exemplified as the recognition result for which fueling is not permitted. However, the recognition result for which fueling is not permitted is not limited to those exemplified in the above embodiments. For example, when the image recognition unit 1005 recognizes a plurality of fuelers based on the images captured by the cameras 130_1 and 130_2, the first determination unit or the second determination unit may be configured to determine that fueling is not permitted.

[0510] Accordingly, for example, in a scene where there are a plurality of fuelers around one fuel dispenser, the first fueling monitoring unit or the second fueling monitoring unit can determine that fueling is not permitted.

[0511] Also, in the above first embodiment, the fire determination unit 1507 is configured to determine whether a temperature equal to or higher than a threshold value has been measured in a region other than the first region 210 in order to avoid misjudging the high-temperature hood of a vehicle included in the first region 210 as fire detection. However, the configuration for avoiding misjudging the high-temperature part of the vehicle as fire detection is not limited to this.

[0512] FIG. 36 is another example of a flowchart showing the flow of processing using the fire determination unit. The differences from the flowchart shown in FIG. 19(b) are steps S3601, S3602, and S3603.

[0513] In step S3601, the fire determination unit 1507 determines whether a temperature equal to or higher than a threshold value Te1 (equal to or higher than the fifth threshold value) has been measured outside the circumscribed rectangle (bounding box) of the vehicle for the currently input thermal sensor data. The threshold value Te1 is, for example, 100°C.

[0514] In step S3601, if it is determined that a temperature equal to or higher than the threshold value Te1 has been measured (YES in step S3601), it is determined that a predetermined non-permission condition is satisfied, and the process proceeds to step S1912.

[0515] On the other hand, in step S3601, if it is determined that a temperature equal to or higher than the threshold value Te1 has not been measured (NO in step S3601), the process proceeds to step S3602.

[0516] In step S3602, the fire determination unit 1507 determines whether a temperature equal to or higher than a threshold Te2 (equal to or higher than the sixth threshold) has been measured within the circumscribed rectangle (bounding box) of the vehicle for the currently input thermal sensor data. Assume that Te2 (the sixth threshold) > Te1 (the fifth threshold), and Te2 is set to a temperature higher than the temperature of the high-temperature hood or muffler of the vehicle. The threshold Te2 is, for example, 300°C.

[0517] In step S3602, if it is determined that a temperature equal to or higher than the threshold Te2 has been measured (if YES in step S3602), it is determined that a predetermined non-permission condition has been satisfied, and the process proceeds to step S3603.

[0518] In step S3603, the fire determination unit 1507 determines that refueling is not permitted. After acquiring the fourth information, the process proceeds to step S1913.

[0519] In step S3602, if it is determined that a temperature equal to or higher than the threshold Te2 has not been measured (if NO in step S3602), the process directly proceeds to step S1913.

[0520] In this way, when determining fire detection, by changing the threshold for determining non-permission to refuel outside and inside the circumscribed rectangle of the vehicle (that is, outside and inside the area of the vehicle), · It is possible to avoid misjudging the high-temperature part of the vehicle as fire detection. · It is possible to detect a vehicle fire.

[0521] Note that the area of the vehicle may be specified in the form of a polygon (that is, a circumscribed polygon) instead of the circumscribed rectangle.

[0522] Also, here, when determining fire detection, a configuration was described in which the threshold for determining no refueling permission is changed between outside the circumscribed rectangle of the vehicle and inside the circumscribed rectangle of the vehicle (that is, between outside the area of the vehicle and inside the area of the vehicle). However, when determining fire detection, it may be configured to change the threshold for determining no refueling permission between outside the circumscribed rectangle of the refueler and inside the circumscribed rectangle of the refueler (that is, between outside the area of the refueler and inside the area of the refueler). Specifically, the fire determination unit 1507 · When it is determined that the thermal data is equal to or higher than the threshold Te3 (equal to or higher than the seventh threshold) in the area outside the area of the refueler, it is determined that there is fire detection. · When it is determined that the thermal data is equal to or higher than the threshold Te4 (equal to or higher than the eighth threshold) in the area of the refueler, it is determined that there is fire detection.

[0523] At this time, the threshold Te4 is set lower than the threshold Te3 (Te3 (seventh threshold) > Te4 (eighth threshold)). Specifically, the threshold Te3 is 100 °C, and the threshold Te4 is 60 °C. The threshold Te3 may be the same value as or different from the threshold Te1. Thereby, · It is possible to detect small fires (for example, lighters, cigarettes, etc.) that the refueler has without missing them.

[0524] Note that the area of the refueler may be the area of a specific part of the refueler. The area of a specific part of the refueler refers to, for example, the area of the upper body of the refueler, the area of the head of the refueler, the area of the fingertips of the refueler, etc. Also, the area of the refueler (or the area of a specific part of the refueler) may be specified in the form of a polygon (that is, a circumscribed polygon) instead of a circumscribed rectangle.

[0525] Also, when the area of the refueler or the area of a specific part of the refueler overlaps with the area of the vehicle, the threshold of the area of the refueler may be prioritized, and when it is equal to or higher than the threshold Te4 (equal to or higher than the eighth threshold), it may be determined that there is fire detection.

[0526] Alternatively, when the area of the fueler or the area of a specific part of the fueler overlaps with the area of the vehicle, a new threshold value Te5 (the ninth threshold value) may be set, and when it is equal to or higher than the threshold value Te5 (equal to or higher than the ninth threshold value), it may be determined that a fire has been detected. Here, the new threshold value Te5 (the ninth threshold value) may be, for example, equal to or higher than the threshold value Te4 (equal to or higher than the eighth threshold value) and less than the threshold value Te1 (less than the fifth threshold value). Specifically, the threshold value Te5 may be a value higher than 60°C and lower than 100°C.

[0527] When the threshold value Te5 is set to a value higher than 60°C and lower than 100°C (for example, when set to 80°C), there are the following advantages. · The fact that the area of the fueler or the area of a specific part of the fueler overlaps with the area of the vehicle means that there is a fueler near the vehicle. And the fact that there is a fueler near the vehicle means that the area of the vehicle is not extremely hot (for example, 80°C or higher). For this reason, the area of the vehicle will not be misjudged as detecting a fire. · Since it is lower than the threshold value Te3 set in an area other than the area of the fueler or an area other than the area of a specific part of the fueler, it is possible to detect a small fire (for example, a lighter, a cigarette, etc.) that the fueler has without missing it.

[0528] Also, in the above first embodiment, the message output by the output control unit 1010 is configured to be divided according to cause information (see the output information 2500 and 2510 in FIG. 25). However, the message output by the output control unit 1010 may be further configured to be divided by section. Also, in the above fifth embodiment, when the refueling act of the fueler is included in the first section or the second section, before determining that refueling is not permitted, a message prompting improvement regarding the refueling act of the fueler is notified. However, even when the refueling act of the fueler is included in the third section, before determining that refueling is not permitted, a message prompting improvement regarding the refueling act of the fueler may be notified.

[0529] That is, the information processing device 160 is · Regardless of whether the refueling operation of the refueler is included in any of the first to third intervals, before determining refueling not permitted, a message prompting improvement for the refueling operation of the refueler is notified, and · It may be configured to change the message prompting improvement depending on whether the refueling operation of the refueler is included in any of the first to third intervals.

[0530] FIG. 37 is a diagram showing another example of output information. In FIG. 37, only the message notified to the refueler who refuels the first refueling type is described, and the description of the message notified to the refueler who refuels the second refueling type is omitted, but the same applies to the case of the second refueling type. Also, in FIG. 37, for reasons of the layout of the drawing, the description of "display information of the simple monitor of the fuel dispenser" is omitted, but "display information of the simple monitor of the fuel dispenser" may be the same regardless of the interval, similar to FIG. 25. Furthermore, in FIG. 37, even in the case where a plurality of the above-described refuelers are recognized, the cause information is additionally noted and the corresponding message is described.

[0531] As shown in the output information 3700, in the present embodiment, regardless of whether the refueling operation of the refueler is included in any of the first to third intervals, when the "cause information" is "nozzle indeterminate", before determining refueling not permitted, a message prompting improvement is notified. However, as shown in the output information 3700, the message notified to the refueler is different depending on whether the refueling operation of the refueler is included in the first interval, the second interval, or the third interval. Specifically, when the refueling operation of the refueler is included in the first interval, a message premised on calling the staff is notified. On the other hand, when the refueling operation of the refueler is included in the second or third interval, a message premised on not calling the staff is notified.

[0532] By configuring to change the message prompting improvement between the first interval before the start of refueling and the second and third intervals after the start of refueling in this way, it becomes possible to reduce the frequency with which the refueler calls the staff of the service station 110'.

[0533] Note that the configurations and the like described in the above embodiments are not limited to the configurations shown here, such as combinations with other elements. Regarding these points, it is possible to make changes without departing from the spirit of the present invention, and it can be appropriately determined according to the application form.

Explanation of Reference Numerals

[0534] 110, 111: Service Station 110S': Fueling System 120_1: First Fueling Machine 120_2: Second Fueling Machine 121_1, 121_2: Fueling Nozzle 122_1, 122_2: Voice Output Device 123_1, 123_3: Fueling Data Display 124_1, 124_2: Fueling Valve 125_1, 125_2: Fueling Pump 130_1, 130_2: Camera 170_1, 170_2: Thermal Sensor 150: Learning Device 160: Information Processing Device 160': Information Processing Device 210: First Region 220: Second Region 211: Vehicle 221: Fueler 231: Container 241: Fueler 500A: Learning Data Generation Unit 500B: Learning Unit 600: First Learning Data 700: Second Learning Data 912: Display Device 921_1: First Fueling Sensor 921_2: Second Fueling Sensor 1001: First Fuel Type Image Data Acquisition Unit 1002: Second Fuel Type Image Data Acquisition Unit 1003: First Fueling Sensor Data Acquisition Unit 1004: Second fuel supply sensor data acquisition unit 1005: Image recognition unit 1006, 1007: Section determination unit 1009_1: First fuel supply monitoring unit 1009_1': First fuel supply monitoring unit 1009_2: Second fuel supply monitoring unit 1009_2': Second fuel supply monitoring unit 1010: Output control unit 1302: First trained model 1312: Second trained model 1503, 1513: First determination logic 1504, 1514: Second determination logic 1505, 1515: Third determination logic 2710: Fuel supply controller 2801, 2811: Transmission unit 2902: First trained model 2912: Second trained model

Claims

1. A fuel tanker and A camera that photographs an area including the tanker; A refueling system having an information processing device, The information processing device includes: an image recognition unit that recognizes, from image data captured by the camera, a fuel dispenser holding a fuel dispenser nozzle in an appropriate position or a fuel dispenser not holding a fuel dispenser nozzle in an appropriate position; A determination unit that determines whether or not refueling is permitted based on the recognition result, the image recognition unit or the determination unit relaxes the conditions for outputting a determination result of permission for refueling, or strengthens the conditions for outputting a determination result of non-permission for refueling, when refueling by a refueler is started; Refueling system.

2. To recognize a fuel dispenser holding the fuel dispenser nozzle in an appropriate posture, at least The method includes recognizing a fuel dispenser who holds a fuel dispenser nozzle inserted into a fuel dispenser opening in an appropriate posture, To recognize a fuel dispenser who is not holding the fuel dispenser nozzle in an appropriate position, at least To recognize a fuel dispenser holding a fuel dispenser nozzle that is not inserted into a fuel filler opening; To recognize a fuel dispenser whose grip on a fuel dispenser nozzle cannot be determined because the fuel dispenser nozzle is hidden behind the fuel dispenser; and recognizing a fuel dispenser who holds a fuel dispenser nozzle inserted into a fuel dispenser opening in an inappropriate position. The fueling system of claim 1 .

3. The determination unit is If the fuel dispenser nozzle is hidden behind the fuel dispenser before the fuel dispenser starts to dispense fuel, it is determined that the fuel dispenser is not permitted to dispense fuel; When the fuel dispenser nozzle is hidden behind the fuel dispenser after the fuel dispenser starts filling the tank, the fuel dispenser does not determine that the fuel dispenser is not permitted to fill the tank.

3. The fueling system of claim 2.

4. The image recognition unit includes: outputting a recognition result indicating that a fuel dispenser holding the fuel dispenser nozzle in an appropriate posture has been recognized when a likelihood of the trained model recognizing the fuel dispenser as holding the fuel dispenser nozzle in an appropriate posture is equal to or greater than a first threshold value; When refueling by a refueler is started, the first threshold value is switched to a value lower than that before the start of refueling.

3. The fueling system of claim 2.

5. The image recognition unit includes: outputting a recognition result indicating that a fuel dispenser not holding the fuel dispenser nozzle in an appropriate position has been recognized when the likelihood of the trained model recognizing the fuel dispenser as not holding the fuel dispenser nozzle in an appropriate position is equal to or greater than a second threshold value; When refueling by a refueler is started, the second threshold value is switched to a value higher than that before the start of refueling.

3. The fueling system of claim 2.

6. The image recognition unit includes: The system has model parameters of trained models that have been trained using different training data to recognize a fuel dispenser holding the fuel dispenser nozzle in an appropriate posture, When refueling by a refueler is started, a model parameter of the trained model is switched.

3. The fueling system of claim 2.

7. The image recognition unit includes: The system has model parameters of trained models that have been trained using different training data to recognize a fuel dispenser that is not holding the fuel dispenser nozzle in an appropriate position, When refueling by a refueler is started, a model parameter of the trained model is switched.

3. The fueling system of claim 2.

8. The determination unit is When the image recognition unit recognizes a fuel dispenser that is not holding the fuel dispenser nozzle in an appropriate position before the fuel dispenser starts to dispense fuel, the image recognition unit determines that fuel dispense is not permitted, When the image recognition unit recognizes a fuel dispenser who is not holding the fuel dispenser nozzle in an appropriate posture after the fuel dispenser starts to dispense fuel, and when the image recognition unit recognizes a fuel dispenser who is not holding the fuel dispenser nozzle in an appropriate posture because the fuel dispenser nozzle is hidden behind the fuel dispenser, the image recognition unit does not determine that the fuel dispenser is being held.

3. The fueling system of claim 2.

9. The determination unit is When a recognition result that a fuel dispenser holding the fuel dispenser nozzle in an appropriate posture is recognized continues for a time period equal to or longer than a third threshold, it is determined that refueling is permitted; when refueling by a refueler is started, the third threshold value is switched to a time shorter than that before the start of refueling; The fueling system of claim 1 .

10. The determination unit is When a recognition result that a fuel dispenser is not holding the fuel dispenser nozzle in an appropriate position continues for a time period equal to or greater than a fourth threshold, it is determined that fuel dispenser is not permitted; When refueling by a refueler is started, the fourth threshold value is switched to a time longer than that before the start of refueling. The fueling system of claim 1 .

11. If the staff does not respond within a predetermined time after the determination unit notifies the staff that refueling is not permitted, use of the determination result by the determination unit is suspended. The fueling system of claim 1 .

12. When the image recognition unit recognizes a fuel dispenser in which it cannot be determined that the fuel dispenser is holding the fuel dispenser nozzle because the fuel dispenser nozzle is hidden behind the fuel dispenser, and when the image recognition unit recognizes a fuel dispenser holding the fuel dispenser nozzle in an appropriate posture after the determination unit notifies a staff member that refueling is not permitted and before the staff member takes any action, the determination unit determines that refueling is permitted.

3. The fueling system of claim 2.

13. Further comprising a thermal sensor; The determination unit is When it is determined that the temperature measured by the thermal sensor is equal to or higher than a fifth threshold value in an area other than an area where a vehicle is recognized, it is determined that a fire has been detected; When it is determined that the temperature measured by the thermal sensor in the area where the vehicle is recognized is equal to or higher than a sixth threshold value that is higher than the fifth threshold value, it is determined that a fire has been detected. The fueling system of claim 1 .

14. Further comprising a thermal sensor; The determination unit is When it is determined that the temperature measured by the thermal sensor is equal to or higher than a seventh threshold value in an area other than the area where the fueler is recognized or in an area other than the area where a specific part of the fueler is recognized, it is determined that a fire has been detected; When it is determined that the temperature measured by the thermal sensor in the area where the fueler is recognized or in the area where a specific part of the fueler is recognized is equal to or higher than an eighth threshold value that is lower than the seventh threshold value, it is determined that a fire has been detected. The fueling system of claim 1 .

15. The determination unit is When it is determined that the temperature measured by the thermal sensor is equal to or higher than a seventh threshold value in an area other than the area where the fueler is recognized or in an area other than the area where a specific part of the fueler is recognized, it is determined that a fire has been detected; When it is determined that the temperature measured by the thermal sensor in the area where the fueler is recognized or in the area where a specific part of the fueler is recognized is equal to or higher than an eighth threshold value that is lower than the seventh threshold value, it is determined that a fire has been detected; When it is determined that the temperature measured by the thermal sensor in an area where the fueler is recognized or an area where a specific part of the fueler is recognized overlaps with an area where the vehicle is recognized, a fire is detected; The ninth threshold value is higher than the eighth threshold value and lower than the fifth threshold value.

14. The fueling system of claim 13.

16. When the image recognition unit recognizes a plurality of refueling persons based on the image data captured by the camera, the determination unit determines that refueling is not permitted. The fueling system of claim 1 .

17. The fuel supply system further includes an alarm device for notifying a fuel supplier of a message, When the refueling action of the refueler is included in the first or second section of a first section before the start of refueling, a second section during refueling, and a third section after the end of refueling, the notification device issues a message encouraging the refueler to improve the refueling action before the determination unit determines that refueling is not permitted. The fueling system of claim 1 .

18. an image recognition unit that recognizes, from image data captured by the camera, a fuel dispenser who is holding the fuel dispenser nozzle in an appropriate position or a fuel dispenser who is not holding the fuel dispenser nozzle in an appropriate position; A determination unit that determines whether or not refueling is permitted based on the recognition result, the image recognition unit or the determination unit relaxes the conditions for outputting a determination result of permission for refueling, or strengthens the conditions for outputting a determination result of non-permission for refueling, when refueling by a refueler is started; Information processing device.

19. an image recognition step of recognizing a fuel dispenser who is holding the fuel dispenser nozzle in an appropriate position or a fuel dispenser who is not holding the fuel dispenser nozzle in an appropriate position from image data captured by the camera; a determination step of determining whether or not refueling is permitted based on the recognition result, the image recognition step or the determination step relaxes the conditions for outputting a determination result of permission for refueling, or strengthens the conditions for outputting a determination result of non-permission for refueling, when refueling by a refueler is started. Refueling control method.

20. an image recognition step of recognizing a fuel dispenser who is holding the fuel dispenser nozzle in an appropriate position or a fuel dispenser who is not holding the fuel dispenser nozzle in an appropriate position from image data captured by the camera; and a determination step of determining whether or not refueling is permitted based on the recognition result. the image recognition step or the determination step relaxes the conditions for outputting a determination result of permission for refueling, or strengthens the conditions for outputting a determination result of non-permission for refueling, when refueling by a refueler is started. Fuel supply control program.

Images