Refueling system, information processing device, refueling control method, and refueling control program

Abstract

To provide a refueling system capable of executing processing according to a refueling scene. [Solution] A refueling system having a refueling machine and an information processing device, in which the information processing device judges the state of the refueling nozzle and the state of refueling based on notification from the refueling machine, and if it is determined that refueling is not permitted based on judgment logic depending on which of the following three sections the refueling person's refueling action falls within: a first section before refueling begins, a second section during refueling, or a third section after refueling is completed, or based on judgment logic depending on which two consecutive sections the refueling action falls within, the information processing device prohibits or stops refueling by the refueling machine.

Description

[Technical field]

[0001] The present disclosure relates to a refueling system, an information processing device, a refueling control method, and a refueling control program. [Background technology]

[0002] Development is underway for a refueling system aimed at service stations that monitors the refueling actions of customers and uses AI and other technologies to determine whether or not to allow refueling. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-83377 A Summary of the Invention [Problem to be solved by the invention]

[0004] On the other hand, there are various refueling situations at service stations, and when developing a refueling system, it is desirable to have a configuration in which the method of determining whether or not refueling is permitted and the method of outputting the determination result can be changed according to the refueling situation.

[0005] One aspect of the present invention is to provide a refueling system capable of executing processing appropriate to a refueling scene. [Means for solving the problem]

[0006] According to one aspect, a fueling system includes: A fuel tanker and A camera that photographs an area including the tanker; A refueling system having an information processing device, The information processing device includes: determining a state of the refueling nozzle and a state of refueling based on the notification from the refueling aircraft; When it is determined that refueling is not permitted based on a determination logic according to whether the refueling action of the refueler is included in the first section before the start of refueling, the second section during refueling, or the third section after the end of refueling, or based on a determination logic according to whether the refueling action is included in two consecutive sections, refueling by the refueling machine is prohibited or stopped. death, The first determination logic according to the first section is configured to determine whether a recognition result obtained when a recognition process is performed on image data captured by the camera is: If the permission conditions are met for a certain period of time, refueling is permitted. When a predetermined non-permission condition is satisfied for a certain period of time, it is determined that refueling is not permitted, and when a permission condition is satisfied for a certain period of time thereafter, it is determined that refueling is permitted; If other disapproval conditions are met for a certain period of time, refueling is deemed to be disapproved. The second judgment logic according to the second section is configured to determine whether or not a recognition result obtained when a recognition process is performed on image data captured by the camera is: If the disallowance condition is met for a certain period of time, a warning is output. If the disapproval conditions are met for a certain period of time after the warning is output, it is determined that refueling is not permitted. If the permission conditions are met for a certain period of time after the warning is output, it is determined that refueling is permitted. The third determination logic according to the third section is configured to determine whether a recognition result obtained when a recognition process is performed on image data captured by the camera is: If the permission conditions are met for a certain period of time, refueling is permitted. If the non-permission conditions are met for a certain period of time, it is determined that refueling is not permitted, and if the permission conditions are met for a certain period of time thereafter, it is determined that refueling is permitted. . Effect of the Invention

[0007] It is possible to provide a refueling system capable of executing processing according to the refueling scene. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram showing a learning flow for constructing the refueling system according to the first embodiment. [Figure 2A] FIG. 1 is a first diagram showing an example of a refueling scene at a service station. [Figure 2B] FIG. 2 is a second diagram showing an example of a refueling scene at a service station. [Figure 3A] FIG. 13 is a diagram showing an overview of a process for generating first learning data. [Figure 3B] FIG. 13 is a diagram showing an overview of a process for generating second learning data. [Figure 4] FIG. 2 illustrates an example of a hardware configuration of a learning device. [Diagram 5] FIG. 2 illustrates an example of a functional configuration of a learning device. [Figure 6] FIG. 4 is a diagram showing a specific example of first learning data. [Figure 7] FIG. 13 is a diagram showing a specific example of second learning data. [Figure 8] 13 is an example of a flowchart showing the flow of a learning process. [Figure 9] FIG. 2 illustrates an example of a hardware configuration of an information processing device. [Figure 10] FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing device. [Figure 11] 2 is a diagram illustrating an example of a functional configuration of a section determination unit of the information processing device. [Figure 12] 11A and 11B are diagrams illustrating a specific example of processing by a section determination unit of the information processing device. [Figure 13] FIG. 2 is a diagram illustrating an example of a functional configuration of an image recognition unit of the information processing apparatus. [Figure 14A] FIG. 2 is a first diagram showing a specific example of processing by an image recognition unit of an information processing device. [Figure 14B] FIG. 2 is a second diagram showing a specific example of processing by the image recognition unit of the information processing device. [Figure 15A] 2 is a diagram illustrating an example of a functional configuration of a first refueling monitoring unit of the information processing device. FIG. [Figure 15B] 13 is a diagram illustrating an example of a functional configuration of a second refueling monitoring unit of the information processing device. FIG. [Figure 16A] 13 is an example of a first flowchart showing a process flow using a first determination logic of a first refueling monitoring unit. [Figure 16B] 13 is an example of a second flowchart showing a process flow using the first determination logic of the first refueling monitoring unit. [Figure 17A] 13 is an example of a first flowchart showing a process flow using a second determination logic of a first refueling monitoring unit. [Figure 17B] 13 is an example of a second flowchart showing a process flow using a second determination logic of the first refueling monitoring unit. [Figure 17C]13 is an example of a third flowchart showing a process flow using the second determination logic of the first refueling monitoring unit. [Figure 17D] 13 is an example of a fourth flowchart showing a process flow using the second determination logic of the first refueling monitoring unit. [Figure 18A] 13 is an example of a first flowchart showing a process flow using a third determination logic of the first refueling monitoring unit. [Figure 18B] 13 is an example of a second flowchart showing a process flow using a third determination logic of the first refueling monitoring unit. [Figure 19] 13 is an example of a flowchart showing the flow of processing using the plastic tank determination unit of the first refueling monitoring unit and processing using the fire determination unit. [Figure 20A] 13 is an example of a first flowchart showing a process flow using a first determination logic of a second refueling monitoring unit. [Figure 20B] 13 is an example of a second flowchart showing a process flow using the first determination logic of the second refueling monitoring unit. [Figure 21A] 13 is an example of a first flowchart showing a process flow using a second determination logic of a second refueling monitoring unit. [Figure 21B] 13 is an example of a second flowchart showing a process flow using a second determination logic of a second refueling monitoring unit. [Figure 21C] 13 is an example of a third flowchart showing a process flow using the second determination logic of the second refueling monitoring unit. [Figure 21D] 13 is an example of a fourth flowchart showing a process flow using the second determination logic of the second refueling monitoring unit. [Figure 22A] 13 is an example of a first flowchart showing a process flow using a third determination logic of a second refueling monitoring unit. [Figure 22B] 13 is an example of a second flowchart showing a process flow using a third determination logic of the second refueling monitoring unit. [Diagram 23] 13 is an example of a flowchart showing the processing flow using the fire detection unit of the second refueling monitoring unit. [Figure 24]FIG. 2 illustrates an example of a functional configuration of an output control unit of the information processing device. [Diagram 25] FIG. 11 is a diagram illustrating an example of output information. [Figure 26] 4 is an example of a flowchart showing a flow of a refueling control process by the refueling system according to the first embodiment. [Figure 27] FIG. 13 is a diagram illustrating another example of a functional configuration of an information processing device. [Figure 28A] 13 is a diagram illustrating another example of the functional configuration of the first refueling monitoring unit of the information processing device. FIG. [Figure 28B] FIG. 13 is a diagram illustrating another example of the functional configuration of the second refueling monitoring unit of the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0010] [First embodiment] <Learning process> First, a learning flow for constructing the refueling system according to the first embodiment will be described. FIG. 1 is a diagram showing a learning flow for constructing the refueling system according to the first embodiment. As shown in FIG. 1, when constructing the refueling system according to the first embodiment, image data collected at a plurality of service stations (for example, service stations 110, 111, ...) is used. In addition, when different types of refueling can be refueled at one service station, image data collected for each type of refueling is used. Note that the different types of refueling here refer to fuel oil for vehicles such as gasoline, diesel, etc. (hereinafter referred to as fuel oil of the "first type of refueling") and fuel oil for purposes other than vehicles such as kerosene (hereinafter referred to as fuel oil of the "second type of refueling").

[0011] The example of Fig. 1 shows a case where the service station 110 is capable of supplying both the first and second fuel types. Therefore, in the example of Fig. 1, image data of a fuel supply scene 110_1 for the first fuel type and image data of a fuel supply scene 110_2 for the second fuel type are collected.

[0012] The refueling scene 110_1 for the first refueling type includes a refueling action by a refueler (not shown). A camera 130_1 that captures an area including the first refueling aircraft 120_1 as a video captures the first refueling aircraft 120_1 and the refueling action by the refueler, and image data of a predetermined frame period is collected by the image data collection device 140.

[0013] As shown in Fig. 1, a refueling nozzle 121_1 is connected to the first refueling machine 120_1, and a first type of fuel oil stored in a tank 126_1 is supplied to the refueling nozzle 121_1 by a refueling pump 125_1 through a refueling piping. As shown in Fig. 1, a refueling valve 124_1 is installed in the refueling piping. In addition, a voice output device 122_1 and a refueling data display 123_1 (devices that display refueling data and messages from the information processing device 160) are installed as notification devices in the first refueling machine 120_1.

[0014] Although not shown in FIG. 1, similar image data is collected by the image data collection device 140 for the refueling scene 110_2 for the second refueling type.

[0015] When collecting image data, the image data collection device 140 also collects a signal indicating the removal of the refueling nozzle 121_1 from the first refueling aircraft 120_1 and the return of the refueling nozzle 121_1 to the first refueling aircraft 120_1 (a signal indicating the state of the refueling nozzle).

[0016] The image data collected by the image data collection device 140 and the signal indicating the state of the refueling nozzle are transmitted to the learning device 150 via the network 180, and learning data is generated in the learning device 150. The learning data is generated based on image data collected at a plurality of service stations (e.g., service stations 110, 111, ...) and the signal indicating the state of the refueling nozzle. However, the learning data is generated separately for the first refueling type and the second refueling type.

[0017] The learning device 150 is installed in a different location (e.g., a server on the cloud) from multiple service stations (e.g., service stations 110, 111, ...). The learning device 150 learns a first model using learning data for a first refueling type (referred to as first learning data) and generates a first trained model. In addition, the learning device 150 learns a second model using learning data for a second refueling type (referred to as second learning data) and generates a second trained model.

[0018] The first trained model and the second trained model generated by the learning device 150 are transmitted to a plurality of service stations (e.g., service stations 110′, 111′, . . .) via the network 180. As a result, they are stored in the information processing device of each service station (e.g., the information processing device 160 of the service station 110′).

[0019] As shown in FIG. 1, a fuel supply system 110S′ according to a first embodiment applied to a service station 110′ includes the following components for a first fuel supply type: Camera 130_1, First tanker 120_1, Thermal sensor 170_1, The first refueling machine 120_1 is connected to a refueling nozzle 121_1 through which the first refueling type fuel oil stored in a tank 126_1 is supplied by a refueling pump 125_1 via a refueling piping. A refueling valve 124_1 is provided on the refueling piping. Furthermore, the first refueling machine 120_1 is provided with an audio output device 122_1 and a refueling data display 123_1 as notification devices.

[0020] In addition, the refueling system 110S' according to the first embodiment applied to the service station 110' has a similar configuration for the second refueling type. However, when the measurement area of ​​the thermal sensor 170_1 includes both the refueling scene 110_1 of the first refueling type and the refueling scene 110_2 of the second refueling type, the thermal sensor 170_1 may be used for both the first refueling type and the second refueling type.

[0021] Furthermore, as shown in FIG. 1, the fuel supply system 110S′ according to the first embodiment, which is applied to the service station 110′, includes an information processing device 160. The information processing device 160 includes: Cameras for the first refueling type and the second refueling type (cameras 130_1, 130_2), Refueling aircraft (first refueling aircraft 120_1, second refueling aircraft 120_2), Thermal sensors (thermal sensors 170_1, 170_2), Fuel pump (fuel pump 125_1, 125_2), and are communicatively connected.

[0022] In the example of Figure 1, the fuel supply system 110S' of the first embodiment is shown as being applied to the service station 110', but the same configuration will be achieved when applied to other service stations (e.g., service stations 111', ...).

[0023] <Description of the refueling scene> Next, a specific example of a refueling scene at the service station 110 will be described. Figures 2A and 2B are first and second diagrams showing an example of a refueling scene at a service station.

[0024] Of these, Fig. 2A shows an example of a refueling scene for a first refueling type at a service station 110. As shown in Fig. 2A, the service station 110 includes a first area 210 where a vehicle 211 stops, and a second area 220 where a first refueling machine 120_1 for refueling the first refueling type is installed and where a refueling act is performed by a refueler 221.

[0025] In the case of the refueling scene shown in FIG. 2A, the refueling person 221 removes the refueling nozzle 121_1 from the nozzle hanger of the first refueling machine 120_1 while the vehicle 211 is stopped in the first area 210, and inserts the refueling nozzle 121_1 into the refueling port of the vehicle 211. The refueling person 221 also starts refueling by operating the lever of the refueling nozzle 121_1, and continues refueling until a desired amount of refueling is reached. Then, when the desired amount of refueling is reached, the refueling person 221 stops refueling by returning the lever of the refueling nozzle 121_1, removes the refueling nozzle 121_1 from the refueling port, and returns it to the nozzle hanger of the first refueling machine 120_1, thereby completing the refueling action.

[0026] On the other hand, Fig. 2B shows an example of a refueling scene for the second refueling type at the service station 110. As shown in Fig. 2B, the service station 110 includes an area where a container 231 (polyethylene tank) is placed, and an area where a second refueling machine 120_2 for supplying the second refueling type is installed and where a refueling act is performed by a refueler 241.

[0027] In the case of the refueling scene shown in FIG. 2B, the refueling person 241 removes the refueling nozzle 121_2 from the nozzle hanger of the second refueling machine 120_2 while placing the container 231, and inserts the refueling nozzle 121_2 into the refueling opening of the container 231. The refueling person 241 also starts refueling by operating the lever of the refueling nozzle 121_2, and continues refueling until a desired amount of refueling is reached. Then, when the desired amount of refueling is reached, the refueling person 241 stops refueling by returning the lever of the refueling nozzle 121_2, removes the refueling nozzle 121_2 from the refueling opening, and returns it to the nozzle hanger of the second refueling machine 120_2, thereby completing the refueling action.

[0028] <Processing for generating learning data> Next, an overview of a process from when the camera 130_1 captures an area including the first area 210 and the second area 220 as a video until the learning device 150 generates first learning data based on the video will be described. Fig. 3A is a diagram showing a flow of a process for generating the first learning data.

[0029] The image data collection device 140 collects image data at a predetermined frame period captured as a moving image by the camera 130_1, as shown in Fig. 3A. In addition, the image data collection device 140 collects a signal indicating the state of the refueling nozzle 121_1 from the first refueling aircraft 120_1, as shown in Fig. 3A.

[0030] The learning device 150 extracts, from the image data collected by the image data collection device 140, image data from the time when the refueling nozzle 121_1 is removed from the nozzle hanger of the first refueling aircraft 120_1 to the time when it is returned to the nozzle hanger.

[0031] Furthermore, the learning device 150 performs image recognition processing on the extracted image data and recognizes each of the recognition objects (vehicle, container, refueler, refueling nozzle, etc.) included in the extracted image data. Furthermore, the learning device 150 adds annotations to the recognized recognition objects (vehicle, container, refueler, refueling nozzle, etc.).

[0032] Specifically, when a vehicle is recognized in image data, and the vehicle is recognized within a first region 210 that is part of the image data, an annotation indicating that it is a vehicle is added to the circumscribing rectangle (bounding box) of the vehicle.

[0033] Furthermore, when a container is recognized in the image data, an annotation indicating that it is a container is added to the circumscribing rectangle (bounding box) of the container.

[0034] In addition, when a fueler is recognized in the image data, The fueler is recognized in the second region 220, and - The fuel nozzle was held in the correct position. In this case, an annotation is added to the circumscribing rectangle (bounding box) of the fuel dispenser indicating that the fuel dispenser is holding the fuel dispenser nozzle in an appropriate position.

[0035] In addition, when a fueler is recognized in the image data, The fueler is recognized in the second region 220, and The fuel nozzle being held was not recognized. In this case, an annotation (fuel nozzle cannot be recognized) is added to the circumscribing rectangle (bounding box) of the fuel dispenser indicating that the fuel dispenser is not holding the fuel dispenser nozzle in the appropriate position.

[0036] In addition, when the fuel supplier is not recognized in the second area 220 in the image data, For example, the fuel nozzle inserted in the fuel filler neck was recognized. In this case, an annotation indicating that the fuel dispenser is not holding the fuel dispenser nozzle in an appropriate position (fuel dispenser cannot be recognized) is added to the circumscribing rectangle (bounding box) of the recognized fuel dispenser nozzle.

[0037] The learning device 150 generates first learning data using image data to which any of the annotations has been added, from among the extracted image data.

[0038] Next, an overview of the process will be described from when the camera 130_2, which captures a video of the area including the second refueling aircraft 120_2, captures an image of the second refueling aircraft 120_2 and the container placed thereon, to when the learning device 150 generates second learning data based on the video. Fig. 3B is a diagram showing the flow of the process for generating the second learning data.

[0039] The image data collection device 140 collects image data at a predetermined frame period captured as a moving image by the camera 130_2, as shown in Fig. 3B. In addition, the image data collection device 140 collects a signal indicating the state of the refueling nozzle 121_2 from the second refueling aircraft 120_2, as shown in Fig. 3B.

[0040] The learning device 150 extracts, from the image data collected by the image data collection device 140, image data from the time when the refueling nozzle 121_2 is removed from the nozzle hanger of the second refueling aircraft 120_2 to the time when it is returned to the nozzle hanger.

[0041] The learning device 150 also performs image recognition processing on the extracted image data and recognizes each of the recognition objects (container, fuel pump, fuel nozzle, etc.) included in the extracted image data. Furthermore, the learning device 150 adds annotations to the recognized recognition objects (container, fuel pump, fuel nozzle, etc.).

[0042] Specifically, when a container is recognized in image data, an annotation indicating that it is a container is added to the circumscribing rectangle (bounding box) of the recognized container.

[0043] In addition, when a fueler is recognized in the image data, - The fueler was holding the fuel nozzle in the correct position. In this case, an annotation is added to the circumscribing rectangle (bounding box) of the fuel dispenser indicating that the fuel dispenser is holding the fuel dispenser nozzle in an appropriate position.

[0044] In addition, when a fueler is recognized in the image data, - The fueler was unable to recognize the fuel nozzle he was holding, In this case, an annotation (fuel nozzle cannot be recognized) is added to the circumscribing rectangle (bounding box) of the fuel dispenser indicating that the fuel dispenser is not holding the fuel dispenser nozzle in the appropriate position.

[0045] In addition, if the fueler is not recognized in the image data, For example, the fuel nozzle inserted in the fuel filler neck was recognized. In this case, an annotation indicating that the fuel dispenser is not holding the fuel dispenser nozzle in an appropriate position (the fuel dispenser cannot be recognized) is added to the circumscribing rectangle (bounding box) of the fuel dispenser nozzle.

[0046] The learning device 150 generates second learning data using image data to which any of the annotations has been added, from among the extracted image data.

[0047] <Hardware configuration of the learning device> Next, a description will be given of the hardware configuration of the learning device 150. Fig. 4 is a diagram showing an example of the hardware configuration of the learning device.

[0048] 4, the learning device 150 includes a processor 401, a memory 402, an auxiliary storage device 403, an I / F (Interface) device 404, a communication device 405, and a drive device 406. The processor 401, the memory 402, the auxiliary storage device 403, the I / F device 404, the communication device 405, and the drive device 406 of the learning device 150 are connected to each other via a bus 407.

[0049] The processor 401 has various arithmetic devices such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. The processor 401 reads various programs (for example, a learning program, etc.) onto the memory 402 and executes them.

[0050] The memory 402 has a main storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The processor 401 and the memory 402 form a so-called computer, and the processor 401 executes various programs read onto the memory 402, causing the computer to realize various functions.

[0051] The auxiliary storage device 403 stores various programs and various information used when the various programs are executed by the processor 401 .

[0052] The I / F device 404 is a connection device for connecting the learning device 150 to an operation device 411 and a display device 412, which are examples of external devices.

[0053] The communication unit 405 is a communication device for communicating with the image data collection unit 140 or the information processing unit 160 via a network.

[0054] The drive device 406 is a device for setting the recording medium 413. The recording medium 413 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, a magneto-optical disk, etc. The recording medium 413 may also include semiconductor memories that record information electrically, such as ROM, flash memory, etc.

[0055] The various programs to be installed in the auxiliary storage device 403 are installed, for example, by setting the distributed recording medium 413 in the drive device 406 and reading out the various programs recorded in the recording medium 413. Alternatively, the various programs to be installed in the auxiliary storage device 403 may be installed by being downloaded from a network via the communication device 405.

[0056] <Functional configuration of learning device> Next, a description will be given of the functional configuration of the learning device 150. Fig. 5 is a diagram showing an example of the functional configuration of the learning device. As described above, a learning program is installed in the learning device 150, and by executing the program, the learning device 150 functions as a learning data generation unit 500A and a learning unit 500B.

[0057] As shown in FIG. 5, the learning data generation unit 500A further includes a collected data acquisition unit 510, an image recognition unit 520, an annotation assignment unit 530, and a learning data generation unit 540.

[0058] The collected data acquisition unit 510 acquires image data captured by the camera 130_1 and a signal indicating the state of the fuel filling nozzle 121_1 from the image data collection device 140. The collected data acquisition unit 510 also acquires image data captured by the camera 130_2 and a signal indicating the state of the fuel filling nozzle 121_2 from the image data collection device 140.

[0059] The image recognition unit 520 extracts image data from the image data captured by the camera 130_1, covering the period from when the refueling nozzle 121_1 is removed from the nozzle hanger of the first refueling machine 120_1 until when the refueling nozzle 121_1 is returned to the nozzle hanger of the first refueling machine 120_1. The image recognition unit 520 also performs image recognition processing on the extracted image data. As a result, the image recognition unit 520 recognizes the recognition target (vehicle, container, refueler, refueling nozzle, etc.).

[0060] Furthermore, the image recognition unit 520 extracts image data from the image data captured by the camera 130_2, from the time when the refueling nozzle 121_2 is removed from the nozzle hanger of the second refueling machine 120_2 until the time when the refueling nozzle 121_2 is returned to the nozzle hanger of the second refueling machine 120_2. Furthermore, the image recognition unit 520 performs image recognition processing on the extracted image data. As a result, the image recognition unit 520 recognizes the recognition target (container, refueler, refueling nozzle, etc.).

[0061] The annotation adding unit 530 adds an annotation to the bounding box of the recognition target recognized by the image recognition unit 520. Note that the method of adding an annotation has been described with reference to Figs. 3A and 3B, and therefore will not be described here.

[0062] The learning data generation unit 540 generates first learning data and second learning data using the image data to which annotations have been added by the annotation adding unit 530 as input data and the added annotations as correct answer data. The learning data generation unit 540 stores the generated first learning data and second learning data in the learning data storage unit 500C.

[0063] As shown in FIG. 5, the learning unit 500B further includes a learning data reading unit 550, a first model 560_1, a comparison / change unit 570_1, a second model 560_2, and a comparison / change unit 570_2.

[0064] The learning data reading unit 550 reads the first learning data from the learning data storage unit 500C, and inputs the input data included in the first learning data to the first model 560_1, and also inputs the correct answer data to the comparison / change unit 570_1.

[0065] Furthermore, the learning data reading unit 550 reads out the second learning data from the learning data storage unit 500C, inputs the input data included in the second learning data to the second model 560_2, and inputs the correct answer data to the comparison / change unit 570_2.

[0066] The first model 560_1 is a machine learning model that performs image recognition processing, and when input data included in the first learning data is input, it performs image recognition processing and outputs a recognition result.

[0067] The comparison / change unit 570_1 compares the recognition result output from the first model 560_1 with the correct answer data input by the learning data reading unit 550. Furthermore, the comparison / change unit 570_1 updates the model parameters of the first model 560_1 so that the recognition result output from the first model 560_1 matches the correct answer data. In this manner, a learning process is performed on the first model 560_1, and a first trained model is generated.

[0068] The second model 560_2 is a machine learning model that performs image recognition processing, and when input data included in the second learning data is input, the second model 560_2 performs image recognition processing and outputs a recognition result.

[0069] The comparison / change unit 570_2 compares the recognition result output from the second model 560_2 with the correct answer data input by the learning data reading unit 550. Furthermore, the comparison / change unit 570_2 updates the model parameters of the second model 560_2 so that the recognition result output from the second model 560_2 matches the correct answer data. In this manner, a learning process is performed on the second model 560_2, and a second trained model is generated.

[0070] The first trained model and the second trained model generated by the learning unit 500B are transmitted to the information processing device 160, etc.

[0071] <Specific examples of learning data> Next, a specific example of the learning data generated by the learning data generating unit 500A of the learning device 150 will be described.

[0072] (1) Specific examples of the first learning data Fig. 6 is a diagram showing a specific example of the first learning data. As shown in Fig. 6, the first learning data 600 includes, as information items, "input data" and "correct answer data". Moreover, the "input data" further includes "image data ID".

[0073] "Image data ID" stores an identifier for identifying image data to which an annotation has been added by the annotation adding unit 530. In the example of Fig. 6, image data 610 is image data with an image data ID = ID001, and image data 620 is image data with an image data ID = ID00L. Image data 630 is image data with an image data ID = ID00M, and image data 640 is image data with an image data ID = ID00N.

[0074] The “answer data” further includes an “area” and “annotation data.” The “area” stores the first area 210 and the second area 220, and the “annotation data” stores the annotation data added to the bounding box of the recognition target recognized in the corresponding area.

[0075] In the example of FIG. 6, a bounding box 611 of a vehicle recognized in a first region 210 that is a part of image data 610 is "Annotation data that identifies it as a vehicle" 6 shows that the bounding box 612 of the fuel supplier recognized in the second region 220, which is a part of the image data 610, is given the following. "Annotation indicating that the fueler is holding the fuel nozzle in the correct position" This indicates that the certificate has been granted.

[0076] Similarly, in the example of FIG. 6, for a bounding box 621 of a vehicle recognized in a first region 210 that is a part of image data 620, "Annotation data that identifies it as a vehicle" 6 shows that the bounding box 622 of the fuel supplier recognized in the second region 220, which is a part of the image data 620, is given the following. "Annotation indicating that the fuel dispenser is not holding the fuel dispenser nozzle in the correct position (fuel dispenser nozzle cannot be recognized)" This indicates that the certificate has been granted.

[0077] Similarly, in the example of FIG. 6, for a bounding box 631 of a vehicle recognized in a first region 210 that is a part of image data 630, "Annotation data that identifies it as a vehicle" 6 shows that the bounding box 632 of the fuel nozzle recognized in the second region 220, which is a part of the image data 630, is given the following. "Annotation indicating that the fueler is not holding the fuel nozzle in the correct position (fueler cannot be identified)" This indicates that the certificate has been granted.

[0078] Similarly, in the example of FIG. 6, for a bounding box 641 of a container recognized in a first region 210 that is a part of image data 640, "Annotation data indicating that it is a container" 6 shows that the bounding box 642 of the fuel supplier recognized in the second region 220, which is a part of the image data 640, is given the following. "Annotation indicating that the fueler is holding the fuel nozzle in the correct position" This indicates that the certificate has been granted.

[0079] (2) Specific examples of the second learning data Fig. 7 is a diagram showing a specific example of the second learning data. As shown in Fig. 7, the second learning data 700 includes, as information items, "input data" and "correct answer data". Moreover, the "input data" further includes "image data ID".

[0080] "Image data ID" stores an identifier for identifying image data to which an annotation has been added by the annotation adding unit 530. In the example of Fig. 7, image data 710 is image data with an image data ID = ID101, and image data 720 is image data with an image data ID = ID10L. Image data 730 is image data with an image data ID = ID10M, and image data 740 is image data with an image data ID = ID10N.

[0081] The "answer data" further includes "annotation data." The "annotation data" stores annotation data that is added to the bounding box of the recognized recognition target.

[0082] In the example of FIG. 7, for a bounding box 711 of a container recognized in image data 710, "Annotation data indicating that it is a container" In addition, the example of FIG. 7 shows that the bounding box 712 of the fueler recognized in the image data 710 is given the following: "Annotation indicating that the fueler is holding the fuel nozzle in the correct position" This indicates that the certificate has been granted.

[0083] Similarly, the example of FIG. 7 shows that for a bounding box 721 of a container recognized in image data 720, "Annotation data indicating that it is a container" In addition, the example of FIG. 7 shows that the bounding box 722 of the fueler recognized in the image data 720 is given the following: "Annotation indicating that the fuel dispenser is not holding the fuel dispenser nozzle in the correct position (fuel dispenser nozzle cannot be recognized)" This indicates that the certificate has been granted.

[0084] Similarly, the example of FIG. 7 shows that for a bounding box 731 of a container recognized in image data 730, "Annotation data indicating that it is a container" In addition, the example of FIG. 7 shows that the bounding box 732 of the fuel nozzle recognized in the image data 730 is given the following: "Annotation indicating that the fueler is not holding the fuel nozzle in the correct position (fueler cannot be identified)" This indicates that the certificate has been granted.

[0085] Similarly, the example of FIG. 7 shows that for a bounding box 742 of a fueler recognized in image data 740, "Annotation indicating that the fueler is holding the fuel nozzle in the correct position" This indicates that the certificate has been granted.

[0086] <Learning process flow by the learning device> Next, a description will be given of the flow of the learning process performed by the learning device 150. Fig. 8 is an example of a flowchart showing the flow of the learning process.

[0087] In step S801, the learning device 150 acquires image data and a signal indicating the state of the fuel nozzle from the image data collection device 140.

[0088] In step S802, the learning device 150 extracts image data from the time when the fuel nozzle is removed from the nozzle hanger to the time when it is returned to the nozzle hanger, and performs image recognition processing on the extracted image data.

[0089] In step S803, the learning device 150 identifies a first region and a second region in the image data used to generate the first learning data, among the image data on which the image recognition process has been performed.

[0090] In step S804, the learning device 150 adds annotations to the image data used to generate the first learning data. Specifically, the learning device 150 adds annotations to the bounding box of the recognition target included in the identified first region, and also adds annotations to the bounding box of the recognition target included in the identified second region.

[0091] Furthermore, the learning device 150 adds annotations to image data used to generate the second learning data. Specifically, the learning device 150 adds annotations to the bounding box of the recognition target.

[0092] In step S805, the learning device 150 generates first learning data for the first refueling type and generates second learning data for the second refueling type based on the image data to which the annotations have been added.

[0093] In step S806, the learning device 150 performs a learning process on the first model using the first training data to generate a first trained model. Also, the learning device 150 performs a learning process on the second model using the second training data to generate a second trained model.

[0094] In step S807, the learning device 150 transmits the generated first trained model and second trained model to the information processing device 160 or the like, and stores them in the information processing device 160 or the like.

[0095] <Hardware configuration of information processing device> Next, a description will be given of a hardware configuration of the information processing device 160. Fig. 9 is a diagram showing an example of a hardware configuration of the information processing device.

[0096] 9, the information processing device 160 includes a processor 901, a memory 902, an auxiliary storage device 903, a connection device 904, an I / F device 905, a communication device 906, and a drive device 907. The processor 901, memory 902, auxiliary storage device 903, connection device 904, I / F device 905, communication device 906, and drive device 907 of the information processing device 160 are connected to each other via a bus 908.

[0097] The processor 901 has various arithmetic devices such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. The processor 901 reads various programs (for example, a fuel supply control program, etc.) onto the memory 902 and executes them.

[0098] The memory 902 has a main storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The processor 901 and the memory 902 form a so-called computer, and the processor 901 executes various programs read onto the memory 902, causing the computer to realize various functions.

[0099] The auxiliary storage device 903 stores various programs and various information used when the various programs are executed by the processor 901 .

[0100] The connection device 904 is a connection device for connecting the first refueling aircraft 120_1 and the second refueling aircraft 120_2 with the information processing device 160. The connection device 904 receives signals indicating the state of the refueling nozzle, signals indicating the amount of refueling, and the like from the first refueling aircraft 120_1 and the second refueling aircraft 120_2. The signals indicating the state of the refueling nozzle, signals indicating the amount of refueling, and the like are measured by the first refueling sensor 921_1 and the second refueling sensor 921_2. In addition, the connection device 904 transmits messages to be output as voice via the voice output devices 122_1 and 122_2 to the first refueling aircraft 120_1 and the second refueling aircraft 120_2. In addition, the connection device 904 transmits display information to be displayed via the refueling data display devices 123_1 and 123_2 to the first refueling aircraft 120_1 and the second refueling aircraft 120_2. The connection device 904 also transmits signals to the first refueling aircraft 120_1 and the second refueling aircraft 120_2 to instruct them to open and close the refueling valves 124_1 and 124_2. Furthermore, the connection device 904 transmits signals to instruct the refueling pumps 125_1 and 125_2 to start and stop driving.

[0101] The connection device 904 also receives thermal sensor data from the thermal sensors 170_1, 170_2.

[0102] The I / F device 905 is a connection device for connecting an operation device 911, which is an example of an external device, and a display device 912 to the information processing device 160. The operation device 911 is operated by, for example, a staff member of the service station 110', and the display device 912 displays information for, for example, the staff member of the service station 110'.

[0103] The communication device 906 communicates with the cameras 130_1 and 130_2 via the network to receive image data. The communication device 906 also communicates with the learning device 150 to receive the first trained model and the second trained model.

[0104] The drive device 907 is a device for setting the recording medium 913. The recording medium 913 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, a magneto-optical disk, etc. The recording medium 913 may also include semiconductor memories that record information electrically, such as ROM, flash memory, etc.

[0105] The various programs to be installed in the auxiliary storage device 903 are installed, for example, by setting the distributed recording medium 913 in the drive device 907 and reading out the various programs recorded in the recording medium 913. Alternatively, the various programs to be installed in the auxiliary storage device 903 may be installed by downloading them from a network via the communication device 906.

[0106] <Functional configuration of information processing device> Next, the functional configuration of the information processing device 160 will be described. Fig. 10 is a diagram showing an example of the functional configuration of the information processing device. As described above, a refueling control program is installed in the information processing device 160, and by executing the program, the information processing device 160: A first refueling type image data acquisition unit 1001, a second refueling type image data acquisition unit 1002, A first refueling sensor data acquisition unit 1003, a second refueling sensor data acquisition unit 1004, Image recognition unit 1005, Section determination unit 1006, section determination unit 1007, A thermal sensor data acquisition unit 1008; First refueling monitoring unit 1009_1, second refueling monitoring unit 1009_2, Output control unit 1010, has.

[0107] The first refueling type image data acquisition unit 1001 acquires image data captured by the camera 130_1, and notifies the image recognition unit 1005 of the image data.

[0108] The second refueling type image data acquisition unit 1002 acquires image data photographed by the camera 130_2, and notifies the image recognition unit 1005 of the image data.

[0109] The first refueling sensor data acquisition unit 1003 acquires a signal from the first refueling sensor 921_1, and notifies the section determination unit 1006 of the signal.

[0110] The second refueling sensor data acquisition unit 1004 acquires a signal from the second refueling sensor 921_2, and notifies the section determination unit 1007 of the signal.

[0111] The section determination unit 1006 classifies the section from when the refueling nozzle 121_1 is removed from the nozzle hanger of the first refueling machine 120_1 to when the refueling nozzle 121_1 is returned to the nozzle hanger into three sections based on a signal from the first refueling sensor 921_1. Of the three sections, the section before the start of refueling is referred to as the first section, the section during refueling as the second section, and the section after the end of refueling as the third section.

[0112] The section determination unit 1007 classifies the section from when the refueling nozzle 121_2 is removed from the nozzle hanger of the second refueling machine 120_2 to when the refueling nozzle 121_2 is returned to the nozzle hanger into three sections based on a signal from the second refueling sensor 921_2. As with the first refueling type, of the three sections, for the second refueling type as well, the section before the start of refueling is called the first section, the section during refueling is called the second section, and the section after the end of refueling is called the third section.

[0113] The image recognition unit 1005 extracts image data included in any one of the first to third sections from the image data notified from the first refueling type image data acquisition unit 1001. In addition, the image recognition unit 1005 performs image recognition processing on the extracted image data using the first trained model, and notifies the first refueling monitoring unit 1009_1 of the recognition result.

[0114] Moreover, the image recognition unit 1005 extracts image data included in any one of the first to third sections from the image data notified from the second refueling type image data acquisition unit 1002. Moreover, the image recognition unit 1005 performs image recognition processing on the extracted image data using the second trained model, and notifies the second refueling monitoring unit 1009_2 of the recognition result.

[0115] The thermal sensor data acquiring unit 1008 acquires thermal sensor data measured by the thermal sensor 170_1 and notifies the first refueling monitoring unit 1009_1. Also, the thermal sensor data acquiring unit 1008 acquires thermal sensor data measured by the thermal sensor 170_2 and notifies the second refueling monitoring unit 1009_2.

[0116] The first refueling monitoring unit 1009_1 is The recognition result notified by the image recognition unit 1005, The section determination result notified by the section determination unit 1006, The thermal sensor data of the thermal sensor 170_1 notified by the thermal sensor data acquisition unit 1008, The first refueling monitoring unit 1009_1 determines whether or not refueling is permitted for the first refueling type based on the result of the determination of whether or not refueling is permitted. Specifically, the first refueling monitoring unit 1009_1 controls the refueling for the first refueling type by controlling the refueling valve 124_1 or the refueling pump 125_1.

[0117] Moreover, the first refueling monitoring unit 1009_1 notifies the output control unit 1010 of the determination result of whether or not refueling is permitted. When the first refueling monitoring unit 1009_1 determines that refueling is not permitted, the first refueling monitoring unit 1009_1 also notifies the output control unit 1010 of information indicating the cause of the refusal of refueling.

[0118] The second refueling monitoring unit 1009_2 is The recognition result notified by the image recognition unit 1005, The section determination result notified by the section determination unit 1007, The thermal sensor data of the thermal sensor 170_2 notified by the thermal sensor data acquisition unit 1008, The second refueling monitoring unit 1009_2 determines whether or not refueling is permitted for the second refueling type based on the determination result of whether or not refueling is permitted. Specifically, the second refueling monitoring unit 1009_2 controls the refueling for the second refueling type by controlling the refueling valve 124_2 or the refueling pump 125_2.

[0119] Moreover, the second refueling monitoring unit 1009_2 notifies the output control unit 1010 of the determination result of whether or not refueling is permitted. When the second refueling monitoring unit 1009_2 determines that refueling is not permitted, the second refueling monitoring unit 1009_2 also notifies the output control unit 1010 of information indicating the cause of the refusal of refueling.

[0120] When the output control unit 1010 is notified of information indicating the cause of refusal of refueling from the first refueling monitoring unit 1009_1, Controlling so that a message corresponding to the information indicating the cause is output as voice via the voice output device 122_1; Controlling so that display information corresponding to the information indicating the cause is displayed on the refueling data display 123_1 of the first refueling aircraft 120_1; The information processing device 160 controls the display device 912 to display information indicating the cause.

[0121] In addition, when the output control unit 1010 is notified of information indicating the cause of refueling non-permission from the second refueling monitoring unit 1009_2, Controlling so that a message corresponding to the information indicating the cause is output as voice via the voice output device 122_2; Controlling so that display information corresponding to the information indicating the cause is displayed on the refueling data display 123_2 of the second refueling aircraft 120_2; The information processing device 160 controls the display device 912 to display information indicating the cause.

[0122] <Details of each part of the information processing device> Next, each unit of the information processing device 160 (here, the section determination units 1006 and 1007, the image recognition unit 1005, the first refueling monitoring unit 1009_1, the second refueling monitoring unit 1009_2, and the output control unit 1010) will be described in detail.

[0123] (1) Details of the section determination units 1006 and 1007 First, as details of the section determination units 1006 and 1007, a functional configuration of the section determination units 1006 and 1007 and a specific example of the processing performed by the section determination units 1006 and 1007 will be described.

[0124] (1-1) Details of the functional configuration of the section determination units 1006 and 1007 11 is a diagram showing an example of the functional configuration of a section determination unit of an information processing device. Note that since the functional configuration of the section determination unit 1006 and the functional configuration of the section determination unit 1007 are the same, they will be collectively described with reference to FIG.

[0125] As shown in FIG. 11, the section determination units 1006 and 1007 Nozzle state determination unit 1101, A fuel supply amount determination unit 1102, A first interval start determination unit 1103, a second interval transition determination unit 1104, a third interval transition determination unit 1105, a third interval end determination unit 1106, Section determination result output unit 1107, has.

[0126] The nozzle state determination unit 1101 acquires a signal indicating the state of the refueling nozzle 121_1 (or 121_2) from the first refueling sensor data acquisition unit 1003 (or the second refueling sensor data acquisition unit 1004).

[0127] When the nozzle state determination unit 1101 receives a signal indicating that the refueling nozzle 121_1 (or 121_2) has been removed from the nozzle hanger of the first refueling aircraft 120_1 (or the second refueling aircraft 120_2), it notifies the first interval start determination unit 1103.

[0128] In addition, when the nozzle state determination unit 1101 receives a signal indicating that the refueling nozzle 121_1 (or 121_2) has been returned to the nozzle hook of the first refueling aircraft 120_1 (or the second refueling aircraft 120_2), it notifies the third interval end determination unit 1106.

[0129] The fuel supply amount determination unit 1102 acquires a signal indicating the amount of fuel supply from the first fuel supply sensor data acquisition unit 1003 (or the second fuel supply sensor data acquisition unit 1004).

[0130] When the signal indicating the amount of refueling acquired from the first refueling sensor data acquisition unit 1003 (or the second refueling sensor data acquisition unit 1004) changes from zero to a predetermined threshold or greater, the refueling amount determination unit 1102 notifies the second zone transition determination unit 1104.

[0131] In addition, when the signal indicating the amount of refueling acquired from the first refueling sensor data acquisition unit 1003 (or the second refueling sensor data acquisition unit 1004) changes from a predetermined threshold or more to zero, the refueling amount determination unit 1102 notifies the third zone transition determination unit 1105.

[0132] The first section start determination unit 1103 determines that the first section has started when a signal indicating that the fuel nozzle 121_1 (or 121_2) has been removed is notified from the nozzle state determination unit 1101. The first section start determination unit 1103 also notifies the second section transition determination unit 1104 and the section determination result output unit 1107 that the first section has started.

[0133] The second section transition determination unit 1104 determines that the first section has been transitioned to the second section when the fuel supply amount determination unit 1102 notifies the unit 1104 that the fuel supply amount has changed from zero to a predetermined threshold or more after the unit 1104 is notified that the first section has started. The second section transition determination unit 1104 also notifies the third section transition determination unit 1105 and the section determination result output unit 1107 that the unit 1104 has transitioned from the first section to the second section.

[0134] The third zone transition determination unit 1105 determines that the vehicle has transitioned from the second zone to the third zone when the fuel supply amount determination unit 1102 notifies the vehicle of a change in the fuel supply amount from a predetermined threshold or more to zero after the unit is notified of the transition to the second zone. The third zone transition determination unit 1105 also notifies the third zone end determination unit 1106 and the zone determination result output unit 1107 of the transition from the second zone to the third zone.

[0135] The third-section end determination unit 1106 determines that the third section has ended when a signal indicating that the fueling nozzle 121_1 (or 121_2) has been returned is notified from the nozzle state determination unit 1101 after being notified of the transition to the third section. The third-section end determination unit 1106 also notifies the section determination result output unit 1107 that the third section has ended.

[0136] When the section determination result output unit 1107 is notified by the first section start determination unit 1103 that the first section has started, it starts outputting information indicating that it is in the first section. Furthermore, when the section determination result output unit 1107 is notified by the second section transition determination unit 1104 that it has transitioned to the second section, it stops outputting information indicating that it is in the first section and starts outputting information indicating that it is in the second section. Furthermore, when the section determination result output unit 1107 is notified by the third section transition determination unit 1105 that it has transitioned to the third section, it stops outputting information indicating that it is in the second section and starts outputting information indicating that it is in the third section. Furthermore, when the section determination result output unit 1107 is notified by the third section end determination unit 1106 that the third section has ended, it stops outputting information indicating that it is in the third section.

[0137] (1-2) Specific example of processing by the section determination units 1006 and 1007 FIG. 12 is a diagram showing a specific example of processing by the section determination unit of the information processing device. As shown in FIG. 12(a), when a signal indicating that the refueling nozzle 121_1 has been removed from the nozzle hanger is acquired as a signal indicating the state of the refueling nozzle 121_1, the section determination unit 1006 starts outputting information indicating that the first section is in progress. Furthermore, when a signal indicating that the signal indicating the amount of fuel supplied by the first refueling machine 120_1 has transitioned from zero to a predetermined threshold or more is acquired, the section determination unit 1006 starts outputting information indicating that the second section is in progress. Furthermore, when a signal indicating that the signal indicating the amount of fuel supplied by the first refueling machine 120_1 has transitioned from a predetermined threshold or more to zero is acquired, the section determination unit 1006 starts outputting information indicating that the third section is in progress. Furthermore, when a signal indicating that the refueling nozzle 121_1 has been returned to the nozzle hanger is acquired as a signal indicating the state of the refueling nozzle 121_1, the section determination unit 1006 stops outputting information indicating that the third section is in progress.

[0138] Similarly, as shown in FIG. 12(b), when the section determination unit 1007 acquires a signal indicating that the refueling nozzle 121_2 has been removed from the nozzle hanger as a signal indicating the state of the refueling nozzle 121_2, the section determination unit 1007 starts outputting information indicating that the section is in the first section. Furthermore, when the section determination unit 1007 acquires a signal indicating that the signal indicating the amount of refueling of the second refueling machine 120_2 has transitioned from zero to a predetermined threshold or more, the section determination unit 1007 starts outputting information indicating that the section is in the second section. Furthermore, when the section determination unit 1007 acquires a signal indicating that the signal indicating the amount of refueling of the second refueling machine 120_2 has transitioned from a predetermined threshold or more to zero, the section determination unit 1007 starts outputting information indicating that the section is in the third section. Furthermore, when the section determination unit 1007 acquires a signal indicating that the refueling nozzle 121_2 has been returned to the nozzle hanger as a signal indicating the state of the refueling nozzle 121_2, the section determination unit 1007 stops outputting information indicating that the section is in the third section.

[0139] (2) Details of the image recognition unit 1005 Next, as details of the image recognition unit 1005, a functional configuration of the image recognition unit 1005 and a specific example of the processing performed by the image recognition unit 1005 will be described.

[0140] (2-1) Details of the functional configuration of the image recognition unit 1005 Fig. 13 is a diagram showing an example of the functional configuration of an image recognition unit of an information processing device. As shown in Fig. 13, the image recognition unit 1005 has a recognition interval acquisition unit 1301, a first trained model 1302, and an area determination unit 1303. The image recognition unit 1005 also has a recognition interval acquisition unit 1311 and a second trained model 1312.

[0141] The recognition section acquisition unit 1301 extracts image data that is included in any of the first to third sections notified by the section determination unit 1006 from the image data notified by the first refueling type image data acquisition unit 1001, and inputs the extracted image data to the first trained model 1302.

[0142] The first trained model 1302 is a first trained model generated in the learning device 150, and when image data is input, it recognizes the recognition target contained in the image data and outputs the recognition result.

[0143] The area determination unit 1303 determines whether or not the recognition result output by the first trained model 1302 is included in the first area 210 or the second area 220. In addition, the area determination unit 1303 outputs the recognition result determined to be included in the first area 210 or the second area 220 to the first refueling monitoring unit 1009_1.

[0144] The recognition section acquisition unit 1311 extracts image data that is included in any of the first to third sections notified by the section determination unit 1007 from the image data notified by the second refueling type image data acquisition unit 1002, and inputs the extracted image data to the second trained model 1312.

[0145] The second trained model 1312 is a second trained model generated in the learning device 150, and when image data is input, it recognizes the recognition object contained in the image data and outputs the recognition result to the second refueling monitoring unit 1009_2.

[0146] (2-2) Specific example of processing by the image recognition unit 1005 (in the case of the first fuel type) Fig. 14A is a first diagram showing a specific example of processing by the image recognition unit of the information processing device. As shown in Fig. 14A, the image recognition unit 1005 extracts image data included in a section (first to third sections) from when the fuel nozzle 121_1 is removed from the nozzle hook to when it is returned, from the image data captured by the camera 130_1.

[0147] Also, as shown in FIG. 14A, the image recognition unit 1005 performs image recognition processing on the extracted image data using the first trained model 1302, and recognizes the recognition target (vehicle, container, refueler, refueling nozzle, etc.), ·vehicle, ·container, - The fueler holds the fuel nozzle in the correct position; - Fuelers not holding the fuel nozzle in the correct position (unable to recognize the fuel nozzle), - Fuelers not holding the fuel nozzle in the correct position (fueler cannot be identified), is output as the recognition result.

[0148] Moreover, image recognition unit 1005 outputs the recognition result that it is determined that the image is included in first region 210 or second region 220. Recognition results 1401 to 1404 in Fig. 14A show examples of output of the recognition results.

[0149] In the example of FIG. 14A, the recognition result 1401 output by the image recognition unit 1005 is A bounding box in which the vehicle is recognized in the first region 210; In the second region 220, a bounding box is recognized in which the fuel dispenser holds the fuel dispenser nozzle in an appropriate posture; This shows that.

[0150] In the example of FIG. 14A, the recognition result 1402 output by the image recognition unit 1005 is A bounding box in which the vehicle is recognized in the first region 210; In the second region 220, a bounding box in which a fuel dispenser is not holding the fuel dispenser nozzle in an appropriate position is recognized (fuel dispenser nozzle cannot be recognized), This shows that.

[0151] In the example of FIG. 14A, the recognition result 1403 output by the image recognition unit 1005 is: A bounding box in which the vehicle is recognized in the first region 210; In the second region 220, a bounding box in which a fuel dispenser not holding the fuel dispenser nozzle in an appropriate position is recognized (fuel dispenser recognition not possible); This shows that.

[0152] Furthermore, in the example of FIG. 14A, the recognition result 1404 output by the image recognition unit 1005 is A bounding box in which the container is recognized in the first region 210; In the second region 220, a bounding box is recognized in which the fuel dispenser holds the fuel dispenser nozzle in an appropriate posture; This shows that.

[0153] (2-3) Specific example of processing by the image recognition unit 1005 (in the case of the second fuel type) Fig. 14B is a second diagram showing a specific example of processing by the image recognition unit of the information processing device. As shown in Fig. 14B, the image recognition unit 1005 extracts image data included in a section (first to third sections) from when the fuel nozzle 121_2 is removed from the nozzle hook to when it is returned, from the image data captured by the camera 130_2.

[0154] Also, as shown in FIG. 14B, the image recognition unit 1005 performs image recognition processing on the extracted image data using the second trained model 1312, and recognizes the recognition target (a container, a fuel dispenser, a fuel nozzle, etc.), ·container, - The fueler holds the fuel nozzle in the correct position; - Fuelers not holding the fuel nozzle in the correct position (unable to recognize the fuel nozzle), - Fuelers not holding the fuel nozzle in the correct position (fueler cannot be identified), As the recognition result, the recognition results 1411 to 1414 in Fig. 14B show examples of the recognition result output.

[0155] In the example of FIG. 14B, the recognition result 1411 output by the image recognition unit 1005 is - bounding box of the container recognized, - The bounding box of the fueler holding the fuel nozzle in the correct position is recognized. This shows that.

[0156] In the example of FIG. 14B, the recognition result 1412 output by the image recognition unit 1005 is - bounding box of the container recognized, - Bounding box of the fuel dispenser not holding the fuel nozzle in the correct position (fuel nozzle cannot be recognized), This shows that.

[0157] In the example of FIG. 14B, the recognition result 1413 output by the image recognition unit 1005 is - bounding box of the container recognized, - Bounding box of a fueler not holding the fuel nozzle in the correct position (fueler not recognizable), This shows that.

[0158] Furthermore, in the example of FIG. 14B, the recognition result 1414 output by the image recognition unit 1005 is - The bounding box of the fueler holding the fuel nozzle in the correct position is recognized. This shows that.

[0159] (3) Details of the first refueling monitoring unit 1009_1 and the second refueling monitoring unit 1009_2 Next, as details of the first refueling monitoring unit 1009_1 and the second refueling monitoring unit 1009_2, a functional configuration and a processing flow of the first refueling monitoring unit 1009_1 and a functional configuration and a processing flow of the second refueling monitoring unit 1009_2 will be described.

[0160] (3-1) Details of the functional configuration of the first refueling monitoring unit 1009_1 and the second refueling monitoring unit 1009_2 Fig. 15A is a diagram showing an example of a functional configuration of a first refueling monitoring unit of an information processing device. As shown in Fig. 15A, the first refueling monitoring unit 1009_1 has a recognition result acquisition unit 1501, a judgment logic switching unit 1502, a plastic tank judgment unit 1506, a first judgment logic 1503, a second judgment logic 1504, a third judgment logic 1505, and a fire judgment unit 1507. The first refueling monitoring unit 1009_1 also has a refueling control unit 1508 and a cause information acquisition unit 1509. The first judgment logic 1503 to the third judgment logic 1505, the plastic tank judgment unit 1506, and the fire judgment unit 1507 are collectively referred to as a first judgment unit.

[0161] The recognition result acquisition unit 1501 acquires the recognition results for each of the image data included in the first to third sections from the image recognition unit 1005. The recognition result acquisition unit 1501 notifies the acquired recognition results to the determination logic switching unit 1502 and the plastic tank determination unit 1506.

[0162] The determination logic switching unit 1502 inputs the recognition result notified from the recognition result acquisition unit 1501 to any one of the first determination logic 1503 to the third determination logic 1505. Which of the first determination logic 1503 to the third determination logic 1505 to input the recognition result to is determined based on the information notified from the section determination unit 1006 indicating which of the first to third sections the recognition result is in. For example, while the information indicating the first section is notified, the determination logic switching unit 1502 inputs the recognition result to the first determination logic 1503. Also, while the information indicating the second section is notified, the determination logic switching unit 1502 inputs the recognition result to the second determination logic 1504. Also, while the information indicating the third section is notified, the determination logic switching unit 1502 inputs the recognition result to the third determination logic 1505.

[0163] Furthermore, the judgment logic switching unit 1502 notifies the plastic container judgment unit 1506 and the fire judgment unit 1507 of information indicating that the current time is in any one of the first to third intervals.

[0164] When the recognition result is notified from the recognition result acquisition unit 1501 in a case where information indicating that the vehicle is in any one of the first to third sections has been notified, the plastic tank determination unit 1506 determines whether or not a container is included in the notified recognition result. If the plastic tank determination unit 1506 determines that a container is included, it notifies the refueling control unit 1508 that refueling is not permitted.

[0165] The first judgment logic 1503 judges whether or not refueling is permitted based on the recognition result during the first section. The first judgment logic 1503 also notifies the refueling control unit 1508 of the judgment result as to whether or not refueling is permitted during the first section. When the first judgment logic 1503 judges that refueling is not permitted, the first judgment logic 1503 also notifies the refueling control unit 1508 of the judgment result, along with information indicating the reason for refueling being not permitted.

[0166] The second determination logic 1504 determines whether or not refueling is permitted based on the recognition result during the second section. The second determination logic 1504 also notifies the refueling control unit 1508 of the determination result regarding whether or not refueling is permitted during the second section. When the second determination logic 1504 determines that refueling is not permitted, the second determination logic 1504 also notifies the refueling control unit 1508 of the determination result, along with information indicating the cause of the determination that refueling is not permitted.

[0167] The third determination logic 1505 determines whether or not refueling is permitted based on the recognition result during the third section. The third determination logic 1505 also notifies the refueling control unit 1508 of the determination result of whether or not refueling is permitted during the third section. When the third determination logic 1505 determines that refueling is not permitted, the third determination logic 1505 also notifies the refueling control unit 1508 of the determination result.

[0168] The fire determination unit 1507 acquires thermal sensor data measured by the thermal sensor 170_1 from the thermal sensor data acquisition unit 1008. When the fire determination unit 1507 is notified of information indicating that the area is in any of the first to third zones, the fire determination unit 1507 determines whether or not a temperature region above a predetermined threshold is included. When the fire determination unit 1507 determines that a temperature region above a predetermined threshold is included, the fire determination unit 1507 notifies the refueling control unit 1508 that refueling is not permitted.

[0169] The refueling control unit 1508 controls the refueling of the first refueling type by controlling the refueling valve 124_1 or the refueling pump 125_1 based on the refueling permission or the refueling non-permission. For example, when the refueling permission is notified, the refueling control unit 1508 drives the refueling pump 125_1 to open the refueling valve 124_1. In addition, the refueling control unit 1508 When a notice of permission to refuel is received from the plastic tank determination unit 1506, the refueling pump 125_1 is driven to open the refueling valve 124_1. When a refueling denial notice is received from any of the first judgment logic 1503 to the third judgment logic 1505, or When the fire judgment unit 1507 notifies the operator that fueling is not permitted, In this case, refueling is stopped by stopping the operation of the refueling pump 125_1, or refueling is prohibited by stopping the operation of the refueling pump 125_1 and closing the refueling valve 124_1. Note that the definitions of the stop of refueling and the prohibition of refueling mentioned here are merely examples, and the stop of refueling and the prohibition of refueling may be defined by other definitions. Alternatively, without distinguishing between the stop of refueling and the prohibition of refueling, the general state in which refueling is not possible may be referred to as the stop or prohibition of refueling.

[0170] Furthermore, the refueling control unit 1508 notifies the cause information acquisition unit 1509 of refueling permission or refueling denial. When notifying refueling denial, the refueling control unit 1508 also notifies the cause information acquisition unit 1509 of information indicating the cause for determining that refueling is not permitted.

[0171] The cause information acquisition unit 1509 outputs refueling permission or refueling refusal to the output control unit 1010. When outputting refueling refusal, the cause information acquisition unit 1509 also outputs information indicating the cause of refusal of refueling notified by the refueling control unit 1508 (the first to fourth information, warning information, and confirmation request information) to the output control unit 1010.

[0172] Fig. 15B is a diagram showing an example of the functional configuration of the second refueling monitoring unit of the information processing device. The difference from the functional configuration of the first refueling monitoring unit 1009_1 shown in Fig. 15A is that the functional configuration of the second refueling monitoring unit 1009_2 does not include a functional unit corresponding to the plastic tank determination unit 1506. The other functional units of the second refueling monitoring unit 1009_2 are similar to the respective functional units of the first refueling monitoring unit 1009_1, and therefore the description thereof will be omitted here.

[0173] Incidentally, while the first refueling monitoring unit 1009_1 is referred to as a first judgment unit, in the second refueling monitoring unit 1009_2, the first judgment logic 1513 to the third judgment logic 1515 and the fire judgment unit 1517 are collectively referred to as a second judgment unit.

[0174] (3-2) Processing flow using the first determination logic 1503 of the first refueling monitoring unit 1009_1 Next, the flow of processing using the first judgment logic 1503 of the first refueling monitoring unit 1009_1 will be described with reference to Fig. 16A and Fig. 16B. Fig. 16A and Fig. 16B are examples of first and second flowcharts showing the flow of processing using the first judgment logic of the first refueling monitoring unit.

[0175] In step S1601, the first judgment logic 1503 judges whether or not the currently input recognition result is a recognition result that is judged as refueling permission (whether or not a predetermined permission condition is satisfied). Specifically, it judges whether or not a vehicle is recognized in the first area 210 and a refueler holding the refueling nozzle in an appropriate posture is recognized in the second area.

[0176] In step S1601, if it is determined that the recognition result is determined to be permission for refueling (YES in step S1601), the process proceeds to step S1602.

[0177] In step S1602, the first judgment logic 1503 judges whether or not the recognition result that is judged as refueling permission has continued for a predetermined time T1 (whether or not a predetermined permission condition has been satisfied for a certain period of time). If it is judged in step S1602 that the recognition result has continued for the predetermined time T1 (YES in step S1602), the process proceeds to step S1603.

[0178] In step S1603, the first decision logic 1503 decides that refueling is permitted, and the process proceeds to step S1614 in FIG. 16B.

[0179] On the other hand, if it is determined in step S1601 that the recognition result is not one that is determined to indicate permission for refueling (NO in step S1601), the process proceeds to step S1604. Alternatively, if it is determined in step S1602 that the predetermined time T1 has not continued (NO in step S1602), the process proceeds to step S1604.

[0180] In step S1604, the first determination logic 1503 determines whether or not a state in which the recognition result is not determined to be refueling permission has continued for a predetermined time T2 (where T2>T1).

[0181] In step S1604, if it is determined that the predetermined time T2 has not continued (NO in step S1604), the process returns to step S1601. On the other hand, if it is determined that the predetermined time T2 has continued (YES in step S1604), the process proceeds to step S1605.

[0182] In step S1605, the first judgment logic 1503 judges whether the currently input recognition result is a recognition result that is judged to disallow refueling (whether a predetermined disallowance condition is satisfied). Specifically, it judges whether a vehicle is recognized in the first area 210 and a refueler who is not holding the refueling nozzle in an appropriate position is recognized in the second area (fueling nozzle cannot be recognized).

[0183] If it is determined in step S1605 that the recognition result is such that refueling is not permitted (YES in step S1605), it is determined that a predetermined non-permission condition has been satisfied for a certain period of time, and the process proceeds to step S1606.

[0184] In step S1606, the first determination logic 1503 determines that refueling is not permitted, and after acquiring the first information, the process proceeds to step S1611 in FIG. 16B.

[0185] On the other hand, if it is determined in step S1605 that the recognition result is such that refueling is not permitted (NO in step S1605), the process proceeds to step S1617 in FIG. 16B.

[0186] In step S1611, the first judgment logic 1503 judges whether the currently input recognition result is a recognition result that is judged as refueling permission (whether a predetermined permission condition is satisfied). Specifically, it judges whether a vehicle is recognized in the first area 210 and a refueler holding the refueling nozzle in an appropriate posture is recognized in the second area.

[0187] In step S1611, when it is determined that the recognition result is not determined to be refueling permission (NO in step S1611), the first refueling monitoring unit 1009_1 ends the process using the first determination logic 1503.

[0188] On the other hand, if it is determined in step S1611 that the recognition result is determined to be permission for refueling (YES in step S1611), the process proceeds to step S1612.

[0189] In step S1612, the first judgment logic 1503 judges whether or not the recognition result that is judged as refueling permission has continued for a predetermined time T1 (whether or not a predetermined permission condition has been satisfied for a certain period of time). If it is judged in step S1612 that the recognition result has continued for the predetermined time T1 (YES in step S1612), the process proceeds to step S1613.

[0190] In step S1613, the first determination logic 1503 determines that refueling is permitted, and the process proceeds to step S1614.

[0191] In step S1614, judgment logic switching section 1502 judges whether or not it is in the first interval. If it is judged in step S1614 that it is in the first interval, it judges not to end the process by first judgment logic 1503 (judgment is NO in step S1614), and returns to step S1601 in FIG. 16A.

[0192] On the other hand, when it is determined in step S1614 that the vehicle is not in the first section, it is determined that the process using the first judgment logic 1503 is to be ended (YES is determined in step S1614). Then, the first refueling monitoring unit 1009_1 ends the process using the first judgment logic 1503.

[0193] On the other hand, if it is determined in step S1612 that the predetermined time T1 has not continued (NO in step S1612), the process proceeds to step S1615.

[0194] In step S1615, the first determination logic 1503 determines whether or not a state in which the recognition result is not determined to be refueling permission has continued for a predetermined time T3 (where T3>T1).

[0195] In step S1615, if it is determined that the predetermined time T3 has not continued (NO in step S1615), the process returns to step S1611. On the other hand, if it is determined that the predetermined time T3 has continued (YES in step S1615), it is determined that the predetermined disallowance condition has been satisfied for a certain period of time, and the process proceeds to step S1616.

[0196] In step S1616, the first determination logic 1503 determines that refueling is not permitted, and acquires the first information. After that, the first refueling monitoring unit 1009_1 ends the process using the first determination logic 1503.

[0197] On the other hand, when the process proceeds from step S1605 in Fig. 16A to step S1617 in Fig. 16B, the first determination logic 1503 determines that a predetermined disallowance condition has been satisfied for a certain period of time. Specifically, the first determination logic 1503 determines that a vehicle has not been recognized in the first area 210. Alternatively, the first determination logic 1503 determines that a fuel dispenser who is not holding the fuel dispenser nozzle in an appropriate position has been recognized in the second area 220 (fuel dispenser recognition impossible).

[0198] In step S1618, the first judgment logic 1503 judges that refueling is not permitted, and after acquiring the second information, the first refueling monitoring unit 1009_1 ends the process using the first judgment logic 1503.

[0199] (3-3) Processing flow using the second determination logic 1504 of the first refueling monitoring unit 1009_1 Next, the flow of processing using the second judgment logic 1504 of the first refueling monitoring unit 1009_1 will be described with reference to Fig. 17A to Fig. 17D. Fig. 17A to Fig. 17D are examples of first to fourth flowcharts showing the flow of processing using the second judgment logic of the first refueling monitoring unit.

[0200] In step S1701, the second judgment logic 1504 judges whether the currently input recognition result is a recognition result that is judged as refueling permission (whether a predetermined permission condition is satisfied). Specifically, it judges whether a vehicle is recognized in the first area 210 and a refueler holding the refueling nozzle in an appropriate posture is recognized in the second area.

[0201] In step S1701, if it is determined that the recognition result is determined to be permission for refueling (YES in step S1701), it is determined that a predetermined permission condition is satisfied, and the process proceeds to step S1702.

[0202] In step S1702, the second decision logic 1504 decides that refueling is permitted, and the process proceeds to step S1734 in FIG. 17D.

[0203] On the other hand, if it is determined in step S1701 that the recognition result is not one that is determined to indicate permission for refueling (NO in step S1701), the process proceeds to step S1703.

[0204] In step S1703, the second determination logic 1504 determines whether or not a state in which the recognition result is not determined to be permission for refueling has continued for a predetermined time T4.

[0205] In step S1703, if it is determined that the predetermined time T4 has not continued (NO in step S1703), the process returns to step S1701. On the other hand, if it is determined that the predetermined time T4 has continued (YES in step S1703), the process proceeds to step S1704.

[0206] In step S1704, the second judgment logic 1504 judges whether the currently input recognition result is a recognition result that is judged to disallow refueling (whether a predetermined disallowance condition is satisfied). Specifically, it judges whether a vehicle is recognized in the first area 210 and a refueler who is not holding the refueling nozzle in an appropriate position is recognized in the second area (fueling nozzle cannot be recognized).

[0207] In step S1704, if it is determined that the recognition result is such that refueling is not permitted (NO in step S1704), the process proceeds to step S1721 in FIG. 17C.

[0208] On the other hand, if it is determined in step S1704 that the recognition result is such that refueling is not permitted (YES in step S1704), it is determined that the predetermined non-permission condition has been satisfied for a certain period of time, and the process proceeds to step S1705.

[0209] In step S1705, the second determination logic 1504 obtains the warning information.

[0210] In step S1706, the second judgment logic 1504 judges whether the currently input recognition result is a recognition result that is judged as refueling permission (whether a predetermined permission condition is satisfied). Specifically, it judges whether a vehicle is recognized in the first area 210 and a refueler holding the refueling nozzle in an appropriate posture is recognized in the second area.

[0211] In step S1706, if it is determined that the recognition result is determined to be permission for refueling (YES in step S1706), the process proceeds to step S1707.

[0212] In step S1707, the second judgment logic 1504 judges whether the recognition result that is judged to be refueling permission has continued for a predetermined time T1. If it is judged that the recognition result has continued for the predetermined time T1 (YES in step S1707), it is judged that the predetermined permission condition has been satisfied for a certain period of time, and the process proceeds to step S1708.

[0213] In step S1708, the second decision logic 1504 decides that refueling is permitted, and the process proceeds to step S1734 in FIG. 17D.

[0214] On the other hand, if it is determined in step S1706 that the recognition result is not one that is determined to indicate permission for refueling (NO in step S1706), the process proceeds to step S1709. Alternatively, if it is determined in step S1707 that the predetermined time T1 has not continued (NO in step S1707), the process proceeds to step S1709.

[0215] In step S1709, the second determination logic 1504 determines whether or not a state in which the recognition result is not determined to be refueling permission has continued for a predetermined time T2 (where T2>T1).

[0216] In step S1709, if it is determined that the predetermined time T2 has not continued (NO in step S1709), the process returns to step S1706. On the other hand, if it is determined that the predetermined time T2 has continued (YES in step S1709), it is determined that the predetermined disallowance condition has been satisfied for a certain period of time, and the process proceeds to step S1711 in FIG. 17B.

[0217] In step S1711, the second determination logic 1504 determines that refueling is not permitted, and after acquiring the first information, the process proceeds to step S1712.

[0218] In step S1712, the second judgment logic 1504 judges whether or not the currently input recognition result has become a recognition result that is judged as refueling permission (whether or not a predetermined permission condition is satisfied) within the predetermined time T2. Specifically, it judges whether or not a state has been reached in which a vehicle is recognized in the first area 210 and a refueler holding the refueling nozzle in an appropriate posture is recognized in the second area within the predetermined time T2.

[0219] In step S1712, if it is determined that the currently input recognition result has become a recognition result that is determined to be refueling permission within the specified time T2 (if YES in step S1712), it is determined that the specified permission conditions have been satisfied, and the process proceeds to step S1713.

[0220] In step S1713, the second decision logic 1504 decides that refueling is permitted, and the process proceeds to step S1734 in FIG. 17D.

[0221] On the other hand, if it is determined in step S1712 that the recognition result is not judged to be refueling permission within the predetermined time T2 (NO in step S1712), the first refueling monitoring unit 1009_1 terminates the processing using the second judgment logic 1504.

[0222] 17C, the second determination logic 1504 determines that a predetermined disallowance condition has been satisfied for a certain period of time. Specifically, the second determination logic 1504 determines that a vehicle has not been recognized in the first area 210. Alternatively, the second determination logic 1504 determines that a fuel dispenser who is not holding the fuel dispenser nozzle in an appropriate position has been recognized in the second area 220 (fuel dispenser recognition not possible).

[0223] In step S1722, the second determination logic 1504 obtains the warning information.

[0224] In step S1723, the second judgment logic 1504 judges whether the currently input recognition result is a recognition result that is judged as refueling permission (whether a predetermined permission condition is satisfied). Specifically, it judges whether a vehicle is recognized in the first area 210 and a refueler holding the refueling nozzle in an appropriate posture is recognized in the second area.

[0225] In step S1723, if it is determined that the recognition result is determined to be permission for refueling (YES in step S1723), the process proceeds to step S1724.

[0226] In step S1724, the second judgment logic 1504 judges whether the recognition result that is judged to be refueling permission has continued for a predetermined time T1. If it is judged that the recognition result has continued for the predetermined time T1 in step S1724 (YES in step S1724), it is judged that the predetermined permission condition has been satisfied for a certain period of time, and the process proceeds to step S1725.

[0227] In step S1725, the second decision logic 1504 decides that refueling is permitted, and the process proceeds to step S1734 in FIG. 17D.

[0228] On the other hand, if it is determined in step S1723 that the recognition result is not one that is determined to indicate permission for refueling (NO in step S1723), the process proceeds to step S1726. Alternatively, if it is determined in step S1724 that the predetermined time T1 has not continued (NO in step S1724), the process proceeds to step S1726.

[0229] In step S1726, the second determination logic 1504 determines whether or not a state in which the recognition result is not determined to be refueling permission has continued for a predetermined time T2 (where T2>T1).

[0230] In step S1726, if it is determined that the predetermined time T2 has not continued (NO in step S1726), the process returns to step S1723. On the other hand, if it is determined that the predetermined time T2 has continued (YES in step S1726), it is determined that the predetermined disallowance condition has been satisfied for a certain period of time, and the process proceeds to step S1731 in FIG. 17D.

[0231] In step S1731, the second determination logic 1504 determines that refueling is not permitted, and after acquiring the second information, the process proceeds to step S1732.

[0232] In step S1732, the second judgment logic 1504 judges whether or not the currently input recognition result has become a recognition result that is judged as refueling permission (whether or not a predetermined permission condition is satisfied) within the predetermined time T2. Specifically, it judges whether or not a state has been reached in which a vehicle is recognized in the first area 210 and a refueler holding the refueling nozzle in an appropriate posture is recognized in the second area within the predetermined time T2.

[0233] In step S1732, if it is determined that the currently input recognition result has become a recognition result that is determined to be refueling permission within the specified time T2 (if YES in step S1732), it is determined that the specified permission conditions have been satisfied, and the process proceeds to step S1733.

[0234] In step S1733, the second decision logic 1504 decides that refueling is permitted, and the process proceeds to step S1734.

[0235] In step S1734, judgment logic switching section 1502 judges whether or not it is in the second interval. If it is judged in step S1734 that it is in the second interval, it judges not to end the process by second judgment logic 1504 (judgment is NO in step S1734), and returns to step S1701 in FIG. 17A.

[0236] On the other hand, when it is determined in step S1734 that the vehicle is not in the second section, it is determined to end the processing using the second judgment logic 1504 (determined as YES in step S1734). Then, the first refueling monitoring unit 1009_1 ends the processing using the second judgment logic 1504.

[0237] Also, in step S1732, if it is not determined that the recognition result indicates that refueling is permitted within the specified time T2 (if NO in step S1732), the first refueling monitoring unit 1009_1 terminates the processing using the second determination logic 1504.

[0238] (3-4) Processing flow using the third determination logic 1505 of the first refueling monitoring unit 1009_1 Next, the flow of processing using the third determination logic 1505 of the first refueling monitoring unit 1009_1 will be described with reference to Fig. 18A and Fig. 18B. Fig. 18A and Fig. 18B are examples of first and second flowcharts showing the flow of processing using the third determination logic of the first refueling monitoring unit.

[0239] In step S1801, the third judgment logic 1505 judges whether the currently input recognition result is a recognition result that is judged as refueling permission (whether a predetermined permission condition is satisfied). Specifically, it judges whether a vehicle is recognized in the first area 210 and a refueler holding the refueling nozzle in an appropriate posture is recognized in the second area.

[0240] In step S1801, if it is determined that the recognition result is determined to be permission for refueling (YES in step S1801), it is determined that a predetermined permission condition is satisfied, and the process proceeds to step S1802.

[0241] In step S1802, the third decision logic 1505 decides that refueling is permitted, and the process proceeds to step S1813 in FIG. 18B.

[0242] On the other hand, if it is determined in step S1801 that the recognition result is not one that is determined to indicate permission for refueling (NO in step S1801), the process proceeds to step S1803.

[0243] In step S1803, the third determination logic 1505 determines whether or not a state in which the recognition result is not determined to be refueling permission has continued for a predetermined time T5.

[0244] In step S1803, if it is determined that the predetermined time T5 has not continued (NO in step S1803), the process returns to step S1801. On the other hand, if it is determined that the predetermined time T5 has continued (YES in step S1803), the process proceeds to step S1804.

[0245] In step S1804, the third judgment logic 1505 judges whether the currently input recognition result is a recognition result that is judged to disallow refueling (whether a predetermined disallowance condition is satisfied). Specifically, it judges whether a vehicle is recognized in the first area 210 and a refueler who is not holding the refueling nozzle in an appropriate position is recognized in the second area (fueling nozzle cannot be recognized).

[0246] In step S1804, if it is determined that the recognition result is such that refueling is not permitted (NO in step S1804), the process proceeds to step S1806.

[0247] In step S1806, the third determination logic 1505 determines that a predetermined disallowance condition has been satisfied for a certain period of time. Specifically, the third determination logic 1505 determines that a vehicle has not been recognized in the first area 210. Alternatively, the third determination logic 1505 determines that a fuel dispenser who is not holding the fuel dispenser nozzle in an appropriate position has been recognized in the second area 220 (fuel dispenser recognition impossible).

[0248] In step S1807, the third determination logic 1505 determines that refueling is not permitted, and after acquiring the second information, the process proceeds to step S1811 in FIG. 18B.

[0249] On the other hand, if it is determined in step S1804 that the recognition result is such that refueling is not permitted (YES in step S1804), it is determined that the predetermined non-permission condition has been satisfied for a certain period of time, and the process proceeds to step S1805.

[0250] In step S1805, the third decision logic 1505 decides that refueling is not permitted, and after acquiring the first information, the process proceeds to step S1811 in FIG. 18B.

[0251] In step S1811, the third judgment logic 1505 judges whether or not the currently input recognition result has become a recognition result that is judged as refueling permission (whether or not a predetermined permission condition is satisfied) within the predetermined time T2. Specifically, it judges whether or not a state has been reached in which a vehicle is recognized in the first area 210 and a refueler holding the refueling nozzle in an appropriate posture is recognized in the second area within the predetermined time T2.

[0252] In step S1811, if it is determined that the currently input recognition result has become a recognition result that is determined to be refueling permission within the specified time T2 (if YES in step S1811), it is determined that the specified permission conditions have been satisfied, and the process proceeds to step S1812.

[0253] In step S1812, the third decision logic 1505 decides that refueling is permitted, and the process proceeds to step S1813.

[0254] In step S1813, judgment logic switching section 1502 judges whether or not it is in the third interval. If it is judged in step S1813 that it is in the third interval, it judges not to end the process by third judgment logic 1505 (judgment is NO in step S1813), and returns to step S1801 in FIG. 18A.

[0255] On the other hand, when it is determined in step S1813 that the vehicle is not in the third section, it is determined to end the processing by the third judgment logic 1505 (determined as YES in step S1813). Then, the first refueling monitoring unit 1009_1 ends the processing using the third judgment logic 1505.

[0256] On the other hand, if it is determined in step S1811 that the currently input recognition result has not become a recognition result that is determined to indicate permission for refueling within the predetermined time T2 (NO in step S1811), the process proceeds to step S1814.

[0257] In step S1814, the third judgment logic 1505 judges whether or not the state in which the recognition result is not judged as refueling permission has continued for a predetermined time T6. If it is judged in step S1814 that the state in which the recognition result is not judged as refueling permission has not continued for the predetermined time T6 (NO in step S1814), the third judgment logic 1505 waits until the state continues for the predetermined time T6.

[0258] On the other hand, if it is determined in step S1814 that the state in which the recognition result is not determined to be refueling permission has continued for the predetermined time T6 (YES in step S1814), the process proceeds to step S1815.

[0259] In step S1815, the third determination logic 1505 acquires the confirmation request information, and the first refueling monitoring unit 1009_1 ends the process using the third determination logic 1505.

[0260] (3-5) Processing flow using other determination logic of the first refueling monitoring unit 1009_1 Next, the flow of processing using other judgment logic of the first refueling monitoring unit 1009_1 (processing using the plastic tank judgment unit 1506, processing using the fire judgment unit 1507) will be described with reference to Fig. 19. Fig. 19 is an example of a flowchart showing the flow of processing using the plastic tank judgment unit of the first refueling monitoring unit and processing using the fire judgment unit.

[0261] First, a process flow using the plastic tank determination unit 1506 of the first refueling monitoring unit 1009_1 will be described.

[0262] 19A, in step S1901, the plastic container determination unit 1506 determines whether or not a container is included in the currently input recognition result (whether or not a predetermined disapproval condition is satisfied). If it is determined in step S1901 that a container is included (YES in step S1901), it is determined that the predetermined disapproval condition is satisfied, and the process proceeds to step S1902.

[0263] In step S1902, the plastic tank determination unit 1506 determines that refueling is not permitted, and after acquiring the third information, the process proceeds to step S1903.

[0264] On the other hand, if it is determined in step S1901 that a container is not included (NO in step S1901), the process proceeds directly to step S1903.

[0265] In step S1903, the plastic tank determination unit 1506 determines whether or not to end the process. If the process corresponds to any of the first to third sections, in step S1903, the plastic tank determination unit 1506 determines to continue the process (determines NO in step S1903) and returns to step S1901.

[0266] On the other hand, if the vehicle does not fall into any of the first to third sections, the first refueling monitoring unit 1009_1 ends the process using the plastic tank determination unit 1506 in step S1903.

[0267] Next, a process flow using the plastic tank determination unit 1506 of the second refueling monitoring unit 1009_2 will be described.

[0268] 19(b), the fire determination unit 1507 determines whether or not a temperature equal to or higher than a predetermined threshold has been measured (whether or not a predetermined disallowance condition is satisfied) in an area other than the first area 210 for the currently input thermal sensor data. Note that the reason for including an area other than the first area 210 is to avoid erroneous determination of fire detection since the hood of a vehicle included in the first area 210 may be hot.

[0269] If it is determined in step S1911 that a temperature equal to or higher than the predetermined threshold has been measured (YES in step S1911), it is determined that a predetermined disallowance condition has been met, and the process proceeds to step S1912.

[0270] In step S1912, the fire determination unit 1507 determines that refueling is not permitted, and after acquiring the fourth information, the process proceeds to step S1913.

[0271] On the other hand, if it is determined in step S1911 that a temperature equal to or higher than the predetermined threshold has not been measured (NO in step S1911), the process proceeds directly to step S1913.

[0272] In step S1913, the fire determination unit 1507 determines whether or not to end the process. If it corresponds to any of the first to third sections, in step S1913, the fire determination unit 1507 determines to continue the process (determines NO in step S1913), and returns to step S1911.

[0273] On the other hand, if the detected area does not fall into any of the first to third sections, the first refueling monitoring unit 1009_1 ends the process using the fire determination unit 1507 in step S1913.

[0274] (3-6) Processing flow using the first determination logic 1513 of the second refueling monitoring unit 1009_2 Next, the flow of processing using the first judgment logic 1513 of the second refueling monitoring unit 1009_2 will be described with reference to Fig. 20A and Fig. 20B. Fig. 20A and Fig. 20B are examples of first and second flowcharts showing the flow of processing using the first judgment logic of the second refueling monitoring unit.

[0275] In step S2001, the first judgment logic 1513 judges whether the currently input recognition result is a recognition result that is judged as refueling permission (whether a predetermined permission condition is satisfied). Specifically, it judges whether the container is recognized and the refueler who holds the refueling nozzle in an appropriate posture is recognized.

[0276] In step S2001, if it is determined that the recognition result is determined to be permission for refueling (YES in step S2001), the process proceeds to step S2002.

[0277] In step S2002, the first judgment logic 1513 judges whether or not the recognition result that is judged as refueling permission has continued for a predetermined time T1 (whether or not a predetermined permission condition has been satisfied for a certain period of time). If it is judged in step S2002 that the recognition result has continued for the predetermined time T1 (if YES in step S2002), the process proceeds to step S2003.

[0278] In step S2003, the first determination logic 1513 determines that refueling is permitted, and the process proceeds to step S2014 in FIG. 20B.

[0279] On the other hand, if it is determined in step S2001 that the recognition result is not one that is determined to be refueling permission (NO in step S2001), the process proceeds to step S2004. Alternatively, if it is determined in step S2002 that the predetermined time T1 has not continued (NO in step S2002), the process proceeds to step S2004.

[0280] In step S2004, the first determination logic 1513 determines whether or not a state in which the recognition result is not determined to be refueling permission has continued for a predetermined time T2 (where T2>T1).

[0281] In step S2004, if it is determined that the predetermined time T2 has not continued (NO in step S2004), the process returns to step S2001. On the other hand, if it is determined that the predetermined time T2 has continued (YES in step S2004), the process proceeds to step S2005.

[0282] In step S2005, the first judgment logic 1513 judges whether the currently input recognition result is a recognition result that is judged to be refueling not permitted (whether a predetermined refusal condition is satisfied). Specifically, it judges whether the container is recognized and whether a refueler who is not holding the refueling nozzle in an appropriate posture is recognized (whether the refueling nozzle cannot be recognized).

[0283] If it is determined in step S2005 that the recognition result is such that refueling is not permitted (YES in step S2005), it is determined that a predetermined non-permission condition has been satisfied for a certain period of time, and the process proceeds to step S2006.

[0284] In step S2006, the first determination logic 1513 determines that refueling is not permitted, and after acquiring the first information, the process proceeds to step S2011 in FIG. 20B.

[0285] On the other hand, if it is determined in step S2005 that the recognition result is such that refueling is not permitted (NO in step S2005), the process proceeds to step S2017 in FIG. 20B.

[0286] In step S2011, the first judgment logic 1513 judges whether the currently input recognition result is a recognition result that is judged as refueling permission (whether a predetermined permission condition is satisfied). Specifically, it judges whether the container is recognized and whether a refueler who holds the refueling nozzle in an appropriate posture is recognized.

[0287] In step S2011, if it is determined that the recognition result is not one that is determined to indicate permission for refueling (NO in step S2011), the process proceeds to step S2017.

[0288] On the other hand, if it is determined in step S2011 that the recognition result is determined to be refueling permission (YES in step S2011), the second refueling monitoring unit 1009_2 ends the process using the first determination logic 1513.

[0289] In step S2012, the first judgment logic 1513 judges whether or not the recognition result that is judged as refueling permission has continued for a predetermined time T1 (whether or not a predetermined permission condition has been satisfied for a certain period of time). If it is judged in step S2012 that the recognition result has continued for the predetermined time T1 (YES in step S2012), the process proceeds to step S2013.

[0290] In step S2013, the first determination logic 1513 determines that refueling is permitted, and the process proceeds to step S2014.

[0291] In step S2014, judgment logic switching unit 1512 judges whether or not it is in the first interval. If it is judged in step S2014 that it is in the first interval, it judges not to end the process by first judgment logic 1513 (judgment is NO in step S2014), and returns to step S2001 in FIG. 20A.

[0292] On the other hand, when it is determined in step S2014 that the vehicle is not in the first interval, it is determined that the process using the first judgment logic 1513 is to be ended (determined as YES in step S2014). Then, the second refueling monitoring unit 1009_2 ends the process using the first judgment logic 1513.

[0293] On the other hand, if it is determined in step S2012 that the predetermined time T1 has not continued (NO in step S2012), the process proceeds to step S2015.

[0294] In step S2015, the first determination logic 1513 determines whether or not a state in which the recognition result is not determined to be refueling permission has continued for a predetermined time T3 (where T3>T1).

[0295] In step S2015, if it is determined that the predetermined time T3 has not continued (NO in step S2015), the process returns to step S2011. On the other hand, in step S2015, if it is determined that the predetermined time T3 has continued (YES in step S2015), it is determined that the predetermined non-permission condition has been satisfied for a certain period of time. Then, the second refueling monitoring unit 1009_2 ends the process using the first judgment logic 1513.

[0296] On the other hand, when the process proceeds from step S2005 in Fig. 20A to step S2017 in Fig. 20B, the first judgment logic 1513 judges that a predetermined disallowance condition has been satisfied for a certain period of time. Specifically, the first judgment logic 1513 judges that the container has not been recognized. Alternatively, the first judgment logic 1513 judges that a fuel dispenser who does not hold the fuel dispenser nozzle in an appropriate posture has been recognized (fuel dispenser recognition impossible).

[0297] In step S2018, the first judgment logic 1513 judges that refueling is not permitted and acquires the second information, and the second refueling monitoring unit 1009_2 ends the process using the first judgment logic 1513.

[0298] (3-7) Processing flow using the second determination logic 1514 of the second refueling monitoring unit 1009_2 Next, the flow of processing using the second judgment logic 1514 of the second refueling monitoring unit 1009_2 will be described with reference to Fig. 21A to Fig. 21D. Fig. 21A to Fig. 21D are examples of first to fourth flowcharts showing the flow of processing using the second judgment logic of the second refueling monitoring unit.

[0299] In step S2101, the second judgment logic 1514 judges whether the currently input recognition result is a recognition result that is judged as refueling permission (whether a predetermined permission condition is satisfied). Specifically, it judges whether the container is recognized and the refueler who holds the refueling nozzle in an appropriate posture is recognized.

[0300] In step S2101, if it is determined that the recognition result is determined to be permission for refueling (YES in step S2101), it is determined that a predetermined permission condition is satisfied, and the process proceeds to step S2102.

[0301] In step S2102, the second decision logic 1514 decides that refueling is permitted, and the process proceeds to step S2134 in FIG. 21D.

[0302] On the other hand, if it is determined in step S2101 that the recognition result is not one that is determined to indicate permission for refueling (NO in step S2101), the process proceeds to step S2103.

[0303] In step S2103, the second determination logic 1514 determines whether or not a state in which the recognition result is not determined to be refueling permission has continued for a predetermined time T4.

[0304] In step S2103, if it is determined that the predetermined time T4 has not continued (NO in step S2103), the process returns to step S2101. On the other hand, if it is determined that the predetermined time T4 has continued (YES in step S2103), the process proceeds to step S2104.

[0305] In step S2104, the second judgment logic 1514 judges whether the currently input recognition result is a recognition result that is judged to be refueling not permitted (whether a predetermined refusal condition is satisfied). Specifically, it judges whether the container is recognized and whether a refueler who is not holding the refueling nozzle in an appropriate posture is recognized (fueling nozzle cannot be recognized).

[0306] In step S2104, if it is determined that the recognition result is such that refueling is not permitted (NO in step S2104), the process proceeds to step S2121 in FIG. 21C.

[0307] On the other hand, if it is determined in step S2104 that the recognition result is such that refueling is not permitted (YES in step S2104), it is determined that the predetermined non-permission condition has been satisfied for a certain period of time, and the process proceeds to step S2105.

[0308] In step S2105, the second determination logic 1514 obtains the warning information.

[0309] In step S2106, the second judgment logic 1514 judges whether the currently input recognition result is a recognition result that is judged as refueling permission (whether a predetermined permission condition is satisfied). Specifically, it judges whether the container is recognized and the refueler who holds the refueling nozzle in an appropriate posture is recognized.

[0310] In step S2106, if it is determined that the recognition result is determined to be permission for refueling (YES in step S2106), the process proceeds to step S2107.

[0311] In step S2107, the second judgment logic 1514 judges whether the recognition result that is judged to be refueling permission has continued for a predetermined time T1. If it is judged that the recognition result has continued for the predetermined time T1 in step S2107 (YES in step S2107), it is judged that the predetermined permission condition has been satisfied for a certain period of time, and the process proceeds to step S2108.

[0312] In step S2108, the second decision logic 1514 decides that refueling is permitted, and the process proceeds to step S2134 in FIG. 21D.

[0313] On the other hand, if it is determined in step S2106 that the recognition result is not one that is determined to indicate permission for refueling (NO in step S2106), the process proceeds to step S2109. Alternatively, if it is determined in step S2107 that the predetermined time T1 has not continued (NO in step S2107), the process proceeds to step S2109.

[0314] In step S2109, the second determination logic 1514 determines whether or not a state in which the recognition result is not determined to be refueling permission has continued for a predetermined time T2 (where T2>T1).

[0315] In step S2109, if it is determined that the predetermined time T2 has not continued (NO in step S2109), the process returns to step S2106. On the other hand, if it is determined that the predetermined time T2 has continued (YES in step S2109), it is determined that the predetermined disallowance condition has been satisfied for a certain period of time, and the process proceeds to step S2111 in FIG. 21B.

[0316] In step S2111, the second determination logic 1514 determines that refueling is not permitted, and after acquiring the first information, the process proceeds to step S2112.

[0317] In step S2112, the second judgment logic 1514 judges whether or not the currently input recognition result has become a recognition result that is judged as refueling permission (whether or not a predetermined permission condition is satisfied) within the predetermined time T2. Specifically, it judges whether or not a state has been reached in which the container has been recognized and the refueler holding the refueling nozzle in an appropriate posture has been recognized within the predetermined time T2.

[0318] In step S2112, if it is determined that the currently input recognition result has become a recognition result that is determined to be refueling permission within the specified time T2 (if YES in step S2112), it is determined that the specified permission condition has been satisfied, and the process proceeds to step S2113.

[0319] In step S2113, the second decision logic 1514 decides that refueling is permitted, and the process proceeds to step S2134 in FIG. 21D.

[0320] On the other hand, if it is determined in step S2112 that the recognition result is not determined to be refueling permission within the predetermined time T2 (NO in step S2112), the second refueling monitoring unit 1009_2 terminates the processing using the second determination logic 1514.

[0321] On the other hand, when the process proceeds from step S2104 to step S2121 in Fig. 21C, the second judgment logic 1514 judges that a predetermined disallowance condition has been satisfied for a certain period of time. Specifically, the second judgment logic 1514 judges that the container has not been recognized. Alternatively, the second judgment logic 1514 judges that a fuel dispenser who does not hold the fuel dispenser nozzle in an appropriate position has been recognized (fuel dispenser recognition impossible).

[0322] In step S2122, the second determination logic 1504 obtains the warning information.

[0323] In step S2123, the second judgment logic 1514 judges whether the currently input recognition result is a recognition result that is judged as refueling permission (whether a predetermined permission condition is satisfied). Specifically, it judges whether the container is recognized and the refueler who holds the refueling nozzle in an appropriate posture is recognized.

[0324] In step S2123, if it is determined that the recognition result is determined to be permission for refueling (YES in step S2123), the process proceeds to step S2124.

[0325] In step S2124, second judgment logic 1514 judges whether the recognition result that is judged to be refueling permission has continued for a predetermined time T1. If it is judged that the recognition result has continued for the predetermined time T1 in step S2124 (YES in step S2124), it is judged that the predetermined permission condition has been satisfied for a certain period of time, and the process proceeds to step S2125.

[0326] In step S2125, the second decision logic 1514 decides that refueling is permitted, and the process proceeds to step S2134 in FIG. 21D.

[0327] On the other hand, if it is determined in step S2123 that the recognition result is not one that is determined to indicate permission for refueling (NO in step S2123), the process proceeds to step S2126. Alternatively, if it is determined in step S2124 that the predetermined time T1 has not continued (NO in step S2124), the process proceeds to step S2126.

[0328] In step S2126, the second determination logic 1514 determines whether or not a state in which the recognition result is not determined to be refueling permission has continued for a predetermined time T2 (where T2>T1).

[0329] In step S2126, if it is determined that the predetermined time T2 has not continued (NO in step S2126), the process returns to step S2123. On the other hand, if it is determined that the predetermined time T2 has continued (YES in step S2126), it is determined that the predetermined disallowance condition has been satisfied for a certain period of time, and the process proceeds to step S2131 in FIG. 21D.

[0330] In step S2131, the second determination logic 1514 determines that refueling is not permitted, and after acquiring the second information, the process proceeds to step S2132.

[0331] In step S2132, the second judgment logic 1514 judges whether or not the currently input recognition result has become a recognition result that is judged as refueling permission (whether or not a predetermined permission condition is satisfied) within the predetermined time T2. Specifically, it judges whether or not a state has been reached in which a container has been recognized and a refueler holding the refueling nozzle in an appropriate posture has been recognized within the predetermined time T2.

[0332] In step S2132, if it is determined that the currently input recognition result has become a recognition result that is determined to be refueling permission within the specified time T2 (if YES in step S2132), it is determined that the specified permission conditions have been satisfied, and the process proceeds to step S2133.

[0333] In step S2133, the second decision logic 1514 decides that refueling is permitted, and the process proceeds to step S2134.

[0334] In step S2134, judgment logic switching section 1512 judges whether or not it is in the second interval. If it is judged in step S2134 that it is in the second interval, it judges not to end the process by second judgment logic 1514 (judgment is NO in step S2134), and returns to step S2101 in FIG. 21A.

[0335] On the other hand, when it is determined in step S2134 that the vehicle is not in the second interval, it is determined to end the process using the second judgment logic 1514 (determined as YES in step S2134). Then, the second refueling monitoring unit 1009_2 ends the process using the second judgment logic 1514.

[0336] Also, in step S2132, if it is not determined that the recognition result indicates that refueling is permitted within the predetermined time T2 (NO in step S2132), the second refueling monitoring unit 1009_2 terminates the processing using the second determination logic 1514.

[0337] (3-8) Processing flow using the third determination logic 1515 of the second refueling monitoring unit 1009_2 Next, the flow of processing using the third determination logic 1515 of the second refueling monitoring unit 1009_2 will be described with reference to Fig. 22A and Fig. 22B. Fig. 22A and Fig. 22B are examples of the first and second flowcharts showing the flow of processing using the third determination logic of the second refueling monitoring unit.

[0338] In step S2201, the third judgment logic 1515 judges whether the currently input recognition result is a recognition result that is judged as refueling permission (whether a predetermined permission condition is satisfied). Specifically, it judges whether the container is recognized and the refueler who holds the refueling nozzle in an appropriate posture is recognized.

[0339] In step S2201, if it is determined that the recognition result is determined to be permission for refueling (YES in step S2201), it is determined that a predetermined permission condition is satisfied, and the process proceeds to step S2202.

[0340] In step S2202, the third decision logic 1515 decides that refueling is permitted, and the process proceeds to step S2213 in FIG. 22B.

[0341] On the other hand, if it is determined in step S2201 that the recognition result is not one that is determined to indicate permission for refueling (NO in step S2201), the process proceeds to step S2203.

[0342] In step S2203, the third determination logic 1515 determines whether or not a state in which the recognition result is not determined to be refueling permission has continued for a predetermined time T5.

[0343] In step S2203, if it is determined that the predetermined time T5 has not continued (NO in step S2203), the process returns to step S2201. On the other hand, if it is determined that the predetermined time T5 has continued (YES in step S2203), the process proceeds to step S2204.

[0344] In step S2204, the third judgment logic 1515 judges whether the currently input recognition result is a recognition result that is judged to be refueling not permitted (whether a predetermined refusal condition is satisfied). Specifically, it judges whether the container is recognized and whether a refueler who is not holding the refueling nozzle in an appropriate posture is recognized (fueling nozzle cannot be recognized).

[0345] In step S2204, if it is determined that the recognition result is such that refueling is not permitted (NO in step S2204), the process proceeds to step S2206.

[0346] In step S2206, the third determination logic 1515 determines that a predetermined disallowance condition has been satisfied for a certain period of time. Specifically, the third determination logic 1515 determines that the container has not been recognized. Alternatively, the third determination logic 1515 determines that a fuel dispenser who does not hold the fuel dispenser nozzle in an appropriate position has been recognized (fuel dispenser recognition not possible).

[0347] In step S2207, the third decision logic 1515 decides that refueling is not permitted, and after acquiring the second information, the process proceeds to step S2211 in FIG. 22B.

[0348] On the other hand, if it is determined in step S2204 that the recognition result is such that refueling is not permitted (YES in step S2204), it is determined that the predetermined non-permission condition has been satisfied for a certain period of time, and the process proceeds to step S2205.

[0349] In step S2205, the third decision logic 1515 decides that refueling is not permitted, and after acquiring the first information, the process proceeds to step S2211 in FIG. 22B.

[0350] In step S2211, the third judgment logic 1515 judges whether or not the currently input recognition result becomes a recognition result that is judged as refueling permission (whether or not a predetermined permission condition is satisfied) within the predetermined time T2. Specifically, it judges whether or not a state in which a container is recognized and a refueler holding a refueling nozzle in an appropriate posture is recognized within the predetermined time T2.

[0351] In step S2211, if it is determined that the currently input recognition result has become a recognition result that is determined to be refueling permission within the specified time T2 (if YES in step S2211), it is determined that the specified permission condition has been satisfied, and the process proceeds to step S2212.

[0352] In step S2212, the third decision logic 1515 decides that refueling is permitted, and the process proceeds to step S2213.

[0353] In step S2213, judgment logic switching section 1512 judges whether or not it is in the third interval. If it is judged in step S2213 that it is in the third interval, it judges not to end the process by third judgment logic 1515 (NO in step S2213), and returns to step S2201 in FIG. 22A.

[0354] On the other hand, when it is determined in step S2213 that the vehicle is not in the third section, it is determined to end the process using the third judgment logic 1515 (determined as YES in step S2213). Then, the second refueling monitoring unit 1009_2 ends the process using the third judgment logic 1515.

[0355] On the other hand, if it is determined in step S2211 that the currently input recognition result has not become a recognition result that is determined to be refueling permission within the predetermined time T2 (NO in step S2211), the process proceeds to step S2214.

[0356] In step S2214, the third judgment logic 1515 judges whether the state where the recognition result is not judged as refueling permission has continued for a predetermined time T6. If it is judged in step S2214 that the state where the recognition result is not judged as refueling permission has not continued for the predetermined time T6 (NO in step S2214), the third judgment logic 1515 waits until the state has continued for the predetermined time T6.

[0357] On the other hand, if it is determined in step S2214 that the state in which the recognition result is not determined to be refueling permission has continued for the predetermined time T6 (YES in step S2214), the process proceeds to step S2215.

[0358] In step S2215, the third determination logic 1515 acquires the confirmation request information, and the second refueling monitoring unit 1009_2 ends the process using the third determination logic 1515.

[0359] (3-9) Processing flow using other determination logic of the second refueling monitoring unit 1009_2 Next, a flow of processing using another judgment logic of the second refueling monitoring unit 1009_2 (processing using the fire judgment unit 1517) will be described with reference to Fig. 23. Fig. 23 is an example of a flowchart showing a flow of processing using the fire judgment unit of the second refueling monitoring unit.

[0360] In step S2301, the fire determination unit 1517 determines whether or not a temperature equal to or higher than a predetermined threshold has been measured from the currently input thermal sensor data (whether or not a predetermined disallowance condition is satisfied).

[0361] In step S2301, if it is determined that a temperature equal to or higher than the predetermined threshold has been measured (YES in step S2301), it is determined that a predetermined disallowance condition has been met, and the process proceeds to step S2302.

[0362] In step S2302, the fire determination unit 1517 determines that refueling is not permitted, and after acquiring the fourth information, the process proceeds to step S2303.

[0363] On the other hand, if it is determined in step S2301 that a temperature equal to or higher than the predetermined threshold has not been measured (NO in step S2301), the process proceeds directly to step S2303.

[0364] In step S2303, the fire determination unit 1517 determines whether or not to end the process. If the process corresponds to any of the first to third sections, in step S2303, the fire determination unit 1517 determines to continue the process (determines NO in step S2303), and returns to step S2301.

[0365] On the other hand, if the detected area does not fall into any of the first to third sections, the second refueling monitoring unit 1009_2 ends the process using the fire determination unit 1517 in step S2303.

[0366] (4) Details of the output control unit 1010 Next, as details of the output control unit 1010, a specific example of the functional configuration and processing of the output control unit 1010 will be described.

[0367] (4-1) Details of the Functional Configuration of the Output Control Unit 1010 24 is a diagram showing an example of the functional configuration of an output control unit of an information processing device. As shown in FIG. 24, the output control unit 1010 includes an output information acquisition unit 2401 and a transmission unit 2402.

[0368] The output information acquisition unit 2401 receives the following information from the first refueling monitoring unit 1009_1: - information indicating permission to refuel, - Information indicating refueling denial and Information 1 to 4, Warning information, Verification request information, Get the.

[0369] In addition, the output information acquisition unit 2401 receives from the second refueling monitoring unit 1009_2, - information indicating permission to refuel, - Information indicating refueling denial and the first information, second information, and fourth information, Warning information, Verification request information, Get the.

[0370] In addition, the output information acquisition unit 2401 notifies the transmission unit 2402 of the information acquired from the first refueling monitoring unit 1009_1. At this time, the output information acquisition unit 2401 refers to the output information storage unit 2403, -Cause information corresponding to the first information to the fourth information (information indicating the cause of refusal to refuel); Messages corresponding to the first to fourth information, Display information corresponding to the first information to the fourth information; -Messages corresponding to warning information, - Display information corresponding to the warning information; Cause information corresponding to the confirmation request information; A message corresponding to the confirmation request information; Display information corresponding to the confirmation request information; Then, the output information acquiring unit 2401 converts the first to fourth information, warning information, and confirmation request information into corresponding cause information, message, and display information, and notifies the transmitting unit 2402 of them.

[0371] Similarly, the output information acquisition unit 2401 notifies the transmission unit 2402 of the information acquired from the second refueling monitoring unit 1009_2. At this time, the output information acquisition unit 2401 refers to the output information storage unit 2403 and -Cause information corresponding to the first information, second information, and fourth information (information indicating the cause of refusal to refuel); messages corresponding to the first information, the second information, and the fourth information; Display information corresponding to the first information, the second information, and the fourth information; -Messages corresponding to warning information, - Display information corresponding to the warning information; Cause information corresponding to the confirmation request information; A message corresponding to the confirmation request information; Display information corresponding to the confirmation request information; Then, the output information acquiring unit 2401 converts the first information, second information, fourth information, warning information, and confirmation request information into corresponding cause information, message, and display information, and notifies the transmitting unit 2402 of them.

[0372] The transmitting unit 2402 transmits the information notified from the output information acquiring unit 2401 as follows: A display device 912, which is a display device for the staff of the service station 110'; The audio output device 122_1 of the first tanker 120_1 or the audio output device 122_2 of the second tanker 120_2; A refueling data display 123_1 of the first refueling aircraft 120_1 or a refueling data display 123_2 of the second refueling aircraft 120_2; Send to.

[0373] Specifically, the transmission unit 2402 receives the information obtained from the first refueling monitoring unit 1009_1, Controlling the display device 912 to display information indicating permission for refueling, information indicating refueling not being permitted, cause information, and confirmation request information; Controlling so that the message is output as voice via the voice output device 122_1 of the first tanker 120_1; The display information is controlled so as to be displayed on the refueling data display 123_1 of the first refueling aircraft 120_1.

[0374] Similarly, the transmission unit 2402 receives the information obtained from the second refueling monitoring unit 1009_2, Controlling the display device 912 to display information indicating permission for refueling, information indicating refueling not being permitted, cause information, and confirmation request information; Controlling so that the message is output as voice via the voice output device 122_2 of the second tanker 120_2; The display information is controlled so as to be displayed on the refueling data display 123_2 of the second refueling aircraft 120_2.

[0375] (4-2) Specific example of processing by the output control unit 1010 Next, a description will be given of specific examples of cause information, messages, and display information converted by the output control unit 1010, as specific examples of the processing of the output control unit 1010. Fig. 25 is a diagram showing an example of output information.

[0376] 25, the output information 2500 includes cause information, messages, and display information corresponding to the first information to fourth information, warning information, and confirmation request information acquired from the first refueling monitoring unit 1009_1. The output information 2510 includes cause information, messages, and display information corresponding to the first information, second information, fourth information, warning information, and confirmation request information acquired from the second refueling monitoring unit 1009_2.

[0377] As shown in FIG. 25, output information 2500, 2510 includes information items such as "information type", "cause information", "message output to customer", and "display information on the refueling data display of the refueling machine".

[0378] In the "information type", in the case of the output information 2500, any one of the first information to the fourth information is stored, and in the case of the output information 2510, any one of the first information, the second information, or the fourth information is stored.

[0379] In the “cause information”, in the case of output information 2500, one of “nozzle undetermined”, “cannot be determined”, “outside vehicle refueling”, or “fire detected” is stored, and in the case of output information 2510, one of “nozzle undetermined”, “cannot be determined”, or “fire detected” is stored.

[0380] In the "message to be output to the customer", in the case of the output information 2500, messages corresponding to the first information to the fourth information to be output as voice from the voice output device 122_1 are stored. In the case of the output information 2510, messages corresponding to the first information, the second information, and the fourth information to be output as voice from the voice output device 122_2 are stored.

[0381] In the "display information of the refueling data display of the refueling aircraft", in the case of the output information 2500, display information corresponding to each of the first information to fourth information displayed on the refueling data display 123_1 of the first refueling aircraft 120_1 is stored. Also, in the case of the output information 2510, display information corresponding to each of the first information, second information, and fourth information displayed on the refueling data display 123_2 of the second refueling aircraft 120_2 is stored.

[0382] <Flow of refueling control processing by information processing device> Next, a description will be given of the flow of the refueling control process by the refueling system 110S' according to the first embodiment. Fig. 26 is an example of a flowchart showing the flow of the refueling control process by the refueling system according to the first embodiment.

[0383] In step S2601, the information processing device 160 acquires image data captured by the camera 130_1 and the camera 130_2.

[0384] In step S2602, the information processing device 160 acquires signals from the first refueling sensor 921_1 and the second refueling sensor 921_2.

[0385] In step S2603, the information processing device 160 acquires thermal sensor data measured by the thermal sensor 170_1 and the thermal sensor 170_2.

[0386] In step S2604, the information processing device 160 determines which of the first to third intervals the current time falls into.

[0387] In step S2605, information processing device 160 determines whether or not to determine whether refueling is permitted. If it is determined in step S2605 that the vehicle does not fall into any of the first to third sections, it determines that the vehicle is not to be permitted to refuel (NO in step S2605), and proceeds to step S2610.

[0388] On the other hand, if it is determined in step S2605 that the vehicle is in any of the first to third sections, it is determined that a determination is made as to whether refueling is permitted (YES in step S2605), and the process proceeds to step S2606.

[0389] In step S2606, the information processing device 160 performs image recognition processing on the image data using the first trained model or the second trained model.

[0390] In step S2607, the information processing device 160 determines whether or not refueling is permitted based on the recognition result, using a determination logic according to the determined section.

[0391] In step S2608, the information processing device 160 controls refueling by the first refueling machine 120_1 and the second refueling machine 120_2 based on the judgment result of whether refueling is permitted. When the judgment result is that refueling is not permitted, the information processing device 160 controls the prohibition or stop of refueling by the first refueling machine 120_1 or the second refueling machine 120_2 according to the cause information of the refueling not permitted.

[0392] In step S2609, the information processing device 160 outputs the determination result of whether or not refueling is permitted. If the determination result is that refueling is not permitted, the information processing device 160 outputs information on the cause of the refusal of refueling, etc.

[0393] In step S2610, information processing device 160 determines whether or not to end the refueling control process. If it is determined in step S2610 that the refueling control process is to be continued (NO in step S2610), the process returns to step S2601. On the other hand, if it is determined in step S2610 that the refueling control process is to be ended (YES in step S2610), the refueling control process is ended.

[0394] <Summary> As is clear from the above description, the refueling system 110S' according to the first embodiment is configured to divide the period from when the refueling nozzle is removed from the nozzle hanger to when it is returned to the nozzle hanger into three sections, and to determine whether or not refueling is permitted using a determination logic corresponding to each section. As a result, according to the first embodiment, it is possible to provide a refueling system that can execute processing according to the refueling scene.

[0395] In addition, the refueling system 110S' according to the first embodiment is configured to obtain information on the cause of refusal of refueling and output a message according to the cause information when refueling is judged to be not permitted based on the recognition result of the recognition target. As a result, according to the first embodiment, it is possible to provide a refueling system capable of executing a process according to the refueling scene.

[0396] In addition, the refueling system 110S' according to the first embodiment is configured to use a learned model according to the type of refueling when performing image recognition processing on a recognition target. As a result, according to the first embodiment, it is possible to provide a refueling system that can execute processing according to a refueling scene.

[0397] [Second embodiment] In the above first embodiment, the information processing device 160 has been described as performing refueling control of the first refueling machine 120_1, the second refueling machine 120_2, etc. However, the refueling control of the first refueling machine 120_1, the second refueling machine 120_2, etc. may be configured to be performed by a device separate from the information processing device 160 (for example, a refueling controller such as a self-service console provided in a service station). The second embodiment will be described below, focusing on the differences from the above first embodiment.

[0398] <Functional configuration of information processing device> Fig. 27 is a diagram showing another example of the functional configuration of the information processing device. The difference between the functional configuration of the information processing device 160 shown in Fig. 10 and that of the information processing device 160' shown in Fig. 27 is that the functions of the first refueling monitoring unit 1009_1' and the second refueling monitoring unit 1009_2' are different from those of the first refueling monitoring unit 1009_1 and the second refueling monitoring unit 1009_2. The difference between the functional configuration of the information processing device 160 shown in Fig. 10 and that of the information processing device 160' shown in Fig. 27 is that the first refueling monitoring unit 1009_1' and the second refueling monitoring unit 1009_2' are communicably connected to the refueling controller 2710.

[0399] The refueling controller 2710 is a self-service console (SSC) and controls refueling by the first refueling aircraft 120_1 based on the determination result of whether refueling is permitted or not transmitted from the first refueling monitoring unit 1009_1'. Also, the refueling controller 2710 controls refueling by the second refueling aircraft 120_2 based on the determination result of whether refueling is permitted or not transmitted from the second refueling monitoring unit 1009_2'.

[0400] <Details of the First and Second Refueling Monitoring Units> Next, the first refueling monitoring unit 1009_1' and the second refueling monitoring unit 1009_2' will be described in detail. FIG. 28A is a diagram showing another example of the functional configuration of the first refueling monitoring unit of the information processing device. In the first embodiment, the first refueling monitoring unit 1009_1 of the information processing device 160 described using FIG. 15A differs from the functional configuration in the first embodiment in that the first refueling monitoring unit 1009_1' of the information processing device 160' of FIG. 28A has a transmission unit 2801. Also, in the first refueling monitoring unit 1009_1' of the information processing device 160' of FIG. 28A, the transmission unit 2801 is communicably connected to the refueling controller 2710.

[0401] The transmission unit 2801 transmits the determination result of refueling permission or refueling non-permission to the refueling controller 2710. As a result, the refueling controller 2710 controls the refueling valve 124_1 or the refueling pump 125_1 to control the refueling of the first refueling type.

[0402] For example, when a refueling permission is transmitted from the transmission unit 2801, the refueling controller 2710 drives the refueling pump 125_1 to open the refueling valve 124_1. When the fuel supply permission from the plastic tank determination unit 1506 is transmitted from the transmission unit 2801, the fuel supply pump 125_1 is driven to open the fuel supply valve 124_1. When a refusal of refueling from any one of the first judgment logic 1503 to the third judgment logic 1505 is transmitted from the transmission unit 2801, or When the refusal of refueling from the fire determination unit 1507 is transmitted from the transmission unit 2801, In this case, refueling is stopped by stopping the operation of the refueling pump 125_1, or refueling is prohibited by stopping the operation of the refueling pump 125_1 and closing the refueling valve 124_1. Note that the definitions of the stop of refueling and the prohibition of refueling mentioned here are merely examples, and the stop of refueling and the prohibition of refueling may be defined by other definitions. Alternatively, without distinguishing between the stop of refueling and the prohibition of refueling, the general state in which refueling is not possible may be referred to as the stop or prohibition of refueling.

[0403] Furthermore, the transmission unit 2801 notifies the cause information acquisition unit 1509 of the determination result of whether refueling is permitted or not permitted. When notifying that refueling is not permitted, the transmission unit 2801 also notifies the cause information acquisition unit 1509 of information indicating the cause of the determination that refueling is not permitted.

[0404] The transmission unit 2801 may be configured to output "determination impossible" to the cause information acquisition unit 1509 when any of the first determination logic 1503 to the third determination logic 1505 cannot determine whether refueling is permitted. When "determination impossible" is output from the transmission unit 2801, the cause information acquisition unit 1509 may display "determination impossible" on, for example, the display device 912, which is a display device for staff of the service station 110'. With this configuration, even when it is impossible to determine whether refueling is permitted, the staff of the service station 110' can operate the refueling controller 2710 and control refueling.

[0405] The transmission unit 2801 may be configured to monitor the communication status with the refueling controller 2710, and if a communication problem occurs, to output a "communication abnormality" to the cause information acquisition unit 1509. When the "communication abnormality" is output from the transmission unit 2801, the cause information acquisition unit 1509 may display the "communication abnormality" on, for example, the display device 912, which is a display device for staff of the service station 110'. With this configuration, even if the result of the determination as to whether or not refueling is permitted cannot be transmitted to the refueling controller 2710, the staff of the service station 110' can operate the refueling controller 2710 and control refueling.

[0406] Fig. 28B is a diagram showing an example of the functional configuration of the second refueling monitoring unit of the information processing device. Note that the functional configuration of the second refueling monitoring unit 1009_2' shown in Fig. 28B is similar to the functional configuration of the first refueling monitoring unit 1009_1' shown in Fig. 28A, so detailed description will be omitted here.

[0407] <Summary> As is clear from the above description, in the second embodiment, the functions of the information processing device 160 in the first embodiment are realized by the information processing device 160' and the refueling controller 2710. Even if the information processing device 160' and the refueling controller 2710 are configured as separate entities, the second embodiment can provide the same effects as the first embodiment.

[0408] [Third embodiment] In the above first embodiment, the period from when the fuel nozzle is removed from the nozzle hanger to when it is returned to the nozzle hanger is divided into three sections. However, the number of sections is not limited to three, and may be four or more.

[0409] In the above first embodiment, one judgment logic is associated with each section. However, the relationship between the number of sections and the number of judgment logics is not limited to this. For example, one judgment logic may be associated with two consecutive sections.

[0410] In the above first embodiment, the case where there are two types of refueling has been described, but there may be three or more types of refueling. When there are three or more types of refueling, three or more trained models are also generated.

[0411] In the first embodiment, the recognition objects for the first refueling type are a vehicle, a container, a refueler, a refueling nozzle, etc. However, the recognition objects for the first refueling type are not limited to these.

[0412] Similarly, in the first embodiment, the recognition objects for the second refueling type are a container, a refueler, a refueling nozzle, etc. However, the recognition objects for the second refueling type are not limited to these.

[0413] In addition, in the above first embodiment, the first judgment logic to the third judgment logic for the first refueling type are explained using Figures 16A to 18B, but the processing contents of the first judgment logic to the third judgment logic are not limited to these.

[0414] Similarly, in the above first embodiment, the first judgment logic to the third judgment logic for the second refueling type are explained using Figures 20A to 22B, but the processing contents of the first judgment logic to the third judgment logic are not limited to these.

[0415] In addition, in the above first embodiment, no mention is made of the processing that takes place after, for example, the cause information is displayed on the display device 912, which is a display device for staff at the service station 110'. However, after the cause information is displayed, the processing may be switched to manual operation by the staff.

[0416] Although the first embodiment does not refer to the number of times the learning process is executed, the learning process may be executed multiple times. For example, if sufficient recognition accuracy is not obtained in the image recognition process using the first trained model and the second trained model generated by the first learning process, or if the recognition accuracy is reduced, an additional learning process may be executed.

[0417] For example, the learning device 150 may be configured to perform additional learning processing by again collecting image data and signals indicating the state of the fuel nozzle from multiple service stations and generating learning data.

[0418] The learning device 150 may be configured to collect image data and signals indicating the state of the fuel nozzle from multiple service stations at any time. A device may be connected to the network 180 to monitor the recognition accuracy, analyze the cause when the recognition accuracy decreases, and notify the learning device 150 of the result.

[0419] Furthermore, in the first embodiment, the learning device 150 and the information processing device 160 are configured as separate entities, but the learning device 150 and the information processing device 160 may be configured as an integrated entity.

[0420] The present invention is not limited to the configurations shown in the above embodiments, combinations with other elements, etc. These points can be changed without departing from the spirit of the present invention, and can be appropriately determined according to the application form. [Explanation of symbols]

[0421] 110, 111: Service station 110S' : Fuel supply system 120_1: First tanker 120_2: Second tanker 121_1, 121_2: Fuel nozzle 122_1, 122_2: Audio output device 123_1, 123_3: Refueling data display 124_1, 124_2: Fuel supply valve 125_1, 125_2: Fuel pump 130_1, 130_2: Camera 170_1, 170_2: Thermal sensor 150: Learning device 160: Information processing device 160': Information processing device 210: 1st area 220:Second area 211: Vehicle 221: Oiler 231: Container 241: Oiler 500A: Learning data generation unit 500B: Learning Department 600: First training data 700: Second training data 912 :Display device 921_1: First fuel sensor 921_2: Second fuel supply sensor 1001: First refueling type image data acquisition unit 1002: Second refueling type image data acquisition unit 1003: First refueling sensor data acquisition unit 1004: Second refueling sensor data acquisition unit 1005: Image recognition unit 1006, 1007: Section determination section 1009_1: First Refueling Monitoring Division 1009_1': First Refueling Monitoring Unit 1009_2: 2nd Refueling Monitoring Division 1009_2': 2nd Refueling Monitoring Unit 1010: Output control section 1302: First trained model 1312: Second trained model 1503, 1513: First decision logic 1504, 1514: Second decision logic 1505, 1515: Third decision logic 2710 : Fuel supply controller 2801, 2811: Transmitter

Claims

1. A fuel tanker and A camera that photographs an area including the tanker; A refueling system having an information processing device, The information processing device includes: determining a state of the refueling nozzle and a state of refueling based on the notification from the refueling aircraft; When it is determined that refueling is not permitted based on a determination logic according to whether the refueling action of the refueler is included in a first section before the start of refueling, a second section during refueling, or a third section after the end of refueling, or based on a determination logic according to whether the refueling action is included in two consecutive sections, refueling by the refueling aircraft is prohibited or stopped; The first determination logic according to the first section is configured to determine whether a recognition result obtained when a recognition process is performed on image data captured by the camera is: If the permission conditions are met for a certain period of time, refueling is permitted. When a predetermined non-permission condition is satisfied for a certain period of time, it is determined that refueling is not permitted, and when a permission condition is satisfied for a certain period of time thereafter, it is determined that refueling is permitted; If other refusal conditions are met for a certain period of time, refueling is not permitted. The second determination logic according to the second section is configured to determine whether a recognition result obtained when a recognition process is performed on image data captured by the camera is: If the disallowance condition is met for a certain period of time, a warning is output. If the disapproval conditions are met for a certain period of time after the warning is output, it is determined that refueling is not permitted. If the permission conditions are met for a certain period of time after the warning is output, it is determined that refueling is permitted. The third determination logic according to the third section is configured to determine whether a recognition result obtained when a recognition process is performed on image data captured by the camera is: If the permission conditions are met for a certain period of time, refueling is permitted. When the non-permission condition is satisfied for a certain period of time, it is determined that refueling is not permitted, and when the permission condition is satisfied for a certain period of time thereafter, it is determined that refueling is permitted. Refueling system.

2. A state determination unit that determines a state of the refueling nozzle and a state of refueling based on a notification from the refueling machine, When the state determination unit determines that the fuel nozzle has been removed from the nozzle hanger, it is determined that the first section has started; When it is determined that refueling is permitted based on a determination logic corresponding to the first section and the state determination unit determines that refueling has started, it is determined that the vehicle has transitioned to the second section; When the state determination unit determines that refueling has not been performed for a predetermined time, it is determined that the state has entered the third section, and when the state determination unit determines that the refueling nozzle has been returned to the nozzle hook, it is determined that the third section has ended. The fueling system of claim 1 .

3. When the type of fuel is gasoline or diesel, the determination logic A recognition process is performed on the image data captured by the camera, A recognition result that a vehicle is recognized, A recognition result that a fuel dispenser holding a fuel nozzle in an appropriate posture is recognized, A recognition result that a fuel dispenser who is not holding the fuel dispenser nozzle in an appropriate position is recognized, The refueling system according to claim 1 , wherein, when the recognition result is output, it is determined whether or not refueling is permitted based on the recognition result.

4. The decision logic further comprises: The refueling system of claim 3, further comprising a process for performing a recognition process on image data captured by the camera, and determining that refueling is not permitted if a recognition result indicating that a specified container has been recognized is output.

5. Further comprising a thermal sensor; The decision logic further comprises:

2. The refueling system according to claim 1, further comprising: a determination as to whether or not the temperature measured by the thermal sensor is equal to or higher than a predetermined threshold; and if it is determined that the temperature is equal to or higher than the predetermined threshold, it is determined that refueling is not permitted.

6. A recognition result indicating that a vehicle has been recognized in a first area that is a part of the image data captured by the camera is output; A recognition result indicating that a fuel dispenser holding a fuel dispenser nozzle in an appropriate posture has been recognized in a second area, which is a part of the image data captured by the camera and is different from the first area, is output.

4. The fueling system of claim 3.

7. Further comprising a thermal sensor; The determination logic includes: The fuel supply system according to claim 6 , further comprising: determining whether or not the temperature measured by the thermal sensor is equal to or higher than a predetermined threshold in an area not including the first area.

8. When the fuel type is kerosene, the judgment logic is as follows: A recognition process is performed on the image data captured by the camera, A recognition result that a specific container has been recognized, A recognition result that a fuel dispenser holding a fuel nozzle in an appropriate posture is recognized, A recognition result that a fuel dispenser who is not holding the fuel dispenser nozzle in an appropriate position is recognized, The refueling system according to claim 1 , wherein, when the recognition result is output, it is determined whether or not refueling is permitted based on the recognition result.

9. Further comprising a thermal sensor; The decision logic further comprises:

9. The refueling system according to claim 8, further comprising: a determination as to whether or not the temperature measured by the thermal sensor is equal to or higher than a predetermined threshold; and if it is determined that the temperature is equal to or higher than the predetermined threshold, it is determined that refueling is not permitted.

10. Prohibiting or stopping refueling by stopping the refueling pump or closing the refueling valve installed in the refueling pipe, The fueling system of claim 1 .

11. A refueling controller communicating with the information processing device, The refueling controller includes: The refueling system according to claim 1 , wherein, when it is determined based on the determination logic that refueling is not permitted, refueling by the refueling aircraft is prohibited or stopped.

12. Based on the notification from the refueling aircraft, the state of the refueling nozzle and the state of refueling are determined; When it is determined that refueling is not permitted based on a determination logic according to whether the refueling action of the refueler is included in a first section before the start of refueling, a second section during refueling, or a third section after the end of refueling, or based on a determination logic according to whether the refueling action is included in two consecutive sections, refueling by the refueling aircraft is prohibited or stopped; The first determination logic according to the first section is configured to determine whether a recognition result obtained when a recognition process is performed on image data captured by a camera that captures an area including the tanker is: If the permission conditions are met for a certain period of time, refueling is permitted. When a predetermined non-permission condition is satisfied for a certain period of time, it is determined that refueling is not permitted, and when a permission condition is satisfied for a certain period of time thereafter, it is determined that refueling is permitted; If other refusal conditions are met for a certain period of time, refueling is not permitted. The second determination logic according to the second section is configured to determine whether a recognition result obtained when a recognition process is performed on image data captured by the camera is: If the disallowance condition is met for a certain period of time, a warning is output. If the disapproval conditions are met for a certain period of time after the warning is output, it is determined that refueling is not permitted. If the permission conditions are met for a certain period of time after the warning is output, it is determined that refueling is permitted. The third determination logic according to the third section is configured to determine whether a recognition result obtained when a recognition process is performed on image data captured by the camera is: If the permission conditions are met for a certain period of time, refueling is permitted. When the non-permission condition is satisfied for a certain period of time, it is determined that refueling is not permitted, and when the permission condition is satisfied for a certain period of time thereafter, it is determined that refueling is permitted. Information processing device.

13. Based on the notification from the refueling aircraft, the state of the refueling nozzle and the state of refueling are determined; when it is determined that refueling is not permitted based on a determination logic according to whether the refueling action of the refueler is included in a first section before the start of refueling, a second section during refueling, or a third section after the end of refueling, or based on a determination logic according to whether the refueling action is included in two consecutive sections, the computer executes a process of prohibiting or stopping refueling by the refueling aircraft; The first determination logic according to the first section is configured to determine whether a recognition result obtained when a recognition process is performed on image data captured by a camera that captures an area including the tanker is: If the permission conditions are met for a certain period of time, refueling is permitted. When a predetermined non-permission condition is satisfied for a certain period of time, it is determined that refueling is not permitted, and when a permission condition is satisfied for a certain period of time thereafter, it is determined that refueling is permitted; If other refusal conditions are met for a certain period of time, refueling is not permitted. The second determination logic according to the second section is configured to determine whether a recognition result obtained when a recognition process is performed on image data captured by the camera is: If the disallowance condition is met for a certain period of time, a warning is output. If the disapproval conditions are met for a certain period of time after the warning is output, it is determined that refueling is not permitted. If the permission conditions are met for a certain period of time after the warning is output, it is determined that refueling is permitted. The third determination logic according to the third section is configured to determine whether a recognition result obtained when a recognition process is performed on image data captured by the camera is: If the permission conditions are met for a certain period of time, refueling is permitted. When the non-permission condition is satisfied for a certain period of time, it is determined that refueling is not permitted, and when the permission condition is satisfied for a certain period of time thereafter, it is determined that refueling is permitted. Refueling control method.

14. Based on the notification from the refueling aircraft, the state of the refueling nozzle and the state of refueling are determined; when it is determined that refueling is not permitted based on a determination logic according to whether the refueling action of the refueler is included in a first section before the start of refueling, a second section during refueling, or a third section after the end of refueling, or based on a determination logic according to whether the refueling action is included in two consecutive sections, a computer is caused to execute a process of prohibiting or stopping refueling by the refueling machine; The first determination logic according to the first section is configured to determine whether a recognition result obtained when a recognition process is performed on image data captured by a camera that captures an area including the tanker is: If the permission conditions are met for a certain period of time, refueling is permitted. When a predetermined non-permission condition is satisfied for a certain period of time, it is determined that refueling is not permitted, and when a permission condition is satisfied for a certain period of time thereafter, it is determined that refueling is permitted; If other refusal conditions are met for a certain period of time, refueling is not permitted. The second determination logic according to the second section is configured to determine whether a recognition result obtained when a recognition process is performed on image data captured by the camera is: If the disallowance condition is met for a certain period of time, a warning is output. If the disapproval conditions are met for a certain period of time after the warning is output, it is determined that refueling is not permitted. If the permission conditions are met for a certain period of time after the warning is output, it is determined that refueling is permitted. The third determination logic according to the third section is configured to determine whether a recognition result obtained when a recognition process is performed on image data captured by the camera is: If the permission conditions are met for a certain period of time, refueling is permitted. When the non-permission condition is satisfied for a certain period of time, it is determined that refueling is not permitted, and when the permission condition is satisfied for a certain period of time thereafter, it is determined that refueling is permitted. Fuel supply control program.

Images