How to control a vehicle using a VQA model
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- TOYOTA JIDOSHA KK
- Filing Date
- 2025-04-01
- Publication Date
- 2026-08-04
Smart Images
Figure 0007899918000001 
Figure 0007899918000002
Abstract
Description
Background Art
[0001] A visual question answering (VQA) learning model can be mounted on a vehicle and used by the vehicle to acquire an image and prompt a result from the learning model based on the image. The VQA learning model can be used in computer vision, facilitating training / educating a machine to understand the content of an image and answer one or more questions about the image, and can do this in a more accurate way than using non-image data. The machine can answer questions in natural language. Once the learning model is trained, a large amount of resources such as a large storage area, memory, and processing power are required to obtain a result or an inference based on a question asked of the machine about an image. Conventional VQA models are based on images already stored in a database, process images with a negligible delay when the images are captured / generated in real time, create a dataset that pairs an image with a question for obtaining an answer, and have limitations in the ability to perform vehicle operations.
Summary of the Invention
[0002] A non-transitory computer-readable medium includes instructions that, in response to execution by one or more processors, cause an operation including receiving an image of a candidate object of interest captured by a camera mounted on a vehicle. The operation includes inputting a first dataset including the image and a first question associated with the candidate object of interest into a visual question answering (VQA) model. The operation includes obtaining an answer to the first question as an output from the VQA model. The operation includes performing an operation of the vehicle based on the answer.
[0003] Aspects of this disclosure relate to a method for implementing vehicle operation. The method includes receiving images of candidate objects of interest captured by a camera mounted on the vehicle. The method includes inputting a first dataset, which includes the images and a first question associated with the candidate objects of interest, into a VQA model. The method includes obtaining answers to the first questions as output from the VQA model. The method includes performing vehicle operations based on the answers in the first dataset using a controller in the vehicle.
[0004] Aspects of this disclosure relate to a vehicle. The device includes a controller, which includes circuitry configured to perform vehicle operations, including performing vehicle operations based on answers to a first dataset. The device includes a processor configured to receive images of candidate objects of interest captured by a camera mounted on the vehicle, or images of users approaching or near the vehicle detected by sensors on the vehicle. The device includes a processor configured to input a first dataset, which includes images and a first question associated with the candidate objects of interest, into a VQA model. The device includes a processor configured to obtain answers to the first question as output from the VQA model. [Brief explanation of the drawing]
[0005] [Figure 1] Figure 1 is an action flowchart for controlling vehicle operation via a visual question answering (VQA) model, according to at least some embodiments of the present disclosure. [Figure 2] Figure 2 is a block diagram of a hardware configuration for controlling a vehicle via a VQA model, according to at least some embodiments of the present disclosure. [Modes for carrying out the invention]
[0006] The aspects of this disclosure will be best understood from the following detailed description, when read in conjunction with the attached drawings. Please note that various features are not depicted to scale in accordance with standard practice in this industry. In fact, the dimensions of various features may be arbitrarily increased or decreased for the sake of clarity in the discussion.
[0007] The following disclosure provides many different embodiments or examples for implementing various features of the subject matter provided. Specific examples of components, values, operations, materials, arrangements, or equivalents thereof are described below for the sake of simplicity of this disclosure. Naturally, these are merely examples and are not intended to be limiting. Other components, values, operations, materials, arrangements, or equivalents thereof are contemplated. In addition, this disclosure may repeat reference numbers and / or letters in various examples. This repetition is for the purpose of simplification and clarity and does not in itself define relationships between the various embodiments and / or configurations discussed.
[0008] Developing image recognition models suitable for diverse use cases presents challenges such as the difficulty of obtaining data to train the models, which in turn consumes a heavy computational load for training. Furthermore, equipping a vehicle with multiple models to perform image recognition for controlling various vehicle operations leads to an increased computational load on the vehicle. An increased computational load in a vehicle negatively impacts other functions of the vehicle.
[0009] In at least one embodiment, the method relates to inputting a dataset. In at least one embodiment, the dataset includes images and questions. In at least one embodiment, the dataset is input to a VQA model. In at least one embodiment, the VQA model is installed in a vehicle at least once. In at least one embodiment, the VQA model is installed in a vehicle multiple times. In at least one embodiment, the output of the VQA model is the answer to a question. In at least one embodiment, the question is asked in machine / computer language. In at least one embodiment, the question is asked in natural language. In at least one embodiment, the answer is read in machine / computer language. In at least one embodiment, the answer is read in natural language. In at least one embodiment, the VQA model is installed in a vehicle, for example, physically installed in memory in the vehicle. In at least one embodiment, the VQA model is physically installed in a storage device in the vehicle. In at least one embodiment, the VQA model is installed in a cloud storage device. In at least one embodiment, the VQA model installed in a cloud storage device is accessible via a cloud server.
[0010] In at least one embodiment, a dataset is input to a VQA model. In at least one embodiment, the dataset includes at least two datasets. In at least one embodiment, each dataset includes an image and a question. In at least one embodiment, each dataset is input to the VQA model multiple times. In at least one embodiment, the first and second datasets are input to the VQA model in a first instance / time. In at least one embodiment, the first and second datasets are input to the VQA model in a second instance / time.
[0011] In at least one embodiment, vehicle control is based on responses to natural language questions. In at least one embodiment, the VQA model is assigned to a viewpoint or angle from which images are acquired by at least one camera. In at least one embodiment, the image is an image of the driver acquired by at least one camera. In at least one embodiment, at least one camera is located inside the vehicle. In at least one embodiment, the image is an image of the vehicle occupants other than the driver, acquired by at least one camera inside the vehicle. In at least one embodiment, the image is an image of the area surrounding / outside the vehicle, acquired by at least one camera outside the vehicle.
[0012] In at least one embodiment, the image is an image according to the aforementioned image, but is acquired by the vehicle driver's mobile device / terminal. In at least one embodiment, the image is an image according to the aforementioned image, but is acquired by the vehicle occupant other than the driver's mobile device / terminal. In at least one embodiment, the image is an image acquired by a vehicle other than the target vehicle, in other words, by a vehicle adjacent to the target vehicle, within the field of view of the camera of the vehicle other than the target vehicle. In at least one embodiment, the image acquired by the vehicle driver or occupant's mobile device / terminal is transmitted via wired / wireless connection to at least one of the VQA models.
[0013] Figure 1 discloses an operation flowchart for controlling vehicle operation via the VQA model 100 according to at least some embodiments of this disclosure. The following is an example of one use case scenario, but the basic method is the same as for other use case scenarios. In at least one embodiment, user 102 approaches the target vehicle. In at least one embodiment, the user is a person. In at least one embodiment, the user is holding an object 104, for example, luggage. In at least one embodiment, the user approaches the target vehicle while holding the object. In at least one embodiment, the user approaches the target vehicle without holding the object.
[0014] In at least one embodiment, the controller 114 is configured to detect a person approaching the vehicle (116). In at least one embodiment, the controller 114 is configured to detect a person via a motion sensor. In at least one embodiment, the controller 114 is configured to detect a person via an image acquisition unit, such as a camera / video camera.
[0015] In at least some embodiments, the controller 114 communicates with the VQA model 124. In at least some embodiments, the controller 114 is configured to host and execute functions of the VQA model. In at least some embodiments, the controller 114 is configured to perform vehicle operations. In at least some embodiments, the controller 114 includes a microcontroller unit (MCU), microprocessor unit (MPU), or electronic control unit (ECU) configured to execute instructions stored in a memory component.
[0016] In at least some embodiments, the controller 114 includes a memory component configured to store instructions and data, such as random access memory (RAM), read-only memory (ROM), or flash memory. In at least some embodiments, the controller 114 includes a storage component configured to store data for a long period of time, such as a hard disk drive (HDD) or solid-state drive (SSD). In at least some embodiments, the controller 114 includes an input / output (I / O) interface configured to communicate with an external device, such as an image acquisition unit. In at least some embodiments, the controller 114 is an embedded system specifically designed for a vehicle.
[0017] In at least one embodiment, a first dataset is input to the VQA model (106). In at least one embodiment, the first dataset includes images, questions, and answers. In at least one embodiment, the images are captured by a camera, for example, a camera capable of taking still images or videos. In at least one embodiment, the camera focuses on a user holding an object.
[0018] In at least one embodiment, the input is combined / paired with a question / query (118). In at least one embodiment, the question / query combined / paired with the image asks, "Is the user facing the camera?" (118). In at least one embodiment, the VQA model outputs a response (126). In at least one embodiment, the VQA model outputs the response "No," and in response to the response "No," the controller does not perform a vehicle operation (126). In at least one embodiment, the VQA model outputs a response. In at least one embodiment, the VQA model outputs the response "Yes," and in response to the response "Yes," the controller performs a vehicle operation (126).
[0019] In at least one embodiment, in response to the first dataset indicating the answer "Yes," the controller is prompted to input a second dataset into the VQA model / input of the second dataset into the VQA model is triggered (108). In at least one embodiment, the second dataset includes images, questions, and answers. In at least one embodiment, the images are captured by a camera similar to that disclosed above. In at least one embodiment, the camera focuses on the user holding the object.
[0020] In at least one embodiment, the input of a second dataset 108 is paired with a question 120. In at least one embodiment, the question paired with the image asks "Is the user holding something?" (120). In at least one embodiment, the VQA model 124 outputs a response (128). In at least one embodiment, the VQA model outputs the response "No", and in response to the response "No", the controller does not perform the vehicle operation 128 (128). In at least one embodiment, the VQA model outputs a response. In at least one embodiment, the VQA model outputs the response "Yes", and in response to the response "Yes", the controller performs the vehicle operation (128). In at least one embodiment, the VQA model outputs the response "Yes", and in response to the response "Yes", the controller 114 performs the second vehicle operation (134).
[0021] In at least one embodiment, in response to the second dataset indicating the answer "Yes," the controller is prompted to input a third dataset 110 into the VQA model 124 / input of the third dataset 110 into the VQA model 124 is triggered. In at least one embodiment, the third dataset includes images, questions, and answers. In at least one embodiment, the images are captured by a camera similar to that disclosed above. In at least one embodiment, the camera focuses on the user's facial expressions.
[0022] In at least one embodiment, the input to the third dataset 110 is paired with a question (122). In at least one embodiment, the question paired with the image asks, "Is the user making a facial expression?" (122). In at least one embodiment, the VQA model 124 outputs a response. In at least one embodiment, the VQA model 124 outputs the response "No", and in response to the response "No", the controller 114 does not perform a vehicle operation (130). In at least one embodiment, the VQA model 124 outputs a response. In at least one embodiment, the VQA model 124 outputs the response "Yes", and in response to the response "Yes", the controller performs a vehicle operation 130.
[0023] In at least one embodiment, if the VQA model outputs "Yes", the operation proceeds to the next step. In at least one embodiment, the controller receives the response and continues inputting the second dataset. In at least one embodiment, the second dataset contains the same images as the first dataset, but with different questions combined / paired. In at least one embodiment, the subsequent dataset contains the same images as the preceding dataset, but with different questions. In at least one embodiment, the subsequent dataset contains different images with different questions as to the preceding dataset. In at least one embodiment, the subsequent dataset contains different images with the same questions as the preceding dataset. In at least one embodiment, the second dataset is combined / paired with the question, "Is the user holding something?". In at least one embodiment, if the VQA model's output response is "No", the operation stops. In at least one embodiment, if the VQA model's output response is "Yes", the controller continues inputting the third dataset. In at least one embodiment, the third dataset is combined / paired with the question, "Is the user laughing?". In at least one embodiment, if the output response of the VQA model is "No", the operation stops. In at least one embodiment, if the output response of the VQA model is "Yes", the controller continues to perform any vehicle operations that the controller is responsible for, such as opening the back door / trunk. In at least one embodiment, if the controller recognizes that the user is facing the camera and the user is holding luggage, the controller turns on the vehicle's taillights in addition to performing the operation.
[0024] In at least one embodiment, the controller forms a question by referring to a sequence, and based on the sequence, the controller captures an image from a camera, pairs the image with a predefined question to form a dataset, and inputs the dataset into a VQA model. In other words, the commands of the controller are based on a predefined camera and a predefined view of the camera.
[0025] In at least one embodiment, the controller performs a predetermined vehicle control based on a binary "Yes" or "No" answer to a question. In at least one embodiment, the controller performs a predetermined vehicle control based on an answer that is not binary to a question. In at least one embodiment, the answer is an integer. In at least one embodiment, the answer is a fraction.
[0026] In at least one embodiment, instead of generating a spontaneous predefined question to start a sequence for ending the execution of a predetermined vehicle operation, the controller is configured to ask a first question based on a first image of a user, and the first question is associated with at least one predefined question.
[0027] FIG. 2 is a block diagram of a hardware configuration for performing a vehicle operation based on an answer output from a VQA model according to at least some embodiments of the present disclosure.
[0028] An exemplary hardware configuration includes a controller 202 that interacts with a VQA model 218 directly or through a network 214. In at least some embodiments, the network 214 is an Ethernet network, a controller area network (CAN), other wired or wireless networks, or a combination thereof. In at least some embodiments, the controller 202 is a computer system that executes computer-readable instructions to perform a vehicle operation.
[0029] In at least some embodiments, the controller 202 includes a processor or programmable circuit that executes instructions, which cause the processor or programmable circuit to perform operations according to the instructions. In at least some embodiments, the controller 202 includes analog or digital programmable circuitry or any combination thereof. In at least some embodiments, the controller 202 includes physically separated storage or circuitry that interacts through communication. In at least some embodiments, the storage unit 204 includes a non-volatile computer-readable medium capable of storing executable and non-executable data accessed by the controller 202 during instruction execution. The communication face 208 transmits data to and receives data from the network 214. The input / output interface 210 connects to various input and output units via parallel ports, serial ports, keyboard ports, mouse ports, monitor ports and equivalents thereto to accept commands and present information. In some embodiments, the storage unit 204 is external to the controller 202.
[0030] In at least some embodiments, the device is another device capable of processing logical functions to perform the operations described herein. In at least some embodiments, the controller and the storage unit do not need to be entirely separate devices, and in some embodiments they share circuitry or one or more computer-readable media. In at least some embodiments, the storage unit includes a hard drive that stores both computer-executable instructions and data accessed by the controller, and the controller includes a combination of a central processing unit (CPU) and RAM, in which the computer-executable instructions are copied whole or in part to be executed by the CPU during the performance of the operations described herein.
[0031] In at least some embodiments where the device is a computer, a program installed on the computer may cause the computer to function as the device of the embodiments described herein, or to perform operations associated with the device of the embodiments described herein. In at least some embodiments, such a program may be run by a processor to cause the computer to perform specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.
[0032] At least some embodiments are described with reference to flowcharts and block diagrams in which a block represents a step (1) of a process in which an operation is performed, or a section (2) of a controller responsible for performing the operation. In at least some embodiments, specific steps and sections are implemented by a dedicated circuit, a programmable circuit supplied with computer-readable instructions stored on a computer-readable medium, and / or a processor supplied with computer-readable instructions stored on a computer-readable medium. In at least some embodiments, the dedicated circuit includes digital and / or analog hardware circuits, including integrated circuits (ICs) and / or discrete circuits. In at least some embodiments, the programmable circuit includes reconfigurable hardware circuits consisting of logical AND, OR, XOR, NAND, NOR and other logic operations, flip-flops, registers, memory elements, etc., such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), etc.
[0033] In at least some embodiments, a computer-readable storage medium includes a tangible device capable of holding and storing instructions used by an instruction execution device. In some embodiments, the computer-readable storage medium includes, but is not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage mediums includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital multipurpose disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved raised structures having instructions recorded thereon, and any suitable combination thereof. The computer-readable media used herein should not be interpreted as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses passing through fiber optic cables), or transient signals themselves, such as electrical signals transmitted through wires.
[0034] In at least some embodiments, the computer-readable program instructions described herein are downloadable from a computer-readable storage medium to each computing / processing device, or downloadable to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. In at least some embodiments, the network includes copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. In at least some embodiments, a network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium within each computing / processing device.
[0035] In at least some embodiments, the computer-readable program instructions for performing the operations described above are either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++ or equivalents, and conventional procedural programming languages such as the C programming language or similar programming languages. In at least some embodiments, the computer-readable program instructions are fully executed on the user's computer, partially executed on the user's computer as a standalone software package, partially executed on the user's computer and partially executed on a remote computer, or fully executed on a remote computer or server. In at least some embodiments, in the latter scenario, the remote computer is connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection is made to an external computer (for example, through the Internet using an Internet Service Provider). In at least some embodiments, an electronic circuit including, for example, a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) executes a computer-readable program instruction by individualizing the electronic circuit using state information of the computer-readable program instruction in order to perform an aspect of the present invention.
[0036] While embodiments of the present invention have been described, the technical scope of any subject matter of the claims is not limited to the embodiments described above. Those skilled in the art will understand that various modifications and improvements are possible to the embodiments described above. Furthermore, those skilled in the art will understand that such modified or improved embodiments are also included in the technical scope of the present invention.
[0037] The operations, procedures, steps, and stages of each process performed by the apparatus, systems, programs, and methods shown in the claims, embodiments, or drawings may be performed in any order, unless the order is indicated by “before,” “before,” or equivalent, and unless the output from a previous process is used in a later process. Even if the flow of a process is described in the claims, embodiments, or drawings using phrases such as “first” or “next,” such description does not necessarily mean that the processes will be performed in the order described.
[0038] In at least some embodiments, vehicle operations in response to the output of the VQA model are performed by a device including a processor that executes instructions according to the aforementioned operations, or a controller that includes circuits configured to perform the aforementioned operations.
[0039] The features of several embodiments have been outlined above so that those skilled in the art may better understand the aspects of this disclosure. Those skilled in the art should understand that this disclosure can be readily used as a basis for designing or modifying other processes and structures to perform the same purposes and / or achieve the same advantages as the embodiments introduced herein. Those skilled in the art should also understand that such equivalent structures can be made in various ways, substitutions and modifications herein without departing from the spirit and scope of this disclosure.
[0040] Supplementary Note 1 Aspects of this disclosure relate to vehicles and vehicle operations. A non-temporary computer-readable medium includes instructions that, in response to execution by one or more processors, cause an operation to be performed which includes receiving images of candidate objects of interest captured by cameras mounted on the vehicle. The operation includes inputting a first dataset, which includes the images and a first question associated with the candidate objects of interest, into a visual question answer (VQA) model. The operation includes obtaining an answer to the first question as output from the VQA model. The operation includes performing an operation of the vehicle based on the answer.
[0041] Supplementary Note 2 The operation described in Supplementary Note 1 further includes, depending on the answers in the first dataset, causing input to the VQA model of a second dataset which includes images from the first dataset and a second question associated with the candidate object of interest. The operation further includes obtaining answers to the second question as output from the VQA model.
[0042] Supplementary Note 3 The vehicle operation described in Supplementary Note 1 or 2, wherein the input to the first dataset and the second dataset occurs multiple times, and the input to the second dataset occurs after the input to the first dataset.
[0043] Supplementary Note 4 The first question above is a vehicle operation as described in any one of Supplementary Notes 1 to 3, based on the pointing direction of the candidate object of interest relative to the camera.
[0044] Supplementary Note 5 The vehicle operation described in any one of Supplementary Notes 1 to 4, wherein the candidate object of interest includes a change in the pointing direction of the pointing operation of the candidate object of interest with respect to the vehicle occupants, the vehicle's speed, or the interior of the vehicle.
[0045] Supplementary Note 6 The vehicle operation described in any one of Supplementary Notes 1 to 5, wherein the name of the candidate object of interest is highlighted in the graphical user interface (GUI).
[0046] Supplementary Note 7 Vehicle operations as described in any one of Supplementary Notes 1 to 6, further including datasets other than the first dataset and the second dataset.
[0047] Supplementary note 8 Aspects of this disclosure relate to a method for implementing vehicle operation. The method includes receiving images of candidate objects of interest captured by a camera mounted on the vehicle. The method includes inputting a first dataset, which includes the images and a first question associated with the candidate objects of interest, into a VQA model. The method includes obtaining answers to the first questions as output from the VQA model. The method includes performing vehicle operations based on the answers in the first dataset using a controller in the vehicle.
[0048] Supplementary note 9 The vehicle operation method described in Supplementary Note 8 further comprises, in response to the answers in the first dataset, causing input to the VQA model of a second dataset including images from the first dataset and a second question associated with the candidate object of interest. The vehicle operation method further comprises obtaining answers to the second question as output from the VQA model.
[0049] Supplementary note 10 The vehicle operation method described in Supplementary Note 8 or 9, wherein the input of the first dataset and the second dataset occurs multiple times, and the input of the second dataset occurs after the input of the first dataset.
[0050] Supplementary note 11 The first question above relates to a method of operating a vehicle as described in any one of Supplementary Notes 8 to 10, based on the pointing direction of the candidate object of interest relative to the camera.
[0051] Supplementary note 12 A method of operating a vehicle as described in any one of Supplementary Notes 8 to 11, wherein the candidate object of interest includes a change in the pointing direction of the pointing operation of the candidate object of interest with respect to vehicle occupants, the speed of the vehicle, or the interior of the vehicle.
[0052] Supplementary note 13 A method of operating a vehicle as described in any one of Supplementary Notes 8 to 12, wherein the name of the candidate object of interest is highlighted in the GUI.
[0053] Supplementary note 14 A method of operating a vehicle as described in any one of Supplementary Notes 8 to 13, further including datasets other than the first dataset and the second dataset.
[0054] Supplementary note 15 Aspects of this disclosure relate to a vehicle. The apparatus includes a controller, which includes circuitry configured to perform operations on the vehicle, including performing operations on the vehicle based on answers to a first dataset. The apparatus includes a processor configured to receive images of candidate objects of interest captured by a camera mounted on the vehicle. The apparatus includes a processor configured to input a first dataset, which includes the images and a first question associated with the candidate objects of interest, into a VQA model. The apparatus includes a processor configured to obtain answers to the first question as output from the VQA model.
[0055] Supplementary note 16 Furthermore, the apparatus described in Supplementary Note 15 is configured to cause input to the VQA model of a second dataset, which includes images from the first dataset and a second question associated with the candidate object of interest, in response to the answers from the first dataset. The apparatus is further configured to obtain answers to the second question as output from the VQA model.
[0056] Supplementary note 17 The apparatus as described in Supplementary Note 15 or 16, wherein the input to the first dataset and the second dataset occurs multiple times, and the input to the second dataset occurs after the input to the first dataset.
[0057] Supplementary note 18 The apparatus according to any one of Supplementary Notes 15 to 17, further comprising a microphone associated with the vehicle, wherein the microphone is configured to process contextual data including audio data received by the microphone within a predetermined time while an image of the candidate object of interest is acquired.
[0058] Supplementary note 19 The apparatus according to any one of Supplementary Notes 15 to 18, wherein the candidate object of interest includes a change in the pointing direction of the pointing operation of the candidate object of interest with respect to a vehicle occupant, the vehicle's speed, or the interior of the vehicle.
[0059] Supplementary note 20 The device described in any one of Supplementary Notes 15 to 19, wherein the name of the candidate object of interest is highlighted in the GUI.
Claims
1. Receiving images of candidate objects of interest captured by cameras mounted on the vehicle, Inputting a first dataset, which includes the aforementioned image and a first question associated with the candidate object of interest, into a visual question answer (VQA) model, The output from the VQA model is to obtain the answer to the first question, In response to the aforementioned answer, the input to the VQA model is a second dataset containing images from the first dataset and a second question associated with the candidate object of interest. The output from the VQA model is to obtain the answer to the second question, Depending on whether the answer to the second question is a predetermined answer, the operation of the vehicle is performed. A computer program that causes a processor to perform an operation that includes [a specific operation].
2. The computer program according to claim 1, wherein the input of the first dataset and the second dataset occurs multiple times, and the input of the second dataset occurs after the input of the first dataset.
3. The computer program according to claim 1 or 2, further comprising datasets other than the first dataset and the second dataset.
4. A method executed by one or more processors, Receiving images of candidate objects of interest captured by cameras mounted on the vehicle, Inputting a first dataset, which includes the aforementioned image and a first question associated with the candidate object of interest, into the VQA model, The output from the VQA model is to obtain the answer to the first question, In response to the answers in the first dataset, the input to the VQA model of a second dataset, which includes images from the first dataset and a second question associated with the candidate object of interest, The output from the VQA model is to obtain the answer to the second question, Depending on whether the answer to the second question is a predetermined answer, the vehicle's operation is performed by the controller inside the vehicle. Methods that include...
5. The method according to claim 4, wherein the inputs to the first dataset and the second dataset occur multiple times, and the input to the second dataset occurs after the input to the first dataset.
6. The method according to claim 4 or 5, further comprising datasets other than the first dataset and the second dataset.