Vehicle control method using VQA model

Abstract

To provide a vehicle control method using a VQA model.SOLUTION: A system / method involves performing an operation that includes receiving an image of a candidate object of interest or a user (COI / U) captured by a camera mounted on a vehicle. The operation includes inputting a first dataset, which includes the image and a first question associated with the COI / U, into the VQA model. The operation involves obtaining an answer to the first question as output from the VQA model. The operation includes executing vehicle control based on the answer.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] A visual question answering (VQA) learning model can be installed in a vehicle and used by the vehicle to acquire images and prompt results from the learning model based on the images. VQA learning models can be used in computer vision, making it easier to train / educate machines to understand the content of images and answer a question or questions about the images, in a more accurate manner than using non-image data. The machine can answer questions in natural language. Once the learning model is trained, significant resources, such as large amounts of storage space, memory, and processing power, are required to obtain results or interventions based on questions asked of the machine about the images. Traditional VQA models are based on images already stored in a database and are limited in their ability to process images with negligible delay as the images are captured / generated in real time, create datasets that pair images with questions to obtain answers, and perform vehicle operations. Summary of the Invention

[0002] A non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause operations to be performed, including receiving an image of a candidate object of interest captured by a camera mounted on a vehicle, inputting a first dataset including the image and a first question associated with the candidate object of interest into a visual question answering (VQA) model, obtaining an answer to the first question as output from the VQA model, and performing a vehicle operation based on the answer.

[0003] An aspect of the present disclosure relates to a method for implementing vehicle manipulation. The method includes receiving an image of a candidate object of interest captured by a camera mounted on a vehicle. The method includes inputting a first dataset including the image and a first question associated with the candidate object of interest into a VQA model. The method includes obtaining an answer to the first question as an output from the VQA model. The method includes performing, by a controller in the vehicle, a vehicle manipulation based on the answer to the first dataset.

[0004] An aspect of the present disclosure relates to a vehicle. The apparatus includes a controller including a circuit configured to perform a vehicle operation, including performing the vehicle operation based on an answer of a first dataset. The apparatus includes a processor configured to receive images of a candidate object of interest captured by a camera mounted on the vehicle or an image of a user approaching or near the vehicle detected by a sensor on the vehicle. The apparatus includes a processor configured to input a first dataset including the images and a first question associated with the candidate object of interest into a VQA model. The apparatus includes a processor configured to obtain an answer to the first question as an output from the VQA model. [Brief explanation of the drawings]

[0005] [Figure 1] FIG. 1 is an operational flowchart for controlling vehicle operation via a visual question answering (VQA) model, according to at least some embodiments of the present disclosure. [Figure 2] FIG. 2 is a block diagram of a hardware configuration for controlling a vehicle via a VQA model, according to at least some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0006] Aspects of the present disclosure are best understood from the following detailed description when read in conjunction with the accompanying drawings. It should be noted that, in accordance with standard practice in the industry, various features have not been drawn to scale. In fact, the dimensions of various features may be arbitrarily increased or decreased for clarity of discussion.

[0007] The following disclosure provides many different embodiments or examples for implementing various features of the provided subject matter. Specific examples of components, values, operations, materials, arrangements, or equivalents thereof are described below to simplify the disclosure. It should be understood that these are merely examples and are not intended to be limiting. Other components, values, operations, materials, arrangements, or equivalents thereof are contemplated. In addition, the disclosure may repeat reference numerals and / or letters in various examples. This repetition is for purposes of simplicity and clarity and does not, in itself, dictate a relationship between the various embodiments and / or configurations discussed.

[0008] Developing image recognition models suitable for various use cases is difficult due to the difficulty of obtaining data for training the image recognition models, which consumes a heavy computational load to train / educate the models. Furthermore, installing multiple models in a vehicle and performing image recognition to control various vehicle operations increases the computational load on the vehicle. Increasing the computational load on a vehicle adversely affects other functions of the vehicle.

[0009] In at least one embodiment, a method involves inputting a dataset. In at least one embodiment, the dataset includes an image and a question. In at least one embodiment, the dataset is input to a VQA model. In at least one embodiment, the VQA model is installed at least once in a vehicle. In at least one embodiment, the VQA model is installed multiple times in a vehicle. In at least one embodiment, an output of the VQA model is an answer to the question. In at least one embodiment, the question is asked in a machine / computer language. In at least one embodiment, the question is asked in a natural language. In at least one embodiment, the answer is retrieved in a machine / computer language. In at least one embodiment, the answer is retrieved in a natural language. In at least one embodiment, the VQA model is installed in a vehicle, e.g., physically installed in memory at the vehicle. In at least one embodiment, the VQA model is physically installed in a storage device at the vehicle. In at least one embodiment, the VQA model is installed in a cloud storage device. In at least one embodiment, the VQA model installed on the cloud storage device is accessible via a cloud server.

[0010] In at least one embodiment, a dataset is input to a VQA model. In at least one embodiment, the dataset includes at least two datasets. In at least one embodiment, each dataset includes an image and a question. In at least one embodiment, each dataset is input to the VQA model multiple times. In at least one embodiment, a first dataset and a second dataset are input to the VQA model at a first instance / time. In at least one embodiment, the first dataset and the second dataset are input to the VQA model at a second instance / time.

[0011] In at least one embodiment, control of the vehicle is based on answers to natural language questions. In at least one embodiment, the VQA model is assigned to a viewpoint or angle from which images are acquired by at least one camera. In at least one embodiment, the images are images of the driver acquired by at least one camera. In at least one embodiment, the at least one camera is located inside the vehicle. In at least one embodiment, the images are images of vehicle occupants other than the driver acquired by at least one camera inside the vehicle. In at least one embodiment, the images are images of the surrounding / exterior area of ​​the vehicle acquired by at least one camera outside the vehicle.

[0012] In at least one embodiment, the image is in accordance with the image described above, but is acquired by a mobile device / terminal of a driver of the vehicle. In at least one embodiment, the image is in accordance with the image described above, but is acquired by a mobile device / terminal of a vehicle occupant other than the driver of the vehicle. In at least one embodiment, the image is acquired by a vehicle other than the target vehicle, in other words, by a vehicle adjacent to the target vehicle, within the field of view of a camera of the vehicle other than the target vehicle. In at least one embodiment, the image acquired by a mobile device / terminal of a driver or occupant of the vehicle is transmitted to at least one of the VQA models via a wired or wireless connection.

[0013] 1, an operational flowchart for controlling vehicle operation via a VQA model 100 is disclosed in accordance with at least some embodiments of the present disclosure. The following is an example of one use case scenario, although the basic method is similar to other use case scenarios. In at least one embodiment, a user 102 approaches a target vehicle. In at least one embodiment, the user is a person. In at least one embodiment, the user is holding an object 104, e.g., luggage. In at least one embodiment, the user approaches the target vehicle while holding the object. In at least one embodiment, the user approaches the target vehicle without holding the object.

[0014] In at least one embodiment, the controller 114 is configured to detect 116 a person approaching the target vehicle. In at least one embodiment, the controller 114 is configured to detect the person via a motion sensor. In at least one embodiment, the controller 114 is configured to detect the person via an image capture unit, e.g., a camera / video camera.

[0015] In at least some embodiments, controller 114 is in communication with VQA model 124. In at least some embodiments, controller 114 is configured to host and execute the functionality of the VQA model. In at least some embodiments, controller 114 is configured to perform vehicle operations. In at least some embodiments, controller 114 includes a microcontroller unit (MCU), a microprocessor unit (MPU), or an electronic control unit (ECU) configured to execute instructions stored in a memory component.

[0016] In at least some embodiments, controller 114 includes a memory component configured to store instructions and data, such as random access memory (RAM), read-only memory (ROM), or flash memory. In at least some embodiments, controller 114 includes a storage component configured to store data for long-term storage, such as a hard disk drive (HDD) or solid-state drive (SSD). In at least some embodiments, controller 114 includes an input / output (I / O) interface configured to communicate with external devices, such as an image acquisition unit. In at least some embodiments, controller 114 is an embedded system specifically designed for use in a vehicle.

[0017] In at least one embodiment, a first dataset is input to the VQA model 106. In at least one embodiment, the first dataset includes images, questions, and answers. In at least one embodiment, the images are captured by a camera, such as a camera capable of capturing still or video images. In at least one embodiment, the camera is focused on a user holding an object.

[0018] In at least one embodiment, the input is combined / paired with a question / query (118). In at least one embodiment, the question / query combined / paired with the image asks, "Is the user facing the camera?" (118). In at least one embodiment, the VQA model outputs an answer (126). In at least one embodiment, the VQA model outputs the answer "No," and in response to the answer "No," the controller does not perform a vehicle maneuver (126). In at least one embodiment, the VQA model outputs the answer. In at least one embodiment, the VQA model outputs the answer "Yes," and in response to the answer "Yes," the controller performs a vehicle maneuver (126).

[0019] In at least one embodiment, in response to the first data set indicating the answer "Yes," the controller is prompted / caused to input a second data set into the VQA model (108). In at least one embodiment, the second data set includes images, questions, and answers. In at least one embodiment, the images are captured by a camera similar to those disclosed above. In at least one embodiment, the camera is focused on a user holding an object.

[0020] In at least one embodiment, the input of the second dataset 108 is combined / paired with a question 120. In at least one embodiment, the question combined / paired with the image asks, "Is the user holding anything?" (120). In at least one embodiment, the VQA model 124 outputs an answer (128). In at least one embodiment, the VQA model outputs the answer "No," and in response to the answer "No," the controller does not execute (128) a vehicle maneuver 128. In at least one embodiment, the VQA model outputs the answer. In at least one embodiment, the VQA model outputs the answer "Yes," and in response to the answer "Yes," the controller executes (128) a vehicle maneuver. In at least one embodiment, the VQA model outputs the answer "Yes," and in response to the answer "Yes," the controller 114 executes (134) a second vehicle maneuver.

[0021] In at least one embodiment, in response to the second dataset indicating the answer "Yes," the controller is prompted / caused to input a third dataset 110 into the VQA model 124. In at least one embodiment, the third dataset includes images, questions, and answers. In at least one embodiment, the images are captured by a camera similar to those disclosed above. In at least one embodiment, the camera focuses on the user's facial expressions.

[0022] In at least one embodiment, the input of the third dataset 110 is combined / paired with a question (122). In at least one embodiment, the question combined / paired with the image asks, "Is the user making an expression?" (122). In at least one embodiment, the VQA model 124 outputs an answer. In at least one embodiment, the VQA model 124 outputs the answer "No," and in response to the answer "No," the controller 114 does not perform a vehicle maneuver (130). In at least one embodiment, the VQA model 124 outputs the answer. In at least one embodiment, the VQA model 124 outputs the answer "Yes," and in response to the answer "Yes," the controller performs the vehicle maneuver 130.

[0023] In at least one embodiment, if the VQA model outputs "Yes," the operation proceeds. In at least one embodiment, the controller receives the answer and continues inputting a second dataset. In at least one embodiment, the second dataset is the same images as the images in the first dataset, but paired with a different question. In at least one embodiment, the subsequent dataset is the same images with a different question relative to the previous dataset. In at least one embodiment, the subsequent dataset is different images with a different question relative to the previous dataset. In at least one embodiment, the subsequent dataset is different images with the same question relative to the previous dataset. In at least one embodiment, the second dataset is paired with the question "Is the user holding something?" In at least one embodiment, if the VQA model outputs the answer "No," the operation stops. In at least one embodiment, if the VQA model outputs the answer "Yes," the controller continues inputting a third dataset. In at least one embodiment, the third dataset is paired with the question "Is the user smiling?" In at least one embodiment, if the VQA model outputs a "No" answer, the operation stops. In at least one embodiment, if the VQA model outputs a "Yes" answer, the controller continues to perform any vehicle operations for which the controller is responsible, such as opening the back door / trunk. In at least one embodiment, if the controller recognizes that the user is facing the camera and is carrying luggage, the controller turns on the vehicle's backlights in addition to performing the operation.

[0024] In at least one embodiment, the controller formulates a question by referencing a sequence, and based on the sequence, the controller captures images from a camera, pairs the images with a predefined question to form a dataset, and inputs the dataset into the VQA model. In other words, the controller's commands are based on a predefined camera and a predefined view of the camera.

[0025] In at least one embodiment, the controller performs a predetermined vehicle control based on a binary "Yes" or "No" answer to the question. In at least one embodiment, the controller performs a predetermined vehicle control based on a non-binary answer to the question. In at least one embodiment, the answer is an integer. In at least one embodiment, the answer is a fraction.

[0026] In at least one embodiment, rather than generating a spontaneous predetermined question to initiate a sequence to complete execution of a predetermined vehicle operation, the controller is configured to ask a first question based on a first image of the user, the first question being associated with at least one predetermined question.

[0027] FIG. 2 is a block diagram of a hardware configuration for performing vehicle operations based on answer outputs from a VQA model, according to at least some embodiments of the present disclosure.

[0028] An exemplary hardware configuration includes a controller 202 that interacts with a VQA model 218 directly or through a network 214. In at least some embodiments, the network 214 is an Ethernet network, a controller area network (CAN), other wired or wireless network, or a combination thereof. In at least some embodiments, the controller 202 is a computer system that executes computer-readable instructions to perform vehicle operations.

[0029] In at least some embodiments, controller 202 includes a processor or programmable circuit that executes instructions, causing the processor or programmable circuit to perform operations in accordance with the instructions. In at least some embodiments, controller 202 includes analog or digital programmable circuitry, or any combination thereof. In at least some embodiments, controller 202 includes physically separate storage or circuitry that interacts through communications. In at least some embodiments, storage unit 204 includes non-volatile computer-readable media capable of storing executable and non-executable data accessed by controller 202 during execution of instructions. Communication interface 208 transmits data to and receives data from network 214. Input / output interface 210 connects to various input and output units to accept commands and present information via parallel ports, serial ports, keyboard ports, mouse ports, monitor ports, and the like. In some embodiments, storage unit 204 is external to controller 202.

[0030] In at least some embodiments, the apparatus is a separate device capable of processing logical functions to perform the operations herein. In at least some embodiments, the controller and storage unit need not be entirely separate devices, and in some embodiments, share circuitry or one or more computer-readable media. In at least some embodiments, the storage unit includes a hard drive that stores both computer-executable instructions and data accessed by the controller, and the controller includes a combination of a central processing unit (CPU) and RAM, where the computer-executable instructions are copied, in whole or in part, for execution by the CPU during performance of the operations herein.

[0031] In at least some embodiments where the device is a computer, a program installed on the computer can cause the computer to function as a device of the embodiments described herein or to perform operations associated with the device of the embodiments described herein, and in at least some embodiments, such a program can be executed by a processor to cause the computer to perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0032] At least some embodiments are described with reference to flowcharts and block diagrams in which blocks represent processing steps (1) in which operations are performed or sections of a controller (2) responsible for performing operations. In at least some embodiments, particular steps and sections are implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable medium, and / or a processor provided with computer-readable instructions stored on a computer-readable medium. In at least some embodiments, dedicated circuitry includes digital and / or analog hardware circuitry, including integrated circuits (ICs) and / or discrete circuits. In at least some embodiments, programmable circuitry includes reconfigurable hardware circuitry consisting of logical AND, OR, XOR, NAND, NOR, and other logic operations, flip-flops, registers, memory elements, etc., such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), etc.

[0033] In at least some embodiments, a computer-readable storage medium comprises a tangible device capable of holding and storing instructions for use by an instruction execution device. In some embodiments, a computer-readable storage medium includes, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, punch cards or mechanically encoded devices such as ridge structures in grooves having instructions recorded thereon, and any suitable combination thereof. As used herein, computer-readable media should not be construed as ephemeral signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0034] In at least some embodiments, the computer-readable program instructions described herein can be downloaded to a respective computing / processing device from a computer-readable storage medium or can be downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. In at least some embodiments, the network includes copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. In at least some embodiments, a network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.

[0035] In at least some embodiments, the computer-readable program instructions for performing the operations described above are either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, or equivalents, and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. In at least some embodiments, the computer-readable program instructions execute entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In at least some embodiments, in the latter scenario, the remote computer is connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or a connection is made to an external computer (e.g., through the Internet using an Internet Service Provider). In at least some embodiments, electronic circuitry, including, for example, programmable logic circuitry, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), executes computer-readable program instructions by utilizing state information in the computer-readable program instructions to individualize the electronic circuitry to perform aspects of the present invention.

[0036] Although the embodiments of the present invention have been described, the technical scope of all subject matter of the claims is not limited to the above-described embodiments. Those skilled in the art will understand that various modifications and improvements to the above-described embodiments are possible. Furthermore, those skilled in the art will understand from the claims that embodiments to which such modifications or improvements have been made are also included in the technical scope of the present invention.

[0037] The operations, procedures, steps, and stages of each process performed by the devices, systems, programs, and methods shown in the claims, embodiments, or figures can be performed in any order, unless the order is indicated by "before," "before," or equivalents thereof, and unless output from a previous process is used in a later process. Even if the flow of a process is described in the claims, embodiments, or figures using phrases such as "first" or "next," such description does not necessarily mean that the process is performed in the order described.

[0038] In at least some embodiments, vehicle operations responsive to the output of the VQA model are performed by an apparatus including a processor that executes instructions in accordance with the operations, or a controller that includes circuitry configured to perform the operations.

[0039] The foregoing outlines features of several embodiments so that those skilled in the art may better understand aspects of the present disclosure. Those skilled in the art should appreciate that this disclosure may readily be used as a basis for designing or modifying other processes and structures to carry out the same purposes and / or achieve the same advantages as the embodiments introduced herein. Those skilled in the art should also appreciate that such equivalent structures do not depart from the spirit and scope of the present disclosure, and that various changes, substitutions, and alterations can be made herein without departing from the spirit and scope of the present disclosure.

[0040] Supplementary Note 1 Aspects of the present disclosure relate to vehicles and vehicle operation. A non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause operations to be performed, including receiving an image of a candidate object of interest captured by a camera mounted on a vehicle. The operations include inputting a first dataset including the image and a first question associated with the candidate object of interest into a visual question answering (VQA) model. The operations include obtaining an answer to the first question as output from the VQA model. The operations include performing an operation of the vehicle based on the answer.

[0041] Supplementary Note 2 10. The vehicle operation of claim 1, wherein the operation further includes causing input to the VQA model of a second dataset including images from the first dataset and a second question associated with the candidate object of interest in response to the answer from the first dataset, and further including obtaining an answer to the second question as output from the VQA model.

[0042] Supplementary Note 3 3. The vehicle operation of any one of Supplementary Notes 1 to 2, wherein the input of the first data set and the second data set occurs multiple times, and the input of the second data set occurs after the input of the first data set.

[0043] Supplementary Note 4 4. The vehicle operation of any one of Supplementary Notes 1 to 3, wherein the first question is based on a pointing direction of the candidate object of interest relative to the camera.

[0044] Supplementary Note 5 5. A vehicle operation as described in any one of Supplementary Notes 1 to 4, wherein the candidate object of interest includes a vehicle occupant, a speed of the vehicle, or a change in pointing direction of a pointing motion of the candidate object of interest relative to an interior of the vehicle.

[0045] Supplementary Note 6 A vehicle operation described in any one of Supplementary Notes 1 to 5, wherein the names of the candidate objects of interest are highlighted in a graphical user interface (GUI).

[0046] Supplementary Note 7 7. The vehicle operation of any one of Supplementary Notes 1 to 6, further comprising a dataset other than the first dataset and the second dataset.

[0047] Supplementary Note 8 An aspect of the present disclosure relates to a method for implementing vehicle manipulation. The method includes receiving an image of a candidate object of interest captured by a camera mounted on a vehicle. The method includes inputting a first dataset including the image and a first question associated with the candidate object of interest into a VQA model. The method includes obtaining an answer to the first question as an output from the VQA model. The method includes performing, by a controller in the vehicle, a manipulation of the vehicle based on the answer in the first dataset.

[0048] Supplementary Note 9 10. The method of vehicle operation of Supplementary Note 8, further comprising: causing input to the VQA model of a second dataset comprising images of the first dataset and a second question associated with the candidate object of interest in response to the answer of the first dataset. The method of vehicle operation further comprises obtaining an answer to the second question as output from the VQA model.

[0049] Supplementary Note 10 10. The method of vehicle operation of any one of Supplementary Note 8 to Supplementary Note 9, wherein the input of the first data set and the second data set occurs multiple times, and the input of the second data set occurs after the input of the first data set.

[0050] Supplementary Note 11 11. The method of vehicle operation of any one of Supplementary Notes 8 to 10, wherein the first query is based on a pointing direction of the candidate object of interest relative to the camera.

[0051] Supplementary Note 12 12. A method of vehicle operation as described in any one of Supplementary Notes 8 to 11, wherein the candidate object of interest includes a vehicle occupant, a speed of the vehicle, or a change in pointing direction of a pointing motion of the candidate object of interest relative to an interior of the vehicle.

[0052] Supplementary Note 13 A method of vehicle operation described in any one of Supplementary Notes 8 to 12, wherein the names of the candidate objects of interest are highlighted in the GUI.

[0053] Supplementary Note 14 14. The method of vehicle operation of any one of Supplementary Notes 8 to 13, further comprising a dataset other than the first dataset and the second dataset.

[0054] Supplementary Note 15 An aspect of the present disclosure relates to a vehicle. The device includes a controller including a circuit configured to perform a vehicle operation, including performing the vehicle operation based on an answer of a first dataset. The device includes a processor configured to receive images of a candidate object of interest captured by a camera mounted on the vehicle. The device includes a processor configured to input the first dataset, including the images and a first question associated with the candidate object of interest, into a VQA model. The device includes a processor configured to obtain an answer to the first question as an output from the VQA model.

[0055] Supplementary Note 16 Supplementary Note 15. The apparatus of Supplementary Note 15, further configured to: cause input to the VQA model of a second dataset comprising images of the first dataset and a second question associated with the candidate object of interest in response to the answer of the first dataset. The apparatus is further configured to obtain the answer to the second question as output from the VQA model.

[0056] Supplementary Note 17 17. The apparatus of any one of Supplementary Notes 15-16, wherein the input of the first data set and the second data set occurs multiple times, the input of the second data set occurring after the input of the first data set.

[0057] Supplementary Note 18 18. The apparatus of any one of Supplementary Notes 15 to 17, further comprising a microphone associated with the vehicle, the microphone configured to process context data including audio data received by the microphone within a predetermined time period during which an image of the candidate object of interest is acquired.

[0058] Supplementary Note 19 19. The apparatus of any one of Supplementary Notes 15 to 18, wherein the candidate object of interest includes a vehicle occupant, a speed of the vehicle, or a change in pointing direction of a pointing motion of the candidate object of interest relative to an interior of the vehicle.

[0059] Supplementary Note 20 20. The apparatus of any one of Supplementary Notes 15 to 19, wherein the names of the candidate objects of interest are highlighted in the GUI.

Claims

1. receiving an image of a candidate object of interest captured by a camera mounted on a vehicle; inputting a first dataset including the images and a first question associated with the candidate object of interest into a visual question answering (VQA) model; obtaining an answer to the first question as output from the VQA model; performing an operation of the vehicle based on the response; and A computer program that causes a processor to perform operations including:

2. The operation further comprises: triggering input of a second dataset to the VQA model, the second dataset including images from the first dataset and second questions associated with the candidate object of interest, in response to the answers from the first dataset; obtaining an answer to the second question as an output from the VQA model; and 2. The computer program of claim 1, comprising:

3. 3. The computer program of claim 2, wherein the input of the first data set and the second data set occurs multiple times, the input of the second data set occurring after the input of the first data set.

4. The computer program product of claim 1 , wherein the first query is based on a pointing direction of the candidate object of interest relative to the camera.

5. The computer program product of claim 1 , wherein the candidate object of interest comprises a vehicle occupant, a speed of the vehicle, or a change in pointing direction of a pointing motion of the candidate object of interest relative to an interior of the vehicle.

6. 4. The computer program of claim 2 or 3, wherein the names of the candidate objects of interest are highlighted in a graphical user interface (GUI).

7. The computer program of claim 2 or 3, further comprising a dataset other than the first dataset and the second dataset.

8. 1. A method executed by one or more processors, comprising: receiving an image of a candidate object of interest captured by a camera mounted on a vehicle; inputting a first dataset including the images and a first question associated with the candidate object of interest into a VQA model; obtaining an answer to the first question as output from the VQA model; performing, by a controller within the vehicle, an operation of the vehicle based on the response of the first data set; A method comprising:

9. triggering input of a second dataset to the VQA model, the second dataset including images from the first dataset and second questions associated with the candidate object of interest, in response to the answers from the first dataset; obtaining an answer to the second question as an output from the VQA model; and The method of claim 8 further comprising:

10. 10. The method of claim 9, wherein the input of the first data set and the second data set occurs multiple times, the input of the second data set occurring after the input of the first data set.

11. The method of claim 8 , wherein the first query is based on a pointing direction of the candidate object of interest relative to the camera.

12. The method of claim 9 or 10, wherein the potential object of interest comprises a vehicle occupant, a speed of the vehicle, or a change in pointing direction of a pointing motion of the potential object of interest relative to an interior of the vehicle.

13. The method of claim 9 or 10, wherein the names of the potential objects of interest are highlighted in the GUI.

14. The method of claim 9 or 10, further comprising a dataset other than the first dataset and the second dataset.

15. a controller including circuitry configured to perform an operation of the vehicle, including performing an operation of the vehicle based on a response of the first data set; receiving an image of a potential object of interest captured by a camera mounted on the vehicle; inputting a first dataset including the images and a first question associated with the candidate object of interest into a VQA model; Obtaining an answer to the first question as an output from the VQA model. and a processor configured as An apparatus comprising:

16. The processor further comprises: triggering input of a second dataset to the VQA model, the second dataset including images from the first dataset and second questions associated with the candidate object of interest, in response to the answers from the first dataset; Obtaining an answer to the second question as an output from the VQA model.

16. The apparatus of claim 15, configured to:

17. 17. The apparatus of claim 15 or 16, wherein the input of the first data set and the second data set occurs multiple times, the input of the second data set occurring after the input of the first data set.

18. further comprising a microphone associated with the vehicle; 17. The apparatus of claim 15 or 16, wherein the microphone is configured to process context data including audio data received by the microphone within a preset time period during which an image of the candidate object of interest is acquired.

19. The apparatus of claim 16 , wherein the potential object of interest comprises a vehicle occupant, a speed of the vehicle, or a change in pointing direction of a pointing motion of the potential object of interest relative to an interior of the vehicle.

20. The apparatus of claim 16 , wherein names of the candidate objects of interest are highlighted in the GUI.

Images