Automatic answering apparatus, automatic answering method, and computer program for automatic answering

Abstract

To provide an automatic answering device capable of generating an appropriate answer to a question about a specific object shown in an image while suppressing man-hours.SOLUTION: The automatic answering unit 6 includes a preprocessing unit 33 that executes preprocessing on an image so as to extract a predetermined area of the image in response to a question, and an answer generation unit 34 that generates an answer by inputting the preprocessed image and the question to a generation model that has been learned in advance so as to generate an answer to the question.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an automatic answering device, an automatic answering method, and a computer program for automatic answering that automatically answers questions from users. [Background technology]

[0002] A generative model (Vision Language Model, hereafter referred to as VLM) is known that, when an image and a question related to the image given as text are input, generates an answer to the question by referring to the image. A technique has been proposed in which a red circle is added around an object of interest in the image input to this VLM to direct the VLM's attention to that object (see Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Aleksandar Shtedritski et al., "What does CLIP know about a red circle? Visual prompt engineering for VLMs", 2023 IEEE / CVF International Conference on Computer Vision (ICCV)), https: / / dx.doi.org / 10.1109 / ICCV51070.2023.01101 Summary of the Invention [Problem to be solved by the invention]

[0004] The objects associated with the question entered into the VLM vary depending on the question, so the above technology requires marking the image with a red circle every time a question is asked, which is extremely time-consuming.

[0005] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an automatic answering device that can generate appropriate answers to questions about specific objects shown in images while reducing the number of steps required. [Means for solving the problem]

[0006] According to one embodiment, there is provided an automated answering machine including a preprocessing unit that preprocesses an image to extract a predetermined region from the image in response to a question, and an answer generation unit that generates an answer by inputting the preprocessed image and the question into a generative model that has been trained in advance to generate an answer to the question.

[0007] In one embodiment, the automatic response device further has a recognition unit that recognizes each of at least one specified object depicted in the image, and when the question includes identification information of any of the at least one specified object, the preprocessing unit determines that the specified area is an object area in which an object identified by the identification information of the at least one specified object is depicted in the image.

[0008] In one embodiment, the preprocessing unit determines the entire image as the predetermined region if the query does not include any identifying information of the at least one object.

[0009] In one embodiment, the preprocessing unit identifies an area in the image that the person focuses on based on the person's posture associated with the question, and sets the identified area as the predetermined area.

[0010] According to another embodiment, there is provided an automated response method that includes, in response to a question, performing preprocessing on an image to extract a predetermined region of the image, and generating an answer by inputting the preprocessed image and the question to a generative model that has been pre-trained to generate an answer to the question.

[0011] According to yet another embodiment, there is provided a computer program for automated responses, the computer program including instructions for causing a computer to preprocess an image to extract a predetermined region of the image in response to a question, and to generate an answer by inputting the preprocessed image and the question into a generative model that has been pre-trained to generate an answer to the question. [Effects of the Invention]

[0012] The automatic answering device according to the present disclosure has the effect of being able to generate appropriate answers to questions about a specific person depicted in an image while reducing the number of steps. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a schematic diagram of a vehicle equipped with an automatic response device; [Figure 2] FIG. 2 is a diagram illustrating a hardware configuration of an automatic answering device. [Figure 3] FIG. 2 is a functional block diagram of a processor of an automatic answering device. [Figure 4] FIG. 2 is an explanatory diagram of input and output of the answer generation model according to the present embodiment. [Figure 5] 10 is a flowchart illustrating the operation of an automatic answering device. DETAILED DESCRIPTION OF THE INVENTION

[0014] The automatic answering device, and the automatic answering method and computer program executed by the automatic answering device will be described below with reference to the drawings. The automatic answering device generates an answer by inputting a preprocessed image obtained by preprocessing an image to extract a predetermined area from the image and the question into a generative model that has been trained in advance to generate an answer to the question.

[0015] In the following, an embodiment will be described in which an automatic response device is installed in a vehicle to automatically generate answers to questions from vehicle occupants.

[0016] 1 is a schematic diagram of a vehicle equipped with an automatic answering device. In this embodiment, the vehicle 1 has an exterior camera 2, an interior camera 3, at least one microphone 4, a notification device 5, and an automatic answering device 6. The exterior camera 2, the interior camera 3, the microphone 4, the notification device 5, and the automatic answering device 6 are connected to each other so that they can communicate with each other.

[0017] The exterior camera 2 is an example of an exterior imaging unit, and is mounted inside the cabin of the vehicle 1 facing a predetermined area around the vehicle 1 (for example, the area in front of the vehicle 1) so that it can capture images of that area. The exterior camera 2 generates an image representing the predetermined area at each predetermined capture interval and outputs the generated image to the automatic response device 6. Note that, hereinafter, the image generated by the exterior camera 2 may be referred to as an exterior image.

[0018] The in-vehicle camera 3 is an example of an in-vehicle imaging unit, and is mounted near the top edge of the windshield facing toward the interior of the vehicle so that all passengers in the vehicle 1 are included in the imaging target area. The in-vehicle camera 3 generates an image representing the imaging target area at each predetermined imaging cycle and outputs the generated image to the automatic answering device 6. Note that, hereinafter, the image generated by the in-vehicle camera 3 may be referred to as an in-vehicle image.

[0019] At least one microphone 4 picks up the voice uttered by any of the passengers aboard the vehicle 1 and outputs an audio signal representing the voice. To this end, each microphone 4 is attached inside the cabin of the vehicle 1. Note that multiple microphones 4 may be attached in an array, or a microphone 4 may be attached around each seat in the cabin of the vehicle 1.

[0020] The notification device 5 is provided in the passenger compartment of the vehicle 1 and notifies the occupant of the answer generated by the automatic answering device 6. To this end, the notification device 5 has, for example, at least either a speaker or a display device. When the notification device 5 receives an answer signal representing an answer to the occupant from the automatic answering device 6, it notifies the driver of the answer by voice from the speaker or by displaying a message, image, or video on the display device.

[0021] The automatic response device 6 generates an answer to the question from the occupant of the vehicle 1 and notifies the occupant of the vehicle 1 of the generated answer via the notification device 5.

[0022] Fig. 2 is a hardware configuration diagram of the automatic answering device 6. As shown in Fig. 2, the automatic answering device 6 has a communication interface 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 may each be configured as separate circuits, or may be configured integrally as a single integrated circuit.

[0023] The communication interface 21 has an interface circuit for connecting the automatic answering device 6 to other devices inside the vehicle. The communication interface 21 passes the outside image received from the exterior camera 2, the inside image received from the interior camera 3, and the audio signals received from the individual microphones 4 to the processor 23. The communication interface 21 also outputs the response signal received from the processor 23 to the notification device 5.

[0024] The memory 22 is an example of a storage unit and includes, for example, a volatile semiconductor memory and a nonvolatile semiconductor memory. The memory 22 stores various data used in the automatic response process executed by the processor 23. Specifically, the memory 22 stores parameters defining a classifier used to recognize a predetermined object depicted in an interior or exterior image of the vehicle and parameters defining a generative model for generating a response. Furthermore, the memory 22 stores, for each of one or more registered persons registered in advance, a feature vector (hereinafter referred to as a registration vector) representing the characteristics of the registered person and identification information (e.g., the registered person's name, nickname, or identification number). Furthermore, the memory 22 may temporarily store exterior images received from the exterior camera 2, interior images received from the interior camera 3, and audio signals received from each microphone 4.

[0025] The processor 23 includes one or more central processing units (CPUs) and their peripheral circuits. The processor 23 may further include other arithmetic circuits such as a logic unit, a numerical calculation unit, or a graphics processing unit. The processor 23 executes the automatic response process.

[0026] 3 is a functional block diagram of processor 23 related to automatic response processing. Processor 23 has a voice recognition unit 31, an image recognition unit 32, a preprocessing unit 33, a response generation unit 34, and a notification processing unit 35. Each of these units in processor 23 is, for example, a functional module realized by a computer program running on processor 23. Alternatively, each of these units in processor 23 may be a dedicated arithmetic circuit provided in processor 23.

[0027] The speech recognition unit 31 recognizes a question uttered by one of the occupants based on a voice signal representing voices in the vehicle cabin collected by the microphone 4. To this end, the speech recognition unit 31 inputs the voice signal into a speech recognition model to recognize the question expressed in the voice signal. Such a speech recognition model may be configured as a deep neural network (DNN) with an attention mechanism or a recurrent neural network (RNN) with a recursive structure. Alternatively, the speech recognition model may be configured as a GMM-HMM based on a mixture normal distribution and a hidden Markov model, or a DNN-HMM based on a DNN and a hidden Markov model. The speech recognition model then outputs the question expressed in the input voice signal as text data. Note that the speech recognition unit 31 may recognize the question expressed in the voice signal by dividing the voice signal into frames having a predetermined time length, extracting voice features for each frame, and inputting the features for each frame into the speech recognition model in chronological order. The features for each frame may be, for example, a predetermined element of the cepstrum of that frame.

[0028] The voice recognition unit 31 outputs text data representing a question recognized from a voice signal to the image recognition unit 32, the preprocessing unit 33, and the answer generation unit .

[0029] The image recognition unit 32 is an example of a recognition unit, and recognizes at least one predetermined object that may be the subject of a question, which is depicted in an exterior image or an interior image of the vehicle. The predetermined object that may be the subject of a question may be set in advance, or may be an object of a type indicated in text data representing a question. The predetermined object may be, for example, an occupant of the vehicle 1 or a specific part of the occupant, another vehicle traveling around the vehicle 1, a pedestrian, or a specific structure such as a building or a sign.

[0030] When recognizing an occupant of the vehicle 1 as a predetermined object, the image recognition unit 32 detects a person area in the vehicle interior image by inputting the vehicle interior image into a classifier that has been trained in advance to detect an area representing the occupant (hereinafter referred to as a person area). As a result, a person area representing each occupant in the vehicle interior of the vehicle 1 is detected. Such a classifier is configured as a DNN having a convolutional neural network (CNN) type architecture such as a Single Shot MultiBox Detector or a DNN having an attention mechanism such as a Vision Transformer. Alternatively, such a classifier may be configured as a classifier based on a machine learning method other than a DNN, such as an AdaBoost classifier.

[0031] Next, the image recognition unit 32 inputs each detected person region into a feature extractor that has been trained in advance to extract feature vectors representing the characteristics of the occupants depicted in the person region, thereby extracting a feature vector from each person region. Such a feature extractor is configured as a DNN that is trained in advance by so-called unsupervised learning, such as an auto-encoder or stacked what-where auto-encoders. In this case, the feature extractor includes, in order from the input side, an encoder that outputs feature quantities with reduced dimensions compared to the input data (in this embodiment, the person region), and a decoder that receives the feature quantities output from the encoder. The feature extractor is trained in advance using a large number of images depicting various people so that the data input to the encoder and the data output from the decoder are the same. Then, by inputting the person regions into the trained feature extractor, the feature quantities output by the encoder are obtained as feature vectors representing the characteristics of the occupants depicted in the person regions. Furthermore, the feature extractor may be configured as a DNN that is trained using a self-supervised learning method, such as self-supervised learning.

[0032] For each detected person region, the image recognition unit 32 calculates the degree of match (e.g., cosine similarity) between the feature vector extracted from that person region and the registered vector of each registered person who has been registered in advance. The image recognition unit 32 then identifies the registered person with the highest degree of match as the occupant depicted in that person region. Note that if the maximum degree of match is less than a predetermined threshold value, the image recognition unit 32 may determine that the occupant depicted in the person region is not one of the registered persons.

[0033] For each detected person region, the image recognition unit 32 outputs position information indicating the position and range of the person region on the image (for example, the coordinates of the upper left corner and lower right corner of the person region) to the preprocessing unit 33. Furthermore, for each detected person region, the image recognition unit 32 may output identification information of the occupant depicted in the person region to the answer generation unit 34. Note that for an occupant who is different from any registered person, the image recognition unit 32 outputs data indicating that the occupant is an unregistered person (for example, text data saying "guest") as the occupant's identification information.

[0034] Furthermore, in order to recognize a predetermined object existing around the vehicle 1, the image recognition unit 32 detects an object region in which the predetermined object is represented on the vehicle exterior image by inputting the vehicle exterior image into a classifier that has been trained in advance to detect such a predetermined object. The classifier may have a configuration similar to that of a classifier used to detect a person region.

[0035] Furthermore, if the text data representing the question includes a demonstrative term, the image recognition unit 32 may estimate the posture of the occupant based on a person area detected from the in-vehicle image. If the estimated posture points in a specific direction, the image recognition unit 32 may set an object present in that specific direction as the object to be recognized. In this embodiment, all occupants in the vehicle 1 are examples of people related to the question because they may be influenced by the answer to a question asked by any occupant. Therefore, the occupant whose posture is to be estimated may be different from the occupant who asked the question.

[0036] In this case, the image recognition unit 32 may compare pre-registered demonstratives with text data representing a question to determine whether the text data contains a demonstrative. If the demonstratives are included, the image recognition unit 32 inputs each person region detected from the vehicle interior image to a pose estimator to estimate the pose of the occupant depicted in the person region. The pose estimator is configured as a pose estimation model that performs pose estimation based on a characteristic structure such as the human skeleton. Such a pose estimation model may be, for example, a DNN with a CNN-type architecture.

[0037] If the estimated posture of the occupant shows either the left or right hand pointing in a certain direction, the image recognition unit 32 determines that the posture of the occupant is pointing in a specific direction. The image recognition unit 32 determines whether the left or right hand is pointing in a certain direction and the direction the hand is pointing in the vehicle exterior image by template matching between a template representing a hand shape prepared for each pointing direction and either the left or right hand in the estimated posture. The image recognition unit 32 then estimates the direction the hand is pointing in real space based on the direction the hand is pointing in the vehicle interior image. The image recognition unit 32 estimates the direction the hand is pointing in real space by referring to a table that represents the relationship between the direction the hand is pointing in real space and the direction the hand is pointing in the vehicle interior image. Such a table may be stored in the memory 22 in advance. The image recognition unit 32 identifies a position on the outside-of-vehicle image that corresponds to the identified direction based on the estimated hand pointing direction and the installation position, angle of view, and shooting direction of the outside-vehicle camera 2. The image recognition unit 32 then determines, as the object to be recognized, an object depicted in an object region located within a predetermined range from the identified position.

[0038] Furthermore, if the recognized object is a structure, the image recognition unit 32 may refer to map information to identify the name of the structure depicted in the object area. In this case, the image recognition unit 32 identifies a vector leading from the position of the vehicle 1 at the time and date when the vehicle exterior image in which the specified object is detected was generated (hereinafter, sometimes simply referred to as the generation time and date), to the structure depicted in the object area, based on parameters such as the position and orientation of the vehicle 1 at the time and date when the vehicle exterior image in which the specified object is detected was generated, the position of the object area in the vehicle exterior image, and the shooting direction of the exterior camera 2. The image recognition unit 32 then refers to the map information to identify a structure that is located within a predetermined allowable error range from the vector and is of the same type as the structure depicted in the object area, and sets the name of the identified structure as the name of the structure depicted in the object area. The map information may be stored in advance in the memory 22.

[0039] For each detected object region, the image recognition unit 32 outputs position information indicating the position and range of the object region on the image (for example, the coordinates of the upper left and lower right corners of the object region) to the pre-processing unit 33. Furthermore, the image recognition unit 32 notifies the pre-processing unit 33 of the object pointed to by the occupant's posture. Furthermore, for each detected object region, the image recognition unit 32 outputs type information indicating the type of object represented in the object region to the answer generation unit 34. Furthermore, when the name of the object represented in the detected object region is identified, the image recognition unit 32 may output the name of the object represented in the object region to the answer generation unit 34.

[0040] The preprocessing unit 33 preprocesses the vehicle exterior image or the vehicle interior image so as to extract a predetermined area from the vehicle exterior image or the vehicle interior image in response to the question. The image obtained by preprocessing is input to a generation model (hereinafter referred to as an answer generation model) that has been trained in advance to generate an answer to the question. Details of the answer generation model will be described later together with the answer generation unit 34.

[0041] The preprocessing unit 33 references the text data representing the question received from the voice recognition unit 31, the identification information of the occupant depicted in the vehicle interior image, and the name of the object detected in the vehicle exterior image received from the image recognition unit 32. If the text data representing the question contains the identification information of the occupant, the preprocessing unit 33 extracts a person region representing the occupant identified by the identification information contained in the text data as a predetermined region to be input to the answer generation model. The preprocessing unit 33 then preprocesses the vehicle interior image so that regions other than the person region are masked. Similarly, if the text data representing the question contains the name of an object detected in the vehicle exterior image, the preprocessing unit 33 extracts an object region representing the object identified by the name as a predetermined region to be input to the answer generation model. The preprocessing unit 33 then preprocesses the vehicle exterior image so that regions other than the object region are masked.

[0042] As a result, image information about objects other than the object that is the subject of the question is not input to the answer generation model, so that the predetermined area to be extracted is appropriately set, and it becomes easier to generate a more appropriate answer to the question. For example, if the question is "Is Mr. A sleeping?", the vehicle interior image is preprocessed so that areas other than the person area representing the occupant Mr. A are masked. Note that if the text data representing the question includes identification information or names of multiple objects depicted in the image, the preprocessing unit 33 preprocesses the vehicle interior image or vehicle exterior image so that areas other than the person (object) areas corresponding to each of the multiple identification information or names are masked.

[0043] Furthermore, if an object is pointed at by an occupant, the object region in which the object is represented is estimated to be the region the occupant focuses on. Therefore, the preprocessing unit 33 sets the object region in which the object pointed at by the occupant is represented as a predetermined region to be input to the answer generation model. The preprocessing unit 33 then preprocesses the outside-of-vehicle image so that regions other than the object region are masked. For example, if one of the occupants is pointing at a preceding vehicle traveling in front of the vehicle 1, the object region in which the preceding vehicle is represented in the outside-of-vehicle image becomes the predetermined region to be extracted. In this case, too, the predetermined region to be extracted is appropriately set, and image information regarding objects other than the object being queried is not input to the answer generation model, making it easier to generate a more appropriate answer to a question.

[0044] As described above, when the preprocessing unit 33 determines that an area other than a person area representing a specific occupant should be masked, the preprocessing unit 33 generates a preprocessed image in which the area other than the person area is masked by cropping only the person area from the vehicle interior image or by replacing the values ​​of each pixel other than the person area with a predetermined pixel value. Similarly, when the preprocessing unit 33 determines that an area other than an object area representing a specific object should be masked, the preprocessing unit 33 generates a preprocessed image in which the area other than the object area is masked by cropping only the object area from the vehicle exterior image or by replacing the values ​​of each pixel other than the object area with a predetermined pixel value.

[0045] Furthermore, if the text data representing the question does not include identification information representing any occupants depicted in the vehicle interior image or the names of any objects detected in the vehicle exterior image, the preprocessing unit 33 sets the entire vehicle interior image or the entire vehicle exterior image as the predetermined region to be extracted. In this case, the preprocessing unit 33 sets the entire vehicle interior image and the entire vehicle exterior image as the preprocessed image. This is because the question does not relate to a specific occupant or a specific object, and generating an appropriate answer likely requires reference to the status of each occupant depicted in the vehicle interior image or the entire vehicle exterior image. For example, if the question is "Does everyone look hot?", the entire vehicle interior image is the predetermined region to be input to the answer generation model. In this way, if the text data representing the question does not include identification information representing any objects depicted in the images, the predetermined region to be extracted is appropriately set by setting the entire vehicle interior image or the entire vehicle exterior image as the predetermined region.

[0046] The preprocessing unit 33 notifies the answer generating unit 34 of the preprocessed image.

[0047] The answer generation unit 34 generates an answer to the question by inputting the image preprocessed by the preprocessing unit 33 and the question into the answer generation model.

[0048] In this embodiment, the answer generation model is configured as a VLM. The VLM serving as the answer generation model is configured, for example, as a combination of an image encoder that encodes an input preprocessed image and a large-scale language model (LLM) in which multiple blocks, each including an attention layer and a feed forward layer, are stacked. The answer generation unit 34 inputs the preprocessed image to the image encoder and inputs text data representing a question to the input layer of the LLM.

[0049] When the predetermined region to be input to the answer generation model is the entire interior image or the entire exterior image, the answer generation unit 34 determines whether the text data representing the question includes terms related to the surroundings of the vehicle 1. If the text data representing the question includes terms related to the surroundings of the vehicle 1 (e.g., “pedestrian,” “vehicle in front,” “rain,” “traffic jam,” etc.), the answer generation unit 34 inputs the exterior image to the answer generation model. On the other hand, if the text data representing the question does not include terms related to the surroundings of the vehicle 1, the answer generation unit 34 inputs the interior image to the answer generation model. The terms related to the surroundings of the vehicle 1 may be stored in advance in the memory 22. Alternatively, when the predetermined region to be input to the answer generation model is the entire interior image or the entire exterior image, the answer generation unit 34 may input an image obtained by combining the interior image and the exterior image so that the images are aligned horizontally or vertically to the answer generation model. Furthermore, the answer generation unit 34 may input an image obtained by further downsampling the combined images to the answer generation model.

[0050] The question input to the answer generation model may be unrelated to the occupants recognized from the interior image and the objects depicted in the exterior image. In this case, the answer generation model is trained in advance to generate an answer to the question regardless of the input preprocessed image.

[0051] The answer generating unit 34 outputs text data representing the generated answer to the notification processing unit 35.

[0052] The notification processing unit 35 outputs the answer to the question via the notification device 5. For example, the notification processing unit 35 generates an audio signal representing the answer according to a predetermined voice synthesis method based on text data representing the answer received from the answer generation unit 34. The notification processing unit 35 then outputs the generated audio signal to a speaker included in the notification device 5, thereby causing the speaker to output audio representing the answer. Alternatively, the notification processing unit 35 causes a display device included in the notification device 5 to display the text data representing the answer.

[0053] 4 is an explanatory diagram of input and output of the answer generation model according to this embodiment. In this embodiment, an image 401 preprocessed by the preprocessing unit 33 and text data 402 representing a question are input to the answer generation model 400. In this example, the preprocessed image 401 is masked except for a person area 401a representing an occupant identified by the occupant identification information "Mr. A" included in the text data 402, thereby extracting the person area 401a. The answer generation model 400 then references the input preprocessed image 401 and text data 402, and outputs text data 403 representing an answer to the question.

[0054] 5 is a flowchart showing the automatic response process according to this embodiment. The processor 23 executes the automatic response process according to this flowchart.

[0055] The voice recognition unit 31 recognizes a question uttered by any occupant based on a voice signal representing a voice inside the vehicle 1 (step S101). The image recognition unit 32 recognizes a predetermined object shown in the vehicle interior image and the vehicle exterior image (step S102).

[0056] The preprocessing unit 33 determines whether the query contains identification information of a predetermined object (step S103). If the query contains identification information (step S103—Yes), the preprocessing unit 33 sets the area representing the predetermined object identified by the identification information as the predetermined area to be extracted (step S104). If the query does not contain identification information (step S103—No), the preprocessing unit 33 determines whether the occupant of the vehicle 1 is pointing in a specific direction (step S105). If the occupant is pointing in a specific direction (step S105—Yes), the preprocessing unit 33 sets the area representing an object in the specific direction in the vehicle exterior image as the predetermined area to be extracted (step S106). If the occupant is not pointing in a specific direction (step S105—No), the preprocessing unit 33 sets the entire vehicle interior image or the entire vehicle exterior image as the predetermined area (step S107).

[0057] After step S104, S106, or S107, the preprocessing unit 33 preprocesses the interior or exterior image to extract a predetermined region (step S08). The answer generation unit 34 generates an answer to the question by inputting the preprocessed image and the question into an answer generation model (step S109). The notification processing unit 35 then notifies the occupant of the generated answer via the notification device 5 (step S110).

[0058] As described above, this automatic answering device generates an answer to a question by inputting the question and a preprocessed image obtained by performing preprocessing on the image to extract a predetermined area from the image in response to the question into the answer generation model. Therefore, this automatic answering device automatically extracts an area representing an object related to the question from among the individual objects depicted in the image input to the answer generation model, thereby generating an appropriate answer to the question while reducing labor costs.

[0059] According to a variant, position information representing the positions of recognized individual occupants on the interior image may be input to the answer generation model together with the preprocessed image and the question. Similarly, position information representing the positions of detected objects on the exterior image may be input to the answer generation model together with the preprocessed image and the question.

[0060] According to another modification, the images to be input to the answer generation model may be limited to either an image obtained by preprocessing an interior image of the vehicle (including the entire interior image) or an image obtained by preprocessing an exterior image of the vehicle (including the entire exterior image). In this case, the processing by the image recognition unit 32 and the preprocessing unit 33 for the image that is not the input target may be omitted.

[0061] According to yet another modification, the answer to the question may be for executing control of the vehicle 1 or control of a device mounted on the vehicle 1. In this case, the answer generation model outputs text data representing the content of the control. The answer generation unit 34 determines the device to be controlled and the control command by referring to a reference table that represents the correspondence between the text data representing the content of the control, the device to be controlled (including the vehicle 1 itself), and the control command for executing the control. The answer generation unit 34 then outputs the identified control command to the control unit of the device to be controlled via the communication interface 21.

[0062] Furthermore, this automatic response device is not limited to an in-vehicle embodiment, but can be used in various systems that are capable of capturing an image of a predetermined object that may be the subject of a question and that are required to generate an answer to a question about the predetermined object. For example, this automatic response device may be installed in a predetermined space in a facility and generate an answer to a question about one or more objects located in the space. Furthermore, questions may be input via a user interface that allows input of text data, such as a keyboard or a touch panel. In this case, the processing of the voice recognition unit 31 may be omitted.

[0063] A computer program for realizing the automatic response process according to the above embodiment or modification may be provided in a form recorded on a computer-readable portable recording medium.

[0064] As described above, those skilled in the art can make various modifications to the embodiments within the scope of the present invention. [Explanation of symbols]

[0065] REFERENCE SIGNS LIST 1 vehicle, 2 exterior camera, 3 interior camera, 4 microphone, 5 notification device, 6 automatic answering device, 21 communication interface, 22 memory, 23 processor, 31 voice recognition unit, 32 image recognition unit, 33 preprocessing unit, 34 response generation unit, 35 notification processing unit

Claims

1. a preprocessing unit that performs preprocessing on the image in response to a query to extract a predetermined region of the image; an answer generation unit that generates the answer by inputting the preprocessed image and the question into a generative model that has been trained in advance to generate an answer to the question; An automatic answering device having:

2. a recognition unit that recognizes at least one predetermined object depicted in the image; The automatic response device described in claim 1, wherein when the question includes identification information of any of the at least one specified object, the preprocessing unit determines that the specified area is an object area in the image that represents an object identified by the identification information of the at least one specified object.

3. 3. The automatic answering device according to claim 2, wherein the preprocessing unit determines the entire image as the predetermined region when the question does not include any identification information of at least one predetermined object.

4. The automatic answering device according to claim 2 , wherein the preprocessing unit identifies an area in the image that the person focuses on based on the person's posture related to the question, and sets the identified area as the predetermined area.

5. performing pre-processing on the image to extract a predetermined region of the image in response to a query; generating the answer by inputting the preprocessed image and the question into a generative model that has been trained to generate an answer to the question; An automatic response method including:

6. performing pre-processing on the image to extract a predetermined region of the image in response to a query; generating the answer by inputting the preprocessed image and the question into a generative model that has been trained to generate an answer to the question; An automatic response computer program that causes a computer to do the following:

Images