Automatic answering apparatus, automatic answering method, and computer program for automatic answering

Abstract

To provide an automatic answering device capable of generating an appropriate answer to a question about a specific person represented in an image.SOLUTION: The automatic answering unit 5 includes a recognition unit 31 that recognizes any of at least one person represented in an image, and an answer generation unit 34 that generates an answer by inputting identification information for identifying the recognized person, the image, and a question related to the recognized person to a generation model that has been trained in advance to generate an answer to the question.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an automatic answering device, an automatic answering method, and a computer program for automatic answering that automatically answers questions from users. [Background technology]

[0002] A generative model (Vision Language Model, hereinafter referred to as VLM) has been proposed that, when an image and a question related to the image given as text are input, generates an answer to the question by referencing the image (see Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Yash Goyal et al., “Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering”, International Journal of Computer Vision, Volume 127, Issue 4, April 2019, pp 398-414, https: / / doi.org / 10.1007 / s11263-018-1116-0 Summary of the Invention [Problem to be solved by the invention]

[0004] In response to an input question, the VLM will provide a general answer about the objects depicted in the image within the scope of its own knowledge. However, the VLM may not be able to provide an appropriate answer to a question about a specific one of the multiple objects depicted in the image. For example, if an input image shows multiple people, even if a question about a specific one of those people is input, the VLM itself may not be able to identify which of those people is the subject of the question, and therefore may not be able to generate an appropriate answer.

[0005] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an automatic answering device that can generate appropriate answers to questions about a specific person depicted in an image. [Means for solving the problem]

[0006] According to one embodiment, there is provided an automated response device including a recognition unit that recognizes at least one person depicted in an image, and an answer generation unit that generates an answer by inputting identification information for identifying the recognized person, the image, and a question related to the recognized person into a generative model that has been trained in advance to generate answers to questions.

[0007] In one embodiment, the answer generator further inputs, together with the question, position information representing the position of a person region in which a person recognized in the image is depicted into the generative model.

[0008] In one embodiment, the answer generation unit preprocesses the image so that areas other than the person area representing the person recognized in the image are masked, and the preprocessed image is used as the image to be input to the generative model.

[0009] In one embodiment, the automatic response device further has a selection unit that selects, depending on the question, whether to input the entire image or a preprocessed image obtained by preprocessing the image into the model, and the answer generation unit inputs the selected one of the entire image or the preprocessed image into the generative model.

[0010] In one embodiment, the selection unit further selects, in response to the question, whether or not to input location information representing the location of a person area representing a person recognized in the image into the model, and the answer generation unit further inputs the location information into the generative model if it is selected to input the location information into the generative model.

[0011] According to another embodiment, there is provided an automated response method, which includes recognizing any of at least one person depicted in an image, and generating an answer by inputting identification information for identifying the recognized person, the image, and a question related to the recognized person into a generative model pre-trained to generate an answer to the question.

[0012] According to yet another embodiment, there is provided a computer program for an automated response, the computer program for an automated response including instructions for causing a computer to recognize any one of at least one person depicted in an image, and generate an answer by inputting identification information for identifying the recognized person, the image, and a question related to the recognized person into a generative model pre-trained to generate an answer to the question. [Effects of the Invention]

[0013] The automatic answering device according to the present disclosure has the effect of being able to generate appropriate answers to questions about a specific person depicted in an image. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a schematic diagram of a vehicle equipped with an automatic response device; [Figure 2] FIG. 2 is a diagram illustrating a hardware configuration of an automatic answering device. [Figure 3] FIG. 2 is a functional block diagram of a processor of an automatic answering device. [Figure 4] FIG. 2 is an explanatory diagram of input and output of the answer generation model according to the present embodiment. [Figure 5]10 is a flowchart illustrating the operation of an automatic answering device. DETAILED DESCRIPTION OF THE INVENTION

[0015] The automatic answering device, and the automatic answering method and computer program executed by the automatic answering device will be described below with reference to the drawings. This automatic answering device recognizes at least one person depicted in an image, and generates an answer by inputting identification information for identifying the recognized person, the image, and a question related to the recognized person into a generative model that has been trained in advance to generate an answer to the question.

[0016] The following describes an embodiment in which an automatic answering device is implemented in a vehicle to automatically generate answers to questions related to any of the vehicle's occupants.

[0017] 1 is a schematic diagram of a vehicle equipped with an automatic answering device. In this embodiment, the vehicle 1 has a camera 2, at least one microphone 3, a notification device 4, and an automatic answering device 5. The camera 2, the microphone 3, the notification device 4, and the automatic answering device 5 are connected to each other so that they can communicate with each other.

[0018] Camera 2 is an example of an imaging unit, and is mounted near the top edge of the windshield facing the interior of the vehicle so that the imaging area includes all passengers in vehicle 1. Camera 2 generates an image representing the imaging area at each predetermined imaging cycle and outputs the generated image to automatic response device 5.

[0019] At least one microphone 3 picks up the voice uttered by any of the occupants riding in the vehicle 1 and outputs an audio signal representing the voice. To this end, each microphone 3 is attached inside the cabin of the vehicle 1. Note that multiple microphones 3 may be attached in an array, or a microphone 3 may be attached around each seat in the cabin of the vehicle 1.

[0020] The notification device 4 is provided in the passenger compartment of the vehicle 1 and notifies the occupant of the answer generated by the automatic answering device 5. To this end, the notification device 4 has, for example, at least either a speaker or a display device. When the notification device 4 receives an answer signal representing an answer to the occupant from the automatic answering device 5, it notifies the driver of the answer by voice from the speaker or by displaying a message, image, or video on the display device.

[0021] The automatic response device 5 generates an answer to a question related to any of the occupants of the vehicle 1, and notifies the occupant of the vehicle 1 of the generated answer via the notification device 4.

[0022] Fig. 2 is a hardware configuration diagram of the automatic answering device 5. As shown in Fig. 2, the automatic answering device 5 has a communication interface 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 may each be configured as separate circuits, or may be configured integrally as a single integrated circuit.

[0023] The communication interface 21 has an interface circuit for connecting the automatic answering device 5 to other devices in the vehicle. The communication interface 21 passes the images received from the camera 2 and the audio signals received from the individual microphones 3 to the processor 23. The communication interface 21 also outputs the response signals received from the processor 23 to the notification device 4.

[0024] The memory 22 is an example of a storage unit and includes, for example, a volatile semiconductor memory and a nonvolatile semiconductor memory. The memory 22 stores various data used in the automatic response process executed by the processor 23. Specifically, the memory 22 stores parameters that define a classifier used to identify an occupant depicted in an image and parameters that define a generative model for generating a response. Furthermore, the memory 22 stores, for each of one or more registered persons registered in advance, a feature vector (hereinafter referred to as a registration vector) that represents the characteristics of the registered person and identification information (e.g., the registered person's name, nickname, or identification number). Furthermore, the memory 22 may temporarily store images received from the camera 2 and audio signals received from each microphone 3.

[0025] The processor 23 includes one or more central processing units (CPUs) and their peripheral circuits. The processor 23 may further include other arithmetic circuits such as a logic unit, a numerical calculation unit, or a graphics processing unit. The processor 23 executes the automatic response process.

[0026] 3 is a functional block diagram of processor 23 related to automatic response processing. Processor 23 has an image recognition unit 31, a voice recognition unit 32, a selection unit 33, a response generation unit 34, and a notification processing unit 35. Each of these units in processor 23 is a functional module realized by, for example, a computer program running on processor 23. Alternatively, each of these units in processor 23 may be a dedicated arithmetic circuit provided in processor 23.

[0027] The image recognition unit 31 is an example of a recognition unit, and recognizes each occupant depicted in an image representing a captured area inside the cabin of the vehicle 1, which is generated by the camera 2. The occupants of the vehicle 1 are an example of people to be recognized.

[0028] The image recognition unit 31 detects a person region in an image by inputting the image into a classifier that has been trained in advance to detect a region representing an occupant (hereinafter referred to as a person region). As a result, a person region representing each occupant in the cabin of the vehicle 1 is detected. Such a classifier is configured as a deep neural network (DNN) having a convolutional neural network (CNN)-type architecture such as a Single Shot MultiBox Detector, or a DNN having an attention mechanism such as a Vision Transformer. Alternatively, such a classifier may be configured as a classifier based on a machine learning method other than a DNN, such as an AdaBoost classifier.

[0029] Next, the image recognition unit 31 inputs each detected person region into a feature extractor that has been trained in advance to extract feature vectors representing the characteristics of the occupants depicted in the person region, thereby extracting a feature vector from each person region. Such a feature extractor is configured as a DNN that is trained in advance by so-called unsupervised learning, such as an auto-encoder or stacked what-where auto-encoders. In this case, the feature extractor includes, in order from the input side, an encoder that outputs feature quantities with reduced dimensions compared to the input data (in this embodiment, the person region), and a decoder that receives the feature quantities output from the encoder. The feature extractor is trained in advance using a large number of images depicting various people so that the data input to the encoder and the data output from the decoder are the same. Then, by inputting the person regions into the trained feature extractor, the feature quantities output by the encoder are obtained as feature vectors representing the characteristics of the occupants depicted in the person regions. Furthermore, the feature extractor may be configured as a DNN that is trained using a self-supervised learning method, such as self-supervised learning.

[0030] For each detected person region, the image recognition unit 31 calculates the degree of match (e.g., cosine similarity) between the feature vector extracted from that person region and the registered vector of each registered person who has been registered in advance. The image recognition unit 31 then identifies the registered person with the highest degree of match as the occupant depicted in that person region. Note that if the maximum degree of match is less than a predetermined threshold value, the image recognition unit 31 may determine that the occupant depicted in the person region is not one of the registered persons.

[0031] The image recognition unit 31 further calculates the distance between the center of gravity of each detected person region on the image and a reference position on the image corresponding to the position of each seat in the vehicle cabin, and determines that the occupant depicted in that person region is seated in the seat corresponding to the reference position where the distance is smallest.

[0032] For each detected person region, the image recognition unit 31 outputs identification information of the occupant depicted in the person region and position information indicating the position of the person region on the image (e.g., the center of gravity of the person region) to the selection unit 33 and the answer generation unit 34. Note that for an occupant who is different from any registered person, the image recognition unit 31 outputs data indicating that the occupant is an unregistered person (e.g., text data saying "guest") as the occupant's identification information. The position information may also include information indicating the range of the person region (e.g., the coordinates of the upper left and lower right corners of the person region). Furthermore, the position information may also include a flag indicating the seat position where the occupant depicted in the person region corresponding to the position information is seated.

[0033] The speech recognition unit 32 recognizes a question uttered by one of the occupants based on a voice signal representing a voice in the vehicle cabin collected by the microphone 3. To this end, the speech recognition unit 32 inputs the voice signal into a speech recognition model to recognize the question expressed in the voice signal. Such a speech recognition model may be configured as a DNN with a recursive structure, such as a DNN with an attention mechanism or a recurrent neural network (RNN). Alternatively, the speech recognition model may be configured as a GMM-HMM based on a mixture normal distribution and a hidden Markov model, or a DNN-HMM based on a DNN and a hidden Markov model. The speech recognition model then outputs the question expressed in the input speech signal as text data. Note that the speech recognition unit 32 may recognize the question expressed in the speech signal by dividing the speech signal into frames having a predetermined time length, extracting speech features for each frame, and inputting the features for each frame into the speech recognition model in chronological order. The features for each frame may be, for example, a predetermined element of the cepstrum of that frame.

[0034] The voice recognition unit 32 outputs text data representing a question recognized from the voice signal to the selection unit 33 and the answer generation unit .

[0035] The selection unit 33 selects, depending on the question, whether to input the entire image or the preprocessed image to a generation model (hereinafter referred to as an answer generation model) that has been trained in advance to generate an answer to the question. Details of the answer generation model will be described later together with the answer generation unit 34.

[0036] The selection unit 33 references the text data representing the question received from the voice recognition unit 32 and the identification information of the occupant depicted in the image received from the image recognition unit 31. If the text data representing the question does not include the identification information of the occupant depicted in the image, the selection unit 33 selects the entire image as an input target for the answer generation model. This is because the question does not relate to a specific occupant, and it is highly likely that the state of each occupant depicted in the image will need to be referenced in order to generate an appropriate answer. For example, if the question is "Does everyone look hot?", the entire image is selected as an input target for the answer generation model.

[0037] On the other hand, if the text data representing the question includes identification information of one of the occupants depicted in the image and does not include terms related to the occupant's surroundings, the selection unit 33 selects, as an input target for the answer generation model, an image preprocessed so that areas other than the person area representing the occupant identified by the identification information included in the text data are masked. This prevents image information related to occupants other than the occupant being asked the question from being input to the answer generation model, making it easier to generate a more appropriate answer to the question. For example, if the question is "Is Mr. A sleeping?", an image preprocessed so that areas other than the person area representing occupant A are masked is selected as an input target for the answer generation model. Note that if the text data representing the question includes identification information of multiple occupants depicted in the image, the selection unit 33 selects, as an input target for the answer generation model, an image preprocessed so that areas other than the person area corresponding to each of the multiple identification information are masked.

[0038] Furthermore, if the text data representing a question includes identification information for one of the occupants depicted in the image and includes terms related to the occupant's surroundings, the selection unit 33 selects the entire image and the position information of the person region representing the occupant identified by the identification information included in the text data as input targets for the answer generation model. This enables attention to be paid to the occupant and his / her surroundings, making it easier to provide a more appropriate answer to a question regarding the occupant's behavior toward his / her surroundings. For example, if the question is "Who is Mr. / Ms. A talking to?", the entire image and the position information of the person region representing occupant A identified by the identification information included in the text data of the question are selected as input targets for the answer generation model. Note that terms related to the occupant's surroundings may be registered in advance and stored in the memory 22.

[0039] The selection unit 33 notifies the answer generation unit 34 of the information representing the selected input target to the answer generation model.

[0040] The answer generation unit 34 generates an answer to the question by inputting the image selected as the input target by the selection unit 33, the identification information of the recognized occupant, and a question related to the recognized occupant into an answer generation model.

[0041] In this embodiment, the answer generation model is configured as a VLM. The VLM serving as the answer generation model is configured, for example, as a combination of an image encoder that encodes an input image and a large-scale language model (LLM) in which multiple blocks, each including an attention layer and a feed forward layer, are stacked. The answer generation unit 34 then adds text data representing the recognized occupant's identification information (e.g., the occupant's name) before or after the question, thereby combining the identification information and the question into a single piece of text data and inputting the combined text data into the answer generation model. Furthermore, if the selection unit 33 includes occupant position information as an input target to the answer generation model, the answer generation unit 34 adds coordinates on the image indicated by the position information or text data representing the occupant's seating position (e.g., driver's seat, passenger seat, left rear seat, etc.) before or after the question together with the text data representing the occupant's identification information.

[0042] Furthermore, when a preprocessed image in which all areas other than the person area of ​​a specific occupant are masked is selected as an input target to the answer generation model, the answer generation unit 34 generates a preprocessed image in which all areas other than the person area are masked by cropping only the person area representing the occupant from the image or by replacing the values ​​of each pixel other than the person area with a predetermined pixel value.The answer generation unit 34 then inputs the preprocessed image to the answer generation model.On the other hand, when the entire image is selected as an input target to the answer generation model, the answer generation unit 34 simply inputs the entire image to the answer generation model.

[0043] For example, suppose vehicle 1 has three occupants, A, B, and C. If the question is "Does everyone look hot?", the answer generation model references the text data representing the question, the entire image, and the identification information of each occupant depicted in the image, and generates and outputs text data representing an answer such as "A, B, and C all look hot." If the question is "Is A sleeping?", the answer generation model references the text data representing the question, the person region representing A, and the identification information of each occupant depicted in the image, and generates and outputs text data representing an answer such as "Yes" or "A is sleeping." If the question is "Who is A talking to?", the answer generation model references the text data representing the question, the entire image, the position information of occupant A, and the identification information of each occupant depicted in the image, and generates and outputs text data representing an answer such as "A is talking to B" or "A is talking to the person to his right."

[0044] The question input to the answer generation model may be unrelated to the occupant recognized from the image. In this case, the answer generation model is trained in advance to generate an answer to the question regardless of the input image and the occupant's identification information.

[0045] The answer generating unit 34 outputs text data representing the generated answer to the notification processing unit 35.

[0046] The notification processing unit 35 outputs the answer to the question via the notification device 4. For example, the notification processing unit 35 generates an audio signal representing the answer according to a predetermined voice synthesis method based on text data representing the answer received from the answer generation unit 34. The notification processing unit 35 then outputs the generated audio signal to a speaker included in the notification device 4, thereby causing the speaker to output audio representing the answer. Alternatively, the notification processing unit 35 causes a display device included in the notification device 4 to display the text data representing the answer.

[0047] 4 is an explanatory diagram of input and output of the answer generation model according to this embodiment. In this embodiment, an image 401 (the entire image or a preprocessed image) selected by the selection unit 33 and text data 402 representing the identification information of each occupant depicted in the image and the question are input to the answer generation model 400. As described above, the text data 402 may include location information of the occupant related to the question. The answer generation model 400 then references the input image 401 and text data 402 and outputs text data 403 representing the answer to the question.

[0048] 5 is a flowchart showing the automatic response process according to this embodiment. The processor 23 executes the automatic response process according to this flowchart.

[0049] The image recognition unit 31 recognizes each occupant depicted in the image generated by the camera 2 and identifies the seating position of the recognized occupant (step S101). The voice recognition unit 32 recognizes a question uttered by any occupant based on a voice signal representing a voice inside the vehicle 1 (step S102).

[0050] The selection unit 33 selects an image to input into the answer generation model in response to the question (step S103). The answer generation unit 34 then inputs the selected image, the identification information of each recognized occupant, and a question related to the recognized occupant into the answer generation model to generate an answer to the question (step S104). The notification processing unit 35 then notifies the occupant of the generated answer via the notification device 4 (step S105).

[0051] As described above, this automated answering machine recognizes at least one person depicted in an image, and generates an answer by inputting identification information for identifying the recognized person, the image, and a question related to the recognized person into a model that has been trained to generate an answer to the question. Therefore, this automated answering machine can generate an appropriate answer to a question about a specific person depicted in an image.

[0052] According to a modified example, the location information representing the location of each recognized occupant may be input to the answer generation model together with the image, the identification information of the recognized occupant, and the question, regardless of the selection result by the selection unit 33. This allows the answer generation model to generate an appropriate answer even if the question requests the identification information of an occupant who satisfies a specific condition to be answered.

[0053] According to another modification, the answer generation unit 34 may input the entire image, together with the identification information of each recognized occupant and the question, into the answer generation model, regardless of the content of the question. In this case, it is preferable that the answer generation unit 34 also input the position information of each recognized occupant into the answer generation model. In this modification, the answer generation model only needs to be trained in advance so that it can generate an appropriate answer to a question about any of the occupants depicted in the image even when the entire image is input. Alternatively, the answer generation unit 34 may generate a preprocessed image by masking the image except for the person areas of each recognized occupant, regardless of the content of the question, and input the generated preprocessed image into the answer generation model together with the identification information of each recognized occupant and the question. Note that in this modification, the processing of the selection unit 33 may be omitted.

[0054] According to yet another modification, the answer to the question may be for executing control of the vehicle 1 or control of a device mounted on the vehicle 1. In this case, the answer generation model outputs text data representing the content of the control. The answer generation unit 34 determines the device to be controlled and the control command by referring to a reference table that represents the correspondence between the text data representing the content of the control, the device to be controlled (including the vehicle 1 itself), and the control command for executing the control. The answer generation unit 34 then outputs the identified control command to the control unit of the device to be controlled via the communication interface 21.

[0055] Furthermore, this automatic answering device is not limited to an in-vehicle embodiment, but can be used in various systems that are capable of capturing images of multiple people and that require generating answers to questions about any of the people. For example, this automatic answering device may be installed in a predetermined space in a facility and generate answers to any of the questions of one or more people located in the space. Questions may also be input via a user interface that allows input of text data, such as a keyboard or a touch panel. In this case, the processing of the voice recognition unit 32 may be omitted.

[0056] A computer program for realizing the automatic response process according to the above embodiment or modification may be provided in a form recorded on a computer-readable portable recording medium.

[0057] As described above, those skilled in the art can make various modifications to the embodiments within the scope of the present invention. [Explanation of symbols]

[0058] REFERENCE SIGNS LIST 1 vehicle, 2 camera, 3 microphone, 4 notification device, 5 automatic answering device, 21 communication interface, 22 memory, 23 processor, 31 image recognition unit, 32 voice recognition unit, 33 selection unit, 34 answer generation unit, 35 notification processing unit

Claims

1. a recognition unit that recognizes at least one person depicted in the image; an answer generation unit that generates an answer by inputting identification information for identifying the recognized person, the image, and a question related to the recognized person into a generative model that has been trained in advance to generate an answer to the question; An automatic answering device having:

2. The automatic answering device according to claim 1 , wherein the answer generating unit further inputs, together with the question, position information representing a position of a person area in which the person recognized on the image is depicted, into the generative model.

3. The automatic answering device described in claim 1 or 2, wherein the answer generation unit preprocesses the image so that areas other than a person area representing a person recognized in the image are masked, and the preprocessed image is used as an image to be input to the generation model.

4. Further, a selection unit is provided that selects, in response to the question, whether the entire image or a preprocessed image obtained by preprocessing the image is to be input to the generative model; The automatic answering device of claim 1 , wherein the answer generator inputs a selected one of the entire image or the preprocessed image into the generative model.

5. the selection unit further selects, in response to the question, whether or not to input position information representing a position of a person region in which the person recognized in the image is depicted into the generative model; The automatic answering device according to claim 4 , wherein the answer generating unit further inputs the location information to the generation model when inputting the location information to the generation model is selected.

6. Recognizing any of at least one person depicted in the image; generating an answer by inputting identification information for identifying the recognized person, the image, and a question related to the recognized person into a generative model that has been trained in advance to generate an answer to the question; An automatic response method including:

7. Recognizing any of at least one person depicted in the image; generating an answer by inputting identification information for identifying the recognized person, the image, and a question related to the recognized person into a generative model that has been trained in advance to generate an answer to the question; An automatic response computer program that causes a computer to do the following:

Images