Automatic response device, automatic response method, and computer program product

Abstract

The invention relates to an automatic response apparatus, an automatic response method, and a computer program product. An automatic response device (5) is provided with: a position estimation unit (31) that estimates the position of an occupant who speaks out of a plurality of occupants in a vehicle (1); and a response generation unit (33) that generates response information by inputting position information indicating the position of the occupant who speaks and voice information indicating the content of the speaking into a generation model that has been learned in advance so as to generate response information for the content of the speaking.

Description

Technical Field

[0001] This invention relates to an automatic response device, an automatic response method, and a computer program product for automatically responding to statements made by occupants of a vehicle. Background Technology

[0002] A method is proposed to identify the speaking location and content based on the voice of a vehicle passenger, and to select and output a portion of multiple pieces of information related to the facility specified by the gesture of the person speaking at the speaking location detected by the image inside the carriage (see Japanese Patent Application Publication No. 2017-90615).

[0003] Depending on the location of the person speaking, the content of the requested response may vary. Summary of the Invention

[0004] Therefore, the object of the present invention is to provide an automatic response device that can appropriately respond to the speech of one of the multiple passengers in a vehicle.

[0005] According to one embodiment, an automatic response device is provided. This automatic response device includes: a position estimation unit for estimating the position of a speaking passenger among multiple passengers in a vehicle; and a response generation unit for generating response information by inputting position information representing the speaking passenger's position and voice information representing the content of the speech into a generative model that has been pre-learned to generate response information in response to the content of the speech.

[0006] In one embodiment, the automatic response device further includes a control unit that controls the device located at the position of the speaking passenger based on the response information.

[0007] In one embodiment, the automatic response device further includes a determining unit that determines a portion of an interior image representing the vehicle's interior that corresponds to the position of the occupant as indicated by location information. Then, a response generation unit generates response information by further inputting the portion of the image into a generation model.

[0008] According to another embodiment, an automatic response method is provided. The automatic response method includes: estimating the position of a speaking passenger among multiple passengers in a vehicle; and generating response information by inputting location information representing the speaking passenger's position and voice information representing the content of the speech into a generative model pre-learned in a manner that generates response information in response to the content of the speech.

[0009] According to yet another embodiment, an automatic response computer program product is provided. This automatic response computer program product includes instructions that cause a computer to perform the following actions: estimating the position of a speaking passenger among multiple passengers in a vehicle; and generating response information by inputting positional information representing the speaking passenger's position and voice information representing the content of the speech into a generative model pre-learned in a manner that generates response information in response to the content of the speech.

[0010] The automatic response device disclosed herein has the effect of being able to respond appropriately to the speech of one of the multiple passengers in a vehicle. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of a vehicle equipped with an automatic response system.

[0012] Figure 2 This is a hardware configuration diagram of an automatic response device.

[0013] Figure 3 This is a functional block diagram of the processor in an automatic response device.

[0014] Figure 4 This is an explanatory diagram regarding the relationship between input and response information of the generative model according to this embodiment.

[0015] Figure 5 This is a flowchart of the automatic response device's operation. Detailed Implementation

[0016] The automatic response device, the automatic response method executed by the automatic response device, and the computer program for automatic response will be described below with reference to the accompanying drawings. The automatic response device estimates the position of the passenger who is speaking among multiple passengers in a vehicle. Then, the automatic response device generates response information by inputting position information indicating the position of the speaking passenger and voice information indicating the content of the speech into a generative model that has been pre-learned to generate response information in response to the content of the speech.

[0017] Figure 1 This is a schematic diagram of a vehicle equipped with an automatic response device. In this embodiment, vehicle 1 includes a camera 2, multiple microphones 3-1 to 3-n (n is an integer greater than or equal to 2, and 4 in the illustrated example), a notification device 4, and an automatic response device 5. The camera 2, microphones 3-1 to 3-n, and notification device 4 are connected to the automatic response device 5 in a manner that enables them to communicate with each other.

[0018] Camera 2 is an example of an in-vehicle imaging unit. It is mounted near the upper part of the windshield, facing the interior of the vehicle, so that all passengers in vehicle 1 are included in the camera's imaging area. Camera 2 then generates an image representing the interior of vehicle 1 at each predetermined shooting cycle and outputs the generated image to the automatic response device 5. Hereinafter, the image generated by camera 2 will be referred to as an in-vehicle image.

[0019] Microphones 3-1 to 3-n pick up the speech of any passenger in vehicle 1 and output a speech signal representing that speech. Therefore, one of microphones 3-1 to 3-n is installed at each seat in vehicle 1, in a position that allows the speech of the passenger seated there to be picked up. Figure 1 In the example shown, microphones are mounted near each of the driver's seat, front passenger seat, right rear seat, and left rear seat (e.g., on the dashboard for the driver and front passenger seats, and on the back of the seat in front of the rear seats). It should be noted that microphones 3-1 to 3-n can also be mounted in an array at locations where the voice emitted by any passenger in the vehicle can be collected (e.g., near the front ceiling or dashboard). Microphones 3-1 to 3-n output the generated voice signals to the automatic response device 5.

[0020] The notification device 4 is located inside the passenger compartment of vehicle 1 and notifies passengers of the response content represented by the response information generated by the automatic response device 5. Therefore, the notification device 4 may include at least one of a speaker or a display device. Then, when a notification signal representing the response content to the passenger is received from the automatic response device 5, the notification device 4 notifies the driver of the response content via voice from the speaker or by displaying a message, image, or moving image via the display device. It should be noted that the display device or speaker of the notification device 4 may also be configured for each seat facing the passenger seated there. In this case, the display device located at the seat where the speaking passenger is seated may display the response content, or the speaker located at the seat where the speaking passenger is seated may output voice representing the response content.

[0021] Automatic response device 5 generates response information to the speech content made by any of the passengers of vehicle 1, and notifies the passenger of vehicle 1 of the generated response information via notification device 4, or controls any device of vehicle 1 according to the response information.

[0022] Figure 2 This is a hardware configuration diagram of the automatic response device 5. (For example...) Figure 2As shown, the automatic response device 5 has a communication interface 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 can be configured as separate circuits, or they can be integrated into a single integrated circuit.

[0023] The communication interface 21 has interface circuitry for connecting the automatic response device 5 to other devices within the vehicle. The communication interface 21 transmits in-vehicle images received from the camera 2 and voice signals received from each of the microphones 3-1 to 3-n to the processor 23. Furthermore, the communication interface 21 outputs notification signals received from the processor 23 to the notification device 4, or outputs control commands received from the processor 23 to any of the in-vehicle devices.

[0024] Memory 22 is an example of a storage unit, such as having volatile semiconductor memory and non-volatile semiconductor memory. Furthermore, memory 22 stores various data used in the automatic response processing executed by processor 23. Specifically, memory 22 stores parameters that define the generation model used to generate response information. Moreover, memory 22 can also temporarily store in-vehicle images received from camera 2 and voice signals received from each of microphones 3-1 to 3-n.

[0025] Processor 23 has one or more CPUs (Central Processing Units) and their peripheral circuitry. Processor 23 may also have other arithmetic circuitry such as logic units, numerical processing units, or graphics processing units. Furthermore, processor 23 performs automatic response processing.

[0026] Figure 3 This is a functional block diagram of the processor 23 related to automatic response processing. The processor 23 includes a position estimation unit 31, a determination unit 32, a response generation unit 33, a notification processing unit 34, and a control unit 35. These units of the processor 23 are, for example, functional modules implemented by a computer program that operates on the processor 23. Alternatively, these units of the processor 23 may also be dedicated arithmetic circuits provided on the processor 23.

[0027] The position estimation unit 31 estimates the position of the passenger who spoke among multiple passengers. In this embodiment, where a microphone is provided for each seat, if the average volume of any of the speech signals generated by each of the microphones 3-1 to 3-n exceeds a speech detection threshold within a recent specified period (e.g., a few seconds), the position estimation unit 31 estimates that any passenger spoke during that specified period. Then, the position estimation unit 31 estimates the position of the passenger who spoke based on the location of the seat where the microphone with the highest average volume during that specified period is located.

[0028] Furthermore, in the embodiment where microphones 3-1 to 3-n are arranged in an array, if the average volume of any one of the speech signals generated by each microphone exceeds a speech detection threshold during a recent specified period, the position estimation unit 31 estimates that any passenger spoke during that specified period. The position estimation unit 31 calculates the phase difference between the speech signals from each microphone during that specified period and estimates the direction of speech arrival based on the calculated phase difference. Then, the position estimation unit 31 estimates the position of the passenger who spoke, based on the position of the seat closest to the direction of speech arrival observed from the assembly position of microphones 3-1 to 3-n. In this case, the direction from the assembly position of each microphone to each seat can be pre-stored in the memory 22. Then, the position estimation unit 31 determines the seat closest to the direction of speech arrival among the directions to each seat and estimates the position of the determined seat as the position of the passenger who spoke.

[0029] Alternatively, the location estimation unit 31 can also estimate the location of the speaking passenger based on in-vehicle images. In this case, the location estimation unit 31 inputs the multiple in-vehicle images into a classifier that has been pre-learned to estimate the location of the speaking passenger, in the order in which the multiple in-vehicle images were generated. Then, if any passenger speaks during the most recent specified period, the location estimation classifier outputs the location of the speaking passenger. It should be noted that the location estimation classifier is configured as a deep neural network (DNN) with a recurrent structure, such as a recurrent neural network (RNN) or a long short-term memory (LSTM) network.

[0030] The position estimation unit 31 generates position information representing the estimated position of the speaking passenger and outputs the generated position information to the determination unit 32 and the response generation unit 33. It should be noted that the position information includes a string representing the position of the speaking passenger (e.g., a string indicating the passenger's seat, such as driver's seat or front passenger seat) or a vector representing the speaking passenger's position. When the speaking passenger's position is represented by a vector, for example, the vector is generated in such a way that each seat contains different elements, and the value of the element corresponding to the speaking passenger's seat is different from the values ​​of the elements corresponding to other seats.

[0031] The determination unit 32 determines a portion of the in-vehicle image corresponding to the position of the speaking passenger as indicated by the position information. For example, the position and range on the in-vehicle image, set for each seat in a manner that includes the passenger seated in that seat, are pre-stored in the memory 22. Then, the determination unit 32 reads the position and range of the seat corresponding to the position of the speaking passenger as indicated by the position information from the memory 22, and determines the area specified by the read position and range as the portion of the in-vehicle image corresponding to the position of the speaking passenger.

[0032] Alternatively, the determination unit 32 can also detect the regions in the in-vehicle image that represent each passenger by inputting an in-vehicle image into a classifier that has been pre-learned in a manner for detecting passengers. Then, the determination unit 32 determines the region in the region representing each passenger that corresponds to the position of the passenger speaking, as indicated by the location information, as the partial region corresponding to the position of the speaking passenger. In this case, the position of the reference point corresponding to each seat in the in-vehicle image is pre-stored in the memory 22 for each seat. Then, the determination unit 32 determines the region in the region representing each passenger that is closest to the reference point for the seat corresponding to the position of the speaking passenger, as the partial region corresponding to the position of the speaking passenger. It should be noted that the classifier used for passenger detection is configured as a DNN with a convolutional neural network (CNN) architecture, such as a single-shot multibox detector, or a DNN with an attention mechanism, such as a vision transformer. Alternatively, the classifier used for occupant detection can be configured as a classifier based on machine learning methods other than DNNs, such as the AdaBoost classifier.

[0033] The determination unit 32 notifies the response generation unit 33 of the location and extent of a portion of the area where the speaker is presenting the passenger.

[0034] The response generation unit 33 generates response information by inputting location information representing the position of the speaking passenger, speech information representing the content of the speech, and a portion of the speaking passenger's area into a generative model that has been pre-learned in a way that generates response information for the content of the speech.

[0035] To generate speech information representing the content of a speech, the response generation unit 33 inputs a speech signal whose average volume over the most recent predetermined period exceeds a speech detection threshold from each of the speech signals generated by microphones 3-1 to 3-n into a speech recognition model. This model then identifies the content of the speech represented by the speech signal and generates a string representing the speech content as speech information. Such a speech recognition model is configured, for example, as a DNN with an attention mechanism, or a recursive structure such as an RNN or LSTM. Alternatively, the speech recognition model can be configured as a GMM-HMM based on a mixture of normal distributions and hidden Markov models, or a DNN-HMM based on a DNN and a hidden Markov model. It should be noted that the response generation unit 33 segments the speech signal in frames of a predetermined time length, extracts speech features for each frame, and inputs the features of each frame into the speech recognition model in chronological order, thereby recognizing the content of the speech represented by the speech signal. Furthermore, the features of each frame can be, for example, defined elements of the cepstral spectrum of that frame.

[0036] The response generation unit 33 appends a string indicating the position of the passenger indicated by the location information before or after the string indicating the content of the speech represented by the voice information, thereby using a single string to represent the combination of the content of the speech and the position of the passenger.

[0037] Furthermore, the response generation unit 33 crops a portion of the in-vehicle image, defined by the location and extent notified by the determination unit 32, to input the portion of the speaking occupant into the generation model. Alternatively, the response generation unit 33 may also mask out areas outside the designated portion of the in-vehicle image by rewriting the values ​​of pixels outside the designated portion as specified by the determination unit 32 to predetermined values.

[0038] In this embodiment, the generative model is configured as a Vision Language Model (VLM). The VLM, serving as the generative model, is configured, for example, to combine an image encoder that encodes the input image with a Large Language Model (LLM), where the LLM is composed of multiple stacked blocks including attention layers and feed-forward layers. The response generation unit 33 inputs a cropped portion of the in-vehicle image, or an image of the in-vehicle interior that is partially masked, to the image encoder. Furthermore, the response generation unit 33 inputs strings representing the content of the speech and the location of the speaking passenger to the LLM. Thus, the generative model outputs text data representing the response content as response information. In this way, by inputting location information along with speech information representing the speech content to the generative model, the generative model can generate response information representing response content corresponding to the location of the speaking passenger. Moreover, not only speech information and location information are used in the generation of response information, but also a portion of the in-vehicle image showing the speaking passenger is used in the generation of response information; thus, the generative model can use the in-vehicle image to determine the state of the speaking passenger. Therefore, the generative model can generate response information that is more appropriate for the passenger speaking. For example, even if the passenger requests that the car window or door be opened, if the passenger is a young child, the generative model will not unlock the window or door, but will generate a reassuring response such as "Wait a moment." Furthermore, when generating response information to display an image on a display device positioned at the passenger's location, the generative model can consider the passenger's state to determine the content of the displayed image.

[0039] For example, if the person speaking is seated in the rear right seat, a portion of the area representing that seat is in the rear right seat is input into the generative model. Furthermore, if the statement is assumed to be "it's hot," a string like "the rear right seat is hot" is input into the generative model. The model then outputs text data such as "enhance the air conditioning in the rear right seat" as a response.

[0040] It should be noted that the response information may also include information for controlling the vehicle 1 itself or the equipment mounted on the vehicle 1.

[0041] According to a variation, an input layer for location information input, representing the position of the speaking rider, can be set up separately from the image encoder and LLM in the generative model. In this case, only the string corresponding to the speech information is input to the LLM, and the vector representing the position of the speaking rider is input to the location information input layer. Then, the vector representing the position of the speaking rider input to this input layer is similarly taken into the LLM in a block of multiple blocks, which has a cross-attention mechanism that calculates cross-attention between the vector and the output from the previous block, just as it is input to a portion of the image encoder.

[0042] The response generation unit 33 outputs the generated response information along with the location information to the notification processing unit 34 and the control unit 35.

[0043] The notification processing unit 34 outputs response information via the notification device 4. Therefore, the notification processing unit 34 generates a notification signal representing the response content included in the response information and outputs the generated notification signal to the notification device 4 via the communication interface 21. For example, the notification processing unit 34 generates a speech signal representing the response content based on the text data representing the response content included in the response information, using a prescribed speech synthesis method, as the notification signal. Then, the notification processing unit 34 outputs the speech representing the response content from the speaker of the notification device 4 by outputting the notification signal. Alternatively, the notification processing unit 34 includes the text data representing the response content in the notification signal. Then, the notification processing unit 34 causes the display device of the notification device 4 to display the text data representing the response content.

[0044] Furthermore, when the notification device 4 has a display device or speaker installed for each seat, the notification processing unit 34 outputs a notification signal via the communication interface 21 to the display device or speaker located at the seat where the speaker is sitting, as indicated by the location information. Then, the notification processing unit 34 causes the display device at the seat where the speaker is sitting to display text data indicating the response content, or causes the speaker at that seat to output voice indicating the response content. Thus, even if the response content is only addressed to the speaker, the notification processing unit 34 can provide an appropriate response.

[0045] It should be noted that when the response information generated by the generation model represents the control content of the device of vehicle 1, the processing performed by the notification processing unit 34 can be omitted.

[0046] The control unit 35 controls the device specified by the response content contained in the response information according to the response content. The control unit 35 determines the device to be controlled and the control command by referring to the correspondence between the text data representing the response content contained in the response information and the device (including the vehicle 1 itself) and the control command used to execute the control. Then, the control unit 35 outputs the determined control command to the electronic control unit (ECU) of the device to be controlled via the communication interface 21.

[0047] It should be noted that the device to be controlled can also be a device located at the position of the passenger speaking. In this case, the control unit 35 determines the position of the passenger speaking by referring to position information, and determines the device to be controlled based on text data representing the response content, as described above. Then, the control unit 35 outputs a signal indicating the position of the controlled device and a control command to the ECU of the controlled device, thereby controlling the device located at the position of the passenger speaking according to the response information.

[0048] For example, assuming the passenger making the comment is seated in the rear right seat, in response to the comment "It's very hot," a response message is generated that increases the airflow to the rear right seat by a predetermined amount. In this case, the control unit 35 outputs a control command to the ECU that controls the air conditioning system, increasing the airflow from the air vent closest to the rear right seat by the predetermined amount.

[0049] It should be noted that, in addition to the air conditioning unit, the device that controls the vehicle based on the position of the passenger speaking may also include any one of the following: windows, door locks, interior lights, and seats. The control unit 35 performs actions such as opening and closing the window closest to the passenger speaking, locking / unlocking the door closest to the passenger speaking, turning the interior light closest to the passenger speaking on or off, or adjusting the seat position of the passenger speaking. This is done as control of the device according to the response content.

[0050] Furthermore, when the response content indicated by the response information is related to the driving control of vehicle 1, the control unit 35 can output a control command indicating the response content to the ECU that controls the driving of vehicle 1 only when the position of the passenger speaking is the driver's seat, that is, only when the passenger speaking is the driver.

[0051] For example, suppose the content of the statement is "faster". In this case, only if the position of the passenger making the statement, as indicated by the location information, is the driver's seat, will the control unit 35 output a control command to the ECU that controls the movement of vehicle 1, increasing the target speed of vehicle 1 by a predetermined amount. On the other hand, if the position of the passenger making the statement, as indicated by the location information, is a seat other than the driver's seat, such as the front passenger seat, the control unit 35 will not output a control command corresponding to the response content. This prevents accidental control of vehicle 1 by a passenger other than the driver, and consequently prevents the behavior of vehicle 1 from becoming unstable. It should be noted that the control unit 35 determines whether the device controlled, as indicated by the text data representing the response content, is the ECU that controls the movement of vehicle 1 by referring to a control reference table.

[0052] Furthermore, if the text data representing the response content does not contain any of the words identifying the controlled device registered in the control reference table, the response information does not control the device. Therefore, in this case, the control unit 35 does not output a control signal.

[0053] Figure 4 This is an explanatory diagram regarding the relationship between input and response information of the generative model according to this embodiment. In this embodiment, text data 401 containing the statement content ("It's hot") and the location information of the occupant making the statement ("Rear right seat"), and a partial region 402 extracted from an in-vehicle image representing the occupant of the rear right seat, are input to the generative model 400. Then, the generative model 400 outputs text data 403 representing the response content to the statement content ("Enhance the air conditioning in the rear right seat"), referring to the text data 401 and the partial region 402.

[0054] Figure 5 This is a flowchart of the automatic response processing according to this embodiment. The processor 23 performs the automatic response processing according to this flowchart.

[0055] The position estimation unit 31 estimates the position of the passenger who is speaking and generates position information representing that position (step S101). The determination unit 32 determines a portion of the in-vehicle image that corresponds to the position of the passenger who is speaking (step S102).

[0056] The response generation unit 33 generates speech information representing the content of the speech given by the passenger based on the speech signal generated by any one of the microphones 3-1 to 3-n (step S103). Then, the response generation unit 33 generates response information by inputting the speech information, location information, and a partial region corresponding to the location of the passenger who is speaking into the generation model (step S104).

[0057] The notification processing unit 34 notifies all passengers or the passenger who spoke of the response content contained in the response information via the notification device 4 (step S105). In addition, the control unit 35 controls the vehicle 1 itself or the equipment mounted on the vehicle 1, especially the equipment located at the position of the passenger who spoke, according to the response content (step S106).

[0058] As explained above, the automatic response device estimates the location of the speaking passenger among the multiple passengers in the vehicle. Then, the device generates response information by inputting location information indicating the speaking passenger's location and voice information indicating the content of the speech into a generative model that has been pre-learned to generate response information in response to the speech content. Therefore, the automatic response device can respond appropriately to the speaking passenger among the multiple passengers in the vehicle.

[0059] According to a variation, the response generation unit 33 may not input the portion of the speaking passenger's area into the generation model. Instead, it may input location information indicating the speaking passenger's position and speech information indicating the content of the speech into the generation model to generate response information. In this case, an LLM (Local Level Model) is used as the generation model. Furthermore, the processing of the determination unit 32 can be omitted. In this variation, location information is also input into the generation model, thus the automatic response device can respond appropriately to the speaking passenger. Furthermore, according to this variation, processing related to the in-vehicle image is omitted, thus reducing the computational load compared to the embodiment described above.

[0060] In the above-described embodiments or variations, a server capable of communicating via an in-vehicle wireless communication terminal (not shown) may have a generation model, generating response information by executing the processing of the response generation unit 33. In this case, the automatic response device 5 generates a question signal containing text data indicating the location and content of the speaking passenger, as well as a partial area representing the speaking passenger cropped from an in-vehicle image, and sends the generated question signal to the server via the wireless communication terminal. Then, the server generates response information by inputting the text data and partial area contained in the received question signal into the generation model, and sends the generated response information to the wireless communication terminal of the vehicle 1 that sent the question signal. It should be noted that if the partial area is not input into the generation model, the automatic response device 5 may also omit the partial area from the question signal. The automatic response device 5 executes the processing of the notification processing unit 34 and the control unit 35 according to the response information received via the wireless communication terminal. According to this variation, the automatic response device 5 can utilize a generation model larger than that implemented using in-vehicle hardware resources for generating response information, thus generating more appropriate response information.

[0061] According to another variation, the processing of either the notification processing unit 34 or the control unit 35 may also be omitted.

[0062] According to another variation, the speech signal, which is presumed to contain speech content, can be directly input into the generative model along with location information, and the generative model can be pre-learned in a way that it can identify the speech content from the speech signal and generate response information corresponding to the speech content and the location of the speaker.

[0063] The computer program that implements the automatic response processing according to the above-described embodiments or variations can be provided as a computer program product, for example, in the form of a computer-readable removable recording medium.

[0064] As described above, those skilled in the art can make various modifications within the scope of this invention depending on the implementation method.

Claims

1. An automatic response device, comprising: The position estimation unit estimates the position of the passenger who speaks among the multiple passengers in the vehicle; and The response generation unit generates response information by inputting location information representing the position of the passenger making the speech and speech information representing the content of the speech into a generative model that has been pre-learned in a way that generates response information for the content of the speech.

2. The automatic response device according to claim 1, wherein, The automatic response device also includes a control unit that controls the device located at the position of the speaking passenger based on the response information.

3. The automatic response device according to claim 1 or 2, wherein, The automatic response device further includes a determining unit that determines a portion of an interior image representing the interior of the vehicle that corresponds to the position of the occupant as indicated by the location information. The response generation unit generates the response information by further inputting the partial region into the generation model.

4. An automatic response method, comprising: The position of the passenger who speaks among the multiple passengers in the vehicle is estimated; as well as The response information is generated by inputting location information representing the position of the passenger making the speech and speech information representing the content of the speech into a generative model that has been pre-learned to generate response information in response to the content of the speech.

5. A computer program product for automatic response, comprising instructions for causing a computer to perform the following actions: To estimate the position of the passenger who speaks among the multiple passengers in the vehicle; and The response information is generated by inputting location information representing the position of the passenger making the speech and speech information representing the content of the speech into a generative model that has been pre-learned to generate response information in response to the content of the speech.

Images