Automatic response system, automatic response device, automatic response method, and program product

Abstract

The invention relates to an automatic response system, an automatic response device, an automatic response method, and a program product. An automatic response device mounted in a vehicle, which is provided in an automatic response system, generates first response information by inputting input information including voice information indicating the content of an utterance from an occupant of the vehicle into a first generation model that has been learned in advance so as to generate the first response information. And generates inquiry information indicating the content of the utterance on the basis of the input information, and responds to the occupant on the basis of at least one of second response information and the first response information generated on the basis of the inquiry information in a server provided outside the vehicle. Further, the server generates second response information by inputting inquiry information received from the automatic response device to a second generative model, which is a model that has been learned in advance in such a manner as to generate the second response information and that is larger than the first generative model.

Description

Technical Field

[0001] This invention relates to an automatic response system, automatic response device, automatic response method, and computer program product for automatically responding to statements made by occupants of a vehicle. Background Technology

[0002] A method for providing information in accordance with the preferences of vehicle passengers is proposed (see Japanese Patent Application Publication No. 2023-124286). This method identifies passenger actions based on vehicle location information and / or voice recordings made by the passenger, and stores action records as records related to the passenger's actions. Then, the method infers the passenger's preferences based on voice regarding the information to be provided to the passenger, and adjusts the content of the information to be provided to the passenger based on the inferred preferences when providing information related to the action records.

[0003] In order to satisfy passengers' satisfaction with their response to statements made by passengers in the vehicle, the response should include appropriate content that corresponds to the statements made. Summary of the Invention

[0004] Therefore, the object of the present invention is to provide an automatic response system that can appropriately respond to statements made by passengers in a vehicle.

[0005] According to one embodiment, an automatic response system is provided. The automatic response system includes an automatic response device mounted in a vehicle and a server located outside the vehicle. The automatic response device includes: a first response generation unit that generates first response information by inputting input information containing voice information representing the content of a speech from a occupant of the vehicle into a first generation model, and generates query information representing the content of the speech based on the input information, wherein the first generation model is mounted in the vehicle and has been pre-learned in a manner that generates first response information for the content of the speech; a transceiver processing unit that sends the query information to the server via a communication device mounted in the vehicle, and receives second response information generated based on the query information from the server; and a response processing unit that responds to the occupant based on at least one of the first response information and the second response information. Furthermore, the server includes a second response generation unit that generates second response information by inputting the query information into a second generation model, wherein the second generation model is pre-learned in a manner that generates the second response information and is a model on a larger scale than the first generation model.

[0006] In one implementation, the first generative model is pre-learned to generate first response information and query information based on the input information. Then, the first response generation unit generates query information by inputting the input information into the first generative model.

[0007] In one implementation, the response processing unit responds to the passenger based on the generated first response information, and then further responds to the passenger based on the second response information received from the server.

[0008] In one embodiment, the response processing unit generates third response information by inputting second response information into a first generation model, and responds to the passenger based on the generated third response information.

[0009] In one embodiment, during the waiting period from responding to the passenger based on the first response information until receiving the second response information, the response processing unit notifies the passenger of the predetermined waiting response via a notification device mounted on the vehicle.

[0010] According to another embodiment, an automatic response method is provided. The automatic response method includes: generating first response information by inputting input information containing voice information representing the content of a speech given by a passenger in a vehicle to a first generative model, wherein the first generative model is mounted in the vehicle and has been pre-learned in a manner that generates first response information in response to the speech content; generating query information representing the speech content based on the input information; generating second response information by inputting the query information to a second generative model, wherein the second generative model is pre-learned in a manner that generates second response information in response to the query information, is larger in scale than the first generative model, and is located outside the vehicle; and responding to the passenger based on at least one of the first and second response information.

[0011] According to yet another embodiment, an automatic response computer program product is provided. This automatic response computer program product includes instructions for causing a computer to perform the following actions: generating first response information by inputting input information containing voice information representing the content of a speech from a occupant of a vehicle into a first generation model, wherein the first generation model is mounted in the vehicle and has been pre-learned in a manner that generates first response information in response to the content of the speech; generating query information representing the content of the speech based on the input information; generating second response information by inputting the query information into a second generation model, wherein the second generation model is pre-learned in a manner that generates second response information in response to the query information, is larger in scale than the first generation model, and is located outside the vehicle; and responding to the occupant based on at least one of the first and second response information.

[0012] According to another embodiment, an automatic response device is provided. This automatic response device includes: a first response generation unit that generates first response information by inputting input information containing voice information representing the content of a speech given by a occupant of a vehicle into a first generation model, and generates query information representing the content of the speech based on the input information, wherein the first generation model is a model mounted in the vehicle and pre-learned in a manner that generates first response information for the content of the speech; a transceiver processing unit that sends the query information to a server located outside the vehicle via a communication device mounted in the vehicle, and receives second response information generated based on the query information from the server; and a response processing unit that responds to the occupant based on at least one of the first response information and the second response information.

[0013] The automatic response system disclosed herein is designed to respond appropriately to statements made by passengers in the vehicle. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the automatic response system.

[0015] Figure 2 This is a schematic diagram of a vehicle equipped with an automatic response system.

[0016] Figure 3 This is a hardware configuration diagram of an automatic response device.

[0017] Figure 4 This is a functional block diagram of the processor in an automatic response device.

[0018] Figure 5 This is a diagram of the server's hardware configuration.

[0019] Figure 6 This is a functional block diagram of the server's processor.

[0020] Figure 7 (A) is an illustration of automatic response processing.

[0021] Figure 7 (B) is an illustration of automatic response processing.

[0022] Figure 8 This is a flowchart of the automatic response processing action. Detailed Implementation

[0023] The automatic response system, the automatic response method executed in the automatic response system, the computer program for automatic response, and the automatic response device included in the automatic response system will be described below with reference to the accompanying drawings. The automatic response system includes an automatic response device mounted in a vehicle and a server located outside the vehicle. The automatic response device generates first response information in a first generation model on-board in response to a statement made by any passenger in the vehicle, and sends query information corresponding to the content of the statement to the server. The server then generates second response information in response to the query information by inputting the received query information into a second generation model that is larger than the first generation model, and sends the generated second response information to the vehicle. When the second response information is received from the server, the automatic response device responds to the passenger based on at least one of the first and second response information.

[0024] Figure 1 This is a schematic diagram of the automatic response system. In this embodiment, the automatic response system 1 includes a server 4 and an automatic response device 3 mounted on the vehicle 2. The automatic response device 3 accesses, for example, a wireless base station 6 connected to a communication network 5 connected to the server 4 via a gateway (not shown), thereby connecting to the server 4 via the wireless base station 6 and the communication network 5. It should be noted that in Figure 1 The diagram only shows one vehicle 2 and one automatic response device 3, but the automatic response system 1 can also have multiple vehicles 2 equipped with automatic response devices 3. Similarly, multiple wireless base stations 6 can be connected to the communication network 5.

[0025] First, the vehicle 2 and the automatic response device 3 will be described. As mentioned above, the automatic response system 1 may include multiple vehicles 2 equipped with automatic response devices 3. However, regarding the automatic response processing, each vehicle 2 and each automatic response device 3 has the same configuration and performs the same processing. Therefore, one vehicle 2 and one automatic response device 3 will be described below.

[0026] Figure 2This is a schematic diagram of a vehicle 2 equipped with an automatic response device 3. The vehicle 2 includes a camera 11, at least one microphone 12, a wireless communication terminal 13, a notification device 14, and the automatic response device 3. The camera 11, microphone 12, wireless communication terminal 13, and notification device 14 are connected to the automatic response device 3 in a manner that enables them to communicate with each other.

[0027] Camera 11 is an example of an in-vehicle sensor, mounted near the upper end of the windshield facing the interior of the vehicle compartment, such that all occupants of vehicle 2 are included in the camera's field of view. Camera 11 then generates an image representing the interior of vehicle 2 at each predetermined shooting cycle and outputs the generated image to the automatic response device 3. Hereinafter, the image generated by camera 11 will be referred to as an in-vehicle image. Furthermore, the in-vehicle image is an example of an in-vehicle sensor signal.

[0028] At least one microphone 12 is another example of an in-vehicle sensor, which picks up the voice of any of the occupants of the vehicle 2 and outputs a voice signal representing that voice. Therefore, each microphone 12 is mounted inside the passenger compartment of the vehicle 2. It should be noted that multiple microphones 12 can be mounted in an array, or microphones 12 can be mounted around each seat in the passenger compartment of the vehicle 2. Each microphone 12 outputs the generated voice signal to the automatic response device 3. The voice signal generated by each microphone 12 is another example of an in-vehicle sensor signal.

[0029] Wireless communication terminal 13 is an example of a communication device, a device that performs wireless communication processing conforming to a prescribed wireless communication standard, for example, connecting to server 4 via wireless base station 6 and communication network 5 by accessing wireless base station 6. That is, a communication line is established between wireless communication terminal 13 and server 4 via wireless base station 6 and communication network 5. Then, wireless communication terminal 13 generates an uplink wireless signal containing query information received from automatic response device 3 and transmits the wireless signal to wireless base station 6. Thus, query information is sent to server 4. In addition, wireless communication terminal 13 receives a downlink wireless signal containing second response information from wireless base station 6 and outputs the second response information to automatic response device 3. Thus, the second response information generated by server 4 is transmitted to automatic response device 3.

[0030] The notification device 14 is located inside the passenger compartment of vehicle 2 and notifies passengers of the response content represented by the response information generated by the automatic response device 3 or the server 4. Therefore, the notification device 14 may include at least one of a speaker or a display device. Then, when a notification signal representing the response content to the passenger is received from the automatic response device 3, the notification device 14 notifies the driver of the response content via voice from the speaker or by displaying a message, image, or moving image on the display device. It should be noted that the display device or speaker of the notification device 14 may also be configured to face the passenger seated in each seat.

[0031] The automatic response device 3 generates a first response message in response to a statement made by any of the passengers from vehicle 2. Furthermore, the automatic response device 3 generates an inquiry message representing the content of the statement and sends the generated inquiry message to server 4 via wireless communication terminal 13. The automatic response device 3 also receives a second response message from server 4 in response to the inquiry message via wireless communication terminal 13. Then, the automatic response device 3 responds to the passenger based on at least one of the first and second response messages.

[0032] Figure 3 This is a hardware configuration diagram of the automatic response device 3. (For example...) Figure 3 As shown, the automatic response device 3 has a communication interface 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 can be configured as separate circuits, or they can be integrated into a single integrated circuit.

[0033] The communication interface 21 has interface circuitry for connecting the automatic response device 3 to other devices within the vehicle. The communication interface 21 transmits in-vehicle images received from the camera 11 and voice signals received from each microphone 12 to the processor 23. Furthermore, the communication interface 21 outputs notification signals received from the processor 23 to the notification device 14, or outputs control commands received from the processor 23 for any of the in-vehicle devices. Moreover, the communication interface 21 outputs query information received from the processor 23 to the wireless communication terminal 13, and conversely, outputs second response information received from the wireless communication terminal 13 to the processor 23.

[0034] Memory 22 is an example of a storage unit, such as having volatile semiconductor memory and non-volatile semiconductor memory. Furthermore, memory 22 stores various data used in the automatic response processing executed by processor 23. Specifically, memory 22 stores a parameter set defining a first generation model used to generate response information. Memory 22 can also temporarily store in-vehicle images received from camera 11 and voice signals received from each microphone 12. Additionally, memory 22 temporarily stores second response information. Furthermore, memory 22 stores waiting response messages during the waiting period until the reception of the second response information is completed.

[0035] Processor 23 has one or more CPUs (Central Processing Units) and their peripheral circuitry. Processor 23 may also have other arithmetic circuitry such as logic units, numerical processing units, or graphics processing units. Furthermore, processor 23 performs automatic response processing.

[0036] Figure 4 This is a functional block diagram of the processor 23 related to automatic response processing. The processor 23 has a first response generation unit 31, a transceiver processing unit 32, and a response processing unit 33. These units of the processor 23 are, for example, functional modules implemented by a computer program that operates on the processor 23. Alternatively, these units of the processor 23 may also be dedicated arithmetic circuits provided on the processor 23.

[0037] The first response generation unit 31 generates first response information in response to the content of a speech by inputting prescribed input information containing voice information representing the content of a speech by any of the occupants of vehicle 2 into a first generation model. Furthermore, the first response generation unit 31 generates inquiry information corresponding to the content of the speech.

[0038] To generate speech information representing the content of a speech, the first response generation unit 31 inputs a speech signal whose average volume over a recent predetermined period exceeds a speech detection threshold from the speech signals generated by each microphone 12 into a speech recognition model. This model recognizes the content of the speech represented by the speech signal and generates a string representing the speech content as speech information. Such a speech recognition model is configured, for example, as a recursive structure DNN with an attention mechanism, such as a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), or a Long Short-Term Memory (LSTM) network. Alternatively, the speech recognition model can be configured as a GMM-HMM based on a mixture of normal distributions and Hidden Markov Models, or a DNN-HMM based on DNNs and Hidden Markov Models. It should be noted that the first response generation unit 31 can segment the speech signal into frames of a predetermined time length, extract speech features for each frame, and input the features of each frame into the speech recognition model in chronological order, thereby recognizing the content of the speech represented by the speech signal. Furthermore, the feature quantities of each frame can be set as, for example, the specified elements of the cepstrum of that frame.

[0039] In this embodiment, the first generative model is configured as a Large Language Model (LLM). The LLM, serving as the first generative model, is configured, for example, as a model composed of multiple stacked blocks containing attention layers and feedforward layers. The first response generation unit 31 inputs a string representing the content of the speech into the LLM. Thereby, the first generative model outputs text data representing the response content corresponding to the speech content as first response information.

[0040] It should be noted that the first response information is not limited to the response content notified to each passenger via the notification device 14, but may also include response content for controlling the vehicle 2 itself or the equipment mounted on the vehicle 2.

[0041] The first generation model is a computationally smaller model, enabling it to generate a first response to the passenger's speech within a short timeframe, even when installed in the processor 23 of the vehicle's automatic response device 3, without causing the passenger any anxiety. Therefore, the response content in the first response information generated by the first generation model is relatively simpler than the response content in the second response information generated by the second generation model.

[0042] According to a variation, the first response generation unit 31 may also include the in-vehicle image or a portion of the in-vehicle image showing each passenger along with the voice information in the input information fed to the first generation model. In this case, the first generation model is configured as a Vision Language Model (VLM). Thus, by inputting an in-vehicle image or a portion of the passenger's image, the first generation model can generate first response information based on the passenger's state.

[0043] When a portion of the area representing the occupant is input into the generative model, the first response generation unit 31 detects the portion of the area representing the occupant by inputting the in-vehicle image into a classifier that has been pre-learned to detect occupants. The classifier for occupant detection is configured as a Single Shot MultiBox Detector, a Convolutional Neural Network (CNN) such as Faster R-CNN, or a DNN with an attention mechanism such as a Vision Transformer.

[0044] Furthermore, when a portion of the occupant's area is input into the first generation model, the first response generation unit 31 can mask the area outside the portion by cropping the portion of the in-vehicle image for each occupant, or by converting the values ​​of each pixel contained in the area outside the portion of each occupant's area into a predetermined pixel value.

[0045] The first response generation unit 31 can also include at least one of the following in the input information: the current position of the vehicle 2, the destination, sensor signals obtained by sensors installed on the vehicle 2 for sensing the behavior of the vehicle 2, the interior conditions of the vehicle 2, or the conditions around the vehicle 2, the amount of operation performed by the occupant on the vehicle 2, and signals indicating the settings of the in-vehicle equipment. It should be noted that the sensors for sensing the behavior of the vehicle 2 are, for example, speed sensors or acceleration sensors. Furthermore, the sensors for sensing the interior conditions of the vehicle 2 or the conditions around the vehicle 2 are, for example, thermometers, illuminance meters, or rain sensors. Moreover, the amount of operation performed by the occupant on the vehicle 2 is, for example, accelerator opening, braking amount, or steering wheel angle. Additionally, the settings of the in-vehicle equipment are, for example, air conditioning temperature settings, fan speed settings, window opening / closing status, and audio system volume settings. Furthermore, the current position of the vehicle 2 is determined by a satellite positioning system-based positioning device (not shown), such as a GPS (Global Positioning System) receiver installed on the vehicle 2. The destination of the vehicle 2 can be obtained from a navigation device (not shown) installed on the vehicle 2. Hereinafter, these signals and information will be referred to as vehicle state information. The vehicle state information is input into the first generation model, which can then refer to the state of vehicle 2 or the onboard equipment to generate more appropriate response information as the first response information.

[0046] It should be noted that the first response generation unit 31 generates text data input to the first generation model by converting the types and signal values ​​of each sensor signal contained in the vehicle state information into strings, and combining the converted strings with strings representing speech information. Alternatively, an input layer for inputting vehicle state information can be set up separately from the block into which the speech information is input in the first generation model. In this case, only the string corresponding to the speech information is input to the block closest to the input side among the stacked blocks, and the vehicle state information is input to the input layer for vehicle state information input. Then, the vehicle state information input to this input layer is taken into the block of the stacked blocks in the generation model, which has a cross-attention mechanism that calculates cross-attention between the vehicle state information and the output from the previous block. In this case, a tuning method such as LoRA can also be applied to the learning of the first generation model related to the taking in of vehicle state information.

[0047] The first response generation unit 31 also generates query information. In this case, the first response generation unit 31 includes all input information fed to the first generation model in the query information. Alternatively, the first generation model can also be configured to generate both first response information and query information based on the input information. In this case, the first generation model is configured such that one or more blocks that branch from the middle of a stack of blocks to generate the first response information and one or more blocks that generate the query information are arranged side-by-side. Furthermore, each block after the branch is configured to include an attention layer and a feedforward layer. Then, for each of the first response information and the query information, the output probability is determined according to the output probability calculated by performing a softmax operation on the output from the corresponding final-level block. In this case, a tuning method such as LoRA can also be applied to the learning of the first generation model related to the first response information and the query information, which are added to the base model. Moreover, the first generation model is pre-learned in a manner where the query information is text data whose output is text data in which the key points of the speech content represented by the input speech information are explicitly defined. For example, if the passenger's statement is "How long until we reach ○○?", the first generation model outputs text data such as "What is the predicted time required to reach ○○ from △△?" (Note that △△ is the current position of vehicle 2) as the query information. In this way, the first generation model generates query information based on the speech signal representing the passenger's statement, thereby enabling the first response generation unit 31 to generate query information that makes the key points of the statement more explicit.

[0048] The first response generation unit 31 outputs first response information to the response processing unit 33 and outputs query information to the transceiver processing unit 32.

[0049] When a query is received, the transceiver processing unit 32 includes identification information of vehicle 2 or identification information of the wireless communication terminal 13 mounted on vehicle 2 in the query. Then, the transceiver processing unit 32 sends the query containing the identification information of vehicle 2 or wireless communication terminal 13 to the server 4 via the wireless communication terminal 13.

[0050] Furthermore, when receiving the second response information from the server 4 via the wireless communication terminal 13, the transceiver processing unit 32 sequentially outputs the received portion of the second response information to the response processing unit 33.

[0051] The response processing unit 33 performs response processing to respond to the occupants of the vehicle 2 based on at least one of the first response information and the second response information. In this embodiment, the response processing includes not only notifying the occupants via the notification device 14 according to the response information, but also controlling the vehicle 2 or various devices mounted on the vehicle 2 according to the response information.

[0052] Upon receiving a first response message, the response processing unit 33 generates a notification signal representing the response content contained in the first response message and outputs the generated notification signal to the notification device 14 via the communication interface 21. For example, the response processing unit 33 generates a speech signal representing the response content based on text data representing the response content contained in the first response message, using a predetermined speech synthesis method, as the notification signal. Then, the response processing unit 33 outputs the speech representing the response content from the speaker of the notification device 14 by outputting the notification signal. Alternatively, the response processing unit 33 includes text data representing the response content in the notification signal. Then, the response processing unit 33 causes the display device of the notification device 14 to display the text data representing the response content.

[0053] Furthermore, the response processing unit 33 controls the device specified by the response content included in the first response information according to the response content. The response processing unit 33 determines the device to be controlled and the control command by referring to a control reference table, wherein the control reference table indicates the correspondence between text data, the device to be controlled (including the vehicle 2 itself), and the control command used to execute the control, and the text data is data representing the response content included in the first response information. Then, the response processing unit 33 outputs the determined control command to the electronic control unit (ECU) of the device to be controlled via the communication interface 21.

[0054] It should be noted that, in addition to the air conditioning unit, the controlled devices may also include any one of the following: vehicle windows, door locks, interior lights, and seats. The response processing unit 33 performs the opening and closing of vehicle windows, locking / unlocking of vehicle doors, turning interior lights on or off, or adjusting the seat position of any passenger, as a control of the devices according to the response content.

[0055] Furthermore, if the text data representing the response content does not contain any of the words registered in the control reference table that identify the controlled device, the response information does not control the device. Therefore, in this case, the response processing unit 33 does not output a control signal.

[0056] When the response output based on the first response information ends, the response processing unit 33 responds according to the second response information. At this time, the response processing unit 33 performs the same processing on the second response information as it does on the response based on the first response information. It should be noted that if the response content contained in the first response information is the same as the response content contained in the second response information, the response processing unit 33 may omit the response based on the second response information.

[0057] Furthermore, the response processing unit 33 can also generate third response information by inputting the second response information into the first generation model. In this case, the first generation model is pre-learned not only to generate first response information (and, depending on the case, query information) when the aforementioned input information is input, but also to generate a response output to the passenger corresponding to the input when text data representing the response content contained in the second response information is input. Therefore, the response processing unit 33 can simply input the text data representing the response content contained in the second response information into the first generation model. Moreover, the response processing unit 33 can perform the same response processing as the response processing based on the first response information based on the third response information. In this way, by responding based on the third response information generated by reusing the first generation model used to generate the first response information, the response processing unit 33 can provide a more natural response to the passenger. It should be noted that the response processing unit 33 can also input text data obtained by combining the text data representing the response content contained in the first response information and the text data representing the response content contained in the second response information into the first generation model when generating the third response information. Therefore, the response processing unit 33 can further improve the matching between the response content contained in the third response information and the response content contained in the first response information.

[0058] According to a variation, the response processing unit 33 may not perform response processing based on the first response information before the reception of the second response information is completed, but instead, at the point when the reception of the second response information is completed, use the first generation model to generate the third response information as described above. Furthermore, the response processing unit 33 may also perform response processing based on the third response information.

[0059] According to another variation, the response processing unit 33 may not perform response processing based on the first response information, but instead perform response processing based on the second response information at the point in time when the reception of the second response information is completed.

[0060] According to another variation, if the reception of the second response information is not completed at the time when the response processing based on the first response information is completed, the response processing unit 33 may notify the passenger of a predetermined waiting response message via the notification device 14 during the waiting period from the completion of the response processing based on the first response information to the completion of the reception of the second response information. The waiting response message may be, for example, a message informing the passenger that the response is not complete, such as "Please wait" or "Inquiry is currently in progress." Even if there is a certain time difference between the completion of the response processing based on the first response information and the receipt of the second response information, notifying the passenger of such a waiting response message prevents a lack of response. Therefore, the response processing unit 33 can provide a more natural response to the passenger.

[0061] Furthermore, if the response processing unit 33 continues to receive second response information at a point in time when the response processing based on the first response information is completed, it can generate a waiting response message by inputting a portion of the second response information received before that point in time into the first generation model. It should be noted that if a marker indicating the end of the second response information exists in a portion of the received second response information, the response processing unit 33 determines that the reception of the second response information is complete; otherwise, it determines that the reception of the second response information is continuing. In this case, the response processing unit 33 can sequentially input the received portion of the second response information into the first generation model each time it receives a portion. Then, the text data output from the first generation model at the point in time when the response processing based on the first response information is completed can be used as the waiting response message. Alternatively, the response processing unit 33 can also generate a waiting response message by re-inputting the text data output from the first generation model based on a portion of the second response information sequentially input before the point in time when the response processing based on the first response information is completed into the first generation model. It should be noted that if the length of the text data contained in a portion of the second response information received at the time when the response processing based on the first response information is completed is less than a predetermined lower threshold, the response processing unit 33 can notify the message for waiting for response pre-stored in the memory 22 via the notification device 14. This prevents the notification of a meaningless waiting response message due to insufficient portion of the second response information received at the time when the response processing based on the first response information is completed.

[0062] Next, server 4 will be described. A second generation model is installed on server 4, and server 4 uses the second generation model to generate second response information in response to the query information.

[0063] Figure 5 This is a hardware configuration diagram of server 4. Server 4 has a communication interface 41, a storage device 42, a memory 43, and a processor 44. The communication interface 41, the storage device 42, and the memory 43 are connected to the processor 44 via signal lines.

[0064] Communication interface 41 is an example of a communication unit, having interface circuitry for connecting server 4 to communication network 5. Furthermore, communication interface 41 is configured to communicate with automatic response device 3 mounted on vehicle 2 via communication network 5, wireless base station 6, and wireless communication terminal 13 mounted on vehicle 2. Specifically, communication interface 41 transmits query information received from automatic response device 3 of vehicle 2 via wireless communication terminal 13, wireless base station 6, and communication network 5 to processor 44. Additionally, communication interface 41 sends second response information received from processor 44 to automatic response device 3 of vehicle 2 via communication network 5, wireless base station 6, and wireless communication terminal 13 of vehicle 2. Moreover, communication interface 41 transmits various information received from other servers connected via communication network 5 (e.g., servers distributing traffic information or weather information) to processor 44.

[0065] Storage device 42 may include, for example, a solid-state drive, a hard disk drive, or an optical recording medium and its access device. Furthermore, storage device 42 stores parameter sets defining the second generation model. Storage device 42 may also store identification information of vehicle 2 or identification information of the wireless communication terminal 13 mounted on vehicle 2. Moreover, storage device 42 may also store computer programs executed on processor 44 for performing automatic response processing on server 4. Additionally, storage device 42 may store various information received from other servers.

[0066] The memory 43 may include, for example, non-volatile semiconductor memory and volatile semiconductor memory. Furthermore, the memory 43 temporarily stores various types of data generated during the execution of the automatic response process or used in the automatic response process.

[0067] Processor 44 has one or more CPUs (Central Processing Units) and their peripheral circuitry. Processor 44 may also have other arithmetic circuitry such as logic units or numerical processing units. Furthermore, processor 44 performs automatic response processing on the server 4 side.

[0068] Figure 6 This is a functional block diagram of a processor 44 related to automatic response processing on the server side. The processor 44 has a transceiver processing unit 51 and a second response generation unit 52. These units of the processor 44 are, for example, functional modules implemented by a computer program that operates on the processor 44. Alternatively, these units of the processor 44 may also be dedicated arithmetic circuitry provided on the processor 44.

[0069] When an inquiry message is received from the automatic response device 3 of vehicle 2, the transceiver processing unit 51 outputs the information (text data, in-vehicle image, vehicle status information, etc.) contained in the inquiry message to the second response generation unit 52, which is to be input into the second generation model. Furthermore, when the second response message is received from the second response generation unit 52, the transceiver processing unit 51 determines the vehicle 2 that sent the inquiry message or the wireless communication terminal 13 mounted on that vehicle 2 by referring to the identification information of vehicle 2 or wireless communication terminal 13 contained in the inquiry message. Then, the transceiver processing unit 51 sends the second response message to the determined wireless communication terminal 13 of vehicle 2 via the communication interface 41, the communication network 5, and the wireless base station 6. At this time, the transceiver processing unit 51 sends the second response message after the second response message generated by the second generation model is completed. Alternatively, the transceiver processing unit 51 may also send a portion of the second response message each time the second generation model outputs a portion of the second response message (e.g., text data with a predetermined number of characters or words).

[0070] The second response generation unit 52 generates second response information by inputting the text data contained in the query information into the second generation model. Furthermore, if the query information includes an interior image or an interior image showing a portion of the occupant's area, the second response generation unit 52 also inputs the interior image or that portion into the second generation model. Similarly, if the query information includes vehicle status information, the second response generation unit 52 also inputs the vehicle status information into the second generation model.

[0071] The second generative model is configured as an LLM (however, in cases where an in-vehicle image or a portion of the image is also input) in the same way as the first generative model. The second generative model is a larger-scale generative model than the first; for example, the second generative model has more blocks with attention mechanisms and feedforward layers than the first generative model. Furthermore, the dataset used for learning the second generative model is larger than the dataset used for learning the first generative model. Therefore, although the second generative model performs more computations than the first generative model, it can generate second response information containing more detailed or accurate response content than the first response information generated by the first generative model. For example, if a passenger's statement requests an explanation of a specific thing or phenomenon, the second generative model can generate second response information containing a more detailed or accurate explanation of that thing or phenomenon than the first response information generated by the first generative model. For example, if a passenger's request is "Tell me information about the ○○ Building," the first response generated by the first generation model contains simple information such as "The ○○ Building is located at △△." In contrast, the second response generated by the second generation model contains more detailed information such as "The ○○ Building is located at △△, and its height is □□ meters. There is a famous restaurant called ××." Furthermore, if a passenger's request is "How much longer until ○○?", the first response generated by the first generation model queries the navigation device installed in vehicle 2 for the estimated time from vehicle 2's current location to ○○, and the result of this query includes "△ minutes remaining." In contrast, the second response generated by the second generation model references the latest traffic information provided by the traffic information server and the route search results obtained from the route search algorithm implemented on server 4, representing a more accurate estimated time to the destination.

[0072] When the second response information is generated by the second generation model, the second response generation unit 52 outputs the second response information to the transceiver processing unit 51. It should be noted that the second response generation unit 52 may also output a portion of the second response information each time the second generation model outputs it, for example, when outputting text data with a specified number of characters.

[0073] Figure 7 (A) and Figure 7 (B) are explanatory diagrams for automatic response processing. Figure 7In the example shown in (A), an inquiry message 703 is generated by inputting input information 701, which contains voice information representing the content of a passenger's speech, into the first generation model 702 in the automatic response device 3 of vehicle 2. Then, the inquiry message 703 is sent to the server 4 and input into the second generation model 704 in the server 4 to generate second response information 705. When the automatic response device 3 receives the second response information 705 from the server 4, it performs response processing for the passenger based on the second response information 705. It should be noted that, in this example, as described above, the automatic response device 3 can perform response processing based on the second response information 705 itself, or it can perform response processing based on third response information 706 generated by the first generation model 702 by inputting the second response information 705 into the first generation model 702. Furthermore, in this example, the first response information generated by inputting input information 701 into the first generation model 702 may not be used for response processing, or the first response information may be used in the generation of the third response information 706 by being input into the first generation model 702 together with the second response information 705. In this example, response processing is performed according to the response information generated by the second generation model 704, which is larger in scale than the first generation model 702, so the automatic response system 1 can provide a detailed or accurate response. It should be noted that, in this example, it is more preferable to use the first generation model 702 or to perform response processing according to the third response information 706 generated by inputting the second response information 705 into the first generation model 702 in the generation of the inquiry information 703. As described above, by using the first generation model 702 in the generation of the inquiry information 703, the key points of the content of the speech made by the passenger in the inquiry information 703 become clearer, so the response content of the second response information 705 generated based on the inquiry information 703 becomes more appropriate. Furthermore, by performing response processing according to the third response information 706, the automatic response system 1 can respond more naturally as described above.

[0074] exist Figure 7In the example shown in (B), first response information 713 and inquiry information 714 are generated by inputting input information 711, which contains voice information representing the content of a passenger's speech, into the first generation model 712 in the automatic response device 3 of vehicle 2. The automatic response device 3 performs response processing based on the first response information 713 and sends the inquiry information 714 to the server 4. Then, second response information 716 is generated by inputting the inquiry information 714 into the second generation model 715 in the server 4. When the automatic response device 3 receives the second response information 716 from the server 4, it then performs response processing based on the second response information 716, in accordance with the response processing based on the first response information 713. It should be noted that, in this example, as described above, the automatic response device 3 may also perform response processing based on the second response information 716 itself, or it may perform response processing based on the third response information 717 generated by inputting the second response information 716 into the first generation model 712. Furthermore, in this example, the first response information 713 can also be used in the generation of the third response information 717 by being input into the first generation model 712 along with the second response information 716. In this example, while the server 4 is generating the second response information 716, response processing is performed first according to the first response information 713, thus shortening the waiting time from the passenger's statement to the initial response, thereby alleviating the passenger's tension. Moreover, after response processing according to the first response information 713, response processing is performed according to the second response information 716 or the third response information 717, so the passenger can receive a more detailed or more accurate response.

[0075] Figure 8 This is a flowchart of the automatic response processing according to this embodiment. The processor 23 of the automatic response device 3 and the processor 44 of the server 4 execute the automatic response processing according to this flowchart.

[0076] The first response generation unit 31 in the processor 23 of the automatic response device 3 generates first response information by inputting input information containing voice information representing the speech content of any passenger in the vehicle 2 to the first generation model (step S101). Furthermore, the first response generation unit 31 generates inquiry information based on the input information (step S102). It should be noted that, as described above, the input information may also include an in-vehicle image, a portion of the in-vehicle image showing the passenger, or vehicle status information.

[0077] The transceiver processing unit 32 in the processor 23 of the automatic response device 3 sends query information to the server 4 via the wireless communication terminal 13, the wireless base station 6, and the communication network 5 (step S103). Furthermore, the response processing unit 33 in the processor 23 of the automatic response device 3 begins response processing based on the first response information (step S104).

[0078] The second response generation unit 52 in the processor 44 of server 4 generates second response information by inputting query information into the second generation model (step S105). Then, the transceiver processing unit 51 in the processor 44 of server 4 sends the second response information to the automatic response device 3 via the communication network 5, the wireless base station 6, and the wireless communication terminal 13 of vehicle 2 (step S106). When the automatic response device 3 receives the second response information, the response processing unit 33 then performs response processing based on the second response information based on the response processing based on the first response information (step S107). It should be noted that, as described above, the response processing based on the first response information in step S104 can also be omitted. Furthermore, in step S107, the response processing unit 33 can also perform response processing based on the third response information generated by inputting the second response information into the first generation model.

[0079] As explained above, this automatic response system can appropriately respond to the speech of passengers by utilizing a relatively large-scale generative model located outside the vehicle. Moreover, this automatic response system can maintain the appropriateness of the response content by combining a relatively small-scale generative model mounted in the vehicle with a server-side generative model, and can shorten the waiting time until a response to the speech of passengers is received.

[0080] The computer program that implements the automatic response processing according to the above-described embodiments or variations can be provided as a computer program product, for example, in the form of a computer-readable portable recording medium.

[0081] As described above, those skilled in the art can make various modifications within the scope of this invention depending on the implementation method.

Claims

1. An automatic response system comprising an automatic response device mounted on a vehicle and a server disposed externally on the vehicle, wherein, The automatic response device has: The first response generation unit generates first response information by inputting input information containing voice information representing the content of a speech from a passenger of the vehicle into a first generation model, and generates query information representing the content of the speech based on the input information. The first generation model is a model mounted on the vehicle and pre-learned in a manner that generates the first response information for the content of the speech. The transceiver processing unit sends the query information to the server via a communication device mounted on the vehicle, and receives second response information generated based on the query information from the server; and The response processing unit responds to the passenger based on at least one of the first response information and the second response information. The server has a second response generation unit, which generates the second response information by inputting the query information into a second generation model, wherein the second generation model is a model that has been pre-learned in the manner of generating the second response information and is larger in scale than the first generation model.

2. The automatic response system according to claim 1, wherein, The first generative model is pre-learned in a manner that generates the first response information and the query information based on the input information. The first response generation unit generates the query information by inputting the input information into the first generation model.

3. The automatic response system according to claim 1 or 2, wherein, The response processing unit responds to the passenger based on the generated first response information, and then further responds to the passenger based on the second response information received from the server.

4. The automatic response system according to claim 1 or 2, wherein, The response processing unit generates third response information by inputting the second response information into the first generation model, and responds to the passenger based on the generated third response information.

5. The automatic response system according to claim 1 or 2, wherein, During the waiting period from responding to the passenger based on the first response information until receiving the second response information, the response processing unit notifies the passenger of a predetermined waiting response via a notification device mounted on the vehicle.

6. An automatic response method, comprising: First response information is generated by inputting voice information containing the content of a speech from a occupant of the vehicle into a first generative model, wherein the first generative model is mounted on the vehicle and has been pre-learned in a manner that generates the first response information in response to the content of the speech. Based on the input information, generate query information representing the content of the speech; A second response is generated by inputting the query information into a second generative model, wherein the second generative model is pre-learned in a manner that generates the second response information in response to the query information, is larger in scale than the first generative model, and is located outside the vehicle; and The passenger responds based on at least one of the first response information and the second response information.

7. A computer program product for automatic response, comprising instructions for causing a computer to perform the following actions: First response information is generated by inputting voice information containing speech information representing the content of statements made by occupants of the vehicle into a first generative model, wherein... The first generative model is a model mounted on the vehicle and pre-learned in a manner that generates the first response information in response to the content of the speech; Based on the input information, generate query information representing the content of the speech; The second response information is generated by inputting the query information into a second generation model, wherein the second generation model is pre-learned in a manner that generates the second response information in response to the query information, is larger in scale than the first generation model, and is located outside the vehicle. as well as The passenger responds based on at least one of the first response information and the second response information.

8. An automatic response device, comprising: The first response generation unit generates first response information by inputting input information containing voice information representing the content of a speech from a vehicle occupant into a first generation model, and generates query information representing the content of the speech based on the input information. The first generative model is a model mounted on the vehicle and pre-learned in a manner that generates the first response information in response to the content of the speech; The transceiver processing unit sends the query information to a server located outside the vehicle via a communication device mounted on the vehicle, and receives a second response information generated based on the query information from the server. as well as The response processing unit responds to the passenger based on at least one of the first response information and the second response information.

Images