Automated response apparatus, automated response method and computer program product

Abstract

The present invention relates to an automatic response device, an automatic response method, and a computer program product. The automatic response device includes a storage section configured to store each of a plurality of vehicle signals linked to a vehicle operation demand in a vector format, an input processing section configured to vectorize query text corresponding to an input from a user, and a signal extraction section configured to extract a vehicle signal suitable for the query text from the plurality of vehicle signals by comparing the query text vectorized by the input processing section with the vehicle operation demand.

Description

Technical Field

[0001] This invention relates to an automatic response device, an automatic response method, and a computer program product. Background Technology

[0002] Conventionally, it is known to generate responses to user input using natural language by employing generative AI models, such as Large Language Models (LLMs). Patent Document 1 describes a method for improving the accuracy of the LLM's output by selecting a database corresponding to the user's question from multiple databases and inputting a prompt generated by adding information from the database to the question into the LLM.

[0003] [List of Citations]

[0004] [Patent Literature]

[0005] [Patent Document 1] Japanese Patent No. 7441366 Summary of the Invention

[0006] [Technical Issues]

[0007] However, even if a database corresponding to the user's question is selected, it is not always possible to extract the information needed to answer the question from the database. On the other hand, if all additional information related to the user input is fed into the generative AI model, the computational load of the generative AI model increases. There is usually a limit to the number of tokens that can be input into the LLM (Limited Language Model) of a generative AI model. Similar problems may occur when using algorithms other than LLMs to output responses to user input.

[0008] In view of the above problems, the object of the present invention is to efficiently obtain additional information for appropriately responding to user input.

[0009] [Solution to the problem]

[0010] The following is an overview of this disclosure.

[0011] (1) An automatic response device, comprising: a storage unit for storing each of a plurality of vehicle signals linked to a vehicle operation requirement in a vector format; an input processing unit configured to vectorize a query text corresponding to an input from a user; and a signal extraction unit configured to extract a vehicle signal suitable for the query text from the plurality of vehicle signals by comparing the query text vectorized by the input processing unit with the vehicle operation requirement.

[0012] (2) The automatic response device described in part (1) above, wherein, in the storage unit, two or more vehicle signals are linked to a vehicle operation request.

[0013] (3) The automatic response device described in part (1) or (2) above further includes an information request unit configured to request information from a server, wherein the automatic response device is installed in the vehicle and the server is installed outside the vehicle, and the information request unit is configured to send a query text to the server when the vehicle signal is not extracted by the signal extraction unit, and to obtain information related to the query text from the server.

[0014] (4) The automatic response device described in part (3) above further includes an operation determination unit configured to determine a vehicle operation that should be performed in response to a query text based on a vehicle signal extracted by the signal extraction unit, wherein the operation determination unit is configured to determine the vehicle operation based on the information when the vehicle signal is not extracted by the signal extraction unit.

[0015] (5) The automatic response device described in part (1) or (2) above further includes an operation determination unit configured to determine a vehicle operation that should be performed in response to a query text based on a vehicle signal extracted by the signal extraction unit.

[0016] (6) The automatic response device described in part (4) or (5) above further includes a feedback unit configured to obtain feedback from the user on vehicle operation, wherein the operation determination unit is configured to determine vehicle operation based on a predetermined algorithm, and the feedback unit is configured to improve the algorithm based on the user's feedback.

[0017] (7) The automatic response device described in any of the above sections (4) to (6), wherein the operation determination unit is configured to determine vehicle operation based on vehicle signals extracted by the signal extraction unit and images generated by the camera.

[0018] (8) The automatic response device described in any of the above sections (1) to (7) further includes a data management unit configured to manage data stored in the storage unit, wherein the data management unit is configured to add at least one of a new vehicle signal and a new vehicle operation request to the data in response to a predetermined trigger.

[0019] (9) An automatic response method executed by a computer, comprising: storing each of a plurality of vehicle signals linked to a vehicle operation requirement in a vector format; vectorizing a query text corresponding to input from a user; and extracting a vehicle signal suitable for the query text from the plurality of vehicle signals by comparing the vectorized query text with the vehicle operation requirement.

[0020] (10) A computer program product comprising a computer program that enables a computer to: store each of a plurality of vehicle signals linked to vehicle operation requirements in a vector format, vectorize a query text corresponding to input from a user, and extract vehicle signals suitable for the query text from the plurality of vehicle signals by comparing the vectorized query text with the vehicle operation requirements.

[0021] Beneficial effects of the present invention

[0022] According to this disclosure, additional information for appropriately responding to user input can be obtained effectively. Attached Figure Description

[0023] Figure 1 This is a schematic configuration diagram of a vehicle including an automatic response device according to a first embodiment of the present invention.

[0024] Figure 2 It is a schematic configuration view of the UI.

[0025] Figure 3 This is a schematic diagram of the automatic response device.

[0026] Figure 4 This is a functional block diagram of the processor of the automatic response device according to the first embodiment of the present invention.

[0027] Figure 5 This is a flowchart illustrating the control routine for the operation determination process of the first embodiment of the present invention.

[0028] Figure 6 This is a schematic configuration diagram of an automatic response system including an automatic response device according to a second embodiment of the present invention.

[0029] Figure 7 This is a functional block diagram of the processor of the automatic response device according to the second embodiment of the present invention.

[0030] Figure 8 This is a flowchart illustrating the control routine for the operation determination process used in the second embodiment of the present invention.

[0031] Figure 9 This is a schematic configuration diagram of a vehicle including an automatic response device according to a third embodiment of the present invention.

[0032] Figure 10 This is a flowchart illustrating the control routine for the operation determination process used in the third embodiment of the present invention.

[0033] Figure 11 This is a functional block diagram of the processor of the automatic response device according to the fourth embodiment of the present invention.

[0034] Figure 12 This is a flowchart illustrating the control routine for feedback processing according to a fourth embodiment of the present invention.

[0035] Figure 13 This is a functional block diagram of the processor of the automatic response device according to the fifth embodiment of the present invention.

[0036] Figure 14 This is a flowchart illustrating the control routine for data update processing according to the fifth embodiment of the present invention. Detailed Implementation

[0037] Embodiments of the present invention will now be explained in detail with reference to the accompanying drawings. It should be noted that in the following explanation, similar elements will be assigned the same reference numerals.

[0038] <First Embodiment>

[0039] The following will refer to Figures 1 to 5 The first embodiment of the present invention is described. Figure 1 This is a schematic configuration diagram of a vehicle 1 including an automatic response device 10 according to a first embodiment of the present invention. In this embodiment, vehicle 1 is a four-wheeled vehicle. Figure 1 As shown, vehicle 1 includes a user interface (UI) 2 and an automatic response device 10. The UI 2 is electrically connected to the automatic response device 10 via an in-vehicle network that conforms to standards such as Controller Area Network (CAN) or Ethernet.

[0040] UI 2 is set up in the carriage and sends and receives information between vehicle 1 and its occupants (e.g., the driver). Figure 2 This is a schematic configuration view of UI 2. UI 2 includes, for example, input device 21 and output device 22.

[0041] Input device 21 accepts input from occupants of vehicle 1. In this embodiment, input device 21 includes a microphone and accepts voice input from occupants of vehicle 1. In addition to or in place of a microphone, input device 21 may include a touch panel, etc. UI 2 sends the input data entered by occupants of vehicle 1 into input device 21 to automatic response device 10.

[0042] Output device 22 notifies the occupants of vehicle 1. In this embodiment, output device 22 includes at least one of a display and a speaker. UI 2 notifies the occupants of vehicle 1 of information corresponding to the signal sent from automatic response device 10 via output device 22.

[0043] Figure 3 This is a schematic configuration diagram of the automatic response device 10. (See diagram for example.) Figure 3As shown, the automatic response device 10 includes a communication interface 11, a memory 12, a processor 13, and a storage device 14. The communication interface 11, memory 12, and storage device 14 are connected to the processor 13 via signal lines. The communication interface 11, memory 12, and processor 13 can be configured as a single integrated circuit or as multiple separate circuits. The communication interface 11, memory 12, and processor 13 can be configured as a single electronic control unit (ECU) or multiple ECUs.

[0044] The communication interface 11 has interface circuitry for connecting the automatic response device 10 to an in-vehicle network. The automatic response device 10 connects to other in-vehicle devices via the communication interface 11. For example, the communication interface 11 sends signals received from the input device 21 of the UI 2 to the processor 13. Furthermore, the communication interface 11 sends signals output from the processor 13 to the output device 22 of the UI 2.

[0045] The memory 12 includes, for example, volatile semiconductor memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM), etc.) and non-volatile semiconductor memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, etc.). The memory 12 stores temporary data, computer programs for various processes performed by the processor 13 (control programs for the automatic response device 10), setting data for the automatic response device 10, log data, vehicle information, etc.

[0046] Processor 13 has one or more central processing units (CPUs) and their peripheral circuitry. Processor 13 executes computer programs stored in memory 12. Processor 13 may further have other arithmetic circuitry, such as logic arithmetic units, numerical arithmetic units, or graphics processing units.

[0047] Storage device 14 includes, for example, a hard disk drive (HDD), a solid-state drive (SSD), or an optical recording medium and its access device. In this embodiment, storage device 14 stores a database, which will be described later. Storage device 14 is an example of a storage unit.

[0048] An automatic response device 10 is installed in vehicle 1 and responds to input from a user (such as an occupant of vehicle 1). For example, the automatic response device 10 uses a generative AI model (such as a large language model (LLM)) to generate a response to input from the user (hereinafter referred to as "user input"). For example, the automatic response device 10 generates an answer to a question from the user.

[0049] On the other hand, when a user inputs a request for vehicle operation into the automatic response device 10, the automatic response device 10 should output the vehicle operation corresponding to the user's request as a response to the user's input. However, since the generative AI model of the automatic response device 10 does not have the vehicle information required to determine the vehicle operation corresponding to the user's request, there is a risk that it may not be able to generate an appropriate response. Therefore, in this embodiment, the automatic response device 10 uses retrieval-enhanced generation (RAG) technology to supplement the information input to the generative AI model.

[0050] In this scenario, it is conceivable that vehicle signals related to vehicle operation are stored in storage device 14 as vehicle information required to determine appropriate vehicle operation, and that the vehicle signals in storage device 14 are input into the generative AI model in addition to user input. However, user input is unlikely to include words that indicate the vehicle signals themselves related to vehicle operation. Furthermore, even if the desired vehicle operation is the same, the user input used to request vehicle operation will be different for each user.

[0051] Therefore, even if a database containing multiple vehicle signals is built in storage device 14 for applying RAG technology, it is difficult to extract vehicle signals suitable for responding to user input from the database. On the other hand, if all vehicle signals are input into the generative AI model, the computational load of the generative AI model increases. Furthermore, there is usually a limit to the number of lexical units that can be input into the generative AI model's LLM.

[0052] Therefore, in this embodiment, in the database, vehicle operation requests issued by the user can be linked to vehicle signals related to those requests, and appropriate vehicle signals are extracted by comparing user input with the vehicle operation requests. In this embodiment, vector format data of both is used to compare user input and vehicle operation requests. In other words, vector similarity search is used to compare user input and vehicle operation requests. Therefore, when comparing user input and vehicle operation requests, the terms of both can be interpreted flexibly, thereby extracting the vehicle operation requests most relevant to the user input, and thus extracting vehicle signals suitable for processing the user input.

[0053] The specific configurations used to implement the above processing will be described below. Figure 4 This is a functional block diagram of the processor 13 of the automatic response device 10 according to the first embodiment of the present invention. Figure 4As shown, the processor 13 includes an input processing unit 31, a signal extraction unit 32, and an operation determination unit 33. The input processing unit 31, the signal extraction unit 32, and the operation determination unit 33 are functional modules implemented by the processor 13 of the automatic response device 10, which executes a computer program stored in the memory 12 of the automatic response device 10. It should be noted that these functional modules can each be implemented using dedicated arithmetic circuits provided in the processor 13.

[0054] The input processing unit 31 processes user input. User input is typically natural language. First, the input processing unit 31 creates a query text corresponding to the user input. For example, if the user input is voice input, the input processing unit 31 creates the query text by converting the voice data into text data using speech recognition technology. It should be noted that if the user input is text input, the input processing unit 31 can use the text input as is as the query text. The query text is also referred to as a "prompt".

[0055] The input processing unit 31 then vectorizes the query text. Specifically, the input processing unit 31 encodes the query text into a vector format. By vectorizing the query text, the query text, which is non-numerical data, is converted into a high-dimensional numeric vector. For example, the input processing unit 31 uses an embedding model to vectorize the query text. Specific examples of embedding models include sentence transformers, USE (Universal Sentence Encoder), text embedding-ada-002, etc.

[0056] On the other hand, storage device 14 stores each of a plurality of vehicle signals linked to vehicle operation requirements in vector format. The vehicle operation requirements are pre-vectorized using the embedding model described above and stored as vector format data in storage device 14. Specifically, storage device 14 stores a plurality of vehicle operation requirements as a vector database. Vehicle signals are dynamic signals that change according to the state of vehicle 1 and are updated accordingly. It should be noted that vehicle signals, along with their correspondence to vehicle operation requirements, may be stored in a different location than the vehicle operation requirements (e.g., a storage device different from storage device 14).

[0057] Examples of vehicle operation requests include "open the window," "lower the interior temperature," and "turn on the windshield wipers." Examples of vehicle signals include signals indicating the open / closed status of each window of vehicle 1, signals indicating the operating status of the air conditioning system of vehicle 1, and signals indicating the operating status of the windshield wipers of vehicle 1. For example, a signal indicating the open / closed status of each window of vehicle 1 is linked to the vehicle operation request "open the window," a signal indicating the operating status of the air conditioning system is linked to the vehicle operation request "lower the interior temperature," and a signal indicating the operating status of the windshield wipers is linked to the vehicle operation request "turn on the windshield wipers."

[0058] The signal extraction unit 32 extracts vehicle signals suitable for the query text, that is, vehicle signals required to correctly respond to the query text. First, the signal extraction unit 32 compares the query text vectorized by the input processing unit 31 with the vehicle operation requirements stored in the storage device 14 in vector format. In other words, the signal extraction unit 32 compares the vector-formatted query text with the vector-formatted vehicle operation requirements. The query text in vector format is also referred to as the query vector.

[0059] Specifically, the signal extraction unit 32 extracts the vehicle operation request most similar to the query text from multiple vehicle operation requests stored in the vector database in the storage device 14 using vector similarity search. In the vector similarity search, cosine similarity, Euclidean distance, dot product, maximum inner product, etc., are used as indices of similarity between vectors. For example, the input processing unit 31 uses a vector search algorithm (such as the Approximate Nearest Neighbor (ANN) algorithm) to perform the vector similarity search. Specific examples of ANN algorithms include kd-tree, LSH (Locality Sensitive Hash), HNSW (Hierarchical Navigable Small World), etc.

[0060] Next, the signal extraction unit 32 extracts vehicle signals linked to the vehicle operation request extracted through vector similarity search. It should be noted that in the storage device 14, two or more vehicle signals can be linked to a single vehicle operation request. In this way, even if the user input is associated with a complex vehicle operation request involving two or more vehicle signals, vehicle signals suitable for the vehicle operation request can be extracted. For example, a signal indicating the relative speed of vehicle 1 relative to the vehicle in front and a signal indicating the distance from vehicle 1 to the vehicle in front (the distance between vehicle 1 and the vehicle in front) are linked to the vehicle operation request of "following the vehicle in front".

[0061] As described above, the signal extraction unit 32 extracts vehicle signals suitable for the query text from multiple vehicle signals by comparing the query text vectorized by the input processing unit 31 with the vehicle operation requirements stored in the storage device 14 in vector format. Therefore, additional information for appropriately responding to user input can be obtained effectively.

[0062] The operation determination unit 33 determines the vehicle operation that should be performed in response to the query text based on the vehicle signals extracted by the signal extraction unit 32. For example, the operation determination unit 33 inputs the query text before vectorization and the extracted vehicle signals into the generative AI model, and causes the generative AI model to output the vehicle operation that should be performed in response to the query text. In this case, since the vehicle signals required to correctly respond to the query text are also input into the generative AI model in addition to the query text, the generative AI model can output the vehicle operation according to the user request. Therefore, the accuracy of the output of the generative AI model can be improved by adding appropriate vehicle signals to the input of the generative AI model.

[0063] The following will refer to Figure 5 Describe the processing flow used to perform the above controls. Figure 5 This is a flowchart illustrating a control routine for operation determination processing according to a first embodiment of the present invention. This control routine is repeatedly executed by the processor 13 of the automatic response device 10 according to a computer program, for example, stored in the memory 12 of the automatic response device 10.

[0064] First, in step S101, the input processing unit 31 of the processor 13 determines whether user input has been received. For example, if a vehicle occupant inputs voice into the microphone of the input device 21, the input processing unit 31 determines that user input has been received when voice input is sent from the input device 21 to the processor 13. If a vehicle occupant inputs text into the touch panel or similar device of the input device 21, the input processing unit 31 determines that user input has been received when text input is sent from the input device 21 to the processor 13. When it is determined in step S101 that no user input has been received, the control routine ends.

[0065] On the other hand, when it is determined in step S101 that user input has been received, the control routine proceeds to step S102. In step S102, the input processing unit 31 creates a query text corresponding to the user input based on the user input. For example, the input processing unit 31 creates the query text by converting the voice data of the voice input into text data using speech recognition technology. It should be noted that when the user input is text input, the input processing unit 31 can use the text input as the query text without performing data conversion.

[0066] Next, in step S103, the input processing unit 31 vectorizes the query text. It should be noted that before vectorizing the query text, preprocessing other than vectorization can be performed on the query text (e.g., normalization, lexicalization, stemming, etc.).

[0067] Next, in step S104, the signal extraction unit 32 of the processor 13 compares the vector-format query text with the vector-format vehicle operation requirements, and extracts the vehicle operation requirement most similar to the query text from the vector database in the storage device 14. Next, in step S105, the signal extraction unit 32 extracts the vehicle signals linked to the extracted vehicle operation requirements as vehicle signals suitable for the query text. It should be noted that in step S104, multiple vehicle operation requirements (e.g., two or three) can be extracted in descending order of similarity, and in step S105, vehicle signals linked to each of the multiple vehicle operation requirements can be extracted.

[0068] Next, in step S106, the operation determination unit 33 of the processor 13 determines the vehicle operation that should be performed in response to the query text based on the vehicle signal extracted by the signal extraction unit 32. For example, the operation determination unit 33 determines the vehicle operation by using the query text and the vehicle signal as input, so that the generative AI model outputs a response to the query text. In this case, the generative AI model is pre-stored in the memory 12 or storage device 14 of the automatic response device 10. After step S106, the control routine ends.

[0069] It should be noted that, although in this embodiment the vehicle operation determined by the operation determination unit 33 is performed by a device other than the automatic response device 10 (e.g., an application installed in the vehicle 1), the operation determination unit 33 can still execute the determined vehicle operation. In this case, the operation determination unit 33 executes the operation required to achieve the vehicle operation by controlling the actuators or the like of the vehicle 1.

[0070] Furthermore, the processor 13 of the automatic response device 10 can execute the processing steps S101 to S105, and devices other than the automatic response device 10 (e.g., applications installed in the vehicle 1) can execute the processing in step S106. In other words, the operation determination unit 33 can be omitted from the automatic response device 10.

[0071] The following describes a specific example of using vehicle signals to determine vehicle operation in response to a user request. For example, the driver of vehicle 1 inputs the voice command "Open the driver's window halfway through the journey." In this case, "open the window" is extracted as the vehicle operation request most similar to the query text. Therefore, the signal indicating the open / closed state of each window of vehicle 1 is extracted as the vehicle signal associated with this vehicle operation request, and the extracted vehicle signal, along with the query text, is input into the generative AI model.

[0072] In this scenario, the generative AI model is typically expected to output the following responses. For example, when a vehicle signal indicates that the driver's window is fully closed, the generative AI model outputs a vehicle operation to partially open the window of vehicle 1. Conversely, when a vehicle signal indicates that the driver's window is fully open, the generative AI model outputs a vehicle operation to partially close the window of vehicle 1. Furthermore, when a vehicle signal indicates that the driver's window is partially open, the generative AI model outputs a vehicle operation to notify the driver that the driver's window is in the desired state.

[0073] <Second Embodiment>

[0074] Except for the points described below, the configuration and control of the automatic response device according to the second embodiment are substantially the same as those of the automatic response device according to the first embodiment. Therefore, the second embodiment of the present invention will be described below focusing on its differences from the first embodiment.

[0075] Figure 6 This is a schematic configuration diagram of an automatic response system 100 including an automatic response device 10 according to a second embodiment of the present invention. The automatic response system 100 includes a vehicle 1' and a server 40. In addition to... Figure 1 In addition to the UI 2 and the automatic response device 10 shown, the vehicle 1' also includes a communication device 3. The communication device 3 is capable of communicating with the outside of the vehicle 1' and can realize communication between the vehicle 1' and the outside of the vehicle 1'. For example, the communication device 3 is a data communication module (DCM) that realizes wide-area wireless communication.

[0076] Server 40 is located outside vehicle 1' and includes a communication interface, storage device, memory, processor, etc. Server 40 may consist of multiple computers. Vehicle 1' can communicate with server 40 via both communication network 50 (such as an operator network or the Internet) and wireless base station 60. Communication between vehicle 1' and wireless base station 60 is performed by known wireless communication technologies (e.g., 3G, LTE, 4G, 5G, etc.).

[0077] In the first embodiment described above, it is assumed that vehicle operation requests similar to query text corresponding to user input exist in the vector database of storage device 14. However, user input to automatic response device 10 is diverse, and it is also assumed that vehicle operation requests similar to query text may not exist.

[0078] Therefore, in the second embodiment, when there is no vehicle operation request similar to the query text, that is, when there is no vehicle signal suitable for the query text, the automatic response device 10 obtains information about the query text from the server 40 and determines the vehicle operation based on the user request according to that information. Thus, the automatic response device 10 can be used to handle user requests other than vehicle operation requests.

[0079] Figure 7 This is a functional block diagram of the processor 13 of the automatic response device 10 according to the second embodiment of the present invention. Figure 7 As shown, in addition to the input processing unit 31, signal extraction unit 32, and operation determination unit 33, the processor 13 also has an information request unit 34. The input processing unit 31, signal extraction unit 32, operation determination unit 33, and information request unit 34 are functional modules implemented by the processor 13 of the automatic response device 10, which executes a computer program stored in the memory 12 of the automatic response device 10. It should be noted that each of these functional modules can be implemented by dedicated arithmetic circuitry provided in the processor 13.

[0080] The information request unit 34 requests information from the server 40. Specifically, when the signal extraction unit 32 does not extract vehicle signals, the information request unit 34 sends a query text to the server 40 and obtains information related to the query text from the server 40. When the signal extraction unit 32 does not extract vehicle signals, the operation determination unit 33 determines vehicle operation based on the information obtained by the information request unit 34.

[0081] Figure 8 This is a flowchart illustrating the control routine for the operation determination process according to a second embodiment of the present invention. This control routine is repeatedly executed by the processor 13 of the automatic response device 10 according to a computer program, for example, stored in the memory 12 of the automatic response device 10.

[0082] Steps S201 to S205 are in conjunction with Figure 5 Steps S101 to S105 are performed in the same manner. However, in step S205, if no vehicle operation request similar to the query text exists, the signal extraction unit 32 does not extract a vehicle signal suitable for the query text. For example, when the similarity between all vehicle operation requests in the vector database and the query text is equal to or less than a predetermined value, the signal extraction unit 32 determines that no vehicle operation request similar to the query text exists. In this case, the signal extraction unit 32 may return an output indicating that no vehicle signal suitable for the query text exists ("No corresponding vehicle signal").

[0083] Following step S205, in step S206, the information request unit 34 determines whether the vehicle signal has been extracted by the signal extraction unit 32. When it is determined that the vehicle signal has been extracted, the control routine proceeds to step S207, which... Figure 5 Step S106 is performed in the same manner.

[0084] On the other hand, when it is determined in step S206 that the vehicle signal has not yet been extracted, the control routine proceeds to step S208. In step S208, the information request unit 34 requests information about the query text from the server 40. Specifically, the information request unit 34 sends the query text before vectorization to the server 40. It should be noted that the query text sent from vehicle 1' to the server 40 can be vectorized by the input processing unit 31.

[0085] Having received the query text, server 40 retrieves information, for example, by inputting the query text into a generative AI model (e.g., an LLM) stored in the memory or storage device of server 40. Typically, server 40 has a higher power consumption tolerance than vehicle 1'. Furthermore, server 40 can use more processors to build expensive generative AI models than the generative AI models provided in vehicle 1'. Therefore, the number of parameters in server 40's generative AI model can be greater than the number of parameters in vehicle 1', thus allowing server 40's generative AI model to output appropriate answers to various query texts. It should be noted that server 40 can obtain information related to the query text by accessing databases external to server 40 using RAG technology.

[0086] Following step S208, in step S209, the information request unit 34 receives information about the query text from the server 40. Next, in step S207, the operation determination unit 33 determines the vehicle operation to be performed in response to the query text based on the information about the query text sent from the server 40 to vehicle 1', instead of the vehicle signal extracted by the signal extraction unit 32. For example, the operation determination unit 33 determines the vehicle operation by using the query text and information about the query text as input, so that the generative AI model outputs a response to the query text. After step S207, the control routine ends.

[0087] The following describes a specific example of how information obtained by server 40 in response to a user request is used to determine vehicle operation. For example, the driver of vehicle 1 enters a voice message saying, "Tell me the average high temperature in Tokyo next week." In this case, since there is no vehicle operation request similar to the query text, no vehicle signal is retrieved. Therefore, the query text is sent from vehicle 1' to server 40, and vehicle operation (e.g., notifying the user of the answer to their question) is determined based on information obtained by server 40 (e.g., weather forecast information for Tokyo).

[0088] <Third Embodiment>

[0089] Except for the points described below, the configuration and control of the automatic response device according to the third embodiment are substantially the same as those according to the first embodiment. Therefore, the third embodiment of the present invention will be described below focusing on the differences from the first embodiment.

[0090] Figure 9 This is a schematic configuration diagram of a vehicle 1" including an automatic response device 10 according to a third embodiment of the present invention. In addition to UI 2 and the automatic response device 10, vehicle 1" also includes an in-vehicle camera 4. The in-vehicle camera 4 captures the interior of vehicle 1" to generate images of the occupants of vehicle 1". The in-vehicle camera 4 is positioned in the passenger compartment such that all seats of vehicle 1" are included in the capture area. For example, the in-vehicle camera 4 is attached near the top edge of the windshield of vehicle 1". The in-vehicle camera 4 is an example of a camera.

[0091] In the third embodiment, the operation determination unit 33 determines the vehicle operation that should be performed in response to the query text based on the vehicle signal extracted by the signal extraction unit 32 and the image generated by the in-vehicle camera 4. Therefore, taking into account the image information, a more appropriate vehicle operation can be selected in response to the user's request.

[0092] Figure 10 This is a flowchart illustrating the control routine for the operation determination process according to a third embodiment of the present invention. This control routine is repeatedly executed by the processor 13 of the automatic response device 10 according to a computer program, for example, stored in the memory 12 of the automatic response device 10.

[0093] Steps S301 to S305 are in accordance with Figure 5 Steps S101 to S105 are performed in the same manner. After step S305, in step S306, the operation determination unit 33 acquires the image generated by the in-vehicle camera 4.

[0094] Next, in step S307, the operation determination unit 33 determines the vehicle operation that should be performed in response to the query text based on the vehicle signals extracted by the signal extraction unit 32 and the image generated by the in-vehicle camera 4. For example, the operation determination unit 33 determines the vehicle operation by using the query text, vehicle signals, and image as input, so that the generative AI model outputs a response to the query text. In other words, in the third embodiment, the operation determination unit 33 uses a multimodal generative AI model (e.g., LLM) that can take text and images as input to determine the vehicle operation.

[0095] The following describes a specific example of using vehicle signals and images to determine vehicle operation in response to a user request. For example, the driver of vehicle 1 inputs the voice command "Open the rear seat window". In this case, "open the window" is extracted as the vehicle operation request most similar to the query text. Therefore, the signal indicating the open / closed state of each window of vehicle 1 is extracted as the vehicle signal associated with this vehicle operation request, and the extracted vehicle signal and the image generated by the in-vehicle camera 4 are input together with the query text into the generative AI model.

[0096] In this context, the generative AI model is typically expected to output answers as follows. For example, when a vehicle signal indicates that the rear seat window is closed and an image indicates that a child is sitting in the rear seat, the generative AI model outputs a vehicle operation to inform the driver that opening the window may pose a danger to the child, or a vehicle operation to partially open the rear seat window.

[0097] It should be noted that the images referenced to determine vehicle operation are not limited to those generated by the in-vehicle camera 4. For example, images generated by cameras installed in vehicle 1” to capture the surrounding environment of vehicle 1”, cameras installed in surrounding vehicles of vehicle 1”, surveillance cameras installed on the road, etc., can be used. When using images generated by cameras installed outside vehicle 1”, the images are sent to vehicle 1” via vehicle-to-vehicle communication, road-to-vehicle communication, or wide area communication.

[0098] <Fourth Embodiment>

[0099] Except for the points described below, the configuration and control of the automatic response device according to the fourth embodiment are substantially the same as those according to the first embodiment. Therefore, the fourth embodiment of the present invention will be described below focusing on the differences from the first embodiment.

[0100] As described above, the operation determination unit 33 uses a generative AI model, such as LLM, to determine vehicle operation. In the generative AI model, many parameters (weights, etc.) of the neural network are determined through prior learning, and an algorithm for determining vehicle operation is determined based on these parameters. Therefore, the operation determination unit 33 determines vehicle operation based on a predetermined algorithm.

[0101] However, users have different preferences, and the learned algorithm used to determine vehicle operation may not necessarily match the preferences of all users. Therefore, in the fourth embodiment, user feedback on vehicle operation determined in response to user input is obtained, and the algorithm used to determine vehicle operation is improved based on the user feedback. As a result, the likelihood of outputting a response that matches the user's preferences when the automatic response device 10 is used can be increased, thereby increasing user satisfaction.

[0102] Figure 11 This is a functional block diagram of the processor 13 of the automatic response device 10 according to the fourth embodiment of the present invention. Figure 11 As shown, in addition to the input processing unit 31, the signal extraction unit 32, and the operation determination unit 33, the processor 13 also has a feedback unit 35. The input processing unit 31, the signal extraction unit 32, the operation determination unit 33, and the feedback unit 35 are functional modules implemented by the processor 13 of the automatic response device 10 executing a computer program stored in the memory 12 of the automatic response device 10. It should be noted that each of these functional modules can be implemented by dedicated arithmetic circuitry provided in the processor 13.

[0103] The feedback unit 35 acquires user feedback on vehicle operation determined by the operation determination unit 33. The feedback unit 35 then improves the algorithm for determining vehicle operation based on the user feedback.

[0104] In the fourth embodiment, besides Figure 5 In addition to the control routines that determine the operation, the execution also includes... Figure 12 The control routine for feedback processing. Figure 12 This is a flowchart illustrating a control routine for feedback processing according to a fourth embodiment of the present invention. This control routine is repeatedly executed by the processor 13 of the automatic response device 10 according to a computer program, for example, stored in the memory 12 of the automatic response device 10.

[0105] First, in step S401, the feedback unit 35 of the processor 13 determines whether the vehicle operation determined by the operation determination unit 33 has already been executed in the vehicle 1. If it is determined that the vehicle operation has not been executed, the control routine ends. On the other hand, if it is determined that the vehicle operation has been executed, the control routine proceeds to step S402.

[0106] In step S402, the feedback unit 35 determines whether user feedback has been received. For example, when the user cancels an already performed vehicle operation, the feedback unit 35 determines that user feedback has been received. As a specific example, after performing a vehicle operation to open the driver's seat window, when the user performs a vehicle operation to close the driver's seat window, the feedback unit 35 determines that user feedback has been received.

[0107] It should be noted that when performing vehicle operations, the feedback unit 35 can present a notification requesting feedback on vehicle operations to the user via the UI 2. In this case, the user provides feedback through input (such as voice input or touch panel operation), and the user feedback is sent from the input device 21 of the UI 2 to the processor 13.

[0108] If it is determined in step S402 that no user feedback has been received, the control routine ends. On the other hand, if it is determined in step S402 that user feedback has been received, the control routine proceeds to step S403.

[0109] In step S403, the feedback unit 35 improves the algorithm used to determine vehicle operation based on user feedback. For example, the feedback unit 35 updates the parameters of the generative AI model used to determine vehicle operation based on user feedback. In this case, the feedback unit 35 uses methods such as reinforcement learning on human feedback (RLHF) to update the parameters of the generative AI model. After step S403, the control routine ends.

[0110] <Fifth Embodiment>

[0111] Except for the points described below, the configuration and control of the automatic response device according to the fifth embodiment are substantially the same as those according to the first embodiment. Therefore, the fifth embodiment of the present invention will be described below focusing on the differences from the first embodiment.

[0112] As described above, the storage device 14 of the automatic response device 10 stores combinations of vehicle operation requests in vector format and vehicle signals linked to those requests. However, there is a risk that simply having predetermined combinations of data may not be sufficient to respond to various requests from the user. Therefore, in the fifth embodiment, in response to a predetermined trigger, at least one of a new vehicle signal and a new vehicle operation request is added to the data in the storage device 14. Thus, user requests that were not initially anticipated can be responded to, thereby improving user satisfaction.

[0113] Figure 13 This is a functional block diagram of the processor 13 of the automatic response device 10 according to the fifth embodiment of the present invention. Figure 13 As shown, in addition to the input processing unit 31, signal extraction unit 32, and operation determination unit 33, the processor 13 also has a data management unit 36. The input processing unit 31, signal extraction unit 32, operation determination unit 33, and data management unit 36 ​​are functional modules implemented by the processor 13 of the automatic response device 10 executing a computer program stored in the memory 12 of the automatic response device 10. It should be noted that each of these functional modules can be implemented by dedicated arithmetic circuitry provided in the processor 13.

[0114] The data management unit 36 ​​manages the data stored in the storage device 14, and specifically, manages data consisting of combinations of vehicle operation requirements and vehicle signals. For example, in response to a predetermined trigger, the data management unit 36 ​​adds at least one of a new vehicle signal and a new vehicle operation requirement to the data.

[0115] In the fifth embodiment, besides Figure 5In addition to the control routines that determine the operation, the execution also includes... Figure 14 The control routine for data update processing. Figure 14 This is a flowchart illustrating a control routine for data update processing according to a fifth embodiment of the present invention. This control routine is repeatedly executed by the processor 13 of the automatic response device 10 according to a computer program, for example, stored in the memory 12 of the automatic response device 10.

[0116] First, in step S501, the data management unit 36 ​​of the processor 13 determines whether a predetermined trigger has occurred. A predetermined trigger is, for example, a software update for the vehicle 1. In this case, when the software of the vehicle 1 is updated via OTA (Over-The-Air) or the like, the data management unit 36 ​​determines that a predetermined trigger has occurred. It should be noted that a predetermined trigger can also be a user's data addition request. In this case, when the user requests to add data via the input device 21 of the UI 2, the data management unit 36 ​​determines that a predetermined trigger has occurred.

[0117] If it is determined in step S501 that no predetermined trigger has occurred, the control routine ends. On the other hand, if it is determined in step S501 that a predetermined trigger has occurred, the control routine proceeds to step S502.

[0118] In step S502, the data management unit 36 ​​updates the data stored in the storage device 14. Specifically, the data management unit 36 ​​adds at least one of a new vehicle signal and a new vehicle operation request to the data. When a vehicle operation request is added, the data management unit 36 ​​vectorizes the vehicle operation request and adds the new vehicle operation request in vector format to the data. When only a vehicle operation request is added, the new vehicle operation request is linked to at least one existing vehicle signal. On the other hand, when only a vehicle signal is added, the new vehicle signal is linked to at least one existing vehicle operation request.

[0119] When the scheduled trigger is a software update for vehicle 1, the data management unit 36 ​​updates the data according to, for example, the software update procedure. When the scheduled trigger is a user's data addition request, the data management unit 36 ​​updates the data according to, for example, the content input by the user into the input device 21. When a user requests to add a new vehicle operation requirement, the data management unit 36 ​​can use a generative AI model such as LLM to determine the vehicle signal linked to the new vehicle operation requirement.

[0120] After step S502, the control routine ends. It should be noted that the predetermined trigger could be, for example, the download of an application developed by the manufacturer of vehicle 1 or a third party of vehicle 1.

[0121] <Other Embodiments>

[0122] While preferred embodiments of the invention have been described above, the invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the claims. For example, the operation determination unit 33 may use algorithms other than generative AI models to determine the vehicle operation that should be performed in response to the query text.

[0123] Furthermore, in the first, third, fourth, or fifth embodiments, a server or the like, located outside the vehicles 1 and 1” and capable of communicating with the vehicles 1 and 1”, can be used as the automatic response device 10. In this case, for example, the server's storage device is used as a storage unit, and the server's processor is used as an input processing unit 31, a signal extraction unit 32, an operation determination unit 33, a feedback unit 35, and a data management unit 36, and the information required for the operation of these functional modules (e.g., user input) is sent from the vehicles 1 and 1” to the server.

[0124] The computer program that enables the computer to perform the functions of each unit of the processor 13 of the automatic response device 10 may be provided in the form of being stored in a computer-readable recording medium or in the form of being included in a computer program product. The computer-readable recording medium may be, for example, a magnetic recording medium, an optical recording medium, or a semiconductor memory.

[0125] The second to fifth embodiments can be implemented in any combination. For example, when the second and third embodiments are combined, in Figure 8 In the control routine, execute Figure 10 Steps S306 and S307 replace step S207. Furthermore, when the fourth or fifth embodiment is combined with the second embodiment, the following steps are performed: Figure 8 Control routines replace Figure 5 The control routine serves as the control routine for operation determination processing. Similarly, when the fourth or fifth embodiment is combined with the third embodiment, the following is executed: Figure 10 Control routines replace Figure 5 The control routine serves as the control routine for operation determination processing. Furthermore, all embodiments of the second to fifth embodiments can be implemented in combination.

[0126] List of reference numerals

[0127] 10 Automatic Response Devices

[0128] 13 processor

[0129] 14 storage devices

[0130] 31 Input Processing Unit

[0131] 32 Signal Extraction Unit

Claims

1. An automatic response device, comprising: A storage unit, wherein the storage unit is used to store each of a plurality of vehicle signals linked to vehicle operation requirements in a vector format; An input processing unit is configured to vectorize query text corresponding to input from a user. as well as The signal extraction unit is configured to extract vehicle signals suitable for the query text from the plurality of vehicle signals by comparing the query text vectorized by the input processing unit with the vehicle operation requirements.

2. The automatic response device according to claim 1, wherein, In the storage unit, two or more vehicle signals are linked to a vehicle operation requirement.

3. The automatic response device according to claim 1 or 2 further includes an information request unit, the information request unit being configured to request information from a server, wherein... The automatic response device is installed inside the vehicle, and the server is located outside the vehicle. The information request unit is configured to send the query text to the server when the signal extraction unit fails to extract the vehicle signal, and to obtain information related to the query text from the server.

4. The automatic response device according to claim 3, further comprising an operation determination unit configured to determine a vehicle operation to be performed in response to the query text based on the vehicle signal extracted by the signal extraction unit, wherein... The operation determination unit is configured to determine the vehicle operation based on the information when the signal extraction unit fails to extract the vehicle signal.

5. The automatic response device according to claim 1 or 2 further includes an operation determination unit configured to determine a vehicle operation that should be performed in response to the query text based on the vehicle signal extracted by the signal extraction unit.

6. The automatic response device according to claim 4 or 5, further comprising a feedback unit configured to acquire user feedback on vehicle operation, wherein... The operation determination unit is configured to determine the vehicle operation based on a predetermined algorithm, and The feedback unit is configured to improve the algorithm based on the user's feedback.

7. The automatic response device according to any one of claims 4 to 6, wherein, The operation determination unit is configured to determine the vehicle operation based on the vehicle signal extracted by the signal extraction unit and the image generated by the camera.

8. The automatic response device according to any one of claims 1 to 7, further comprising a data management unit configured to manage data stored in the storage unit, wherein... The data management unit is configured to add at least one of a new vehicle signal and a new vehicle operation request to the data in response to a predetermined trigger.

9. An automatic response method executed by a computer, comprising: Each vehicle signal among multiple vehicle signals linked to vehicle operation requirements is stored in vector format; Vectorize the query text corresponding to the user's input; as well as By comparing the vectorized query text with the vehicle operation requirements, vehicle signals suitable for the query text are extracted from the plurality of vehicle signals.

10. A computer program product comprising a computer program, said computer program causing a computer to: Each vehicle signal among multiple vehicle signals linked to vehicle operation requirements is stored in vector format; Vectorize the query text corresponding to the user's input; and By comparing the vectorized query text with the vehicle operation requirements, vehicle signals suitable for the query text are extracted from the plurality of vehicle signals.

Images