Information processing apparatus

Abstract

An information processing apparatus decides one or more external agents, which are specialized for a plurality of prescribed fields and provide a natural language-based dialogue service, as a transmission destination of a speech sentence of a user's speech from among the external agents, transmits the speech sentence to the decided external agent, acquires one or more responses from the external agent and integrates the responses, and provides an integrated response to the user. In addition, the external agent is decided as a transmission destination of a speech sentence based on a score set for each of a plurality of external agents, or the integrated response is decided based on the score, and the score corresponding to at least any one of the one or more external agents is updated based on the user's action after the integrated response is provided.

Description

Technical Field

[0001] This disclosure relates to dialogue technology. Background Technology

[0002] There are techniques for conducting natural language-based dialogues based on input articles. For example, International Publication No. 2019 / 225201 discloses a device capable of conversing with multiple agents via a network.

[0003] In the future, with the development of machine learning, it is believed that dialogue services based on natural language will further increase. Summary of the Invention

[0004] The purpose of this disclosure is to improve the accuracy of dialogues using machine learning models.

[0005] One aspect of this disclosure is an information processing apparatus having a control unit that performs the following processing: obtaining a speech statement based on a user's speech; determining, based on the speech statement, one or more external agents from among multiple external agents, each dedicated to multiple defined domains and providing natural language-based dialogue services, as the destination for sending the speech statement; sending the speech statement to the determined external agents and obtaining one or more responses from the external agents; and integrating the responses and providing the integrated response to the user, wherein the control unit determines the external agents as the destination for sending the speech statement based on scores set for each of the multiple external agents, or determines the integrated object among the responses based on the scores, and updates the score corresponding to at least one of the external agents based on the user's action after providing the integrated response.

[0006] Alternatively, a program for causing a computer to perform the method performed by the aforementioned apparatus, or a computer-readable storage medium that non-temporarily stores the program, can be cited as an example.

[0007] According to this disclosure, the accuracy of dialogues utilizing machine learning models can be improved. Attached Figure Description

[0008] Hereinafter, the features, advantages, and technical and industrial significance of exemplary embodiments of the present invention will be described with reference to the accompanying drawings, in which the same reference numerals denote the same elements, wherein:

[0009] Figure 1A This is a schematic diagram of the dialogue system in the first embodiment;

[0010] Figure 1BThis is a schematic diagram of the dialogue system according to the first embodiment;

[0011] Figure 1C This is a schematic diagram of the dialogue system according to the first embodiment;

[0012] Figure 2 This is a hardware structure diagram of the vehicle-mounted device 10;

[0013] Figure 3 This is a software structure diagram of the vehicle-mounted device 10;

[0014] Figure 4 This is an example of fractional data used by the vehicle-mounted device 10;

[0015] Figure 5 This is a diagram illustrating the process flow performed by the vehicle-mounted device 10 in the first embodiment;

[0016] Figure 6A This is an example of text input and output relative to an external intelligent agent;

[0017] Figure 6B This is an example of text input and output relative to an external intelligent agent;

[0018] Figure 6C This is an example of text input and output relative to an external intelligent agent;

[0019] Figure 7A This is an example of text input and output relative to the local agent;

[0020] Figure 7B This is an example of text input and output relative to the local agent;

[0021] Figure 8 This is a flowchart of the process performed by the vehicle-mounted device 10 in the first embodiment;

[0022] Figure 9 This is a diagram illustrating the process executed by the vehicle-mounted device 10 in the third embodiment;

[0023] Figure 10 This is a flowchart of the process performed by the vehicle-mounted device 10 in the third embodiment;

[0024] Figure 11 This is a diagram illustrating the process flow performed by the vehicle-mounted device 10 in the fourth embodiment; and

[0025] Figure 12 This is a flowchart of the process performed by the vehicle-mounted device 10 in the fourth embodiment. Detailed Implementation

[0026] In recent years, with the development of machine learning, the number of products equipped with language models has been increasing. For example, by incorporating a large-scale language model (LLM), which improves accuracy through machine learning using large-scale datasets, into a product, it is possible to add natural language-based dialogue functionality to the product.

[0027] One area where natural language conversations come into play is the automotive industry. For example, by incorporating LLM (Limited Language Management) into in-vehicle devices, information can be obtained without operating touch panels. This functionality is particularly useful when hands are occupied, such as while driving.

[0028] On the other hand, the large data size of the learned LLM model makes it difficult to integrate into embedded devices due to cost considerations. Therefore, a method is proposed that instead of integrating the LLM into the device, access the LLM via a network. By accessing the LLM over a network, it is possible to utilize large-scale language models that are difficult to integrate into local memory.

[0029] Furthermore, via the network, it is possible to selectively access LLMs dedicated to specific domains. For example, when multiple LLMs are prepared for various domains such as LLMs dedicated to route guidance, LLMs dedicated to location information guidance, and LLMs dedicated to social dialogue, it is possible to utilize the LLM that matches the user's intended purpose. In this regard, there are known technologies that determine the category of a user's speech based on its content and automatically select an LLM that matches that category.

[0030] However, when choosing an LLM to conduct dialogue from multiple domains, there is a problem that dialogue can only be conducted in a specific domain.

[0031] For example, if a user's speech includes something like "about the car's warning lights," and they have selected an LLM specifically for vehicle operation, it's conceivable that they could respond by providing information related to the meaning of the warning lights. However, an LLM specifically for vehicle operation lacks location-specific information and therefore might not be able to provide further services based on the speech content, such as "directing you to the nearest car dealership for maintenance."

[0032] As a solution to this problem, consider sending the content (speech statements) of user speech to multiple LLMs in different domains, integrating the resulting responses and providing them to the user.

[0033] However, sending speech statements unconditionally to multiple LLMs may lead to other problems, such as providing responses from LLMs that do not conform to the user's intent or requirements.

[0034] The information processing apparatus disclosed herein addresses such a problem.

[0035] One aspect of this disclosure relates to an information processing apparatus having a control unit that performs the following processing: obtaining a speech statement based on a user's speech; determining, based on the speech statement, one or more external agents from among multiple external agents that are respectively dedicated to multiple defined domains and provide natural language-based dialogue services as the destination of the speech statement; sending the speech statement to the determined external agents and obtaining one or more responses from the external agents; and integrating the responses and providing the integrated response to the user, wherein the control unit determines the external agents as the destination of the speech statement based on scores set for each of the multiple external agents, or determines the integrated object among the responses based on the scores, and updates the score corresponding to at least one of the external agents based on the user's action after providing the integrated response.

[0036] External agents are systems capable of providing dialogue services using language models that are specialized for multiple domains and can engage in natural language-based conversations. These language models can be large-scale language models, such as those learned to perform natural language-based dialogue tasks. Examples of multiple domains include "route guidance," "location information guidance," "vehicle operation instructions," "tourist guidance," and "social dialogue." These agents are typically configured to be remotely accessible via a network.

[0037] The control unit determines one or more external agents as the destination for the user's speech (speech statement), and sends the speech statement to the determined external agent to obtain a response. The control unit can also send a prompt text containing the user's speech (speech statement) to the determined external agent via a network. Thus, it is possible to obtain one or more responses from the aforementioned external agents.

[0038] In addition, the control unit integrates more than one response obtained from various external agents and provides it to the user. Response integration can be based on a rule base or can utilize other language models. For example, response integration can also be achieved by inputting the responses obtained from various external agents into a locally running agent with a language model (the local agent).

[0039] The language model of the local agent is a language model capable of performing natural language-based dialogue tasks, but it can be configured to be a language model generated at a relatively low cost without performing domain-specific learning.

[0040] Based on this structure, information can be obtained from multiple LLMs with various professional fields and integrated to provide it to users.

[0041] In the above examples, for instance, by integrating responses from an LLM dedicated to vehicle operation guidelines and responses from an LLM dedicated to location information guidance, it is possible to interpret the meaning of the warning light and provide a message directing you to a service point that can resolve the issue.

[0042] On the other hand, when deciding which external agent to send the speech, sometimes an external agent is chosen that is different from the user's intention.

[0043] For example, if there is an external agent belonging to the fields of "current affairs," "economy," "entertainment," "sports," "food," or "art," or if there is an external agent belonging to the field that does not match the user's preferences (e.g., "sports"), it is preferable not to actively select that external agent. Alternatively, even if selected, it is preferable not to actively provide the user with a response from that external agent.

[0044] Therefore, the information processing apparatus of this disclosure uses a score set for each external agent to determine one or more external agents as the destination for the spoken statement. Alternatively, it determines the integrated object in the one or more responses based on the score.

[0045] Furthermore, the information processing device updates the score corresponding to at least one of the more than one external agents based on the user's actions following the provision of the integrated response.

[0046] The information processing device obtains a score set for each external agent and uses this score to integrate the decisions and responses of the external agents.

[0047] Scores can also be set for each user. For example, a user can assign higher scores to LLMs they want to prioritize and lower scores to LLMs they want to avoid. This allows, for example, avoiding sending messages to LLMs that don't suit the user's preferences, or excluding responses from those LLMs from the integration process.

[0048] Furthermore, the score is updated based on the user's actions after the response is provided. For example, if the user acts according to the suggestions of an LLM, the LLM's score may be updated to a larger value. Conversely, if the user ignores the suggestions of an LLM, the LLM's score may be updated to a smaller value.

[0049] This structure can prevent information from LLMs that do not align with the user's preferences from being provided to the user.

[0050] Alternatively, the control unit may also acquire feedback data representing the user's actions after providing the integrated response, and update the score based on the feedback data.

[0051] Feedback data can also include information related to the user's actions, information indicating the destination visited, information indicating the content of speech, search history, information indicating gestures, etc.

[0052] For example, if the feedback data indicates that the user's response to the first external agent is positive, the control unit increases the score for the first external agent; if the feedback data indicates that the user's response to the first external agent is negative, the control unit decreases the score for the first external agent.

[0053] The following description, based on the accompanying drawings, details specific embodiments of this disclosure. Unless otherwise specified, the hardware structures, module structures, and functional structures described in each embodiment are not intended to limit the scope of the disclosed technology to them.

[0054] First Implementation Method

[0055] System Overview

[0056] An overview of the dialogue system according to the first embodiment will be described. The dialogue system of this embodiment is configured to include an in-vehicle device 10 mounted in a vehicle. The in-vehicle device 10 is able to access an intelligent agent (external intelligent agent) that provides natural language-based dialogue services via a network (e.g., a mobile communication network).

[0057] The vehicle-mounted device 10 is mounted on a networked vehicle capable of communicating with any device via wireless communication. The vehicle-mounted device 10 may also include a data communication module (DCM) for connecting vehicle components (e.g., ECU, vehicle terminal, etc.) to a network. In this embodiment, the vehicle-mounted device 10 can access the Internet via a predetermined mobile communication network and connect with intelligent agents providing dialogue services.

[0058] Furthermore, the vehicle (or components mounted on the vehicle) can provide various services by communicating with external devices via the on-board unit 10. Examples of these services include, for instance, navigation services, remote control services (e.g., remote air conditioning), in-vehicle Wi-Fi services, and emergency notification services.

[0059] Furthermore, the vehicle-mounted device 10 has voice input and output functions, enabling it to engage in natural language-based dialogue with the occupants of the vehicle.

[0060] Figure 1A This is a schematic diagram illustrating the existing vehicle-mounted device 10.

[0061] The illustrated vehicle-mounted device 10 is configured to include a control unit for sound input / output and sound recognition, and a local intelligent agent. The local intelligent agent contains a language model learned in a manner capable of engaging in natural language-based dialogue. The local intelligent agent can, for example, provide prescribed functions such as path search and vehicle operation (air conditioning, etc.) via dialogue. In the illustrated configuration, the language model used by the local intelligent agent is stored in a storage device within the vehicle-mounted device 10.

[0062] On the other hand, this approach makes it difficult to provide dialogue services based on large-scale language models. For example, dialogues encompass a wide variety of categories, such as location information, tourist guidance, vehicle-related information (operation guides), and social conversations. However, to cover all these categories, a language model trained on massive amounts of data is required. Such a language model can range from several gigabytes to hundreds of gigabytes in size, making its storage in local memory impractical from a cost perspective.

[0063] Therefore, a system for accessing domain-specific language models for dialogue via a network was proposed.

[0064] Figure 1B This is a schematic diagram illustrating how the vehicle-mounted device 10 can access multiple language models available on the Internet in addition to the local intelligent agent. For example, consider the following approach: an intelligent agent (external intelligent agent) with multiple large-scale language models dedicated to a specific domain is configured in a device connected to the Internet, and the vehicle-mounted device 10 accesses the language model selected according to the request.

[0065] However, this approach has a limitation: it restricts conversations to specific domains. For example, if a user wants information about a particular sport, they might converse with an agent dedicated to that sport. However, this agent cannot provide information outside the sport, such as facility details, and therefore cannot provide responses like introducing venues where the sport can be watched.

[0066] Therefore, in the vehicle-mounted device 10 according to this embodiment, the local agent has the function of integrating responses from multiple external agents. The user's speech content (speech statements) is sent to multiple external agents, and responses are obtained from these multiple external agents. These responses are integrated in the local agent, and the integrated response is provided to the user. Figure 1C This is a schematic diagram of the vehicle-mounted device 10 according to this embodiment.

[0067] When the vehicle-mounted device 10 recognizes a user's speech, it determines "what external agent matches the speech" based on the content. This processing can be based on a rule base or a machine learning model. Furthermore, the control unit sends the speech to one or more of the determined external agents.

[0068] The responses from the selected external agent are integrated by the local agent and provided to the user.

[0069] This enables the user to be appropriately provided with responses obtained from multiple external agents.

[0070] On the other hand, in a model that integrates multiple responses from multiple external agents, a new problem may arise: unwanted information may be mixed in. For example, the following situation may occur: the external agent to be used is selected based on the content of the speech, but the result is that an external agent belonging to a domain that does not conform to the user's preferences is selected.

[0071] To address this situation, in this embodiment, the vehicle-mounted device 10 sets a score for each user regarding the external intelligent agent and controls information collection based on the set score. Thus, for example, it is possible to take measures such as "stopping the use of external intelligent agents that belong to areas that do not conform to the user's preferences."

[0072] Furthermore, in the description of this embodiment, terms such as access to external intelligent agents and connection to external intelligent agents mean access to or connection to external devices that provide dialogue services to external intelligent agents. In this embodiment, among multiple external devices, multiple dialogue services based on a large-scale language model operate as "external intelligent agents," and the vehicle-mounted device 10 can engage in dialogue by connecting to any of the external intelligent agents.

[0073] Hardware Structure

[0074] Next, the hardware structure of each device that makes up the system will be described. Figure 2 This diagram illustrates an example of the hardware structure of the vehicle-mounted device 10 according to this embodiment.

[0075] The vehicle-mounted device 10 can be configured as a computer with a processor (CPU, GPU, etc.), main storage (RAM, ROM, etc.), and auxiliary storage (EPROM, hard disk drive, removable media, etc.). The auxiliary storage stores the operating system (OS), various programs, various tables, etc., and by executing the programs stored therein, various functions (software modules) that meet a predetermined purpose can be achieved, as described later. However, some or all of the functions can also be implemented as hardware modules using hardware circuits such as ASICs or FPGAs.

[0076] The vehicle-mounted device 10 is configured to have a control unit 11, a storage unit 12, a wireless communication module 13, and an input / output unit 14 as hardware.

[0077] The control unit 11 is a computing unit that executes prescribed programs to implement various functions of the vehicle-mounted device 10. The control unit 11 can be implemented, for example, by a hardware processor such as a CPU. Alternatively, the control unit 11 may be configured to include RAM, ROM (Read Only Memory), cache memory, etc.

[0078] The storage unit 12 is a unit for storing information and is composed of storage media such as RAM, disk, and flash memory. The storage unit 12 stores the program executed by the control unit 11 and the data used by the program.

[0079] The wireless communication module 13 is a communication device that communicates wirelessly with a specified network. In this embodiment, the wireless communication module 13 is configured to communicate with a specified mobile communication network. The wireless communication module 13 may be configured to have an eUICC (e.g., a SIM card). The SIM card is configured as a microcomputer with a CPU and a storage device. The SIM card is connected to the mobile communication network and stores information for accepting authentication.

[0080] The input / output unit 14 is a unit that receives input from the user of the device and provides prompts to the user. The input / output unit 14 is typically configured to include devices for inputting and outputting sound, such as a microphone and a speaker. Alternatively, the input / output unit 14 may also include devices that provide visual information (such as a display).

[0081] Software Structure

[0082] Next, the software structure of each device that makes up the system will be explained. Figure 3 This is a diagram schematically illustrating the software structure of the vehicle-mounted device 10 according to this embodiment.

[0083] In this embodiment, the control unit 11 of the vehicle-mounted device 10 is configured with three software modules: a dialogue receiving unit 111, an intelligent agent unit 112, and a route guidance unit 113. Each software module can also be implemented by the control unit 11 (CPU, etc.) executing a program stored in the storage unit 12. Furthermore, information processing performed by the software modules is synonymous with information processing performed by the control unit 11 (CPU, etc.).

[0084] The dialogue receiving unit 111 acquires the speech given by the vehicle occupants (hereinafter also referred to as users) via the input / output unit 14. The dialogue receiving unit 111 performs prescribed processing on the acquired voice data, performing speech recognition. As a result, the content of the speech is converted into text. Furthermore, the dialogue receiving unit 111 sends the text obtained as a result of speech recognition to the intelligent agent unit 112.

[0085] Additionally, the dialogue receiving unit 111 outputs the response (hereinafter, the answer statement) sent from the language model by the intelligent agent unit 112. The dialogue receiving unit 111 converts the answer statement into speech and outputs it via the input / output unit 14.

[0086] The agent unit 112 first provides a dialogue service utilizing a language model stored in the device (the local language model 12A described later). The agent unit 112 can operate as a virtual agent (local agent). The local agent can, for example, provide dialogue services completed within the device (e.g., social dialogue).

[0087] Furthermore, when the intelligent agent unit 112 requires specialized knowledge during a dialogue, it sends spoken statements to one or more available external intelligent agents via the network to obtain the results. For example, if it is determined that the user is seeking information related to a vehicle, the intelligent agent unit 112 selects an external intelligent agent that can provide vehicle operation instructions. Additionally, if it is determined that the user is seeking information related to facilities or shops, the intelligent agent unit 112 selects an external intelligent agent that can provide location information.

[0088] The selection of external agents can also be based on the results of parsing the content of the speech. For example, agent unit 112 can calculate the fitness of each of the multiple external agents based on the words and context contained in the speech, and send the speech to the external agent that has obtained a fitness value of above a specified value.

[0089] The agent unit 112 may also select multiple external agents to send the speech statement. For example, if there are multiple external agents whose suitability for the speech statement is above a predetermined value, the agent unit 112 may send the speech statement to these multiple external agents. In this case, the agent unit 112 obtains the response statements to the speech statement from the multiple external agents, integrates these response statements, and provides them to the user.

[0090] Furthermore, when selecting external agents, the intelligent agent unit 112 utilizes score data pre-set by each user. The score data represents the priority of each of the multiple external agents.

[0091] Figure 4 This is an example of score data. In this embodiment, score data consists of a combination of the domain of the external agent and scores. A score is a numerical value that quantifies the degree of a user's interests or preferences. In the illustrated example, it indicates that user A has a strong interest in current events and sports, but is not very interested in art.

[0092] In this embodiment, the agent unit 112 excludes external agents from the selected objects that correspond to domains with scores below a predetermined value. That is, for external agents in domains with scores less than a predetermined value (e.g., 1.0), no speech statement is sent.

[0093] This allows the system to suppress responses that are not of interest to the user.

[0094] return Figure 3 Let me continue explaining.

[0095] The route guidance unit 113 receives the user's specified origin and destination, and generates a route connecting the origin and destination based on road map data, etc. Furthermore, it outputs information related to the generated route via the input / output unit 14.

[0096] The storage unit 12 of the vehicle-mounted device 10 stores a local language model 12A, agent information 12B, attribute data 12C, and score data 12D.

[0097] The local language model 12A is a language model that has been learned in a way that enables it to perform natural language-based dialogue tasks. Local language model 12A is a language model generated at a relatively low cost and has not undergone learning for domain-specific dialogue. Local language model 12A is sufficient for general conversational use. For example, it can also utilize open-source language models as local language model 12A.

[0098] The local language model 12A is a lighter-weight language model than the large-scale language models possessed by external agents accessible via a network. By utilizing the local language model 12A, agent unit 112 can engage in responsive dialogue. For example, usage scenarios can be differentiated: when specialized conversations are required, the spoken statements are sent to an external agent; when a high-speed response is needed, the local agent processes the spoken statements.

[0099] Agent information 12B is a collection of data related to multiple external agents that can be utilized by the vehicle-mounted device 10. Agent information 12B may include, for example, the identifier and name of the external agent, information related to a specific domain, and information related to the destination (e.g., providing the network address of the external device of the external agent). By referring to agent information 12B, agent unit 112 can determine the external agent that will be the destination for the spoken statement.

[0100] Attribute data 12C is a collection of data related to the attributes of the user interacting with the in-vehicle device 10. Attribute data 12C may include, for example, personal information such as the user's gender, age, height, and weight. Furthermore, if there are multiple users in the vehicle, attribute data 12C may also contain data for multiple individuals.

[0101] For reference Figure 4 As explained, Score Data 12D represents the degree of matching between the domain expertise of multiple external agents and the target user. Score Data 12D is defined for each of the multiple users who may be taking the ride.

[0102] Dialogue process

[0103] Next, a summary of the processes performed by the control unit 11 will be explained. Figure 5 This is a diagram illustrating the processing flow performed by the control unit 11 of the vehicle-mounted device 10 from the user receiving the speech.

[0104] The dialogue receiving unit 111 acquires speech from the user via the input / output unit 14. For example, the input / output unit 14 converts speech acquired via a microphone or the like into voice data, which the dialogue receiving unit 111 then acquires. The dialogue receiving unit 111 performs prescribed speech recognition processing on the acquired voice data, converting it into text. While retaining the response to the speech, the dialogue receiving unit 111 sends the text obtained through speech recognition to the intelligent agent unit 112. This text is referred to as the "speech statement".

[0105] Upon receiving the spoken statement, agent 112 determines the destination of the statement—the external agent—based on the content of the speech, after judging the speaker's intent (what the user seeks) and the user's current state (what they are currently doing). The speaker's intent can be determined using the local language model 12A, or other machine learning models. Alternatively, it can be determined based on a rule base.

[0106] Furthermore, the user's status can also be determined using sensor data other than speech. For example, the intelligent agent unit 112 can also determine the user's status using data related to vehicle speed, current location, and direction of travel obtained from sensors on the vehicle. Examples of user status could include "moving on a regular road," "moving on a highway," "parked," "waiting for a traffic light," "moving towards the workplace," and "going home."

[0107] The agent unit 112 obtains information (agent information 12B) related to external agents from the storage unit 12, and determines the external agent to which the speech statement should be sent based on the agent information 12B, the determined intent, and the situation. For example, the agent unit 112 may also calculate the fitness level for each of multiple external agents and determine to send the speech statement to the external agent that has obtained a fitness level of a predetermined value or higher.

[0108] In this example, the following three are shown as external intelligent agents.

[0109] (1) Vehicle intelligent agent

[0110] An external intelligent agent capable of monitoring the vehicle's interior and configuring its devices. It has already studied the vehicle's owner's manual and can provide information about the vehicle's functions.

[0111] (2) Social dialogue intelligent agent

[0112] External intelligent agents specifically designed for non-task-oriented social conversations.

[0113] (3) Regional Information Intelligent Agent

[0114] An external intelligent agent capable of providing information about the surrounding area, including store directions and sightseeing guides.

[0115] In this example, the vehicle agent and the local information agent were selected as the external agents that correspond to the speech.

[0116] Furthermore, the agent unit 112 may sometimes determine that it is unnecessary to send a speech statement to an external agent. For example, this could be done by the device itself, such as when the user requests a path or requires a high-speed response. In such cases, the speech statement is processed solely by the agent unit 112 (and the local language model 12A).

[0117] Next, the intelligent agent unit 112 filters the selected external intelligent agents based on the user's preferences. Based on the score data 12D, the intelligent agent unit 112 excludes external intelligent agents in the domain with scores below a predetermined value from the selected objects.

[0118] Then, agent unit 112 generates prompt text to be input to the selected external agent. Figure 6A This is an example of a prompt text input from agent 112 to an external agent.

[0119] In addition to the spoken statements, the prompt text also includes background information relevant to the user.

[0120] In this embodiment, background information is information attached to the spoken statements and is related to the user's attribute information and the user's movement. Background information may include, for example, personal information such as the user's gender, age, height, and weight. This information can also be obtained from attribute data 12C stored in storage unit 12.

[0121] In addition, background information may include information related to the user's movement. Such information may include details related to the vehicle's origin, destination, route, and transit points. This information can also be obtained from the route guidance unit 113.

[0122] External agents can then generate responses based on this background information.

[0123] The intelligent agent 112 sends the generated prompt text to the external device of the external intelligent agent that provides the target via the network.

[0124] The external agent that has been given the spoken statement responds to the spoken statement by outputting a response. Figure 6B This is an example of a response from a vehicle's intelligent agent. Figure 6C This is an example of a response from a local information agent. As shown in the figure, each agent outputs a response relevant to its area of ​​expertise. For example, the vehicle agent outputs information related to the specific meaning of a warning light, and the area information agent outputs information about the location where the vehicle can be repaired.

[0125] When agent 112 receives responses from an external agent that sent the spoken statement, it integrates these responses. The integration of responses can be performed using the local language model 12A.

[0126] For example, agent 112 inputs a prompt text containing one or more received responses into the local language model 12A to integrate the responses. This prompt text may include, for example, statements made by the user, responses obtained from multiple external agents, and instructions to integrate the responses to generate a new text. Figure 7A This is an example of the prompt text input into the local language model 12A.

[0127] As a result, the integrated response statement is output from the local language model 12A. Figure 7B This is an example of a response statement obtained as a result of integration. The integrated response statement is sent from the agent unit 112 to the dialogue receiving unit 111.

[0128] When the dialogue receiving unit 111 receives a response statement, it generates voice data based on the response statement and outputs it via the input / output unit 14. Furthermore, the dialogue receiving unit 111 may generate only the voice data of the response statement, or it may generate information accompanying the voice data. For example, a user interface screen may be displayed. This user interface screen may also contain the response statement written in text.

[0129] The dialogue system in this embodiment operates seamlessly. Figure 5 The series of processes shown are as follows. Thus, when observed from the perspective of a user who has spoken, it is possible to achieve an effect similar to having a conversation with the vehicle-mounted device 10.

[0130] flow chart

[0131] Next, the details of the processing performed by the vehicle-mounted device 10 will be explained. Figure 8 This is a flowchart of the processing performed by the vehicle-mounted device 10. Figure 8 The process shown begins upon detection of a user's (vehicle occupant's) speech. The start of a speech can be detected, for example, by predefined keywords.

[0132] First, in step S11, the dialogue receiving unit 111 identifies the content of the speech. The dialogue receiving unit 111 acquires the voice data output from the input / output unit 14, performs speech recognition processing, and thereby converts the speech into text. The converted text (speech statement) is sent to the intelligent agent unit 112.

[0133] Next, in step S12, the intelligent agent 112 acquires the score data corresponding to the user who spoke. The user who spoke can be identified based on voice, and if the vehicle is equipped with a personal authentication device (e.g., a facial recognition device or a fingerprint authentication device), identification can also be based on the authentication result.

[0134] In step S13, the intelligent agent unit 112 determines the external intelligent agent that will send the speech statement. The external intelligent agent that will send the speech statement may, for example, infer the user's intention (what the user requests) and situation (what the user is currently doing) based on the speech statement, and make a decision based on the result.

[0135] For example, the agent unit 112 may also read information (agent information 12B) related to the characteristics of the external agent from the storage unit 12, and decide on the external agent to be utilized based on the information.

[0136] Furthermore, as described above, the agent unit 112 filters external agents that send speech statements based on score data. Based on score data 12D, the agent unit 112 excludes external agents in the selected field that have scores below a predetermined value.

[0137] Next, in step S14, the agent unit 112 obtains the user's background information. If the background information is related to the user's movement, this information can also be obtained from the path guidance unit 113. Alternatively, if the background information is the user's attribute information, this information (attribute data 12C) can also be obtained from the storage unit 12.

[0138] In step S15, the intelligent agent unit 112 sends a speech statement and background information to the external intelligent agent determined in step S13. For example, the intelligent agent unit 112 accesses an external device that provides services to the external intelligent agent via the wireless communication module 13 and sends the speech statement and background information to that external intelligent agent.

[0139] In step S16, the agent unit 112 obtains the response corresponding to the spoken statement from the external agent of the object.

[0140] In step S17, the agent unit 112 uses the local language model 12A to integrate these responses. Specifically, the agent unit 112 generates a prompt text containing a list of responses obtained from multiple external agents and an instruction expressing the intention to integrate these responses, and inputs the prompt text into the local language model 12A. Additionally, it obtains the integrated response statement output from the local language model 12A.

[0141] In step S18, the intelligent agent unit 112 sends the response statement obtained as a result of integration to the dialogue receiving unit 111. The dialogue receiving unit 111 generates voice data based on the response statement and outputs it via the input / output unit 14. The dialogue receiving unit 111 may also convert the response statement into voice data using speech synthesis technology. The voice data is output via the input / output unit 14 (speaker, etc.) and provided to the user.

[0142] As explained above, the vehicle-mounted device 10 of this embodiment determines which external agent the local agent should send the content of the vehicle occupants' speech to. The local agent integrates and outputs the response statements obtained from the external agents. Thus, it is possible to provide the user with response content from multiple external agents handling highly specialized topics.

[0143] Furthermore, by using the user's personal preferences to filter the external agents that send the speech, responses from areas that are not of interest to the user can be excluded.

[0144] Second Implementation Method

[0145] In the first embodiment, when selecting an external agent in step S13, score data is used to exclude external agents that are unsuitable for the user. Alternatively, even if a speech statement is sent to an external agent without considering the user's preferences, responses unsuitable for the user can be excluded when integrating the responses.

[0146] In the second embodiment, filtering by the external agent using the score data is not performed in step S13. That is, in the second embodiment, the speech statement is sent to the external agent without filtering based on the user's preferences.

[0147] On the other hand, in the second embodiment, information filtering is performed during the integration of responses in step S17. Specifically, responses from external agents with scores below a predetermined value are excluded from the integration. According to this structure, it is also possible to achieve the objective of "providing responses from areas of no interest to the user to the customer."

[0148] Third Implementation Method

[0149] In the first and second embodiments, pre-generated score data is used to select the external agent to be utilized. Alternatively, the score data can be updated as needed. The third embodiment updates the score data based on user feedback.

[0150] In the third embodiment, the intelligent agent unit 112 is configured to acquire data (hereinafter referred to as feedback data) related to the user's actions after providing a response to the user. Examples of user actions include speaking and moving. Feedback data includes, for example, the content of the user's speech, the result of speech parsing, and movement records (e.g., where the user went after receiving a response).

[0151] For example, if a user who receives a response immediately responds with an affirmative statement, it can be assumed that the user holds an affirmative attitude toward at least one domain belonging to the external agent being used. Similarly, if a user who receives a response acts in accordance with the response (e.g., actually visiting a location recommended by the external agent), it can be assumed that the user holds an affirmative attitude toward at least one domain belonging to the external agent being used.

[0152] In this case, it is preferable to increase the scores included in the score data for the domain to which the corresponding external agent belongs.

[0153] The third implementation updates the score data based on user feedback after a response is provided.

[0154] Figure 9 This is a diagram illustrating the processing flow performed by the control unit 11 of the vehicle-mounted device 10 receiving a speech from the user in the third embodiment. As shown in the figure, in the third embodiment, the intelligent agent unit 112 is configured to acquire feedback data.

[0155] Feedback data can be the text of the user's speech after a response is given, or the result of parsing that speech (e.g., a numerical value indicating affirmation). Additionally, feedback data can also represent the user's reaction after a response is given. For example, feedback data could be data representing the user's gestures (nodding, shaking their head, etc.). Furthermore, feedback data can also be data related to the user's actions after a response is given (e.g., location information shifts, identifiers of visited points, etc.).

[0156] In addition, the intelligent agent unit 112 evaluates each external intelligent agent based on the feedback data and updates the score data.

[0157] Figure 10 This is a flowchart of the processing performed by the intelligent agent unit 112 in the third embodiment. The processing shown in the figure is performed after step S18 (processing to output the integrated response statement).

[0158] First, in step S21, the agent unit 112 acquires feedback data representing the content of feedback from the user. When feedback is provided via voice, the agent unit 112 acquires the user's speech and obtains the result of analyzing it as feedback data. When feedback is provided via user actions, the agent unit 112 acquires data related to the user's actions and obtains the result of analyzing it as feedback data. Of course, other user actions can also be detected as feedback.

[0159] Then, in step S22, the agent unit 112 updates the score data of each external agent based on the feedback data.

[0160] For example, if an external agent suggests visiting a specific location, and the visit to the suggested location is actually confirmed based on feedback data, the agent unit 112 performs a process that increases the score of the domain to which the external agent belongs.

[0161] Furthermore, when feedback data indicates that the user has given a positive response to the output of a specific external agent, the agent unit 112 performs processing to increase the score of the domain to which the external agent belongs. Positive responses include, for example, affirmative speech or affirmative actions.

[0162] Furthermore, if feedback data indicates that the user has given a negative response to the output of a specific external agent, the agent unit 112 can also reduce the score based on the domain to which the external agent belongs. Negative responses include, for example, negative speech or negative actions.

[0163] Furthermore, the intelligent agent unit 112 can also determine which external intelligent agent's feedback the user provided was directed to. For example, if the feedback is about moving to a certain location, it can be determined that the external intelligent agent that requested access to that location is the object of evaluation.

[0164] Furthermore, for example, when the feedback is a user's speech, it's possible to determine which external agent the feedback is directed to by analyzing the content of that speech. For instance, if an external agent belonging to the "art" domain provides art news, and the user then expresses a lack of interest in art, it can be known that the external agent belonging to the "art" domain is the subject of the evaluation.

[0165] Furthermore, if it is unclear which of the multiple external agents is the recipient of the feedback, agent 112 can also ask the user further questions.

[0166] For example, if a suggestion to watch a baseball game is made after integrating responses from external agents belonging to the "sports" domain and those belonging to the "baseball" domain, but the user makes a negative comment, it's impossible to determine whether the user is uninterested in baseball or interested in sports in general. Therefore, the agent unit 112 can further question the user to determine which external agent provided the feedback.

[0167] Furthermore, the intelligent agent unit 112 can also be configured to directly obtain feedback from the user regarding external intelligent agents. For example, the intelligent agent unit 112 can also prompt the user with a list of external intelligent agents being utilized and input ratings for each. In this case, the feedback data includes the ratings given by the user.

[0168] Furthermore, the intelligent agent 112 can update the score data each time it detects positive or negative feedback from the user, or it can accumulate feedback data and update the score data at predetermined times based on the accumulated feedback data. In this case, new feedback data can be given greater weight.

[0169] As described above, according to the third embodiment, the score data can be updated appropriately based on feedback from users.

[0170] Fourth Implementation Method

[0171] Imagine a future where many third-party external intelligent agents emerge. In such a scenario, it would be preferable not to process user feedback locally, but to share it across multiple vehicles for evaluation by the external intelligent agents.

[0172] The fourth embodiment is an implementation in which an intelligent agent (hereinafter, the evaluation agent) is configured to evaluate an external intelligent agent, and the evaluation agent evaluates the external intelligent agent based on feedback data sent from multiple vehicle-mounted devices 10.

[0173] In the fourth embodiment, multiple vehicle-mounted devices 10 receive the results of the evaluation agent's level and update the score data 12D based on the results.

[0174] Furthermore, while the feedback data in the third embodiment reflects personal preferences, the feedback data in the fourth embodiment is used to evaluate the performance and reliability of external agents, such as the accuracy of information and the appropriateness of the timing of information delivery. Therefore, in the fourth embodiment, the user's ratings of each external agent are used as feedback data, rather than the user's actions. Additionally, the score data is not domain-specific but defined for each of the multiple external agents.

[0175] Figure 11 This diagram illustrates the processing flow performed by the control unit 11 of the vehicle-mounted device 10 upon receiving a speech from the user in the fourth embodiment. As shown, in the fourth embodiment, the agent unit 112 sends the acquired feedback data to the evaluation agent. As described above, the feedback data in this embodiment includes a rating for the external agent used. Therefore, the agent unit 112 can also provide the user with a list of external agents used and input ratings.

[0176] The intelligent agent unit 112 sends the feedback data to the evaluation server in a way that establishes an association between the feedback data and the external intelligent agent of the object. If there are multiple external intelligent agents that are the objects of evaluation, multiple sets of feedback data are also sent.

[0177] The feedback data sent is statistically analyzed by the evaluation server. The evaluation server statistically analyzes the scores contained in the feedback data obtained from multiple vehicle-mounted devices 10 and calculates an integrated score for each external agent. The integrated score can also be calculated considering the influence of each user. The calculated integrated score is provided based on requests from the vehicle-mounted devices 10.

[0178] The intelligent agent unit 112 updates the score data 12D based on the integrated score provided by each external intelligent agent from the evaluation server. The acquisition of the integrated score from the evaluation server and the updating of the score data can be performed either at a predetermined period or after the processing of sending feedback data to the evaluation server.

[0179] Figure 12 This is a flowchart of the processing performed by the intelligent agent unit 112 in the fourth embodiment. The illustrated processing is... Figure 10 Similarly, this is executed after step S18 (processing of the response statement).

[0180] First, in step S31, the agent unit 112 receives input from the user the ratings of each external agent being utilized.

[0181] Next, in step S32, the agent unit 112 generates feedback data containing a score and sends it to the evaluation agent. In the case of an external agent containing multiple objects, multiple sets of feedback data are also generated.

[0182] Next, in step S33, the agent unit 112 receives the integrated scores for each of the multiple external agents from the evaluation agent and reflects them in the score data 12D possessed by the device. In this step, for example, a lower score can be assigned to an external agent with a lower integrated score, and a higher score can be assigned to an external agent with a higher integrated score.

[0183] According to this embodiment, the evaluations given by other users can be reflected in the score data 12D. Based on this structure, external agents that receive higher evaluations from multiple users can be utilized more actively, while external agents that receive lower evaluations can be utilized less actively.

[0184] In addition, Figure 4The diagram illustrates an example where a single score is defined in score data 12D, but the score data can also contain both scores reflecting an individual's preferences and scores reflecting information from the evaluating agent. Agent unit 112 can integrate these scores.

[0185] Variations

[0186] The above-described implementation is merely an example, and this disclosure can be implemented with appropriate modifications without departing from its spirit.

[0187] For example, the processing and methods described in this disclosure can be freely combined and implemented as long as they do not create technical contradictions.

[0188] Furthermore, while the embodiment illustrates configuring the external agent on an external device connected to the Internet, the external agent can also be configured on other devices. For example, the external agent can also be configured on an edge server accessible by the vehicle. Alternatively, the external agent can be distributed across multiple edge servers or the cloud.

[0189] Furthermore, while an in-vehicle device is exemplified in the embodiments, the information processing apparatus of this disclosure can also be implemented as a non-in-vehicle device or equipment.

[0190] Furthermore, when an external agent is configured on multiple edge servers, the vehicle-mounted device 10 can obtain the vehicle's location information and, based on this location information, determine the edge server where the target external agent is located. For example, an external agent providing regional information may be configured on multiple edge servers divided by region. In this case, for example, the vehicle-mounted device 10 can determine the external agent configured on the geographically nearest edge server based on the vehicle's location information. The regional information processed by the external agent may vary depending on the edge server. For example, an external agent providing regional information for region A may be configured on an edge server configured in region A, and an external agent providing regional information for region B may be configured on an edge server configured in region B. By determining the edge server corresponding to the region where the vehicle is located and accessing the external agent provided by that edge server, the vehicle-mounted device 10 can obtain the regional information for that region.

[0191] Furthermore, a process described as being performed by one device can also be performed by multiple devices. Alternatively, a process described as being performed by different devices can also be performed by one device. In a computer system, the hardware configuration (server configuration) through which various functions are implemented can be flexibly changed.

[0192] This disclosure can also be implemented by providing a computer with a computer program that has the functions described in the above embodiments installed, and having the program read and executed by one or more processors owned by the computer. Such a computer program can be provided to the computer via a non-transitory computer-readable storage medium capable of being connected to a computer system bus, or via a network. Non-transitory computer-readable storage media include, for example, any type of disk (floppy disk, hard disk drive (HDD), etc.), optical disk (CD-ROM, DVD, Blu-ray disc, etc.), read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic cards, flash memory, optical cards, and any type of medium suitable for storing electronic instructions.

Claims

1. An information processing device, wherein, The information processing device has a control unit that performs the following processing: The speech is obtained based on the user's speech. Based on the spoken statement, one or more external agents are selected from multiple external agents that are dedicated to multiple specified domains and provide natural language-based dialogue services, to serve as the destination of the spoken statement. The speech statement is sent to one or more external agents that make the decision, and one or more responses are obtained from the one or more external agents. as well as The system integrates the responses from one or more sources and provides the integrated response to the user. The control unit determines one or more external agents as the destination for the spoken statement based on scores set for each of the plurality of external agents, or determines the integrated object in the one or more responses based on the scores. Furthermore, based on the user's actions after providing the integrated response, the score corresponding to at least one of the more than one external agents is updated.

2. The information processing apparatus according to claim 1, wherein, The control unit also acquires feedback data representing the user's actions after providing the integrated response, and updates the score based on the feedback data.

3. The information processing apparatus according to claim 2, wherein, If the feedback data indicates that the user's response to the first external agent is positive, the control unit increases the score for the first external agent.

4. The information processing apparatus according to claim 2, wherein, If the feedback data indicates that the user's response to the first external agent is negative, the control unit reduces the score for the first external agent.

5. The information processing apparatus according to claim 2, wherein, The feedback data includes data indicating the user's destination or the content of the user's speech.

Images