Information processing apparatus

Abstract

The present application relates to an information processing apparatus. One or more external agents respectively specialized for a plurality of prescribed domains, which provide a natural language-based dialogue service, are decided based on a dialogue sentence of a speech made by a user, the dialogue sentence is transmitted to the one or more external agents, one or more responses are acquired from the one or more external agents, the one or more responses are integrated, and the integrated response is provided to the user. In addition, the one or more external agents are decided based on a score set for the plurality of external agents, or the object of the integration among the one or more responses is decided based on the score, and the score is updated based on information acquired from an evaluation agent that evaluates each of the plurality of external agents.

Description

Technical Field

[0001] This disclosure relates to dialogue technology. Background Technology

[0002] There exists a technique for conducting natural language-based dialogues based on input text. For example, International Publication No. 2019 / 225201 discloses a device capable of conversing with multiple agents via a network.

[0003] In the future, with the development of machine learning, it is expected that dialogue services based on natural language will further increase. Summary of the Invention

[0004] The purpose of this disclosure is to improve the accuracy of dialogues that utilize machine learning models.

[0005] One aspect of this disclosure relates to an information processing apparatus comprising a control unit that performs the following processing: obtaining a speech statement based on a user's speech; determining, based on the speech statement, one or more external agents from among a plurality of external agents that are respectively dedicated to multiple defined domains and provide natural language-based dialogue services as the destination of the speech statement; sending the speech statement to the determined one or more external agents; obtaining one or more responses from the one or more external agents; and integrating the one or more responses and providing the integrated response to the user, wherein the control unit determines the one or more external agents as the destination of the speech statement based on ratings set for each of the plurality of external agents, or determines the object of integration in the one or more responses based on the ratings, and updates the ratings set for each of the plurality of external agents based on information obtained from the evaluation agent that evaluates each of the plurality of external agents.

[0006] Alternatively, a program for causing a computer to perform the method described above, or a computer-readable storage medium that non-temporarily stores the program, may be cited as an example.

[0007] According to this disclosure, the accuracy of dialogues utilizing machine learning models can be improved. Attached Figure Description

[0008] The features, advantages, and technical and industrial significance of exemplary embodiments of the present invention will now be described with reference to the accompanying drawings. In the drawings, the same reference numerals denote the same elements, wherein:

[0009] Figure 1A This is a schematic diagram of the dialogue system involved in the first embodiment.

[0010] Figure 1BThis is a schematic diagram of the dialogue system involved in the first embodiment.

[0011] Figure 1C This is a schematic diagram of the dialogue system involved in the first embodiment.

[0012] Figure 2 This is a hardware configuration diagram of the vehicle-mounted device 10.

[0013] Figure 3 This is a software configuration diagram of the vehicle-mounted device 10.

[0014] Figure 4 This is an example of the scoring data used by the vehicle-mounted device 10.

[0015] Figure 5 This is a diagram illustrating the process flow performed by the vehicle-mounted device 10 in the first embodiment.

[0016] Figure 6A This is an example of text input from an external intelligent agent.

[0017] Figure 6B This is an example of text output by an external intelligent agent.

[0018] Figure 6C This is an example of text output by an external intelligent agent.

[0019] Figure 7A This is an example of text input to a local agent.

[0020] Figure 7B This is an example of text output by a local agent.

[0021] Figure 8 This is a flowchart of the process performed by the vehicle-mounted device 10 in the first embodiment.

[0022] Figure 9 This is a diagram illustrating the process flow performed by the vehicle-mounted device 10 in the third embodiment.

[0023] Figure 10 This is a flowchart of the process performed by the vehicle-mounted device 10 in the third embodiment.

[0024] Figure 11 This is a diagram illustrating the process flow performed by the vehicle-mounted device 10 in the fourth embodiment.

[0025] Figure 12 This is a flowchart of the process performed by the vehicle-mounted device 10 in the fourth embodiment. Detailed Implementation

[0026] In recent years, with the development of machine learning, products equipped with language models have been increasing. For example, by assembling a large-scale language model (LLM) with improved accuracy through machine learning using large-scale datasets into a product, it is possible to add natural language-based dialogue functionality to the product.

[0027] One area where natural language-based dialogue is useful is the automotive industry. For example, by incorporating LLM into in-vehicle devices, information can be obtained without operating touch panels. This functionality is particularly useful when hands cannot be released while driving.

[0028] On the other hand, the large data size of the learned LLM model makes it difficult to integrate into embedded devices due to cost considerations. Therefore, a method is proposed that instead of integrating the LLM into a device, access the LLM via a network. By accessing the LLM over a network, large-scale language models that are difficult to integrate into local storage can be utilized.

[0029] Furthermore, via the network, it is possible to selectively access LLMs dedicated to specific domains. For example, in situations where multiple LLMs are prepared for each domain, such as LLMs dedicated to route guidance, LLMs dedicated to location information guidance, and LLMs dedicated to casual conversation, it is possible to utilize an LLM suitable for the user's intended purpose. In this regard, a technique is known to determine the category of a user's speech based on its content and automatically select an LLM suitable for that category.

[0030] However, in the case of an LLM that selects to conduct dialogue from multiple domains, there is a problem that dialogue can only be conducted in a specific domain.

[0031] For example, imagine a user speaking something like "The car's warning lights are on," and selecting an LLM (Local Management Model) specifically for vehicle operation guidance. The system would then provide information related to the meaning of the warning lights in response to the speech. However, because LLMs for vehicle operation guidance lack location-specific information, it's possible that services beyond the scope of the speech, such as "Guide to the nearest car dealership for inspection," might not be available.

[0032] As a solution to this problem, one approach is to send the content of the user's speech (speech statements) to multiple LLMs in different domains, and then synthesize and provide the resulting responses to the user.

[0033] However, when multiple third parties provide diverse LLMs, sending speech statements to a low-quality LLM may result in situations where the quality of the response cannot be guaranteed.

[0034] The information processing apparatus disclosed herein addresses such a problem.

[0035] One aspect of this disclosure relates to an information processing apparatus comprising a control unit that performs the following processing: obtaining a speech statement based on a user's speech; determining, based on the speech statement, one or more external agents from among a plurality of external agents that are dedicated to multiple defined domains and provide natural language-based dialogue services as the destination of the speech statement; sending the speech statement to the determined one or more external agents; obtaining one or more responses from the one or more external agents; and integrating the one or more responses and providing the integrated response to the user, wherein the control unit determines the one or more external agents as the destination of the speech statement based on ratings set for each of the plurality of external agents, or determines the object of integration in the one or more responses based on the ratings, and updates the ratings set for each of the plurality of external agents based on information obtained from the evaluation agent that evaluates each of the plurality of external agents.

[0036] External agents are systems capable of engaging in natural language-based dialogue and providing dialogue services using language models specifically designed for multiple domains. These language models can be large-scale language models, such as those learned to perform natural language-based dialogue tasks. Examples of multiple domains include "route guidance," "location information guidance," "vehicle operation guidance," "sightseeing guidance," and "casual conversation." These agents are typically configured to be remotely accessible via a network.

[0037] The control unit determines one or more external agents as the destination for the user's speech (speech statement), and sends the speech statement to the determined external agent to obtain a response. The control unit can also send a prompt text containing the user's speech (speech statement) to the determined external agent via a network. Thus, it is possible to obtain one or more responses from the one or more external agents.

[0038] In addition, the control unit integrates more than one response obtained from various external agents and provides it to the user. Response integration can be rule-based or utilize other language models. For example, response integration can be achieved by inputting responses obtained from various external agents into an agent with a language model (the local agent) through local actions.

[0039] The language model possessed by the local agent is a language model capable of performing natural language-based dialogue tasks, but it can be configured to be a language model generated at a relatively low cost without performing domain-specific learning.

[0040] Based on this structure, information can be obtained from multiple LLMs with various professional fields, and this information can be integrated and provided to users.

[0041] In the above examples, for instance, by integrating responses obtained from an LLM dedicated to vehicle operation guidance and responses obtained from an LLM dedicated to geographic information guidance, it is possible to provide the meaning of the warning light and guide the service location that can resolve the problem.

[0042] On the other hand, in situations where multiple third parties provide LLMs, if the quality of the LLMs is inconsistent, inappropriate responses may sometimes be provided to users. For example, if an external agent with low accuracy of information is used, the quality of the information provided to the user will also be reduced.

[0043] In view of this, the information processing apparatus of this disclosure uses a score set according to each external agent to determine one or more external agents as the destination for the transmitted speech statement. Alternatively, the integrated object in the one or more responses is determined based on the score.

[0044] The scores set for each of the multiple external agents are updated based on information obtained from the evaluation agent that evaluates each of the multiple external agents.

[0045] An evaluating agent is an agent that evaluates multiple external agents. For example, an evaluating agent can assess external agents based on the accuracy of information. For instance, an external agent that receives more user ratings for "providing accurate information" can be assigned a higher score. Conversely, an external agent that receives a rating for "lacking in the correctness of information" can be assigned a lower score.

[0046] According to this configuration, it is possible to avoid providing users with information from LLMs that are not preferred for the user (e.g., low-quality LLMs).

[0047] In addition, the control unit may also obtain feedback data representing the user's evaluation of the integrated response, and further send the feedback data to the evaluation agent.

[0048] Feedback data is data used by users to evaluate external intelligent agents. Feedback data can include both direct and indirect evaluations of the external intelligent agents.

[0049] Direct evaluation can be a score assigned by the user to the external agent, while indirect evaluation can be an assessment of "what actions the user took after receiving a response from the external agent (e.g., whether the user acted in accordance with the external agent's proposal)".

[0050] Hereinafter, specific embodiments of the present disclosure will be described based on the accompanying drawings. Unless otherwise specified, the hardware configurations, module configurations, and functional configurations described in each embodiment are not intended to limit the scope of the disclosed technology to them.

[0051] First Implementation Method

[0052] System Overview

[0053] An overview of the dialogue system according to the first embodiment will be described. The dialogue system according to this embodiment is configured to include an in-vehicle device 10 mounted in a vehicle. The in-vehicle device 10 is able to access an intelligent agent (external intelligent agent) that provides natural language-based dialogue services via a network (e.g., a mobile communication network).

[0054] The vehicle-mounted device 10 is mounted on a networked vehicle capable of communicating with any device via wireless communication. The vehicle-mounted device 10 may also include a data communication module (DCM) for connecting vehicle components (e.g., ECU, vehicle terminal, etc.) to a network. In this embodiment, the vehicle-mounted device 10 can access the Internet via a pre-determined mobile communication network and connect with intelligent agents providing dialogue services.

[0055] Furthermore, the vehicle (or components mounted on the vehicle) can provide various services by communicating with external devices via the onboard unit 10. Examples of these services include navigation services, remote control services (e.g., remote air conditioning), in-vehicle Wi-Fi services, and emergency notification services.

[0056] Furthermore, the in-vehicle device 10 has voice input and output functions, enabling it to engage in natural language-based dialogues with vehicle occupants.

[0057] Figure 1A This is a schematic diagram illustrating the vehicle-mounted device 10 involved in the existing method.

[0058] The illustrated vehicle-mounted device 10 is configured to include a control unit for voice input / output and voice recognition, and a local intelligent agent. The local intelligent agent contains a language model learned in a manner capable of engaging in natural language-based dialogue. The local intelligent agent can, for example, provide prescribed functions such as route search and vehicle operation (air conditioning, etc.) via dialogue. In the illustrated configuration, the language model used by the local intelligent agent is stored in a storage device within the vehicle-mounted device 10.

[0059] On the other hand, this approach makes it difficult to provide dialogue services based on large-scale language models. For example, dialogues encompass a wide variety of categories, such as location information, tourist guidance, vehicle-related information (operation guides), and casual conversation. Covering all these categories requires language models trained on massive amounts of data. Such language models can range in size from several gigabytes to hundreds of gigabytes, making storage in local memory impractical from a cost perspective.

[0060] Therefore, a system is proposed for accessing a language model that is dedicated to dialogue in a specific domain via a network.

[0061] Figure 1B This is a schematic diagram illustrating how the vehicle-mounted device 10 can access multiple language models available on the Internet in addition to the local intelligent agent. For example, consider the method of configuring an intelligent agent (external intelligent agent) with multiple large-scale language models dedicated to a specific domain in a device connected to the Internet, and accessing the language model selected according to the request from the vehicle-mounted device 10.

[0062] However, this approach has a limitation: it only allows dialogue related to a specific domain. For example, if a user wants information about a particular sport, it could be envisioned as engaging in dialogue with an agent specifically designed for that sport. However, since this agent cannot provide information outside of the sport, such as facility information, it cannot provide guidance on facilities that allow viewing of the sport.

[0063] Therefore, in the vehicle-mounted device 10 of this embodiment, the local intelligent agent has the function of integrating responses from multiple external intelligent agents. The user's speech content (speech statements) is sent to multiple external intelligent agents, and responses are obtained from these multiple external intelligent agents. These responses are integrated in the local intelligent agent, and the integrated response is provided to the user. Figure 1C This is a schematic diagram of the vehicle-mounted device 10 according to this embodiment.

[0064] If the vehicle-mounted device 10 recognizes a user's speech, it determines which external agent is suitable for the speech based on its content. This processing can be rule-based or performed by a machine learning model. Furthermore, the control unit sends the speech to one or more of the determined external agents.

[0065] The responses from the selected external agent are integrated and provided to the user by the local agent.

[0066] This enables the appropriate delivery of responses from multiple external agents to the user.

[0067] On the other hand, integrating multiple responses from multiple external agents may introduce a new problem: unwanted information may be mixed in. For example, when selecting external agents based on the content of the speech, it is possible that an external agent belonging to a domain that does not align with the user's preferences may be chosen.

[0068] To address this situation, in this embodiment, the vehicle-mounted device 10 sets a score for each external agent according to the user's settings, and controls information collection based on the set score. This allows for actions such as "stopping the use of external agents belonging to domains of no interest to the user."

[0069] In this embodiment, phrases such as "access to an external intelligent agent" and "connection to an external intelligent agent" refer to accessing or connecting to an external device that provides dialogue services through an external intelligent agent. In this embodiment, multiple dialogue services based on a large-scale language model operate as "external intelligent agents" in multiple external devices, and the vehicle-mounted device 10 can engage in dialogue by connecting to any of these external intelligent agents.

[0070] Hardware configuration

[0071] Next, the hardware configuration of each device constituting the system will be explained. Figure 2 This is a diagram that schematically illustrates an example of the hardware structure of the vehicle-mounted device 10 according to this embodiment.

[0072] The vehicle-mounted device 10 can be configured as a computer with a processor (CPU, GPU, etc.), main storage (RAM, ROM, etc.), and auxiliary storage (EPROM, hard disk drive, removable media, etc.). The auxiliary storage stores the operating system (OS), various programs, various tables, etc. By executing the programs stored therein, various functions (software modules) that meet the specified purpose can be achieved, as described later. However, some or all of the functions can also be implemented as hardware modules using hardware circuits such as ASICs or FPGAs.

[0073] The vehicle-mounted device 10 is configured to have a control unit 11, a storage unit 12, a wireless communication module 13, and an input / output unit 14 as hardware.

[0074] The control unit 11 is a computing unit that executes a prescribed program to implement various functions of the vehicle-mounted device 10. The control unit 11 can be implemented, for example, by a hardware processor such as a CPU. Furthermore, the control unit 11 can be configured to include RAM, ROM (Read-Only Memory), cache memory, etc.

[0075] The storage unit 12 is a mechanism for storing information and is composed of storage media such as RAM, disk, and flash memory. The storage unit 12 stores the program executed by the control unit 11 and the data used by the program.

[0076] The wireless communication module 13 is a communication device that performs wireless communication with a specified network. In this embodiment, the wireless communication module 13 is configured to communicate with a specified mobile communication network. The wireless communication module 13 may be configured to have an eUICC (e.g., a SIM card). The SIM card is configured as a microcomputer equipped with a CPU and a storage device. The SIM card is connected to the mobile communication network and stores information for accepting authentication.

[0077] The input / output unit 14 is a unit that receives input from the user of the device and provides prompts to the user. The input / output unit 14 is typically configured to include a device for inputting and outputting voice, such as a microphone or speaker. Additionally, the input / output unit 14 may include a device (such as a display) that provides information based on vision.

[0078] Software composition

[0079] Next, the software configuration of each device constituting the system will be explained. Figure 3 This diagram schematically illustrates the software configuration of the vehicle-mounted device 10 according to this embodiment.

[0080] In this embodiment, the control unit 11 of the vehicle-mounted device 10 is configured with three software modules: a dialogue receiving unit 111, an intelligent agent unit 112, and a route guidance unit 113. Each software module can be implemented by the control unit 11 (CPU, etc.) executing a program stored in the storage unit 12. The information processing executed by the software modules is synonymous with the information processing executed by the control unit 11 (CPU, etc.).

[0081] The dialogue receiving unit 111 acquires the speech given by the vehicle occupants (hereinafter also referred to as users) via the input / output unit 14. The dialogue receiving unit 111 performs prescribed processing on the acquired voice data, performing speech recognition. As a result, the content of the speech is converted into text. Furthermore, the dialogue receiving unit 111 sends the text obtained as a result of speech recognition to the intelligent agent unit 112.

[0082] Additionally, the dialogue receiving unit 111 outputs the response (hereinafter referred to as the answer statement) sent from the language model by the intelligent agent unit 112. The dialogue receiving unit 111 converts the answer statement into speech and outputs it via the input / output unit 14.

[0083] The agent unit 112 first provides a dialogue service utilizing a language model stored in the device (the local language model 12A described later). The agent unit 112 can operate as a virtual agent (local agent). The local agent can, for example, provide dialogue services completed within the device (e.g., casual conversation).

[0084] Furthermore, when specialized knowledge is required during a conversation, the intelligent agent unit 112 sends spoken statements via a network to one or more external intelligent agents that can be utilized, and obtains the results. For example, if it is determined that the user is requesting information related to a vehicle, the intelligent agent unit 112 selects an external intelligent agent that can provide guidance on vehicle processing. Additionally, if it is determined that the user is requesting information related to facilities or shops, the intelligent agent unit 112 selects an external intelligent agent that can provide location information.

[0085] The selection of external agents can be based on the results of parsing the content of the speech. For example, the agent unit 112 can calculate the fitness of each of the multiple external agents based on the words contained in the speech and the context, and send the speech to the external agents that have obtained a fitness value of above a certain value.

[0086] The agent unit 112 may also select multiple external agents to send the speech statement. For example, if there are multiple external agents whose suitability for the speech statement is above a predetermined value, the agent unit 112 may send the speech statement to these multiple external agents. In this case, the agent unit 112 obtains the response statements to the speech statement from the multiple external agents, integrates these response statements, and provides them to the user.

[0087] Furthermore, when selecting an external agent, the intelligent agent unit 112 utilizes pre-set rating data for each user. The rating data represents the priority of each of the multiple external agents.

[0088] Figure 4 This is an example of rating data. In this embodiment, rating data consists of a combination of the domain of an external intelligent agent and ratings. A rating is a numerical representation of the degree of a user's interest or preference. In the illustrated example, this represents user A having a strong interest in current events and sports, but a weak interest in art.

[0089] In this embodiment, the agent unit 112 excludes external agents corresponding to domains with scores below a predetermined value from the selected objects. That is, external agents with scores below a predetermined value (e.g., 1.0) are not sent speech statements.

[0090] This allows the system to suppress responses that are of little interest to the user.

[0091] Return to Figure 3 Let's continue with the explanation.

[0092] The route guidance unit 113 receives the user's specified origin and destination, and generates a route connecting the origin and destination based on road map data and the like. Furthermore, it outputs information related to the generated route via the input / output unit 14.

[0093] The storage unit 12 of the vehicle-mounted device 10 stores a local language model 12A, agent information 12B, attribute data 12C, and rating data 12D.

[0094] The local language model 12A is a language model learned in a way that enables it to perform natural language-based dialogue tasks. Local language model 12A is a language model generated at a relatively low cost, without undergoing learning for domain-specific dialogue. Local language model 12A only needs to be a language model for general conversations. For example, as local language model 12A, it is also possible to utilize language models that are publicly available as open source.

[0095] The local language model 12A is a lightweight language model compared to the large-scale language models possessed by external agents accessible via a network. By utilizing the local language model 12A, agent unit 112 can conduct responsive dialogues. For example, it is possible to differentiate between sending speech statements to external agents when specialized conversations are required, and having the local agent handle speech statements for conversations requiring high-speed responses.

[0096] Agent information 12B is a collection of data related to multiple external agents that can be utilized by the vehicle-mounted device 10. Agent information 12B may include, for example, the identifier and name of the external agent, information related to a specific domain, and information related to the destination (e.g., providing the network address of the external device of the external agent). Agent unit 112 can determine the external agent to be the destination of the spoken statement by referring to agent information 12B.

[0097] Attribute data 12C is a collection of data related to the attributes of the user interacting with the vehicle-mounted device 10. Attribute data 12C may include, for example, personal information such as the user's gender, age, height, and weight. Furthermore, if there are multiple users in the vehicle, attribute data 12C may also contain data for multiple individuals.

[0098] For reference Figure 4 As explained, rating data 12D represents the degree to which a user is suitable for a given user within the respective domains of multiple external agents. Rating data 12D is defined for each of the multiple users who may be riding in the vehicle.

[0099] Dialogue process

[0100] Next, a summary of the processes performed by the control unit 11 will be explained. Figure 5 This is a diagram illustrating the processing flow performed by the control unit 11 of the vehicle-mounted device 10 from the user receiving the speech.

[0101] The dialogue receiving unit 111 acquires speech from the user via the input / output unit 14. For example, the input / output unit 14 converts speech acquired via a microphone, etc., into voice data, which the dialogue receiving unit 111 then acquires. The dialogue receiving unit 111 performs prescribed speech recognition processing on the acquired voice data, converting the voice data into text. While suspending the response to the speech, the dialogue receiving unit 111 sends the text obtained from the speech recognition to the intelligent agent unit 112. This text is referred to as the "speech statement".

[0102] Upon receiving the spoken statement, agent 112 determines the external agent to which the statement should be sent, based on the content of the speech, the intent of the speech (what the user is seeking), and the user's current state (what they are currently doing). The intent of the speech can be determined using the local language model 12A, or other machine learning models. Alternatively, it can be determined based on rules.

[0103] Furthermore, the user's status can also be determined using sensor data other than speech. For example, the intelligent agent unit 112 can also determine the user's status using data related to vehicle speed, current location, and direction of travel obtained from sensors on the vehicle. Examples of user status could include "moving on a regular road," "moving on a highway," "parked," "waiting at a traffic light," "moving towards the workplace," and "going home."

[0104] The agent unit 112 obtains information (agent information 12B) related to external agents from the storage unit 12, and determines the external agent to which the speech statement should be sent based on the agent information 12B, the determined intent, and the situation. For example, the agent unit 112 can calculate the fitness level for each of multiple external agents and decide to send the speech statement to the external agent that has obtained a fitness level of a predetermined value or higher.

[0105] In this example, the following three are shown as external intelligent agents.

[0106] (1) Vehicle intelligent agent

[0107] An external intelligent agent capable of monitoring the vehicle's interior and configuring its devices. It has already studied the vehicle's owner's manual and can provide information about the vehicle's functions.

[0108] (2) Chatting agent

[0109] An external agent specifically designed for non-task-oriented conversations.

[0110] (3) Regional Information Intelligent Agent

[0111] An external intelligent agent capable of providing information about the surrounding area, including store directions and sightseeing guides.

[0112] In addition, in this example, vehicle intelligence and local information intelligence were selected as suitable external intelligent agents for speaking.

[0113] In some cases, the agent unit 112 may determine that it is unnecessary to send a speech statement to an external agent. For example, this may be the case when the user's request is for path guidance or other situations that can be handled by the device itself, or when a high-speed response is required. In such cases, the speech statement is processed only by the agent unit 112 (and the local language model 12A).

[0114] Next, the intelligent agent unit 112 filters the selected external intelligent agents based on the user's preferences. The intelligent agent unit 112 excludes external intelligent agents with scores below a predetermined value from the selected objects based on the rating data 12D.

[0115] Then, agent unit 112 generates prompt text for input to the selected external agent. Figure 6A This is an example of a prompt text input from agent 112 to an external agent.

[0116] In addition to the spoken words, the prompt text also includes context information relevant to the user.

[0117] In this embodiment, background information is information accompanying spoken statements and is related to the user's attribute information and the user's movement. Background information may include, for example, personal information such as the user's gender, age, height, and weight. This information can be obtained from attribute data 12C stored in storage unit 12.

[0118] Additionally, the background information may include information about the user's movement. This information can include details related to the vehicle's origin, destination, route, and transit points. This information can be obtained from the route guidance unit 113.

[0119] External agents can then generate responses based on this background information.

[0120] The intelligent agent 112 sends the generated prompt text to the external device of the external intelligent agent that provides the target via the network.

[0121] The external agent that has been given the spoken statement responds to the spoken statement by outputting a response. Figure 6B This is an example of a response from a vehicle's intelligent agent. Figure 6C This is an example of a response from a local information agent. As shown in the figure, each agent outputs a response relevant to its respective domain of expertise. For example, the vehicle agent outputs information about the specific meaning of a warning light, while the local information agent outputs information about locations where the vehicle can be inspected.

[0122] If agent 112 receives a response from an external agent that sent the spoken statement, it integrates these responses. This integration can be performed using the local language model 12A.

[0123] For example, agent 112 inputs a prompt text containing one or more received responses into the local language model 12A, enabling it to integrate the responses. This prompt text may include, for example, the user's speech, responses from multiple external agents, and an instruction to integrate the responses to generate a new text. Figure 7A This is an example of prompt text that is input into the local language model 12A.

[0124] The result is that the integrated response statement is output from the local language model 12A. Figure 7B This is an example of a response statement obtained as a result of integration. The integrated response statement is sent from the agent unit 112 to the dialogue receiving unit 111.

[0125] If the dialogue receiving unit 111 receives a response statement, it generates voice data based on the response statement and outputs it via the input / output unit 14. The dialogue receiving unit 111 may generate only the voice data of the response statement, or it may generate information accompanying the voice data. For example, a user interface screen may be provided. This user interface screen may contain the response statement written in text.

[0126] The dialogue system described in this embodiment operates seamlessly. Figure 5 The series of processes shown are as follows. Thus, when observed from the perspective of a user who has spoken, it is possible to achieve an effect similar to having a conversation with the vehicle-mounted device 10.

[0127] flow chart

[0128] Next, the details of the processing performed by the vehicle-mounted device 10 will be explained. Figure 8 This is a flowchart of the processing performed by the vehicle-mounted device 10. Figure 8 The process shown begins upon detection of a user's (vehicle occupant's) speech. The start of a speech can be detected, for example, by predefined keywords.

[0129] First, in step S11, the dialogue receiving unit 111 identifies the content of the speech. The dialogue receiving unit 111 acquires the voice data output from the input / output unit 14, performs speech recognition processing, and thereby converts the speech into text. The converted text (speech statement) is sent to the intelligent agent unit 112.

[0130] Next, in step S12, the intelligent agent 112 acquires the scoring data corresponding to the user who spoke. The user who spoke can be identified based on voice, or, if the vehicle is equipped with a personal authentication device (e.g., a facial recognition device or a fingerprint authentication device), can be identified based on their authentication result.

[0131] In step S13, the intelligent agent unit 112 determines the external intelligent agent that will send the speech statement. The external intelligent agent that sends the speech statement can, for example, infer the user's intention (what the user requests) and situation (what the user is currently doing) based on the speech statement, and make a decision based on the result.

[0132] For example, the agent unit 112 can read information (agent information 12B) related to the characteristics of an external agent from the storage unit 12, and decide on the external agent to be utilized based on this information.

[0133] Furthermore, as described above, the agent unit 112 filters external agents that send speech statements based on scoring data. The agent unit 112 excludes external agents with scores below a predetermined value from the selected objects based on the scoring data 12D.

[0134] Next, in step S14, the agent unit 112 obtains the user's background information. If the background information is related to the user's movement, this information can be obtained from the path guidance unit 113. Alternatively, if the background information is the user's attribute information, this information (attribute data 12C) can be obtained from the storage unit 12.

[0135] In step S15, the agent unit 112 sends a speech statement and background information to the external agent determined in step S13. For example, the agent unit 112 accesses an external device that provides services related to the external agent providing the object via the wireless communication module 13, and sends the speech statement and background information to that external agent.

[0136] In step S16, the agent unit 112 obtains the response corresponding to the spoken statement from the external agent of the object.

[0137] In step S17, the agent unit 112 uses the local language model 12A to integrate these responses. Specifically, the agent unit 112 generates a prompt text containing a list of responses obtained from multiple external agents and an instruction to integrate these responses, and inputs the prompt text into the local language model 12A. Additionally, it obtains the integrated response statement output from the local language model 12A.

[0138] In step S18, the intelligent agent unit 112 sends the response statement obtained as a result of integration to the dialogue receiving unit 111. The dialogue receiving unit 111 generates voice data based on the response statement and outputs it via the input / output unit 14. The dialogue receiving unit 111 can also convert the response statement into voice data using speech synthesis technology. The voice data is output to the user via the input / output unit 14 (speaker, etc.).

[0139] As explained above, the vehicle-mounted device 10 of this embodiment determines which external agent the local agent should send the content of the vehicle occupants' speech to. Furthermore, the local agent integrates and outputs the responses obtained from the external agents. Thus, it is possible to provide the user with responses from multiple external agents handling highly specialized topics simultaneously.

[0140] Furthermore, since the external agent sending the speech is filtered using the user's personal preferences, responses from areas that are not of interest to the user can be excluded.

[0141] Second Implementation Method

[0142] In the first embodiment, when an external agent is selected in step S13, scoring data is used to exclude external agents that are unsuitable for the user. Alternatively, even if a speech statement is sent to an external agent without considering the user's preferences, responses unsuitable for the user can be excluded during the timing of the integrated response.

[0143] In the second embodiment, filtering by the external agent utilizing the rating data is not performed in step S13. That is, in the second embodiment, the speech statements are sent to the external agent without filtering based on the user's preferences.

[0144] On the other hand, in the second embodiment, filtering of timing execution information for the integrated response is performed in step S17. Specifically, responses from external agents with scores below a predetermined value are excluded from the integrated object. Based on this configuration, it is also possible to achieve the objective of "providing responses from domains that are not of interest to the user."

[0145] Third Implementation Method

[0146] In the first and second embodiments, pre-generated rating data is used to select the external agent to be utilized. On the other hand, the rating data can be updated appropriately. The third embodiment updates the rating data based on user feedback.

[0147] In the third embodiment, the intelligent agent unit 112 is configured to acquire data (hereinafter referred to as feedback data) related to the user's actions after a response statement has been provided to the user. Examples of user actions include speaking and moving. Feedback data includes, for example, the content of the user's speech, the result of speech parsing, and actual movement (e.g., where the user went after receiving the response statement).

[0148] For example, if a user who receives a response immediately responds with an affirmative statement, it can be assumed that the user holds an affirmative view of at least one domain belonging to the external agent used. Furthermore, if a user who receives a response acts in accordance with the response (e.g., actually visiting a location recommended by the external agent), it can be assumed that the user holds an affirmative view of at least one region belonging to the external agent used.

[0149] In this case, it is preferable to increase the scores included in the scoring data for the domain to which the corresponding external agent belongs.

[0150] The third implementation updates the rating data based on user feedback after the user provides an answer.

[0151] Figure 9 This is a diagram illustrating the processing flow performed by the control unit 11 of the vehicle-mounted device 10 receiving speech from the user in the third embodiment. As shown in the diagram, in the third embodiment, the intelligent agent unit 112 is configured to acquire feedback data.

[0152] Feedback data can be the text of a user's speech after a response has been given, or the result of parsing that speech (e.g., a numerical value indicating approval). Additionally, feedback data can also represent the user's reaction after a response has been given. For example, feedback data could be data representing the user's gestures (nodding, shaking their head, etc.). Furthermore, feedback data can also be data related to the user's actions after a response has been given (e.g., location updates, identifiers of visited locations, etc.).

[0153] In addition, the intelligent agent unit 112 evaluates each external intelligent agent based on feedback data and updates the score data.

[0154] Figure 10This is a flowchart of the process executed by the intelligent agent unit 112 in the third embodiment. The process shown in the figure is executed immediately after step S18 (processing to output the integrated response statement).

[0155] First, in step S21, the agent unit 112 acquires feedback data representing the content of feedback from the user. When feedback is provided via voice, the agent unit 112 acquires the user's speech and obtains the result of analyzing it as feedback data. When feedback is provided via user actions, the agent unit 112 acquires data related to the user's actions and obtains the result of analyzing it as feedback data. Of course, other user actions can also be detected as feedback.

[0156] Then, in step S22, the agent unit 112 updates the score data of each external agent based on the feedback data.

[0157] For example, when an external agent proposes a visit to a specific location, and the visit to the proposed location is actually confirmed based on feedback data, the agent unit 112 performs a process that increases the score of the domain to which the external agent belongs.

[0158] Furthermore, when the feedback data indicates that the user has given a positive response to the output of a specific external agent, the agent unit 112 performs processing to increase the score of the domain to which the external agent belongs. Positive responses include, for example, affirmative speech or positive reactions.

[0159] Furthermore, if the feedback data indicates that the user has given a negative response to the output of a specific external agent, the agent unit 112 can perform processing to reduce the score based on the domain to which the external agent belongs. Negative responses include, for example, negative speech or negative reactions.

[0160] The agent unit 112 can also determine which external agent's feedback the user gave. For example, if the feedback is about moving to a certain location, it can be determined that the external agent that requested access to that location is the object of evaluation.

[0161] Furthermore, for example, when the feedback is a user's speech, the content of that speech can be analyzed to determine which external agent the feedback is directed to. For instance, if an external agent belonging to the "art" domain provides art news, and the user makes a statement indicating no interest in art, it can be known that the external agent belonging to the "art" domain is the subject of the evaluation.

[0162] In cases where it is unclear which of the multiple external agents is the recipient of the feedback, agent 112 can ask the user further questions.

[0163] For example, if the result of integrating responses from external agents belonging to the "sports" domain and external agents belonging to the "baseball" domain is to suggest watching a baseball game, and a negative response is given, the user cannot determine whether they are uninterested in baseball or in the sport itself. Therefore, agent 112 can further question the user to determine which external agent gave the feedback.

[0164] Furthermore, the intelligent agent unit 112 can also be configured to directly obtain feedback from the user regarding external intelligent agents. For example, the intelligent agent unit 112 can prompt the user with a list of external intelligent agents that have been utilized and ask the user to input evaluation scores for each. In this case, the feedback data includes the evaluation scores assigned by the user.

[0165] Furthermore, the intelligent agent 112 can update the rating data each time it detects positive or negative feedback from the user, or it can accumulate feedback data and update the rating data at predetermined intervals based on the accumulated feedback data. In this case, new feedback data can be given greater weight.

[0166] As explained above, according to the third embodiment, the rating data can be appropriately updated based on feedback from users.

[0167] Fourth Implementation Method

[0168] It is conceivable that many third-party external intelligent agents will emerge in the future. In such a case, it would be preferable not to process user feedback only locally, but to share and utilize the evaluations of external intelligent agents across multiple vehicles.

[0169] The fourth embodiment is an implementation in which an intelligent agent (hereinafter referred to as the evaluation intelligent agent) is configured to evaluate an external intelligent agent, and the evaluation intelligent agent evaluates the external intelligent agent based on feedback data sent from multiple vehicle-mounted devices 10.

[0170] In the fourth embodiment, multiple vehicle-mounted devices 10 receive rating results from the evaluation agent and update the rating data 12D based on the results.

[0171] In the third embodiment, the feedback data reflects personal preferences. In contrast, in the fourth embodiment, the feedback data is used to evaluate the performance and reliability of external agents, such as the accuracy of information and the appropriateness of the timing of information delivery. Therefore, in the fourth embodiment, the user's ratings assigned to each external agent are used as feedback data, rather than the user's actions. Furthermore, the rating data is not based on a domain definition, but on the definition of each of the multiple external agents.

[0172] Figure 11This diagram illustrates the processing flow performed by the control unit 11 of the vehicle-mounted device 10 upon receiving a speech from the user in the fourth embodiment. As shown, in the fourth embodiment, the agent unit 112 sends the acquired feedback data to the evaluation agent. As described above, the feedback data in this embodiment includes evaluation scores for the external agents utilized. Therefore, the agent unit 112 can provide the user with a list of utilized external agents and allow them to input evaluation scores.

[0173] The intelligent agent unit 112 sends the feedback data to the evaluation server, establishing an association between the feedback data and the external intelligent agent of the object. If there are multiple external intelligent agents being evaluated, multiple sets of feedback data are also sent.

[0174] The feedback data sent is statistically analyzed by the evaluation server. The evaluation server statistically analyzes the evaluation scores contained in the feedback data obtained from multiple vehicle-mounted devices 10, and calculates an integrated evaluation score for each external agent. Alternatively, the integrated evaluation score can be calculated considering the influence of each user. The calculated integrated evaluation score is provided based on requests from the vehicle-mounted devices 10.

[0175] Additionally, the intelligent agent unit 112 updates the scoring data 12D based on the integrated evaluation score of each external intelligent agent provided by the evaluation server. The acquisition of the integrated evaluation score from the evaluation server and the updating of the scoring data can be performed at a predetermined period, or immediately after the feedback data is sent to the evaluation server.

[0176] Figure 12 This is a flowchart of the processing performed by the intelligent agent unit 112 in the fourth embodiment. The illustrated processing is... Figure 10 Similarly, step S18 (processing the response statement) is executed immediately afterward.

[0177] First, in step S31, the intelligent agent unit 112 receives the evaluation scores of each external intelligent agent used by the user.

[0178] Next, in step S32, the agent unit 112 generates feedback data containing the evaluation score and sends it to the evaluation agent. If multiple external agents exist for the object, multiple sets of feedback data are also generated.

[0179] Next, in step S33, the agent unit 112 receives the integrated evaluation scores for each of the multiple external agents from the evaluation agent and reflects them in the scoring data 12D possessed by the device. In this step, for example, a lower score can be assigned to an external agent with a lower integrated evaluation score, and a higher score can be assigned to an external agent with a higher integrated evaluation score.

[0180] According to this embodiment, the evaluations given by other users can be reflected in the rating data 12D. With this configuration, it is possible to more actively utilize external agents that receive high evaluations from multiple users, and less actively utilize external agents that receive low evaluations.

[0181] in, Figure 4 The diagram shows an example of a single rating defined in the rating data 12D, but the ratings included in the rating data can also include ratings reflecting personal preferences and ratings reflecting information from the evaluating agent. Furthermore, the agent unit 112 can integrate and use these ratings.

[0182] Variations

[0183] The above-described implementation is merely an example, and this disclosure can be implemented with appropriate modifications without departing from its spirit.

[0184] For example, the processing and methods described in this disclosure can be freely combined and implemented as long as they do not create technical contradictions.

[0185] Furthermore, while the embodiment illustrates configuring an external intelligent agent on an external device connected to the Internet, the external intelligent agent can also be configured on other devices. For example, the external intelligent agent can also be configured on an edge server accessible from the vehicle. Additionally, the external intelligent agent can be distributed across multiple edge servers or the cloud.

[0186] Furthermore, while an in-vehicle device is exemplified in the embodiments, the information processing device disclosed herein can also be implemented as an in-vehicle device or equipment.

[0187] In cases where external agents are configured on multiple edge servers, the vehicle-mounted device 10 can obtain the vehicle's location information and determine the edge server where the target external agent is configured based on this location information. For example, there may be cases where external agents providing geographic information are configured on multiple edge servers arranged geographically. In this case, the vehicle-mounted device 10 can, for example, determine the external agent configured on the geographically nearest edge server based on the vehicle's location information. The geographic information processed by the external agent may differ for each edge server. For example, an external agent providing geographic information for region A may be configured on an edge server configured in region A, and an external agent providing geographic information for region B may be configured on an edge server configured in region B. The vehicle-mounted device 10 determines the edge server corresponding to the region where the vehicle is located and obtains the geographic information for that region by accessing the external agent provided by that edge server.

[0188] Furthermore, a process described as being performed by one device can also be executed by multiple devices. Alternatively, a process described as being performed by different devices can be executed by a single device. In a computer system, it is possible to flexibly change the hardware configuration (server configuration) used to implement various functions.

[0189] This disclosure can also be implemented by supplying a computer program containing the functions described in the above embodiments to a computer, which then reads and executes the program using one or more processors. Such a computer program can be provided to the computer via a non-transitory computer-readable storage medium that can be connected to the system bus, or via a network. Non-transitory computer-readable storage media include, for example, any type of disk such as a hard disk (floppy disk, hard disk drive (HDD), etc.), an optical disk (CD-ROM, DVD, Blu-ray disc, etc.), read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic cards, flash memory, optical cards, and any type of medium suitable for storing electronic instructions.

Claims

1. An information processing device, wherein, It has a control unit that performs the following processes: The speech statements are obtained based on the user's speech. Based on the spoken statement, determine one or more external agents from among multiple external agents that are dedicated to multiple specified domains and provide natural language-based dialogue services as the destination of the spoken statement; Send the speech statement to one or more external agents that have been determined, and obtain one or more responses from the one or more external agents; as well as The multiple responses are combined, and the combined response is provided to the user. The control unit determines one or more external agents as the destination for the speech statement based on scores set for each of the plurality of external agents, or determines the integrated object in the one or more responses based on the scores, and updates the scores set for each of the plurality of external agents based on information obtained from the evaluation agent that evaluates each of the plurality of external agents.

2. The information processing apparatus according to claim 1, wherein, The control unit also obtains feedback data indicating the user's evaluation of the integrated response, and further sends the feedback data to the evaluation agent.

3. The information processing apparatus according to claim 1, wherein, The evaluation includes an assessment of the accuracy of the information provided by the plurality of external intelligent agents.

4. The information processing apparatus according to claim 1, wherein, The control unit transmits the speech to an external agent that has a score exceeding a predetermined value.

5. The information processing apparatus according to claim 1, wherein, The control unit will exclude responses from the integrated objects from those received by external agents with scores below a specified value.

Images