Vehicle-mounted device, mobile terminal, information presentation method, and information presentation system

Abstract

Upon detecting a request from a vehicle user, an inquiry destination determining unit (S120) determines a target LLM that is to serve as an inquiry destination, from among a plurality of LLMs, on the basis of the result of text analysis of the content of the request. An inquiry execution unit (S130) executes an inquiry with respect to the target LLM determined by the inquiry destination determining unit, via an accessing unit. A presenting unit (S140) presents, to the vehicle user, an answer from the target LLM obtained in response to the inquiry executed by the inquiry execution unit.

Description

In-vehicle device, mobile terminal, information presentation method, and information presentation system CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This international application claims priority based on Japanese Patent Application No. 2024-089419, filed with the Japan Patent Office on May 31, 2024, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to a technology for proposing services to vehicle users by utilizing multiple types of large-scale language models.

[0003] The following Patent Document 1 describes a technology relating to a dialogue system that uses a large-scale language model (hereinafter referred to as LLM) to obtain output using data accumulated in a vehicle.

[0004] JP 2024-43564 A

[0005] There are a wide variety of LLMs used in dialogue systems, including cloud LLMs provided by clouds such as Chat GPT, and in-vehicle LLMs installed in vehicles. However, conventional technologies assume the use of a single predetermined language model, which results in biased responses from the LLM, and a problem has been found in that accurate responses corresponding to the situation at hand are not necessarily obtained.

[0006] For example, the information source for constructing an in-vehicle LLM is limited to information about the vehicle. Moreover, the period during a user's life cycle when using a vehicle is short. This means that the accuracy of an in-vehicle LLM is reduced for personalized answers for situations unrelated to the vehicle's life cycle, or answers that require broad knowledge that cannot be covered by personal experience or knowledge.

[0007] One aspect of the present disclosure provides a technology that enables highly accurate answers to be easily obtained from a system that uses a large-scale language model.

[0008] One aspect of the present disclosure is an in-vehicle device comprising an access unit, a query determination unit, a query execution unit, and a presentation unit. The access unit is configured to access each of a plurality of LLMs, including an in-vehicle LLM and a mobile LLM. The in-vehicle LLM is a large-scale language model mounted on a vehicle and trained using information acquired from the vehicle. The mobile LLM is a large-scale language model mounted on a mobile terminal carried by a vehicle user and trained using information acquired from the mobile terminal. The query determination unit is configured to, upon detecting a request from the vehicle user, determine a target LLM to be queried from among the plurality of LLMs based on a result of text analysis of the content of the request. The query execution unit is configured to execute a query to the target LLM determined by the query determination unit via the access unit. The presentation unit is configured to present to the vehicle user a response from the target LLM obtained by the query execution unit.

[0009] With this configuration, by including the mobile LLM in the inquiry destination, it becomes possible to provide answers to inquiries from vehicle users that reflect information collected while they are dismounting the vehicle, as well as personal preferences unrelated to vehicle control, thereby providing highly accurate answers.

[0010] One aspect of the present disclosure is a mobile terminal carried by a vehicle user, comprising a mobile LLM and a mobile control unit. The mobile LLM is a large-scale language model that learns using information acquired from the mobile terminal. When the mobile control unit receives a query based on a request from the vehicle user from an in-vehicle device, it acquires a response to the query from the mobile LLM and returns the response to the in-vehicle device. The in-vehicle device is configured to be able to access each of multiple LLMs, including the in-vehicle LLM and the mobile LLM, which are large-scale language models that learn using information acquired from the vehicle.

[0011] With this configuration, it is possible to provide a response to an inquiry from the in-vehicle device that reflects information collected while the driver is out of the vehicle and personal preferences that are not related to vehicle control.

[0012] One aspect of the present disclosure is an information presentation method executed by an in-vehicle device. The in-vehicle device is configured to be able to access each of a plurality of LLMs, including an in-vehicle LLM and a mobile LLM. The in-vehicle LLM is a large-scale language model that is mounted on a vehicle and trains using information acquired from the vehicle. The mobile LLM is a large-scale language model that is mounted on a mobile terminal carried by a vehicle user and trains using information acquired from the mobile terminal. The information presentation method includes, upon detecting a request from a vehicle user, determining a target LLM to be queried from among the plurality of LLMs based on a result of text analysis of the content of the request, executing a query to the determined target LLM, and presenting information based on a response from the target LLM obtained by the query to the vehicle user.

[0013] By carrying out such a method, it is possible to obtain the same effects as those of the above-mentioned in-vehicle device.

[0014] One aspect of the present disclosure is an information presentation system including an on-board LLM, a mobile terminal, and an on-board device. The on-board LLM is a large-scale language model that is mounted on a vehicle and trains using information acquired from the vehicle. The mobile terminal is a mobile terminal carried by a vehicle user and is equipped with a mobile LLM, which is a large-scale language model that trains using information acquired from the mobile terminal. The on-board device is configured to acquire a request from the vehicle user and to be able to access each of multiple LLMs including the on-board LLM and the mobile LLM. The on-board device also includes an access unit, a query destination determination unit, a query execution unit, and a presentation unit. The on-board device is similar to the on-board device described above as one aspect of the present disclosure.

[0015] With this configuration, it is possible to obtain the same effects as the above-described in-vehicle device.

[0016] 1 is a block diagram showing the device configuration of a service proposal system. FIG. 2 is a block diagram showing the functional configuration of the service proposal system. FIG. 3 is an explanatory diagram illustrating the contents of a request pattern table. FIG. 4 is a flowchart of processing executed by an LLM orchestrator. FIG. 5 is a sequence diagram showing the operation of the system before determining a request pattern. FIG. 6 is a sequence diagram showing the operation of the system when the request pattern is A or B1. FIG. 7 is a sequence diagram showing the operation of the system when the request pattern is B2. FIG. 8 is a sequence diagram showing the operation of the system when an LLM orchestrator merges the results of inquiries to multiple LLMs to make a service proposal. FIG. 9 is a sequence diagram showing the operation of the system when an LLM orchestrator makes an inquiry to another LLM depending on the content of a response to an inquiry to a certain LLM. FIG. 10 is a sequence diagram showing the operation of the system when an LLM that has received an inquiry makes an inquiry to yet another LLM.

[0017] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0018] 1. Configuration The service proposal system 1 shown in FIG. 1 includes an in-vehicle device 10, a mobile terminal 30, and a cloud server 50.

[0019] The in-vehicle device 10 is mounted on a vehicle. Hereinafter, the vehicle mounting the in-vehicle device 10 will be referred to as the mounted vehicle 2. The in-vehicle device 10 communicates with a mobile terminal 30 and a cloud server 50. The in-vehicle device 10 is connected to an in-vehicle network of the mounted vehicle 2, and collects various vehicle data and performs various vehicle controls via the in-vehicle network. Any application program (hereinafter, referred to as an app) can be installed on the in-vehicle device 10, and various functions may be realized by executing the installed app.

[0020] The in-vehicle device 10 includes a control unit 11 , a communication unit 12 , a storage unit 13 , and a vehicle I / F unit 14 .

[0021] The control unit 11 is an electronic control device mainly composed of a microcomputer including a CPU 111, a ROM 112, a RAM 113, and the like.

[0022] The communication unit 12 performs wireless communication with the mobile terminal 30 and the cloud server 50. For example, Bluetooth may be used for communication with the mobile terminal 30. Bluetooth is a registered trademark. For example, a wide area communication network may be used for communication with the cloud server 50.

[0023] The storage unit 13 stores at least an in-vehicle large-scale language model (hereinafter referred to as in-vehicle LLM) 25, which will be described later. The LLM is a language model constructed using an extremely large data set and deep learning technology, and is capable of fluent conversation close to that of a human, and can perform various processes using natural language with high accuracy.

[0024] The vehicle I / F unit 14 is connected to various on-board devices via an on-board network of the vehicle 2 and acquires various information from the on-board devices. The on-board network may include CAN and Ethernet. CAN is an abbreviation for Controller Area Network. Ethernet is a registered trademark. The on-board devices connected to the vehicle I / F unit 14 may include devices that are originally installed in the vehicle as well as exterior devices that are added later. The on-board devices may include sensors, cameras, audio devices, display devices, etc.

[0025] The mobile terminal 30 is, for example, a smartphone, a tablet, or the like carried by a vehicle user of the vehicle 2. The mobile terminal 30 communicates with the in-vehicle device 10 and the cloud server 50. The mobile terminal 30 may execute various application programs that are arbitrarily installed.

[0026] The mobile terminal 30 includes a control unit 31 , a communication unit 32 , and a storage unit 33 .

[0027] The control unit 31 is an electronic control device mainly composed of a microcomputer including a CPU 311, a ROM 312, a RAM 313, and the like.

[0028] The communication unit 32 performs wireless communication with the in-vehicle device 10 and the cloud server 50. For example, Bluetooth may be used for communication with the in-vehicle device 10. For example, a wide area communication network may be used for communication with the cloud server 50.

[0029] The storage unit 33 stores at least a mobile LLM 41, which will be described later.

[0030] The cloud server 50 is connected to, for example, a wide area communication network that can be accessed by the in-vehicle device 10 and the mobile terminal 30 .

[0031] The cloud server 50 includes a control unit 51 , a communication unit 52 , and a storage unit 53 .

[0032] The control unit 51 is an electronic control device mainly composed of a microcomputer including a CPU 511, a ROM 512, a RAM 513, and the like.

[0033] The communication unit 52 performs wireless communication with the in-vehicle device 10 and the mobile terminal 30. For communication with the in-vehicle device 10 and the mobile terminal 30, a wide area communication network may be used, for example.

[0034] The storage unit 53 stores at least a cloud LLM 61, which will be described later.

[0035] The various functions of the microcomputers in the above-described control units 11, 31, and 51 are realized by the CPUs 111, 311, and 511 executing programs stored in non-transitory storage media. In this example, the ROMs 112, 312, and 512 correspond to the non-transitory storage media storing the programs. Furthermore, the execution of these programs results in the execution of methods corresponding to the programs. Note that some or all of the functions executed by the CPUs 111, 311, and 511 may be implemented in hardware using one or more ICs, etc. Furthermore, the number of microcomputers constituting each control unit 11, 31, and 51 may be one or more.

[0036] [2. Functional Configuration] Various functions provided in the service proposal system 1 will be described.

[0037] 2, the mobile terminal 30 includes a mobile LLM 41, a mobile information source 42, and an LLM learning unit 43. These functions of the mobile terminal 30 may be realized by processing executed by a microcomputer constituting the control unit 31.

[0038] The mobile LLM 41 is a large-scale language model that learns using various information acquired from a mobile information source 42. The information acquired from the mobile information source 42 may include information generated by the mobile terminal 30 and information obtained from an application running on the mobile terminal 30. The information acquired from the mobile information source 42 includes personalized information that reflects the personal characteristics, hobbies, and preferences of the user of the mobile terminal 30 (i.e., the vehicle user).

[0039] The information acquired from the mobile information source 42 may include application usage information 421, terminal usage information 422, health care information 423, and history information 424. These pieces of information may be collected by executing an application installed on the mobile terminal 30. Note that the application may be pre-installed on the mobile terminal 30, or may be installed arbitrarily by the user of the mobile terminal 30.

[0040] Specifically, personalized information may be acquired by collecting and learning related information from public APIs, such as a healthcare app, a map app, a music app, or a search app (i.e., a browser search function). An app (hereinafter, a mobile LLM app) that builds a personal profile based on the collected and learned related information is then installed and run on the user's mobile device. The mobile LLM app may also have a function to collect and learn about topics of interest and build a personal profile by collecting conversation history from the mobile device's voice recognition function or a chat function provided in the mobile LLM app.

[0041] The application usage information 421 is information about the usage status and usage history of various applications installed on the mobile terminal 30. The applications may be classified by category, such as for work, for hobbies, and for recording.

[0042] The terminal usage information 422 is information about where the mobile terminal 30 is being used (e.g., location information), what the mobile terminal 30 is being used for, etc. For example, this may include use as a communication terminal, use as a terminal for listening to music or video content, use as a terminal for acquiring or transmitting various information, etc. The health care information 423 is information about the user's health condition and behavioral history collected using the functions of the mobile terminal 30 or a device linked to the mobile terminal 30. The history information 424 may include information about the history of use of the search function of the mobile terminal 30, and the history of outgoing and incoming calls and emails. Furthermore, the history information 424 may include information about the user's comment history collected using the voice recognition function of the mobile terminal 30 or a chat function that allows the mobile terminal 30 to converse with the user using voice, text, etc.

[0043] In other words, the mobile information source 42 provides various information that reflects the user's hobbies, preferences, and behavioral trends throughout the user's life cycle. Therefore, when a user makes an inquiry to the mobile LLM 41, the user can receive an answer that takes into account the user's characteristics. The mobile LLM 41 may perform text analysis of the inquiry content and transfer the inquiry to the cloud LLM 61 based on the results. For example, if there is a 3rd Party app 623 that may be able to quote information in a related genre, the inquiry may be transferred to the cloud LLM 61.

[0044] The LLM training unit 43 uses information obtained from the mobile information source 42 and training data generated by the LLM orchestrator 24 to train and retrain the mobile LLM 41 .

[0045] The cloud server 50 includes a cloud LLM 61, a cloud information source 62, and an LLM learning unit 63. These functions of the cloud server 50 may be realized by processing executed by a microcomputer constituting the control unit 51.

[0046] The cloud LLM 61 is a large-scale language model trained using large-scale knowledge obtained from cloud information sources 62 .

[0047] The cloud information source 62 may include a public LLM 621, various Web sites 622, and a 3rd Party application 623, etc.

[0048] The public LLM 621 may include, for example, Chat GPT, GPTs, etc. The various Web 622 may include a search engine, a video distribution service, an SNS, etc. The 3rd Party app may include various word-of-mouth sites, a map information providing site, etc.

[0049] The LLM learning unit 63 uses information obtained from the cloud information source 62 and learning data generated by the LLM orchestrator 24 to learn and re-learn the cloud LLM 61 .

[0050] The cloud server 50 is able to make a wide range of proposals that are not biased towards individual preferences, based on the large amount of knowledge that the cloud LLM 61 possesses.

[0051] In other words, a wide range of large-scale information can be obtained from an unspecified number of people from the cloud information source 62. Therefore, when making an inquiry to the cloud LLM 61, it is possible to obtain a response that takes into account general trends in society and a response that goes beyond the range that an individual can imagine.

[0052] The in-vehicle device 10 includes an HMI unit 21, an image recognition unit 22, a voice recognition unit 23, an LLM orchestrator 24, an in-vehicle LLM 25, and an LLM learning unit 26. These functions of the in-vehicle device 10 may be realized by processing executed by a microcomputer constituting the control unit 11. In addition, the in-vehicle devices connected via the vehicle I / F unit 14 function as in-vehicle information sources 7.

[0053] The on-board LLM 25 is a large-scale language model trained using information obtained from the on-board information source 7 .

[0054] The LLM learning unit 26 uses information obtained from the in-vehicle information source 7 and learning data generated by the LLM orchestrator 24 to perform learning and relearning of the in-vehicle LLM 25 .

[0055] The information obtained from the in-vehicle information source 7 may include occupant characteristic information 71, image information 72, vehicle state information 73, input signal / connection history information 74, operation history information 75, and the like.

[0056] The occupant characteristic information 71 may include the age and gender of the occupant, emotions and drowsiness read from the occupant's facial expressions, the duration of the ride, etc. The image information 72 may include video and still image information obtained from an in-vehicle camera, an outside vehicle camera, a drive recorder, etc. The vehicle status information 73 may include diagnostic information transmitted through an in-vehicle network, control communication information for vehicle control, and detection information from various sensors. The input signal / connection history information 74 may include input audio signals, communication connection information such as Wi-Fi and Bluetooth, and connection information for external devices such as USB. Wi-Fi is a registered trademark. The operation history information 75 may include a history of vehicle operations and in-vehicle device operations, such as steering wheel operation, accelerator / brake operation, navigation operation, air conditioner operation, and door opening / closing operation. Navigation operations may include destination setting and music playback.

[0057] That is, information that reflects the vehicle user's hobbies, preferences, and behavioral tendencies, etc., can be obtained from the in-vehicle information source 7 through the vehicle user's behavior inside the vehicle. Therefore, when an inquiry is made to the in-vehicle LLM 25, an answer that reflects the characteristics of the vehicle user in a limited situation related to the vehicle can be obtained.

[0058] The HMI unit 21 includes an audio device, a display device, etc., and is used to receive voice input from a vehicle user of the vehicle 2 and to present various types of information to the vehicle user audibly or visually. HMI is an abbreviation for Human Machine Interface.

[0059] The image recognition unit 22 executes processing to extract, for example, information that serves as a trigger for processing from the image information provided by the in-vehicle information source 7 .

[0060] The voice recognition unit 23 recognizes voice input via the HMI unit 21 and converts it into text data.

[0061] The LLM orchestrator 24 receives requests from the user via the HMI unit 21 and executes queries to the in-vehicle LLM 25, the mobile LLM 41, and the cloud LLM 61. The LLM orchestrator 24 also proposes services to the user in accordance with the request based on the responses obtained from each of the LLMs 25, 41, and 61. Furthermore, the LLM orchestrator 24 obtains the user's response to the proposed service via the HMI unit 21, and executes the proposed service and re-proposes the service.

[0062] The LLM orchestrator 24 performs processing by, for example, executing a pre-installed application. By making the application a target for OTA and updating it as needed, it becomes possible to easily change the control logic used by the LLM orchestrator 24. OTA stands for Over The Air, and refers to updating software via wireless communication.

[0063] The LLM orchestrator 24 analyzes the textual request, determines which request pattern it corresponds to according to a pattern table prepared in advance, and executes processing according to the determined request pattern.

[0064] As shown in FIG. 3, the pattern table is a table that associates request patterns with processing patterns that indicate how to process LLM groups.

[0065] The request patterns may be classified into, for example, those without characteristics (hereinafter, request pattern A) and those with characteristics (hereinafter, request pattern B). Furthermore, those with characteristics may be classified into driving-related (hereinafter, request pattern B1), behavior-related (hereinafter, request pattern B2), and general-related (hereinafter, request pattern B3). The request patterns are not limited to request patterns A and B1 to B3, and may be classified into more or fewer patterns.

[0066] Request pattern A is a vague request that includes non-descriptive keywords. For example, a request from a vehicle user such as "Please suggest some services" corresponds to request pattern A. In this case, the processing pattern is to query each LLM in turn according to a query list preset for each LLM 25, 41, 61 belonging to the LLM group, and then propose a service to the user based on the query results. If a negative response is received from the user to the proposal, a query is made to the next LLM set in the order according to the query list. Thereafter, the same process is repeated until a positive response is received from the user or until all of the ordered LLMs have been queried.

[0067] In this embodiment, in the case of request pattern A, the inquiry list is set in the order of the in-vehicle LLM 25, the mobile LLM 41, and the cloud LLM 61.

[0068] Request pattern B1 is a request that includes keywords related to driving characteristics. For example, requests such as "Tell me about malfunction information," "Do ... from in-car camera information," and "What is the in-car environment like?" Keywords related to driving characteristics may include words related to in-car devices, words related to driving operations, words related to the state of the vehicle, etc.

[0069] In the processing pattern associated with request pattern B1, the first query destination in the query list is set to the on-board LLM 25, and thereafter, an order is set arbitrarily for the LLMs 41 and 61 other than the on-board LLM 25. In other words, in request pattern B1, the same processing as in request pattern A is executed, except that the second and subsequent query destinations in the query list may be different from those in request pattern A.

[0070] Request pattern B2 is a request that includes keywords related to personal characteristics. For example, requests such as "Play music like...", "Is there a place like... nearby?", "Provide a car space that suits my health condition", etc. Keywords related to personal characteristics may also include keywords that are closely related to personal hobbies and preferences.

[0071] In the processing pattern associated with request pattern B2, the first query destination in the query list is set to mobile LLM 41, and thereafter, an arbitrary order is set for LLMs 25 and 61 other than mobile LLM 41. In other words, in request pattern B2, the same processing as in request patterns A and B1 is executed, except that the first query destination in the query list is different from request patterns A and B1.

[0072] Request pattern B3 is a request that includes keywords related to general information. For example, requests such as "generally...," "popular...," "recommended...," "search for...," etc. Keywords related to general information may include keywords that suggest disregarding personal hobbies and preferences.

[0073] In the processing pattern associated with request pattern B3, the first query destination in the query list is set to cloud LLM 61, and thereafter, an arbitrary order is set for LLMs 25 and 41 other than cloud LLM 61. In other words, in request pattern B3, the same processing as in request patterns A, B1, and B2 is executed except that the first query destination in the query list is different from that in request patterns A, B1, and B2.

[0074] [1-2. Processing] Next, the processing executed by the control unit 11 functioning as the LLM orchestrator 24 will be described with reference to the flowchart of FIG.

[0075] This process is started when a voice request from the user is input via the HMI section 21 and converted into text by the voice recognition section 23 .

[0076] In S110, the control unit 11 analyzes the request content that has been converted into text.

[0077] In S120, the control unit 11 performs pattern determination to determine whether the input request corresponds to request pattern A, B1, or B3, in accordance with the analysis result in S110.

[0078] In S130, the control unit 11 refers to the pattern table based on the pattern determination result in S120, extracts a processing pattern associated with the determination result, and sets the LLM set at the top of the inquiry list in the extracted processing pattern as the target LLM.The control unit 11 then executes an inquiry to the target LLM according to the request content and obtains a response to the inquiry from the target LLM.

[0079] In addition, if the waiting time until a response is received from the target LLM exceeds the allowable time or there is a possibility that the allowable time will be exceeded, the control unit 11 may notify the vehicle user via the HMI unit 21 that it will take some time to receive a response.

[0080] In S140, the control unit 11 proposes a service to the vehicle user via the HMI unit 21 in accordance with the answer acquired in S130. The control unit 11 also checks the vehicle user's response to the proposed service via the HMI unit 21. If the response from the vehicle user is affirmative, the control unit 11 determines that the vehicle user intends to implement the service, and if the response from the vehicle user is negative, the control unit 11 determines that the vehicle user does not intend to implement the service. Note that the response from the vehicle user may include, in addition to a voice response, a response by a gesture of the vehicle user obtained from an image of the vehicle user's upper body.

[0081] In S150, the control unit 11 generates learning data that associates the inquiry content to the target LLM, the response from the target LLM, the service proposal content based on the response, and the vehicle user's response to the proposal, and transmits the data to each LLM 25, 41, 61. The learning data may be transmitted each time it is generated, or the generated learning data may be temporarily stored and transmitted collectively at a predetermined transmission timing. The learning data may also be transmitted to all LLMs, or may be transmitted only to the target LLM that was the destination of the inquiry. Each LLM 25, 41, 61 may re-learn the LLM 25, 41, 61 according to the notified learning data.

[0082] In S160, the control unit 11 confirms the vehicle user's intention to carry out the operation in S140, and if it is confirmed that the vehicle user has the intention to carry out the operation, it proceeds to S170, and if it is confirmed that the vehicle user has no intention to carry out the operation, it proceeds to S190.

[0083] In S170, the control unit 11 obtains consent from the vehicle user for the implementation of the proposed service. Specifically, the control unit 11 requests consent from the vehicle user via the HMI unit 21, and determines that consent has been obtained when a response from the vehicle user indicating consent is detected via the HMI unit 21. The response from the vehicle user indicating consent may be an operation of a predetermined in-vehicle switch or the like, or may be a voice or gesture indicating consent.

[0084] In S180, the control unit 11 instructs the processing unit that provides the service for which the vehicle user's consent was obtained in S170 to provide the service, and then ends the process. The processing unit that provides the service may include, for example, an air conditioner, a music player, etc. If the vehicle user's consent is not obtained within a certain time, the control unit 11 may notify the vehicle user via the HMI unit 21 that the process will end due to time-out, and then end the process.

[0085] In S190, the control unit 11 determines whether or not there is an unqueried LLM in the query list for the processing pattern extracted in S130. If the control unit 11 determines that there is an unqueried LLM, the control unit 11 proceeds to S130. If the control unit 11 determines that there is no unqueried LLM, the control unit 11 proceeds to S200.

[0086] When the process proceeds from S190 to S130, the control unit 11 performs a query on the first LLM among the unqueried LLMs in the query list as the target LLM.

[0087] In S200, the control unit 11 outputs an instruction to the vehicle user via the HMI unit 21 to prompt the vehicle user to make a request again.

[0088] In S210, the control unit 11 determines whether a re-request from the vehicle user has been detected, and if a re-request is detected within a certain period of time, the control unit 11 returns the processing to S110, and if a re-request is not detected after the certain period of time has elapsed, the control unit 11 terminates the processing.

[0089] 3. Operation 3-1. Common Sequence A sequence showing the operation up to when the LLM orchestrator 24 determines the pattern of a request from a vehicle user will be described with reference to FIG.

[0090] In S1, each sensor belonging to the in-vehicle information source 7 notifies the LLM orchestrator 24 of its detection result.

[0091] In S2, the LLM orchestrator 24 analyzes the detection result to detect a service proposal trigger. The service proposal trigger is used by the LLM orchestrator 24 to initiate a conversation that requests a request from the vehicle user, triggered by some behavior of the vehicle user or a change in the environment including the vehicle user. Specifically, the service proposal trigger may be the entry of occupants including the vehicle user, the occupant's gestures, the occupant's conversation, a change in the occupant's physical condition, a change in the vehicle's driving state, a change in the vehicle interior environment, or the like. Changes in the vehicle interior environment may include a change in the temperature inside the vehicle, as well as confirmation of a Bluetooth connection between the in-vehicle device 10 and the mobile terminal 30.

[0092] In S3, the LLM orchestrator 24 executes an inquiry to the vehicle user via the HMI unit 21. The inquiry here may be, for example, a general inquiry such as "Please let me know if you need anything," triggered by the vehicle user getting into the vehicle. Alternatively, the inquiry may be a personalized inquiry such as "Is there anywhere you are looking for?" or "Shall I play some music?" based on trends estimated from stored content of interactions during past rides.

[0093] In S4, a request from the vehicle user is input to the LLM orchestrator 24 via the HMI unit 21.

[0094] In S5, the LLM orchestrator 24 analyzes the request content and determines the request pattern, as described above in S120.

[0095] In S6, the LLM orchestrator 24 executes a query to each of the LLMs 25, 41, and 61 according to the query list of the processing pattern associated with the determined request pattern.

[0096] The request in S4 may be input regardless of S1 to S3. For example, speech input after pressing the speech recognition start button, or speech input after a trigger to start speech recognition, such as "Hey XX" or "OK ​​XX," may be recognized as a request.

[0097] 3-2. Request Patterns A and B1 Next, the sequence for request patterns A and B1 will be described with reference to FIG.

[0098] In both request patterns A and B1, the in-vehicle LLM 25 is set as the first inquiry destination in the inquiry list, and therefore the operations are the same.

[0099] In S11, the LLM orchestrator 24 executes an inquiry according to the request content to the in-vehicle LLM 25, which is the first inquiry destination in the inquiry list.

[0100] In S12, the in-vehicle LLM 25 returns a response to the inquiry to the LLM orchestrator 24.

[0101] As shown in S13, the LLM orchestrator 24 may transmit a processing status notification to the vehicle user via the HMI unit 21, informing the vehicle user that it will take some time to respond to the request. In accordance with the processing status notification, the HMI unit 21 notifies the vehicle user by, for example, issuing a voice message or a message on the navigation screen saying, "We are currently responding, so please wait a moment."

[0102] The processing status notification may be sent, for example, when no response is received within an allowable time from the inquiry to the target LLM, or when it is expected that it will take longer than the allowable time to receive a response. The processing status notification may be sent whenever an inquiry is made to the target LLM, or may be sent when no response is received within a predetermined time set to be less than the allowable time after the inquiry is made. Whether or not to send a processing status notification may be determined for each LLM.

[0103] In S14, the LLM orchestrator 24 proposes a service corresponding to the response received from the in-vehicle LLM 25 to the vehicle user via the HMI unit 21. Only one or more proposed services may be displayed.

[0104] In S15, the HMI unit 21 checks whether the vehicle user intends to use the proposed service based on the vehicle user's response to the proposed service. If a positive response is obtained by voice, gesture, or the like, it is determined that the vehicle user intends to use the proposed service. If multiple services are proposed, it is also determined which service the vehicle user has selected.

[0105] In S16, the HMI unit 21 transmits the response from the vehicle user to the LLM orchestrator 24 as a result notification.

[0106] In S17, the LLM orchestrator 24 generates learning data based on the result notification and transmits it to each of the LLMs 25, 41, and 61.

[0107] In S18, each of the LLMs 25, 41, and 61 may re-learn the LLMs 25, 41, and 61 based on the received learning data. Each of the LLMs 25, 41, and 61 can arbitrarily determine which learning data to use for re-learning and at what timing.

[0108] If the result notification obtained in the previous S16 is positive, that is, if the vehicle user has an intention to implement the proposed service, in S21 the LLM orchestrator 24 notifies the vehicle user of an acceptance request via the HMI unit 21. The acceptance request confirms with the vehicle user whether or not the proposed service is really acceptable to be implemented.

[0109] In S22, the HMI unit 21 waits for the vehicle user to input an approval operation.

[0110] In S23, when the vehicle user inputs an approval operation, the HMI unit 21 transmits a result notification to the LLM orchestrator 24.

[0111] If the LLM orchestrator 24 receives a positive result notification regarding the implementation of the service, it instructs each part of the vehicle to execute the proposed service in S24, thereby providing the service to the vehicle user.

[0112] If the result notification obtained in the previous S16 is negative, i.e., if the user has no intention to implement the proposed service, in S25, the first LLM among the unqueried LLMs in the query list is set as the target LLM, and a query is executed to the target LLM. If the target LLM is the mobile LLM 41, the processes of S31 to S33 and subsequent processes in FIG. 7, which will be described later, are executed. If the target LLM is the cloud LLM 61, the processes of S41 to S43 and subsequent processes in FIG. 8, which will be described later, are executed.

[0113] [3-3. Case of Request Pattern B2] The sequence in the case of request pattern B2 will be described with reference to FIG.

[0114] In S31, the LLM orchestrator 24 executes an inquiry according to the request content to the mobile LLM 41 set as the first inquiry destination in the inquiry list.

[0115] In S32, the mobile LLM 41 returns a response to the inquiry to the LLM orchestrator 24.

[0116] The LLM orchestrator 24 may transmit a processing status notification to the HMI unit 21 in S33, as in the cases of request patterns A and B1.

[0117] The subsequent processes from S14 to S18 and S21 to S25 are the same as those in the cases of request patterns A and B1.

[0118] [3-4. Request Pattern B3] The sequence for request pattern B3 will be described with reference to FIG.

[0119] In S41, the LLM orchestrator 24 executes an inquiry according to the request content to the cloud LLM 61 set as the first inquiry destination in the inquiry list.

[0120] In S42, the cloud LLM 61 returns a response to the inquiry to the LLM orchestrator 24.

[0121] The LLM orchestrator 24 may transmit a processing status notification to the HMI unit 21 in S43, as in the cases of request patterns A and B1.

[0122] The subsequent processes from S14 to S18 and S21 to S25 are the same as those in the cases of request patterns A and B1.

[0123] 4. Use Cases 4-1. Baby Soothing Function A case in which the LLMs 25, 41, and 61 cooperate to propose a strategy to stop a baby from crying will be described with reference to FIGS.

[0124] In this case, the baby's cry is the trigger for suggesting a service that is detected in step S2. Then, in step S3, a voice message is sent to the vehicle user, asking, for example, "Are you OK?"

[0125] In S4, when a voice request such as "Is there any way to stop a baby from crying?" is input from the vehicle user, in S5, the request pattern is determined to be, for example, request pattern B3 (i.e., characteristic: general type).

[0126] Based on this determination result, the LLM orchestrator 24 first inquires of the cloud LLM 61 about "ways to stop a baby from crying" in S41. The cloud LLM 61 responds in S42 with "playing music that calms the baby" from among the general ways to stop a baby from crying stored in the cloud LLM 61. The LLM orchestrator 24 proposes a service to the vehicle user in S14 according to the response.

[0127] If the vehicle user responds positively to the service proposal in S15, the LLM orchestrator 24, having received the result notification in S16, receives consent to the implementation of the service from the vehicle user in S21 to S23, and then implements the service of "playing music to calm the baby" in S24.

[0128] If the vehicle user responds negatively to the service proposal in S15, such as "No, because it will have the opposite effect," the LLM orchestrator 24, which has received the result notification in S16, inquires of the in-vehicle LLM 25, which is the next inquiry destination shown in the inquiry list, about "how to stop a baby from crying." Based on the information stored in the in-vehicle LLM 25 that "children often cry when riding in the car in the evening," the in-vehicle LLM 25 responds by suggesting "opening and closing windows to let in fresh air." The LLM orchestrator 24 proposes a service to the vehicle user according to the response.

[0129] If the vehicle user responds positively to the service proposal, the LLM orchestrator 24 receives consent from the vehicle user to implement the service and then implements the service of "changing the air by opening and closing the windows."

[0130] If the vehicle user responds negatively to the service proposal, the LLM orchestrator 24 inquires of the mobile LLM 41, which is the next contact on the inquiry list, about "how to stop a baby from crying." The mobile LLM 41 responds by offering to "provide guidance to nearby facilities where breastfeeding is available" based on information such as "frequency of milk drinking and time elapsed since the last time" from a growth record app stored in the mobile LLM 41. The LLM orchestrator 24 proposes a service to the vehicle user based on the response.

[0131] If the vehicle user responds positively to the proposed service, the service is implemented in the same manner as described above. If the vehicle user responds negatively to the proposed service, the process may be terminated by notifying the vehicle user that there are no services that can be proposed, or the process may be repeated by changing the way questions are posed to the LLM.

[0132] [4-2. Interior Space Creation] A case where the LLMs 25, 41, and 61 cooperate to make suggestions regarding interior space creation will be described with reference to FIGS. 5 and 7. FIG.

[0133] In this case, for example, the service suggestion trigger detected in S2 is the establishment of a Bluetooth connection between the mobile terminal 30 and the in-vehicle device 10. Then, in S3, for example, a voice prompt is issued to the vehicle user asking, "Is there anything you would like us to do?"

[0134] In S4, when the vehicle user inputs a voice request, for example, "play some music," in S5, the request is determined to be request pattern B2 (i.e., characteristic: behavioral) based on the behavioral keyword "music."

[0135] Based on this determination result, the LLM orchestrator 24 first inquires about "music" from the mobile LLM 41 in S31. The mobile LLM 41 responds in S32 by selecting a "song by a certain artist" based on information about "a certain artist whose music the vehicle user regularly searches for or purchases from on the mobile terminal 30" stored in the mobile LLM 41. Based on the response, the LLM orchestrator 24 proposes to the vehicle user in S14 a service that plays the selected "song by a certain artist."

[0136] If the vehicle user responds affirmatively to the service proposal, the service is implemented as described above. If the vehicle user responds negatively to the service proposal, such as "I want a different song," the LLM orchestrator 24 queries the cloud LLM 61, the next query destination in the query list, for "songs by a certain artist." In other words, it can be interpreted that the vehicle user's response does not reject the "certain artist" part. Based on the "abundant information on popular songs by a certain artist" stored in the cloud LLM 61, the cloud LLM 61 responds with "the latest songs by a certain artist," "songs popular among core fans of a certain artist," "songs by other artists with similar atmospheres," etc. The LLM orchestrator 24 proposes a service to the vehicle user based on the response.

[0137] The subsequent operations are the same as those explained above and will be omitted.

[0138] [4-3. Guidance to the best restaurant] The function of providing guidance to the best restaurant using LLM will be described.

[0139] In this case, there is no service suggestion trigger, and the LLM orchestrator 24 recognizes the voice input of "I'm hungry" uttered by the vehicle user as a request in S4, and determines it as request pattern A (i.e., no features) in S5.

[0140] Based on this determination result, the LLM orchestrator 24 first inquires about "restaurants" from the onboard LLM 25 in S11. In S32, the onboard LLM 25 responds with "restaurants near the current location of the vehicle" using information obtained from the onboard information source 7. The LLM orchestrator 24 suggests restaurants near the vehicle to the vehicle user based on the response.

[0141] If the vehicle user responds positively to the proposed service, a service of route guidance to the proposed restaurant is provided using the same procedure as described above. If the vehicle user responds negatively to the proposed service, the LLM orchestrator 24 queries the cloud LLM 61, which is the next query destination listed in the query list for request pattern A, for a "restaurant." The cloud LLM 61 replies with highly rated restaurants regardless of genre, based on, for example, a wealth of information stored in the cloud LLM 61, such as review sites. The LLM orchestrator 24 suggests the "restaurant" to the vehicle user according to the content of the response.

[0142] If the vehicle user responds positively to the service proposal, the service is implemented as described above. If the vehicle user again responds negatively to the service proposal, the LLM orchestrator 24 queries the mobile LLM 41, which is the next query destination listed in the query list for request pattern A, for "restaurants." Based on information stored in the mobile LLM 41 that the vehicle user's preference is "salt ramen," the mobile LLM 41 responds with "highly rated salt ramen restaurants near the vehicle." The LLM orchestrator 24 proposes a service to the vehicle user based on the response.

[0143] The subsequent operations are the same as those explained above and will be omitted.

[0144] [4. Correspondence of Terms] In this embodiment, the communication units 12, 32, and 52 correspond to the access unit of the present disclosure. In this embodiment, S120 corresponds to the inquiry destination determination unit of the present disclosure, S130 corresponds to the inquiry execution unit of the present disclosure, and S140 corresponds to the presentation unit and response confirmation unit of the present disclosure. In this embodiment, S150 corresponds to the learning data provision unit, and S180 corresponds to the instruction unit. In this embodiment, S1 to S3 correspond to the request promotion unit of the present disclosure. In this embodiment, the control unit 31 of the mobile terminal 30, which performs processing to obtain a response to an inquiry from the LLM orchestrator 24 from the mobile LLM 41 and return it to the LLM orchestrator 24, corresponds to the mobile control unit of the present disclosure.

[0145] 5. Effects According to the first embodiment described above in detail, the following effects are achieved.

[0146] (5a) In the service proposal system 1, the LLM orchestrator 24 coordinates and uses multiple LLMs 25, 41, and 61, each of which has a different information source 7, 42, and 62, to propose services in response to requests from vehicle users. Therefore, it is possible to realize a variety of service proposals with high accuracy, not limited to services that focus on individual characteristics or services based on a broad perspective that is not dependent on individual characteristics.

[0147] (5b) The service proposal system 1 uses the in-vehicle LLM 25 present in the in-vehicle device 10, the mobile LLM 41 mounted on the mobile terminal 30 connected to the in-vehicle device 10 via proximity communication such as Bluetooth, and the cloud LLM 61 mounted on the cloud server 50 on the wide-area communication network in cooperation with each other. Therefore, even if cooperation with the cloud LLM 61 is not possible due to a poor communication environment, for example, service proposals can be realized with a certain level of accuracy or higher by cooperation between the in-vehicle LLM 25 and the mobile LLM 41.

[0148] (5c) The LLM orchestrator 24 not only waits for user-initiated requests, but also detects triggers and initiates conversations to elicit requests from vehicle users. This allows for efficient provision of services according to the circumstances that triggered the trigger.

[0149] (5d) If the waiting time from the request to the service proposal is long or there is a possibility that it will be long, such as if it takes a long time to receive a response to an inquiry to the LLM, the LLM orchestrator 24 notifies the vehicle user of this fact by sending a processing status notification via the HMI unit 21. This reduces the stress of the vehicle user who uses the service proposal system 1.

[0150] 6. Other Embodiments Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments and can be implemented in various modifications.

[0151] (6a) In the above embodiment, the LLM orchestrator 24 proposes a service to the vehicle user each time it executes a query to one LLM according to the query list. For example, as shown in FIG. 9 , queries may be executed in parallel to multiple LLMs, and the responses may be merged to propose a service to the vehicle user. In this case, the query list may indicate that queries are executed simultaneously to multiple LLMs.

[0152] 9 , in S51 and S53, the LLM orchestrator 24 executes an inquiry according to the request content to the in-vehicle LLM 25 and the mobile LLM 41, which are the first candidates in the inquiry list. In S52, the in-vehicle LLM 25 returns a response to the inquiry to the LLM orchestrator 24. In S54, the mobile LLM 41 returns a response to the inquiry to the LLM orchestrator 24.

[0153] As in the case of request patterns A and B1, the LLM orchestrator 24 may send a processing status notification to the HMI unit 21 in S55, thereby notifying the vehicle user via the HMI unit 21 that it will take some time to receive a response.

[0154] When the LLM orchestrator 24 receives responses from the in-vehicle LLM 25 and the mobile LLM 41, in S56, the LLM orchestrator 24 creates a service proposal by merging the response contents.

[0155] In S57, the LLM orchestrator 24 proposes a service to the vehicle user in accordance with the created proposal content via the HMI unit 21. The proposed service may be one or more.

[0156] For example, in the use case shown in [4-3. Guidance to the Best Restaurant], if the vehicle user responds negatively to a service proposal based on a response from the in-vehicle LLM 25, subsequent queries to the cloud LLM 61 and the mobile LLM 41 may be made simultaneously rather than sequentially.

[0157] (6b) In the above embodiment, the response obtained from the LLM is used as is to make a service proposal. For example, if the response obtained from the LLM is analyzed and it is determined that a further inquiry is necessary, a further inquiry may be made to another LLM, and a service proposal may be made by merging the results of the further inquiry.

[0158] In this case, as shown in FIG. 10, steps S51 to S55 are the same as those described with reference to FIG.

[0159] When the LLM orchestrator 24 receives responses from the onboard LLM 25 and the mobile LLM 41, the LLM orchestrator 24 analyzes the response content in S61. If the LLM orchestrator 24 determines that it is necessary to narrow down the response content and propose it based on the preferences of the vehicle user, for example, the LLM orchestrator 24 executes a query determined to be necessary from the analysis results to the mobile LLM 41 in S62. In S63, the mobile LLM 41 returns a response to the query to the LLM orchestrator 24.

[0160] When the LLM orchestrator 24 receives a response from the mobile LLM 41, in S64, the LLM orchestrator 24 creates a service proposal by merging the responses obtained in S52 and S54 with the response obtained in S63. In S65, the LLM orchestrator 24 makes a service proposal to the vehicle user via the HMI unit 21 in accordance with the created proposal.

[0161] For example, in the use case shown in [4-3. Guidance to the best restaurant], the service proposal based on the response from the cloud LLM 61 may be omitted and an inquiry to the mobile LLM 41 may be continued.

[0162] (6c) In the above embodiment, each LLM 25, 41, 61 responds to inquiries from the LLM orchestrator 24 within the scope of its own information source. For example, as shown in Fig. 11, each LLM may analyze the content of the inquiry from the LLM orchestrator 24, and if necessary, further query other LLMs, and return a response to the LLM orchestrator 24 taking into account the responses from the other LLMs.

[0163] In this case, as shown in FIG. 11 , in S71, the LLM orchestrator 24 executes a query corresponding to the request content to the mobile LLM 41 according to the query list. In S72, the mobile LLM 41 analyzes the query content. If the analysis determines that a query to another LLM (e.g., the cloud LLM 61) is necessary, the mobile LLM 41 executes the query required by the analysis to the cloud LLM 61 in S73. In S74, the cloud LLM 61 returns a response to the query to the mobile LLM 41. Upon receiving a response from the cloud LLM 61, the mobile LLM 41 generates a response to the query from the LLM orchestrator 24 by merging the information obtained from the mobile information source 42 with the response from the cloud LLM 61 in S75.

[0164] In S76, the mobile LLM 41 returns the created response to the LLM orchestrator 24. The processing status notification in S77 is the same as that described in S13.

[0165] In S78, based on the response from the mobile LLM 41, a service proposal is made to the vehicle user via the HMI unit 21.

[0166] (6d) In the above embodiment, if no affirmative response to the proposed service is received from the vehicle user even after all LLMs listed in the query list have been queried, the process is terminated. In this case, for example, instead of terminating the process, the content (e.g., wording) of the query to the LLM may be changed and the query to the LLM according to the query list may be repeated. Furthermore, a voice or text message may be presented to the vehicle user to prompt them to input a new request with the modified request. Furthermore, the query content may be changed not only after all LLMs in the query list have been queried, but also during the list.

[0167] When a service proposal is made to the vehicle user, in addition to an affirmative response accepting the proposal or a negative response rejecting the proposal, a response indicating a desire to change the request (a natural language response to "Do you want to change the request?") may be accepted. In this case, the inquiry content may be changed and the LLM to be contacted may be determined again. In addition, as response options to the proposal, an affirmative response, a negative response, and a response indicating a desire to change the request may be presented to the vehicle user. The options may be presented by voice or displayed on a touch panel of the in-vehicle display.

[0168] (6e) The control unit 11, 31, 51 and its method described herein may be implemented by a special-purpose computer configured by configuring a processor and memory programmed to execute one or more functions embodied in a computer program. Alternatively, the control unit 11, 31, 51 and its method described herein may be implemented by a special-purpose computer configured by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the control unit 11, 31, 51 and its method described herein may be implemented by one or more special-purpose computers configured by combining a processor and memory programmed to execute one or more functions with a processor configured with one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by a computer on a computer-readable non-transitory tangible recording medium. The method for implementing the functions of each unit included in the control unit 11, 31, 51 does not necessarily need to include software; all of the functions may be implemented using one or more hardware.

[0169] (6f) Multiple functions of one component in the above embodiments may be realized by multiple components, or one function of one component may be realized by multiple components. Also, multiple functions of multiple components may be realized by one component, or one function realized by multiple components may be realized by one component. Also, part of the configuration of the above embodiments may be omitted. Also, at least part of the configuration of the above embodiments may be added to or substituted for the configuration of another of the above embodiments.

[0170] (6g) In addition to the above-described in-vehicle device, the present disclosure can also be realized in various forms, such as a system including the in-vehicle device as a component, a program for causing a computer to function as the in-vehicle device, a non-transient tangible recording medium such as a semiconductor memory on which this program is recorded, and a service proposal method.

[0171] [7. Technical Ideas Disclosed in the Present Specification] [Item 1] An in-vehicle device comprising: an access unit (12, 32, 52) configured to access each of a plurality of LLMs, including an in-vehicle LLM that is a large-scale language model mounted on a vehicle and trained using information acquired from the vehicle, and a mobile LLM that is a large-scale language model mounted on a mobile terminal carried by a vehicle user and trained using information acquired from the mobile terminal; a query destination determination unit (11: S120) configured to, upon detecting a request from the vehicle user, determine a target LLM to be queried from among the plurality of LLMs based on a result of text analysis of the content of the request; a query execution unit (11: S130) configured to execute a query to the target LLM determined by the query destination determination unit via the access unit; and a presentation unit (11: S140) configured to present to the vehicle user a response from the target LLM obtained by the query by the query execution unit.

[0172] [Item 2] The in-vehicle device according to Item 1, wherein the query execution unit is configured to set the query content to the target LLM assuming that the request is a content requesting a service proposal, and each LLM belonging to the plurality of LLMs is configured to output a service corresponding to the request as the response.

[0173] [Item 3] The in-vehicle device according to item 1 or 2, further comprising: a response confirmation unit (11: S140) configured to confirm a response of the vehicle user to the content presented by the presentation unit; and when a negative response is confirmed by the response confirmation unit, the inquiry destination determination unit is configured to set another LLM that belongs to the plurality of LLMs and is different from the target LLM as a new target LLM.

[0174] [Item 4] The in-vehicle device according to Item 3, wherein, when the response confirmation unit confirms a negative response for all LLMs included in the plurality of LLMs, the query execution unit prompts the vehicle user to input a new request that changes the request.

[0175] [Item 5] The in-vehicle device according to Item 3 or Item 4, wherein, when the response confirmation unit confirms a response indicating a request change, the query execution unit changes the content of the query, and the query destination determination unit re-determines the target LLM.

[0176] [Item 6] The in-vehicle device according to any one of items 3 to 5, wherein the presentation unit proposes a plurality of services based on the response, and the response confirmation unit confirms a result of the vehicle user selecting one of the plurality of services proposed via the presentation unit.

[0177] [Item 7] The in-vehicle device according to any one of Items 3 to 6, further comprising a learning data providing unit (11: S150) configured to generate learning data that associates content presented by the presentation unit with responses of the vehicle user confirmed by the response confirmation unit, and to provide the generated learning data to each LLM belonging to the plurality of LLMs.

[0178] [Item 8] The in-vehicle device according to any one of items 3 to 7, further comprising an instruction unit (11: S180) configured to, when the response confirmation unit confirms a positive response from the vehicle user, inquire of the vehicle user about whether or not the proposal content for which the positive response was obtained can be implemented, and, when permission to implement the proposal content is obtained, instruct a processing unit that provides a service that realizes the proposal content to implement the service.

[0179] [Item 9] The in-vehicle device according to any one of items 1 to 8, wherein the inquiry destination determination unit is configured to set at least the in-vehicle LLM as the target LLM when the request includes a keyword related to a driving characteristic of the vehicle.

[0180] [Item 10] The in-vehicle device according to any one of items 1 to 9, wherein the inquiry destination determination unit is configured to set at least the mobile LLM as the target LLM when the request includes a keyword related to a personal characteristic of the vehicle user.

[0181] [Item 11] The in-vehicle device according to any one of items 1 to 10, wherein the plurality of LLMs includes a cloud LLM that is a large-scale language model that is trained using large-scale data obtained on a cloud.

[0182] [Item 12] The in-vehicle device according to Item 11, wherein the inquiry destination determination unit is configured to set at least the cloud LLM as the target LLM when the request includes a keyword related to general information.

[0183] [Item 13] The in-vehicle device according to any one of items 1 to 12, further comprising a request prompting unit (11: S1 to S3) configured to detect, as a service proposal trigger, a change in the behavior of the vehicle user or an environment including the vehicle user, which is related to a situation in which the vehicle user is likely to make the request, and to notify the vehicle user to prompt the vehicle user to input the request when the service proposal trigger is detected.

[0184] [Item 14] The in-vehicle device according to Item 13, wherein the service suggestion trigger includes at least one of the following: the vehicle user getting into the vehicle; the mobile terminal and the in-vehicle device being connected in a state where they can communicate; and the in-vehicle device obtaining information indicating the health condition of the vehicle user via the mobile terminal.

[0185] [Item 15] The in-vehicle device according to any one of items 1 to 14, wherein the query execution unit transmits a processing status notification to the vehicle user, indicating that it will take some time to obtain a response, when a waiting time until a response is obtained from the target LLM exceeds an allowed time or there is a possibility that the allowed time will be exceeded.

[0186] [Item 16] The in-vehicle device according to any one of items 1 to 15, wherein the presentation unit presents a positive response, a negative response, and a response indicating a desire to change the request as response options to the answer presented to the vehicle user.

[0187] [Item 17] A mobile terminal carried by a vehicle user, comprising: a mobile LLM (41) that is a large-scale language model that learns using information acquired from the mobile terminal; and a mobile control unit (31) that, upon receiving a query based on a request from the vehicle user from an in-vehicle device (10) that is mounted on the vehicle and configured to be able to access each of a plurality of LLMs including an in-vehicle LLM (25) that is a large-scale language model that learns using information acquired from the vehicle, and the mobile LLM, acquires a response to the query from the mobile LLM and returns the response to the in-vehicle device.

[0188] [Item 18] An information presentation method executed by an on-board device (10) configured to be able to access each of a plurality of LLMs, including an on-board LLM (25) that is a large-scale language model mounted on a vehicle and trained using information acquired from the vehicle, and a mobile LLM (41) that is a large-scale language model mounted on a mobile terminal carried by a vehicle user and trained using information acquired from the mobile terminal, the information presentation method comprising the steps of: upon detecting a request from the vehicle user, determining a target LLM to be queried from among the plurality of LLMs based on a result of text analysis of the content of the request (S120); executing a query to the determined target LLM (S130); and presenting information based on a response from the target LLM obtained by the query to the vehicle user (S140).

[0189] [Item 19] A system including: an on-board LLM (25) that is mounted on a vehicle and is a large-scale language model that learns using information acquired from the vehicle; a mobile terminal (30) that is carried by a vehicle user and that is mounted with a mobile LLM (41) that is a large-scale language model that learns using information acquired from the mobile terminal; and an on-board device (10) that receives a request from the vehicle user and is configured to be able to access each of a plurality of LLMs including the on-board LLM and the mobile LLM, wherein the on-board device comprises: an access unit (12, 32, 52) configured to execute access to each of the plurality of LLMs; a query destination determination unit (11: S120) configured to, upon detecting a request from the vehicle user, determine a target LLM to be queried from among the plurality of LLMs based on a result of text analysis of the content of the request; and a query execution unit (11: S130) configured to execute a query for the target LLM determined by the query destination determination unit via the access unit. a presentation unit (11: S140) configured to present to the vehicle user a response from the target LLM obtained by the query executed by the query execution unit.

[0190] [Item 20] The information presentation system according to Item 19, wherein the mobile LLM learns using usage information of an application program installed on the mobile terminal.

[0191] [Item 21] The information presentation system according to item 19 or 20, wherein the mobile LLM learns using information collected by a voice recognition function of the mobile terminal.

[0192] [Item 22] The information presentation system according to any one of items 19 to 21, wherein the mobile LLM learns using a statement history of the vehicle user collected by a chat function in which the mobile terminal converses with the vehicle user by voice or text.

[0193] [Item 23] The information presentation system according to any one of items 19 to 22, further comprising a cloud LLM that is a large-scale language model that learns using large-scale data obtained on a cloud, wherein the mobile LLM transfers the query to the cloud LLM based on a result of text analysis of the content of the query.

[0194] [Item 24] The information presentation system according to any one of Items 19 to 23, further comprising a response confirmation unit (11: S140) configured to confirm a response of the vehicle user to content presented by the presentation unit, wherein the presentation unit proposes a plurality of services based on the answer, and the response confirmation unit is configured to confirm a selection result in which the vehicle user selects one of the plurality of services proposed via the presentation unit, and each of the plurality of LLMs learns using the selection result.

Claims

1. An in-vehicle device comprising: an access unit (12, 32, 52) configured to access each of a plurality of LLMs, including an in-vehicle LLM, which is a large-scale language model mounted on a vehicle and trained using information acquired from the vehicle, and a mobile LLM, which is a large-scale language model mounted on a mobile terminal carried by a vehicle user and trained using information acquired from the mobile terminal; a query destination determination unit (11: S120) configured to, upon detecting a request from the vehicle user, determine a target LLM to be queried from among the plurality of LLMs based on a result of text analysis of the content of the request; a query execution unit (11: S130) configured to execute a query to the target LLM determined by the query destination determination unit via the access unit; and a presentation unit (11: S140) configured to present to the vehicle user a response from the target LLM obtained by the query executed by the query execution unit.

2. An in-vehicle device according to claim 1, wherein the query execution unit is configured to set the query content to the target LLM assuming that the request is a request for a service proposal, and each LLM belonging to the plurality of LLMs is configured to output a service corresponding to the request as the response.

3. An in-vehicle device according to claim 1 or claim 2, comprising: a response confirmation unit (11: S140) configured to confirm the vehicle user's response to the content presented by the presentation unit; and when a negative response is confirmed by the response confirmation unit, the inquiry determination unit is configured to set another LLM that belongs to the plurality of LLMs and is different from the target LLM as the new target LLM.

4. An in-vehicle device according to claim 3, wherein the query execution unit prompts the vehicle user to input a new request that modifies the request when the response confirmation unit confirms a negative response for all LLMs included in the plurality of LLMs.

5. An in-vehicle device according to claim 3, wherein, when the response confirmation unit confirms a response indicating a request change, the query execution unit changes the content of the query, and the query destination determination unit re-determines the target LLM.

6. An in-vehicle device according to claim 3, wherein the presentation unit proposes a plurality of services based on the response, and the response confirmation unit confirms a result of the vehicle user selecting one of the plurality of services proposed via the presentation unit.

7. An in-vehicle device according to claim 3, further comprising a learning data providing unit (11: S150) configured to generate learning data that associates the content presented by the presentation unit with the response of the vehicle user confirmed by the response confirmation unit, and to provide the generated learning data to each LLM belonging to the plurality of LLMs.

8. An in-vehicle device according to claim 3, further comprising an instruction unit (11: S180) configured to, when the response confirmation unit confirms a positive response from the vehicle user, inquire of the vehicle user as to whether or not the proposal content for which the positive response was obtained can be implemented, and, when permission to implement the proposal content is obtained, instruct a processing unit that provides a service that realizes the proposal content to implement the service.

9. An in-vehicle device according to claim 1, wherein the inquiry destination determination unit is configured to set at least the in-vehicle LLM as the target LLM when the request includes a keyword related to the driving characteristics of the vehicle.

10. An in-vehicle device according to claim 1, wherein the inquiry destination determination unit is configured to set at least the mobile LLM as the target LLM when the request includes a keyword related to a personal characteristic of the vehicle user.

11. The in-vehicle device according to claim 1, wherein the plurality of LLMs includes a cloud LLM, which is a large-scale language model that trains using large-scale data obtained on a cloud.

12. An in-vehicle device according to claim 11, wherein the inquiry destination determination unit is configured to set at least the cloud LLM as the target LLM when the request includes a keyword related to general information.

13. An in-vehicle device according to claim 1, further comprising a request prompting unit (11: S1 to S3) configured to detect, as a service proposal trigger, a change in the vehicle user's behavior or the environment including the vehicle user, which is related to a situation in which the vehicle user may make the request, and to notify the vehicle user to prompt the vehicle user to input the request when the service proposal trigger is detected.

14. An in-vehicle device according to claim 13, wherein the service proposal trigger includes at least one of the following: the vehicle user getting into the vehicle; the mobile terminal and the in-vehicle device being connected in a state where they can communicate; and the in-vehicle device obtaining information indicating the health condition of the vehicle user via the mobile terminal.

15. An in-vehicle device according to claim 1, wherein the query execution unit sends a processing status notification to the vehicle user indicating that it will take some time to obtain a response when the waiting time until a response is obtained from the target LLM exceeds an allowable time or there is a possibility that the allowable time will be exceeded.

16. An in-vehicle device according to claim 1, wherein the presentation unit presents a positive response, a negative response, and a response indicating a desire to change the request as response options to the answer presented to the vehicle user.

17. A mobile terminal comprising: a mobile LLM (41) that is a mobile terminal carried by a vehicle user and is a large-scale language model that learns using information acquired from the mobile terminal; and a mobile control unit (31) that, upon receiving a query based on a request from the vehicle user from an in-vehicle device (10) configured to be able to access an in-vehicle LLM (25) that is a large-scale language model that learns using information acquired from the vehicle and a plurality of LLMs including the mobile LLM, acquires a response to the query from the mobile LLM and returns the response to the in-vehicle device.

18. An information presentation method executed by an on-board device (10) configured to be able to access multiple LLMs, including an on-board LLM (25), which is a large-scale language model mounted on a vehicle and trained using information acquired from the vehicle, and a mobile LLM (41), which is a large-scale language model mounted on a mobile terminal carried by a vehicle user and trained using information acquired from the mobile terminal, the information presentation method comprising the steps of: upon detecting a request from the vehicle user, determining a target LLM to be queried from among the multiple LLMs based on a result of text analysis of the content of the request (S120); executing a query to the determined target LLM (S130); and presenting information based on a response from the target LLM obtained by the query to the vehicle user (S140).

19. A system comprising: an on-board LLM (25) that is a large-scale language model mounted on a vehicle and trains using information generated in the vehicle; a mobile terminal (30) carried by a vehicle user and that is mounted with a mobile LLM (41) that is a large-scale language model that trains using information generated in the mobile terminal; and an on-board device (10) that receives a request from the vehicle user and is configured to be able to access each of a plurality of LLMs including the on-board LLM and the mobile LLM, wherein the on-board device comprises: an access unit (12, 32, 52) configured to execute access to each of the plurality of LLMs; a query destination determination unit (11: S120) configured to, upon detecting a request from the vehicle user, determine a target LLM to be queried from among the plurality of LLMs based on a result of text analysis of the content of the request; and a query execution unit (11: S130) configured to execute a query for the target LLM determined by the query destination determination unit via the access unit. a presentation unit (11: S140) configured to present to the vehicle user a response from the target LLM obtained by the query executed by the query execution unit.

20. An information presentation system according to claim 19, wherein the mobile LLM learns using usage information of an application program installed on the mobile terminal.

21. An information presentation system according to claim 19, wherein the mobile LLM learns using information collected by a voice recognition function of the mobile terminal.

22. An information presentation system according to claim 19, wherein the mobile LLM learns using the vehicle user's speech history collected through a chat function in which the mobile terminal converses with the vehicle user via voice or text.

23. An information presentation system as described in claim 19, further comprising a cloud LLM, which is a large-scale language model that learns using large-scale data obtained on the cloud, and the mobile LLM transfers the query to the cloud LLM based on the results of text analysis of the content of the query.

24. An information presentation system as described in claim 19, further comprising a response confirmation unit (11: S140) configured to confirm the vehicle user's response to the content presented by the presentation unit, wherein the presentation unit proposes a plurality of services based on the answer, and the response confirmation unit is configured to confirm a selection result from having the vehicle user select one of the plurality of services proposed via the presentation unit, and each of the plurality of LLMs learns using the selection result.

Images