Service providing system, edge apparatus, server, service providing method, and program

Abstract

In the present invention, an information management unit acquires, regarding a vehicle and one or more devices other than the vehicle related to a user of the vehicle as related devices, user information which is related to the user of the vehicle and is possessed by the plurality of related devices, and situation information indicating the situation of the related devices or the user. An event detection unit (S110 to S120) detects occurrence of an event related to the vehicle or the user. When the event detection unit detects the occurrence of an event, a control instruction unit (S130 to S160) uses a large-scale language model to generate a control instruction for achieving a service suitable for the user information and the situation information at the time of the occurrence of the event, and transmits the control instruction to at least one of the related devices.

Description

Service Providing System, Edge Device, Server, Service Providing Method, and Program Cross - reference to Related Applications

[0001] This international application claims the benefit of Japanese Patent Application No. 2024 - 162614, filed with the Japan Patent Office on September 19, 2024, and Japanese Patent Application No. 2025 - 105101, filed with the Japan Patent Office on June 20, 2025, and the entire disclosure thereof is incorporated herein by reference.

[0002] This disclosure relates to a technology for providing services to vehicle users using a large - language model.

[0003] Patent Document 1 below describes a technology related to a dialogue system using a large - language model (hereinafter, LLM) that obtains an output using data stored in an automobile.

[0004] Japanese Unexamined Patent Application Publication No. 2024 - 43564

[0005] However, as a result of the inventors' detailed examination, the following problems have been found in the prior art.

[0006] That is, in the prior art, although it may be possible to make proposals suitable for everyone according to the tendency extracted from past data collected and stored for a large number of users, there is a problem that it is difficult to correctly understand the characteristics of individual users and the scenes in which the users are placed and make proposals based on that situation. Also, if the information obtained from the vehicle is limited when grasping the scene in which the user is placed, the understanding of the scene becomes limited. [[ID=!21]]

[0007] In one aspect of this disclosure, a technology for realizing the provision of services based on the characteristics of users and the situations in which the users are placed is provided.

[0008] One aspect of this disclosure is a service provision system comprising an information management unit, an event detection unit, and a control instruction unit. The information management unit is configured to acquire user information relating to the vehicle user and status information indicating the status of the associated devices or users, with each of the vehicle and one or more devices other than the vehicle relating to the vehicle user being associated devices. The event detection unit is configured to detect the occurrence of events relating to the vehicle or user. When the event detection unit detects the occurrence of an event, the control instruction unit is configured to use a large-scale language model to generate control instructions for realizing a service that matches the user information and status information at the time of the event, and to transmit these instructions to at least one of the associated devices.

[0009] This configuration makes it possible to provide services that take into account the characteristics and circumstances of the users.

[0010] One aspect of this disclosure is an edge device mounted on a vehicle, comprising an information management unit, an event detection unit, and a control instruction unit.

[0011] One aspect of this disclosure is a server that communicates with a vehicle, comprising an information management unit, an event detection unit, and a control instruction unit.

[0012] One aspect of this disclosure is a service delivery method implemented by a vehicle or a server communicating with a vehicle. The service delivery method includes acquiring user information about the vehicle user and status information indicating the status of the associated devices or the user, with the vehicle and one or more devices other than the vehicle related to the vehicle's user each being associated devices. The service delivery method includes detecting the occurrence of an event related to the vehicle or the user. When the occurrence of an event is detected, the service delivery method includes using a large-scale language model to generate control instructions for realizing a service that matches the user information and status information at the time of the event, and transmitting these instructions to at least one of the associated devices.

[0013] One aspect of this disclosure is a program that causes a computer to function as an information management unit, an event detection unit, and a control instruction unit.

[0014] This is a block diagram showing the device configuration of the service delivery system. This is a block diagram showing the functional configuration of the service delivery system. This is an explanatory diagram showing the information stored in the integrated memory unit. This is an explanatory diagram illustrating the contents of an instruction file. This is an explanatory diagram illustrating the contents of control information. This is a flowchart of the main processing performed by the controller. This is a sequence diagram showing the system's operation when a baby is detected being left unattended. This is a sequence diagram showing the system's operation when a driver's sleep deprivation is detected. This is an explanatory diagram showing the information stored in the local memory unit.

[0015] Embodiments of this disclosure will be described below with reference to the drawings.

[0016] [1. Overall Configuration] The service provision system 1 of this embodiment shown in Figure 1 comprises a vehicle 2, a mobile terminal 30, and a cloud server 50. The vehicle 2, the mobile terminal 30, and the cloud server 50 may be connected to each other so as to be able to communicate with each other. The vehicle 2, the mobile terminal 30, and the cloud server 50 all correspond to the devices of this disclosure.

[0017] Vehicle 2 is equipped with an edge device 10.

[0018] The edge device 10 is connected to the vehicle's in-vehicle network and collects various vehicle data and performs various vehicle controls via the in-vehicle network. The edge device 10 can have any application program (hereinafter referred to as "app") installed on it. The edge device 10 may perform various functions by running the installed app.

[0019] The edge device 10 comprises a control unit 11, a communication unit 12, a storage unit 13, and a vehicle I / F unit 14.

[0020] The control unit 11 is an electronic control device centered around a computer equipped with a CPU 111, ROM 112, RAM 113, and the like.

[0021] The communication unit 12 performs wireless communication with the mobile terminal 30 and the cloud server 50. For communication with the mobile terminal 30, for example, Bluetooth may be used. Bluetooth is a registered trademark. For communication with the cloud server 50, for example, a wide-area communication network may be used.

[0022] The memory unit 13 stores at least information necessary to realize the in-vehicle large-scale language model (hereinafter referred to as the in-vehicle LLM), which will be described later. The LLM is a language model constructed using a very large dataset and deep learning technology, enabling fluent conversation similar to that of a human, and performing various processes using natural language with high accuracy.

[0023] The vehicle interface unit 14 is connected to various in-vehicle devices via the vehicle's in-vehicle network, etc., and acquires various information from these devices. The in-vehicle network may include CAN and Ethernet. CAN stands for Controller Area Network. Ethernet is a registered trademark. The in-vehicle devices connected to the vehicle interface unit 14 may include devices that are originally installed in the vehicle, as well as aftermarket exterior devices. The in-vehicle devices may include sensors, cameras, sound devices, and display devices.

[0024] The mobile terminal 30 is, for example, a smartphone or tablet carried by the occupants of the vehicle 2. The mobile terminal 30 communicates with the edge device 10 and the cloud server 50. The mobile terminal 30 may run various application programs that are optionally installed.

[0025] The mobile terminal 30 comprises a control unit 31, a communication unit 32, and a storage unit 33.

[0026] The control unit 31 is an electronic control device centered around a computer equipped with a CPU 311, ROM 312, RAM 313, and the like.

[0027] The communication unit 32 performs wireless communication with the edge device 10 and the cloud server 50. For communication with the edge device 10, for example, Bluetooth may be used. For communication with the cloud server 50, for example, a wide-area communication network may be used.

[0028] The memory unit 33 stores at least the information necessary to realize the mobile LLM described later.

[0029] The cloud server 50 is connected to a wide-area communication network accessible by, for example, the edge device 10 and the mobile terminal 30.

[0030] The cloud server 50 comprises a control unit 51, a communication unit 52, and a storage unit 53.

[0031] The control unit 51 is an electronic control unit that is primarily composed of a computer equipped with a CPU 511, ROM 512, RAM 513, and the like.

[0032] The communication unit 52 performs wireless communication with the edge device 10 and the mobile terminal 30. For communication with the edge device 10 and the mobile terminal 30, a wide-area communication network may be used, for example.

[0033] The memory unit 53 stores at least the controller LLM, which will be described later.

[0034] The various computer functions of each of the control units 11, 31, and 51 described above are realized by the CPUs 111, 311, and 511 executing programs stored in non-transitional physical recording media. In this example, ROMs 112, 312, and 512 correspond to non-transitional physical recording media that store the programs. Furthermore, the execution of these programs executes methods corresponding to the programs. Note that some or all of the functions executed by the CPUs 111, 311, and 511 may be configured in hardware using one or more ICs. Also, the number of computers constituting each of the control units 11, 31, and 51 may be one or more.

[0035] [2. Functional Configuration] The various functions provided by the service provision system 1 will be described below.

[0036] [2-1. Cloud Server] As shown in Figure 2, the cloud server 50 includes a controller 60.

[0037] The controller 60 comprises an information management unit 61, an integrated storage unit 62, and a controller AI unit 63. In this embodiment, a configuration in which the controller 60 is located on a cloud server 50 is described as an example, but the controller 60 may be located on, for example, a vehicle 2.

[0038] The information management unit 61 has the function of acquiring user information and status information from the vehicle 2 and the mobile terminal 30 and storing them in the integrated storage unit 62. User information is information about the occupants of the vehicle 2 (hereinafter referred to as vehicle users) and terminal users who are users of the mobile terminal 30, and is static information whose values ​​are unlikely to change and which will remain the same for a long period of time. In this embodiment, the vehicle user and the terminal user are the same, and will be simply referred to as users below. Status information is information about the status of the vehicle 2 and the mobile terminal 30, and is dynamic information whose values ​​change in the short term.

[0039] The information management unit 61 may have the function of acquiring general information from an external information source 70, and further, the function of storing the acquired information in the integrated storage unit 62. The information acquired from the external information source 70 may include information related to the determination of event occurrence and information related to control instructions for realizing services provided to the user. The information management unit 61 may also have the function of requesting user information and status information from the vehicle 2 and the mobile terminal 30 as needed.

[0040] As shown in Figure 3, the integrated storage unit 62 stores LLM parameters 621, profiles 622, and contexts 623. The integrated storage unit 62 may also store history information 624, instruction files 625, control information 626, and general information 627, etc.

[0041] The LLM parameters 621 are a set of parameters that constitute a large-scale language model (i.e., LLM). The LLM realized by the LLM parameters stored in the integrated memory unit 62 is called the controller LLM.

[0042] The profile 622 is user information collected by the information management unit 61 from the vehicle 2 and the mobile terminal 30. The context 623 is situation information collected by the information management unit 61 from the vehicle 2 and the mobile terminal 30. The general information 627 is general information obtained from an external information source 70. The external information source 70 is, for example, another server connected to the cloud server 50 via a communication network. The general information 627 may include, for example, weather information, traffic congestion information, review site information, news, and music rankings, etc.

[0043] The history information 624 may include the interaction / conversation history between the controller LLM and other LLMs. Here, other LLMs refer to the in-vehicle LLMs and mobile LLMs installed in the vehicle 2 and the mobile terminal 30 that make up the service providing system 1. The in-vehicle LLMs and mobile LLMs will be described later. The history information 624 may include the history information 254, 454 provided from the vehicle 2 and the mobile terminal 30. The history information 624 may be used for generating prompts and control instructions, and for learning and tuning the controller LLM, etc.

[0044] The instruction file 625 is a file that shows prerequisite information for controlling the conversation between LLMs by imposing certain constraints on the controller LLM, the in-vehicle LLMs, and the mobile LLMs. That is, the instruction file 625 is used to learn and tune each LLM so that an appropriate answer can be obtained from each LLM.

[0045] For example, the role of the controller LLM may be described in the instruction file 625. For example, the role of the in-vehicle LLM may be described in the instruction file 255 to be described later, and for example, the role of the mobile LLM may be described in the instruction file 455 to be described later.

[0046] In the instruction file 625, as shown in FIG. 4, the following may be described as the role of the LLM. For example, it may be described that the controller LLM is required to serve as an assistant to the occupant, and is required to collect information necessary for user support from the vehicle 2 or the mobile terminal 30 as necessary. It may also be described that the in-vehicle LLM and the mobile LLM are required to generate queries for accessing a database or the like where user information and situation information are managed for a given question, and that only the query is to be provided without adding explanations or additional information when generating the query.

[0047] The instruction file 625 may further include information specifying a format (for example, JSON format) used when the LLM converses with a device or a user. The format may include information indicating the conversation content, information specifying the related device that is the conversation partner, and information specifying the output format of the conversation. The related devices include the vehicle 2 and the mobile terminal 30, and may further include the in-vehicle LLM and in-vehicle equipment 23 of the vehicle 2, and the mobile LLM and mobile application 43 of the mobile terminal 30.

[0048] The control information 626 describes information for correctly handling related devices such as the in-vehicle equipment 23 and the mobile application 43 to be controlled in the service providing system 1. That is, the control information 626 is loaded into the LLM to train and tune the LLM so that the LLM can generate accurate instructions for the related devices. The control information 626 may include the name of the control target, the description of the control target, the state of the control target, and the like. The control information 626 is described, for example, as shown in FIG. 5.

[0049] The controller AI unit 63 has a function to detect the occurrence of an event indicating a situation requiring assistance to the user, based on predetermined event occurrence conditions, using information stored in the integrated memory unit 62 or information acquired by the information management unit 61 for temporary use. The event occurrence conditions may include, for example, the elapsed time since the user boarded or alighted. The event occurrence conditions may also include the notification to the controller AI unit 63 of an event detected by a related device (i.e., the vehicle 2 or mobile terminal 30). Event detection may be performed according to a predetermined logic, or it may be performed by a controller LLM that has been learned or tuned to detect the occurrence of events by taking user information and situation information as input.

[0050] The controller AI unit 63 has the function of acquiring the status of related devices and the user by communicating with related devices based on the content of the events that have occurred, and issuing control instructions to the related devices as necessary. Control instructions may include instructions to activate equipment on the related devices, and instructions to provide information on the related devices.

[0051] The controller AI unit 63 has a function to generate prompts used for conversations with the controller LLM and related devices (i.e., the vehicle 2 and the mobile terminal 30). A prompt is text that gives instructions or questions to the AI, etc. In other words, it is an input sentence that tells the AI ​​what to do. The controller AI unit 63 may generate the prompt in the controller LLM and then use the generated prompt to query the related devices. In this case, the controller AI unit 63 can obtain a response from the queried related device that takes into account the profile and context. The prompt may also specify the format to be used for describing the response.

[0052] The controller AI unit 63 may use the controller LLM to determine whether or not to acquire further information about user information and status information, and to generate control instructions for devices to realize services according to the user information and status information.

[0053] For example, in the case of sending control instructions regarding the air conditioner to a vehicle, the conditions for sending the control instructions may differ depending on the user information. Specifically, normally, a control instruction is sent when an event is detected indicating that the interior temperature of the vehicle has reached a standard temperature (e.g., 25 degrees Celsius or higher). However, if there is user information indicating that the user is sensitive to heat, it is conceivable that a control instruction would be sent when an event is detected indicating that the temperature has reached a lower standard temperature than usual (e.g., 23 degrees Celsius or higher).

[0054] [2-2. Vehicle] Vehicle 2 comprises an in-vehicle information source 21, an information collection unit 22, in-vehicle equipment 23, a control device 24, a local storage unit 25, and an in-vehicle AI unit 26. The information collection unit 22, the local storage unit 25, and the in-vehicle AI unit 26 are mounted on the edge device 10.

[0055] The in-vehicle information source 21 includes in-vehicle sensors such as cameras and radar, as well as an in-vehicle ECU mounted in the vehicle. ECU stands for Electronic Control Unit.

[0056] The information gathering unit 22 has the function of collecting user information and status information from the in-vehicle information source 21 and storing it in the local storage unit 25. The user information collected by the information gathering unit 22 includes personal information such as the user's age, gender, and driving characteristics, and is information that indicates the static characteristics and circumstances of the user related to the vehicle. The status information collected by the information gathering unit 22 includes in-vehicle image information and vehicle status information, and is information that indicates the dynamic characteristics and circumstances of the user related to the vehicle. The information gathering unit 22 periodically updates the user information and status information stored in the local storage unit 25. User information that does not change much may be updated less frequently, while status information that is constantly changing may be updated more frequently (for example, in real time). The in-vehicle LLM may be retrained and tuned according to the updated content each time the information in the local storage unit 25 is updated.

[0057] The information collection unit 22 has the function of synchronizing the user information and status information stored in the local storage unit 25 with the controller 60 by transmitting the user information and status information stored in the local storage unit 25 to the controller 60 according to predetermined criteria. For example, the information collection unit 22 may perform the synchronization of user information and status information periodically. User information that does not change often may be synchronized less frequently (for example, once a day), while status information that is constantly changing may be synchronized more frequently (for example, synchronized in real time each time the status information in the local storage unit 25 is updated). Furthermore, the synchronization of user information and status information may be performed according to instructions from the controller 60.

[0058] The in-vehicle equipment 23 consists of various pieces of equipment necessary to provide services that support the user. The in-vehicle equipment 23 may include, for example, an audio device, a display device, a communication device, and various actuators (for example, actuators that operate the air conditioner or power windows).

[0059] The control device 24 is an electronic control device that controls the in-vehicle equipment 23 and has the function of controlling the in-vehicle equipment 23 according to control instructions from the controller 60. The in-vehicle information source 21, the in-vehicle equipment 23, and the control device 24 are in-vehicle devices connected via the vehicle I / F unit 14.

[0060] As shown in Figure 9, the local storage unit 25 stores LLM parameters 251, profiles 252, context 253, history information 254, instruction files 255, control information 256, and the like.

[0061] The LLM parameter 251 is a set of parameters that constitute a large-scale language model. The LLM implemented by the LLM parameter 251 installed in vehicle 2 is the in-vehicle LLM.

[0062] Profile 252 is user information collected by the information collection unit 22. Context 253 is status information collected by the information collection unit 22. Profile 252 and context 253 may include user information and status information generated by the in-vehicle LLM based on information obtained from the in-vehicle information source 21. Profile 252 and context 253 may also include user information and status information notified by the controller 60.

[0063] The history information 254 may include the history of interactions and conversations between the in-vehicle LLM and the user, and the history of interactions and conversations between the in-vehicle LLM and other LLMs. Here, "other LLMs" refers to the controller LLM installed in the controller 60 that constitutes the service provision system 1, and the mobile LLM installed in the mobile terminal 30.

[0064] The in-vehicle AI unit 26 has the function of generating a response to an inquiry (for example, status information) using the in-vehicle LLM when it receives an inquiry from the controller 60, and transmitting it to the controller 60. The in-vehicle AI unit 26 has the function of searching for information stored in the local storage unit 25 in order to generate a response to an inquiry, and the function of exchanging information with the control device 24. The response to the inquiry may be generated based on the history information 254.

[0065] The in-vehicle AI unit 26 has the function of generating responses in conversations with the controller AI unit 63.

[0066] The in-vehicle AI unit 26 may have a function to store in the local storage unit 25 the results of recognizing the user's characteristics and the in-vehicle situation using the in-vehicle LLM, based on information such as information inside and around the vehicle 2 acquired from the in-vehicle information source 21, as user information and situation information.

[0067] The in-vehicle AI unit 26 may have a function to operate the in-vehicle equipment 23 in the control device 24 according to control instructions from the controller 60. The control instructions from the controller 60 may be instructions in a format that the control device 24 can execute (hereinafter referred to as JOB), or they may be conversational instructions that indicate the items necessary for generating a JOB. If the control instructions from the controller 60 are JOBs, the in-vehicle AI unit 26 may transfer the JOBs directly to the control device 24. If the control instructions from the controller 60 are conversational instructions, the in-vehicle AI unit 26 may convert the conversational instructions into JOBs using the in-vehicle LLM and transfer them to the control device 24.

[0068] [2-3. Mobile Devices] Mobile devices 30 are, for example, smartphones or tablet devices that can be carried by the user.

[0069] The mobile terminal 30 comprises a mobile information source 41, an information collection unit 42, a mobile application 43, a control device 44, a local storage unit 45, and a mobile AI unit 46.

[0070] The mobile information source 41 includes sensors installed in the mobile terminal 30, an application program (hereinafter referred to as a mobile app) 43 executed by the mobile terminal 30, and an OS that manages the execution of the mobile app. The mobile app 43 may include an app that collects healthcare information regarding the user's health status, and an app that manages the user's schedule, etc.

[0071] The information obtained from the mobile information source 41 may include information specific to the terminal user and information indicating the status of the mobile terminal 30. Information specific to the terminal user may include, for example, usage information of the mobile application 43 installed on the mobile terminal 30, healthcare information of the terminal user obtained by running the mobile application 43, and the terminal user's planned activities and activity history. Information indicating the status of the mobile terminal 30 may include, for example, location information.

[0072] The information gathering unit 42 has the function of saving user information and status information from the mobile information source 41 to the local storage unit 45. Other functions of the information gathering unit 42 are the same as those of the information gathering unit 22 of the vehicle 2.

[0073] The control device 44 is an electronic control device that controls the equipment of the mobile terminal 30 and the mobile application 43 installed on the mobile terminal 30, and has the function of controlling the mobile application 43, etc., in accordance with control instructions from the controller 60.

[0074] As shown in Figure 9, the local storage unit 45 stores LLM parameters 451, profiles 452, context 453, history information 454, instruction files 455, control information 456, and the like.

[0075] The LLM parameter 451 is a set of parameters that constitute a large-scale language model. The LLM implemented by the LLM parameter 451 installed on the mobile terminal 30 is called mobile LLM.

[0076] Profile 452 is user information collected by the information collection unit 42. Context 453 is status information collected by the information collection unit 42. Profile 452 and context 453 may include user information and status information generated by the mobile LLM based on information obtained from the mobile information source 41. Profile 452 and context 453 may also include user information and status information notified by the controller 60.

[0077] The history information 454 may include the history of interactions and conversations between the mobile LLM and the terminal user, and the history of interactions and conversations between the mobile LLM and other LLMs. Here, "other LLMs" refers to the controller LLM installed in the controller 60 that constitutes the service provision system 1, and the in-vehicle LLM installed in the vehicle 2.

[0078] The mobile AI unit 46 has the same functions as the in-vehicle AI unit 26. However, in the description of the mobile AI unit 46, the in-vehicle LLM in the description of the in-vehicle AI unit 26 should be read as mobile LLM, and the vehicle 2 as mobile terminal 30. Furthermore, the in-vehicle information source 21, in-vehicle equipment 23, control device 24, and local storage unit 25 should be read as mobile information source 41, mobile application 43, control device 44, and local storage unit 45, respectively.

[0079] [3. Processing] The main processing performed by the controller 60 in order to function as the controller AI unit 63 will be explained using the flowchart in Figure 6. In addition to the main processing, the controller 60 also performs tuning processing to cause the controller LLM to read the instruction file 625 and control information 626 when the controller 60 is started up and when the instruction file 625 and control information 626 are updated.

[0080] The main process is repeatedly executed by the controller 60.

[0081] In S110, the controller 60 monitors the user information and status information stored in the integrated storage unit 62 and performs event detection processing to detect the occurrence of an event by determining whether the user information and status information meet the pre-set event occurrence conditions.

[0082] In S120, the controller 60, if an event has been detected in the event detection process, proceeds to S130; otherwise, it returns to S110.

[0083] In S130, the controller 60 determines whether additional information is needed to determine the content of a service (i.e., support for the user) corresponding to the event that occurred. The need for additional information may be determined by referring to a table prepared in advance that associates events with the necessary information and the source of that information. Alternatively, the need for additional information may be determined by generating a prompt and querying the controller LLM. In this case, the controller LLM outputs a response that includes the necessary information and the source of that information.

[0084] If the controller 60 determines that additional information is needed, it proceeds to process S140; if it determines that additional information is not needed, it proceeds to process S150.

[0085] In S140, the controller 60 retrieves the necessary information from the related device that is the source of the information, according to the information obtained from the table or controller LLM in S130, and returns the process to S130. Information may be retrieved from multiple related devices, or additional information may be retrieved based on the information obtained. The related device may access its own local storage units 25 and 45 in response to the information retrieval request from the controller 60 to obtain the necessary information. Alternatively, the related device may obtain the necessary information by querying its own LLM.

[0086] In S150, the controller 60 generates a control instruction using the controller LLM. The controller 60 causes the controller LLM to generate a control instruction based on the event that occurred, the user information of the user involved in the event, and the status information of the vehicle and the user. The user information and status information include information stored in the integrated storage unit 62 and information acquired in S140. The control instruction is written, for example, in JSON format and indicates the related device to which the control instruction is to be sent and the content of the control instruction. The control instruction may be generated for multiple related devices.

[0087] In S160, the controller 60 sends the control instruction generated in S150 to the destination related device and terminates the process.

[0088] The related device that receives the control instruction executes the processing according to the control instruction. For example, it may activate an actuator, output audio, or display information on a screen. Furthermore, it may send notifications to necessary locations. The related device may also notify the controller 60 of the control result of the control instruction.

[0089] [4. Use Cases] Use cases for Service Delivery System 1 will be described.

[0090] [4-1. Baby Abandonment Detection] This use case assumes a situation where a baby is left alone in a car on a hot day.

[0091] As shown in Figure 7, in S11, when the edge device 10 of the vehicle 2 detects that the user has disembarked based on information from the in-vehicle information source 21, it notifies the controller 60 of the cloud server 50 of the detection of the user's disembarkation.

[0092] Upon receiving notification of passenger disembarkation, the controller 60 detects the occurrence of an event in S12, as the event conditions are met. Since this event requires information gathering, the controller 60 inquires about the in-vehicle conditions from the edge device 10 of the vehicle 2 that sent the disembarkation notification.

[0093] Upon receiving the inquiry, the edge device 10 acquires status information indicating the in-vehicle situation from an in-vehicle information source 21 such as an in-vehicle camera or from a local storage unit 25 in S13. Furthermore, if the edge device 10 detects from the acquired status information that there is only one occupant in the vehicle, such as a baby, it notifies the controller 60 of the detected information.

[0094] Upon receiving notification of additional information from the edge device 10, the controller 60 determines in S14 that additional information is needed to determine the support to be provided to the user, and queries the mobile terminal 30 held by the user for location information, time spent at the location, etc.

[0095] Upon receiving the inquiry, the mobile terminal 30 obtains the location information of the mobile terminal 30 (i.e., the user possessing the mobile terminal 30) and the user's estimated average stay time at the facility from the mobile information source 41 in S15. The mobile terminal 30 also notifies the controller 60 of the obtained location information and average stay time as additional information indicating the user's status. For the average stay time, for example, information pre-stored in the local storage unit 45 of the mobile terminal 30 linked to the facility's location information is used. In this use case, it is assumed that an entertainment facility that the user regularly visits is notified as location information, and the user's average stay time at that entertainment facility is 60 minutes. Note that information on the average stay time at the entertainment facility may also be pre-stored as user information in the integrated storage unit 62 of the controller 60.

[0096] Upon receiving notification of additional information from the mobile terminal 30, the controller 60, in S16, uses the controller LLM to make a situation assessment and derive a solution based on the acquired additional information and the information stored in the integrated storage unit 62. In this use case, to implement the derived solution, for example, the controller 60 may send a control instruction to the edge device 10 to activate the air conditioner and a control instruction to the mobile terminal 30 to output a voice prompt urging the user to return to their vehicle.

[0097] [4-2. Driver Sleep Deprivation Detection] This use case assumes a situation where a user who is sleep-deprived and stressed is commuting to work by car.

[0098] The integrated storage unit 62 of the controller 60 may have user information pre-stored in it, such as the user's vehicle driving time (for example, 7:00 AM when they go to work) and driving characteristics, which are obtained from a mobile terminal.

[0099] As shown in Figure 8, in S21 and S22, when the mobile terminal 30 detects from a mobile information source 41, such as a healthcare app, that the user is sleep-deprived and that the user is under stress, it notifies the controller 60 of the detected information as status information indicating the user's condition.

[0100] In step S23, when the mobile terminal 30 detects from a mobile information source 41, such as a schedule app, that a meeting is scheduled for early morning, it notifies the controller 60 of the detected information as status information indicating the user's status.

[0101] The controller 60 sequentially updates the context 623 of the integrated storage unit 62 based on the status information notified from the mobile terminal 30.

[0102] The controller 60 acquires information related to vehicle operation from the external information source 70 as needed and sequentially updates the context 623 of the integrated storage unit 62. In this case, the information acquired from the external information source 70 includes traffic congestion information.

[0103] In S24, the controller 60 determines, based on traffic congestion information and schedule information notified from the mobile terminal 30, that the vehicle's route to its destination is congested, and detects that the conditions for an event to occur have been met, thereby detecting the occurrence of an event.

[0104] In S25, the controller 60 uses the controller LLM to make a situation assessment and derive a solution based on the user information and situation information that is known at the time the event occurs. At this time, the controller LLM is input with situation information indicating the user's situation, including not only that there is traffic congestion, but also that a meeting is scheduled early in the morning, that the user is sleep-deprived, and that the user is under stress, and derives a solution according to the user's situation. In this use case, in order to implement the derived solution, the controller 60 may send a voice output control instruction to the mobile terminal 30 suggesting a change to the usual departure time. Furthermore, the controller 60 may send a control instruction to the edge device 10 suggesting setting an alternative route to avoid the congested route, or, taking into account the user's anxiety about being late for a meeting, a control instruction to guide them to a place where they can safely park and make a phone call.

[0105] Upon receiving a control instruction for voice output, the mobile terminal 30, in S26, uses its built-in sound device to output a voice suggestion for changing the departure time. Upon receiving a control instruction for setting a detour, the edge device 10, in S27, processes the navigation system or the like installed in the vehicle 2 to set a detour. Furthermore, upon receiving a control instruction for guidance to a safe location, the edge device 10, in S28, searches for a safe location and processes the navigation system or the like to guide the user to that safe location.

[0106] [5. Correspondence of Terms] In this embodiment, the processing S110 to S120 executed by the controller 60 corresponds to the event detection unit of this disclosure, and the processing S130 to S160 corresponds to the control instruction unit of this disclosure. In this embodiment, the instruction file corresponds to the instruction information of this disclosure.

[0107] [6. Effects] The embodiments described in detail above produce the following effects.

[0108] (6a) The service provision system 1 integrates user information (i.e., profile) and situation information (i.e., context) provided from the vehicle 2 and related devices such as the mobile terminal 30 to comprehensively understand the user's characteristics and the situation in which the user is placed. Then, using LLM, it generates control instructions to realize the optimal service for the driver derived from the understood information, and controls the related devices with the generated control instructions. Therefore, it is possible to provide accurate services that take into account the scene in which the user is placed and the user's characteristics.

[0109] (6b) According to the service provision system 1, when an event is detected, additional user information and status information are acquired as needed and used to generate control instructions. Therefore, even for the same event, multiple types of services can be provided to the user depending on the situation.

[0110] [7. Other Embodiments] Although embodiments of the present disclosure have been described above, the present disclosure is not limited to the embodiments described above and can be implemented in various modified forms.

[0111] (7a) In the above embodiment, an example was described in which the controller 60 is located on the cloud server 50, but the location where the controller 60 is located is not limited to the cloud server. For example, the controller 60 may be located on an edge device 10 of the vehicle 2 or on a related device such as a mobile terminal 30. When the controller 60 is located on the vehicle 2, the vehicle 2 does not necessarily have to be equipped with both the on-board LLM and the controller LLM; it may be equipped with only the controller LLM. Also, if the controller 60 is not located on the cloud server 50, a cloud LLM may be separately installed, which is used to reflect information obtained from an external information source 70, etc., in the processing of the controller 60.

[0112] (7b) In the above embodiment, the vehicle 2 and the mobile terminal 30 are given as examples of related devices, but the related devices may also include home appliances, etc., that can communicate with at least one of the vehicle 2, the mobile terminal 30, and the controller 60. In this case, either or both of the vehicle 2 and the mobile terminal 30 may be excluded from the related devices.

[0113] (7c) In the above embodiment, each of the related devices is equipped with an LLM, but LLMs other than the controller LLM may be omitted. Also, each LLM may be replaced or updated as appropriate.

[0114] (7d) In the above embodiment, the vehicle 2, the mobile terminal 30, and the controller 60 communicate with each other, but there are no particular restrictions on the communication protocol, and any communication protocol can be used.

[0115] (7e) In the above embodiment, the controller 60 detects the occurrence of an event, but the related device may also detect the occurrence of an event and notify the controller 60 that it has detected an event. Alternatively, detection by the controller 60 and detection by the related device may be used in combination.

[0116] (7f) The controller 60, edge device 10, and mobile terminal 30 (hereinafter collectively referred to as the controller 60, etc.) and the method described herein may be implemented by a dedicated computer provided by configuring a processor and memory programmed to execute one or more functions embodied by a computer program. Alternatively, the controller 60, etc. and the method described herein may be implemented by a dedicated computer provided by configuring a processor by one or more dedicated hardware logic circuits. Alternatively, the controller 60, etc. and the method described herein may be implemented by one or more dedicated computers configured by a combination of a processor and memory programmed to execute one or more functions and a processor configured by one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by the computer on a computer-readable non-transitional tangible recording medium. The method for realizing the functions of each part included in the controller 60, etc. does not necessarily need to include software, and all of its functions may be realized using one or more hardware components.

[0117] (7g) Multiple functions of one component in the above embodiment may be realized by multiple components, or one function of one component may be realized by multiple components. Also, multiple functions of multiple components may be realized by one component, or one function realized by multiple components may be realized by one component. Furthermore, some of the configurations of the above embodiment may be omitted. Furthermore, at least some of the configurations of the above embodiment may be added to or replaced with the configurations of other above embodiments.

[0118] (7h) In addition to the service provision system 1 described above, this disclosure can also be realized in various forms, such as the edge device 10 and server 50 which are components of the service provision system 1, a program for causing a computer to function as the controller 60 of the service provision system 1, a non-transitional physical recording medium such as semiconductor memory on which this program is recorded, and a service provision method.

[0119] [8. Technical Concept Disclosed in This Specification] [Item 1] A service provision system comprising: an information management unit (61) configured to acquire user information relating to the user of the vehicle and status information indicating the status of the associated devices or the user, with each of the vehicle and one or more devices other than the vehicle relating to the user of the vehicle being associated devices; an event detection unit (26: S110 to S120) configured to detect the occurrence of an event relating to the vehicle or the user; and a control instruction unit (S130 to S160) configured to generate control instructions using a large-scale language model to realize a service that matches the user information and status information at the time of the event occurrence, and to transmit these instructions to at least one of the associated devices when the event detection unit detects the occurrence of the event.

[0120] [Item 2] A service provision system according to Item 1, wherein when the occurrence of the event is detected, the control instruction unit further acquires additional information from at least one of the user information and status information related to the control instruction from at least one of the associated devices, and uses the acquired additional information to generate the control instruction.

[0121] [Item 3] A service provision system as described in Item 2, wherein the control instruction unit, upon detection of the occurrence of the event, generates a prompt for querying the additional information and transmits it to the associated device.

[0122] [Item 4] A service provision system according to any one of items 1 to 3, wherein the information management unit is configured to further acquire information relating to the occurrence of the event and information relating to the control instructions for realizing the service from an external information source (70) other than the related device.

[0123] [Item 5] A service provision system according to any one of items 1 to 4, wherein the large-scale language model used by the control instruction unit is learned and tuned using control information describing the functions of the equipment of the related device.

[0124] [Item 6] A service provision system according to any one of Items 1 to 5, wherein the large-scale language model used by the control instruction unit is learned and tuned using instruction information describing the roles of the control instruction unit and the related devices.

[0125] [Item 7] A service provision system as described in Item 6, wherein the instruction information includes information that defines a format used for conversation between the large-scale language model used by the control instruction unit and the large-scale language model possessed by the associated device.

[0126] [Item 8] A service provision system as described in Item 7, wherein the format includes information indicating the content of a conversation, information specifying the related device that will be the conversation partner, and information specifying the output format of the conversation.

[0127] [Item 9] A service provision system according to any one of items 1 to 8, wherein the control instruction includes an instruction to activate equipment on the associated device.

[0128] [Item 10] A service provision system according to any one of items 1 to 9, comprising an integrated storage unit (62) that integrates and stores user information and status information acquired from a plurality of related devices by the information management unit, and an event detection unit that detects the occurrence of an event based on the information stored in the integrated storage unit.

[0129] [Item 11] A service provision system according to any one of Items 1 to 10, wherein the related device comprises a local storage unit (25, 45) that stores the user information and status information relating to the related device, and the event detection unit is provided on the related device and detects the occurrence of the event based on the information stored in the local storage unit.

[0130] [Item 12] A service provision system according to any one of Items 1 to 9, further comprising: an integrated storage unit (62) that integrates and stores user information and status information provided by a plurality of related devices by the information management unit; and local storage units (25, 45) provided on the related devices that store user information and status information acquired by the related devices, wherein the information management unit has a function to synchronize the information stored in the integrated storage unit and the local storage units.

[0131] [Item 13] A service provision system as described in Item 12, wherein the information management unit periodically performs the synchronization in accordance with the updates of the information held by the local storage unit.

[0132] [Item 14] A service provision system according to Item 12 or Item 13, wherein the information management unit performs the synchronization in real time in accordance with the update of information held by the local storage unit.

[0133] [Item 15] A service provision system according to any one of items 1 to 14, wherein the control instruction generated by the control instruction unit is a JOB-format instruction that can be executed by the computer of the associated device.

[0134] [Item 16] A service provision system according to any one of items 1 to 15, wherein the control instruction generated by the control instruction unit is an instruction to cause the large-scale language model of the associated device to generate a JOB-format instruction that can be executed by the computer of the associated device.

[0135] [Item 17] A service delivery system according to any one of items 1 to 16, wherein the related device includes a mobile terminal (30) carried by the user of the vehicle.

[0136] [Item 18] A service provision system according to any one of items 1 to 17, wherein the control instruction unit further acquires history information including the history of communication with the related device, and uses the acquired history information to generate the control instruction.

[0137] [Item 19] A service provision system according to any one of items 1 to 18, wherein the control instruction unit, when acquiring information from the associated device, acquires a response based on historical information including the history of communication with the control instruction unit stored in the associated device, and uses the response to generate the control instruction.

[0138] [Item 20] An edge device mounted on a vehicle, comprising: an information acquisition unit configured to acquire user information relating to the user of the vehicle and status information indicating the status of the associated devices or the user, with each of the vehicle and one or more devices other than the vehicle relating to the user of the vehicle being associated devices; an event detection unit configured to detect the occurrence of an event relating to the vehicle or the user; and a control instruction unit configured to generate a control instruction using a large-scale language model to realize a service that matches the user information and status information at the time of the event occurrence, and to transmit it to at least one of the associated devices, when the event detection unit detects the occurrence of the event.

[0139] [Item 21] A server that communicates with a vehicle, comprising: an information management unit configured to acquire user information relating to the user of the vehicle and status information indicating the status of the associated devices or the user, with each of the vehicle and one or more devices other than the vehicle relating to the user of the vehicle being associated devices; an event detection unit configured to detect the occurrence of an event relating to the vehicle or the user; and a control instruction unit configured to generate control instructions using a large-scale language model to realize a service that matches the user information and status information at the time the event occurred, and to transmit these instructions to at least one of the associated devices, when the event detection unit detects the occurrence of the event.

[0140] [Item 22] A service provision method implemented by a vehicle or a server communicating with the vehicle, comprising: acquiring user information relating to the user of the vehicle and status information indicating the status of the associated devices or the user, with each of the vehicle and one or more devices other than the vehicle relating to the user of the vehicle being associated devices; detecting the occurrence of an event relating to the vehicle or the user; and, when the occurrence of the event is detected, generating control instructions using a large-scale language model to realize a service that conforms to the user information and status information at the time the event occurred, and transmitting these instructions to at least one of the associated devices.

[0141] [Item 23] A program that causes a computer to function as: an information management unit configured to acquire user information relating to the user of the vehicle and status information indicating the status of the associated devices or the user, with each of the vehicle and one or more devices other than the vehicle relating to the user of the vehicle being associated devices; an event detection unit configured to detect the occurrence of an event relating to the vehicle or the user; and a control instruction unit configured to generate control instructions using a large-scale language model to realize a service that matches the user information and status information at the time of the event occurrence, and to transmit these instructions to at least one of the associated devices, when the event detection unit detects the occurrence of the event.

Claims

1. A service provision system comprising: an information management unit (61) configured to acquire user information relating to the user of the vehicle and status information indicating the status of the associated devices or the user, with each of the vehicle and one or more devices other than the vehicle relating to the user of the vehicle being treated as associated devices; an event detection unit (26: S110 to S120) configured to detect the occurrence of an event relating to the vehicle or the user; and a control instruction unit (S130 to S160) configured to generate control instructions using a large-scale language model to realize a service that matches the user information and status information at the time of the event occurrence, and to transmit these instructions to at least one of the associated devices when the event detection unit detects the occurrence of the event.

2. A service provision system according to claim 1, wherein when the occurrence of the event is detected, the control instruction unit further acquires additional information from at least one of the user information and status information relating to the control instruction from at least one of the associated devices, and uses the acquired additional information to generate the control instruction.

3. A service provision system according to claim 2, wherein the control instruction unit, upon detection of the occurrence of the event, generates a prompt for querying the additional information and transmits it to the associated device.

4. A service provision system according to claim 1, wherein the information management unit is configured to further acquire information relating to the occurrence of the event and information relating to the control instructions for realizing the service from an external information source (70) other than the related device.

5. A service provision system according to claim 1, wherein the large-scale language model used by the control instruction unit is learned and tuned using control information describing the functions of the equipment of the associated device.

6. A service provision system according to claim 1, wherein the large-scale language model used by the control instruction unit is learned and tuned using instruction information describing the roles of the control instruction unit and the related devices.

7. A service provision system according to claim 6, wherein the instruction information includes information defining a format used for conversation between the large-scale language model used by the control instruction unit and the large-scale language model possessed by the associated device.

8. A service provision system according to claim 7, wherein the format includes information indicating the content of a conversation, information specifying the related device that is the conversation partner, and information specifying the output format of the conversation.

9. A service provision system according to claim 1, wherein the control instruction includes an instruction to activate equipment of the associated device.

10. A service provision system according to claim 1, comprising an integrated storage unit (62) that integrates and stores user information and status information acquired from a plurality of related devices by the information management unit, and an event detection unit that detects the occurrence of an event based on the information stored in the integrated storage unit.

11. A service provision system according to claim 1, wherein the related device comprises a local storage unit (25, 45) that stores the user information and the status information acquired by the related device, and the event detection unit is provided on the related device and detects the occurrence of the event based on the information stored in the local storage unit.

12. A service provision system according to claim 1, further comprising: an integrated storage unit (62) that integrates and stores user information and status information provided by a plurality of related devices by the information management unit; and local storage units (25, 45) provided on the related devices that store user information and status information acquired by the related devices, wherein the information management unit has a function to synchronize the information held by the integrated storage unit with the information held by the local storage units.

13. A service provision system according to claim 12, wherein the information management unit periodically performs the synchronization in accordance with the update of information held in the local storage unit.

14. A service provision system according to claim 12, wherein the information management unit performs the synchronization in real time in accordance with the update of information held by the local storage unit.

15. A service provision system according to claim 1, wherein the control instruction generated by the control instruction unit is a JOB-format instruction that can be executed by a computer of the associated device.

16. A service provision system according to claim 1, wherein the control instruction generated by the control instruction unit is an instruction to cause the large-scale language model of the associated device to generate a JOB-format instruction that can be executed by the computer of the associated device.

17. A service provision system according to claim 1, wherein the associated device includes a mobile terminal (30) carried by the user of the vehicle.

18. A service provision system according to claim 1, wherein the control instruction unit further acquires history information (624) including the history of communication with the related device, and uses the acquired history information to generate the control instruction.

19. A service provision system according to claim 1, wherein the control instruction unit, when acquiring information from the associated device, acquires a response based on history information (254, 454) including the history of communication with the control instruction unit stored in the associated device, and uses the response to generate the control instruction.

20. An edge device mounted on a vehicle, comprising: an information management unit configured to acquire user information relating to the vehicle's user and status information indicating the status of the associated devices or the user, with each of the vehicle and one or more devices other than the vehicle relating to the vehicle's user being associated devices; an event detection unit configured to detect the occurrence of an event relating to the vehicle or the user; and a control instruction unit configured to generate control instructions using a large-scale language model to realize a service that matches the user information and status information at the time of the event, and transmit them to at least one of the associated devices, when the event detection unit detects the occurrence of the event.

21. A server that communicates with a vehicle, comprising: an information acquisition unit configured to acquire user information relating to the user of the vehicle and status information indicating the status of the associated devices or the user, with each of the vehicle and one or more devices other than the vehicle relating to the user of the vehicle being associated devices; an event detection unit configured to detect the occurrence of an event relating to the vehicle or the user; and a control instruction unit configured to generate control instructions using a large-scale language model to realize a service that matches the user information and status information at the time of the event occurrence, and to transmit these instructions to at least one of the associated devices, when the event detection unit detects the occurrence of the event.

22. A service provision method implemented by a vehicle or a server communicating with the vehicle, comprising: acquiring user information relating to the vehicle's user and status information indicating the status of the associated devices or the user, with each of the vehicle and one or more devices other than the vehicle relating to the vehicle's user being associated devices; detecting the occurrence of an event relating to the vehicle or the user; and, when the occurrence of the event is detected, generating control instructions using a large-scale language model to realize a service that conforms to the user information and status information at the time of the event, and transmitting these instructions to at least one of the associated devices.

23. A program that causes a computer to function as: an information management unit configured to acquire user information relating to the user of the vehicle and status information indicating the status of the associated devices or the user, with each of the vehicle and one or more devices other than the vehicle relating to the user of the vehicle being treated as associated devices; an event detection unit configured to detect the occurrence of an event relating to the vehicle or the user; and a control instruction unit configured to generate control instructions using a large-scale language model to realize a service that matches the user information and status information at the time of the event, and to transmit these instructions to at least one of the associated devices, when the event detection unit detects the occurrence of the event.

Images