Driving assistance function recommendation system and method using llm

Abstract

The present application relates to a driving assistance function recommendation system and method using LLM. The present application aims to make it easy for a user to use a driving assistance function by recommending a driving assistance function that seems to be helpful to the user but that the user can not know in a driving assistance function recommendation system using LLM. The driving assistance function recommendation system using LLM includes an acquisition unit that acquires record information obtained when a user drives a vehicle having a prescribed type of driving assistance function, an estimation unit that generates a situation description text in an estimated manner by LLM based on the record information to register it in a situation description text DB, and a first search unit that searches a manual related to the driving assistance function to recommend a driving assistance function that is used based on the user from the situation description text DB.

Description

Technical Field

[0001] This invention relates to the technical field of a driver assistance function recommendation system using LLM applicable to vehicles or automobiles with various driver assistance functions that are capable of autonomous, semi-autonomous, or manual driving. Background Technology

[0002] As a technology somewhat similar to the recommendation of this driving assistance function, the following technology is suggested (refer to Patent Document 1): although it is not a prompt for driving assistance functions or a prompt based on language information such as dialogue logs, it prompts for navigation or multimedia-related actions. The following technology is suggested (refer to Patent Document 2): taking at least one of driver information, dialogue information, and traffic information as input, allowing the learned model to complete the missing information, and using this information as the basis for driving assistance. Furthermore, the following technology is suggested (refer to Patent Document 3): acquiring vehicle information from multiple vehicles as big data, and recommending the use of vehicle functions (but not envisioning the use of language or voice information).

[0003] Patent Document 1: Japanese Patent Publication No. 2014-524380

[0004] Patent Document 2: Japanese Patent Application Publication No. 2024-159078

[0005] Patent Document 3: Japanese Patent Application Publication No. 2019-74803 Summary of the Invention

[0006] However, based on the aforementioned background technology, it is unclear to the user whether the vehicle possesses driver assistance functions that appear to be helpful. Even with highly advanced or diverse driver assistance functions, users cannot fully grasp what driver assistance functions the vehicle is equipped with, leading to the problem of not being able to make good use of these valuable features.

[0007] The objective of this invention is to provide a driver assistance function recommendation system that utilizes LLM: for example, by recommending driver assistance functions that seem helpful to the user but that the user may not be aware of, making it easier for the user to utilize the functions.

[0008] To address the aforementioned problem, the present invention provides a method for a driver assistance function recommendation system utilizing LLM, comprising: an acquisition unit that acquires recorded information obtained when a user drives a vehicle with a specified type of driver assistance function; an estimation unit that, based on the acquired recorded information, generates a situation description text of the user driving the vehicle using LLM in an estimation manner, and registers it in a situation description text database; a first search unit that, based on the registered situation description text database, searches and recommends driver assistance functions used by the user from manuals related to the vehicle's driver assistance functions; and a suggestion unit that suggests the searched driver assistance functions to the user in a specified form.

[0009] Invention Effects

[0010] According to one aspect of the driving assistance function recommendation system of the present invention, it is possible to recommend driving assistance functions that seem helpful to the user but that the user may not be aware of, and to make it easy for the user to utilize driving assistance functions that can function appropriately according to the situation.

[0011] Based on this effect of the present invention, it will be explained more clearly through the following description of the embodiments of the invention. Attached Figure Description

[0012] Figure 1 This is a block diagram illustrating the overall structure of the driver assistance function recommendation system involved in the implementation method.

[0013] Figure 2 This is a flowchart illustrating an example of processing in a driver assistance function recommendation system according to the implementation method. Detailed Implementation

[0014] First, referring to Figure 1, the overall structure of the driver assistance function recommendation system (hereinafter appropriately referred to as the "driver assistance function recommendation system") utilizing LLM (Large Language Models) involved in the implementation method will be described.

[0015] In this embodiment, the LLM used can be a unimodal LLM that takes verbal data as input, or a multimodal LLM that takes not only verbal data as input but also other forms of data that are easily understood by the LLM. This embodiment, for example, uses LLM or multimodal LLM to estimate the situation of each scenario and verbalize it based on recorded information such as camera data, CAN (Controlled Area Network) data, voice data, and usage records of driving assistance functions. The system is constructed as follows: The verbalized situation description is compared with the situation estimation results when the user or other users utilize driving assistance functions or the descriptions of driving assistance functions in the manual. The system estimates the driving assistance functions that the user can utilize and makes recommendations to the user.

[0016] Furthermore, in addition to traditional AI learning systems employing supervised, unsupervised, or reinforcement learning methods, this type of LLM or AI learning can also utilize newly practical, currently under development, or future emerging technologies such as generative AI or generative LLM. For example, this LLM or AI learning can be constructed using neural networks that perform efficient learning through techniques such as representation learning, transfer learning, feature selection, fine-tuning or hyperparameter optimization, and ensemble learning.

[0017] like Figure 1 As shown, the driving assistance function recommendation system according to the embodiment comprises an input unit 100 connected to a vehicle (not shown) via a communication network, a situation description generation unit 102, a situation description search unit 103, a manual search unit 104, a suggestion generation unit 105, and an output unit 107.

[0018] The input unit 100 constitutes an example of the "acquisition unit" according to this embodiment, including an interface for inputting image data or real-time recorder data, CAN data, voice data picked up by an in-vehicle microphone, usage records of driver assistance functions, and other data or recorded data 101 from other vehicles (including vehicles not shown) of the user's vehicle via a communication network using a camera such as an in-vehicle CCD camera, or driving assistance data. This recorded data 101 can be input to the input unit 100 from the vehicle in real-time or almost real-time, or it can be temporarily stored in a storage device or database (not shown) housed in the same communication network, and then input from the input unit 100 at a desired time or in a desired quantity.

[0019] The various recorded data 101 entered here may include data from cameras installed in the user's vehicle, as well as data from cameras installed in other vehicles. The camera data may include image or video data from dashcams that record inside and outside the vehicle, as well as data from roadside cameras or drone cameras outside the vehicle. The CAN data may include data from other vehicles, in addition to data from the user's vehicle. The voice data may include voice data picked up by the user's vehicle's microphone, as well as data picked up by microphones from other vehicles.

[0020] The situation description generation unit 102 constitutes an example of the "estimation unit" involved in this embodiment. It is configured to generate a situation description of the user driving the vehicle in an estimation manner by means of LLM based on the recording information represented by various recording data 101 input or acquired by the input unit 100, and register it in the situation description DB201.

[0021] The status description search unit 103 constitutes an example of the "second search unit" in this embodiment, and the manual search unit 104 constitutes an example of the "first search unit" in this embodiment. These search units work together to search for and recommend driving assistance functions based on the user's usage.

[0022] Specifically, the manual search unit 104 is configured to search for recommended driving assistance functions from the manual data 202 based on the situation description document DB. Furthermore, the situation description document search unit 103 searches the situation description document DB for similar cases involving driving assistance functions. In this case, the manual search unit 104 is configured to search for recommended driving assistance functions from the manual data 202 based on the situation description document DB and the found similar cases. With this structure, based on similar cases, the suggestion generation unit 105 and output unit 107 can recommend driving assistance functions that seem helpful to the user but that the user may not be aware of.

[0023] Furthermore, the situation description generation unit 102 can also be configured to estimate the situation description using LLM based on the recorded data 101 and / or information related to the user's driving skills or preferences, which are part of the data. This configuration allows for the recommendation of driving assistance functions that may seem helpful to the user but that the user might not be aware of, based on consideration of similar cases, or in addition to considering similar cases, taking into account the user's driving skills or preferences.

[0024] The suggestion generation unit 105 and the output unit 107 constitute an example of the "suggestion unit" according to this embodiment. The suggestion generation unit 105 suggests the searched driving assistance functions to the user in the form of a driving assistance function suggestion text 106. The output unit 107 is configured to output the suggested suggestion text 106 as display output data or voice data in a prescribed format, or a combination thereof. At this time, the suggestion generation unit 105 can be configured to prompt the driving assistance functions with LLM-textualized annotation statements. In this way, by also utilizing LLM in the generation of the suggestion text, it is possible to recommend driving assistance functions that seem helpful to the user but that the user may not be aware of in a text format that is easier for the user to understand.

[0025] Such a driver assistance function recommendation system can be primarily or at least partially built into the user's smartphone, personal computer (PC), tablet, smartwatch, or other electronic devices that are connected to the user's vehicle via a communication network and are separate from the vehicle. Alternatively, this driver assistance function recommendation system can be primarily or at least partially built into a server device or similar device connected to the user's vehicle via a communication network, with the user's smartphone or similar device primarily or specifically configured as a browser. This configuration allows for the suggestion of driver assistance functions to the user before getting into the vehicle or driving, either through display output or voice output, in the user's home or indoor / outdoor facilities, in the wild, inside or outside the vehicle, etc.

[0026] Alternatively, the driver assistance function recommendation system configured in this way can be partially or wholly integrated with a vehicle agent. In this case, driver assistance functions can be recommended not only before driving, but also in real time during driving or while parked, through the display output or voice output of the output unit 107 of the vehicle agent installed in the vehicle.

[0027] In any case, the driver assistance function recommendation system may be configured as a computer-mounted device or a computer device that performs centralized or distributed processing. In other words, the driver assistance function recommendation system is constructed as a system that performs centralized or distributed processing of large-scale data such as the situation description document DB201.

[0028] Furthermore, the output unit 107 can be integrated into all of the user's smartphones, etc., to output to the user outside the vehicle or when not driving (e.g., as part of the user's post-driving evaluation / comment or rating), or to output to the user inside the vehicle or while driving or in real time (e.g., in the form of prompting the user or driver to recommend support functions at the current time).

[0029] Alternatively, in the LLM of the aforementioned situation description generation unit or the LLM of the suggestion generation unit 105, external relevant knowledge acquired outside the vehicle and capable of fine-tuning or over-tuning in the LLM of the driver assistance function recommendation system can be combined with the situation description document DB201 or manual data 202 to impart domain knowledge. In addition to or in lieu of this external relevant knowledge, "general common sense" related to the vehicle, vehicle movement, driving route, street view, buildings, general drivers, pedestrians, road structure, road maps, etc., can also be used. Alternatively, in addition to or in lieu of this external relevant knowledge, map information such as road maps or multi-destination maps can also be used. In this case, map information, by plotting the vehicle's position information on a map and using it as data related to the position information, can be utilized as data more easily understood by the LLM.

[0030] Next, refer to Figure 1 block diagram and Figure 2 The flowchart illustrates an example of processing in the driver assistance function recommendation system according to this embodiment.

[0031] exist Figure 2 First, through the input unit 100 (reference) Figure 1 (Step S1) Acquire image data, voice data, CAN data, and usage records of driver assistance functions from the camera.

[0032] Next, through the situation description generation unit 102 (reference) Figure 1 The system uses LLM to estimate the conditions of each scenario (i.e., the various situations that the user or vehicle experiences or encounters) to generate a situation description (including whether or not the driver assistance function is used) (step S2).

[0033] Next, through the situation description document search department 103 (reference) Figure 1 (Step S3) Search for similar cases (events using driving assistance functions) from the situation description text DB201 accumulated from the situation description text generated in step S2.

[0034] After the search in step S3, or in parallel or before / after the search in step S3, or without performing the search in step S3, the search can be performed through the manual search section 104 (see reference). Figure 1 The system searches the vehicle's manual for driver assistance features that the user can use (or that can be used but are not yet used) (step S4). At this time, following the processing in step S3, the system also searches for driver assistance features using similar cases.

[0035] Then, through the suggestion document generation unit 105 and the output unit 107 (see reference) Figure 1 The system then suggests the searched driving assistance functions to the user (step S5), concluding the series of processes. This series of processes can be executed based on commands from the application involved in the driving assistance function recommendation system, triggered by user smartphone operations or similar actions. Alternatively, in the case of a structure that provides real-time suggestions while the vehicle is in motion, it can be invoked and executed appropriately, particularly as a subroutine that can be repeatedly executed within a short period.

[0036] Next, regarding as Figure 2 The specific examples of the recommendations are explained in detail based on the results of a series of processing steps.

[0037] First, let's illustrate the example of recommending the "intersection collision avoidance support function".

[0038] In situations such as "During the last drive, a pedestrian suddenly rushed out while turning right, posing a danger," "This fact was recorded in the camera log or CAN data log," and "PCS (Pre-Collision Safety System) is available but is currently disabled," the LLM generates a situation report based on information recorded by cameras, voice, CAN data, etc., and searches the situation report database or manual. The result, "During the last drive, a pedestrian suddenly rushed out while turning right, almost causing a collision," indicates that the vehicle has intersection collision avoidance support, which should be available in this situation, but is currently disabled. Should we try turning it ON?

[0039] In addition, the "PCS" in this example refers to a function that assists the driver in collision avoidance maneuvers by issuing warnings or controlling braking force when sensors detect a work object in the driving path and the system determines that there is a high probability of collision. Furthermore, the "Intersection Collision Avoidance Support Function" is a function that provides support based on collision warnings and pre-collision braking when the system detects pedestrians or bicycles crossing the driving path while turning left / right, or when turning right at an intersection and crossing the driving path of oncoming vehicles, and the system determines that there is a high probability of collision. This is one of the functions of PCS.

[0040] Next, examples of recommended “radar cruise control and LTA” will be explained.

[0041] In situations where "We are currently driving on a congested highway" and "LTA seems useful, but it cannot be used because radar cruise control is off," the LLM generates a situation description based on recorded information, destination setting information, congestion information, etc., and searches for situation description DB201 or manual data 202. Then, the suggestion is made: "This vehicle has radar cruise control, which determines the presence or distance of a vehicle ahead and ensures an appropriate distance, and can be used on highways, but it is currently off. Furthermore, the LTA function, which uses a forward camera or radar to identify lanes or vehicles ahead / around and supports the steering wheel operations required for lane keeping, can only be used when this function is ON, which is helpful when driving in congested traffic. Should we try turning radar cruise control ON and using radar cruise control and LTA?"

[0042] Additionally, in this example, "radar cruise control" is a function that determines the presence or distance of a vehicle ahead and ensures an appropriate distance. This function is used on highways or dedicated motorway roads. Furthermore, "LTA" is a function that, when driving on a road with lane markings and in the operation of radar cruise control, uses a forward camera or radar to identify the lane or the vehicle ahead / surrounding vehicles and supports the steering wheel operations required for lane keeping.

[0043] In the generation process utilizing the LLM estimation in the situation description generation unit 102 or suggestion generation unit 105 described in detail above, for efficient processing, it is preferable to perform vectorization processing using LLM after all the converted graphs or charts have been converted into articles. Furthermore, in each of the above-described LLM processes, large amounts of text data can be used to fine-tune a large-scale language model using LLM. Therefore, it can be adapted to various Natural Language Processing (NLP) tasks such as text classification or sentiment analysis, information extraction, article summarization, text generation, and question answering.

[0044] Furthermore, in this embodiment, the aforementioned situation description generation unit 102 or suggestion generation unit 105, and further the situation description search unit 103 or manual search unit 104, are expressed as separate parts as functional blocks that perform separate processing. However, these can be constructed as hardware by a single processor or the like, or they can be configured as separate processing units in software.

[0045] In the above embodiments, since it is difficult for an LLM to directly understand the recorded data 101 such as camera, CAN, and voice data, it is highly advantageous to visualize the data on a map or in a graphical format, using an expression that is easily readable by an LLM or multimodal LLM, thereby significantly improving the accuracy of situation estimation. Furthermore, according to this embodiment, by using LLM with linguistic data as input, it is not only possible to extract relevant features from character data such as papers and related literature from common sense, but also to perform situation estimation at a higher dimension.

[0046] As detailed above, according to this embodiment, in the LLM-based driver assistance function recommendation system, by recommending driver assistance functions that seem helpful to the user but that the user may not be aware of, the function can be easily utilized.

[0047] [Postscript]

[0048] Regarding the implementation methods described above, the following notes are further disclosed.

[0049] [Postscript 1]

[0050] The LLM-based driver assistance function recommendation system described in Appendix 1 of this invention comprises: an acquisition unit that acquires recorded information obtained when a user drives a vehicle with a specified type of driver assistance function; an estimation unit that, based on the acquired recorded information, generates a situation description text of the user driving the vehicle in an estimation manner using LLM, and registers it in a situation description text DB; a first search unit that, based on the registered situation description text DB, searches and recommends driver assistance functions used by the user from manuals related to the vehicle's driver assistance functions; and a suggestion unit that suggests the searched driver assistance functions to the user in a specified form.

[0051] According to the driving assistance function recommendation system described in Appendix 1, if the acquisition unit (e.g., the aforementioned input unit 100) acquires recorded information (e.g., the aforementioned camera, CAN, voice, etc. recorded data 101), the estimation unit (e.g., the aforementioned situation description generation unit 102) generates a situation description based on the recorded information using LLM-based estimation and registers it in the situation description database (e.g., the aforementioned situation description database 201). Then, the first search unit (e.g., the aforementioned manual search unit 104) searches for recommended driving assistance functions from the manual (e.g., the aforementioned manual data 202) based on the situation description database. Then, the suggestion unit (e.g., the aforementioned suggestion generation unit 105 and output unit 107) suggests the searched driving assistance functions in a prescribed form (e.g., the aforementioned driving assistance function suggestion text 106). These results enable the recommendation of driving assistance functions that seem helpful to the user but that the user may not be aware of.

[0052] [Postscript 2]

[0053] The driving assistance function recommendation system described in Appendix 2 of this invention is characterized by further comprising: a second search unit that searches for similar cases related to the driving assistance function based on the registered status description document DB; and a first search unit that searches the manual for recommended driving assistance functions based on the user's usage based on the registered status description document DB and the similar cases searched by the second search unit.

[0054] According to Appendix 2 of the present invention, the driving assistance function recommendation system further utilizes a second search unit (e.g., the aforementioned situation description search unit 103) to search for similar cases related to the driving assistance function based on the situation description document DB. Then, a first search unit (e.g., the aforementioned manual search unit 104) searches the manual for recommended driving assistance functions based on user usage, using the situation description document DB and the found similar cases. These results enable the recommendation of driving assistance functions that appear helpful to the user but which the user may not be aware of, taking into account similar cases.

[0055] [Postscript 3]

[0056] The driving assistance function recommendation system described in Appendix 1 or 2 of this invention is characterized in that the suggestion section, as the prescribed form, suggests the searched driving assistance functions in the form of LLM textualized annotation statements.

[0057] According to Appendix 3 of the present invention, the driving assistance function recommendation system uses an LLM-textualized annotation to suggest driving assistance functions. Therefore, it is possible to recommend seemingly helpful but potentially unknown driving assistance functions to the user in a more easily understood textual form (e.g., display output or voice output in a smartphone or in-vehicle agent).

[0058] [Postscript 4]

[0059] The driving assistance function recommendation system described in Appendix 1 or 3 of this invention is characterized in that the suggestion unit makes suggestions to the user before the user drives the vehicle next time, based on at least one of the display output and voice output of an electronic device constructed separately from the vehicle.

[0060] According to Appendix 4 of the present invention, the driving assistance function recommendation system, by a suggestion unit, recommends driving assistance functions to the user before getting into the vehicle or driving, based on at least one of the display output and voice output of an electronic device (e.g., a smartphone, PC, tablet, etc.) built separately from the vehicle, in places such as the user's residence or indoor / outdoor facilities, in the wild, or inside or outside the vehicle. Therefore, it is possible to recommend driving assistance functions that seem helpful to the user but that the user may not be aware of, allowing sufficient time for pre-training of the user.

[0061] [Postscript 5]

[0062] The driving assistance function recommendation system described in Appendix 5 of this invention is the same as the driving assistance function recommendation system described in any one of Appendices 1 to 3, characterized in that the suggestion unit provides suggestions to the user in real time when the user is driving the vehicle or riding in the vehicle, based on at least one of the display output and voice output of the vehicle agent installed in the vehicle.

[0063] According to Appendix 5 of the present invention, the driving assistance function recommendation system comprises a suggestion unit that, based on at least one of the display output and voice output of the vehicle agent installed in the vehicle, suggests driving assistance functions in real time, such as while driving or while parked. Therefore, it is possible to recommend driving assistance functions that appear helpful to the user but may be unknown to the user, based on the current road conditions, vehicle conditions, and user conditions.

[0064] [Postscript 6]

[0065] The driving assistance function recommendation system described in Appendix 6 of this invention is the driving assistance function recommendation system described in any one of Appendices 1 to 5, characterized in that the estimation unit estimates the situation description text based on at least one of the acquired record information and / or a portion of the acquired record information, information related to the user's driving skills, and information related to the user's preferences.

[0066] The driving assistance function recommendation system according to Appendix 6 of the present invention comprises an estimation unit that estimates a situation description using LLM based on at least one of recorded information and / or information relating to the user's driving skills and preferences, which are part of the recorded information. Therefore, it is possible to recommend driving assistance functions that appear helpful to the user but which the user may not be aware of, taking into account the user's driving skills or preferences.

[0067] [Postscript 7]

[0068] The driving assistance function recommendation method using LLM described in Appendix 7 of this invention comprises the following steps: obtaining recorded information obtained when a user drives a vehicle with a specified type of driving assistance function; generating a situation description text of the user driving the vehicle in an estimation manner using LLM based on the obtained recorded information, and registering it in a situation description text DB; searching and recommending driving assistance functions based on the user's use from manuals related to the driving assistance functions of the vehicle based on the registered situation description text DB; and suggesting the searched driving assistance functions to the user in a specified form.

[0069] The driving assistance function recommendation method described in Appendix 7 of the present invention, similar to the driving assistance function recommendation system described in Appendix 1, can recommend driving assistance functions that seem helpful to the user but that the user may not be aware of.

[0070] The present invention can be suitably modified without departing from the spirit or concept of the invention as can be read from the scope of the claims and the entire specification, and the driving assistance function recommendation system and method accompanying such modifications are also included in the technical concept of the present invention.

[0071] Symbol Explanation

[0072] 100-Input Section, 102-Status Description Generation Section, 103-Status Description Search Section, 104-Manual Search Section, 105-Recommendation Generation Section, 107-Output Section, 201-Status Description DB, 202-Manual Data.

Claims

1. A driver assistance function recommendation system utilizing LLM, characterized in that, have: The acquisition unit acquires recorded information obtained when a user drives a vehicle with a specified type of driver assistance function; The estimation department, based on the acquired record information, generates a status statement of the user driving the vehicle in an estimation manner using LLM, and registers it in the status statement DB; The first search unit searches and recommends driving assistance functions based on the user's usage from the manuals related to the vehicle's driving assistance functions, according to the registered status description document DB. and The suggestion department recommends the searched driving assistance functions to the user in a prescribed manner.

2. The driving assistance function recommendation system utilizing LLM according to claim 1, characterized in that, It also has: The second search unit searches for similar cases related to the driver assistance function based on the registered status description document DB. The first search unit searches the manual for recommended driving assistance functions based on the user's usage, according to the registered status description document DB and similar cases found by the second search unit.

3. The driver assistance function recommendation system utilizing LLM according to claim 1 or 2, characterized in that, The suggestion section, as a prescribed form, suggests the searched driver assistance functions through LLM-textualized annotation statements.

4. The driver assistance function recommendation system utilizing LLM according to any one of claims 1 to 3, characterized in that, The suggestion unit makes suggestions to the user before the user drives the vehicle next time, based on at least one of the display output and voice output of an electronic device built separately from the vehicle.

5. The driver assistance function recommendation system utilizing LLM according to any one of claims 1 to 3, characterized in that, The suggestion unit provides suggestions to the user in real time while the user is driving or riding in the vehicle, based on at least one of the display output and voice output of the vehicle agent installed in the vehicle.

6. The driving assistance function recommendation system according to any one of claims 1 to 5, characterized in that, The estimation unit, as part of the acquired recorded information and / or the acquired recorded information, estimates the situation description based on at least one of the information related to the user's driving skills and the information related to the user's preferences.

7. A method for recommending driver assistance functions using LLM, characterized in that, The following steps are required: Acquire recorded information when a user drives a vehicle with specified types of driver assistance functions; Based on the obtained record information, a situation description text of the user driving the vehicle is generated in an estimation manner using LLM, and it is registered in the situation description text DB; Based on the registered status description document DB, search the manuals related to the vehicle's driving assistance functions for recommendations based on the user's usage; and The searched driving assistance functions are suggested to the user in a prescribed manner.

Images