Vehicle and driver status estimation system using LLM

Abstract

In a vehicle and driver situation estimation system using LLM, various situations can be identified without defining the scenes to be identified. [Solution] A vehicle (100) and driver status estimation system using LLM includes a conversion unit (205) that converts first-type input data relating to at least one of the vehicle status and the user status into second-type input data that is easier to understand than the first-type input data using LLM, and an estimation unit (202) that takes the converted input data as input and estimates the data relating to at least one of the above using LLM.

Description

Technical Field

[0001] The present invention relates to the technical field of a system for estimating the situation of a vehicle and a driver using an LLM.

Background Art

[0002] ' As a technology related to this type of system, there has been proposed a technology for determining the situation of a vehicle or an automobile and the situation of a driver or a user by combining various data related to the vehicle and the driver and using manual labor or a rule algorithm in which each situation is defined in advance (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, according to the above-mentioned background art, in order to estimate the situation of a vehicle or a driver using various data, it is necessary to develop an algorithm that combines the data well and matches the situation to be discriminated in advance. In order to automatically estimate the situation of a vehicle or a driver, there is a technical problem that a lot of learning of data related to the situation of the vehicle or the driver to be discriminated and data related to the scene to be discriminated is required.

[0005] An object of the present invention is to provide a system for estimating the situation of a vehicle and a driver using an LLM that can discriminate various situations with little or almost no or no definition of the scene to be discriminated.

Means for Solving the Problems

[0006] One embodiment of the vehicle and driver status estimation system using LLM according to the present invention, in order to solve the above problems, comprises a conversion unit that converts first type input data relating to at least one of the vehicle status and the user status into second type input data that is easier to understand than the first type using a predetermined type of LLM, and an estimation unit that takes the converted input data as input and estimates the data relating to at least one of the above using the LLM. [Effects of the Invention]

[0007] According to one aspect of the system according to the present invention, the input data is converted to data that is easy for LLM to understand, such as data expressed in language, and then estimated using LLM. Therefore, various common sense based on the expressed content becomes part of the basis for estimation when LLM is making estimations. This reduces the amount of raw data related to vehicles and users that needs to be input for estimation, and makes it possible to distinguish various situations without defining the scenes to be distinguished much, almost none, or at all.

[0008] The effects and benefits of the present invention will be further clarified by the embodiments of the invention described below. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing the overall configuration of the system according to the embodiment. [Figure 2] This is a flowchart showing an example of processing in the system according to the embodiment. [Modes for carrying out the invention]

[0010] First, with reference to Figure 1, the overall configuration of the vehicle and driver situation estimation system using LLM (Large Language Models) according to this embodiment (hereinafter simply referred to as the "situation estimation system") will be described.

[0011] The LLM used for estimation according to this embodiment may be a single-modal LLM that takes data in a verbalized format as input, or a multimodal LLM that takes not only data in a verbalized format as input, but also other data in a format that is easy for the LLM to understand. This embodiment is constructed as a system in which, for example, first-type input data relating to the status of the vehicle or the user (e.g., location information data from GPS, raw data from various sensors, etc.) is converted into second-type input data that is easier to understand than the first type using the type of LLM adopted in this embodiment (e.g., data plotting the locations indicated by the location information data on a map, data graphing the raw sensor data), the converted data is taken as input, the data that has not been verbalized from the converted data is verbalized, and then the LLM estimates the status of the vehicle or the user based on the verbalized data.

[0012] Furthermore, such AI learning or LLM learning can employ not only traditional AI learning systems such as supervised learning, unsupervised learning, or reinforcement learning, but also new technologies such as generative AI or LLM that have recently been put into practical use, are currently under development, or will be developed in the future. For example, the AI ​​learning or LLM learning described here may be constructed using a neural network that performs efficient learning through representation learning, transfer learning, feature selection, fine tuning or hyperparameter tuning, ensemble learning, etc.

[0013] As shown in Figure 1, the situation estimation system according to the embodiment is configured to include an on-board unit 101 and a server unit 200 mounted on a vehicle 100. The on-board unit 101 and the server unit 200 are connected to a communication network 10, such as the Internet or a dedicated network line. Other multiple or numerous vehicles 100 are also connected to the communication network 10 in the same way.

[0014] The communication network 10 also includes a general knowledge collection unit 301 and a map data collection unit 302 for collecting "general knowledge." In addition, the communication network 10 may include an external related knowledge collection unit (not shown) for collecting information obtained outside the vehicle 100 that can be used to perform fine-tuning or hyper-tuning on the situation estimation system to impart domain knowledge (i.e., "external related knowledge"). These general knowledge collection unit 301, map data collection unit 302, and the external related knowledge collection unit (not shown) may be at least partially located within the server unit 200 or within the facility where the server unit 200 is located, or within the vehicle-mounted unit 101 or inside the vehicle.

[0015] The server unit 200 is connected to a database (DB) 300 which stores various data, including data used in the situation estimation system. The DB 300 may be connected to the server unit 200 or the vehicle-mounted unit 101 via a communication network 10. The server unit 200 is composed of various computer-equipped devices and various computer devices that perform centralized or distributed processing. In other words, the situation estimation system is constructed as a system that performs centralized or distributed processing using the large-scale data in the DB 300.

[0016] In Figure 1, vehicle 100 may be, for example, a so-called HEV (Hybrid Electric Vehicle), PHEV (Plugin HEV), FCEV (Fuel Cell EV), BEV (Battery Electric Vehicle), etc., or it may be a vehicle primarily powered by an internal combustion engine, but it is constructed as a so-called connected car.

[0017] The in-vehicle unit 101 is comprised of an interface unit 111, a camera unit 112, a location information unit 113, various sensor units 114, a processing unit 115, and a communication unit 116.

[0018] The interface unit 111 is configured to communicate with the driver or user in the vehicle using voice and images. Specifically, the interface unit 111 is configured to allow input of, for example, the destination of the vehicle 100 and the conditions for selecting a planned route to the destination, using voice input or predetermined operations on an image. The selection of the planned route (i.e., the navigation function) may be configured to be executed entirely or partially by the processing unit 115, or partially or entirely by the processing unit 202 on the server unit 200 side (in other words, the in-vehicle unit 101 side may exclusively perform the function of a browser). The interface unit 111 is further configured to output data showing the estimation results obtained from the server unit 200 side in some format as voice or images as appropriate.

[0019] The camera unit 112 includes one or more cameras such as CCDs and is configured to function as a drive recorder that captures the interior and exterior of the vehicle. The camera unit 112 is configured to capture drive recorder video under the control of the processing unit 115 and to transmit the video data as appropriate from the communication unit 116 to the server unit 200 via the communication network 10.

[0020] The location information unit 113 includes a GPS device, an autonomous navigation positioning device, etc., and sequentially outputs the current position of the vehicle 100. The location information unit 113 is configured to generate raw location information data, such as latitude, longitude, and coordinates, under the control of the processing unit 115, and to transmit it as appropriate from the communication unit 116 to the server unit 200 via the communication network 10.

[0021] The various sensor units 114 each include various sensors such as a vehicle speed sensor, an acceleration sensor, a distance sensor, an engine speed sensor, a temperature sensor, an altitude sensor, a pressure sensor, a battery level sensor, and a user vital sensor, which are respectively deployed at predetermined positions inside the vehicle. The various sensor units 114 detect the current driving state of the vehicle 100 and various information 102a related to the driver of the vehicle 100, and are configured to be appropriately transmitted from the communication unit 116 to the server unit 200 side via the communication network 10 under the control of the processing unit 103 as CAN (Controlled Area Network) data or the like.

[0022] The processing unit 115 has a CPU, a memory, etc. that control the interface unit 111, the camera unit 112, the position information unit 113, the various sensor units 114, and the communication unit 116. Raw data detected or generated at various parts related to the vehicle 100 and its driver is appropriately transmitted from the communication unit 104 to the server unit 200 side in the form of data in a predetermined format. Furthermore, it is configured to appropriately receive data indicating the estimation result from the server unit 200 side via the communication unit 104.

[0023] The communication unit 116 includes a modem or the like configured to be communicable with the outside of the vehicle via the communication network 10. The communication unit 116 is configured to appropriately transmit various raw data collected by the vehicle 100 as input data of the first type to the server device 200 via the communication network 10 under the control of the processing unit 103.

[0024] In FIG. 1, the server device 200 is configured to include a communication unit 201, a processing unit 202, and a data conversion unit 205.

[0025] The communication unit 201 appropriately receives various data collected by the vehicle 100 via the communication network 10 under the control of the processing unit 202, appropriately receives general knowledge data collected by the general knowledge collection unit 3, and appropriately receives map data collected by the map data collection unit 302 via the communication network 10. The communication unit 201 is configured such that the received data is passed to the data conversion unit 205.

[0026] The data conversion unit 205 is configured to convert Type 1 input data (for example, raw data transmitted from the in-vehicle unit 101 received by the communication unit 201) into Type 2 input data that is easier for LLM to understand. The conversion process will be described in detail later with reference to Figure 2.

[0027] The processing unit 202 is configured to take this converted second-type input data as input and perform processing related to situation estimation using LLM. This processing will also be described in detail later with reference to Figure 2.

[0028] DB300 is configured to include a large-scale and high-speed data input / output storage device that stores various data received by the server unit 200 via the communication network 10, in particular various data necessary for estimation processing using LLM, data related to the estimation results or intermediate results generated by the processing unit 202, and data converted by the data conversion unit 205.

[0029] The general knowledge collection unit 301 is configured to collect "general knowledge" about drivers and vehicles, such as vehicles 100, the movement of vehicles 100, roads, streetscapes, buildings, general drivers, pedestrians, road structures, and road maps, and to transmit it to the server unit 200 as appropriate via the communication network 10. When the processing unit 202 in the server unit 200 performs estimation processing using LLM, the amount of raw data required for estimation by LLM can be reduced by utilizing such general knowledge.

[0030] The map data collection unit 302 is configured to collect road maps, multipurpose maps, etc., and transmit them to the server unit 200 as appropriate via the communication network 10. Prior to the processing unit 202 in the server unit 200 performing estimation processing using LLM, the location information of the vehicle 100 is plotted on the collected map as described later with reference to Figure 2, and the location information data is converted into data that is easier to understand using LLM.

[0031] Next, with reference to the block diagram in Figure 1 and the flowchart in Figure 2, an example of processing in the situation estimation system according to this embodiment (in particular, processing executed using LLM in the processing unit 202 of the server unit 200) will be explained.

[0032] In Figure 2, first, image data related to the drive recorder video captured by the camera unit 112 is input to the conversion unit 205 either directly from the communication unit 201 or via the processing unit 202 (step S1).

[0033] Then, the conversion unit 205 performs a data conversion process on the input image data (step S2) using LLM to verbalize the situation of the user, driver, or vehicle as determined from common sense and the drive recorder video. In other words, the conversion unit 205 generates language data corresponding to the drive recorder video as one of the second type of data. That is, here the image data is an example of the first type of data, and the converted language data is an example of the second type of data. The conversion unit 205 performs this conversion process using LLM (i.e., an LLM process different from the LLM process for situation estimation in the processing unit 202) as preparation for situation estimation.

[0034] In parallel with or preceding steps S1 and S2 described above, the conversion unit 205 receives location information data generated, acquired, or determined by the location information unit 113, either directly from the communication unit 201 or via the processing unit 202 (step S3).

[0035] Then, the conversion unit 205 generates data in a format in which the locations indicated by the location information data are plotted on a map that has been previously collected by the map information collection unit 302 and includes the locations indicated by the location information data (step S4). Furthermore, it applies a data conversion process to the input location information data using LLM to verbalize the situation of the user, driver, or vehicle as determined from common sense and the plot on the map (step S5). In other words, the conversion unit 205 generates language data corresponding to the location information data as one of the second type of data. That is, the location information data is an example of the first type of data, and the converted language data is an example of the second type of data. The conversion unit 205 performs this conversion process using LLM (i.e., an LLM process different from the LLM process for situation estimation in the processing unit 202) as preparation for situation estimation.

[0036] In step S5, for example, the location data is translated into language data such as, "The most recent location is the intersection in front of Tokyo Tower. Tokyo Tower is visible to the right."

[0037] In parallel with or preceding steps S1 and S2 and steps S3 to S5 described above, the conversion unit 205 receives various sensor value data generated, acquired, or determined by the various sensor units 114, either directly from the communication unit 201 or via the processing unit 202 (step S6).

[0038] Then, the conversion unit 205 generates data in a graph format of the various sensor value data (step S7), and further performs a data conversion process on the input various sensor value data using LLM to verbalize the user, driver, or vehicle situation as determined from common sense and the graph (step S8). In other words, the conversion unit 205 generates language data corresponding to the various sensor value data as one of the second type of data. That is, here the various sensor value data is an example of first type data, and the converted language data is an example of second type data. The conversion unit 205 performs this conversion process using LLM (i.e., an LLM process different from the LLM process for situation estimation in the processing unit 202) as preparation for situation estimation.

[0039] In step S8, for example, language data corresponding to various sensor value data is generated, such as "User A is driving. The vehicle is coming to a stop from a speed of 50 kph over 20 seconds."

[0040] Once the above-described language processing (steps S2, S5, and S8) is completed, the processing unit 202 performs a situation analysis in the LLM by combining information from general knowledge, language data input from the conversion unit 205, and unlanguaged data (step S9). For example, it estimates a languageized situation such as, "User A is heading from Keio University to Otemachi Building. Currently, they are stopped at the Tokyo Tower intersection."

[0041] Next, the processing unit 202 uses LLM to estimate the user and vehicle status (step S10). For example, it makes an estimation such as, "It can be estimated that the vehicle is currently waiting at a traffic light at the Tokyo Tower intersection," and outputs a situation description as appropriate.

[0042] In order to efficiently perform the language conversion process (steps S2, S5, S8) and the situation estimation process (steps S9 and S10) using LLM in the conversion unit 205 and processing unit 202, it is advisable to convert all converted plots and graphs into text using LLM and then vectorize them. Furthermore, in each of the LLM-based processes described above, a large-scale language model may be fine-tuned using LLM with a large amount of text data. This makes it possible to adapt to various natural language processing (NLP) tasks such as text classification, sentiment analysis, information extraction, text summarization, text generation, and question answering.

[0043] In this embodiment, the conversion unit 205 and the processing unit 202 are represented as separate functional blocks that perform different processing functions. However, they may be composed of a single processor or the like as hardware, or they may be configured to be functionally treated separately in software.

[0044] Once the situation estimation using LLM by processing unit 202 is complete, the series of processes is terminated. This series of processes may be called periodically or irregularly as a subroutine that can be executed repeatedly in a short time, and executed as needed.

[0045] As explained in detail above, this embodiment allows for the identification of various situations without defining the scenes to be identified, by taking advantage of the LLM's characteristics that enable reasoning based on common sense. In this case, common sense becomes part of the basis for estimation when LLM is used in processes such as steps S2, S5, S8, S9, and S10, thus reducing the amount of raw data input related to vehicles and users required for estimation.

[0046] Since it is difficult to directly make vehicle data understandable to an LLM, as in this embodiment, visualizing the data on a map in step S5, for example, or graphing sensor values ​​in step S8, for example, enables a representation that is easy for an LLM or multimodal LLM to read, which is extremely advantageous from the standpoint of efficiently improving the accuracy of situation estimation.

[0047] According to this embodiment, by using LLM with not only numerical data, such as in previous technologies or background technologies, but also verbalized data as input, it becomes possible to extract relevant features not only from common sense but also from textual data such as papers and related literature, and to perform situation estimation at a higher level.

[0048] Note The following additional information is disclosed regarding the embodiments described above.

[0049] [Note 1] The situation estimation system described in Appendix 1 of the present invention comprises a conversion unit that converts first type input data relating to at least one of the vehicle situation and the user situation into second type input data that is easier to understand than the first type using a predetermined type of LLM, and an estimation unit that takes the converted input data as input and estimates the data relating to at least one of the above using the LLM.

[0050] According to the situation estimation system described in Appendix 1, LLM estimates data that has been verbalized or otherwise made easier to understand as at least part of the input. Therefore, various common sense facts based on verbalized content (for example, vehicles travel on roads, vehicles stop at red lights, the location of known landmarks, the spatial relationships between specific buildings, etc.) become part of the basis for estimation when LLM is used in the estimation unit. This reduces the amount of raw data input related to vehicles and users required for estimation. Consequently, it becomes possible to determine various situations with relatively high accuracy and efficiency, even without defining the scenes to be determined in much, little, or no way.

[0051] [Note 2] The situation estimation system according to Appendix 2 of the present invention is characterized in that the input data of the first type includes at least one of the vehicle's sensor value data and the vehicle's location information, the conversion unit converts the sensor value data into graphed data, and in lieu of or in addition to converts the location information into data plotted on a map, and the LLM of the estimation unit includes a multimodal LLM that includes the graphed data and the plotted data as input for estimation.

[0052] According to the situation estimation system described in Appendix 2 of the present invention, the estimation unit uses LLM to estimate data that is easier to understand with LLM than raw sensor value data, or data plotted on a map that is easier to understand with LLM than raw location information, as at least part of the input. This makes it possible to improve the accuracy and efficiency of the estimation process using LLM.

[0053] [Note 3] The situation estimation system described in Appendix 3 of the present invention is a situation estimation system described in Appendix 1 or 2, characterized in that the estimation unit includes verbalized navigation information, including the vehicle's departure information and destination information, which has not been converted or has been converted by the conversion unit, as input for estimation.

[0054] According to the situation estimation system described in Appendix 3 of the present invention, the estimation unit estimates verbalized navigation information, rather than location information of the departure point and destination or link information connecting them, using LLM as at least part of the input. Therefore, it is possible to improve the accuracy and efficiency of the estimation process using LLM.

[0055] [Note 4] The situation estimation system described in Appendix 4 of the present invention is a situation estimation system described in any one of Appendix 1 to 4, characterized in that the conversion unit converts the input data of the first type to the input data of the second type using LLM.

[0056] According to the situation estimation system described in Appendix 4 of the present invention, the conversion unit performs data conversion using LLM, so that not only the estimation unit but also the conversion unit can perform conversion by estimation at a high level by using text data such as common sense, papers, and related literature.

[0057] [Note 5] The situation estimation system described in Appendix 5 of the present invention is the situation estimation system described in Appendix 4, characterized in that the input data of the first type includes image data of the vehicle's drive recorder video, the conversion unit converts the image data into text data by language processing using the LLM, and the LLM of the estimation unit includes the converted text data as input for estimation.

[0058] According to the situation estimation system described in Appendix 5 of the present invention, the conversion unit converts image data into text data by language processing using LLM. Therefore, not only the estimation unit but also the conversion unit can perform conversions based on high-dimensional estimation using common sense or text data such as papers and related literature.

[0059] [Note 6] The analysis and improvement suggestion method described in Appendix 6 of the present invention comprises the steps of: converting input data of a first type relating to at least one of the vehicle's condition and the user's condition into input data of a second type that is easier to understand than the first type using a predetermined type of LLM; and estimating the data relating to at least one of the above using the LLM with the converted input data as input.

[0060] According to the situation estimation method described in Appendix 6 of the present invention, similar to the situation estimation system described in Appendix 1, various common sense based on verbalized content becomes part of the estimation basis when LLM is used for estimation in the estimation unit, and various situations can be determined relatively efficiently and with high accuracy even without defining the scenes to be determined little, almost, or at all.

[0061] The present invention may be modified as appropriate, without contradicting the gist or spirit of the invention as can be inferred from the claims and the specification as a whole, and situation estimation systems and methods involving such modifications are also included in the technical concept of the present invention. [Explanation of Symbols]

[0062] Vehicles...100 Camera section... 112 Location information department (GPS)……113 Various sensor units...102 Processing section...115 Server section...200 Processing section...202 Data conversion section...205 DB...300 General Knowledge Collection Department...301 Map data collection department...302

Claims

1. A conversion unit that converts input data of type 1 relating to at least one of the vehicle status and the user status into input data of type 2 which is easier to understand than type 1 using a predetermined type of LLM, The LLM uses the converted input data as input to estimate the data relating to at least one of the above. A vehicle and driver status estimation system using LLM, characterized by comprising the following:

2. The input data for the first type includes at least one of the vehicle's sensor value data and the vehicle's location information. The conversion unit converts the sensor value data into graphed data, and in exchange for or in addition to this, converts the location information into data plotted on a map. The LLM of the estimation unit includes a multimodal LLM that includes the graphed data and the plotted data as inputs for estimation. A vehicle and driver status estimation system using LLM as described in feature 1.

3. The estimation unit takes into account verbalized navigation information, including the vehicle's departure and destination information, which may or may not have been converted by the conversion unit, as input for estimation. A vehicle and driver status estimation system using LLM according to claim 1 or 2, characterized by the above.

4. The conversion unit is characterized in that it converts the input data of the first type to the input data of the second type using LLM, as described in any one of claims 1 to 4, for estimating the status of a vehicle and driver using LLM.

5. The input data for the first type includes image data from the vehicle's drive recorder, The conversion unit converts the image data into text data by language processing using the LLM. The LLM of the estimation unit performs estimation by including the converted text data as input. A vehicle and driver status estimation system using LLM according to feature 4.

6. A step of converting input data of type 1 relating to at least one of the vehicle status and the user status into input data of type 2 which is easier to understand than type 1 using a predetermined type of LLM, The steps include: using the converted input data as input, estimating the data relating to at least one of the above in the LLM; A method for estimating the situation of a vehicle and driver using LLM, characterized by comprising the following:

Images