Fully localized vehicle condition monitoring AI device and driving assistance method

Abstract

It is difficult for drivers to intuitively understand the vehicle's condition based solely on numerical information from the vehicle diagnostic port, and cloud-based AI services have communication delays and privacy risks. Furthermore, they do not provide sufficient detailed advice that takes into account the driving environment, such as traffic congestion, and the driver's condition. [Solution] This device is equipped with a vehicle information acquisition means that acquires information such as coolant temperature from an OBD-standard vehicle diagnostic port, a first camera that captures images of the road ahead, a second camera that captures images of the driver, and a microphone that captures the driver's voice. Features are generated from this multimodal information and the driver profile, and a compact language model specialized for the vehicle field outputs a vehicle status description, risk level, and recommended actions in natural language, taking congestion levels into account. All processing is completed within the edge device without using the cloud, and the device can be used in multiple vehicles as a portable unit.

Description

Technical Field

[0001] The present invention relates to vehicle condition monitoring and safe driving support. More specifically, the present invention integrates sensor information obtained from a vehicle diagnostic port, a plurality of cameras that image the road ahead and the driver, and multi-modal information obtained from a microphone, and generates driving support messages using a language model specialized for the vehicle field without using the cloud. The present invention relates to a completely local type vehicle condition monitoring AI device and a driving support method.

Background Art

[0002] Conventionally, in order to check the conditions of a vehicle such as water temperature, oil temperature, and error codes, a simple diagnostic device connected to a vehicle diagnostic port compliant with the OBD standard or a scan tool that cooperates with a smartphone has been used. These notify abnormalities by numerical display or the lighting of a warning lamp, but it is difficult for a driver to understand the transition of detailed values during driving, and it is impossible to give detailed advice according to the usage and anxiety of each driver's vehicle. In addition, a telematics service has been proposed that links a large-scale language model on the cloud with in-vehicle devices, transmits information from the vehicle to the cloud, and generates explanations and advice. However, these depend on the communication environment, and there are concerns about communication delays, communication interruptions, and privacy due to the transmission of driving logs and location information to an external server. On the other hand, due to the progress of edge AI, technologies for performing image recognition and voice recognition on a small computer and technologies for operating a small language model specialized for a limited field within a terminal are known. However, a portable device that integrates a plurality of modal information such as vehicle diagnostic data, front road video, driver video, and voice, evaluates the vehicle condition, traffic condition, and driver condition together, and generates customized advice for each driver and vehicle in a completely local and real-time manner has not been fully realized.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

[0004] [Non-Patent Document 1] Non-patent document 1: MAA Khan et al., “A Machine Learning Approach for Driver Identification Using On-Board Diagnostics (OBD-II) Data,” arXiv:2207.10807, 2022. Non-patent document 2: A. Halin et al., “Survey and Synthesis of State of the Art in Driver Monitoring,” Sensors, 21(16), 5558, 2021. [Overview of the project] [Problems that the invention aims to solve]

[0005] The objective of the present invention is to provide a fully local vehicle condition monitoring AI device and driving assistance method that integrates numerical information acquired from a vehicle diagnostic port with video and audio of the road ahead and the driver, and presents it to the driver as an easy-to-understand Japanese message using a vehicle-specific language model without relying on the cloud. This allows for early detection of signs of vehicle trouble such as overheating, while also considering driving conditions such as traffic congestion to reduce driver anxiety. [Means for solving the problem]

[0006] A vehicle condition monitoring AI device according to one aspect of the present invention comprises: a vehicle information acquisition means connected to a vehicle diagnostic port; a first camera means for imaging the road ahead; a second camera means for imaging the driver; a microphone for capturing the driver's voice; a feature generation means for generating multimodal features based on data from the vehicle information acquisition means, the first and second camera means, and the microphone; a generative AI inference means including a vehicle domain-specific language model that generates the vehicle's condition, risk level, and recommended actions in natural language using the multimodal features and driver profile as input; a display means and a speech synthesis means for displaying the output of the generative AI inference means; a control means for executing each of the means on a single edge device without communicating with a cloud server; and a portable housing equipped with a connector connected to a vehicle diagnostic port and a power supply means capable of being powered from an in-vehicle power supply or a mobile battery. In a preferred embodiment, the vehicle information acquisition means acquires at least one of the following from a vehicle diagnostic port compliant with the OBD standard: coolant temperature, oil temperature, oil pressure, engine speed, vehicle speed, and diagnostic trouble code. The generating AI inference means evaluates the risk of overheating and the risk of poor lubrication based on these time-series changes and features indicating the traffic congestion on the road ahead, and generates a driver assistance message including the evaluation results. In a more preferred embodiment, the vehicle-specific language model is obtained by distilling and fine-tuning a small language model using texts related to vehicle engineering and maintenance manuals, etc., and responses obtained from a large language model as training data, and is quantized to a size that allows real-time inference on edge devices. Another aspect of the present invention relates to a driving assistance method that includes the steps of: using the above-described vehicle condition monitoring AI device to acquire vehicle diagnostic data and video and audio from first and second cameras; generating multimodal features based on these; generating driving assistance messages using the vehicle domain-specific language model; and displaying and outputting the generated messages. [Effects of the Invention]

[0007] According to the present invention, vehicle status monitoring and natural language-based driving assistance can be achieved using only edge devices within the vehicle, without the need for the cloud. This eliminates the impact of communication delays and interruptions, and also offers privacy advantages as driving logs and location information are not transmitted externally. Furthermore, the device of the present invention is configured to be portable and can be easily carried between multiple vehicles using a cigarette lighter socket or mobile battery, making it applicable to a wide range of vehicles, including sports cars, classic cars, rental cars, and company cars. In addition, because the components are modular, it can be used as a basic technology when filing separate applications for each function in the future, such as a driving environment recognition function, a driver state estimation function, a vehicle diagnosis and predictive maintenance function, and an interactive driving assistance function. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example configuration of the vehicle condition monitoring AI device of the present invention. [Figure 2] This figure shows an example of the processing flow in the driver assistance method of the present invention. [Figure 3] This is a schematic diagram showing an example of a coolant temperature monitoring screen and driver assistance messages during traffic congestion. [Modes for carrying out the invention]

[0009] 1) Hardware configuration As shown in Figure 1, the vehicle condition monitoring AI device 10 of the present invention comprises a vehicle information acquisition unit 12 connected to a vehicle diagnostic port 11, a first camera 13A for imaging the road ahead, a second camera 13B for imaging the driver, a microphone 14 for capturing the driver's speech, an edge computing device 15, a display unit 16, an audio output unit 17, a power supply unit 18, and a housing 19. The vehicle diagnostic port 11 is a vehicle diagnostic port compliant with the OBD standard, and the vehicle information acquisition unit 12 periodically acquires at least one of the following: coolant temperature, engine speed, vehicle speed, oil temperature, oil pressure, and diagnostic trouble codes (DTCs). The acquisition period can be, for example, 0.5 seconds to 2 seconds. The edge computing device 15 consists of a single-board computer and an edge AI accelerator. Specifically, a general-purpose operating system may run on the single-board computer, and neural network models may be inferred at high speed on the edge AI accelerator. The edge computing device 15 does not require a constant connection to the internet or an external cloud server and can complete inference processing independently. The display unit 16 may be a small display or a display screen wirelessly connected to the driver's mobile device. The audio output unit 17 consists of a vehicle-built speaker or an external speaker. The power supply unit 18 may include a cigarette lighter socket connection plug for receiving power from the vehicle's power supply and a power input unit for receiving power from a mobile battery or the like. The first camera 13A captures images of the road ahead and the driving environment, including the brake lights of the preceding vehicle, and is used to determine congestion and low-speed driving conditions. The second camera 13B captures images of the driver's face and upper body and is used to estimate the driver's condition, such as blinking frequency, gaze, and posture. These multiple cameras allow for a comprehensive evaluation of not only the vehicle condition but also the driving environment and the driver's condition. The following software modules run on the edge computing device 15. OBD data acquisition module 20: Controls the vehicle information acquisition unit 12 to acquire various sensor values ​​and convert them into physical quantities. Image acquisition module 21: Acquires images from the first camera 13A and the second camera 13B, and extracts the forward road area, preceding vehicle, instrument panel area, and driver's face area as needed. Voice acquisition module 22: Acquires the audio waveform from microphone 14 and sends it to the voice recognition module. Feature generation module 23: Integrates time-dependent changes in OBD data, feature vectors such as congestion levels and stop / start patterns extracted from forward road images, alertness indicators extracted from driver images, and speech recognition results as multimodal features. Vehicle-specific language model 24: A compact language model specialized for vehicle state descriptions and driving advice, which generates natural language messages using the multimodal features and driver profile 25 as input. Speech synthesis module 26: Synthesizes the generated text into Japanese speech and outputs it from the speech output unit 17. Log recording module 27: Encrypts driving history and speech history and saves them to local storage. The driver profile 25 records information such as the type of vehicle, engine specifications, normal water and oil temperature ranges, past trouble cases, and the driver's preferred explanation style (concise / detailed, etc.). This allows for the generation of different comments depending on the vehicle and driver, even with the same numerical values. As shown in Figure 2, the driving assistance method of the present invention generally consists of the following steps. 1) OBD data acquisition step S1: The OBD data acquisition module 20 acquires various sensor values ​​from the vehicle diagnostic port 11. 2) Image acquisition step S2: The image acquisition module 21 acquires frames from the first camera 13A and the second camera 13B and analyzes the traffic conditions on the road ahead and the driver's condition. 3) Voice acquisition step S3: The voice acquisition module 22 acquires the driver's speech and converts it to text using speech recognition. 4) Feature Integration Step S4: The feature generation module 23 generates multimodal features by integrating the temporal changes in OBD data, the degree of congestion and the number of stops and starts extracted from the road image ahead, the alertness index extracted from the driver image, the speech recognition results, and the driver profile 25. 5) Language model inference step S5: The vehicle domain-specific language model 24 takes the multimodal features as input and generates driver assistance messages including a description of the vehicle state, risk level, and recommended actions. 6) Output step S6: The generated message is displayed on the display unit 16 and read aloud from the voice output unit 17 by the speech synthesis module 26. For example, when driving slowly on a congested road, it is detected that the brake lamp of the preceding vehicle frequently lights up in the image obtained from the first camera 13A, and the vehicle speed in the OBD data repeatedly alternates between low speed and stop in a short cycle. When the cooling water temperature rapidly rises immediately after the start of operation in such a situation and is higher than the normal range of the vehicle recorded in the driver profile 25, the vehicle field-specific language model 24 can generate an explanation such as "Currently, you are driving slowly on a congested road and the coolant temperature is higher than normal. Avoid long-term idling. If possible, take an early break after leaving the congested section. If the temperature does not drop, please have it inspected at a repair shop." and notify the driver. When the risk level reaches a predetermined high level, a warning sound or a highly urgent message is output from the voice output unit 17, and an instruction such as "Immediately stop on the shoulder and turn off the engine." is given as necessary. Based on the driver profile 25, it is also possible to switch to a mode of issuing a warning earlier, a mode of giving a concise explanation, etc. In the above embodiment, an internal combustion engine vehicle is exemplified, but the present invention can also be applied to monitoring the battery temperature and inverter temperature in hybrid vehicles and electric vehicles. In that case, the parameters acquired as OBD data may be replaced with motor temperature, battery remaining amount, charge / discharge current, etc. Also, it may be configured to omit imaging the forward road by the first camera 13A and generate a driving support message only from the temporal change of the OBD data and the driver's voice, or to be configured to specialize only in estimating the driver's state by the second camera 13B. Conversely, it may be configured to detect dangerous driving and near misses only from the forward road video and the driver video without using OBD data. These modified examples are obtained by separating the components of the present invention by function, and will serve as a technical basis when filing divisional applications as a driving environment recognition device, a driver state estimation device, a vehicle state diagnosis device, a field-specific language model device, etc. respectively in the future. Furthermore, in response to the driver's voice input, a conversation function can be added so that the device of the present invention can answer questions such as "What is the current water temperature?" and "Is there any burden on the engine in this traffic jam?" in natural language by referring to the latest OBD data and the degree of traffic congestion on the forward road. Such an interactive driving support function is also included in this specification. Note that the company names, product names, and software names described in this specification are generally registered trademarks or trademarks of each company.

Example

[0010] As a specific example of the present invention, a configuration using Raspberry Pi 5 as a single-board computer and Hailo-8L as an edge AI accelerator can be mentioned. Also, as the vehicle field-specific language model, a model distilled and fine-tuned using a learning dataset consisting of vehicle engineering, maintenance manuals, actual vehicle test logs, etc. based on an open-source small language model can be used. However, it is not limited to these specific examples, and other hardware and software having equivalent functions may be used.

Industrial Applicability

[0011] The fully local vehicle state monitoring AI device and driving support method of the present invention can be widely used not only for personal passenger cars and sports cars but also for the operation management of company cars, transport vehicles, car-sharing vehicles, rental cars, etc., and further for vehicle state monitoring during circuit driving. Since it does not depend on the cloud, it is easy to introduce in areas with poor communication environments and applications with strict privacy requirements, and is useful in the entire automotive-related industry. It can also be applied to preventive maintenance and safe driving education in fields such as transportation, bus operators, construction machinery, and agricultural machinery.

Explanation of Signs

[0012] 10 Vehicle state monitoring AI device 11 Vehicle diagnostic port 12 Vehicle information acquisition unit 13A First camera (forward road imaging) 13B Second camera (driver imaging) 14 microphones 15 Edge Computing Devices 15A Single-board computer (e.g., Raspberry Pi 5) 15B Edge AI accelerator (e.g., Hailo-8L) 16 displays 17. Audio output section 18 power supply section 19 cabinets 20 to 27 various software modules

Claims

1. A portable vehicle condition monitoring AI device connected to a vehicle having a vehicle diagnostic port, (A) A vehicle information acquisition means electrically connected to the vehicle diagnostic port and periodically acquiring at least one of the following: coolant temperature, oil temperature, oil pressure, engine speed, vehicle speed, and diagnostic trouble code; (B) A first camera means for imaging the road ahead, (C) A second camera means for imaging the driver, (D) A microphone for recording the driver's voice, (E) A feature generation means that generates multimodal features based on data from the vehicle information acquisition means, image data from the first camera means and the second camera means, and audio data from the microphone, (F) A generative AI inference means including a vehicle domain-specific language model that takes the multimodal features and driver profile information as input to generate driver assistance messages in natural language indicating the vehicle status, risk level, and recommended actions, (G) Display means for displaying the driving assistance message and speech synthesis means for outputting the driving assistance message as synthesized speech, (H) A control means that executes the vehicle information acquisition means, feature generation means, generation AI inference means and speech synthesis means on a single edge device without communicating with a cloud server, (I) A housing equipped with a connector for connecting to the vehicle diagnostic port and a power supply means capable of being powered from an on-board power supply or a mobile battery, A vehicle condition monitoring AI device characterized by comprising the following:

2. In the vehicle condition monitoring AI device according to claim 1, The aforementioned vehicle diagnostic port complies with the OBD standard, A vehicle condition monitoring AI device characterized in that the generating AI inference means evaluates the risk of overheating based on the temporal changes in the coolant temperature and oil temperature, and generates a driving support message including the evaluation result.

3. In the vehicle condition monitoring AI device according to claim 1, The feature generation means calculates a congestion index based on the forward road image obtained from the first camera means and the vehicle speed data obtained from the vehicle information acquisition means. The vehicle condition monitoring AI device is characterized in that the generating AI inference means generates a driver assistance message, including recommended actions regarding engine cooling during traffic congestion, based on the traffic congestion index and the temporal change in the coolant temperature.

4. In the vehicle condition monitoring AI device according to claim 1, The vehicle-specific language model is obtained by distilling and fine-tuning a small language model based on training data including texts related to vehicle engineering and maintenance manuals and responses obtained from a large language model, and is quantized to enable inference on an edge device. This is the characteristics of the vehicle condition monitoring AI device.

5. In the vehicle condition monitoring AI device according to claim 1, The system further includes a profile storage means for storing driver profile information, The vehicle status monitoring AI device is characterized in that the generating AI inference means updates the driver profile information based on past driving history and the driver's speech history, and adjusts the content of the driving assistance message or the warning threshold according to the updated driver profile information.

6. In the vehicle condition monitoring AI device according to claim 1, The vehicle condition monitoring AI device is characterized in that the generating AI inference means instructs the voice synthesis means to output an emergency warning message when the risk level reaches a predetermined high level, and displays to the display means a message indicating that the vehicle should stop driving and park in a safe place.

7. A driving assistance method using the vehicle condition monitoring AI device described in claim 1, The process involves obtaining sensor values ​​and diagnostic trouble codes from the vehicle's diagnostic port, The process involves acquiring an image of the road ahead using the first camera, The process involves acquiring an image of the driver using a second camera, The process of acquiring the driver's voice using a microphone, A step of generating multimodal features based on the acquired data, The process involves inputting the multimodal features and driver profile information into the vehicle domain-specific language model and generating driver assistance messages in natural language. The process involves displaying and outputting the generated driver assistance message as audio, A driving assistance method comprising the above steps, characterized in that at least the multimodal feature generation and the inference by the language model are completed within the vehicle condition monitoring AI device without going through a cloud server.

Images