Completely local vehicle condition monitoring AI device and driving assistance method

Abstract

It is difficult for drivers to intuitively understand the vehicle's condition based solely on numerical information from the vehicle diagnostic port, and cloud-based AI services have communication delays and privacy risks. Furthermore, they do not provide sufficient detailed advice that takes into account the driving environment, such as traffic congestion, and the driver's condition. [Solution] This device is equipped with a vehicle information acquisition means that acquires information such as coolant temperature from an OBD-standard vehicle diagnostic port, a first camera that captures images of the road ahead, a second camera that captures images of the driver, and a microphone that captures the driver's voice. Features are generated from this multimodal information and the driver profile, and a compact language model specialized for the vehicle field outputs a vehicle status description, risk level, and recommended actions in natural language, taking congestion levels into account. All processing is completed within the edge device without using the cloud, and the device can be used in multiple vehicles as a portable unit.

Description

[Technical Field]

[0001] The present invention relates to automobile state monitoring and safe driving assistance, and more particularly to a fully local vehicle state monitoring AI device and driving assistance method that integrates sensor information acquired from a vehicle diagnosis port with multimodal information acquired from multiple cameras that capture images of the road ahead and the driver, and from microphones, and generates driving assistance messages using a language model specialized for the vehicle field without using the cloud. [Background technology]

[0002] Conventionally, simple diagnostic devices that connect to vehicle diagnostic ports that comply with the OBD standard or scan tools that connect to smartphones have been used to check the status of a vehicle's water temperature, oil temperature, error codes, etc. These notify of abnormalities by displaying numerical values ​​or turning on warning lights, but it is difficult for drivers to understand the detailed changes in values ​​while driving, and they are unable to provide detailed advice tailored to each driver's vehicle usage habits and concerns. In addition, telematics services have been proposed that link large-scale language models on the cloud with in-vehicle devices, sending information from the vehicle to the cloud to generate explanations and advice. However, these services depend on the communication environment, and there are concerns about communication delays and interruptions, as well as privacy issues due to the transmission of driving logs and location information to external servers. Meanwhile, advances in edge AI have led to the development of technologies for image and voice recognition on small computers, as well as technologies for running small language models specialized for limited fields within devices. However, portable devices that integrate multi-modal information (vehicle diagnostic data, road ahead images, driver images, and voice), evaluate vehicle status, traffic conditions, and driver status, and then generate customized advice for each driver and vehicle in real time, completely locally, have not yet been fully realized. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Document 1: JP 2002-257690 A, "Vehicle diagnostic system and automobile using said system" Patent Document 2: JP 2009-301367 A, "Driver state estimation device" [Non-patent literature]

[0004] [Non-Patent Document 1] Non-Patent Document 1: MAA Khan et al., “A Machine Learning Approach for Driver Identification Using On-Board Diagnostics (OBD-II) Data,” arXiv:2207.10807, 2022. Non-Patent Document 2: A. Halin et al., “Survey and Synthesis of State of the Art in Driver Monitoring,” Sensors, 21(16), 5558, 2021. Summary of the Invention [Problem to be solved by the invention]

[0005] The object of the present invention is to provide a completely local vehicle condition monitoring AI device and driving assistance method that multi-modally integrates numerical information obtained from a vehicle diagnosis port with images of the road ahead and the driver, and the driver's voice, and presents the information to the driver as an easy-to-understand Japanese message using a vehicle-specific language model without relying on the cloud, thereby enabling early detection of signs of vehicle trouble such as overheating while also taking into account driving conditions such as traffic congestion to reduce driver anxiety. [Means for solving the problem]

[0006] A vehicle condition monitoring AI device according to one embodiment of the present invention comprises: a vehicle information acquisition means connected to a vehicle diagnosis port; a first camera means for capturing an image of the road ahead; a second camera means for capturing an image of the driver; a microphone for collecting the driver's voice; a feature generation means for generating multimodal features based on data from the vehicle information acquisition means, the first and second camera means, and the microphone; a generative AI inference means including a vehicle-specific language model that uses the multimodal features and a driver profile as input to generate the vehicle condition, risk level, and recommended actions in natural language; a display means and a speech synthesis means for displaying the output of the generative AI inference means; a control means for executing each of the above means on a single edge device without communicating with a cloud server; and a portable housing equipped with a connector for connecting to the vehicle diagnosis port and a power supply means capable of supplying power from an on-board power supply or a mobile battery. In a preferred embodiment, the vehicle information acquisition means acquires at least one of coolant temperature, oil temperature, oil pressure, engine RPM, vehicle speed, and diagnostic trouble code from a vehicle diagnostic port that complies with the OBD standard, and the AI ​​generation inference means evaluates the risk of overheating or poor lubrication based on time-series changes in these data and features that indicate the traffic congestion situation on the road ahead, and generates a driving assistance message that includes the evaluation results. In a further preferred embodiment, the vehicle domain-specific language model is obtained by distilling and fine-tuning a small language model using text related to vehicle engineering and maintenance manuals, etc. and responses obtained from a large-scale language model as training data, and is quantized to a size that allows real-time inference on an edge device. A driving assistance method according to another aspect of the present invention includes the steps of using the above-mentioned vehicle condition monitoring AI device to acquire vehicle diagnostic data and video and audio from the first and second cameras, generating multimodal features based on the data, generating a driving assistance message using the vehicle field-specific language model, and displaying and audibly outputting the generated message. [Effects of the Invention]

[0007] According to the present invention, vehicle condition monitoring and driving assistance using natural language can be achieved using only edge devices inside the vehicle without using the cloud, which means that there is no impact from communication delays or interruptions, and there are privacy benefits in that driving logs and location information are not transmitted to the outside. Furthermore, the device of the present invention is configured as a portable device and can be easily carried between multiple vehicles using cigarette lighter power supply or a mobile battery, making it applicable to a wide range of vehicles, including sports cars, classic cars, rental cars, company cars, etc. In addition, because the components are modularized, it can also be used as the basic technology for filing divisional applications for each function in the future, such as a driving environment recognition function, a driver state estimation function, a vehicle diagnosis / sign detection function, and an interactive driving assistance function. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of the configuration of a vehicle state monitoring AI device according to the present invention; [Figure 2] FIG. 1 is a diagram showing an example of a processing flow in a driving assistance method of the present invention. [Figure 3] 10 is a schematic diagram showing an example of a cooling water temperature monitoring screen and a driving assistance message during traffic congestion. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0009] 1) Hardware configuration As shown in Figure 1, the vehicle condition monitoring AI device 10 of the present invention includes a vehicle information acquisition unit 12 connected to a vehicle diagnosis port 11, a first camera 13A that captures an image of the road ahead, a second camera 13B that captures an image of the driver, a microphone 14 that picks up the driver's speech, an edge calculation device 15, a display unit 16, an audio output unit 17, a power supply unit 18, and a housing 19. The vehicle diagnostic port 11 is a vehicle diagnostic port that complies with the OBD standard, and the vehicle information acquisition unit 12 periodically acquires at least one of the following: coolant temperature, engine speed, vehicle speed, oil temperature, oil pressure, and diagnostic trouble code (DTC). The acquisition period can be, for example, 0.5 to 2 seconds. The edge computing device 15 is composed of a single-board computer and an edge AI accelerator. As a specific example, a general-purpose operating system may run on the single-board computer, and a neural network model may be inferred at high speed on the edge AI accelerator. The edge computing device 15 can complete inference processing independently without requiring a constant connection to the Internet or an external cloud server. The display unit 16 may be a small display or a display screen wirelessly connected to the driver's mobile terminal. The audio output unit 17 is composed of a speaker built into the vehicle or an external speaker. The power supply unit 18 may include a cigarette lighter plug for receiving power from an on-board power supply and a power input unit for receiving power from a mobile battery or the like. The first camera 13A captures images of the driving environment, including the road ahead and the brake lights of the preceding vehicle, and is used to determine congestion and low-speed driving conditions. The second camera 13B captures images of the driver's face and upper body, and is used to estimate the driver's state, such as blinking frequency, line of sight, and posture. These multiple cameras make it possible to collectively evaluate not only the vehicle state but also the driving environment and driver state. The following software modules run on the edge calculation device 15. OBD data acquisition module 20: Controls the vehicle information acquisition unit 12 to acquire various sensor values ​​and converts them into physical quantities. Image acquisition module 21: Acquires images from first camera 13A and second camera 13B, and extracts the area of ​​the road ahead, the vehicle ahead, the meter panel area, and the driver's face area as needed. Voice acquisition module 22: Acquires voice waveforms from the microphone 14 and sends them to the voice recognition module. Feature generation module 23: Integrates time-varying OBD data, feature vectors such as congestion levels and stop-start patterns extracted from images of the road ahead, alertness indicators extracted from driver images, and voice recognition results as multimodal features. Vehicle field-specific language model 24: A small language model specialized for vehicle state explanation and driving advice, which generates a natural language message using the multimodal features and the driver profile 25 as input. Speech synthesis module 26: Synthesizes the generated text into Japanese speech and outputs it from the speech output unit 17. Logging module 27: Encrypts driving history and speech history and stores them in local storage. The driver profile 25 records information such as the vehicle type, engine specifications, normal water and oil temperature ranges, past trouble cases, and the driver's preferred explanation style (brief / detailed, etc.). This makes it possible to generate different comments depending on the vehicle and driver, even if the same numerical values ​​are used. As shown in FIG. 2, the driving assistance method of the present invention mainly comprises the following steps: 1) OBD Data Acquisition Step S1: The OBD data acquisition module 20 acquires various sensor values ​​from the vehicle diagnostic port 11. 2) Image acquisition step S2: The image acquisition module 21 acquires frames from the first camera 13A and the second camera 13B, and analyzes the traffic situation on the road ahead and the state of the driver. 3) Speech acquisition step S3: The speech acquisition module 22 acquires the driver's speech and converts it into text through speech recognition. 4) Feature integration step S4: The feature generation module 23 generates multimodal features by integrating the temporal changes in the OBD data, the congestion level and the number of stops and starts extracted from the image of the road ahead, the alertness index extracted from the image of the driver, the voice recognition results, and the driver profile 25. 5) Language model inference step S5: The vehicle domain-specific language model 24 receives the multimodal features as input and generates a driving assistance message including a description of the vehicle state, a risk level, and a recommended action. 6) Output step S6: The generated message is displayed on the display unit 16, and is also read aloud from the voice output unit 17 by the voice synthesis module 26. For example, when the vehicle is traveling slowly on a congested road, the image obtained from the first camera 13A may show that the brake lights of the vehicle ahead are frequently illuminated, and the OBD data may show that the vehicle is repeatedly slowing down and stopping in short cycles. In such a situation, if the coolant temperature rises sharply immediately after the start of driving and remains higher than the normal range for the vehicle recorded in the driver profile 25, the vehicle domain-specific language model 24 may generate an explanation to inform the driver, such as, "You are currently traveling slowly on a congested road, and the coolant temperature is higher than normal. Avoid idling for long periods of time, and if possible, take an early break after passing through the congested section. If the temperature does not decrease, have the vehicle checked at a repair shop." If the risk level reaches a predetermined high level, the audio output unit 17 outputs a warning sound or a message of high urgency, and if necessary, issues an instruction such as "Please pull over to the side of the road immediately and turn off the engine." Based on the driver profile 25, it is also possible to switch to a mode that issues an early warning or a mode that provides a simple explanation. While the above embodiment illustrates an internal combustion engine vehicle, the present invention can also be applied to monitoring battery temperature and inverter temperature in hybrid vehicles and electric vehicles. In this case, the parameters acquired as OBD data can be replaced with motor temperature, remaining battery capacity, charge / discharge current, etc. Furthermore, the system may be configured to omit capturing images of the road ahead using the first camera 13A and generate a driving assistance message based only on the time variation of the OBD data and the driver's voice, or may be configured to specialize in estimating the driver's state using the second camera 13B. Conversely, the system may be configured to detect dangerous driving and near-miss events based only on images of the road ahead and the driver, without using OBD data. These modified examples separate the components of the present invention by function, and will serve as the technical foundation for filing divisional applications in the future as a driving environment recognition device, a driver state estimation device, a vehicle state diagnosis device, a field-specific language model device, etc. Furthermore, a conversation function can be added in which the device of the present invention responds to questions such as "What's the water temperature now?" or "Is this traffic jam putting a strain on the engine?" in response to voice input from the driver, by referring to the latest OBD data and the degree of traffic congestion on the road ahead and responding in natural language. Such interactive driving assistance functions are also encompassed in this specification. The company names, product names, and software names mentioned in this specification are generally registered trademarks or trademarks of the respective companies. [Example]

[0010] One specific example of the present invention is a configuration using a Raspberry Pi 5 as a single-board computer and a Hailo-8L as an edge AI accelerator. Furthermore, a vehicle-specific language model can be a model based on an open-source small language model, distilled and fine-tuned using a training dataset consisting of vehicle engineering, maintenance manuals, actual vehicle test logs, etc. However, the present invention is not limited to these specific examples, and other hardware and software with equivalent functions may also be used. [Industrial Applicability]

[0011] The present invention's fully local vehicle condition monitoring AI device and driving assistance method can be widely used for operational management of not only privately owned passenger cars and sports cars, but also company cars, transport vehicles, car-sharing vehicles, rental cars, etc., as well as for monitoring vehicle conditions while driving on a circuit. Because it does not rely on the cloud, it is easy to introduce in areas with poor communication environments or for applications with strict privacy requirements, making it useful in the automotive industry in general. It can also be applied to preventive maintenance and safe driving education in fields such as transportation, bus operators, construction machinery, and agricultural machinery. [Explanation of symbols]

[0012] 10 AI vehicle condition monitoring device 11 Vehicle Diagnostic Port 12 Vehicle Information Acquisition Department 13A First Camera (forward road imaging) 13B Second camera (driver imaging) 14 microphones 15 Edge Computing Device 15A single-board computer (e.g. Raspberry Pi 5) 15B Edge AI Accelerator (e.g. Hailo-8L) 16 displays 17 Audio output section 18 power supply section 19 cabinets 20 to 27 various software modules

Claims

1. A portable vehicle condition monitoring AI device connected to a vehicle having a vehicle diagnostic port, comprising: (A) a vehicle information acquisition means electrically connected to the vehicle diagnostic port for periodically acquiring at least one of a coolant temperature, an oil temperature, an oil pressure, an engine speed, a vehicle speed, and a diagnostic trouble code; (B) a first camera means for capturing an image of a road ahead; (C) a second camera means for capturing an image of the driver; (D) a microphone for picking up the driver's voice; (E) a feature generating means for generating multimodal features based on data from the vehicle information acquiring means, image data from the first camera means and the second camera means, and audio data from the microphone; (F) a generative AI inference means including a vehicle field-specific language model that receives the multimodal features and driver profile information as inputs and generates a driving assistance message in natural language indicating the vehicle state, risk level, and recommended actions; (G) a display means for displaying the driving assistance message and a voice synthesis means for outputting the driving assistance message as a synthesized voice; (H) a control means for executing the vehicle information acquisition means, the feature generation means, the generation AI inference means, and the voice synthesis means on a single edge device without communicating with a cloud server; (I) a housing having a connector to be connected to the vehicle diagnostic port and a power supply means capable of supplying power from an in-vehicle power supply or a mobile battery; A vehicle condition monitoring AI device comprising:

2. 2. The vehicle state monitoring AI device according to claim 1, The vehicle diagnostic port is OBD compliant; A vehicle condition monitoring AI device characterized in that the generation AI inference means evaluates the risk of overheating based on changes in the coolant temperature and oil temperature over time, and generates a driving assistance message including the evaluation results.

3. 2. The vehicle state monitoring AI device according to claim 1, the feature generating means calculates a congestion index based on a forward road image obtained from the first camera means and vehicle speed data obtained from the vehicle information acquiring means; The vehicle condition monitoring AI device is characterized in that the generation AI inference means generates a driving assistance message including recommended actions for cooling the engine during traffic congestion based on the congestion degree index and the change in the coolant temperature over time.

4. 2. The vehicle state monitoring AI device according to claim 1, The vehicle field-specific language model is obtained by distilling and fine-tuning a small language model based on training data including text related to vehicle engineering and maintenance manuals and responses obtained from a large-scale language model, and is quantized so that inference can be performed on an edge device.

5. 2. The vehicle state monitoring AI device according to claim 1, Further comprising a profile storage means for storing driver profile information; A vehicle condition monitoring AI device characterized in that the generation AI inference means updates the driver profile information based on past driving history and the driver's speech history, and adjusts the expression content of the driving assistance message or the warning threshold according to the updated driver profile information.

6. 2. The vehicle state monitoring AI device according to claim 1, A vehicle condition monitoring AI device characterized in that, when the risk level reaches a predetermined high level, the generation AI inference means instructs the voice synthesis means to output an emergency warning message and causes the display means to display a message including a message to stop driving and stop the vehicle in a safe place.

7. A driving assistance method using the vehicle state monitoring AI device according to claim 1, obtaining sensor values ​​and diagnostic trouble codes from a vehicle diagnostic port; acquiring an image of the road ahead with a first camera; capturing an image of the driver with a second camera; acquiring a voice of a driver by a microphone; generating multimodal features based on the acquired data; inputting the multimodal features and driver profile information into the vehicle domain-specific language model to generate a driving assistance message in natural language; a step of displaying and audibly outputting the generated driving assistance message; wherein at least the generation of multimodal features and the inference using the language model in the steps are completed within the vehicle state monitoring AI device without going through a cloud server.

Images