AI-powered vehicle driver assistance device and driver assistance method for a fully localized communication-restricted environment.

Abstract

In lunar vehicles, we provide a fully localized driver assistance system that can immediately present occupants with driving environment risks and recommended actions, along with explanations, even in environments where communication delays or interruptions may occur. [Solution] The system includes a first camera for capturing images of the driving environment ahead, a second camera for capturing images of the passenger, a microphone for acquiring passenger voices, a feature generation means for generating multimodal features based on images and voices, a generation AI inference means for generating driving support messages in natural language indicating risk levels and recommended actions using the feature and passenger profile information as input, a display means, a speech synthesis means, and a control means that executes these on a single edge device without communicating with a cloud server.

Description

Technical Field

[0001] The present invention relates to Used in areas with poor communication infrastructure or in environments where communication delays or interruptions may occur. vehicles vehicle condition monitoring and safe driving assistance, and more specifically, relates to a vehicle driving assistance AI apparatus and driving assistance method that integrates information obtained from a camera that images the forward driving environment of a vehicle, a camera that images the driver, and a microphone that acquires the driver's voice, and generates driving assistance messages in natural language on a single edge device without using a cloud For environments with complete local communication restrictions. .

Background Art

[0002] Conventionally, in vehicle condition monitoring and driving assistance, technologies for issuing alerts by combining in-vehicle sensors, driving environment images, driver images and voice have been proposed. Further, services have also been proposed in which computing resources or large language models on a cloud are linked with in-vehicle equipment, and information obtained from a vehicle is transmitted to an external server to generate explanations and advice. However, In configurations that rely on external communication, the communication environment may be poor or there may be communication delays or communication issues. In environments where disconnections may occur, support may be delayed. Also, driving environment, driver image, and sound. Privacy concerns arising from the transmission of voice or location / driving history information to external parties. There are also others.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0004]

Non-Patent Document 1

[0005] The objective of the present invention is, Use in areas with poor communication infrastructure or in environments where communication delays or interruptions may occur. Vehicles that can be there In this system, forward driving environment images, driver images, and driver voice are integrated, and explanations, risk levels, and recommended actions for the driver are generated and presented instantly in natural language such as Japanese, without relying on the cloud. AI-powered vehicle driver assistance system for environments with complete local communication restrictions. Installation and driving assistance methods The objective is to provide. [Means for solving the problem]

[0006] According to one aspect of the present invention AI-powered vehicle driver assistance system for environments with complete local communication restrictions. teeth, Communication environment Vehicles used in areas with limited communication resources or in environments where communication delays or interruptions may occur.The system comprises a housing equipped with a first camera means for capturing images of the forward driving environment, a second camera means for capturing images of the driver of the vehicle, a microphone for capturing the driver's voice, a feature generation means for generating multimodal features based on image data obtained from the first and second camera means and voice data obtained from the microphone, a generation AI inference means including a vehicle domain-specific language model that generates driving assistance messages in natural language indicating descriptions of the vehicle's driving environment or the driver's condition, risk levels, and recommended actions based on the multimodal features and driver profile information, a display means, a speech synthesis means, a control means that executes each of the means on a single edge device without communicating with a cloud server, and a power supply means. In a preferred embodiment, the generating AI inference means integrates features based on the image of the forward driving environment and features based on the image of the driver, and reflects the integration result in the driving assistance message. In a preferred embodiment, the device further comprises speech recognition means for converting the driver's speech into text, and the generating AI inference means adjusts the content of the driver assistance message according to the transcribed speech content. In a preferred embodiment, the driver profile information includes the driver's preferred descriptive style, and the generating AI inference means adjusts the descriptive expression according to the driver profile information. [Effects of the Invention]

[0007] According to the present invention, even in environments where communication delays or communication interruptions may occur, Inside the vehicle or connected to the vehicle On a single edge device, the generation and presentation of driver assistance messages can be completed. Therefore, driver assistance can continue without relying on external cloud servers. Furthermore, because it can present not only the risk level but also an explanation in natural language, drivers can easily understand the purpose of the warning and make decisions such as slowing down, checking for stopping, changing routes, taking a break, or inspecting the vehicle. Furthermore, it can process driving environment images, driver images, and audio information without sending them to an external server. Therefore, privacy concerns can be reduced. [Brief explanation of the drawing]

[0008] [Figure 1]It is a block diagram showing a configuration example of a vehicle driving support AI device for a complete local communication restricted environment according to the present invention. [Figure 2] It is a diagram showing a processing flow of a driving support method for a complete local communication restricted environment according to the present invention. [Figure 3] It is a diagram showing a conceptual example of explanation generation in the present invention. MODE FOR CARRYING OUT THE INVENTION

[0009] As shown in Figure 1, the apparatus 10 of the present invention comprises a first camera means 11, a second camera means 12, and ICROPHONE 13, Feature generation means 14, Profile storage means 15, Generative AI inference means 1 6. The device comprises a display means 17, a speech synthesis means 18, a control means 19, and a power supply means 20. Option 10 is used in areas with poor communication infrastructure or in environments where communication delays or interruptions may occur. It is mounted on or connected to the vehicle.

[0010] The first camera means 11 captures images of the driving environment in front of the vehicle and takes image data that shows the situation ahead. To benefit. The second camera means 12 displays image data showing the driver's face, gaze, posture, or movement state. The microphone 13 acquires the driver's speech or response voice. The feature generation means 14 includes a first camera means 11, a second camera means 12, and a microphone 1 Based on the input data obtained from 3, the forward driving environment, driver status, and voice content are represented. It generates multimodal features. The profile storage means 15 stores the driver's preferred explanatory style. Driver profile information, including this information, is stored locally. The generative AI inference means 16 uses the multimodal features and the driver profile information to generate AI inference. This includes explanations regarding the vehicle's driving environment or driver conditions, risk levels, and recommended actions. The system generates support messages in natural language. The generating AI inference means 16 uses a language model specific to the vehicle sector. Including Dell, quantized or lightweighted to a size that allows for real-time inference on edge devices. It's fine if you do that. Display means 17 displays the risk level, recommended actions, and explanations on the same screen. Speech synthesis Means 18 output all or a summary of the driver assistance message as audio. Control means 19: The feature generation means 14, the generation AI inference means 16, and the speech synthesis means 18 are located on a cloud server. It runs on a single edge device without communicating with the other devices. Power supply means 20 is connected to device 10 It supplies power to it. As shown in Figure 2, in the method of the present invention, first an image of the forward driving environment is acquired (S1), and then The driver's image is acquired (S2), and then the driver's voice is acquired (S3). Next, multi-modal Features are generated (S4), and local AI inference is used to explain, determine the risk level, and recommend actions. Generate a driver assistance message including (S5), and display and output the result as audio (S6) )。 As shown in Figure 3, inputs related to the forward driving environment, inputs related to the driver's condition, or the driver's When utterances are input, the generating AI inference means 16 evaluates and explains the relationships between those inputs. The text generates the message: "Pay attention to the situation ahead. Slow down and prioritize safety checks." It can output messages such as "Please do so." INDUSTRIAL APPLICABILITY

[0011] This invention does not rely on constant communication with a cloud server and is installed in or on a vehicle. Using only connected local computing resources, it monitors the driving environment and driver status, and uses natural language... Because it can provide driving assistance, it is suitable for areas with poor communication environments or where there is communication delay or interruption. Even if such a situation may occur, driver assistance can continue to be provided to the driver. Therefore, the present invention provides support for safe driving of vehicles, monitoring of the driver's condition, and attention according to the driving environment. It can be widely used industrially as a driver assistance technology that provides alerts and uses natural language. DESCRIPTION OF REFERENCE NUMERALS

[0012] 10 device 11 First camera means 12 Second camera means 13 Microphone 14 Feature generation means 15. Profile storage means 16 Generative AI inference means 17 Display means 18. Speech synthesis means 19 Control means 20 Power means

Claims

1. A vehicle driving support AI device for a fully local communication restricted environment that provides driving support to the driver of a vehicle used in an area with poor communication environment or in an environment where communication delays or communication interruptions may occur, A first camera means for imaging the forward driving environment of the vehicle, A second camera means for capturing images of the driver of the vehicle, A microphone for capturing the driver's voice, A feature generation means generates multimodal feature quantities representing the forward driving environment, the driver's state, and the audio content based on image data obtained from the first camera means and the second camera means, and audio data obtained from the microphone. A generative AI inference means including a vehicle-specific language model that takes the aforementioned multimodal features and driver profile information as input to generate driver assistance messages in natural language that describe the vehicle's driving environment or the driver's condition, the risk level, and recommended actions, A display means for displaying the aforementioned driver assistance message and a speech synthesis means for outputting the aforementioned driver assistance message as synthesized speech, Control means for executing the feature generation means, the generation AI inference means, and the speech synthesis means on a single edge device without communicating with a cloud server, A housing equipped with a power supply, A vehicle driving assistance AI device for a completely local communication restriction environment, characterized by comprising the following features.

2. The generating AI inference means integrates feature quantities based on the image of the forward driving environment and feature quantities based on the image of the driver, and reflects the integration result in the driving assistance message, as described in claim 1, for a vehicle driving assistance AI device for a fully local communication restricted environment.

3. The vehicle driving support AI device for a fully local communication restricted environment according to claim 1 or 2, further comprising speech recognition means for converting the driver's speech into text, and the generating AI inference means adjusting the content of the driving support message according to the transcribed speech content.

4. The vehicle driving support AI device for a fully local communication restricted environment according to claim 1, wherein the driver profile information includes the driver's preferred descriptive style, and the generating AI inference means adjusts the descriptive expression according to the driver profile information.

5. The display means displays the risk level, recommended actions, and explanations on the same screen, and the speech synthesis means outputs a driving support message including the explanations as voice, according to claim 1, for use as a vehicle driving support AI device for a fully local communication restricted environment.

6. A driving assistance method for a fully local communication restriction environment that provides driving assistance to a driver of a vehicle used in an area with poor communication environment or in an environment where communication delays or communication interruptions may occur, The process of acquiring an image of the forward driving environment of the vehicle, The process of acquiring the driver's image and voice, A step of generating multimodal features representing the forward driving environment, the driver's state, and the audio content based on the forward driving environment image, the driver's image, and the audio; The process involves using the aforementioned multimodal features and driver profile information as input to generate a driver assistance message in natural language that describes the vehicle's driving environment or the driver's condition, the risk level, and recommended actions. The process of displaying and outputting the generated driver assistance message as sound, Includes, A driver support method for a completely local communication-restricted environment, characterized in that each of the aforementioned steps is performed on a single edge device without communicating with a cloud server.

Images