AI device and driver assistance system for fully localized lunar rover driving environment and passenger monitoring.

Abstract

In lunar vehicles, we provide a fully localized driver assistance system that can immediately present occupants with driving environment risks and recommended actions, along with explanations, even in environments where communication delays or interruptions may occur. [Solution] The system includes a first camera for capturing images of the driving environment ahead, a second camera for capturing images of the passenger, a microphone for acquiring passenger voices, a feature generation means for generating multimodal features based on images and voices, a generation AI inference means for generating driving support messages in natural language indicating risk levels and recommended actions using the feature and passenger profile information as input, a display means, a speech synthesis means, and a control means that executes these on a single edge device without communicating with a cloud server.

Description

Technical Field

[0001] The present invention relates to state monitoring and driving support for passengers in a lunar vehicle. More specifically, it integrates information obtained from a camera that images the forward driving environment of the lunar vehicle, a camera that images the passengers, and a microphone that acquires passenger voices, and uses local AI without using the cloud to generate risk assessments and recommended actions in natural language. It relates to a completely local type of driving environment and passenger monitoring AI device and a driving support method.

Background Art

[0002] Conventionally, in vehicle state monitoring and driving support, technologies have been proposed that combine in-vehicle sensors, driving environment images, and driver images to issue warnings. However, many configurations assume ground vehicles, and there are problems such as dependence on the communication environment and simply stopping at warning displays.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

[0005] The object of the present invention is to provide a fully local AI device and driving support method for lunar rover vehicles that integrates forward driving environment images, occupant images, and occupant voices, and generates and presents risk levels, recommended actions, and the reasons for them in natural language such as Japanese, without relying on the cloud. [Means for solving the problem]

[0006] An AI device for monitoring the driving environment and occupants of a fully local lunar rover according to one aspect of the present invention comprises a housing equipped with a first camera means for imaging the driving environment in front of the lunar rover, a second camera means for imaging the occupants, a microphone for capturing the occupants' voices, a feature generation means for generating multimodal features based on image data and voice data, a generation AI inference means for generating driving support messages in natural language indicating risk levels and recommended actions based on the feature data and occupant profile information, a display means, a speech synthesis means, a control means for executing each of the means on a single edge device without communicating with a cloud server, and a power supply means. In a preferred embodiment, the generating AI inference means estimates potential obstacles, reduced visibility, uneven terrain, or signs of difficulty in driving based on the driving environment image. In a preferred embodiment, the device further comprises speech recognition means for converting the passenger's speech into text, and adjusting explanatory or warning expressions according to the content of the speech. In a preferred embodiment, the passenger profile information includes the passenger's explanation style, past alert history, or assistance history. [Effects of the Invention]

[0007] According to the present invention, even in environments where communication delays or interruptions may occur, risk assessment and recommended actions can be completed on a single edge device within the lunar rover, making it easier to ensure operational continuity. Furthermore, since not only the risk level but also the reasons can be explained in natural language, occupants can easily understand the intent of the warning and make decisions such as reducing speed, confirming a stop, or changing the route. [Brief explanation of the drawing]

[0008] [Figure 1] This block diagram shows an example configuration of an AI system for monitoring the driving environment and occupants of a fully localized lunar rover. [Figure 2] This diagram shows the processing flow of a fully localized driving assistance method. [Figure 3] This figure shows a conceptual example of explanation generation. [Modes for carrying out the invention]

[0009] As shown in Figure 1, the apparatus 10 of the present invention comprises a first camera means 11 for imaging the forward driving environment of the lunar vehicle, a second camera means 12 for imaging the occupant, a microphone 13 for acquiring occupant voice, a feature generation means 14, an occupant profile storage means 15, a generation AI inference means 16, a display means 17, a speech synthesis means 18, a control means 19, and a power supply means 20. [Examples]

[0010] The first camera means 11 acquires the terrain ahead, potential obstacles, or signs of reduced visibility. The second camera means 12 acquires the passenger's gaze, posture, or movement. The microphone 13 acquires the passenger's speech or response voice. The feature generation means 14 generates multimodal features from each of the input data. The generation AI inference means 16 generates a driver assistance message that includes the risk level in the lunar vehicle's driving environment, recommended actions, and the reasons for those actions, based on these features and the passenger profile information. Display means 17 displays the risk level, recommended actions, and reasons on the same screen. Speech synthesis means 18 outputs the same content or a summary of the content as speech. Control means 19 controls each of the above means on a single edge device without communicating with a cloud server. As shown in Figure 2, the method of the present invention first acquires an image of the driving environment (S1), then an image of the passenger (S2), and then the passenger's voice (S3). Subsequently, multimodal features are generated (S4), risk levels, recommended actions, and reasons are generated by local AI inference (S5), and the results are displayed and output as audio (S6). As shown in Figure 3, when inputs indicate reduced forward visibility, the presence of potential obstacles, or passenger speech, the generating AI inference means 16 evaluates the relationships between these inputs to generate an explanatory text and can output a message such as, "Forward visibility is reduced. Slow down and prioritize safety checks." [Industrial applicability]

[0011] This invention can monitor the driving environment and occupant status and provide natural language driving or behavioral support using only local computing resources mounted on a mobile vehicle, without relying on constant communication with a cloud server or ground-based computing resources. Therefore, it can be applied to vehicles operating in special environments, non-road vehicles, mobile vehicles in extreme environments, and similar exploration vehicles. Specific examples include a manned lunar rover, an unmanned lunar exploration vehicle, a planetary exploration rover, a local support vehicle used in a remote environment where communication delays or outages are likely to occur, and a special vehicle that is operated with people on board in a closed or dangerous environment or while receiving remote support. In addition, the present invention can integrate and process image information of the driving environment, image information of passengers, and voice information by a local AI, and generate recommended actions, warning messages, or explanatory messages in natural language. Therefore, even in an environment where communication cannot be relied upon, it contributes to improving the safety of passengers, the speed of operation judgment, and the continuity of operation. Therefore, the present invention is not limited to the field of lunar exploration, and in the future, it can be widely used industrially as a mobile body support technology used in planetary exploration, polar exploration, disaster response, underground work, closed space work, and other special environments.

Explanation of Signs

[0012] 10 Device 11 First Camera Means 12 Second Camera Means 13 Microphone 14 Feature Quantity Generation Means 15 Passenger Profile Storage Means 16 Generated AI Inference Means 17 Display Means 18 Voice Synthesis Means 19 Control Means 20 Power Supply Means

Claims

1. A fully localized lunar rover driving environment and occupant monitoring AI device that provides driving assistance to the occupants of a lunar rover, A first camera means for imaging the forward driving environment of the lunar vehicle, A second camera means for imaging the occupants of the lunar vehicle, A microphone for capturing the voice of the aforementioned passenger, A feature generation means that generates multimodal features based on image data obtained from the first camera means and the second camera means, and audio data obtained from the microphone, A generative AI inference means including a vehicle-specific language model that takes the aforementioned multimodal features and passenger profile information as input to generate driver assistance messages in natural language indicating the risk level and recommended actions in the driving environment of the lunar vehicle, A display means for displaying the aforementioned driver assistance message and a speech synthesis means for outputting the aforementioned driver assistance message as synthesized speech, Control means for executing the feature generation means, the generation AI inference means, and the speech synthesis means on a single edge device without communicating with a cloud server, A housing equipped with a power supply means that can be powered from a power supply mounted on the lunar vehicle or a portable power supply, A fully localized lunar rover driving environment and occupant monitoring AI system characterized by comprising the following features.

2. The generating AI inference means estimates at least one of an obstacle candidate, reduced visibility, uneven terrain, or signs of difficulty in driving based on the driving environment image, and reflects the estimation result in the driving support message, as described in claim 1, for a fully localized lunar vehicle driving environment and passenger monitoring AI device.

3. The device further comprises speech recognition means for converting the utterances of the passengers into text, and the generating AI inference means adjusts the content of the driving support message according to the content of the transcribed utterances, as described in claim 1 or 2.

4. The passenger profile information includes the passenger's explanation style, past driving assistance history, or warning history, and the generating AI inference means adjusts the warning threshold or explanation expression according to the passenger profile information, as described in claim 1, for a fully localized lunar vehicle driving environment and passenger monitoring AI device.

5. The display means displays the risk level, recommended action and the reason on the same screen, and the voice synthesis means outputs a driving support message including the reason, according to claim 1, for a fully localized lunar vehicle driving environment and passenger monitoring AI device.

6. A fully localized driving assistance method that provides driving assistance to the occupants of a lunar vehicle, The process of acquiring images of the driving environment ahead, The process of acquiring passenger images and passenger audio, A step of generating multimodal features based on the forward driving environment image, the passenger image, and the passenger's voice, The process involves using the aforementioned multimodal features and occupant profile information as input to generate driver assistance messages in natural language that indicate the risk level and recommended actions in the driving environment of the lunar vehicle, The process of displaying and outputting the generated driver assistance message as sound, Includes, A fully local driving support method characterized in that each of the aforementioned steps is performed on a single edge device without communicating with a cloud server.

Images