Vehicle-mounted display method and system and vehicle

Abstract

The invention provides a vehicle-mounted display method and system and a vehicle. AI intelligent layout interface elements are driven through multi-mode message data and are rendered and displayed. The multi-modal data enables the displayed information to be more comprehensive, the display is more flexible and diverse, and various information including the vehicle running state, the navigation information, the road condition prompt and the like can be provided for a driver.

Description

Technical Field

[0001] The present application relates to the field of display technology, and in particular to a vehicle-mounted display method and system, and a vehicle. Background Art

[0002] A heads-up display (HUD) projects key information directly onto a transparent display within the driver's field of view, allowing drivers to access information without having to look down at the instrument panel. HUDs allow drivers to focus more closely on the road and traffic, improving driving safety, a crucial feature particularly at high speeds or in complex road conditions. With technological advancements, HUDs are evolving from simple information displays to advanced augmented reality (AR) integration, enhancing the driving experience.

[0003] Existing HUDs are based on preset trigger conditions, loading corresponding images and other resources and rendering to generate the UI interface. With the rapid development of AI technology, generative AI is widely used in numerous industries, providing an effective and feasible method for HUD interface generation. It can continuously learn and optimize, adapt to the environment, and provide a better method and experience for augmented reality. However, current AI-generated image technology has limitations and significant room for improvement. Especially in the automotive field, there is still a lack of technical means to generate images from vehicle information through AI technology.

[0004] Therefore, the existing HUD technology still has some problems that need to be solved:

[0005] 1. Poor multimodal data fusion: Existing HUD display technology primarily triggers image rendering based on CAN / LIN or Ethernet signals. Typically, received messages are received separately from a single source, such as a CAN frame, a LIN frame, and an Ethernet frame. This multimodal data received from multiple data sources cannot guarantee real-time and synchronization. The individual source messages are then parsed separately, resulting in a more fragmented message analysis. For example, speed is parsed and rendered for display, or a mini-map or zoomed-in intersection view is parsed and rendered for display. In a dynamically changing driving environment, insufficient multimodal data fusion leads to insufficient real-time and synchronization, which can cause information mismatch and affect display accuracy.

[0006] 2. The layout of traditional HUD is fixed and cannot be optimized according to driver habits or environment. The fixed layout of traditional HUD can easily lead to information stacking in complex road conditions. For example, navigation routes, ADAS warnings, and entertainment information appear at the same time, causing visual confusion. Static UI layout cannot adapt to the changing patterns of visual focus of different drivers. For example, novice drivers pay more attention to vehicle speed, while experienced drivers pay more attention to navigation previews. In addition, commercial-oriented UI design may induce attention deviation, such as over-magnifying promotional advertisements and weakening safety prompts. Therefore, the UI module of automotive HUD requires more intelligent spatial layout logic. Existing HUD technology focuses more on information content generation and display accuracy optimization, and has not yet clearly applied AI algorithms to the spatial layout logic of UI modules (such as information density adaptation, driver attention hot zone matching, etc.). Therefore, AI-based intelligent layout of HUD interface has room for further exploration in the direction of differentiated innovation.

[0007] 3. AR fusion is the process of fitting virtual images into real scenes. The UI layout of virtual images in real scenes is currently pre-calibrated. Existing HUDs lack effective methods for high-quality augmented reality (AR) fusion with real-scene images. The fusion of virtual information and the real world is not natural enough, affecting the driving experience. Summary of the Invention

[0008] This application proposes an in-vehicle display method and system, as well as a vehicle, that uses multimodal message data to drive AI to intelligently layout interface elements and render the display. Multimodal data enables more comprehensive and flexible display of information, including vehicle operating status, navigation information, road condition prompts, and other information to the driver.

[0009] On the one hand, the present application provides an in-vehicle display method, which includes: collecting and preprocessing multimodal data; stylized definition of multi-source data and customized layout of UI graphics; and realization of AR effects.

[0010] Optionally, the collecting and preprocessing of multimodal data includes: acquiring CAN / LIN bus and Ethernet signals; collecting image data; and preprocessing data.

[0011] Optionally, the CAN / LIN bus data includes but is not limited to speed, turning angle, vehicle body position, environmental perception, and vehicle status.

[0012] Optionally, the Ethernet signal includes but is not limited to navigation data, camera image data, LiDAR point cloud data, and data from an in-vehicle infotainment system.

[0013] Optionally, the collected image data is obtained by acquiring image data of the vehicle's surrounding environment through a camera; For the collection Image data identification timestamp, This timestamp is matched to the CAN signal.

[0014] Optionally, the preprocessed data includes but is not limited to: denoising and correcting images; standardizing CAN bus data and Ethernet signal data, and cleaning CAN bus data and Ethernet signal data; decoding, structuring, and data timing alignment of multimodal message data from multiple data sources.

[0015] Optionally, the pre-processed data includes but is not limited to environmental perception information, driving status dynamic data, and driver dynamic information.

[0016] Optionally, the environmental perception information includes but is not limited to ambient light intensity and temperature.

[0017] Optionally, the driving status dynamic data includes but is not limited to driving status.

[0018] Optionally, the driver's dynamic information includes but is not limited to eye position, sitting posture, driving status, and identity information.

[0019] Optionally, when performing time alignment on the multimodal message data from multiple data sources, multiple frames of data are first cached for each source message data, and the caching form of the message data includes but is not limited to a queue and a tree.

[0020] Optionally, when performing time alignment on the multimodal message data from multiple data sources, the message data from each source in the cache is matched, and the message data with the same timestamp are formed into a complete frame of multimodal data.

[0021] Optionally, the stylized definition of multi-source data and the customized layout of UI graphics include: AI processing of multimodal message data from multiple data sources and stylized mapping, intelligent random layout of UI modules and arrangement optimization.

[0022] Optionally, by performing feature extraction, graphic mapping, AI training, and semantic verification on multimodal data, the stylized definition of multi-source data and the customized layout of UI graphics can be completed.

[0023] Optionally, the AI ​​processing of multimodal message data and stylized mapping from multiple data sources includes but is not limited to: data preprocessing and multimodal fusion message parsing, time alignment, and multimodal feature fusion. Optionally, the data preprocessing and multimodal fusion message parsing includes but is not limited to: decoding and structuring CAN messages, LIN messages, and sensor data; wherein the sensor data includes but is not limited to vehicle speed, temperature, and fault codes.

[0024] Optionally, the timing alignment is specifically to synchronize sensor data of different frequencies through timestamps.

[0025] Optionally, the multimodal feature fusion is specifically: integrating numerical data, textual fault descriptions, and time series patterns into a unified feature vector.

[0026] Optionally, the intelligent random layout UI module arrangement optimization is specifically: AI-driven layout generation, layout optimization based on genetic algorithm, modeling of component relationships through graph neural network, user interaction and dynamic adjustment.

[0027] Optionally, the in-vehicle display process is implemented end-to-end.

[0028] Optionally, the end-to-end model is continuously iterated and optimized.

[0029] Optionally, the continuous iteration and optimization of the end-to-end model specifically includes: continuously collecting usage data and outputting feedback to users.

[0030] Optionally, the feedback output to the user is realized as personalized recommendation and drag optimization; the personalized recommendation is used to record the user's historical selection preferences and recommend high-frequency display module combinations through collaborative filtering; the drag optimization is used to calculate the optimal position of the remaining components in real time after the user manually adjusts it.

[0031] Optionally, the AR effect realization includes: constructing a three-dimensional scene based on multimodal data, locating dynamic targets, and generating a virtual and real HUD image through neural rendering fusion, wherein the multimodal data includes images and point clouds.

[0032] Optionally, the display priority and fusion strategy of the virtual content are adapted according to the user's focus and the characteristics of the scene.

[0033] Optionally, real-time rendering via a graphics processing unit.

[0034] Optionally, the rendered image is adjusted in real time based on the environmental perception information.

[0035] Optionally, an image is generated based on driving status dynamic data.

[0036] In one aspect, the present application provides a vehicle-mounted display system, comprising:

[0037] a vehicle bus for transmitting multimodal signals;

[0038] The controller receives the multimodal signal transmitted by the bus, pre-processes the multimodal data, generates a virtual image based on the model, and renders the virtual image to generate a HUD image that combines virtual and real images; the optical machine receives the HUD image data and displays it.

[0039] Optionally, the vehicle-mounted display system uses the aforementioned vehicle-mounted display method for display.

[0040] On the other hand, the present application provides a vehicle, which is equipped with the aforementioned in-vehicle display system and executes the aforementioned in-vehicle display method.

[0041] Beneficial effects of this application:

[0042] This application proposes a method that systematically addresses the long-standing conflict between "information effectiveness" and "visual friendliness" in the HUD field through dynamic spatial topology optimization and resource adaptation, driving the evolution of human-computer interaction from "functional presentation" to "cognitive symbiosis." Compared with existing technologies, this invention significantly improves real-time performance, diversity, interactivity, and driving experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] To more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0044] In order to more completely understand the present application and its beneficial effects, the following description will be given in conjunction with the accompanying drawings, wherein the same drawing numbers represent the same parts in the following description.

[0045] Figure 1 This is a schematic diagram of the application scenario of in-vehicle AR-HUD;

[0046] Figure 2 Data processing and rendering flow chart for the in-vehicle AR-HUD system;

[0047] Figure 3 Schematic diagram of current in-vehicle AR-HUD hardware. DETAILED DESCRIPTION

[0048] The technical solution in this application will be described below with reference to the accompanying drawings.

[0049] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0050] In the embodiments of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0051] It should be understood that the terms used in the description of the various examples herein are for the purpose of describing the particular examples only and are not intended to be limiting. As used in the description of the various examples, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0052] In this application, "at least one" means one, two, or more, and "more than one" means more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0053] It should also be understood that in this application, unless otherwise clearly specified and limited, the term "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a sliding connection, a detachable connection, or an integral connection, etc.; it can be a direct connection or an indirect connection through an intermediate medium.

[0054] It will also be understood that the term “comprise” (also known as “includes,” “including,” “comprises,” and / or “comprising”) when used in this specification specifies the presence of stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0055] It should be understood that references throughout this specification to "one embodiment," "another embodiment," or "a possible design" mean that specific features, structures, or characteristics associated with an embodiment or implementation are included in at least one embodiment of this application. Therefore, the appearance of "in one embodiment of this application," "in another embodiment of this application," or "a possible design" throughout this specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0056] It should also be understood that the specific numerical values ​​mentioned in the embodiments of this application do not limit the specific dimensions of specific features or structures. The relevant numerical values ​​may be provided for ease of explanation or may be the theoretically optimal value of a certain feature. In practice, the relevant dimensions may be within a range of the values, for example, the range may be ±50% of the optimal theoretical value. In practice, the range shall prevail to achieve the corresponding technical effect.

[0057] The embodiment of the present application provides a vehicle display method and system, a vehicle, and a vehicle. Figure 1 The vehicle-mounted display method and system are applied inside the vehicle. Based on the vehicle-mounted display method and system, the driver can see a virtual and real HUD in the cabin.

[0058] In some embodiments, see Figure 2 and Figure 3 This embodiment provides a data processing and rendering flow chart for an in-vehicle display method, as well as an in-vehicle display hardware system. A bus transmits multimodal data to a controller via LIN / CAN or Ethernet protocols. The multimodal data includes, but is not limited to, vehicle data, driving environment data, and driver data. The controller utilizes AI to decode, structure, and time-align multimodal message data from multiple data sources. When synchronizing multimodal message data from multiple data sources, multiple frames are extracted from each source. AI intelligently adjusts the timing, extracts features from the same timing, and reframes the data to achieve multi-source data synchronization. Feature extraction, graphic mapping, AI training, and semantic verification are then used to stylize the multi-source data and customize the UI graphic layout. These are then transformed into intuitive visual elements, and intelligently arranged to create personalized module combinations for the two-dimensional projection area. AI technology is used to transform multimodal message data from multiple data sources into stylized visual graphics and intelligently arrange them. The core concept is to achieve "data as interface" through semantic understanding and personalized decision-making. Specifically, in a multi-focus HUD, the display content for each focus plane is laid out. For example, there are display focus plane 1 and display focus plane 2. Multimodal message data from multiple data sources are assembled into data frame 1 and data frame 2, converted into stylized visualization graphics, and intelligently arranged on display focus plane 1 and display focus plane 2.

[0059] In some embodiments, to output a 3D reconstructed point cloud of the street scene, AI methods are used to achieve the 3D reconstruction and real-time rendering requirements of the HUD (Head-Up Display) fusion projection system. The AI ​​models the 3D scene based on multimodal data, including but not limited to image data and point cloud data. Dynamic targets are simultaneously located and then the modeled 3D scene and dynamic targets are fused through neural network rendering to generate a combined virtual and real HUD image. The key technologies in generating this combined virtual and real HUD image lie in cross-modal data alignment and real-time neural rendering.

[0060] In some embodiments, regarding the vehicle-mounted display method, the specific steps are as follows:

[0061] 1. Collect and preprocess multimodal data

[0062] Acquisition of CAN / LIN and Ethernet signals: The system collects CAN / LIN bus data (such as speed, cornering angle, vehicle position, environmental perception, driving status (acceleration, braking, steering, gear position, vehicle status, etc.) and Ethernet signals (such as navigation, camera images, LiDAR point clouds, driver dynamic information, and in-vehicle infotainment system data) from the entire vehicle. Deep learning models are used to analyze image data collected by the vehicle's cameras to perform tasks such as object detection, driver monitoring, and road recognition.

[0063] Data preprocessing: De-noises and corrects images, standardizes and cleans CAN / LIN and Ethernet signals to ensure data quality. Multimodal message data from multiple data sources is decoded, structured, and time-aligned.

[0064] 1) The above data includes environmental perception information, such as ambient light perception: using sensors to detect ambient light intensity and temperature (such as daylight, cloudy days, nighttime, etc.).

[0065] 2) The above data includes dynamic driving status data, including driving status (such as acceleration, braking, steering, navigation, etc.), and also includes displaying more details when parking (such as vehicle status, etc.).

[0066] 3) The above data includes driver dynamic information, such as eye position, sitting posture, driving status, identity, etc.

[0067] 4) When aligning data timing, use a camera to capture image data of the vehicle's surroundings. Ensure that the timestamps of the image acquisition match those of the CAN / LIN and Ethernet signals for effective fusion.

[0068] Cache multiple frames of data separately in advance. The cache can use queues, trees, buffers, etc. For example: after receiving data source source1, data source source2...data source sourceN, cache source1_buffer1, source1_buffer2, source1_buffer3, source1_buffer2, source2_buffer2, source2_buffer3,...source1_buffer2, source2_buffer2, source2_buffer3 respectively.

[0069] Two or more data sources come from CAN / LIN and / or Ethernet; data from source1 and source2 are first stored in a cache, and the data from source1 and source2 in the cache are time stamp matched to form a complete frame of multimodal data.

[0070] 2. Multimodal data is transformed into stylized definitions and customized UI graphic layouts through feature extraction, graphic mapping, AI training, and semantic verification. This is then transformed into intuitive view components, and intelligently arranged to create personalized modules for the 2D projection area, tailored to each user. The core goal is to achieve "data as interface" through semantic understanding and personalized decision-making.

[0071] Using AI technology to convert multimodal message data from multiple data sources into stylized visual graphics and achieve intelligent arrangement requires combining data feature extraction, generative AI models, and layout optimization algorithms to ultimately generate an intuitive interface that meets user needs.

[0072] 1) AI processing and stylized mapping of multimodal message data from multiple data sources

[0073] Data preprocessing and multimodal fusion message parsing: Decode and structure CAN / LIN, Ethernet messages, and sensor data (such as vehicle speed, temperature, and fault codes).

[0074] Timing alignment: Synchronize sensor data of different frequencies through timestamps, such as 1Hz GPS and 100Hz accelerometer.

[0075] Multimodal feature fusion: Integrate numerical data, textual fault descriptions, and time series patterns into a unified feature vector.

[0076] 2) Intelligent random layout and arrangement optimization of UI modules

[0077] The complete Smart Layout solution aims to optimize the random layout and arrangement of UI modules through AI technology, improving user experience and interface aesthetics. This solution combines multiple technologies, including reinforcement learning, genetic algorithms, and graph neural networks (GNNs), to form an efficient, flexible, and personalized layout generation and adjustment system.

[0078] ①AI-driven layout generation

[0079] AI-driven layout generation is a core module of intelligent layout. It primarily uses reinforcement learning to train an agent to learn from excellent UI layout examples. During training, the agent interacts with the virtual environment, gradually mastering the arrangement patterns and aesthetic principles of UI components. The design of the reward function is key to reinforcement learning and primarily encompasses metrics such as information density, visual balance, and focus guidance efficiency. The agent's action space includes translation, scaling, and rotation of UI components. Through continuous iteration and optimization, the agent is able to generate high-quality layout solutions that meet user needs.

[0080] The AI-driven layout generation process also introduces multimodal data fusion and attention hotspot prediction technology to enhance the intelligence of layout strategies. By using the Transformer model to fuse multimodal data (such as user behavior data, historical layout preferences, module functional attributes, etc.), the system can more comprehensively understand user needs and scenario characteristics. At the same time, based on CNN or RNN processing of user eye movement data, the system can predict the user's attention hotspots, thereby optimizing information priority calculations and ensuring that key information is displayed first.

[0081] ②Optimize layout based on genetic algorithm

[0082] In the application of genetic algorithms, spatial topology modeling technology is used to model the logical relationship between UI components, simulating the biological evolution process and further optimizing the generation process of layout solutions. Its core steps include crossover, mutation, and selection operations. Genetic coding encodes the position, size, and hierarchical information of each UI component into gene fragments to form the "genome" of the layout. The crossover operation extracts some gene fragments from two excellent layout solutions and combines them to generate a new layout solution. The mutation operation randomly adjusts the gene fragments to introduce new layout possibilities. The selection operation evaluates the pros and cons of the layout solution according to the fitness function, and retains the solution with high fitness for the next iteration. The fitness function mainly evaluates the aesthetics and functionality of the layout, including indicators such as symmetry, alignment, color coordination, readability, ease of operation, and information accessibility.

[0083] ③Modeling the relationship between neural networks and components

[0084] Graph neural networks (GNNs) model the logical relationships between UI components as a graph structure, which is used to predict the optimal relative positions of components. Nodes represent UI modules (such as instruments, charts, buttons, etc.), and edges represent the logical associations between nodes. For example, "vehicle speed" and "revolution speed" are often displayed adjacent to each other, or the "play" and "pause" buttons need to be aligned. By analyzing historical data and user behavior, association weights between components are constructed to guide layout generation. The advantage of GNNs lies in their ability to capture complex relationships between components and predict the optimal arrangement of components based on these relationships, thereby generating a layout solution that meets user expectations.

[0085] ④User interaction and dynamic adjustment

[0086] User interaction and dynamic adjustment functions further enhance the flexibility and personalization of the system. Personalized recommendations record users' layout preferences and historical choices, and use collaborative filtering algorithms to recommend frequently displayed module combinations. For example, if users often put the "Weather" and "Calendar" modules together, the system will prioritize similar layout solutions. The drag-and-drop optimization function allows users to manually adjust the position of UI components. After the adjustment is completed, the system will calculate the optimal position of the remaining components in real time, and use rendering optimization technology to ensure that the dynamic adjustment process of the layout is smooth and efficient, similar to Figma's automatic layout function. This dynamic adjustment mechanism using AI optimization can enhance users' creative freedom while ensuring the overall coordination of the layout.

[0087] Ultimately, the intelligent layout system integrates the strengths of reinforcement learning, genetic algorithms, and GNNs to form an efficient and flexible layout generation and optimization framework. Initial layout generation utilizes reinforcement learning and genetic algorithms to generate diverse layout solutions. Relationship modeling and optimization uses GNNs to model the logical relationships between components and further optimize the layout. User interaction and dynamic adjustment combine personalized user needs with manual adjustments to optimize layout solutions in real time. The system generates beautiful, efficient, and easy-to-use UI layout solutions while meeting individual user needs. This significantly improves UI design efficiency, reduces designer workload, and provides users with a more personalized and optimized user experience.

[0088] 3. AR effect realization

[0089] Using AI methods to achieve the 3D reconstruction and real-time rendering requirements of the HUD virtual-reality fusion projection system, the goal is to output a 3D reconstructed point cloud map of the street scene;

[0090] AI models three-dimensional scenes through multimodal data (images + point clouds), locates dynamic targets at the same time, and then uses neural networks to render and fuse the three-dimensional scenes and dynamic targets, finally generating a HUD image that combines virtual and real. The key technology lies in cross-modal data alignment and real-time neural network rendering.

[0091] The display priority and fusion strategy of virtual content can be adaptively adjusted according to the user's focus and the characteristics of the scene.

[0092] Leveraging the graphics processing unit (GPU) for efficient computing, the system accelerates real-time rendering and displays AR effects on the head-up display, enabling users to interact with the environment.

[0093] besides:

[0094] 1) Rendered images and screens can be adjusted and optimized in real time based on environmental perception information. For example, generative AI automatically adjusts the HUD's contrast and brightness based on environmental perception information to ensure clear visibility in various lighting conditions. Using PQ (picture quality) optimization algorithms, the HUD optimizes the color contrast of the displayed image interface. For example, in bright sunlight, the HUD can select high-contrast color combinations to improve readability; in low-light environments, it can use softer tones to reduce visual fatigue.

[0095] 2) Generate richer images based on dynamic driving status data. For example, the HUD can generate relevant images based on driving status (such as acceleration, braking, steering, etc.). When driving fast, a simpler image containing key information (such as speed and navigation) is generated and displayed, while more details (such as entertainment information and vehicle status) are displayed when the car is parked. Behavior prediction is also possible. Generative AI can analyze driving behavior and anticipate the driver's needs (such as automatically generating a zoomed-in navigation instruction image when entering a complex road section).

[0096] 4. Iteration and Optimization

[0097] Continuous data collection: Continue to collect data in real-world usage to improve the model.

[0098] Feedback mechanism: Establish a user feedback mechanism to continuously optimize the model's effectiveness and user experience.

[0099] Feedback can be in the form of personalized recommendations and drag-and-drop optimization:

[0100] Personalized recommendations can record user historical selection preferences and recommend high-frequency display module combinations through collaborative filtering.

[0101] Drag optimization means that after the user manually adjusts the layout, AI recalculates the optimal position of the remaining components in real time (similar to Figma's automatic layout).

[0102] Furthermore, in some embodiments, the in-vehicle display process of the multimodal data is an end-to-end process, which is manifested as follows:

[0103] During the input phase: users select display modules and style preferences, such as "sports mode" or "night theme," and input multimodal message data from multiple data sources into the controller. During the processing phase: stylized UI components are generated through pre-trained generative models (such as cGAN) to achieve the conversion from input data to mapping graphics; multiple candidate layout solutions are then output through reinforcement learning / genetic algorithms; and finally, the candidate layout solutions are ranked based on layout scores based on aesthetics, information priority, and historical user behavior. During the output phase: a visual interface output from the controller is rendered in real time, outputting a virtual-real HUD screen that supports dynamic updates and interactive adjustments.

[0104] To further illustrate the data-to-image conversion process, in some embodiments, the automotive HUD system realizes the dynamic generation of the entire link from raw signals to UI images through multimodal data fusion and intelligent algorithms. The entire technology first relies on 128-line lidar, 8-megapixel camera and biosensor to collect environmental point cloud, road image and driver eye movement data in real time, and realizes the spatiotemporal synchronization of multi-source signals through the IEEE 1588v2 protocol (time error <1ms), and maps the lidar point cloud and camera image to a unified vehicle coordinate system (spatial error <3cm). Subsequently, the system uses the BEV bird's-eye view network to build a 3D semantic map to identify key road elements, combines the Transformer model to analyze driving intentions, and uses the LSTM neural network to predict the probability distribution of gaze focus in the next 2 seconds to form a dynamic attention hot zone with a resolution of 1024×768.

[0105] In the information generation stage, the reinforcement learning model (Q-Learning) dynamically evaluates the information priority according to the real-time scenario (such as vehicle speed and curve curvature). When multiple UI elements have spatial conflicts, the Shapley value algorithm is used to quantify the layout contribution of each module, so as to realize intelligent adjustments such as stretching the navigation prompt along the tangent and automatically placing the emergency warning on the top. The optical rendering process projects different information to multiple focal planes such as 3.2m and 4.5m through depth of field layering technology, combines aspherical lenses (curvature radius 120mm) with DLSS super-resolution technology to eliminate image distortion (error <0.1 pixel), and dynamically adjusts the HUD brightness (200-1200cd / m 2 ) and color temperature (covering sRGB 98% color gamut).

[0106] Information Type Instant Value Q Long-term value V value Forward Collision Warning 0.93 0.88 Lane keeping reminder 0.75 0.82 Music playback control 0.32 0.15

[0107] Table 1. Example of constructing an information value matrix based on the reinforcement learning Q-learning algorithm

[0108] The final system monitors the driver's cognitive load in real time through changes in pupil diameter (3-7mm) and blinking frequency (12-20 times / minute), and uses A / B testing in a digital twin environment to continuously optimize the layout strategy. Compared with traditional HUD systems, key indicators have achieved breakthrough improvements: end-to-end latency has been reduced from 120ms to 80ms, the contrast ratio in strong light environments has been increased by 3 times to 15:1, the eye tracking response speed has been accelerated to 80ms, and the information density adaptation range has been expanded from a fixed 3 levels to dynamic 16 levels of adjustment, forming a closed-loop optimization system for human-machine collaboration.

[0109]

[0110]

[0111] Table 2. Technical characteristics comparison layout classification style and characteristics:

[0112]

[0113] The images generated by this solution are rendered using a graphics processing unit (such as a GPU) and sent to a head-up display.

[0114] Through the above steps, the present invention utilizes AI to decode, structure, and time-align multimodal message data from multiple data sources. Through feature extraction, graphic mapping, AI training, and semantic verification, it achieves stylized definition of multi-source data and customized UI graphic layout solutions. AI methods are used to achieve the 3D reconstruction and real-time rendering requirements of the HUD virtual-reality fusion projection system, outputting a 3D reconstructed point cloud of the street scene and generating a virtual-reality HUD image.

[0115] This invention enables real-time generation and rendering of HUD display interfaces, providing richer, more diverse, timely, interactive, and AR-integrated display information. Leveraging generative AI techniques, complex and creative imagery and visual effects can be created. This combination typically involves combining generative models with GPU rendering technology, generating image data before rendering using the GPU. This significantly improves performance and content richness.

[0116] In addition, the embodiment of the present application provides a vehicle-mounted display system, see Figure 3 The vehicle-mounted display system includes a vehicle bus, a controller, and an optical engine. The vehicle bus is used to transmit multimodal signals, and bus types include but are not limited to LIN / CAN bus and in-vehicle Ethernet. The controller receives the multimodal signals transmitted by the bus, pre-processes the multimodal data, generates a virtual image based on a model, and renders the virtual image to generate a virtual and real HUD image. The controller includes but is not limited to an MCU (microcontroller), a CPU (central processing unit), a GPU (graphics processing unit), an NPU (neural network processor), and a DSP (digital signal processor). The communication and control interfaces of each controller include but are not limited to GPIO (general purpose input / output), SPI (universal asynchronous receiver / transmitter), and UART (serial peripheral interface). The optical engine receives and displays HUD image data, and the HUD optical engine display interface includes but is not limited to LVDS, MIPI DSI. The vehicle-mounted display system implements the aforementioned vehicle-mounted display method to achieve real-time and virtual and real HUD display. Based on this vehicle-mounted display system, real-time synchronous processing of multimodal data in driving scenarios and real-time generation of virtual and real HUD images are achieved.

[0117] In addition, the embodiment of the present application provides a vehicle, see Figure 1The vehicle is equipped with the vehicle display system described in the above embodiment and executes the above vehicle display method. The vehicle has all the beneficial effects of the above vehicle display method, which will not be repeated in this application.

[0118] The vehicle may be a fuel vehicle, a plug-in hybrid vehicle or a new energy vehicle, etc., and this application does not make any specific restrictions on this.

[0119] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

[0120] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0121] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the scope of protection of this application includes the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0122] This document uses specific examples to illustrate the working principle and implementation method of the omnidirectional antenna of the present application. The description of the above embodiments is only used to help understand the specific settings and core ideas of the present application. At the same time, for those skilled in the art, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present application.

[0123] The above content is only a specific embodiment of this application, but the scope of protection of this application is not limited to this. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A vehicle-mounted display method, characterized in that: The method comprises: Collect and preprocess multimodal data; Stylized definition of multi-source data and customized layout of UI graphics; AR effect is achieved.

2. The vehicle-mounted display method according to claim 1, characterized in that: The collecting and preprocessing of multimodal data includes: acquiring CAN / LIN bus data and Ethernet signals; collecting image data; and preprocessing data.

3. The vehicle-mounted display method according to claim 2, characterized in that: The CAN / LIN bus data includes but is not limited to speed, turning angle, vehicle position, environmental perception, and driving status.

4. The vehicle-mounted display method according to claim 2, characterized in that: The Ethernet signal includes but is not limited to navigation data, camera image data, lidar point cloud data, driver dynamic information, and data from the in-vehicle infotainment system.

5. The vehicle-mounted display method according to claim 2, characterized in that: The image data acquisition includes acquiring image data of the vehicle's surrounding environment through a camera; and marking a timestamp on the acquired image data, wherein the timestamp matches the CAN / LIN signal.

6. The vehicle-mounted display method according to claim 2, characterized in that: The pre-processing data includes but is not limited to: image denoising and correction; Standardize CAN / LIN bus data and Ethernet signal data, and cleanse CAN / LIN bus data and Ethernet signal data; Decode, structure, and time-align multimodal message data from multiple data sources.

7. The vehicle-mounted display method according to claim 6, characterized in that: The pre-processed data includes but is not limited to environmental perception information, driving status dynamic data, and driver dynamic information.

8. The vehicle-mounted display method according to claim 7, characterized in that: The environmental perception information includes but is not limited to ambient light intensity and temperature.

9. The vehicle-mounted display method according to claim 7, characterized in that: The driving status dynamic data includes but is not limited to driving status.

10. The vehicle-mounted display method according to claim 7, characterized in that: The driver dynamic information includes but is not limited to eye position, sitting posture, driving status, and identity information.

11. The vehicle-mounted display method according to claim 6, characterized in that: When performing time sequence alignment on the multimodal message data from multiple data sources, multiple frames of data are first cached for each source message data, and the caching form of the message data includes but is not limited to a queue and a tree.

12. The vehicle-mounted display method according to claim 11, characterized in that: When performing time alignment on the multimodal message data from multiple data sources, the message data from each source in the cache is matched, and the message data with the same timestamp are formed into a complete frame of multimodal data.

13. The vehicle-mounted display method according to claim 1, characterized in that: The stylized definition of multimodal message data from multiple data sources and the customized layout of UI graphics include: AI processing of multimodal message data from multiple data sources and stylized mapping, intelligent random layout of UI modules and arrangement optimization.

14. The vehicle-mounted display method according to claim 13, characterized in that: By performing feature extraction, graphic mapping, AI training, and semantic verification on multimodal data, we can complete the stylized definition of multi-source data and the customized layout of UI graphics.

15. The vehicle-mounted display method according to claim 14, characterized in that: The AI ​​processing of multimodal message data and stylized mapping from multiple data sources includes but is not limited to: data preprocessing and multimodal fusion message parsing, time series alignment, and multimodal feature fusion.

16. The vehicle-mounted display method according to claim 15, characterized in that: The data preprocessing and multimodal fusion message parsing include but are not limited to: decoding and structuring CAN messages, LIN messages, Ethernet messages, and sensor data; wherein the sensor data includes but is not limited to vehicle speed, temperature, and fault code.

17. The vehicle-mounted display method according to claim 15, characterized in that: The timing alignment specifically involves synchronizing sensor data of different frequencies through timestamps.

18. The vehicle-mounted display method according to claim 15, characterized in that: The multimodal feature fusion specifically includes integrating numerical data, textual fault descriptions, and time series patterns into a unified feature vector.

19. The vehicle-mounted display method according to claim 13, characterized in that: The intelligent random layout UI module arrangement optimization is specifically: AI-driven layout generation, layout optimization based on genetic algorithm, modeling of component relationships through graph neural network, user interaction and dynamic adjustment.

20. The vehicle-mounted display method according to any one of claims 1 to 19, characterized in that: The in-vehicle display process is implemented end-to-end.

21. The vehicle-mounted display method according to claim 20, characterized in that: The end-to-end model is continuously iterated and optimized.

22. The vehicle-mounted display method according to claim 21, characterized in that: The continuous iteration and optimization of the model specifically includes: continuously collecting usage data and outputting feedback to users.

23. The vehicle-mounted display method according to claim 22, characterized in that: The feedback output to the user is realized as personalized recommendation and drag optimization; the personalized recommendation is used to record the user's historical selection preferences and recommend high-frequency display module combinations through collaborative filtering; the drag optimization is used to calculate the optimal position of the remaining components in real time after the user manually adjusts them.

24. The vehicle-mounted display method according to claim 1, characterized in that: The AR effect realization includes: constructing a three-dimensional scene based on multimodal data, locating dynamic targets, and generating a virtual and real HUD image through neural rendering fusion, wherein the multimodal data includes images and point clouds.

25. The vehicle-mounted display method according to claim 24, characterized in that: Adapt the display priority and fusion strategy of virtual content based on the user's focus and the characteristics of the scene.

26. The vehicle-mounted display method according to claim 24, characterized in that: Real-time rendering via graphics processing units.

27. The vehicle-mounted display method according to claim 26, characterized in that: Adjust rendered images in real time based on environmental awareness information.

28. The vehicle-mounted display method according to claim 26, characterized in that: Generate images based on driving status dynamic data.

29. A vehicle-mounted display system, characterized in that: The system comprises: a vehicle bus for transmitting multimodal signals; A controller that receives multimodal signals transmitted by the bus, preprocesses the multimodal data, generates a virtual image based on the model, and renders the virtual image to generate a virtual-real HUD image; Optical machine, receives HUD image data and displays it.

30. The vehicle-mounted display system according to claim 29, characterized in that: The in-vehicle display system uses the in-vehicle display method described in one of claims 1-28 for display.

31. A vehicle, characterized in that: Display is performed using the vehicle-mounted display method described in any one of claims 1 to 28.

Images