Augmented reality head-up display methods, devices, vehicle controllers, and vehicles

Abstract

This application relates to an augmented reality head-up display method, device, vehicle controller, and vehicle. The method includes: acquiring vehicle environment-related data; the vehicle environment-related data includes at least one of environmental perception data, navigation map data, and vehicle motion state data; inputting the vehicle environment-related data into a large model, using the large model to comprehensively understand and infer the driving scene outside the vehicle to obtain information of interest to the vehicle; and displaying the information of interest on the display interface of the in-vehicle augmented reality head-up display. This method can effectively improve the driver's situational awareness efficiency in complex road conditions, reduce the cognitive load of information filtering, and enhance driving safety and interactive experience.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to an augmented reality head-up display method, device, vehicle controller, and vehicle. Background Technology

[0002] With the rapid evolution of intelligent driving technology and the continuous upgrading of human-machine interaction experience, augmented reality (AR) technology is accelerating its penetration into automotive cockpits, becoming a core component of the intelligent cockpit ecosystem. In recent years, the global automotive industry has been undergoing a transformation from electrification to intelligentization, and in-vehicle display technology, as the core interface connecting vehicles, roads, and users, is becoming increasingly important.

[0003] Augmented reality head-up displays (AR-HUDs), with their ability to deeply integrate virtual information with real-world scenes, are considered by the industry as a next-generation display solution to enhance driving safety and interactive immersion, attracting significant investment from numerous OEMs and technology companies. Currently, AR-HUDs typically use vehicle sensors to perceive the external environment and directly acquire the user's navigation information. The in-vehicle AR-HUD is then displayed based on these perceptions, typically showing lane lines, other vehicles, and the navigation destination.

[0004] However, the aforementioned augmented reality head-up display methods suffer from low levels of intelligence. Summary of the Invention

[0005] Therefore, it is necessary to provide an augmented reality head-up display method, device, vehicle controller, and vehicle that can improve the intelligence level of augmented reality head-up display in response to the above-mentioned technical problems.

[0006] Firstly, this application provides an augmented reality head-up display method, including:

[0007] Acquire vehicle environment-related data; vehicle environment-related data includes at least one of the following: environmental perception data, navigation map data, and vehicle motion status data;

[0008] Vehicle environment-related data is input into a large model, which then performs a comprehensive understanding and reasoning of the driving scenario outside the vehicle to obtain the vehicle's attention-grabbing information.

[0009] The information to be noted is displayed on the in-vehicle augmented reality head-up display interface.

[0010] In one embodiment, the method further includes:

[0011] Acquire user-related data; user-related data includes user profiling information and / or scenario triggering information;

[0012] Vehicle environment-related data is input into a large model, which then performs a comprehensive understanding and inference of the external driving scenario to obtain the vehicle's key information, including:

[0013] User-related data and vehicle-related environmental data are input into a large model. The large model then performs a comprehensive understanding and reasoning of the driving scenarios outside the vehicle to obtain the vehicle's key information.

[0014] In one embodiment, the aforementioned information to be of interest includes user-related recommendation information; the aforementioned input of user-related data and vehicle environment-related data into a large model, and the comprehensive understanding and reasoning of the driving scenario outside the vehicle through the large model, yields the vehicle's information to be of interest, including:

[0015] User-related data and vehicle environment-related data are input into a large model. The large model then comprehensively understands and infers the driving scenario outside the vehicle to obtain user-related recommendation information.

[0016] The navigation map data includes information on shops and / or attractions located within a preset distance from the vehicle's current location; the scene trigger information is the vehicle entering the first driving road, and / or the current date is a special date specified by the user.

[0017] In one embodiment, the aforementioned information of concern also includes vehicle hazard warning information; the aforementioned input of vehicle environment-related data into a large model, and the comprehensive understanding and reasoning of the driving scenario outside the vehicle through the large model, yields the vehicle's information of concern, including:

[0018] The vehicle navigation map is determined based on environmental perception data, navigation map data, and vehicle motion status data;

[0019] The vehicle navigation map, environmental perception data, and vehicle motion status data are input into the large model. The large model then performs a comprehensive understanding and reasoning of the driving scenario outside the vehicle to obtain hazard warning information for the vehicle.

[0020] In one embodiment, the aforementioned environmental perception data includes: state information of multiple traffic participants; the state information includes at least one of the following: type, size, position, speed, and direction of the traffic participants;

[0021] The navigation map data includes information about the surrounding roads outside the vehicle, including: information about the intersections in front of the vehicle, and / or, information about the lane markings where the vehicle is located.

[0022] Vehicle motion status data includes at least one of the vehicle's current position, direction of travel, and speed.

[0023] In one embodiment, the method further includes:

[0024] Acquire drone perception information around the vehicle;

[0025] Vehicle environment-related data is input into a large model, which then performs a comprehensive understanding and inference of the external driving scenario to obtain the vehicle's key information, including:

[0026] The drone's perception information and vehicle environment-related data are input into a large model. The large model then performs a comprehensive understanding and reasoning of the driving scenario outside the vehicle to obtain the vehicle's attention-grabbing information.

[0027] In one embodiment, displaying the information of interest on the in-vehicle augmented reality head-up display interface includes:

[0028] Based on the priority of the information to be monitored, the information to be monitored will be displayed differently on the display interface.

[0029] Secondly, this application also provides an augmented reality head-up display device, comprising:

[0030] The acquisition module is used to acquire vehicle environment-related data; the vehicle environment-related data includes at least one of the following: environmental perception data, navigation map data, and vehicle motion status data.

[0031] The reasoning module is used to input vehicle environment-related data into the large model, and through the large model, to comprehensively understand and reason about the driving scenario outside the vehicle to obtain the vehicle's attention information;

[0032] The display module is used to display the information to be viewed on the display interface of the in-vehicle augmented reality head-up display.

[0033] Thirdly, this application also provides a vehicle controller having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0034] Acquire vehicle environment-related data; vehicle environment-related data includes at least one of the following: environmental perception data, navigation map data, and vehicle motion status data;

[0035] Vehicle environment-related data is input into a large model, which then performs a comprehensive understanding and reasoning of the driving scenario outside the vehicle to obtain the vehicle's attention-grabbing information.

[0036] The information to be noted is displayed on the in-vehicle augmented reality head-up display interface.

[0037] Fourthly, this application also provides a vehicle including the vehicle controller described in the third aspect above.

[0038] The aforementioned augmented reality head-up display method, device, vehicle controller, and vehicle integrate multi-source vehicle environment information such as environmental perception data, navigation maps, and vehicle motion status. By leveraging the powerful comprehensive understanding and reasoning capabilities of large models, it intelligently identifies key information of interest in the driving scenario and presents it intuitively on the head-up display interface in an augmented reality manner. This effectively improves the driver's situational awareness efficiency in complex road conditions, reduces the cognitive load of information filtering, and enhances driving safety and interactive experience. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a diagram illustrating the application environment of an augmented reality head-up display method in one embodiment.

[0041] Figure 2 This is a flowchart illustrating an augmented reality head-up display method in one embodiment;

[0042] Figure 3 This is a flowchart illustrating an augmented reality head-up display method in another embodiment;

[0043] Figure 4 This is a flowchart illustrating an augmented reality head-up display method in another embodiment;

[0044] Figure 5 This is a vehicle navigation map in one embodiment;

[0045] Figure 6 This is a flowchart illustrating an augmented reality head-up display method in another embodiment;

[0046] Figure 7 This is a flowchart illustrating an augmented reality head-up display method in another embodiment;

[0047] Figure 8 This is a structural block diagram of an augmented reality head-up display device in one embodiment;

[0048] Figure 9 This is an internal structural diagram of the vehicle controller in one embodiment. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] With the rapid evolution of intelligent driving technology and the continuous upgrading of human-machine interaction experience, augmented reality (AR) technology is accelerating its penetration into automotive cockpits, becoming a core component of the intelligent cockpit ecosystem. In recent years, the global automotive industry has been undergoing a transformation from electrification to intelligentization, and in-vehicle display technology, as the core interface connecting vehicles, roads, and users, is becoming increasingly important.

[0051] Augmented reality head-up displays (AR-HUDs), with their ability to deeply integrate virtual information with real-world scenes, are considered by the industry as a next-generation display solution to enhance driving safety and interactive immersion, attracting significant investment from numerous OEMs and technology companies. Currently, AR-HUDs typically use vehicle sensors to perceive the external environment and directly acquire the user's navigation information. The in-vehicle AR-HUD is then displayed based on these perceptions, typically showing lane lines, other vehicles, and the navigation destination.

[0052] However, the aforementioned augmented reality head-up display methods suffer from low levels of intelligence. Therefore, this application provides a novel augmented reality head-up display method to address these issues.

[0053] The augmented reality head-up display method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, the application environment includes a data storage system 102 and a server 104. The data storage system 102 can store the data that the server 104 needs to process. The data storage system 102 can be integrated onto the server 104, or it can be located in the cloud or on other network servers. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The data storage system 102 can communicate with the server 104. For example, the server 104 can send a data request to the data storage system to obtain data from the data storage system 102, and then process the obtained data. In this application, real-time vehicle environment-related data acquired by sensors can be stored in the data storage system 102. Then, the server can obtain vehicle environment-related data from the data storage system 102 and process the vehicle environment-related data to understand and infer the driving scene outside the vehicle.

[0054] It should be noted that the aforementioned server can be a vehicle controller, or it can be applied to other terminals, such as mobile apps or cloud platforms, to process data and send the processing results to the vehicle.

[0055] In one exemplary embodiment, such as Figure 2 As shown, an augmented reality heads-up display method is provided, which is applied to... Figure 1 Taking server 104 as an example, the explanation includes:

[0056] S201. Obtain vehicle environment-related data; vehicle environment-related data includes at least one of the following: environmental perception data, navigation map data, and vehicle motion status data.

[0057] The environmental perception data includes dynamic information of traffic participants obtained from the intelligent driving perception system. This dynamic information includes the status of various traffic participants, such as type (cars, trucks, electric two-wheelers, etc.), length, width, position, speed, direction, and turn signal status. For example, when the traffic participant is a vehicle, the status information includes the vehicle type, length, width, current position, current speed, current direction, and current turn signal status. Similarly, when the traffic participant is not a vehicle, the status information includes the non-vehicle type (e.g., person, fixed obstacle, etc.), current position, current speed, and current direction.

[0058] The navigation map data includes road information surrounding the vehicle's current location, obtained from the navigation map system. This surrounding road information includes: intersection information ahead of the vehicle, and / or lane information, such as the intersection conditions (whether there are merging or branching roads), the location and current status of traffic lights, the location and type of the vehicle's current lane, and the speed limit location and value for the vehicle's current lane. Additionally, the navigation map data may also include information on shops and / or tourist attractions located within a preset distance from the vehicle's current location, obtained from the navigation map system.

[0059] The vehicle motion status data includes information such as the vehicle's position, direction of travel, and speed.

[0060] In this embodiment, when intelligent display is required on the display screen of augmented reality head-up display, the server can first obtain at least one of environmental perception data, navigation map data and vehicle motion state data, so that the server can subsequently predict the vehicle's attention information based on at least one of environmental perception data, navigation map data and vehicle motion state data.

[0061] S202. Input vehicle environment-related data into the large model, and use the large model to comprehensively understand and reason about the driving scenario outside the vehicle to obtain the vehicle's attention information.

[0062] The vehicle's attention information includes hazard warnings and user-related recommendations. Hazard warnings may include information such as a lane change ahead where a vehicle may suddenly cut in, a pedestrian crossing ahead where a pedestrian may jaywalk, or traffic congestion ahead where an accident may occur. User-related recommendations may include information such as tourist attractions and shops that the user may be interested in.

[0063] The large model can be pre-trained based on sample vehicle environment-related data to enable it to comprehensively understand and reason about the driving scenario outside the vehicle. This large model can be a Large Language Model (LLM).

[0064] For example, the large model can also be a multimodal visual language large model. This model employs a serial encoder-decoder architecture. The encoder consists of an image feature extraction layer and a temporal feature fusion layer, while the decoder carries scene inference and classification output branches. When implementing the large model's understanding and inference of driving scenarios outside the vehicle, the vehicle environment-related data can first be uniformly converted into a standard input format and fed into the encoder. The encoder first extracts visual spatial features and temporal change features from the vehicle environment-related data, then fuses these features and feeds them into the decoder to understand and infer driving scenarios, identifying risk scenarios such as vehicle weaving, pedestrian crossings, and road congestion. Finally, it combines user travel profiles to match nearby businesses and attractions, ultimately classifying and outputting two types of information to be considered: hazard warning information and personalized recommendation information.

[0065] Another example is the edge-side lightweight general-purpose large model. The basic structure of this model is a multi-layered Transformer stacked architecture, with built-in perception and understanding modules, causal reasoning modules, and information filtering and output modules. When implementing the large model's understanding and reasoning of the driving scenario outside the vehicle, vehicle environment-related data can be aggregated, including at least one of environmental perception data, navigation map data, and vehicle motion state data. This data is then normalized and preprocessed before being input into the edge-side lightweight general-purpose large model. The built-in perception and understanding module of the model associates spatial and temporal data from the vehicle environment-related data. The associated data is then input into the causal reasoning module to deduce the scenario evolution trend and predict potential driving hazards. Simultaneously, the information filtering and output module retrieves service content that matches the travel scenario, filtering out complete vehicle information of interest.

[0066] In this embodiment, after obtaining the vehicle environment-related data, the vehicle environment-related data can be input into a large model. The large model can then comprehensively understand and reason about the driving scenario outside the vehicle to obtain the vehicle's attention-related information.

[0067] S203. Display the information to be noted on the display interface of the in-vehicle augmented reality head-up display.

[0068] In this embodiment, after obtaining the information to be concerned, the information to be concerned can be displayed on the display interface of the in-vehicle augmented reality head-up display.

[0069] For example, when the information to be concerned is a hazard warning, the hazard warning can be displayed directly on the display interface of the in-vehicle augmented reality head-up display. Alternatively, the navigation map can be displayed in the in-vehicle augmented reality head-up display, and the hazard warning can be marked and displayed on the navigation map.

[0070] For example, when the information to be paid attention to is a recommendation related to the user, the recommendation can be directly displayed on the display interface of the in-vehicle augmented reality head-up display. Alternatively, the navigation map can be displayed in the in-vehicle augmented reality head-up display, and the recommendation can be marked and displayed on the navigation map.

[0071] In this embodiment, in addition to highlighting the information to be of interest on the display interface, appropriate reminders can also be given to the user through instruments, central control screens, or voice commands.

[0072] In this embodiment, by integrating multi-source vehicle environment information such as environmental perception data, navigation maps, and vehicle motion status, and leveraging the powerful comprehensive understanding and reasoning capabilities of the large model, key information requiring attention in the driving scenario is intelligently identified and presented intuitively on the head-up display interface in an augmented reality manner. This effectively improves the driver's situational awareness efficiency in complex road conditions, reduces the cognitive load of information filtering, and enhances driving safety and interactive experience.

[0073] In one exemplary embodiment, in the above Figure 2 Based on the illustrated embodiments, see also Figure 3 The above methods also include:

[0074] S204. Obtain user-related data; user-related data includes user profiling information and / or scene trigger information.

[0075] The user profile information includes the user's birthday, gender, interests, tastes, etc., the scenario trigger information is the vehicle entering the road for the first time, and / or the current date is a special date for the user, etc.

[0076] It should be noted that the user-related data here can be driver-related data or vehicle occupant-related data. Large-scale models can be trained based on different user-related data and vehicle environment-related data to obtain large-scale models applicable to different users.

[0077] It should be noted that, in order to prevent repeated triggering, each scenario trigger information will only be triggered once within a preset time period (such as one day or one week).

[0078] In this embodiment, to further improve the understanding and reasoning of the large model regarding the driving scenario outside the vehicle and make it more in line with user preferences, user-related data can also be acquired. This allows for subsequent large model predictions based on user-related data and vehicle environment-related data, resulting in more relevant information that aligns with user preferences.

[0079] The aforementioned S202 includes: inputting user-related data and vehicle environment-related data into a large model, and using the large model to comprehensively understand and reason about the driving scenario outside the vehicle to obtain the vehicle's attention information.

[0080] The large model can be pre-trained based on sample vehicle environment data and sample user data to enable it to comprehensively understand and reason about the driving scenario outside the vehicle. This large model can be a Large Language Model (LLM).

[0081] For example, the large model can be a model structure that integrates Convolutional Neural Networks (CNN) and Graph Neural Networks (GNN). This model structure consists of a data access layer, a CNN visual feature extraction layer, a GNN spatial topology association layer, a fusion inference layer, and a result output layer. When the large model understands and infers the driving scene outside the vehicle, vehicle environment-related data is first synchronously fed into the model through the data access layer. Then, the CNN visual feature extraction layer extracts visual features of roads, vehicles, and pedestrians. The GNN spatial topology association layer analyzes the positional and motion relationships between road condition elements. Furthermore, the fusion inference layer combines user data to perform correlation inference, identify and predict dangerous situations such as lane changes, congestion, and pedestrian crossings, and matches appropriate location recommendation information. Finally, the result output layer outputs the information of interest.

[0082] For example, the large model can adopt the spatiotemporal convolutional model STGCN structure, which consists of a spatiotemporal convolution module, a feature aggregation module, a scene prediction module, and an information sorting module in sequence. When the large model understands and infers the driving scene outside the vehicle, it uses spatiotemporal convolution to capture the spatial distribution features and temporal dynamic change features of vehicle environment-related data simultaneously. Then, the feature aggregation module aggregates the spatial distribution features and temporal dynamic change features, and inputs the aggregated features into the scene prediction module to infer the subsequent evolution trend of the scene. Finally, the information sorting module combines user attribute data to filter and adapt service content, distinguish between safety warnings and life recommendation information, and organizes and generates vehicle attention information.

[0083] In this embodiment, after obtaining user-related data and vehicle environment-related data as described above, the user-related data and vehicle environment-related data can be input into a large model for prediction, so as to comprehensively understand and reason about the driving scenario outside the vehicle through the large model and obtain the vehicle's attention information.

[0084] For example, user preference information from user-related data and navigation map data from vehicle environment-related data can be input into a large model to predict the user's frequently visited locations, so as to recommend navigation maps from the current location to the frequently visited locations to the user.

[0085] For example, after inputting user preference information from user-related data and navigation map data from vehicle environment-related data into the large model, corresponding prompts can be entered, such as, "You are an understanding in-car life assistant who cares a lot about the user's physical health and emotional well-being. Now you have discovered <insert trigger condition>, based on user characteristics <insert user profile information>, and current driving road information <insert road point of interest information>, do you think there are any locations on the road ahead that are worth the user's attention? Please output the points of interest and the reasons in JSON format, such as {"point of interest": "Holiland Cake Shop", "reason": "Today is the user's birthday"}.

[0086] This large model can combine triggering conditions, user profiles, and road interest points to make personalized recommendations for users. For example, if a user likes hot pot, highly-rated hot pot restaurants can be recommended when the user first enters a road, or if a user likes hiking, scenic spots and mountains can be suggested when the user first enters a scenic area.

[0087] Optionally, the following provides a specific implementation method for inputting user-related data and vehicle environment-related data into a large model to predict the vehicle's information of concern. The above-mentioned S202 further includes:

[0088] User-related data and vehicle-environment-related data are input into a large model. The large model then performs a comprehensive understanding and reasoning of the driving scenario outside the vehicle to obtain user-related recommendation information.

[0089] The navigation map data includes information on shops and / or attractions located within a preset distance from the vehicle's current location; the scene trigger information is the vehicle entering the first driving road, and / or the current date is a special date specified by the user.

[0090] In this embodiment, when the vehicle enters the initial driving route and / or the current date is a special date for the user, user profile information, and / or scenario trigger information, as well as vehicle environment-related data, can be input into the large model. The large model then comprehensively understands and infers the external driving scenario to obtain user-related recommendation information. It should be noted that the scenario trigger information here refers to the vehicle entering the initial driving route and / or the current date being a special date for the user. The navigation map data in the vehicle environment-related data includes shop information and / or attraction information within a preset distance range from the vehicle's current location.

[0091] In this embodiment, by integrating user profile information with scenario triggering information (such as the first driving route, special dates for the user, etc.) and vehicle environmental data, a large model is used to achieve personalized scenario understanding and intelligent reasoning. It proactively recommends related information such as nearby shops and attractions to users and enhances the presentation of augmented reality, thereby significantly improving the accuracy and timeliness of information services in driving scenarios, enhancing the personalized experience of human-computer interaction, effectively mining potential user needs, and improving the enjoyment and convenience of the driving process.

[0092] In one exemplary embodiment, in the above Figure 2 Based on the illustrated embodiment, the aforementioned information of concern also includes vehicle hazard warning information; furthermore, by inputting vehicle environmental data into a large model, and through comprehensive understanding and reasoning of the driving scenario outside the vehicle, the specific implementation methods of the vehicle's information of concern can be predicted. See [link to relevant documentation]. Figure 4 The aforementioned S202 includes:

[0093] S301. Determine the vehicle navigation map based on environmental perception data, navigation map data, and vehicle motion status data.

[0094] The environmental perception data includes: status information of multiple traffic participants; the status information includes at least one of the following: type, size, location, speed, and direction of the traffic participants.

[0095] The navigation map data includes information about the surrounding roads outside the vehicle, including: information about intersections in front of the vehicle, and / or, information about the lane lines where the vehicle is located.

[0096] The vehicle motion status data includes at least one of the vehicle's current position, direction of travel, and speed.

[0097] In this embodiment, environmental perception data, navigation map data, and vehicle motion status data can be plotted into a vehicle navigation map according to the correct scale and position. See [link / reference]. Figure 5 Furthermore, the vehicle size, movement information, indicator light information, and other information are marked with text on the vehicle navigation map, and a correct scale is drawn for easy viewing by the user.

[0098] S302. Input the vehicle navigation map, environmental perception data and vehicle motion state data into the large model. Through the large model, a comprehensive understanding and reasoning of the driving scenario outside the vehicle is performed to obtain the vehicle's danger warning information.

[0099] In this embodiment, after obtaining the vehicle navigation map, environmental perception data, and vehicle motion state data, the vehicle navigation map, environmental perception data, and vehicle motion state data can be input into the large model. The large model can then comprehensively understand and reason about the driving scenario outside the vehicle to obtain the vehicle's hazard warning information.

[0100] Optionally, the large-scale model can adopt a model structure that combines a SAM image segmentation large-scale model and a 3D perception and reasoning model. This model structure includes a data preprocessing layer, a multi-source feature extraction layer, a spatial target association layer, a risk inference layer, and a warning output layer. When enabling the large-scale model to understand and reason about the driving scene outside the vehicle, navigation map data, environmental perception data, and vehicle motion data are uniformly formatted and input into the model's data preprocessing layer for preprocessing. The preprocessed data is then input into the multi-source feature extraction layer to segment and identify scene targets such as roads, vehicles, pedestrians, and zebra crossings. Based on the spatial target relationship layer, the coordinate positions and driving speeds of traffic participants are combined to construct object motion relationships. The risk inference layer predicts potential hazards such as sudden lane changes by vehicles ahead, pedestrians crossing the road, and road congestion. Finally, the warning output layer outputs corresponding danger warning information.

[0101] For example, the environmental perception data and the vehicle's motion status data are organized into JSON format, as shown in the following example:

[0102] {

[0103] "Bicycle":

[0104] {

[0105] Speed: 6km / h

[0106] "Heading Angle": "-5 degrees"

[0107] Vehicle length: 4.5m

[0108] "Car frame": "2m"

[0109] },

[0110] "Other traffic participants": [

[0112] {

[0113] Type: Truck

[0114] "Relative position to vehicle": {"x": "6m", "y": "4m"},

[0115] Speed: 10km / h

[0116] "Heading Angle": "-10 degrees"

[0117] Vehicle length: 6m

[0118] Vehicle width: 2m

[0119] },

[0120] {

[0121] Type: Two-wheeled electric vehicle

[0122] "Relative position to vehicle": {"x": "3m", "y": "0.5m"},

[0123] Speed: 5km / h

[0124] "Heading Angle": "15 degrees"

[0125] Vehicle length: 1.5m

[0126] Vehicle width: 1m

[0127] } ]

[0129] };

[0130] The environmental perception data and vehicle motion status data, organized into JSON format, are input into a large model along with the vehicle navigation map. The large model then performs a comprehensive understanding and reasoning of the external driving scenario to obtain hazard warning information for the vehicle.

[0131] For example, after inputting the vehicle navigation map, environmental perception data, and vehicle motion status data into the large model, corresponding prompts can be entered, such as, "As an experienced driver, please determine whether there are potential dangers or situations that require attention based on the vehicle navigation map, environmental perception data, and vehicle motion status data. This may include <insert a preset potential danger> or other possible dangerous situations. Please output the potential danger and the location of concern in JSON format, such as {"potential danger": "pedestrian running a red light", "relative vehicle position": {"x": "9m", "y": "2m"}, "danger level": "moderate"}.

[0132] In this embodiment, by fusing vehicle navigation map with environmental perception data and vehicle motion state data and inputting it into a large model, the powerful scene comprehensive understanding and reasoning capabilities are utilized to accurately identify and generate hazard warning information in the external driving scene of the vehicle. This effectively compensates for the perception blind spots of a single sensor or traditional navigation system, improves the accuracy and real-time performance of hazard warnings in complex road conditions, significantly enhances driving safety, and reduces the risk of accidents.

[0133] In one exemplary embodiment, in the above Figures 2-5 Based on any of the embodiments shown, see Figure 6 The above methods also include:

[0134] S205. Acquire drone perception information around the vehicle.

[0135] Among them, the drone's perception information includes traffic conditions in front of the vehicle, such as whether there is congestion within 5 kilometers in front of the vehicle.

[0136] In this embodiment, to further improve the understanding and reasoning of the large model regarding the driving scene outside the vehicle and make it more in line with user preferences, drone perception information around the vehicle can also be obtained. This allows the large model to make predictions based on the drone perception information and vehicle environment-related data, resulting in more user-friendly information to focus on.

[0137] The aforementioned S202 includes: inputting the drone's perception information and vehicle environment-related data into a large model, and using the large model to comprehensively understand and reason about the driving scenario outside the vehicle to obtain the vehicle's attention information.

[0138] The large model can be pre-trained based on sample vehicle environment data and sample user data to enable it to comprehensively understand and reason about the driving scenario outside the vehicle. This large model can be a Large Language Model (LLM).

[0139] Optionally, a multimodal spatiotemporal fusion model can be used for the large-scale model. This model's overall structure consists of a data preprocessing layer, a cross-source feature encoding layer, a spatiotemporal correlation inference layer, and an information classification output layer. When enabling the large-scale model to understand and infer the driving scenario outside the vehicle, the data preprocessing layer can normalize and format-adapt the UAV-perceived information and vehicle environment-related data. Then, the cross-source feature encoding layer extracts high-altitude global scene features and local environmental features around the vehicle from the preprocessed data and fuses these two types of features. Based on the fused features, the spatiotemporal correlation inference layer identifies safety risks such as vehicle weaving, road congestion, and road surface anomalies. Simultaneously, it combines scene matching with relevant service content. Finally, the information classification output layer generates hazard warnings and scene-recommended vehicle information for attention.

[0140] In this embodiment, after obtaining the drone perception information and vehicle environment-related data, the drone perception information and vehicle environment-related data can be input into a large model for prediction, so as to comprehensively understand and reason about the driving scene outside the vehicle through the large model and obtain the vehicle's attention information.

[0141] For example, navigation map data from drone perception information and vehicle environment-related data can be input into a large model to predict the user's frequently visited locations, so as to recommend navigation maps from the current location to the frequently visited locations, and highlight the congestion status in the navigation route.

[0142] In this embodiment, by introducing UAV perception information around the vehicle and fusing it with the vehicle's local environmental data into a large model, the model leverages its cross-perspective comprehensive understanding and reasoning capabilities to overcome the physical limitations of onboard sensors. This enables the perception of external driving scenarios with a wide range, high angle, and no blind spots, thereby significantly expanding the vehicle's environmental perception range, identifying potential risks in distant or obscured areas in advance, and comprehensively improving situational awareness and driving safety in complex traffic scenarios.

[0143] In one exemplary embodiment, in the above Figures 2-5 Based on any of the embodiments shown, S203 above includes: displaying the information to be concerned on the display interface differently according to the priority of the information to be concerned.

[0144] The priority of the information to be monitored refers to its level of priority. For example, the priority of a traffic jam ahead, which may lead to an accident, is lower than the priority of a pedestrian crossing ahead, which may lead to a pedestrian jaywalking. Optionally, the priority of each piece of information to be monitored can be predetermined and stored in a database so that the priority of the information to be monitored can be directly retrieved from the database.

[0145] Furthermore, the information to be followed can be displayed differently on the interface based on its priority. For example, high-priority information can be highlighted in red, medium-priority information in yellow, and low-priority information in green. As another example, high-priority recommendations can be highlighted in red, medium-priority recommendations in yellow, and low-priority recommendations in green.

[0146] In this embodiment, by differentiating the display of information based on its priority, a hierarchical management of information presentation is achieved, enabling drivers to quickly capture key information, effectively reducing cognitive load and visual interference, improving information acquisition efficiency, and preventing important warnings from being buried by secondary information, thus significantly enhancing the intuitiveness of human-computer interaction and driving safety.

[0147] In one exemplary embodiment, see Figure 7 It also provides an augmented reality head-up display method, including:

[0148] T1. Acquire vehicle environment-related data; vehicle environment-related data includes at least one of the following: environmental perception data, navigation map data, and vehicle motion status data;

[0149] T2. Obtain user-related data; user-related data includes user profiling information and / or scene trigger information;

[0150] T3. Input user-related data and vehicle environment-related data into the large model. The large model will comprehensively understand and reason about the driving scenario outside the vehicle to obtain user-related recommendation information. Among them, the navigation map data includes store information and / or scenic spot information within a preset distance range from the vehicle's current location; the scenario trigger information is the vehicle entering the road for the first time, and / or the current date is a special date for the user.

[0151] T4. Determine the vehicle navigation map based on environmental perception data, navigation map data, and vehicle motion status data;

[0152] T5. Input the vehicle navigation map, environmental perception data and vehicle motion status data into the big model. The big model will then comprehensively understand and reason about the driving scenario outside the vehicle to obtain the vehicle's hazard warning information.

[0153] T6. Based on the priority of the information to be concerned, display the information to be concerned on the display interface in a differentiated manner; the information to be concerned includes vehicle hazard warning information and user-related recommendation information.

[0154] It should be noted that the descriptions of T1-T5 above can be found in the relevant descriptions in the above embodiments, and their effects are similar, so they will not be repeated here.

[0155] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0156] Based on the same inventive concept, this application also provides an augmented reality head-up display device for implementing the augmented reality head-up display method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more embodiments of the augmented reality head-up display device provided below can be found in the limitations of the augmented reality head-up display method described above, and will not be repeated here.

[0157] In one exemplary embodiment, such as Figure 8 As shown, an augmented reality head-up display device is provided, including: an acquisition module 10, an inference module 11, and a display module 12, wherein:

[0158] The acquisition module 10 is used to acquire vehicle environment-related data; the vehicle environment-related data includes at least one of the following: environmental perception data, navigation map data, and vehicle motion status data.

[0159] The reasoning module 11 is used to input vehicle environment-related data into the large model, and through the large model, to comprehensively understand and reason about the driving scene outside the vehicle to obtain the vehicle's attention information.

[0160] Display module 12 is used to display the information to be viewed on the display interface of the in-vehicle augmented reality head-up display.

[0161] In one exemplary embodiment, the above-described apparatus further includes:

[0162] The data acquisition module is used to acquire user-related data, including user profiling information and / or scene triggering information.

[0163] The aforementioned reasoning module 11 is also used to input user-related data and vehicle environment-related data into the large model, and to comprehensively understand and reason about the driving scenario outside the vehicle through the large model to obtain the vehicle's attention information.

[0164] In an exemplary embodiment, the inference module 11 is further configured to input user-related data and vehicle environment-related data into a large model, and to comprehensively understand and infer the driving scenario outside the vehicle through the large model to obtain user-related recommendation information; wherein, the navigation map data includes store information and / or scenic spot information within a preset distance range from the vehicle's current location; the scenario trigger information is the vehicle entering the first driving road, and / or the current date is a special date for the user.

[0165] In an exemplary embodiment, the inference module 11 described above includes:

[0166] The determining unit is specifically used to determine the vehicle navigation map based on environmental perception data, navigation map data, and vehicle motion status data;

[0167] The reasoning unit is specifically used to input vehicle navigation map, environmental perception data and vehicle motion state data into the large model. Through the large model, the vehicle's external driving scenario is comprehensively understood and reasoned to obtain the vehicle's danger warning information.

[0168] In an exemplary embodiment, the environmental perception data mentioned above includes: state information of multiple traffic participants; the state information includes at least one of the following: type, size, position, speed, and direction of the traffic participants;

[0169] The navigation map data includes information about the surrounding roads outside the vehicle, including: information about the intersections in front of the vehicle, and / or, information about the lane markings where the vehicle is located.

[0170] Vehicle motion status data includes at least one of the vehicle's current position, direction of travel, and speed.

[0171] In one exemplary embodiment, the above-described apparatus further includes:

[0172] The information acquisition module is used to acquire drone perception information around the vehicle;

[0173] The aforementioned reasoning module 11 is also used to input the drone's perception information and vehicle environment-related data into the large model, and to comprehensively understand and reason about the driving scene outside the vehicle through the large model to obtain the vehicle's attention information.

[0174] In an exemplary embodiment, the display module 12 is further configured to differentiate the information to be concerned on the display interface according to the priority of the information to be concerned.

[0175] The modules in the aforementioned augmented reality head-up display device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0176] In one exemplary embodiment, a vehicle controller is provided, which may be a server, and its internal structure diagram may be as follows. Figure 9 As shown, the vehicle controller includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media. The database stores vehicle environment-related data. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an augmented reality head-up display method.

[0177] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0178] In one exemplary embodiment, a vehicle controller is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0179] Acquire vehicle environment-related data; vehicle environment-related data includes at least one of the following: environmental perception data, navigation map data, and vehicle motion status data;

[0180] Vehicle environment-related data is input into a large model, which then performs a comprehensive understanding and reasoning of the driving scenario outside the vehicle to obtain the vehicle's attention-grabbing information.

[0181] The information to be noted is displayed on the in-vehicle augmented reality head-up display interface.

[0182] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0183] Acquire user-related data; user-related data includes user profiling information and / or scenario triggering information;

[0184] Vehicle environment-related data is input into a large model, which then performs a comprehensive understanding and inference of the external driving scenario to obtain the vehicle's key information, including:

[0185] User-related data and vehicle-related environmental data are input into a large model. The large model then performs a comprehensive understanding and reasoning of the driving scenarios outside the vehicle to obtain the vehicle's key information.

[0186] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0187] User-related data and vehicle environment-related data are input into a large model. The large model then comprehensively understands and infers the driving scenario outside the vehicle to obtain user-related recommendation information.

[0188] The navigation map data includes information on shops and / or attractions located within a preset distance from the vehicle's current location; the scene trigger information is the vehicle entering the first driving road, and / or the current date is a special date specified by the user.

[0189] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0190] The vehicle navigation map is determined based on environmental perception data, navigation map data, and vehicle motion status data;

[0191] The vehicle navigation map, environmental perception data, and vehicle motion status data are input into the large model. The large model then performs a comprehensive understanding and reasoning of the driving scenario outside the vehicle to obtain hazard warning information for the vehicle.

[0192] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0193] Acquire drone perception information around the vehicle;

[0194] Vehicle environment-related data is input into a large model, which then performs a comprehensive understanding and inference of the external driving scenario to obtain the vehicle's key information, including:

[0195] The drone's perception information and vehicle environment-related data are input into a large model. The large model then performs a comprehensive understanding and reasoning of the driving scenario outside the vehicle to obtain the vehicle's attention-grabbing information.

[0196] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0197] Based on the priority of the information to be monitored, the information to be monitored will be displayed differently on the display interface.

[0198] In one embodiment, a vehicle is provided, including the aforementioned vehicle controller.

[0199] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0200] Acquire vehicle environment-related data; vehicle environment-related data includes at least one of the following: environmental perception data, navigation map data, and vehicle motion status data;

[0201] Vehicle environment-related data is input into a large model, which then performs a comprehensive understanding and reasoning of the driving scenario outside the vehicle to obtain the vehicle's attention-grabbing information.

[0202] The information to be noted is displayed on the in-vehicle augmented reality head-up display interface.

[0203] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0204] Acquire user-related data; user-related data includes user profiling information and / or scenario triggering information;

[0205] Vehicle environment-related data is input into a large model, which then performs a comprehensive understanding and inference of the external driving scenario to obtain the vehicle's key information, including:

[0206] User-related data and vehicle-related environmental data are input into a large model. The large model then performs a comprehensive understanding and reasoning of the driving scenarios outside the vehicle to obtain the vehicle's key information.

[0207] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0208] User-related data and vehicle environment-related data are input into a large model. The large model then comprehensively understands and infers the driving scenario outside the vehicle to obtain user-related recommendation information.

[0209] The navigation map data includes information on shops and / or attractions located within a preset distance from the vehicle's current location; the scene trigger information is the vehicle entering the first driving road, and / or the current date is a special date specified by the user.

[0210] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0211] The vehicle navigation map is determined based on environmental perception data, navigation map data, and vehicle motion status data;

[0212] The vehicle navigation map, environmental perception data, and vehicle motion status data are input into the large model. The large model then performs a comprehensive understanding and reasoning of the driving scenario outside the vehicle to obtain hazard warning information for the vehicle.

[0213] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0214] Acquire drone perception information around the vehicle;

[0215] Vehicle environment-related data is input into a large model, which then performs a comprehensive understanding and inference of the external driving scenario to obtain the vehicle's key information, including:

[0216] The drone's perception information and vehicle environment-related data are input into a large model. The large model then performs a comprehensive understanding and reasoning of the driving scenario outside the vehicle to obtain the vehicle's attention-grabbing information.

[0217] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0218] Based on the priority of the information to be monitored, the information to be monitored will be displayed differently on the display interface.

[0219] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0220] Acquire vehicle environment-related data; vehicle environment-related data includes at least one of the following: environmental perception data, navigation map data, and vehicle motion status data;

[0221] Vehicle environment-related data is input into a large model, which then performs a comprehensive understanding and reasoning of the driving scenario outside the vehicle to obtain the vehicle's attention-grabbing information.

[0222] The information to be noted is displayed on the in-vehicle augmented reality head-up display interface.

[0223] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0224] Acquire user-related data; user-related data includes user profiling information and / or scenario triggering information;

[0225] Vehicle environment-related data is input into a large model, which then performs a comprehensive understanding and inference of the external driving scenario to obtain the vehicle's key information, including:

[0226] User-related data and vehicle-related environmental data are input into a large model. The large model then performs a comprehensive understanding and reasoning of the driving scenarios outside the vehicle to obtain the vehicle's key information.

[0227] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0228] User-related data and vehicle environment-related data are input into a large model. The large model then comprehensively understands and infers the driving scenario outside the vehicle to obtain user-related recommendation information.

[0229] The navigation map data includes information on shops and / or attractions located within a preset distance from the vehicle's current location; the scene trigger information is the vehicle entering the first driving road, and / or the current date is a special date specified by the user.

[0230] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0231] The vehicle navigation map is determined based on environmental perception data, navigation map data, and vehicle motion status data;

[0232] The vehicle navigation map, environmental perception data, and vehicle motion status data are input into the large model. The large model then performs a comprehensive understanding and reasoning of the driving scenario outside the vehicle to obtain hazard warning information for the vehicle.

[0233] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0234] Acquire drone perception information around the vehicle;

[0235] Vehicle environment-related data is input into a large model, which then performs a comprehensive understanding and inference of the external driving scenario to obtain the vehicle's key information, including:

[0236] The drone's perception information and vehicle environment-related data are input into a large model. The large model then performs a comprehensive understanding and reasoning of the driving scenario outside the vehicle to obtain the vehicle's attention-grabbing information.

[0237] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0238] Based on the priority of the information to be monitored, the information to be monitored will be displayed differently on the display interface.

[0239] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0240] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0241] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0242] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An augmented reality head-up display method, characterized in that, The method includes: Acquire vehicle environment-related data; the vehicle environment-related data includes at least one of: environmental perception data, navigation map data, and vehicle motion status data; The vehicle environment-related data is input into a large model, which then performs a comprehensive understanding and reasoning of the driving scenario outside the vehicle to obtain the vehicle's information of interest. The information to be noted is displayed on the in-vehicle augmented reality head-up display interface.

2. The method according to claim 1, characterized in that, The method further includes: Acquire user-related data; the user-related data includes user profiling information and / or scene triggering information; The process involves inputting the vehicle environment-related data into a large model, which then performs a comprehensive understanding and reasoning of the external driving scenario to obtain the vehicle's information of interest, including: The user-related data and the vehicle environment-related data are input into a large model. The large model then performs a comprehensive understanding and reasoning of the driving scenario outside the vehicle to obtain the vehicle's attention-grabbing information.

3. The method according to claim 2, characterized in that, The information to be noted includes user-related recommendation information; the process of inputting the user-related data and the vehicle environment-related data into a large model, and using the large model to comprehensively understand and reason about the driving scenario outside the vehicle to obtain the vehicle's information to be noted includes: The user-related data and the vehicle environment-related data are input into the large model. The large model is then used to comprehensively understand and reason about the driving scenario outside the vehicle to obtain the user-related recommendation information. The navigation map data includes information on shops and / or attractions located within a preset distance range from the vehicle's current location; the scenario trigger information is the vehicle entering the first driving road, and / or the current date being a special date specified by the user.

4. The method according to claim 1, characterized in that, The information to be concerned also includes vehicle hazard warning information; the process of inputting the vehicle environment-related data into a large model, and using the large model to comprehensively understand and reason about the driving scenario outside the vehicle to obtain the vehicle's information to be concerned, includes: The vehicle navigation map is determined based on the environmental perception data, the navigation map data, and the vehicle motion status data. The vehicle navigation map, the environmental perception data, and the vehicle motion state data are input into a large model. The large model then performs a comprehensive understanding and reasoning of the driving scenario outside the vehicle to obtain the vehicle's hazard warning information.

5. The method according to claim 4, characterized in that, The environmental perception data includes: status information of multiple traffic participants; the status information includes at least one of the traffic participants' type, size, location, speed, and direction; The navigation map data includes information about the surrounding roads outside the vehicle, including: information about intersections in front of the vehicle, and / or, information about the lane lines where the vehicle is located. The vehicle motion status data includes at least one of the vehicle's current position, driving direction, and driving speed.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: Acquire drone perception information around the vehicle; The process involves inputting the vehicle environment-related data into a large model, which then performs a comprehensive understanding and reasoning of the external driving scenario to obtain the vehicle's information of interest, including: The UAV's perception information and the vehicle's environmental data are input into a large model. The large model then performs a comprehensive understanding and reasoning of the driving scenario outside the vehicle to obtain the vehicle's information of interest.

7. The method according to any one of claims 1-5, characterized in that, The step of displaying the information to be noted on the display interface of the in-vehicle augmented reality head-up display includes: Based on the priority of the information to be monitored, the information to be monitored is displayed differently on the display interface.

8. An augmented reality head-up display device, characterized in that, The device includes: The acquisition module is used to acquire vehicle environment-related data; the vehicle environment-related data includes at least one of environmental perception data, navigation map data, and vehicle motion status data. The reasoning module is used to input the vehicle environment-related data into the large model, and through the large model, to comprehensively understand and reason about the driving scene outside the vehicle to obtain the vehicle's attention information; The display module is used to display the information to be concerned on the display interface of the in-vehicle augmented reality head-up display.

9. A vehicle controller, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A vehicle having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images