Head-up display method of vehicle and computer equipment

Abstract

The invention provides a head-up display method of a vehicle and computer equipment. The method comprises the following steps: acquiring image data and map navigation information of vehicle driving in real time; a predicted road attribute and a predicted weather attribute of vehicle driving are determined by a first network model obtained through pre-training according to the preset frame number of image data; determining a road lighting value according to the preset frame number of image data, and determining whether to switch a current projection mode of head-up display according to the predicted road attribute, the predicted weather attribute, the road lighting value and map navigation information; if yes, determining a target projection mode according to the predicted road attribute, the predicted weather attribute and the road lighting value, and switching the current projection mode into the target projection mode; and verifying and correcting the first network model by a second network model obtained by pre-training according to the preset frame number of image data so as to improve the prediction precision of the first network model and the accuracy of projection mode switching.

Description

Technical Field

[0001] The present application relates to the field of automobile technology, and in particular to a head-up display method and computer equipment for a vehicle. Background Art

[0002] In the automotive field, the Augmented Reality Head Up Display (AR-HUD) as a driving assistance system can project navigation, speed and other information during the vehicle's driving process directly into the driver's field of vision. The driver does not need to lower his head to check the dashboard, which reduces the frequency of the driver lowering his head and reduces the risk of "blind driving", thereby improving driving safety.

[0003] In different driving environments, the projection background keeps changing, which makes the projection effect of AR-HUD stand out in the background in some environments. At present, the projection mode of AR-HUD needs to be manually adjusted by the driver in different external environments. For example, the driver needs to manually adjust the brightness and display effect of AR-HUD according to different weather, time periods and road conditions, which may distract the driver's attention, increase driving risks and affect the driving safety of the vehicle.

[0004] Therefore, the manual switching of the projection mode of AR-HUD in different driving environments in the prior art has certain limitations. Summary of the invention

[0005] The purpose of this application is to provide a head-up display method and computer device for a vehicle in view of the above-mentioned deficiencies in the prior art, so as to solve the practical problem of low driving safety when manually switching the display mode of AR-HUD under different driving environments in the prior art.

[0006] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0007] In a first aspect, an embodiment of the present application provides a head-up display method for a vehicle, the method comprising:

[0008] Acquire the image data of the vehicle and the vehicle system data in real time; the vehicle system data at least includes map navigation information;

[0009] The first network model obtained by pre-training determines the predicted road attributes and the predicted weather attributes of the vehicle according to the image data of a preset number of frames;

[0010] Determine a road lighting value according to the preset number of frames of image data, and determine whether to switch a current projection mode of a head-up display according to the predicted road attributes, the predicted weather attributes, the road lighting value, and the map navigation information;

[0011] If yes, determining a target projection mode according to the predicted road attributes, the predicted weather attributes and the road lighting value, and switching the current projection mode to the target projection mode;

[0012] The second network model obtained by pre-training verifies and corrects the first network model according to the image data of the preset number of frames, wherein the prediction accuracy of the second network model is higher than the prediction accuracy of the first network model, and the prediction efficiency of the second network model is lower than the prediction efficiency of the first network model.

[0013] As an optional implementation, the training process of the first network model includes:

[0014] Pre-training the pre-built back-propagation neural network model to obtain a target back-propagation neural network;

[0015] The pre-trained large model is used as the teacher model, and the target back propagation neural network model is used as the student model. Through knowledge distillation, the student model is trained so that the student model learns the data distribution of the teacher model, and the student model at the end of the training is used as the first network model.

[0016] As an optional implementation, the first network model obtained by pre-training determines the predicted road attributes and predicted weather attributes of the vehicle according to the image data of a preset number of frames, including:

[0017] The image data of the preset number of frames are sequentially input into the first network model in a forward direction to obtain a predicted probability distribution of each frame of image data, wherein the predicted probability distribution includes a predicted probability distribution of each frame of image data in each road attribute and a predicted probability distribution of each weather attribute;

[0018] Determining predicted road attributes for vehicle travel according to predicted probability distribution of each frame of image data at each road attribute;

[0019] The predicted weather attribute for vehicle travel is determined based on the predicted probability distribution of each weather attribute for each frame of image data.

[0020] As an optional implementation, determining whether to switch the current projection mode of the head-up display according to the predicted road attribute, the predicted weather attribute, the road lighting value, and the map navigation information includes:

[0021] Performing data synchronization on the predicted road attributes, the predicted weather attributes, the road lighting value, and the map navigation information to obtain target map navigation information;

[0022] If it is determined according to the target map navigation information that the vehicle is currently traveling in a non-specific area, determining whether the road lighting value matches the predicted weather attribute;

[0023] If yes, determining the target projection brightness type according to the road lighting value and the lighting thresholds corresponding to each brightness type;

[0024] Determine whether to switch the current projection mode of the head-up display according to the target projection brightness type, the predicted road attributes, the predicted weather attributes, and the projection brightness type, road attributes, and weather attributes in the current projection mode.

[0025] As an optional implementation, the step of synchronizing the predicted road attributes, the predicted weather attributes, the road lighting value, and the map navigation information to obtain target map navigation information includes:

[0026] Determining the time interval corresponding to the predicted road attribute, the predicted weather attribute, and the road lighting value according to the acquisition time of the image data of the preset number of frames;

[0027] The map navigation information corresponding to the time interval is used as the target map navigation information.

[0028] As an optional implementation manner, determining whether the road lighting value matches the predicted weather attribute includes:

[0029] Obtaining the current time and determining the current time period to which the current time belongs;

[0030] Obtaining the lighting threshold corresponding to the predicted weather attribute in the current time period;

[0031] The road lighting value is compared with the lighting threshold, and whether the road lighting value matches the predicted weather attribute is determined according to the comparison result.

[0032] As an optional implementation manner, determining whether to switch the current projection mode of the head-up display according to the target projection brightness type, the predicted road attribute, the predicted weather attribute, and the projection brightness type, road attribute, and weather attribute in the current projection mode includes:

[0033] If the target projection brightness type is consistent with the projection brightness type in the current projection mode, the predicted road attribute is consistent with the road attribute in the current projection mode, and the predicted weather attribute is consistent with the weather attribute in the current projection mode, then determining not to switch the current projection mode of the head-up display;

[0034] Otherwise, determine to switch the current projection mode of the head-up display.

[0035] As an optional implementation, the second network model obtained by pre-training verifies and corrects the first network model according to the image data of the preset number of frames, including:

[0036] Determining, by the second network model, a target road attribute and a target weather attribute for vehicle travel according to the image data of the preset number of frames;

[0037] Determining a road score according to the target road attribute and the predicted road attribute;

[0038] Determining a weather score according to the target weather attribute and the predicted weather attribute;

[0039] Determining a target score according to the road score, the weather score, a preset road weight, and a preset weather weight;

[0040] The first network model is verified and corrected according to the target score and the preset value.

[0041] As an optional implementation, the verifying and correcting the first network model according to the target score and the preset value includes:

[0042] If the target score is less than the preset value, the image data of the preset number of frames are labeled according to the prediction results of the second network model for the image data of the preset number of frames, and the labeled image data of the preset number of frames are added to the training database of the first network model to continue training the first network model according to the training database.

[0043] In a second aspect, an embodiment of the present application provides a head-up display device for a vehicle, the device comprising:

[0044] An acquisition module, used to acquire the image data of the vehicle and the vehicle system data in real time; the vehicle system data at least includes map navigation information;

[0045] A determination module, configured to determine predicted road attributes and predicted weather attributes of a vehicle according to a preset number of frames of image data using a pre-trained first network model;

[0046] The determination module is further used to determine the road lighting value according to the image data of the preset number of frames, and determine whether to switch the current projection mode of the head-up display according to the predicted road attributes, the predicted weather attributes, the road lighting value and the map navigation information;

[0047] The determination module is further configured to determine a target projection mode according to the predicted road attributes, the predicted weather attributes and the road lighting value, and switch the current projection mode to the target projection mode;

[0048] A correction module is used to verify and correct the first network model according to the image data of the preset number of frames using a pre-trained second network model, wherein the prediction accuracy of the second network model is higher than the prediction accuracy of the first network model, and the prediction efficiency of the second network model is lower than the prediction efficiency of the first network model.

[0049] As an optional implementation manner, the determining module is specifically used for:

[0050] Pre-training the pre-built back-propagation neural network model to obtain a target back-propagation neural network;

[0051] The pre-trained large model is used as the teacher model, and the target back propagation neural network model is used as the student model. Through knowledge distillation, the student model is trained so that the student model learns the data distribution of the teacher model, and the student model at the end of the training is used as the first network model.

[0052] As an optional implementation manner, the determining module is specifically used for:

[0053] The image data of the preset number of frames are sequentially input into the first network model in a forward direction to obtain a predicted probability distribution of each frame of image data, wherein the predicted probability distribution includes a predicted probability distribution of each frame of image data in each road attribute and a predicted probability distribution of each weather attribute;

[0054] Determining predicted road attributes for vehicle travel according to predicted probability distribution of each frame of image data at each road attribute;

[0055] The predicted weather attribute for vehicle travel is determined based on the predicted probability distribution of each weather attribute for each frame of image data.

[0056] As an optional implementation manner, the determining module is specifically used for:

[0057] Performing data synchronization on the predicted road attributes, the predicted weather attributes, the road lighting value, and the map navigation information to obtain target map navigation information;

[0058] If it is determined according to the target map navigation information that the vehicle is currently traveling in a non-specific area, determining whether the road lighting value matches the predicted weather attribute;

[0059] If yes, determining the target projection brightness type according to the road lighting value and the lighting thresholds corresponding to each brightness type;

[0060] Determine whether to switch the current projection mode of the head-up display according to the target projection brightness type, the predicted road attributes, the predicted weather attributes, and the projection brightness type, road attributes, and weather attributes in the current projection mode.

[0061] As an optional implementation manner, the determining module is specifically used for:

[0062] Determining the time interval corresponding to the predicted road attribute, the predicted weather attribute, and the road lighting value according to the acquisition time of the image data of the preset number of frames;

[0063] The map navigation information corresponding to the time interval is used as the target map navigation information.

[0064] As an optional implementation manner, the determining module is specifically used for:

[0065] Obtaining the current time and determining the current time period to which the current time belongs;

[0066] Obtaining the lighting threshold corresponding to the predicted weather attribute in the current time period;

[0067] The road lighting value is compared with the lighting threshold, and whether the road lighting value matches the predicted weather attribute is determined according to the comparison result.

[0068] As an optional implementation manner, the determining module is specifically used for:

[0069] If the target projection brightness type is consistent with the projection brightness type in the current projection mode, the predicted road attribute is consistent with the road attribute in the current projection mode, and the predicted weather attribute is consistent with the weather attribute in the current projection mode, then determining not to switch the current projection mode of the head-up display;

[0070] Otherwise, it is determined to switch the current projection mode of the head-up display.

[0071] As an optional implementation, the correction module is specifically used for:

[0072] Determining, by the second network model, a target road attribute and a target weather attribute for vehicle travel according to the image data of the preset number of frames;

[0073] Determining a road score according to the target road attribute and the predicted road attribute;

[0074] Determining a weather score according to the target weather attribute and the predicted weather attribute;

[0075] Determining a target score according to the road score, the weather score, a preset road weight, and a preset weather weight;

[0076] The first network model is verified and corrected according to the target score and the preset value.

[0077] As an optional implementation, the correction module is specifically used for:

[0078] If the target score is less than the preset value, the image data of the preset number of frames are labeled according to the prediction results of the second network model for the image data of the preset number of frames, and the labeled image data of the preset number of frames are added to the training database of the first network model to continue training the first network model according to the training database.

[0079] In a third aspect, an embodiment of the present application provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the vehicle head-up display method as described in the first aspect above.

[0080] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the vehicle head-up display method as described in the first aspect above are executed.

[0081] The beneficial effects of this application are:

[0082] The present application provides a head-up display method and computer device for a vehicle, which obtains the image data of the vehicle driving and the subway navigation information in the vehicle system data in real time, and inputs the image data into a pre-trained first network model as a group according to a preset number of frames. Since the structure of the first network model is relatively simple, the first network model performs real-time road attribute prediction and weather attribute prediction on a group of image data of the preset number of frames, and can efficiently determine the predicted road attribute and predicted weather attribute of the vehicle driving. According to the image data of the preset number of frames, the road lighting value representing the lighting degree when the vehicle is driving is calculated. Combined with the predicted road attribute, predicted weather attribute, road lighting value and map navigation information, it is judged whether the three environmental factors of the road, weather or lighting degree when the vehicle is currently driving have changed significantly, so as to determine whether to switch the current projection mode of the head-up display. If so, the target projection mode that best matches the road, weather and lighting degree when the vehicle is currently driving is determined in combination with the road lighting value and the predicted road attribute and predicted weather attribute output by the first network model, and the current projection mode is automatically and quickly switched to the target projection mode to adapt to the obvious changes in environmental factors, avoid abrupt projection effects, and avoid the driving safety hazards caused by the driver manually switching the projection mode. A second network model with a more complex structure than the first network model is obtained in advance through model training. Due to the more complex structure of the second network model, the prediction accuracy of the second network model is higher than that of the first network model and the prediction efficiency of the second network model is lower than that of the first network model. The first network model is verified by the accurate prediction results output by the second network model for each group of preset frame number of image data, and the model parameters of the first network model are corrected according to the image data, so as to continuously improve the prediction accuracy of the first network model, and the advantages of high prediction efficiency of the first network model and high prediction accuracy of the second network model are combined to ensure that the first network model accurately and efficiently outputs predicted road attributes and predicted weather attributes during driving, thereby ensuring the accuracy of switching of projection modes. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0084] Figure 1 A schematic diagram of a process flow of a head-up display method for a vehicle provided in an embodiment of the present application;

[0085] Figure 2 A schematic diagram of a process for training a first network model of a head-up display method for a vehicle provided in an embodiment of the present application;

[0086] Figure 3 A schematic diagram of an interface for training a first network model provided in an embodiment of the present application;

[0087] Figure 4 A schematic diagram of a process of determining predicted road attributes and predicted weather attributes for vehicle travel in a head-up display method for a vehicle provided in an embodiment of the present application;

[0088] Figure 5 A schematic diagram of a flow chart of determining whether to switch the current projection mode of the head-up display of a vehicle provided in an embodiment of the present application;

[0089] Figure 6 A schematic diagram of a data synchronization process of a head-up display method for a vehicle provided in an embodiment of the present application;

[0090] Figure 7 A schematic diagram of a flow chart of determining whether a road lighting value matches a predicted weather attribute in a head-up display method for a vehicle provided in an embodiment of the present application;

[0091] Figure 8 Another flowchart of the head-up display method for a vehicle provided in an embodiment of the present application for determining whether to switch the current projection mode of the head-up display;

[0092] Fig. 9 A schematic diagram of a process of verifying and correcting a first network model of a head-up display method for a vehicle provided in an embodiment of the present application;

[0093] Fig.10 A module structure diagram of a head-up display device for a vehicle provided in an embodiment of the present application;

[0094] Fig.11 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0095] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0096] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0097] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0098] In the automotive field, AR-HUD projects navigation, speed and other information during vehicle driving directly to the front of the driver's field of vision, reducing the frequency of the driver lowering his head and the risk of "blind driving", thereby improving driving safety. At present, the driver needs to manually switch the projection mode of AR-HUD according to the external environment such as different weather and road conditions and changes in different time periods, which can easily distract the driver's attention, increase driving risks, and affect the driving safety of the vehicle.

[0099] Based on the above problems, the embodiment of the present application proposes a head-up display method for a vehicle, which obtains image data of the vehicle driving by photographing the road environment in real time, and uses a first network model to predict weather attributes and road attributes of the image data according to a deep learning classification algorithm, and calculates the road lighting value of the image data in combination with an image statistical algorithm. Based on the road lighting value and the predicted weather attributes and predicted road attributes predicted by the first network model, the optimal projection mode is determined and the current projection mode is automatically switched, without the need for the driver to switch manually, thereby improving the driving safety of switching the projection mode. The first network model is verified and corrected by a second network model with higher prediction accuracy, which continuously improves the prediction accuracy of the first network model, thereby improving the accuracy of the projection mode switching. Since the prediction efficiency of the first network model is higher than that of the second network model, the efficiency of the projection mode switching is also improved.

[0100] Figure 1 The flowchart of the head-up display method for a vehicle provided in the embodiment of the present application is shown in FIG. 1 , and the execution subject of the method can be any computer device with computing processing capabilities. Figure 1 As shown, the method includes:

[0101] S101. Acquire vehicle driving image data and vehicle system data in real time; the vehicle system data at least includes map navigation information.

[0102] Optionally, an exterior perception camera is installed at the front of the vehicle. In the Advanced Driving Assistance System (ADAS) installed in the vehicle, the exterior perception camera turns on the Digital Video Recorder (DVR) function, which can capture the video stream of the vehicle in real time, that is, the image data of the road the vehicle is traveling on, including the road information and weather information when the vehicle is traveling. Among them, there can be one or more exterior perception cameras installed at the front of the vehicle, and there is no specific restriction on the number of exterior perception cameras.

[0103] The computer device communicates with the ADAS's off-vehicle perception camera in real time to capture the image data of the vehicle's driving from the video stream during the vehicle's driving process. Since the image data is captured from the video stream during the vehicle's driving process, that is, the image data of the road the vehicle is driving on, the image data also contains the road information and weather information when the vehicle is driving. The computer device also communicates with the vehicle system in real time to obtain map navigation information, vehicle speed and other vehicle system data in real time, wherein the map navigation information is used to represent the latitude and longitude of the vehicle's current driving and the name of the road.

[0104] S102, determining predicted road attributes and predicted weather attributes of vehicle travel according to a preset number of frames of image data using a pre-trained first network model.

[0105] Optionally, the first network model is obtained in advance through model training, and the structure of the first network model is relatively simple, so that the prediction efficiency of the first network model is relatively fast. The real-time intercepted image data is input into the first network model as a group of preset frames, and the first network model uses a deep learning classification algorithm to perform real-time road attribute prediction and weather attribute prediction on a group of image data of the preset frame number, and efficiently determine the predicted road attributes and predicted weather attributes of the vehicle. Among them, the preset number of frames can be 20 frames, and a predicted road attribute and a predicted weather attribute are output respectively with 20 frames of image data as a group. There is no specific restriction on the value of the preset number of frames.

[0106] Exemplarily, road attributes may include urban roads, rural roads, mountains, snowy areas, and deserts, and weather attributes may include sunny days, cloudy days, rainy days, snowy days, and foggy days. The subdivision categories of road attributes and weather attributes are not specifically limited herein. For example, the first network model uses a deep learning classification algorithm to classify and predict the road information and weather information contained in a set of image data of a preset number of frames, and efficiently judges and outputs the predicted road attribute of the vehicle as an urban road and the predicted weather attribute as a rainy day.

[0107] S103, determining a road lighting value based on image data of a preset number of frames, and determining whether to switch a current projection mode of the head-up display based on predicted road attributes, predicted weather attributes, road lighting values, and map navigation information.

[0108] Optionally, for the image data of a preset number of frames, the brightness histogram and color histogram of each frame of the image are calculated frame by frame, the brightness features and color features of each frame of the image are extracted, and the brightness features and color features of each frame of the image are filtered using a filtering algorithm to calculate the road lighting value corresponding to the image data of the preset number of frames. The road lighting value can represent the lighting when the vehicle is driving. Among them, the brightness histogram represents the distribution of light and darkness of the image, and the color histogram represents the frequency or quantity of various colors in the image, that is, it represents the distribution of different colors in the image. It is worth noting that the road lighting value can also be directly measured by a high-precision illuminance meter.

[0109] Combined with the predicted road attributes, predicted weather attributes, road lighting values, and map navigation information obtained in real time, it is determined whether the three environmental factors of the road, weather, or lighting when the vehicle is currently traveling have changed significantly, and whether to switch the current projection mode of the AR-HUD is determined based on the judgment result. Specifically, if any environmental factor of the road, weather, or lighting when the vehicle is currently traveling has changed significantly, it means that the current projection mode of the AR-HUD is no longer suitable for the current driving environment of the vehicle, and it is determined that the current projection mode of the AR-HUD needs to be switched.

[0110] S104: If yes, determine the target projection mode according to the predicted road attributes, predicted weather attributes and road lighting values, and switch the current projection mode to the target projection mode.

[0111] Optionally, if it is determined that the current projection mode of the AR-HUD needs to be switched, the road lighting value and the predicted road attributes and predicted weather attributes output by the first network model are combined to determine the optimal matching mode that matches the road, weather and lighting conditions of the vehicle at the time of current travel as the target projection mode.

[0112] The computer device quickly switches the current projection mode of the AR-HUD to the target projection mode to adapt to the obvious changes in environmental factors such as the road, weather or lighting when the vehicle is currently driving, avoid abrupt projection effects, and avoid the driver manually switching the projection mode of the AR-HUD, so that the driver can focus more on vehicle driving and improve driving safety.

[0113] S105. The pre-trained second network model verifies and corrects the first network model according to a preset number of frames of image data, wherein the prediction accuracy of the second network model is higher than the prediction accuracy of the first network model, and the prediction efficiency of the second network model is lower than the prediction efficiency of the first network model.

[0114] Optionally, after the current projection mode is switched to the target projection mode, during the driving of the vehicle, the first network model continues to output the predicted road attributes and weather prediction attributes of the vehicle according to the next set of preset frame number images.

[0115] The second network model is obtained in advance through model training. The structure of the second network model is more complex than that of the first network model, so that the prediction accuracy of the second network model is higher than that of the first network model, but at the same time, the prediction efficiency of the second network model is lower than that of the first network model. Although the second network model can accurately output prediction results that are more in line with actual road conditions and actual weather conditions, the prediction efficiency of the second network model is low and it is impossible to output prediction results in real time during vehicle driving. The first network model is verified by the accurate prediction results output by the second network model for image data, and the first network model is corrected according to the image data, so that the prediction accuracy of the first network model can be continuously improved, ensuring that the first network model can accurately and efficiently output predicted road attributes and predicted weather attributes during driving.

[0116] Specifically, in an offline environment where the vehicle stops driving, all groups of image data of a preset number of frames predicted by the first network model are sequentially input into the second network model, and the second network model uses a deep learning classification algorithm to predict the road attributes and weather attributes of each group of image data of the preset number of frames, and compares them one by one with each predicted road attribute and each weather road attribute output by the first network model to verify whether the prediction accuracy of the first network model for each group of image data of the preset number of frames meets the standard. If it does not meet the standard, the model parameters in the first network model are corrected in an offline environment where the vehicle stops driving according to each group of image data of the preset number of frames whose prediction accuracy does not meet the standard, so as to improve the prediction accuracy of the first network model, so that the predicted road attributes and predicted weather attributes of the vehicle driving output by the first network model during the next vehicle driving process are more accurate and more in line with the actual road conditions and actual weather conditions.

[0117] In this embodiment, the image data of the vehicle driving and the subway navigation information in the vehicle system data are obtained in real time, and the image data is input into the pre-trained first network model as a group according to the preset number of frames. Since the structure of the first network model is relatively simple, the first network model performs real-time road attribute prediction and weather attribute prediction on a group of image data of the preset number of frames, and the predicted road attribute and predicted weather attribute of the vehicle driving can be efficiently determined. According to the image data of the preset number of frames, the road lighting value representing the lighting degree when the vehicle is driving is calculated. Combined with the predicted road attribute, predicted weather attribute, road lighting value and map navigation information, it is judged whether the three environmental factors of the road, weather or lighting degree when the vehicle is currently driving have changed significantly, so as to determine whether to switch the current projection mode of the head-up display. If so, the target projection mode that best matches the road, weather and lighting degree when the vehicle is currently driving is determined in combination with the road lighting value and the predicted road attribute and predicted weather attribute output by the first network model, and the current projection mode is automatically and quickly switched to the target projection mode to adapt to the obvious changes in environmental factors, avoid abrupt projection effects, and avoid driving safety hazards caused by the driver manually switching the projection mode. A second network model with a more complex structure than the first network model is obtained in advance through model training. Due to the more complex structure of the second network model, the prediction accuracy of the second network model is higher than that of the first network model and the prediction efficiency of the second network model is lower than that of the first network model. The first network model is verified by the accurate prediction results output by the second network model for each group of preset frame number of image data, and the model parameters of the first network model are corrected according to the image data, so as to continuously improve the prediction accuracy of the first network model, and the advantages of high prediction efficiency of the first network model and high prediction accuracy of the second network model are combined to ensure that the first network model accurately and efficiently outputs predicted road attributes and predicted weather attributes during driving, thereby ensuring the accuracy of switching of projection modes.

[0118] The following is a detailed description of the training process of the first network model.

[0119] Figure 2 A schematic diagram of a process flow of training a first network model for a head-up display method for a vehicle provided in an embodiment of the present application, such as Figure 2 As shown, the training process of the first network model includes:

[0120] S201. Pre-train the pre-built back-propagation neural network model to obtain a target back-propagation neural network.

[0121] Optionally, image data of vehicles traveling on various road attributes and various weather attributes are collected in advance and annotated to obtain standard image data, and each standard image data is divided into a training set, a test set, and a validation set in a ratio of 7:2:1. Exemplarily, standard image data of vehicles traveling on sunny, cloudy, rainy, snowy, and foggy urban roads, standard image data of vehicles traveling on sunny, cloudy, rainy, snowy, and foggy rural roads, standard image data of vehicles traveling on sunny, cloudy, rainy, snowy, and foggy mountains, standard image data of vehicles traveling on sunny, cloudy, rainy, snowy, and foggy snowy areas, and standard image data of vehicles traveling on sunny, cloudy, rainy, snowy, and foggy areas in the desert are collected in advance.

[0122] A back propagation (BP) neural network model is pre-constructed, and each batch of standard image data is forward input into the BP neural network model. The BP neural network model outputs the predicted probability distribution of each frame of standard image data in each batch of standard image data through a deep learning classification algorithm and a target detection algorithm, wherein the predicted probability distribution includes the predicted probability distribution of each frame of standard image data in each road attribute and the predicted probability distribution of each weather attribute.

[0123] According to the loss function of the weather category and the loss function of the road category, the loss value of each batch of each category is determined respectively, and the gradient is determined according to each loss value. The weight and bias of the BP neural network model are iteratively adjusted according to each gradient and iterative prediction is performed until the loss value of each category meets the preset conditions, and the target BP neural network model is obtained, where the loss value is the error between the predicted probability distribution and the preset true probability distribution. The target detection algorithm uses a separate output of the probability distribution of a single category attribute, and suppresses the prediction of non-category during training. That is, a specific loss function is designed for each category during training, so that during back propagation, only the loss of the target category will be calculated and used to update the model parameters, while the prediction of the non-target category is suppressed and does not participate in the gradient calculation, so that the weather attribute is suppressed when predicting the road attribute, and the road attribute is suppressed when predicting the weather attribute, thereby improving the accuracy of the target BP neural network model.

[0124] S202. Use the pre-trained large model as the teacher model, use the target back propagation neural network model as the student model, and train the student model through knowledge distillation so that the student model learns the data distribution of the teacher model. Use the student model at the end of the training as the first network model.

[0125] Optionally, Figure 3 The interface diagram of the training first network model provided in the embodiment of the present application is shown in FIG. Figure 3, the pre-trained large and complex model is used as the teacher model, and the pre-trained target BP neural network model is used as the student model. The teacher model is a large model that has been fully trained and performs well, with strong learning and generalization capabilities. The training of the student model is guided by knowledge distillation, so that the student model learns to imitate the teacher model and minimizes the difference between the student model and the teacher model. Through knowledge transfer, the data distribution of the student model is ensured to be as close as possible to the data distribution of the teacher model, and a well-trained, efficient and lightweight student model is obtained, and the trained student model is used as the first network model.

[0126] In this embodiment, a target back-propagation neural network model is obtained by pre-training a pre-built back-propagation neural network model and iteratively updating the weights and biases in the back-propagation neural network model. The large model obtained by pre-training is used as the teacher model, and the target back-propagation neural network model is used as the student model. The teacher model guides the training of the student model through knowledge distillation, so that the student model learns to imitate the teacher model, minimizes the difference between the teacher model and the student model, ensures that the data distribution of the student model is as close as possible to the data distribution of the teacher model, obtains a trained efficient and lightweight student model, and uses the trained student model as the first network model. The prediction accuracy of the first network model is improved through pre-training and training guided by the teacher model.

[0127] The following describes in detail the process of determining predicted road attributes and predicted weather attributes of a vehicle according to a preset number of frames of image data using a pre-trained first network model.

[0128] Figure 4 A schematic diagram of a process for determining predicted road attributes and predicted weather attributes of a vehicle traveling in a head-up display method for a vehicle provided in an embodiment of the present application, such as Figure 4 As shown, the step of determining the predicted road attributes and the predicted weather attributes of the vehicle according to the image data of a preset number of frames by the first network model obtained by pre-training in the above step S102 includes:

[0129] S401. Input a preset number of frames of image data into the first network model in sequence to obtain a predicted probability distribution of each frame of image data, wherein the predicted probability distribution includes a predicted probability distribution of each frame of image data in each road attribute and a predicted probability distribution of each weather attribute.

[0130] Optionally, a set of image data with a preset number of frames is forward input into the first network model frame by frame, and the first network model uses a deep learning classification algorithm to analyze each frame of image data frame by frame and output the predicted probability distribution of each road attribute and the predicted probability distribution of each weather attribute.

[0131] Exemplarily, for a frame of image data, the predicted probability distribution of each road attribute output by the first network model can be [0.4 0.2 0.2 0.1 0.1], indicating that the first network model predicts that the probability of the vehicle traveling on urban roads, rural roads, mountains, snow and deserts are 0.4, 0.2, 0.2, 0.1 and 0.1 respectively; the output predicted probability distribution of each weather attribute can be [0.1 0.3 0.2 0.2 0.2], indicating that the first network model predicts that the probability of the vehicle traveling on sunny days, cloudy days, rainy days, snowy days and foggy days is 0.1, 0.3, 0.2, 0.2 and 0.2.

[0132] S402: Determine predicted road attributes for vehicle travel based on predicted probability distribution of each frame of image data for each road attribute.

[0133] Optionally, the first network model determines the predicted road attribute corresponding to each frame of image data based on the predicted probability distribution of each road attribute for each frame of image data in a set of preset frame number image data. Specifically, for a frame of image data, the road attribute with the highest probability in the predicted probability distribution of road attributes for the frame of image data is used as the predicted road attribute corresponding to the frame of image data. Exemplarily, if the first network model predicts that the probability of a vehicle traveling on an urban road, a rural road, a mountain, a snowy area, and a desert is 0.4, 0.2, 0.2, 0.1, and 0.1, respectively, for the frame of image data, and the probability of an urban road is the highest, then the predicted road attribute corresponding to the frame of image data is determined to be an urban road.

[0134] The first network model determines the predicted road attribute of the vehicle according to the predicted road attribute corresponding to each frame of image data in a set of preset frame number image data. Specifically, the road attribute with the most hits of the predicted road attribute corresponding to each frame of image data in the preset frame number is used as the predicted road attribute of the vehicle. Exemplarily, if the preset number of frames is 20 frames, among which the predicted road attribute corresponding to 15 frames of image data is an urban road, the predicted road attribute corresponding to 3 frames of image data is a rural road, and the predicted road attribute corresponding to 2 frames of image data is a desert, then the predicted road attribute of the vehicle is determined to be an urban road, and the urban road is used as the predicted road attribute of the vehicle corresponding to this set of preset frame number image data.

[0135] S403: Determine the predicted weather attribute for vehicle travel according to the predicted probability distribution of each weather attribute for each frame of image data.

[0136] Optionally, the first network model determines the predicted weather attribute corresponding to each frame of image data based on the predicted probability distribution of each weather attribute for each frame of image data in a set of preset frame number image data. Specifically, for a frame of image data, the weather attribute with the highest probability in the predicted probability distribution of weather attributes for the frame of image data is used as the predicted weather attribute corresponding to the frame of image data. Exemplarily, if the first network model predicts that the probability of the vehicle traveling on a sunny day, cloudy day, rainy day, snowy day and foggy day is 0.1, 0.3, 0.2, 0.2 and 0.2 for the frame of image data, and the probability of cloudy day is the highest, then the predicted weather attribute corresponding to the frame of image data is determined to be cloudy day.

[0137] The first network model determines the predicted weather attribute for vehicle travel according to the predicted weather attribute corresponding to each frame of image data in a set of preset frame number image data. Specifically, the weather attribute with the most hits for the predicted weather attribute corresponding to each frame of image data in the preset frame number is used as the predicted weather attribute for vehicle travel. Exemplarily, if the preset number of frames is 20 frames, among which the predicted weather attribute corresponding to 16 frames of image data is cloudy, the predicted weather attribute corresponding to 3 frames of image data is rainy, and the predicted weather attribute corresponding to 1 frame of image data is snowy, then the predicted weather attribute for vehicle travel is determined to be cloudy, and cloudy is used as the predicted weather attribute for vehicle travel corresponding to this set of preset frame number image data.

[0138] In this embodiment, a set of image data of a preset number of frames is input into the first network model frame by frame, and the first network model uses a deep learning classification algorithm to analyze and output the predicted probability distribution of each frame of image data in each road attribute and the predicted probability distribution of each weather attribute. The first network model determines the predicted road attribute corresponding to each frame of image data based on the predicted probability distribution of each road attribute of each frame of image data in a set of preset number of frames, and determines the predicted road attribute of vehicle travel based on the predicted road attribute corresponding to each frame of image data in a set of preset number of frames. The first network model determines the predicted weather attribute corresponding to each frame of image data based on the predicted probability distribution of each weather attribute of each frame of image data in a set of preset number of frames, and determines the predicted weather attribute of vehicle travel based on the predicted weather attribute corresponding to each frame of image data in a set of preset number of frames. The accuracy and prediction efficiency of the predicted road attribute and predicted weather attribute of vehicle travel are improved.

[0139] The following describes in detail the process of determining whether to switch the current projection mode of the head-up display according to the predicted road attributes, predicted weather attributes, road lighting values, and map navigation information.

[0140] Figure 5 A schematic diagram of a flow chart of determining whether to switch the current projection mode of the head-up display of a vehicle provided in an embodiment of the present application. Figure 5 As shown, the step of determining whether to switch the current projection mode of the head-up display according to the predicted road attributes, predicted weather attributes, road lighting values ​​and map navigation information in the above step S103 includes:

[0141] S501: Synchronize the predicted road attributes, predicted weather attributes, road lighting values, and map navigation information to obtain target map navigation information.

[0142] Optionally, the predicted road attributes, predicted weather attributes, and road lighting values ​​are time-aligned with a set of preset frame image data, and after the time alignment, the target map navigation information is determined based on the map navigation information obtained in real time. The target map navigation information includes the latitude and longitude of the vehicle and the road name.

[0143] S502: If it is determined according to the target map navigation information that the vehicle is currently traveling in a non-specific area, it is determined whether the road lighting value matches the predicted weather attribute.

[0144] Optionally, it is determined whether the vehicle is currently traveling in a non-specific area according to the navigation information of the target map, wherein the non-specific area can be any area other than a garage, parking lot or tunnel, that is, it is determined whether the vehicle is currently traveling in an outdoor area. If not, it means that the vehicle is traveling in a closed indoor section such as a garage, parking lot or tunnel, and is not affected by the external road, weather and lighting. There is no need to switch the AR-HUD projection mode according to the external road, weather and lighting, and the current projection mode can be maintained unchanged.

[0145] If so, it means that the vehicle is driving in an outdoor area and is easily affected by the external road, weather and lighting. It is necessary to determine whether the road lighting value and the predicted weather attribute match to perform logical judgment to verify whether the predicted weather attribute output by the first network model is accurate.

[0146] S503: If yes, determine the target projection brightness type according to the road lighting value and the lighting thresholds corresponding to each brightness type.

[0147] Optionally, if the road lighting value matches the predicted weather attribute, indicating that the predicted weather attribute output by the first network model is relatively accurate, then the projection brightness type that best matches the road lighting value is determined based on the difference between the road lighting value and the preset lighting thresholds corresponding to each brightness type. Specifically, the brightness type with the smallest difference between the road lighting value and the lighting thresholds corresponding to each brightness type is used as the target projection brightness type.

[0148] Exemplarily, the projection brightness type may include a projection brightness type corresponding to early morning or evening, a projection brightness type corresponding to noon, and a projection brightness type corresponding to night. The projection brightness type corresponding to early morning or evening corresponds to a first lighting threshold, the projection brightness type corresponding to noon corresponds to a second lighting threshold, and the projection brightness type corresponding to night corresponds to a third lighting threshold. The first difference, second difference, and third difference between the road lighting value and the first lighting threshold, the second lighting threshold, and the third lighting threshold are determined respectively. If the first difference is less than the second difference and the third difference, the target projection brightness type is the projection brightness type corresponding to early morning or evening.

[0149] Among them, the number of preset brightness types is not specifically limited here. Based on the requirements of switching sensitivity of different projection brightness, more or fewer types of brightness types can be set. The above projection brightness types include the projection brightness type corresponding to the early morning or evening, the projection brightness type corresponding to noon, and the projection brightness type corresponding to night, which is only an example of an embodiment of the present application.

[0150] S504: Determine whether to switch the current projection mode of the head-up display according to the target projection brightness type, the predicted road attributes, the predicted weather attributes, and the projection brightness type, the road attributes, and the weather attributes in the current projection mode.

[0151] Optionally, the projection brightness type, road attributes and weather attributes of the AR-HUD in the current projection mode are obtained, and compared one by one with the target projection brightness type, predicted road attributes and predicted weather attributes, and it is determined whether to switch the current projection mode of the AR-HUD based on the comparison results.

[0152] In this embodiment, the predicted road attributes, predicted weather attributes and road lighting values ​​are synchronized in time series. After data synchronization, the target map navigation information is determined according to the map navigation information obtained in real time. According to the latitude and longitude of the vehicle and the road name represented by the target map navigation information, it is determined whether the vehicle is currently traveling in a non-specific area. If so, it is necessary to determine whether the road lighting value and the predicted weather attribute match to perform logical judgment to verify whether the predicted weather attribute output by the first network model is accurate. If the road lighting value and the predicted weather attribute match, the projection brightness type that best matches the road lighting value is determined according to the difference between the road lighting value and the lighting threshold corresponding to each preset brightness type. Compare the target projection brightness type, predicted road attributes, predicted weather attributes, and the projection brightness type, road attributes and weather attributes in the current projection mode, and determine whether to switch the current projection mode of the head-up display according to the comparison results. The accuracy of the predicted weather attributes output by the first network model is verified, and the accuracy of switching the current projection mode of the head-up display is improved.

[0153] The following describes in detail the process of performing data synchronization on predicted road attributes, predicted weather attributes, road lighting values, and map navigation information to obtain target map navigation information.

[0154] Figure 6 A schematic diagram of a data synchronization process of a head-up display method for a vehicle provided in an embodiment of the present application, such as Figure 6 As shown, the step of synchronizing the predicted road attributes, predicted weather attributes, road lighting values ​​and map navigation information in the above step S501 to obtain target map navigation information includes:

[0155] S601. Determine a time interval corresponding to predicted road attributes, predicted weather attributes, and road lighting values ​​according to the acquisition time of image data of a preset number of frames.

[0156] Optionally, each frame of image data in a set of preset frame numbers of image data includes a timestamp, and the acquisition time of each frame of image data is determined based on the timestamp of each frame of image data in a set of preset frame numbers, and then the acquisition time of a set of preset frame numbers of image data is determined, that is, the difference between the timestamp of the first frame of image data and the timestamp of the last frame of image data in the set of preset frame numbers of image data.

[0157] The acquisition time of this set of image data with a preset frame number is used as the time interval for the predicted road attributes and predicted weather attributes output by the first network model based on the set of image data with a preset frame number, as well as the time interval for the calculated road lighting values, so that the set of image data with a preset frame number acquired at this acquisition time corresponds to the predicted road attributes, predicted weather attributes and road lighting values ​​in this time interval.

[0158] S602: Use the map navigation information corresponding to the time interval as target map navigation information.

[0159] Optionally, the map navigation information corresponding to the time interval is intercepted from the map navigation information acquired in real time as the target map navigation information when collecting the image data of the preset number of frames. The latitude and longitude of the vehicle and the road name in the target map navigation information can represent the positioning of the vehicle when collecting the image data of the preset number of frames.

[0160] In this embodiment, the acquisition time of each frame of image data is determined according to each time stamp contained in each frame of image data in a set of preset frames, and then the acquisition time of a set of preset frames of image data is determined. The acquisition time of this set of preset frames of image data is used as the time interval of the predicted road attributes and predicted weather attributes output by the first network model according to the set of preset frames of image data, and the time interval of the calculated road lighting value, so that the set of preset frames of image data collected at the acquisition time corresponds to the predicted road attributes, predicted weather attributes and road lighting values ​​in the time interval. The map navigation information corresponding to the time interval is intercepted from the map navigation information obtained in real time as the target map navigation information when the set of preset frames of image data is collected, so as to represent the positioning when the set of preset frames of image data is collected. The predicted road attributes, predicted weather attributes, road lighting values ​​and map navigation information are synchronized based on the timestamp, and the target map navigation information is accurately determined.

[0161] The following is a detailed description of the process of determining whether the road lighting value matches the predicted weather attribute.

[0162] Figure 7 A schematic diagram of a process for determining whether a road lighting value matches a predicted weather attribute in a head-up display method for a vehicle provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the step of determining whether the road lighting value matches the predicted weather attribute in the above step S502 includes:

[0163] S701. Obtain the current time, and determine the current time period to which the current time belongs.

[0164] Optionally, the vehicle system data also includes time, and the computer device acquires the current time of the vehicle in real time by communicating with the vehicle system in real time, and determines the current time period to which the current time belongs according to the current time. For example, the current time period may be early morning, noon, evening or night, or may be a specific hourly interval, such as the interval from 9 am to 10 am, the interval from 4 pm to 5 pm, etc.

[0165] S702: Obtain the lighting threshold corresponding to the predicted weather attribute in the current time period.

[0166] Optionally, the daylighting thresholds corresponding to different weather attributes in each time period are pre-set, and according to the current time period and the predicted weather attribute output by the first network model, the daylighting thresholds corresponding to the different weather attributes in each time period are searched for the daylighting thresholds corresponding to the predicted weather attribute in the current time period. The daylighting thresholds may include a minimum daylighting threshold and a maximum daylighting threshold, i.e., the upper and lower limits of the daylighting value, which characterize the normal daylighting range of the daylighting value corresponding to different weather attributes in each time period.

[0167] Exemplarily, if the predicted weather attribute output by the first network model is cloudy and the current time period is early morning, the daylight threshold for cloudy early morning is obtained from the daylight thresholds corresponding to different weather attributes in each time period, including the upper and lower limits of the daylight value for cloudy early morning.

[0168] S703: Compare the road lighting value with the lighting threshold, and determine whether the road lighting value matches the predicted weather attribute based on the comparison result.

[0169] Optionally, the road lighting value is compared with the upper and lower limits of the lighting value in the lighting threshold. If the road lighting value is greater than or equal to the lower limit of the lighting value and less than or equal to the upper limit of the lighting value, that is, the road lighting value is within the normal lighting interval of the lighting value, then the road lighting value is determined to match the predicted weather attribute, and the predicted weather attribute output by the first network model is verified to be relatively accurate. If the road lighting value is greater than the lower limit of the lighting value or the upper limit of the lighting value when it rains, that is, the road lighting value is not within the normal lighting interval of the lighting value, then the road lighting value is determined to not match the predicted weather attribute, and the accuracy of the predicted weather attribute output by the first network model is verified to be low.

[0170] In this embodiment, the current time of the vehicle when it is traveling is obtained in real time by communicating with the vehicle system in real time, and the current time period to which the current time belongs is determined based on the current time. According to the current time period and the predicted weather attributes output by the first network model, the lighting threshold corresponding to the predicted weather attribute in the current time period is found from the preset lighting thresholds corresponding to different weather attributes in each time period. The road lighting value is compared with the lighting threshold, and it is determined whether the road lighting value matches the predicted weather attribute based on the comparison result. The accuracy of the predicted weather attributes output by the first network model is accurately verified.

[0171] The following describes in detail the process of determining whether to switch the current projection mode of the head-up display according to the target projection brightness type, predicted road attributes, predicted weather attributes, and the projection brightness type, road attributes, and weather attributes in the current projection mode.

[0172] Figure 8 Another flowchart of the head-up display method for a vehicle provided in an embodiment of the present application for determining whether to switch the current projection mode of the head-up display is shown in FIG. Figure 8 As shown, the step of determining whether to switch the current projection mode of the head-up display according to the target projection brightness type, the predicted road attributes, the predicted weather attributes, and the projection brightness type, the road attributes, and the weather attributes in the current projection mode in step S504 includes:

[0173] S801. If the target projection brightness type is consistent with the projection brightness type in the current projection mode, the predicted road attributes are consistent with the road attributes in the current projection mode, and the predicted weather attributes are consistent with the weather attributes in the current projection mode, determine not to switch the current projection mode of the head-up display.

[0174] Optionally, the target projection brightness type is compared with the projection brightness type in the current projection mode to obtain a first comparison result; the predicted road attributes are compared with the road attributes in the current projection mode to obtain a second comparison result; and the predicted weather attributes are compared with the weather attributes in the current projection mode to obtain a third comparison result.

[0175] Combined with the first comparison result, the second comparison result and the third comparison result, a comprehensive judgment is made as to whether to switch the current projection mode of the AR-HUD. Specifically, if the first comparison result, the second comparison result and the third comparison result are all consistent, it means that the external lighting, road and weather have not changed significantly when the vehicle is currently driving, and they all meet the current projection mode, then there is no need to switch the current projection mode of the AR-HUD.

[0176] S802: Otherwise, determine to switch the current projection mode of the head-up display.

[0177] Optionally, if at least one of the first comparison result, the second comparison result or the third comparison result is inconsistent, indicating that one or more factors of the external lighting, road or weather have changed significantly when the vehicle is currently driving, and do not conform to the current projection mode, it is determined that the current projection mode of the AR-HUD needs to be switched.

[0178] In this embodiment, by combining the first comparison result of whether the target projection brightness type is consistent with the projection brightness type in the current projection mode, the second comparison result of whether the predicted road attributes are consistent with the road attributes in the current projection mode, and the third comparison result of whether the predicted weather attributes are consistent with the weather attributes in the current projection mode, a comprehensive judgment is made as to whether to switch the current projection mode of the head-up display. If the target projection brightness type, the predicted road attributes, and the predicted weather attributes are all consistent with the projection brightness type, road attributes, and weather attributes in the current projection mode, it is determined not to switch the current projection mode of the head-up display, otherwise, it is determined to switch the current projection mode of the head-up display. The accuracy of switching the current projection mode of the head-up display is improved, and abrupt projection effects caused by obvious changes in the external environment are avoided.

[0179] The following describes in detail the process of verifying and correcting the first network model based on the image data of a preset number of frames using the second network model obtained by pre-training.

[0180] Fig. 9A schematic diagram of a process of verifying and correcting a first network model of a vehicle head-up display method provided in an embodiment of the present application, such as Figure 8 As shown, the step of verifying and correcting the first network model according to the preset number of frames of image data by the pre-trained second network model in the above step S105 includes:

[0181] S901. Determine, by a second network model, a target road attribute and a target weather attribute for vehicle travel based on image data of a preset number of frames.

[0182] Optionally, the image data of a preset number of frames are sequentially forward-inputted into a second network model with a more complex model structure and higher prediction accuracy, and the second network model uses a deep learning classification algorithm to analyze frame by frame and output the target probability distribution of each frame of image data in each road attribute and the target probability distribution of each weather attribute, and determine the target road attribute and target weather attribute corresponding to each frame of image data based on the target probability distribution of each road attribute and the target weather distribution of each weather attribute in a set of image data with a preset number of frames. Then, the second network model determines the target road attribute and target weather attribute for vehicle travel based on the target road attribute and target weather attribute corresponding to each frame of image data in a set of image data with a preset number of frames. Specifically, the road attribute with the most hits for the target road attribute corresponding to each frame of image data in the preset number of frames is used as the target road attribute for vehicle travel, and the weather attribute with the most hits for the target weather attribute corresponding to each frame of image data in the preset number of frames is used as the target weather attribute for vehicle travel.

[0183] It is worth noting that since the second network model is more complex, the calculation process is longer, and the prediction accuracy is higher, that is, the prediction result is more in line with the actual road and weather environment, the target road attributes predicted and output by the second network model can be used as the actual road attributes, and the target weather attributes predicted and output by the second network model can be used as the actual weather attributes.

[0184] S902: Determine a road score according to the target road attributes and the predicted road attributes.

[0185] Optionally, for the same set of image data of preset frames, the target road attributes output by the second network model are compared with the predicted road attributes output by the first network model to see if they are consistent, and the predicted road attributes output by the first network model for the same set of image data of preset frames are scored according to the comparison results to obtain the road scores of the first network model for each set of image data of preset frames. The road scores of the first network model for each set of image data of preset frames are used to characterize the accuracy of the road attributes output by the first network model for each set of image data of preset frames.

[0186] Specifically, if for the same set of image data with a preset number of frames, the target road attributes output by the second network model are consistent with the predicted road attributes output by the first network model, the road score of the first network model for this set of image data with a preset number of frames is assigned to 1; if for the same set of image data with a preset number of frames, the target road attributes output by the second network model are inconsistent with the predicted road attributes output by the first network model, the road score of the first network model for this set of image data with a preset number of frames is assigned to 0.

[0187] Exemplarily, if for the same set of image data with a preset number of frames, the target road attribute output by the second network model is an urban road, and the predicted road attribute output by the first network model is a rural road, then the road score of the first network model for this set of image data with a preset number of frames is 0; if for the same set of image data with a preset number of frames, the target road attribute output by the second network model and the predicted road attribute output by the first network model are both rural roads, then the road score of the first network model for this set of image data with a preset number of frames is 1.

[0188] S903: Determine a weather score according to the target weather attributes and the predicted weather attributes.

[0189] Optionally, for the same set of image data of preset frames, the target weather attributes output by the second network model are compared with the predicted weather attributes output by the first network model to see if they are consistent, and the predicted weather attributes output by the first network model for the same set of image data of preset frames are scored according to the comparison results to obtain the weather scores of the first network model for each set of image data of preset frames. The weather scores of the first network model for each set of image data of preset frames are used to characterize the accuracy of the weather attributes output by the first network model for each set of image data of preset frames.

[0190] Specifically, if for the same set of image data with a preset number of frames, the target weather attribute output by the second network model is consistent with the predicted weather attribute output by the first network model, the weather score of the first network model for this set of image data with a preset number of frames is assigned a value of 1; if for the same set of image data with a preset number of frames, the target weather attribute output by the second network model is inconsistent with the predicted weather attribute output by the first network model, the weather score of the first network model for this set of image data with a preset number of frames is assigned a value of 0.

[0191] Exemplarily, if for the same set of image data with a preset number of frames, the target weather attribute output by the second network model is sunny, and the predicted weather attribute output by the first network model is cloudy, then the weather score of the first network model for this set of image data with a preset number of frames is 0; if for the same set of image data with a preset number of frames, the target road attribute output by the second network model and the predicted road attribute output by the first network model are both cloudy, then the weather score of the first network model for this set of image data with a preset number of frames is 1.

[0192] S904: Determine a target score according to the road score, the weather score, a preset road weight, and a preset weather weight.

[0193] Optionally, the road weight and weather weight are preset. Since weather changes in the external environment have a greater impact on the projection mode of the AR-HUD than road changes, the preset weather weight is greater than the preset road weight. The sum of the preset weather weight and the preset road weight is 1. For example, the preset weather weight may be 0.7, and the preset road weight may be 0.3. According to the road score, weather score, and budgeted road weight and weather weight of the first network model for each group of preset frame image data, the target score of the first network model for each group of preset frame image data is determined.

[0194] Specifically, a first product is determined based on the road score and the preset road weight of the first network model for this group of preset frame number image data; a second product is determined based on the weather score and the preset weather weight of the first network model for this group of preset frame number image data; and the sum of the first product and the second product is used as the target score of the first network model for this group of preset frame number image data.

[0195] S905: Verify and correct the first network model according to the target score and the preset value.

[0196] Optionally, the first network model is verified according to the relationship between the target score of each group of preset frame numbers of image data of the first network model and a preset value.

[0197] Specifically, according to the relationship between the target score of the first network model for this group of preset frame image data and the preset value, check whether the prediction accuracy of the first network model for this group of preset frame image data meets the standard. If it does, discard this group of preset frame image data, and continue to determine the target score of the first network model for the next group of preset frame image data, so as to check whether the prediction accuracy of the first network model for the next group of preset frame image data meets the standard according to the relationship between the target score of the first network model for the next group of preset frame image data and the preset value. If it does not meet the standard, retain the next group of preset frame image data, and continue to check whether the prediction accuracy of the first network model for the next group of preset frame image data meets the standard.

[0198] Exemplarily, the preset value can be 0.6. If the target score of the first network model for this set of image data with a preset number of frames is 0.8, it is determined that the prediction accuracy of the first network model for this set of image data with a preset number of frames meets the standard; if the target score of the first network model for this set of image data with a preset number of frames is 0.5, it is determined that the prediction accuracy of the first network model for this set of image data with a preset number of frames does not meet the standard, and it is necessary to retain this set of image data with a preset number of frames to facilitate subsequent correction of the first network model.

[0199] After the verification is completed, the first network model is corrected according to all the retained image data of each group of preset frames whose prediction accuracy does not meet the standard, and each model parameter in the first network model is updated to obtain a first network model with higher prediction accuracy after training.

[0200] In this embodiment, the image data of the preset number of frames are sequentially forward-inputted into the second network model with a more complex model structure and higher prediction accuracy, and the second network model uses a deep learning classification algorithm to analyze and output the target road attributes and target weather attributes of the vehicle driving frame by frame. For the same set of image data of the preset number of frames, the predicted road attributes output by the first network model for the same set of image data of the preset number of frames are scored according to the comparison result of whether the target road attributes output by the second network model are consistent with the predicted road attributes output by the first network model, and the road score of the first network model for each set of image data of the preset number of frames is obtained to characterize the accuracy of the road attributes output by the first network model for each set of image data of the preset number of frames. And according to the comparison result of whether the target weather attributes output by the second network model are consistent with the predicted weather attributes output by the first network model, the predicted weather attributes output by the first network model for the same set of image data of the preset number of frames are scored to obtain the weather score of the first network model for each set of image data of the preset number of frames to characterize the accuracy of the weather attributes output by the first network model for each set of image data of the preset number of frames. According to the road score, weather score, preset road weight and preset weather weight of the first network model for each group of preset frame image data, the target score of the first network model for each group of preset frame image data is determined, and according to the relationship between the target score and the preset value of the first network model for each group of preset frame image data, the first network model is verified, and the image data of each group of preset frame whose prediction accuracy does not meet the standard is retained, and the model parameters of the first network model are corrected based on this. The corrected first network model has the dual effects of high prediction accuracy and fast prediction efficiency.

[0201] As an optional implementation, the step of verifying and correcting the first network model according to the target score and the preset value in the above step S905 includes:

[0202] If the target score is less than a preset value, the preset number of frames of image data are labeled according to the prediction results of the second network model for the preset number of frames of image data, and the labeled preset number of frames of image data are added to the training database of the first network model to continue training the first network model according to the training database.

[0203] Optionally, for a group of image data with a preset number of frames, if the target score of the first network model for this group of image data with a preset number of frames is less than a preset value, then this group of image data with a preset number of frames is retained, and road attribute labels and weather attribute labels are performed on this group of image data with a preset number of frames based on the target road attributes and target weather attributes in the prediction results output by the second network model for this group of image data with a preset number of frames.

[0204] Repeat the above steps until all groups of preset frame numbers of image data with target scores less than the preset value are labeled, and all labeled image data are added as new standard image data to the original training database of the first network model to obtain a new training database, and continue to train the first network model based on the new training database to continuously improve the prediction accuracy of the first network model for road attributes and weather attributes.

[0205] In this embodiment, the image data of each group of preset frames whose target score is less than the preset value is retained, and the road attribute and weather attribute of each group of preset frame image data retained are annotated according to the target road attribute and target weather attribute in the prediction results output by the second network model for each group of preset frame image data, and all the annotated image data are added as new standard image data to the original training database of the first network model to obtain a new training database, and the first network model is continuously trained according to the new training database. The prediction accuracy of the first network model for road attributes and weather attributes is greatly improved.

[0206] Based on the same inventive concept, the embodiment of the present application also provides a head-up display device for a vehicle corresponding to the head-up display method for a vehicle. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the head-up display method for the vehicle in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0207] Fig.10 The module structure diagram of the head-up display device of the vehicle provided in the embodiment of the present application is as follows: Fig.10 As shown, the device comprises:

[0208] The acquisition module 1001 is used to acquire the image data of the vehicle driving and the vehicle system data in real time; the vehicle system data at least includes map navigation information.

[0209] The determination module 1002 is used to determine the predicted road attributes and predicted weather attributes of the vehicle according to the image data of a preset number of frames using the pre-trained first network model.

[0210] The determination module 1002 is also used to determine the road lighting value based on the image data of a preset number of frames, and determine whether to switch the current projection mode of the head-up display based on the predicted road properties, predicted weather properties, road lighting values ​​and map navigation information.

[0211] The determination module 1002 is further configured to, if yes, determine the target projection mode according to the predicted road attributes, the predicted weather attributes and the road lighting value, and switch the current projection mode to the target projection mode.

[0212] The correction module 1003 is used to verify and correct the first network model according to the image data of a preset number of frames using the pre-trained second network model, wherein the prediction accuracy of the second network model is higher than the prediction accuracy of the first network model, and the prediction efficiency of the second network model is lower than the prediction efficiency of the first network model.

[0213] As an optional implementation, the determination module 1002 is specifically configured to:

[0214] The pre-built back-propagation neural network model is pre-trained to obtain the target back-propagation neural network.

[0215] The pre-trained large model is used as the teacher model, and the target back-propagation neural network model is used as the student model. Through knowledge distillation, the student model is trained so that the student model learns the data distribution of the teacher model, and the student model at the end of training is used as the first network model.

[0216] As an optional implementation, the determination module 1002 is specifically configured to:

[0217] The image data of a preset number of frames are sequentially input into the first network model to obtain the predicted probability distribution of each frame of image data, wherein the predicted probability distribution includes the predicted probability distribution of each frame of image data in each road attribute and the predicted probability distribution of each weather attribute.

[0218] The predicted road attribute for vehicle travel is determined based on the predicted probability distribution of each frame image data on each road attribute.

[0219] The predicted weather attribute for vehicle driving is determined based on the predicted probability distribution of each weather attribute for each frame of image data.

[0220] As an optional implementation, the determination module 1002 is specifically configured to:

[0221] The predicted road attributes, predicted weather attributes, road lighting values, and map navigation information are synchronized to obtain target map navigation information.

[0222] If it is determined according to the target map navigation information that the vehicle is currently traveling in a non-specific area, it is determined whether the road lighting value matches the predicted weather attribute.

[0223] If so, the target projection brightness type is determined according to the road lighting value and the lighting threshold corresponding to each brightness type.

[0224] Determine whether to switch the current projection mode of the head-up display according to the target projection brightness type, predicted road attributes, predicted weather attributes, and the projection brightness type, road attributes, and weather attributes in the current projection mode.

[0225] As an optional implementation, the determination module 1002 is specifically configured to:

[0226] According to the acquisition time of the image data of the preset frame number, the time interval corresponding to the predicted road attribute, the predicted weather attribute and the road lighting value is determined.

[0227] The map navigation information corresponding to the time interval is used as the target map navigation information.

[0228] As an optional implementation, the determination module 1002 is specifically configured to:

[0229] Get the current time and determine the current time period to which the current time belongs.

[0230] Get the daylight threshold corresponding to the predicted weather attribute in the current period.

[0231] The road lighting value is compared with the lighting threshold, and whether the road lighting value matches the predicted weather attribute is determined based on the comparison result.

[0232] As an optional implementation, the determination module 1002 is specifically configured to:

[0233] If the target projection brightness type is consistent with the projection brightness type in the current projection mode, the predicted road attributes are consistent with the road attributes in the current projection mode, and the predicted weather attributes are consistent with the weather attributes in the current projection mode, it is determined not to switch the current projection mode of the head-up display.

[0234] Otherwise, it is determined to switch the current projection mode of the head-up display.

[0235] As an optional implementation, the correction module 1003 is specifically used for:

[0236] The second network model determines the target road attributes and target weather attributes of the vehicle based on the image data of a preset number of frames.

[0237] The road score is determined based on the target road attributes and the predicted road attributes.

[0238] The weather score is determined based on the target weather attributes and the predicted weather attributes.

[0239] The target score is determined based on the road score, the weather score, the preset road weight, and the preset weather weight.

[0240] The first network model is verified and corrected according to the target score and the preset value.

[0241] As an optional implementation, the correction module 1003 is specifically used for:

[0242] If the target score is less than a preset value, the preset number of frames of image data are labeled according to the prediction results of the second network model for the preset number of frames of image data, and the labeled preset number of frames of image data are added to the training database of the first network model to continue training the first network model according to the training database.

[0243] The present application also provides a computer device, such as Fig.11 1 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application, including: a processor 111, a memory 112 and a bus 113. The memory 112 stores machine-readable instructions executable by the processor 111 (for example, Fig.10 In the device, the execution instructions corresponding to the acquisition module 1001, the determination module 1002 and the correction module 1003 are obtained, etc. When the computer device is running, the processor 111 and the memory 112 communicate through the bus 113. When the machine-readable instructions are executed by the processor 111, the steps of the head-up display method for the vehicle in the above embodiment are executed.

[0244] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the vehicle head-up display method in the above embodiment are executed.

[0245] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0246] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0247] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A head-up display method for a vehicle, characterized in that: include: Acquire vehicle driving image data and vehicle system data in real time; The vehicle system data at least includes map navigation information; The first network model obtained by pre-training determines the predicted road attributes and the predicted weather attributes of the vehicle according to the image data of a preset number of frames; Determine a road lighting value according to the preset number of frames of image data, and determine whether to switch a current projection mode of a head-up display according to the predicted road attributes, the predicted weather attributes, the road lighting value, and the map navigation information; If yes, determining a target projection mode according to the predicted road attributes, the predicted weather attributes and the road lighting value, and switching the current projection mode to the target projection mode; The second network model obtained by pre-training verifies and corrects the first network model according to the image data of the preset number of frames, wherein the prediction accuracy of the second network model is higher than the prediction accuracy of the first network model, and the prediction efficiency of the second network model is lower than the prediction efficiency of the first network model.

2. The method according to claim 1, characterized in that The training process of the first network model includes: Pre-training the pre-built back-propagation neural network model to obtain a target back-propagation neural network; The pre-trained large model is used as the teacher model, and the target back propagation neural network model is used as the student model. Through knowledge distillation, the student model is trained so that the student model learns the data distribution of the teacher model, and the student model at the end of the training is used as the first network model.

3. The method according to claim 1, characterized in that The first network model obtained by pre-training determines the predicted road attributes and predicted weather attributes of the vehicle according to the image data of a preset number of frames, including: The image data of the preset number of frames are sequentially input into the first network model in a forward direction to obtain a predicted probability distribution of each frame of image data, wherein the predicted probability distribution includes a predicted probability distribution of each frame of image data in each road attribute and a predicted probability distribution of each weather attribute; Determining predicted road attributes for vehicle travel according to predicted probability distribution of each frame of image data at each road attribute; The predicted weather attribute for vehicle travel is determined based on the predicted probability distribution of each weather attribute for each frame of image data.

4. The method according to claim 1, characterized in that The determining whether to switch the current projection mode of the head-up display according to the predicted road attribute, the predicted weather attribute, the road lighting value and the map navigation information includes: Performing data synchronization on the predicted road attributes, the predicted weather attributes, the road lighting value, and the map navigation information to obtain target map navigation information; If it is determined according to the target map navigation information that the vehicle is currently traveling in a non-specific area, determining whether the road lighting value matches the predicted weather attribute; If yes, determining the target projection brightness type according to the road lighting value and the lighting thresholds corresponding to each brightness type; Determine whether to switch the current projection mode of the head-up display according to the target projection brightness type, the predicted road attributes, the predicted weather attributes, and the projection brightness type, road attributes, and weather attributes in the current projection mode.

5. The method according to claim 4, characterized in that The step of synchronizing the predicted road attributes, the predicted weather attributes, the road lighting value, and the map navigation information to obtain target map navigation information includes: Determining the time interval corresponding to the predicted road attribute, the predicted weather attribute, and the road lighting value according to the acquisition time of the image data of the preset number of frames; The map navigation information corresponding to the time interval is used as the target map navigation information.

6. The method according to claim 4, characterized in that The determining whether the road lighting value matches the predicted weather attribute includes: Obtaining the current time and determining the current time period to which the current time belongs; Obtaining the lighting threshold corresponding to the predicted weather attribute in the current time period; The road lighting value is compared with the lighting threshold, and whether the road lighting value matches the predicted weather attribute is determined according to the comparison result.

7. The method according to claim 4, characterized in that The determining whether to switch the current projection mode of the head-up display according to the target projection brightness type, the predicted road attribute, the predicted weather attribute, and the projection brightness type, road attribute, and weather attribute in the current projection mode includes: If the target projection brightness type is consistent with the projection brightness type in the current projection mode, the predicted road attribute is consistent with the road attribute in the current projection mode, and the predicted weather attribute is consistent with the weather attribute in the current projection mode, then determining not to switch the current projection mode of the head-up display; Otherwise, it is determined to switch the current projection mode of the head-up display.

8. The method according to claim 1, characterized in that The second network model obtained by pre-training verifies and corrects the first network model according to the image data of the preset number of frames, including: Determining, by the second network model, a target road attribute and a target weather attribute for vehicle travel according to the image data of the preset number of frames; Determining a road score according to the target road attribute and the predicted road attribute; Determining a weather score according to the target weather attribute and the predicted weather attribute; Determining a target score according to the road score, the weather score, a preset road weight, and a preset weather weight; The first network model is verified and corrected according to the target score and the preset value.

9. The method according to claim 8, characterized in that The verifying and correcting the first network model according to the target score and the preset value includes: If the target score is less than the preset value, the image data of the preset number of frames are labeled according to the prediction results of the second network model for the image data of the preset number of frames, and the labeled image data of the preset number of frames are added to the training database of the first network model to continue training the first network model according to the training database.

10. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the head-up display method for a vehicle as described in any one of claims 1 to 9.

Images