Vehicle HUD projection jitter prediction and optimization method and system based on multi-source data fusion

Abstract

The invention discloses a vehicle HUD projection jitter prediction and optimization method and system based on multi-source data fusion, and belongs to the technical field of intelligent driving and vehicle information display, and the method comprises the steps: confirming optimization parameters which comprise motion data of a vehicle body, vibration data of an HUD and a vibration image of the HUD in the driving process of the vehicle; acquiring a reference image based on the vibration image when the vehicle runs on the road surface meeting the flatness requirement; predicting a future vibration image of the HUD as a predicted image based on the data of the optimization parameters when the vehicle runs on the road surface to be measured; acquiring a displacement compensation value of the predicted image based on the reference image and the predicted image; and optimizing the predicted image based on the displacement compensation value. According to the method, the image displayed by the vehicle HUD in the future is adjusted in real time after the jitter is predicted, the problem of image jitter caused by factors such as vehicle vibration and road surface unevenness is solved, the stability and definition of the HUD image are improved, and therefore the visual experience and driving safety of a driver are improved.

Description

Technical Field

[0001] This application belongs to the field of intelligent driving and vehicle information display technology, specifically relating to a method and system for predicting and optimizing vehicle HUD projection jitter based on multi-source data fusion. Background Technology

[0002] With the rapid development of intelligent driving technology, vehicle HUD (Head-Up Display) systems, as an important driving assistance tool, have been widely used in modern cars and motorcycles. HUDs can project key driver assistance information (such as navigation) into the driver's field of vision in real time, allowing the driver to obtain crucial information without looking down at the car's center console, thereby improving driving safety and convenience.

[0003] Currently, plug-and-play HUDs are becoming increasingly popular due to their flexible installation. However, this installation method is unreliable. During actual driving, especially on roads with complex or bumpy conditions, the vehicle's HUD image is easily affected by changes in the external environment, causing image jitter and severely impacting the driving experience. Besides external environmental factors, the structure and installation process of the HUD device itself can also cause image jitter. For example, if the HUD device is not securely fixed during installation, or if the mechanical bracket is loose, or if internal components become mechanically loose or slightly deformed due to vibration, aging, or temperature changes during long-term use, the HUD projection image may drift, jitter, or become blurry. This type of "intrinsic" HUD image instability not only exacerbates the complexity of the jitter problem but also places higher demands on the long-term reliability of HUD projection.

[0004] Image jitter not only affects the driver's visual experience but also reduces driving safety, especially at high speeds or in complex driving scenarios, where the jittery image can impair the driver's judgment and reaction. Whether caused by impacts from the vehicle's external driving environment or by defects in the HUD's structure and installation, image jitter severely impacts the stability and practical application effectiveness of the HUD system. Summary of the Invention

[0005] Purpose of the invention: This application develops a method and system for predicting and optimizing vehicle HUD projection jitter based on multi-source data fusion, aiming to solve the technical problem of HUD projection image jitter in the prior art.

[0006] Technical Solution: In a first aspect, this application provides a method for predicting and optimizing vehicle HUD projection jitter based on multi-source data fusion, including:

[0007] The optimization parameters are confirmed, including the vehicle body motion data, HUD vibration data, and HUD vibration images during vehicle operation;

[0008] A reference image is obtained based on the vibration images of the vehicle driving on a road surface that meets the smoothness requirements;

[0009] Based on the data of optimized parameters when the vehicle is driving on the test road surface, the future vibration image of the HUD is predicted, which is the predicted image;

[0010] The displacement compensation value of the predicted image is obtained based on the reference image and the predicted image;

[0011] The predicted image is optimized based on the displacement compensation value.

[0012] In some embodiments, the step of acquiring the reference image includes:

[0013] Based on chronological order, vibration images of vehicles traveling on roads meeting smoothness requirements, collected at multiple time points, are arranged to obtain a reference image. The representation formula of the reference image includes:

[0014]

[0015] in, The reference image; I s (t1),I s (t2),…,I s (t n ) represent the times at multiple time points t1, t2, ..., t n The vibration images collected when the vehicle is driving on a road surface that meets the flatness requirements.

[0016] In some embodiments, the step of predicting future vibration images of the HUD includes:

[0017] The vibration images of the HUD when the vehicle is driving on the test road are processed by the ResNet-18 deep residual network model to obtain spatial features;

[0018] Based on standard scores, the motion data of the vehicle body, the vibration data of the HUD and the spatial features when the vehicle is driving on the test road surface are standardized to obtain standardized data;

[0019] Standardized data from multiple time points are categorized according to parameter type, and the standardized data of each parameter are sorted based on time series to obtain time series data;

[0020] The temporal features of the time-series data are learned by using an LSTM (Long Short-Term Memory) network to obtain the temporal features of the hidden state. The temporal features are used to characterize the motion data of the vehicle body and the vibration data of the HUD, as well as the spatiotemporal relationship with the vibration image.

[0021] The future vibration images of the HUD are predicted based on the temporal features.

[0022] In some embodiments, the step of acquiring spatial features includes:

[0023] The vibration image of the HUD when the vehicle is driving on the test road is processed through the convolutional layer of the ResNet-18 deep residual network model to obtain the convolutional output;

[0024] The convolutional output is processed through the pooling layer of the ResNet-18 deep residual network model to obtain the pooled output;

[0025] The pooling output is processed through the residual blocks of the deep residual network model ResNet-18 to obtain the residual output;

[0026] The spatial features are obtained by performing global average pooling on the residual output.

[0027] In some embodiments, the representation formula for the standardized data includes:

[0028]

[0029] Among them, A T This represents the three-axis acceleration data of the vehicle body at time point T, used to characterize the linear disturbance behavior of the vehicle body; G T B represents the three-axis angular velocity data of the vehicle body at time point T, used to characterize the change in the vehicle body's rotational attitude; T This represents the three-axis acceleration data of the HUD at time point T, used to characterize the vibration intensity of the HUD; H T This represents the three-axis angular velocity data of the HUD at time point T, used to characterize the attitude change of the HUD; D t The spatial features at time point T; μ A This represents the mean of the vehicle's three-axis acceleration data at multiple time points; μ G This represents the mean of the three-axis angular velocity data of the vehicle body at multiple time points; μ B This represents the mean of the three-axis acceleration data of the HUD at multiple time points; μ H This represents the mean of the three-axis angular velocity data of the HUD at multiple time points; μ D σ represents the mean of the spatial features at multiple time points; A σ represents the standard deviation of the vehicle's three-axis acceleration data at multiple time points. G σ represents the standard deviation of the vehicle's three-axis angular velocity data at multiple time points. B σ represents the standard deviation of the three-axis acceleration data of the HUD at multiple time points. Hσ represents the standard deviation of the three-axis angular velocity data of the HUD at multiple time points. D A′ represents the standard deviation of the spatial characteristics at multiple time points. T This represents the standardized three-axis acceleration data of the vehicle body at time point T; G′ T This represents the standardized three-axis angular velocity data of the vehicle body at time point T; B′ T This represents the standardized three-axis acceleration data of the HUD at time point T; H T This represents the standardized three-axis angular velocity data of the HUD at time point T; D t This represents the standardized spatial features at time point T.

[0030] In some embodiments, the step of predicting future vibration images of the HUD based on the temporal features includes:

[0031] For each future time point t+i (i = 1, 2, ..., k), the hidden state h of the LSTM Long Short-Term Memory network is... t+i All inputs are fed into the regression layer to generate vibration images at time point t+i. The formula for representing the vibration image at time point t+i includes:

[0032]

[0033] Among them, W h b is the weight matrix of the regression layer; h For bias terms; The vibration image at a future time point t+i; h t+i It is the hidden output state of the LSTM Long Short-Term Memory network at a future time point t+i.

[0034] In some embodiments, the step of obtaining the displacement compensation value of the predicted image based on the reference image and the predicted image includes:

[0035] Obtain the displacement compensation value from the previous time point, and combine it with the reference image and the predicted image to obtain the state vector at the current time point. The representation formula of the state vector includes:

[0036]

[0037] Among them, S t This is the state vector at the current time point t; The predicted image for time points t+1 to t+k; The reference image; Δ t-1 The displacement compensation value at the previous time point t-1, Δ t-1 =(Δx) t-1 ,Δy t-1), Δx t-1 Δy represents the pixel-level compensation amount applied in the horizontal direction at the previous time point t-1. t-1 This represents the pixel-level compensation amount applied in the vertical direction at the previous time point t-1;

[0038] Based on the state vector, the jitter level and direction of the HUD at the current time point are predicted, and combined with the displacement compensation value at the previous time point, the displacement compensation value at the current time point is obtained. The formula representing the displacement compensation value at the current time point includes:

[0039] Δ t =(Δx) t ,Δy t );

[0040] Where, Δ t The displacement compensation value at the current time point t; Δx t Δy represents the pixel-level compensation amount applied in the horizontal direction at the current time point t. t This represents the pixel-level compensation amount applied in the horizontal direction at the current time point t.

[0041] In some embodiments, it also includes:

[0042] Determine the reward function, which includes a pixel-level difference index and a structural similarity index;

[0043] The reward feedback between the optimized predicted image and the reference image is obtained based on the reward function, and the optimized predicted image is adjusted based on the reward feedback; the formula for the reward function includes:

[0044]

[0045] Among them, R t The reward function is used to characterize the reward feedback; MSE is the pixel-level difference index; SSIM is the structural similarity index used to characterize brightness, contrast, and structural similarity; α is the adjustable weight of the pixel-level difference index; β is the adjustable weight of the structural similarity index. The optimized predicted image; I s (t+i) is the target smoothed image subsequence corresponding to the time step of the reference image.

[0046] In some embodiments, the vehicle is provided with:

[0047] A first attitude sensor is disposed on the chassis of the vehicle and is used to acquire motion data of the vehicle body.

[0048] A second attitude sensor is disposed at the projection position of the HUD and is used to acquire vibration data of the HUD.

[0049] A camera, wherein the image acquisition port of the camera is oriented toward the projection position of the HUD, is used to acquire vibration images of the HUD during vehicle operation.

[0050] Secondly, embodiments of this application also provide a vehicle HUD projection jitter prediction and optimization system based on multi-source data fusion, comprising:

[0051] The parameter determination module is used to confirm the optimization parameters, including the vehicle body motion data, HUD vibration data, and HUD vibration images during vehicle operation.

[0052] A reference image module is used to acquire a reference image based on vibration images of a vehicle traveling on a road surface that meets the smoothness requirements;

[0053] A prediction image module is used to predict future vibration images of the HUD based on data of optimized parameters when the vehicle is driving on the test road surface.

[0054] A compensation acquisition module is used to acquire the displacement compensation value of the predicted image based on the reference image and the predicted image;

[0055] An image compensation module is used to optimize the predicted image based on the displacement compensation value.

[0056] Beneficial Effects: Compared with existing technologies, the present application provides a method for predicting and optimizing vehicle HUD projection jitter based on multi-source data fusion. This method includes identifying optimization parameters, which include vehicle motion data, HUD vibration data, and vibration images of the HUD during vehicle operation; obtaining a reference image based on the vibration images of the vehicle traveling on a road surface meeting smoothness requirements; predicting the future vibration image of the HUD based on the data of the optimization parameters when the vehicle is traveling on a test road surface; obtaining a displacement compensation value for the predicted image based on the reference image and the predicted image; and optimizing the predicted image based on the displacement compensation value. This application, by predicting jitter and adjusting the future image displayed by the vehicle HUD in real time, solves the image jitter problem caused by vehicle vibration and uneven road surfaces, improving the stability and clarity of the HUD image, thereby enhancing the driver's visual experience and driving safety. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 A flowchart illustrating the steps of a vehicle HUD projection jitter prediction and optimization method based on multi-source data fusion provided in an embodiment of this application;

[0059] Figure 2 A flowchart illustrating the steps of predicting future vibration images of the HUD in the vehicle HUD projection jitter prediction and optimization method based on multi-source data fusion provided in this application embodiment;

[0060] Figure 3 A flowchart illustrating the steps for obtaining spatial features in the vehicle HUD projection jitter prediction and optimization method based on multi-source data fusion provided in this application embodiment;

[0061] Figure 4 A flowchart illustrating the steps for obtaining displacement compensation values ​​in a vehicle HUD projection jitter prediction and optimization method based on multi-source data fusion, as provided in this application embodiment;

[0062] Figure 5 A flowchart illustrating the steps of providing reward feedback to the optimized predicted image in the vehicle HUD projection jitter prediction and optimization method based on multi-source data fusion provided in this application embodiment;

[0063] Figure 6 A schematic diagram of a vehicle in the vehicle HUD projection jitter prediction and optimization method based on multi-source data fusion provided in the embodiments of this application;

[0064] Figure 7 A module connection diagram of a vehicle HUD projection jitter prediction and optimization system based on multi-source data fusion provided in an embodiment of this application;

[0065] Figure 8 The flowchart of the method for predicting and optimizing vehicle HUD projection jitter based on multi-source data fusion provided in this application embodiment;

[0066] Figure 9 The flowchart shows the process of preprocessing and feature extraction of vehicle HUD projection jitter prediction data in the vehicle HUD projection jitter prediction and optimization method based on multi-source data fusion provided in the embodiments of this application.

[0067] Figure 10The flowchart of the program for predicting jitter images using the LSTM long short-term memory network in the vehicle HUD projection jitter prediction and optimization method based on multi-source data fusion provided in the embodiments of this application;

[0068] Figure 11 A flowchart of the reinforcement learning-based HUD image displacement compensation procedure in the vehicle HUD projection jitter prediction and optimization method based on multi-source data fusion provided in the embodiments of this application.

[0069] Reference numerals: 1. First attitude sensor; 2. Second attitude sensor; 3. Camera; 10. Parameter determination module; 20. Reference image module; 30. Predicted image module; 40. Compensation acquisition module; 50. Image compensation module. Detailed Implementation

[0070] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0071] With the rapid development of intelligent driving technology, vehicle HUD (Head-Up Display) systems, as an important driving assistance tool, have been widely used in modern cars and motorcycles. HUDs can project key driver assistance information (such as navigation) into the driver's field of vision in real time, allowing the driver to obtain crucial information without looking down at the car's center console, thereby improving driving safety and convenience.

[0072] Currently, plug-and-play HUDs are becoming increasingly popular due to their flexible installation. However, this installation method is unreliable. During actual driving, especially on roads with complex or bumpy conditions, the vehicle's HUD image is easily affected by changes in the external environment, causing image jitter and severely impacting the driving experience. Besides external environmental factors, the structure and installation process of the HUD device itself can also cause image jitter. For example, if the HUD device is not securely fixed during installation, or if the mechanical bracket is loose, or if internal components become mechanically loose or slightly deformed due to vibration, aging, or temperature changes during long-term use, the HUD projection image may drift, jitter, or become blurry. This type of "intrinsic" HUD image instability not only exacerbates the complexity of the jitter problem but also places higher demands on the long-term reliability of HUD projection.

[0073] Image jitter not only affects the driver's visual experience but also reduces driving safety, especially at high speeds or in complex driving scenarios, where the jittery image can impair the driver's judgment and reaction. Whether caused by impacts from the vehicle's external driving environment or by defects in the HUD's structure and installation, image jitter severely impacts the stability and practical application effectiveness of the HUD system.

[0074] Here, because the content to be displayed in the HUD (such as navigation information) is obtained from the server a few seconds in advance, it possesses the characteristic of "predictability." This means that before the image is actually displayed, the system can adjust the HUD screen in advance based on the vehicle's impending attitude changes, making it feasible to predict and pre-compensate for image jitter.

[0075] In view of this, embodiments of this application provide a method for predicting and optimizing vehicle HUD projection jitter based on multi-source data fusion. The method includes identifying optimization parameters, which include vehicle motion data, HUD vibration data, and vibration images of the HUD during vehicle operation; obtaining a reference image based on the vibration images of the vehicle traveling on a road surface meeting smoothness requirements; predicting the future vibration image of the HUD based on the data of the optimization parameters when the vehicle is traveling on a test road surface; obtaining a displacement compensation value for the predicted image based on the reference image and the predicted image; and optimizing the predicted image based on the displacement compensation value. This application, by predicting jitter and adjusting the future image displayed by the vehicle HUD in real time, solves the image jitter problem caused by vehicle vibration and uneven road surfaces, improves the stability and clarity of the HUD image, and thus enhances the driver's visual experience and driving safety.

[0076] In some embodiments, please refer to Figure 1 and Figure 8 , Figure 1 A flowchart illustrating the steps of the vehicle HUD projection jitter prediction and optimization method based on multi-source data fusion provided in this application embodiment. Figure 8 The flowchart of the vehicle HUD projection jitter prediction and optimization method based on multi-source data fusion provided in this application embodiment is shown. The vehicle HUD projection jitter prediction and optimization method in this application embodiment is specifically implemented through steps 100 to 500:

[0077] Step 100: Confirm the optimization parameters, which include the vehicle's motion data, the HUD's vibration data, and the HUD's vibration images during vehicle operation.

[0078] In some embodiments, please refer to Figure 6 , Figure 6 This is a schematic diagram of a vehicle in the vehicle HUD projection jitter prediction and optimization method based on multi-source data fusion provided in the embodiments of this application. The vehicle in this application is equipped with:

[0079] The first attitude sensor 1 is installed on the vehicle chassis to acquire vehicle motion data. Specifically, the first attitude sensor 1 uses a TELESKY WT901C 9-axis attitude sensor kit, collecting data including three-axis acceleration and three-axis angular velocity to comprehensively reflect the vehicle's motion state. By analyzing its acceleration data, dynamic events such as acceleration / deceleration, bumps, and tire-obstacle contact can be identified, which is an important basis for determining whether there are original disturbances causing HUD image jitter. The angular velocity data is used to capture the vehicle's attitude changes in pitch, roll, and yaw directions, assisting in the identification of dynamic behaviors such as steering and tilting, and providing complete attitude information support for jitter modeling.

[0080] The second attitude sensor 2 is positioned at the projection location of the HUD and is used to acquire vibration data of the HUD. Specifically, the second attitude sensor 2 adopts a TELESKY WT901C 9-axis attitude sensor kit, and the data types acquired include three-axis acceleration and three-axis angular velocity, which reflect the actual vibration intensity generated after the vibration is transmitted to the HUD device through the vehicle structure. This data can be used to evaluate the damping effect of the shock absorption system on shaking. At the same time, its angular velocity data is used to capture the attitude disturbance of the HUD in three-dimensional space, which is an important basis for judging changes in image stability.

[0081] Camera 3 is positioned directly in front of the HUD projection screen on the vehicle's dashboard. The image acquisition port of camera 3 faces the projection position of the HUD and is used to simulate the driver's perspective to acquire vibration images of the HUD during vehicle operation. Specifically, camera 3 is a Feynman M1 RGB-D camera 3 from Zhaoguan Electronics.

[0082] Step 200: Obtain a reference image based on the vibration image of the vehicle driving on a road surface that meets the smoothness requirements.

[0083] Understandably, when acquiring reference images, embodiments of this application select high-grade flat roads with smooth surfaces and no obvious depressions to ensure that vehicles will hardly experience bumps or vibrations during driving. Such road conditions can effectively avoid complex interference from the external environment and ensure that the acquired HUD image data can reflect the stable state of the vehicle during driving.

[0084] Specifically, when acquiring the reference image, a Feynman M1 RGB-D camera from Zhaoguan Electronics, mounted directly in front of the HUD projection screen on the vehicle's dashboard, is used. This camera features high-resolution imaging and depth information acquisition capabilities. The camera is securely mounted using a dedicated bracket to ensure that the field of view consistently covers the HUD projection area throughout the entire process. The camera's position is... Figure 6Position 3 in the system is used to simulate the driver's viewing angle of the HUD projection and is continuously powered by the vehicle's power system to ensure the stability and continuity of data acquisition.

[0085] In the data acquisition experiment, the vehicle traveled smoothly at a constant speed, and the camera continuously acquired dynamic images of the HUD projection screen. The entire acquisition process was monitored by designated personnel to ensure that the camera was functioning properly and that the acquired images were unobstructed and free from glare. After acquisition, vibration images of the vehicle traveling on a road surface meeting the smoothness requirements, acquired at multiple time points, were compiled based on chronological order to obtain a reference image. The characterization formula for the reference image includes:

[0086]

[0087] in, For reference image; I s (t1),I s (t2),…,I s (t n ) represent the times at multiple time points t1, t2, ..., t n Vibration images of vehicles traveling on roads that meet the flatness requirements.

[0088] Understandably, these high-quality HUD images, acquired under conditions without significant shaking or vibration, reflect the ideal performance of HUD images under normal vehicle driving conditions. This type of data not only provides an accurate reference standard for the training and performance evaluation of jitter reduction algorithms, but also allows for comparison with jitter data under complex road conditions, systematically analyzing the variation patterns of HUD images under different operating conditions, and helping the model to more effectively distinguish and identify jitter features. By constructing a dataset of stable operating conditions, the scientific rigor, robustness, and practical application value of the jitter prediction and optimization methods are further improved, helping to ensure the reliability and generalization ability of the algorithm in real road environments.

[0089] Step 300: Based on the data of optimized parameters when the vehicle is driving on the test road surface, predict the future vibration image of the HUD, which is the predicted image.

[0090] Specifically, when acquiring the data to be measured, the vehicle travels on a road surface where the smoothness of the road surface is not measured, such as a complex gravel road surface with a large amount of loose gravel and potholes. Under such conditions, the vehicle is prone to bumps and vibrations. The HUD image jitter data collected under such conditions is used to comprehensively characterize the jitter process of the HUD projection.

[0091] In some embodiments, the data for optimizing parameters when organizing vehicles to drive on the test road surface in chronological order can be represented as follows:

[0092] {(X1,(A1,G1,B1,H1)),(X2,(A2,G2,B2,H2)),…,(X T ,(A T G T B T H T ))};

[0093] Among them, X t This represents the vibration image of the HUD at time point t, where t = 1, 2...t, and 1, 2...T are multiple time points ordered in a time series; A t and G t A represents the motion data of the vehicle body at time point t. t This represents the three-axis acceleration data of the vehicle body at time point t, used to characterize the linear disturbance behavior of the vehicle body. G t B represents the three-axis angular velocity data of the vehicle body at time point t, used to characterize the change in the vehicle body's rotational attitude; t and H t B represents the vibration data of the HUD at time point t. t This represents the three-axis acceleration data of the HUD at time point t, used to characterize the vibration intensity of the HUD. t This represents the three-axis angular velocity data of the HUD at time point t, used to characterize the attitude change of the HUD.

[0094] In some embodiments, please refer to Figure 2 and Figure 9 , Figure 2 The flowchart illustrates the steps of predicting future vibration images of the HUD in the vehicle HUD projection jitter prediction and optimization method based on multi-source data fusion provided in this application embodiment. Figure 9 This is a flowchart illustrating the preprocessing and feature extraction of vehicle HUD projection jitter prediction data in the multi-source data fusion-based vehicle HUD projection jitter prediction and optimization method provided in this application embodiment. The method for predicting future vibration images of the HUD is specifically implemented through steps 310 to 350:

[0095] Step 310: The vibration image of the HUD when the vehicle is driving on the test road surface is processed by the ResNet-18 deep residual network model to obtain spatial features.

[0096] Understandably, the ResNet-18 deep residual network model uses residual connections, which helps avoid the gradient vanishing problem common in deep neural network training and can extract richer spatial features.

[0097] In some embodiments, please refer to Figure 3 , Figure 3This is a flowchart illustrating the steps for obtaining spatial features in the vehicle HUD projection jitter prediction and optimization method based on multi-source data fusion provided in this application embodiment. The method for obtaining spatial features in this application embodiment is implemented through steps 311 to 314:

[0098] Step 311: The vibration image of the HUD when the vehicle is driving on the test road surface is processed through the convolutional layer of the ResNet-18 deep residual network model to obtain the convolutional output.

[0099] In some embodiments, the input image is X t X t It is an RGB color image with a size of 224*224*3, where 224*224 represents the width and height of the image, and 3 represents the number of channels. A ResNet-18 model performs convolution operations to extract the original features of the image, with a kernel size of 7*7 and a stride of 2. The final output is:

[0100] F conv1 =RELU(X t *W conv1 +b conv1 );

[0101] Among them, F conv1 For convolution output; ReLU is the corrected linear unit; X t For the input image; W conv1 b represents the weights of the first layer convolution; conv1 is the bias of the first convolutional layer; conv1 is the first convolutional layer.

[0102] The first convolutional layer of ResNet-18 uses 64 7×7 convolutional kernels, resulting in 64 channels in the output feature map. A feature map is the output image generated after a convolution operation. It can be viewed as a spatial representation of the features or patterns extracted from the input image after processing by the convolutional kernels. Each convolutional kernel extracts a set of features from the input image, and the number of output feature maps is equal to the number of convolutional kernels (64), resulting in an output size of 112*112*64.

[0103] Step 312: Process the convolution output through the pooling layer of the ResNet-18 deep residual network model to obtain the pooled output.

[0104] In some embodiments, the convolutional output is passed through a pooling layer with a stride of 2, and the final output is a pooled output F. pool1 pool1 is the first pooling layer, with a size of 56*56*64.

[0105] Step 313: Process the pooling output through the residual block of the deep residual network model ResNet-18 to obtain the residual output.

[0106] In some embodiments, in ResNet-18, each residual block contains two 3×3 convolutional layers. The first layer transforms and extracts the input features, and the second layer extracts and enhances features at depth. Within the residual block, the input is directly added to the output, forming skip connections. This operation allows the network to directly learn the residual, i.e., the difference between the input and output, thereby training deeper networks more effectively.

[0107] In the residual block, the input is F pool1 The output is F res1 The mathematical representation of the residual block is:

[0108] F res1 =E(F pool1 W i )+F pool1 ;

[0109] Among them, F res1 For residual output, res represents residual connection; E represents the combination of two convolutional layer operations in the residual block; W i This represents the weight parameters in the residual block. After the last residual block, the output feature maps of the convolutional layers are retained as key features, which contain rich semantic information.

[0110] Step 314: Perform global average pooling on the residual output to obtain spatial features.

[0111] In some embodiments, global average pooling is performed on the residual output to obtain a fixed-length feature vector, which is a vector of spatial features. The formula is as follows:

[0112]

[0113] H and W: The height and width of the feature map, representing the size of each feature map;

[0114] f i : Represents the pixel index (from 1 to H) in the height direction of the feature map, that is, the vertical pixel position of the feature map;

[0115] f j : Represents the pixel index (from 1 to W) in the width direction of the feature map, that is, the horizontal pixel position of the feature map;

[0116] f k : Represents a specific channel index or feature map index, indicating the k-th feature map currently being processed;

[0117] The pixel value of the k-th channel at position (i,j) in the feature map;

[0118] The average value of the k-th channel is calculated by normalizing the sum of the pixel count H*W.

[0119] γ k : The average pixel value of the k-th channel (features extracted through convolution and pooling).

[0120] Furthermore, after convolution and pooling operations, an activation function (ReLU) is typically added after each convolutional layer in the network to introduce a non-linear transformation, helping the model learn complex spatial features. These non-linear transformations facilitate the gradual construction of the network from low-level features to high-level features.

[0121] The formula for applying the activation function is as follows:

[0122] D t =ReLU(γ) k );

[0123] Where, γ k It is the feature extracted by convolutional and pooling layers. The ReLU function sets all negative values ​​to zero and retains positive values ​​to ensure that non-linear features in the feature map can be effectively learned.

[0124] Understandably, through the convolutional layers, residual modules, pooling layers, and activation functions of the ResNet-18 network, a feature vector D containing rich spatial information is ultimately obtained. t It contains video frame X at that moment. t The spatial features of the input image. The output feature map is typically a small vector that represents high-level semantic information of the input image.

[0125] Step 320: Based on the standard scores, standardize the vehicle's motion data, HUD vibration data, and spatial characteristics when the vehicle is driving on the test road surface to obtain standardized data.

[0126] Understandably, the core purpose of standardization is to uniformly convert features of all dimensions into a normal distribution with a mean of 0 and a standard deviation of 1, so that all types of features can be fairly used by the model under the same scale, avoiding training instability or model bias problems caused by differences in numerical scale.

[0127] In some embodiments, the formula for representing standardized data includes:

[0128]

[0129] Among them, A T This represents the three-axis acceleration data of the vehicle body at time point T, used to characterize the linear disturbance behavior of the vehicle body; G TB represents the three-axis angular velocity data of the vehicle body at time point T, used to characterize the change in the vehicle body's rotational attitude; T This represents the three-axis acceleration data of the HUD at time point T, used to characterize the vibration intensity of the HUD; H T This represents the three-axis angular velocity data of the HUD at time point T, used to characterize the attitude change of the HUD; D t μ represents the spatial characteristics at time point T. A This represents the mean of the vehicle's three-axis acceleration data at multiple time points; μ G This represents the mean of the three-axis angular velocity data of the vehicle body at multiple time points; μ B This represents the mean of the three-axis acceleration data of the HUD at multiple time points; μ H This represents the mean of the three-axis angular velocity data of the HUD at multiple time points; μ D σ represents the mean of spatial characteristics at multiple time points; A σ represents the standard deviation of the vehicle's three-axis acceleration data at multiple time points. G σ represents the standard deviation of the vehicle's three-axis angular velocity data at multiple time points. B σ represents the standard deviation of the three-axis acceleration data of the HUD at multiple time points. H σ represents the standard deviation of the three-axis angular velocity data of the HUD at multiple time points. D A′ represents the standard deviation of spatial characteristics at multiple time points. T This represents the standardized three-axis acceleration data of the vehicle body at time point T; G′ T This represents the standardized three-axis angular velocity data of the vehicle body at time point T; B′ T This represents the standardized three-axis acceleration data of the HUD at time point T; H T This represents the standardized three-axis angular velocity data of the HUD at time point T; D t This represents the standardized spatial features at time point T. Understandably, through standardization, the mean of all data is 0 and the standard deviation is 1. This allows data from different dimensions to be processed on the same scale, avoiding the negative impact of excessively large or small numerical ranges of a particular feature on the model training process.

[0130] Step 330: Classify the standardized data from multiple time points according to the type of parameters, and sort the standardized data of each parameter based on the time series to obtain time series data.

[0131] Understandably, after standardization, all data is converted into a time-series format to provide rich spatiotemporal information for subsequent spatiotemporal feature learning models. The time-series format of the data means that the input at each time step t includes a fusion of image features and sensor data, helping the model capture the patterns of data change over time.

[0132] In some embodiments, the representation formula for time series data includes:

[0133] {D1′,D2′,…,D′ T},{A′1,A′2,…,A′ T},{G1′,G2′,…,G′ T},{B1′,B2′,…,B′ T},{H1′,H2′,…,H′ T};

[0134] in:

[0135] {D1′,D2′,…,D′ T} represents the standardized feature vectors at time points 1, 2, ..., T, ensuring that image and motion sensor data can be uniformly represented in the model, which is beneficial for multi-source information fusion.

[0136] {A′1,A′2,…,A′ T} represents the standardized acceleration data from the first sensor at time points 1, 2, ..., T, so that the model learning is not affected by the data units and value range;

[0137] {G1′,G2′,…,G′ T} represents the standardized angular velocity data from the first sensor at time points 1, 2, ..., T, enabling the model to more accurately understand the vehicle's attitude dynamics and avoiding interference from inconsistent feature scales on learning.

[0138] {B1′,B2′,…,B′ T} represents the standardized HUD acceleration in the second sensor at time points 1, 2, ..., T, which helps the model to more effectively distinguish the HUD response characteristics under different vibration conditions;

[0139] {H1′,H2′,…,H′ T} represents the standardized angular velocity in the second sensor at time points 1, 2, ..., T, which helps the model capture high-frequency perturbations in HUD attitude changes and improves the accuracy of image stability judgment.

[0140] Understandably, by effectively fusing and preprocessing multi-source data, different types of data, such as vehicle video data, first sensor data, and second sensor data, can be transformed into a unified format, and spatiotemporal feature learning can be performed on this basis. This will help the model learn the potential correlations between data sources, thereby enabling it to accurately predict potential HUD image jitter during vehicle operation and providing a solid data foundation for training the jitter removal algorithm.

[0141] Step 340: Based on the LSTM long short-term memory network, perform temporal feature learning on the temporal data to obtain the temporal features of the hidden state. The temporal features are used to characterize the motion data of the vehicle body and the vibration data of the HUD, as well as the spatiotemporal relationship with the vibration image.

[0142] Understandably, LSTM (Long Short-Term Memory) networks are specifically designed to capture long-term dependencies in temporal data. This application uses LSTM to process the spatiotemporal relationships between video data and sensor data. The core of LSTM is to control the flow and updating of information through a gating mechanism, thereby remembering important information and discarding irrelevant information.

[0143] In some embodiments, the LSTM long short-term memory network includes an input gate i t The input gate determines the update level of the current input information. The calculation formula is as follows:

[0144] i t =σ(W i [h t-1 ,x t ]+b i );

[0145] Among them, h t-1 It is the hidden state of the previous time step t-1, that is, the temporal characteristics of the previous time step t-1; x t It is the input for the current time step t; W i b is the weight matrix of the input gate; i σ is the bias term of the input gate; σ is the sigmoid activation function, whose output value is in the range [0,1], and determines what proportion of information needs to be updated.

[0146] In some embodiments, LSTM long short-term memory networks include a forgetting gate. t The forgetting gate determines which memories should be forgotten. The calculation formula is as follows:

[0147] f t =σ(W f [h t-1 ,x t ]+b f );

[0148] W f This is the weight matrix of the forget gate; b f It is the bias term of the forget gate.

[0149] In some embodiments, an LSTM (Long Short-Term Memory) network includes an output gate. t The output gate determines the output information at the current moment. The calculation formula is as follows:

[0150] o t =σ(Wo [h t-1 ,x t ]+b o );

[0151] W o This is the weight matrix of the output gate; b o It is the bias term of the output gate.

[0152] In some embodiments, the LSTM long short-term memory network includes candidate memory units. Candidate memory units generate candidate memories for the current time step, calculated using the following formula:

[0153]

[0154] W C It is the weight matrix of the candidate memory units; b C is the bias term for candidate memory units; tanh is the hyperbolic tangent activation function.

[0155] In some embodiments, an LSTM long short-term memory network includes a memory unit (C t At each time step, the LSTM updates the memory cells through the input gate, forget gate, and candidate memory cells. The calculation formula is as follows:

[0156]

[0157] C t-1 It is the memory unit of the previous time step t-1, used to store long-term memory;

[0158] f t It is the Gate of Oblivion, which determines how much of the previous memories are retained;

[0159] i t It is the input gate, which determines how much information is retained from the current input;

[0160] C t It is a candidate memory unit at the current moment, representing new memory content.

[0161] In some embodiments, the LSTM (Long Short-Term Memory) network generates the hidden state h at the current time step through the output gate. t It incorporates spatiotemporal characteristics. The calculation formula is as follows:

[0162] h t =o t ·tanh(C t );

[0163] o t It is an output gate used to control the update of the hidden state; tanh(C t) is the activation output of the current memory cell, used to generate the hidden state at the current moment.

[0164] Understandably, the LSTM (Long Short-Term Memory) network, through recursive computation over multiple time steps, captures not only the spatial features of image data but also the temporal features of sensor data. Ultimately, the LSTM network uses hidden states h... t Learn the spatiotemporal relationship between image and sensor data. The hidden state h at each time step. t It includes temporal characteristics of image information (such as vehicle position, obstacles, road conditions, etc.) and sensor data (such as acceleration, vehicle speed, projection device displacement, etc.).

[0165] Step 350: Predict future vibration images of the HUD based on temporal features.

[0166] In some embodiments, please refer to Figure 10 , Figure 10 The flowchart shows the program of LSTM Long Short-Term Memory Network predicting jitter images in the vehicle HUD projection jitter prediction and optimization method based on multi-source data fusion provided in the embodiments of this application. This application uses LSTM Long Short-Term Memory Network to predict the jitter change trend of HUD projection images in the future, that is, to output a sequence of time-series jitter images of multiple consecutive frames.

[0167] Specifically, the current input sequence is R1, R2, ..., R T , where R T This represents the fused input vector (containing image features and sensor data) at time T. The LSTM (Long Short-Term Memory) network, after temporal modeling, obtains the hidden state h at the current time step. t And based on this state, the hidden state {h} can be recursively predicted for the next k time steps. t+1 ,...,h t+k For each future time point t+i (i = 1, 2, ..., k), the hidden state h of the LSTM Long Short-Term Memory network is... t+i All inputs are fed into the regression layer to generate vibration images at time point t+i. The formulas representing the vibration image at time point t+i include:

[0168]

[0169] Among them, W h b is the weight matrix of the regression layer; h This is a bias term used to offset the weighted result, helping the model fit more complex outputs; The vibration image at a future time point t+i; h t+iThe hidden state h is the output of the LSTM (Long Short-Term Memory) network at a future time point t+i. It contains current and historical time-series information to characterize the jitter features of the HUD image at future time steps. For each future time step t+i (i ranges from 1 to k), the LSTM network outputs a hidden state h. t+i It represents predictive information for future moments.

[0170] Therefore, the LSTM (Long Short-Term Memory) network model can ultimately output a predicted temporal jitter image sequence of length k:

[0171]

[0172] Predicted image for each frame These data are obtained through recursive prediction based on the integration of current and historical input features, and can reflect the jitter development trend of HUD images over a future period. This data can provide more comprehensive and forward-looking data support for subsequent jitter compensation, helping to improve the real-time performance and stability of the entire system.

[0173] Step 400: Obtain the displacement compensation value of the predicted image based on the reference image and the predicted image.

[0174] In some embodiments, please refer to Figure 4 and Figure 11 , Figure 4 This is a flowchart illustrating the steps involved in obtaining the displacement compensation value of the predicted image in a vehicle HUD projection jitter prediction and optimization method based on multi-source data fusion, as provided in an embodiment of this application. Figure 11 This is a flowchart of the reinforcement learning-based HUD image displacement compensation procedure in the vehicle HUD projection jitter prediction and optimization method based on multi-source data fusion provided in the embodiments of this application. The method for obtaining the displacement compensation value of the predicted image in the embodiments of this application is specifically implemented through steps 410 to 420:

[0175] Step 410: Obtain the displacement compensation value at the previous time point, and combine it with the reference image and the predicted image to obtain the state vector at the current time point. The representation formula of the state vector includes:

[0176]

[0177] Among them, S t This is the state vector at the current time point t; The predicted image for time points t+1 to t+k; For reference image; Δ t-1 The displacement compensation value at the previous time point t-1, Δ t-1 =(Δx) t-1 ,Δy t-1 ), Δx t-1Δy represents the pixel-level compensation amount applied in the horizontal direction at the previous time point t-1. t-1 This represents the pixel-level compensation amount applied in the vertical direction at the previous time point t-1;

[0178] Understandably, to measure the target of jitter compensation, the system uses an ideal HUD image sequence acquired under stable operating conditions as a reference. At each decision-making time, the state not only contains the predicted jitter image sequence output by the current LSTM Long Short-Term Memory network, but also... It will also extract the target smoothed image subsequence corresponding to its time step. This is so that subsequent compensation actions will have a clear reference target.

[0179] Furthermore, considering the continuity of the compensation strategy and the effect of dynamic optimization, the state vector also incorporates historical displacement compensation information. Let Δ t-1 =(Δx) t-1 ,Δy t-1 Let Δ0 be the displacement compensation amount applied in the previous time step. If there is no compensation at the initial time step, then let Δ0 = (0,0). By incorporating the compensation history into the state space, reinforcement learning can better capture the long-term impact of compensation behavior on image stability, thereby improving the global optimization capability of the compensation strategy.

[0180] Step 420: Based on the state vector, predict the jitter level and direction of the HUD at the current time point, and combine this with the displacement compensation value from the previous time point to obtain the displacement compensation value at the current time point. The formula representing the displacement compensation value at the current time point includes:

[0181] Δ t =(Δx) t ,Δy t );

[0182] Where, Δ t The displacement compensation value at the current time point t; Δx t Δy represents the pixel-level compensation amount applied in the horizontal direction at the current time point t. t This represents the pixel-level compensation amount applied in the horizontal direction at the current time point t.

[0183] Step 500: Optimize the predicted image based on the displacement compensation value.

[0184] Understandably, this compensation action Δ t It is based on the current state S t The goal of the learned optimal action output is to compensate the predicted image sequence to a position consistent with the target reference image as much as possible through spatial displacement operations, thereby achieving smooth image display.

[0185] Furthermore, Δxt and Δy t These represent the pixel-level displacement compensation amounts applied along the horizontal and vertical directions at the current moment, respectively. For a predicted jitter image with a sequence length of k... Action A t You can choose to apply the same compensation amount to all future frames, or you can design it as a multi-step action sequence {(Δx)} depending on the actual task. t+1 ,Δy t+1 ),…,(Δx t+k ,Δy t+k To enhance flexibility.

[0186] Whenever the agent outputs a compensation action Δ t Subsequently, the environment will apply this action to the future time-series jittery image sequence, specifically by performing corresponding translation adjustments to the spatial coordinates of each frame. The motion-compensated HUD image sequence can be denoted as:

[0187]

[0188] Here, Shift(·) represents the image translation operation, which means moving each pixel in the image from position (x, y) to a new position (x + Δx). t ,y+Δy t ). Where Δx t and Δy t These represent the translation amounts in the horizontal and vertical directions, respectively. That is, moving each pixel (x, y) in the original image to the new position (x + Δx). t ,y+Δy t ). This is the adjusted predicted jitter image, which is the result after applying a translation compensation operation to the original predicted jitter image. Through the translation operation, each pixel of the image is moved to a new position, thereby further optimizing the image stability.

[0189] The effectiveness of the action will be quantitatively evaluated in the subsequent reward function design. Ideally, after action A... t Adjusted and compensated jitter image sequence Able to approximate the target smooth image sequence as closely as possible This design ensures a close connection between the action space and the state space—only by fully understanding the dynamic jitter of the current and future images can the agent output the most targeted compensation strategy.

[0190] By naturally connecting the action space with the aforementioned state space, the reinforcement learning agent can achieve accurate perception of the future jitter trend of the HUD image and dynamically output the optimal displacement compensation operation, effectively improving the stability and visual experience of the HUD image under various complex working conditions.

[0191] In some embodiments, please refer to Figure 5 , Figure 5 The flowchart illustrates the steps of providing reward feedback to the optimized predicted image in the vehicle HUD projection jitter prediction and optimization method based on multi-source data fusion provided in this application embodiment. Specifically, the method for providing reward feedback to the optimized predicted image in this application embodiment is implemented through steps 610 to 620:

[0192] Step 610: Determine the reward function, which includes pixel-level difference metrics and structural similarity metrics.

[0193] In some embodiments, the core idea of ​​the reward function is: if action Δ t A higher reward is given if the compensated image sequence is highly consistent with the target smoothed image; conversely, if the compensation effect is poor, the reward value is reduced or even negative. To reflect the multidimensional effect of image compensation, the reward function combines two indicators: pixel-level difference (MSE) and structural similarity (SSIM).

[0194] Step 620: Obtain the reward feedback between the optimized predicted image and the reference image based on the reward function, and adjust the optimized predicted image based on the reward feedback; the formula for the reward function includes:

[0195]

[0196] Among them, R t α is the reward function, used to characterize reward feedback; MSE is the pixel-level difference index; SSIM is the structural similarity index, used to characterize brightness, contrast, and structural similarity; α is the adjustable weight of the pixel-level difference index; β is the adjustable weight of the structural similarity index. The optimized predicted image; I s (t+i) is the target smoothed image subsequence corresponding to the time step of the reference image.

[0197] Understandably, through this reward function design, the RL agent can continuously adjust and optimize its strategy based on the immediate reward brought by each compensation action during continuous interaction with the environment. Positive feedback drives the agent to continuously output better displacement compensation operations, achieving adaptive and precise control of HUD image jitter. Furthermore, to ensure that the agent can effectively converge and obtain the optimal image compensation strategy, this invention further sets strict performance evaluation indicators and training stopping conditions. Specifically, during the training process, the system uses the mean square error (MSE) and structural similarity (SSIM) between the compensated HUD image and the target smoothed image as the core performance evaluation indicators. After each complete training episode, the system calculates the mean MSE and SSIM of the compensation results to dynamically evaluate the optimization effect of the current strategy.

[0198] To achieve an efficient and stable training process, the system sets the following multiple stopping conditions:

[0199] First, the cumulative reward is used as the primary convergence criterion. Specifically, within N consecutive training rounds (e.g., N=100), if the average cumulative reward improvement is lower than a preset threshold ε (set to ε=0.005 in this patent), the strategy is considered to have essentially converged, and the training process can be terminated early. Simultaneously, if, during training, the average MSE of the compensated HUD image decreases to a preset target (less than 0.005), or the average SSIM increases to above a set threshold (greater than 0.98), this can also be used as a basis for terminating training. These image-level metrics comprehensively reflect the model's improvement effect on HUD image stability, ensuring that the compensation results meet the requirements of practical applications.

[0200] Furthermore, to prevent the model from getting stuck in local optima or overfitting, a maximum number of rounds (5000 rounds) is set as a hard stopping condition. When training reaches the limit, training will automatically end regardless of whether the reward level is met, and the best-performing policy parameters will be saved.

[0201] By setting the aforementioned multiple metrics and stopping conditions, the reinforcement learning agent can continuously optimize the HUD image jitter compensation strategy while ensuring model performance and training efficiency, ultimately achieving high-quality and stable output of HUD projection images under various complex conditions. This not only enhances the driver's visual experience but also provides a solid guarantee for driving safety.

[0202] In some embodiments, the model training in the learning phase is completed using the vehicle HUD projection jitter prediction and optimization method based on multi-source data fusion provided in this application, resulting in an image prediction and compensation model. Then, in the application phase, jitter prediction data provided by the patent (application number 2025107474454, entitled "A Method for Multi-Source Jitter Prediction for Driver's HUD Projectors") is input into the image prediction and compensation model, and the compensated displacement of the image is output in real time to perform frame-level correction on the HUD image. The entire process requires no additional training or real-time evaluation, directly outputting a stable and clear HUD image, significantly improving the reliability of driving information display and user experience, and effectively meeting the needs of practical engineering deployment for efficiency and practicality. Understandably, addressing the problem of vehicle HUD projection easily generating jitter under complex road conditions, affecting driving safety and visual experience, this application adopts a two-stage design of "learning phase + application phase" to achieve a complete closed loop from model training to practical application. During the learning phase, the system collects multimodal data from the vehicle, such as video, acceleration, and angular velocity information. It utilizes a deep learning model to extract spatial features from the image and model temporal features, accurately predicting future HUD image jitter trends. Simultaneously, a reinforcement learning mechanism is introduced, combining ideal smooth images with historical compensation behaviors to construct state and action spaces, forming an adaptive compensation strategy. The compensation effect is comprehensively evaluated using pixel-level error (MSE) and structural similarity (SSIM), continuously optimizing model performance. In the application phase, the system directly calls the pre-trained image prediction and compensation model. Based on the jitter prediction data provided in patent application number 2025107474454 ("A Method for Multi-Source Jitter Prediction for Driver's HUD Projectors"), it outputs real-time image displacement compensation, achieving frame-by-frame correction and stable display of the HUD image, meeting the real-time and robustness requirements of actual driving environments.

[0203] Specifically, in the application phase, the known jitter prediction data from the patent application (application number 2025107474454, titled "A Method for Multi-Source Predictive Jitter of a Driver's HUD Projector") is directly used to correct jitter in the vehicle's HUD projection image using the image prediction and compensation model trained during the learning phase. The specific process is as follows: First, the system acquires displacement and attitude change data of the HUD projection device over a future period, predicted by the previous method, from the second attitude sensor 2. This data accurately depicts the jitter trend of the HUD projection image during driving in a time-series format. Then, the compensation module inputs this known jitter prediction data into the image prediction and compensation model trained during the learning phase, according to a unified time-series input format. Based on the input jitter amplitude and direction, and combined with the compensation rules learned from historical data, the model automatically calculates the spatial translation amount to be applied to each frame of the HUD image, such as pixel-level horizontal and vertical displacement, and applies this compensation amount to the original HUD projection image. During the compensation process, no additional data collection, model fine-tuning, or real-time effect evaluation is required. All compensation operations are based on known jitter prediction data and existing model parameters. Through frame-by-frame compensation processing, the corrected HUD projection image can eliminate the original spatial jitter to the greatest extent possible, restoring image stability and clarity under ideal conditions. Ultimately, the compensated HUD image provides drivers with clearer and more stable information display, significantly improving driving safety and user experience. The entire application process is efficient and automated, requiring only jitter prediction data input to complete real-time or batch compensation of HUD images, greatly simplifying the complexity of actual engineering deployment.

[0204] Understandably, the vehicle HUD projection jitter prediction and optimization method based on multi-source data fusion provided in the embodiments of this application includes: identifying optimization parameters, which include vehicle motion data, HUD vibration data, and HUD vibration images during vehicle operation; obtaining a reference image based on the vibration images of the vehicle driving on a road surface meeting smoothness requirements; predicting the future vibration image of the HUD based on the data of the optimization parameters when the vehicle is driving on a test road surface, resulting in a predicted image; obtaining a displacement compensation value for the predicted image based on the reference image and the predicted image; and optimizing the predicted image based on the displacement compensation value. This application, by predicting jitter and adjusting the future image displayed by the vehicle HUD in real time, solves the image jitter problem caused by vehicle vibration and uneven road surfaces, improves the stability and clarity of the HUD image, thereby enhancing the driver's visual experience and driving safety.

[0205] Accordingly, please refer to Figure 7 , Figure 7This is a module connection diagram of a vehicle HUD projection jitter prediction and optimization system based on multi-source data fusion provided in this application embodiment. The vehicle HUD projection jitter prediction and optimization system based on multi-source data fusion provided in this application embodiment includes:

[0206] The parameter determination module 10 is used to confirm the optimization parameters, including the vehicle body motion data, HUD vibration data, and HUD vibration images during vehicle operation.

[0207] Reference image module 20 is used to acquire a reference image based on the vibration image of the vehicle driving on a road surface that meets the smoothness requirements;

[0208] The prediction image module 30 is used to predict future vibration images of the HUD based on data of optimized parameters when the vehicle is driving on the test road surface.

[0209] The compensation acquisition module 40 is used to acquire the displacement compensation value of the prediction image based on the reference image and the prediction image.

[0210] Image compensation module 50 is used to optimize the predicted image based on displacement compensation value.

[0211] In some embodiments, the reference image module 20 is specifically used for:

[0212] Based on time sequence, vibration images of vehicles traveling on roads meeting smoothness requirements, collected at multiple time points, are arranged to obtain a reference image. The representation formula for the reference image includes:

[0213]

[0214] in, For reference image; I s (t1),I s (t2),…,I s (t n ) represent the times at multiple time points t1, t2, ..., t n Vibration images of vehicles traveling on roads that meet the flatness requirements.

[0215] In some embodiments, the image prediction module 30 is specifically used for:

[0216] The vibration images of the HUD when the vehicle is driving on the test road are processed by the ResNet-18 deep residual network model to obtain spatial features;

[0217] Standardized data is obtained by standardizing the vehicle's motion data, HUD vibration data, and spatial characteristics when the vehicle is driving on the test road surface based on standard scores.

[0218] Standardized data from multiple time points are categorized according to parameter type, and the standardized data of each parameter are sorted based on time series to obtain time series data;

[0219] Temporal features of hidden states are learned by using LSTM (Long Short-Term Memory) network to characterize the motion data of the vehicle body and the vibration data of the HUD, as well as the spatiotemporal relationship with the vibration images.

[0220] Predict future vibration images of the HUD based on temporal features.

[0221] In some embodiments, the image prediction module 30 is specifically used for:

[0222] The vibration image of the HUD when the vehicle is driving on the test road is processed through the convolutional layer of the ResNet-18 deep residual network model to obtain the convolutional output;

[0223] The convolutional output is processed through the pooling layer of the ResNet-18 deep residual network model to obtain the pooled output;

[0224] The pooling output is processed through residual blocks of the ResNet-18 deep residual network model to obtain residual output;

[0225] Global average pooling is performed on the residual output to obtain spatial features.

[0226] In some embodiments, the image prediction module 30 is specifically used for:

[0227] For each future time point t+i (i = 1, 2, ..., k), the hidden state h of the LSTM Long Short-Term Memory network is... t+i All inputs are fed into the regression layer to generate vibration images at time point t+i. The formulas representing the vibration image at time point t+i include:

[0228]

[0229] Among them, W h b is the weight matrix of the regression layer; h For bias terms; The vibration image at a future time point t+i; h t+i It is the hidden output state of the LSTM Long Short-Term Memory network at a future time point t+i.

[0230] In some embodiments, the compensation acquisition module 40 is specifically used for:

[0231] Obtain the displacement compensation value from the previous time point, and combine it with the reference image and the predicted image to obtain the state vector at the current time point. The formula for representing the state vector includes:

[0232]

[0233] Among them, S t This is the state vector at the current time point t; The predicted image for time points t+1 to t+k; For reference image; Δ t -1 represents the displacement compensation value at the previous time point t-1, Δ t-1 =(Δx) t-1 ,Δy t-1 ), Δx t-1 Δy represents the pixel-level compensation amount applied in the horizontal direction at the previous time point t-1. t-1 This represents the pixel-level compensation amount applied in the vertical direction at the previous time point t-1;

[0234] Based on the state vector, the jitter level and direction of the HUD at the current time point are predicted. Combined with the displacement compensation value from the previous time point, the displacement compensation value at the current time point is obtained. The formula representing the displacement compensation value at the current time point includes:

[0235] Δ t =(Δx) t ,Δy t );

[0236] Where, Δ t The displacement compensation value at the current time point t; Δx t Δy represents the pixel-level compensation amount applied in the horizontal direction at the current time point t. t This represents the pixel-level compensation amount applied in the horizontal direction at the current time point t.

[0237] In some embodiments, the vehicle HUD projection jitter prediction and optimization system based on multi-source data fusion is specifically used for:

[0238] Determine the reward function, which includes pixel-level difference metrics and structural similarity metrics;

[0239] The reward function is used to obtain the reward feedback between the optimized predicted image and the reference image, and the optimized predicted image is adjusted based on the reward feedback; the formula for the reward function includes:

[0240]

[0241] Among them, R tα is the reward function, used to characterize reward feedback; MSE is the pixel-level difference index; SSIM is the structural similarity index, used to characterize brightness, contrast, and structural similarity; α is the adjustable weight of the pixel-level difference index; β is the adjustable weight of the structural similarity index. The optimized predicted image; I s (t+i) is the target smoothed image subsequence corresponding to the time step of the reference image.

[0242] This application has provided a detailed description of a method and system for predicting and optimizing vehicle HUD projection jitter based on multi-source data fusion, as provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for predicting and optimizing vehicle HUD projection jitter based on multi-source data fusion, characterized in that, include: The optimization parameters are confirmed, including the vehicle's motion data, the HUD's vibration data, and the HUD's vibration images during vehicle operation; A reference image is obtained based on the vibration images of the vehicle driving on a road surface that meets the smoothness requirements; Based on the data of optimized parameters when the vehicle is driving on the test road surface, the future vibration image of the HUD is predicted, which is the predicted image; The displacement compensation value of the predicted image is obtained based on the reference image and the predicted image; The predicted image is optimized based on the displacement compensation value.

2. The method for predicting and optimizing vehicle HUD projection jitter based on multi-source data fusion according to claim 1, characterized in that, The step of obtaining the reference image includes: Based on chronological order, vibration images of vehicles traveling on roads meeting smoothness requirements, collected at multiple time points, are arranged to obtain a reference image. The representation formula of the reference image includes: in, The reference image; I s (t1),I s (t2),…,I s (t n ) represent the times at multiple time points t1, t2, ..., t n The vibration images collected when the vehicle is driving on a road surface that meets the flatness requirements.

3. The method for predicting and optimizing vehicle HUD projection jitter based on multi-source data fusion according to claim 1, characterized in that, The step of predicting the future vibration images of the HUD includes: The vibration images of the HUD when the vehicle is driving on the test road are processed by the ResNet-18 deep residual network model to obtain spatial features; Based on standard scores, the motion data of the vehicle body, the vibration data of the HUD and the spatial features when the vehicle is driving on the test road surface are standardized to obtain standardized data; Standardized data from multiple time points are categorized according to parameter type, and the standardized data of each parameter are sorted based on time series to obtain time series data; The temporal features of the time-series data are learned by using an LSTM (Long Short-Term Memory) network to obtain the temporal features of the hidden state. The temporal features are used to characterize the motion data of the vehicle body and the vibration data of the HUD, as well as the spatiotemporal relationship with the vibration image. The future vibration images of the HUD are predicted based on the temporal features.

4. The method for predicting and optimizing vehicle HUD projection jitter based on multi-source data fusion according to claim 3, characterized in that, The steps for obtaining spatial features include: The vibration image of the HUD when the vehicle is driving on the test road is processed through the convolutional layer of the ResNet-18 deep residual network model to obtain the convolutional output; The convolutional output is processed through the pooling layer of the ResNet-18 deep residual network model to obtain the pooled output; The pooling output is processed through the residual blocks of the deep residual network model ResNet-18 to obtain the residual output; The spatial features are obtained by performing global average pooling on the residual output.

5. The method for predicting and optimizing vehicle HUD projection jitter based on multi-source data fusion according to claim 3, characterized in that, The formula for representing the standardized data includes: Among them, A T This represents the three-axis acceleration data of the vehicle body at time point T, used to characterize the linear disturbance behavior of the vehicle body; G T B represents the three-axis angular velocity data of the vehicle body at time point T, used to characterize the change in the vehicle body's rotational attitude; T This represents the three-axis acceleration data of the HUD at time point T, used to characterize the vibration intensity of the HUD; H T This represents the three-axis angular velocity data of the HUD at time point T, used to characterize the attitude change of the HUD; D t The spatial features at time point T; μ A This represents the mean of the vehicle's three-axis acceleration data at multiple time points; μ G This represents the mean of the three-axis angular velocity data of the vehicle body at multiple time points; μ B This represents the mean of the three-axis acceleration data of the HUD at multiple time points; μ H This represents the mean of the three-axis angular velocity data of the HUD at multiple time points; μ D σ represents the mean of the spatial features at multiple time points; A σ represents the standard deviation of the vehicle's three-axis acceleration data at multiple time points. G σ represents the standard deviation of the vehicle's three-axis angular velocity data at multiple time points. B σ represents the standard deviation of the three-axis acceleration data of the HUD at multiple time points. H σ represents the standard deviation of the three-axis angular velocity data of the HUD at multiple time points. D A′ represents the standard deviation of the spatial characteristics at multiple time points. T This represents the standardized three-axis acceleration data of the vehicle body at time point T; G′ T This represents the standardized three-axis angular velocity data of the vehicle body at time point T; B′ T This represents the standardized three-axis acceleration data of the HUD at time point T; H T This represents the standardized three-axis angular velocity data of the HUD at time point T; D t This represents the standardized spatial features at time point T.

6. The method for predicting and optimizing vehicle HUD projection jitter based on multi-source data fusion according to claim 3, characterized in that, The step of predicting future vibration images of the HUD based on the temporal features includes: For each future time point t+i (i = 1, 2, ..., k), the hidden state h of the LSTM Long Short-Term Memory network is... t+i All inputs are fed into the regression layer to generate vibration images at time point t+i. The formula for representing the vibration image at time point t+i includes: Among them, W h b is the weight matrix of the regression layer; h For bias terms; The vibration image at a future time point t+i; h t+i It is the hidden output state of the LSTM Long Short-Term Memory network at a future time point t+i.

7. The method for predicting and optimizing vehicle HUD projection jitter based on multi-source data fusion according to claim 1, characterized in that, The step of obtaining the displacement compensation value of the predicted image based on the reference image and the predicted image includes: Obtain the displacement compensation value from the previous time point, and combine it with the reference image and the predicted image to obtain the state vector at the current time point. The representation formula of the state vector includes: Among them, S t This is the state vector at the current time point t; The predicted image for time points t+1 to t+k; The reference image; Δ t-1 The displacement compensation value at the previous time point t-1, Δ t-1 =(Δx) t-1 ,Δy t-1 ), Δx t-1 Δy represents the pixel-level compensation amount applied in the horizontal direction at the previous time point t-1. t-1 This represents the pixel-level compensation amount applied in the vertical direction at the previous time point t-1; Based on the state vector, the jitter level and direction of the HUD at the current time point are predicted, and combined with the displacement compensation value at the previous time point, the displacement compensation value at the current time point is obtained. The formula representing the displacement compensation value at the current time point includes: D t =(Δx t ,Dy t ); Where, Δ t The displacement compensation value at the current time point t; Δx t Δy represents the pixel-level compensation amount applied in the horizontal direction at the current time point t. t This represents the pixel-level compensation amount applied in the horizontal direction at the current time point t.

8. The method for predicting and optimizing vehicle HUD projection jitter based on multi-source data fusion according to claim 1, characterized in that, Also includes: Determine the reward function, which includes a pixel-level difference index and a structural similarity index; The reward feedback between the optimized predicted image and the reference image is obtained based on the reward function, and the optimized predicted image is adjusted based on the reward feedback. The formula for representing the reward function includes: Among them, R t The reward function is used to characterize the reward feedback; MSE is the pixel-level difference index; SSIM is the structural similarity index used to characterize brightness, contrast, and structural similarity; α is the adjustable weight of the pixel-level difference index; β is the adjustable weight of the structural similarity index. The optimized predicted image; I s (t+i) is the target smoothed image subsequence corresponding to the time step of the reference image.

9. The method for predicting and optimizing vehicle HUD projection jitter based on multi-source data fusion according to claim 1, characterized in that, The vehicle is equipped with: A first attitude sensor (1) is disposed on the chassis of the vehicle and is used to acquire motion data of the vehicle body; The second attitude sensor (2) is disposed at the projection position of the HUD and is used to acquire the vibration data of the HUD; The camera (3) has its image acquisition port facing the projection position of the HUD, and is used to acquire vibration images of the HUD during vehicle operation.

10. A vehicle HUD projection jitter prediction and optimization system based on multi-source data fusion, characterized in that, include: The parameter determination module (10) is used to confirm the optimization parameters, including the motion data of the vehicle body, the vibration data of the HUD and the vibration image of the HUD during vehicle operation; Reference image module (20), the reference image module (20) is used to obtain a reference image based on the vibration image of the vehicle when driving on a road surface that meets the flatness requirements; The prediction image module (30) is used to predict the future vibration image of the HUD based on the data of the optimized parameters when the vehicle is driving on the test road surface, which is the prediction image; The compensation acquisition module (40) is used to acquire the displacement compensation value of the prediction image based on the reference image and the prediction image; Image compensation module (50) is used to optimize the predicted image based on the displacement compensation value.

Images