Methods, apparatus, electronic devices and storage media for generating vehicle-mounted holographic images

Abstract

This application relates to a method, apparatus, electronic device, and storage medium for generating in-vehicle holographic images. The method includes: acquiring user behavior data, current vehicle operating condition data, and current in-vehicle imaging adjustment data; determining a preliminary user intent based on the user behavior data, wherein the preliminary user intent indicates the user's control needs for the vehicle; if the vehicle operating condition data meets the safe operating condition execution conditions corresponding to the preliminary user intent, then the preliminary user intent is taken as a valid user intent, wherein the safe operating condition execution conditions limit the range of vehicle driving conditions corresponding to the safe execution of the preliminary user intent; generating calibration parameters based on the in-vehicle imaging adjustment data, wherein the calibration parameters are used to adjust the quality of the in-vehicle imaging; and generating an in-vehicle holographic image based on the valid user intent and the calibration parameters. This application improves driving safety in in-vehicle holographic interaction scenarios.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and in particular to a method, apparatus, electronic device and storage medium for generating vehicle-mounted holographic images. Background Technology

[0002] With the continuous development and iteration of intelligent vehicle technology, in-vehicle interaction methods are gradually upgrading towards a buttonless, immersive interactive experience. Holographic projection technology, with its advantages of overcoming the spatial limitations of traditional flat displays and effectively improving the utilization of in-vehicle space, has been gradually applied to many in-vehicle interaction scenarios such as central control, gear shift prompts, and navigation displays, becoming an important development direction in the field of in-vehicle human-machine interaction. However, in the actual application of in-vehicle scenarios, related in-vehicle holographic projection technologies still struggle to adapt to the usage specifications of driving scenarios and meet the driving safety requirements of the in-vehicle environment. Summary of the Invention

[0003] This application provides a method, apparatus, electronic device, and storage medium for generating vehicle-mounted holographic images to address the issue of low driving safety in vehicle-mounted holographic interaction scenarios.

[0004] In a first aspect, this application provides a method for generating a vehicle-mounted holographic image, the method comprising: Acquire user behavior data, current vehicle operating condition data, and current vehicle imaging adjustment data, wherein the user behavior data is used to indicate the user's interactive control behavior towards the vehicle; A preliminary user intent is determined based on the user behavior data, wherein the preliminary user intent is used to indicate the user's control needs for the vehicle; If the vehicle operating condition data meets the operating condition safety execution conditions corresponding to the preliminary user intent, then the preliminary user intent is regarded as a valid user intent, wherein the operating condition safety execution conditions are used to limit the range of vehicle driving operating conditions corresponding to the safe execution of the preliminary user intent. Calibration parameters are generated based on the vehicle imaging adjustment data, wherein the calibration parameters are used to adjust the quality of the vehicle imaging. Generate an in-vehicle holographic image based on the valid user intent and the calibration parameters.

[0005] Optionally, if the vehicle operating condition data meets the operating condition safety execution conditions corresponding to the preliminary user intent, then the preliminary user intent is considered a valid user intent, including: Based on the preset mapping relationship between user intent and security sensitivity level, the target security sensitivity level corresponding to the initial user intent is determined; Based on the preset binding relationship between the safety sensitivity level and the operating condition safety execution conditions, the target operating condition safety execution conditions bound to the target safety sensitivity level are determined; If the vehicle speed and gear in the vehicle operating data both meet the target operating condition safety execution conditions, then the preliminary user intent will be considered a valid user intent.

[0006] Optionally, generating calibration parameters based on the vehicle-mounted imaging adjustment data includes: The vehicle imaging adjustment data is input into the environmental prediction model to predict the phase offset reference value, brightness reference value and contrast reference value. The phase offset reference value is used to compensate for the imaging offset caused by vehicle vibration, the brightness reference value is used to determine the imaging brightness, and the contrast reference value is used to determine the imaging transmittance. Obtain the actual brightness value sent by the brightness sensor; If the deviation between the brightness reference value and the actual brightness value exceeds a preset deviation range, a preset correction mechanism is used to synchronously and iteratively correct the phase offset reference value, the brightness reference value, and the contrast reference value. If the detected deviation is within the preset deviation range, the correction is stopped, and the final phase offset, brightness data, and contrast data are obtained. The phase offset, brightness data, and contrast data are then used as the calibration parameters.

[0007] Optionally, the vehicle imaging adjustment data includes illumination data, temperature and humidity data, and vehicle vibration data. The vehicle imaging adjustment data is input into an environmental prediction model to predict phase shift reference values, brightness reference values, and contrast reference values, including: Input the illumination data, the temperature and humidity data, and the vehicle vibration data into the environmental prediction model; Global feature extraction and fusion are performed through the shared feature extraction layer of the environmental prediction model to obtain shared features; The vehicle vibration data is processed by the first prediction branch of the environmental prediction model, and adaptive calibration is performed by combining the shared features to predict the phase offset reference value. The illumination data is processed by the second prediction branch of the environmental prediction model, and adaptive calibration is performed in conjunction with the shared features to predict the brightness reference value. The illumination data and temperature and humidity data are processed by the third prediction branch of the environmental prediction model, and adaptive calibration is performed in conjunction with the shared features to predict the contrast benchmark value.

[0008] Optionally, generating the vehicle-mounted holographic image based on the valid user intent and the calibration parameters includes: Obtain a user profile, wherein the user profile is used to characterize the user's holographic interaction usage preferences; The in-vehicle holographic image is generated by combining the user profile, the valid user intent, and the calibration parameters.

[0009] Optionally, generating an in-vehicle holographic image by combining the user profile, the valid user intent, and the calibration parameters includes: The user profile and the effective user intent are fused to generate holographic scene description text, wherein the holographic scene description text includes imaging layout information of user preferences and imaging content information to be displayed; The holographic scene description text is processed by an in-vehicle imaging model to obtain interface layout elements and holographic imaging elements, and a phase distribution map is generated based on the interface layout elements and the holographic imaging elements, wherein the phase distribution map is a non-visual optical data matrix. Using the phase distribution map as the basis for optical modulation, the pixel phase delay, pixel transmittance, and driving power of the light source of the spatial light modulator are adjusted according to the calibration parameters, and the final vehicle-mounted holographic image is obtained after optical transmission.

[0010] Optionally, after generating the vehicle-mounted holographic image based on the valid user intent and the calibration parameters, the method further includes: Generate vehicle control commands based on the valid user intent, and control the vehicle to perform corresponding control actions based on the vehicle control commands; Obtain user feedback data for the control action, and statistically analyze the accuracy of intent recognition of the valid user intent based on the feedback data within a preset sliding window; If the accuracy of intent recognition is lower than a preset accuracy threshold, the parameters of the multimodal fusion model are adjusted according to the feedback data. The feedback data includes at least one of the user's touch operation, voice operation, or gesture operation on the vehicle holographic image. The multimodal fusion model is used to determine the preliminary user intent based on the multimodal user behavior data.

[0011] Secondly, this application provides an apparatus for generating vehicle-mounted holographic images, the apparatus comprising: The acquisition module is used to acquire user behavior data, current vehicle operating condition data, and current vehicle imaging adjustment data, wherein the user behavior data is used to indicate the user's interactive control behavior towards the vehicle. The first determining module is used to determine a preliminary user intent based on the user behavior data, wherein the preliminary user intent is used to indicate the user's control needs for the vehicle; The second determining module is used to determine the preliminary user intent as a valid user intent if the vehicle operating condition data meets the operating condition safety execution conditions corresponding to the preliminary user intent, wherein the operating condition safety execution conditions are used to limit the range of vehicle driving conditions corresponding to the safe execution of the preliminary user intent. The first generation module is used to generate calibration parameters based on the vehicle imaging adjustment data, wherein the calibration parameters are used to adjust the quality of the vehicle imaging. The second generation module is used to generate an in-vehicle holographic image based on the valid user intent and the calibration parameters.

[0012] Thirdly, this application provides an electronic device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus.

[0013] Fourthly, this application also provides a computer storage medium storing computer-executable instructions for executing the method for generating vehicle-mounted holographic images as described in any of the preceding claims.

[0014] The technical solutions provided in this application have the following advantages compared with the prior art: After acquiring user behavior data, current vehicle operating condition data, and current in-vehicle imaging adjustment data, the initial user intent is first determined based on the user behavior data. If the vehicle operating condition data meets the operating condition safety execution conditions corresponding to the initial user intent, it indicates that the operation to be performed by the vehicle is compatible with the current driving state, avoiding driving safety hazards caused by mismatch between user intent and vehicle operating condition. At this time, the initial user intent is determined as a valid user intent. Then, calibration parameters are generated based on the in-vehicle imaging adjustment data to improve the image quality of the holographic imaging. Finally, an in-vehicle holographic image is generated based on the valid user intent and the calibration parameters. This application effectively avoids safety risks caused by operating condition mismatch by co-verifying user intent with real-time vehicle operating condition data, executing the corresponding operation only when the vehicle operating condition data meets the operating condition safety execution conditions corresponding to the initial user intent. Simultaneously, the calibration parameters ensure the clarity of the holographic imaging in real time, reducing user misoperation caused by poor imaging quality and improving driving safety in in-vehicle holographic interaction scenarios. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0018] Figure 1 A schematic diagram of the vehicle-mounted holographic image generation system provided in this application embodiment; Figure 2 A flowchart illustrating a method for generating a vehicle-mounted holographic image, as provided in an embodiment of this application; Figure 3 This application provides an overall flowchart for generating vehicle-mounted holographic images according to an embodiment of the present application. Figure 4 A schematic diagram of the structure of a vehicle-mounted holographic image generation device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0021] To address the issue of low driving safety in in-vehicle holographic interaction mentioned in the background technology, this application verifies user intent using vehicle operating data to ensure that vehicle operation is adapted to the current driving state; at the same time, it optimizes holographic imaging quality using calibration parameters, thereby improving driving safety in in-vehicle holographic interaction scenarios.

[0022] The embodiments of this application are applied to in-vehicle holographic interactive control scenarios, including but not limited to: in-vehicle holographic gesture gear shifting, holographic navigation confirmation, holographic air conditioning adjustment, and holographic volume control, etc.

[0023] Optionally, in the embodiments of this application, the above-described method for generating vehicle-mounted holographic images can be applied to, for example... Figure 1 The hardware environment shown consists of sensing component 101 and processor 103. Figure 1 As shown, the sensing component 101 sends the acquired sensing data to the processor 103. The processor 103 processes this sensing data to generate a non-visual phase distribution map, which is then projected onto the holographic interaction area via an in-vehicle holographic prism. The sensing components include, but are not limited to, cameras, radar, voice sensors, vehicle speed sensors, and vibration sensors. A database 105 can be configured on or independently of the processor to provide data storage services to the processor 103.

[0024] The following will describe in detail a method for generating vehicle-mounted holographic images according to an embodiment of this application, taking an application to a processor as an example. Figure 2 As shown, the specific steps are as follows: Step 201: Acquire user behavior data, current vehicle operating condition data, and current vehicle imaging adjustment data, wherein the user behavior data is used to indicate the user's interactive control behavior towards the vehicle; Step 202: Determine the preliminary user intent based on user behavior data, wherein the preliminary user intent is used to indicate the user's control needs for the vehicle; Step 203: If the vehicle operating condition data meets the safe operating condition execution conditions corresponding to the initial user intent, then the initial user intent is taken as a valid user intent. The safe operating condition execution conditions are used to limit the range of vehicle driving conditions corresponding to the safe execution of the initial user intent. Step 204: Generate calibration parameters based on the vehicle imaging adjustment data, wherein the calibration parameters are used to adjust the quality of the vehicle imaging; Step 205: Generate an in-vehicle holographic image based on valid user intent and calibration parameters.

[0025] In step 201, the processor acquires the three-dimensional spatial coordinates and motion trajectory of the user's hand through a visual acquisition device. Specifically, the OpenPose model can be used to extract the three-dimensional spatial coordinates of 21 joints of the user's hand and 18 joints of the limbs. The instantaneous hand posture is identified using these joint coordinates, and then the three-dimensional spatial coordinates from multiple time points are concatenated to form the motion trajectory. Finally, the user's hand posture is determined by combining the instantaneous hand posture and the motion trajectory. Simultaneously, the processor uses millimeter-wave radar to spatially locate the user's hand, obtaining the relative distance between the user's hand and the holographic interaction area. This relative distance is used to determine the validity of the user's interaction action. When the relative distance is less than a preset distance threshold, it indicates that the hand is within the holographic interaction area, at which point the subsequent preliminary user intent recognition is triggered. Furthermore, the processor collects the user's voice data through a voice acquisition device to extract the user's voice control needs.

[0026] The processor integrates the aforementioned three-dimensional spatial coordinates of the hand, the hand movement trajectory, the relative distance between the hand and the holographic interaction area, and voice data into user behavior data, which is used to clearly indicate the user's interactive control behavior towards the vehicle.

[0027] The processor collects the current vehicle speed through the vehicle speed sensor and the current vehicle gear through the gear position sensor, and uses the vehicle speed and vehicle gear as vehicle operating condition data. This vehicle operating condition data directly reflects the current driving status of the vehicle and is used for subsequent adaptation and verification of user intentions.

[0028] The processor collects light intensity through a light sensor located on the vehicle's dashboard, temperature and humidity data inside the vehicle through a temperature and humidity sensor, and vehicle vibration data through a vibration sensor. It then uses the light intensity, temperature and humidity data, and vehicle vibration data as vehicle imaging adjustment data for subsequent generation of calibration parameters for holographic imaging.

[0029] In step 202, the processor first retrieves the relative distance between the user's hand and the holographic interaction area from the user behavior data. Based on the comparison between this relative distance and a preset distance threshold, it determines whether the user's hand is within the holographic interaction area. If the relative distance is greater than the preset distance threshold, it indicates that the user's hand is not within the holographic interaction area, and the interaction is deemed invalid. The subsequent intent recognition process is not initiated to avoid false triggering. If the relative distance is less than or equal to the preset distance threshold, it indicates that the user's hand is within the holographic interaction area, and the processor initiates user intent recognition to ensure the validity of the interaction.

[0030] Subsequently, the processor inputs the hand gestures and voice data from the user behavior data into a preset multimodal fusion model. The multimodal fusion model performs weighted fusion processing on the hand gestures and voice data to determine the initial user intent. This initial user intent is used to indicate the user's control needs for the vehicle, clearly reflecting the specific operation the user wants the vehicle to perform. Specifically, when the visual recognition detects a gesture to shift to D gear and the voice command is to switch to D gear, the two intents are combined... Figure 1 If the gesture is recognized but the voice is not, or if the voice is recognized but the gesture is not, or if the gesture and voice conflict, the intention is deemed invalid and no preliminary user intention is generated. This further avoids false triggering and ensures that the preliminary user intention accurately matches the user's actual interaction control needs, thereby reducing intention recognition errors.

[0031] The multimodal fusion model can be a lightweight dual-stream Transformer fusion model. The model parameters are set according to the features of the multimodal interaction data. The input dimension matches the 256-dimensional spatial features extracted from gesture key points and the 128-dimensional speech temporal features extracted from Mel frequency cepstral coefficients. It is set with 2 encoder layers, 4 attention heads, and a hidden layer dimension of 512. The output dimension is 64-dimensional to match the classification of user intent. Dropout is set to 0.2 to prevent overfitting and meet the real-time inference requirements of the vehicle terminal.

[0032] In step 203, after determining the initial user intent, the processor retrieves the vehicle speed and gear from the current vehicle operating condition data to perform an adaptation verification on the initial user intent, determining whether the vehicle operating condition data meets the operating condition safety execution conditions corresponding to the initial user intent. Operating condition safety execution conditions refer to the constraints that allow the safe execution of the user intent based on different vehicle driving states. These conditions are used to avoid driving risks caused by conflicts between the user intent and the vehicle's driving state, ensuring that user interaction operations are coordinated and adapted to the actual vehicle operating conditions.

[0033] If the processor's vehicle operating condition data meets the operating condition safety execution conditions corresponding to the initial user intent, such as the operating condition safety execution conditions corresponding to a gear shifting intent being a vehicle speed of 0 and a gear in P, and the current vehicle speed and gear both meet these conditions, then the initial user intent is determined as a valid user intent. This ensures that the operation to be performed by the vehicle is compatible with the current driving state, avoiding potential driving safety hazards caused by a mismatch between intent and operating conditions.

[0034] If the processor determines that the vehicle operating condition data does not meet the operating condition safety execution conditions corresponding to the initial user intent, such as when the vehicle is traveling at high speed and the current operating condition data does not meet the operating condition safety execution conditions corresponding to the intention to shift to reverse, the processor will directly discard the initial user intent and trigger a feedback prompt to inform the user that the current intent cannot be executed, thereby avoiding invalid operations and safety risks.

[0035] This application establishes a dynamic adaptation mechanism between user intent and vehicle operating conditions, and only executes the corresponding operation when the vehicle operating condition data meets the safe execution conditions corresponding to the user intent. This can effectively prevent unsafe interactive behaviors caused by user misoperation or intent recognition deviation, and ensure driving safety from the source of interaction control.

[0036] In step 204, the processor retrieves the collected vehicle imaging adjustment data, including vehicle vibration data, light intensity, and temperature and humidity data. Based on the above data, the processor accurately generates calibration parameters, which include phase offset, brightness data, and contrast data.

[0037] Among them, the phase offset is generated based on vehicle vibration data and is used to accurately compensate for the holographic imaging offset caused by vibration during vehicle operation, so as to avoid problems such as image jitter and ghosting; the brightness data is generated based on light intensity to adapt to the visual needs of different lighting environments and ensure that the vehicle holographic image can be clearly seen in both strong light and low light environments; the contrast data is generated by combining light intensity and temperature and humidity data to reasonably determine the light transmittance of imaging pixels, improve the sense of brightness and darkness of the image, and avoid image blurring and loss of details.

[0038] The processor ensures that the generated calibration parameters are fully adapted to the current vehicle environment through collaborative analysis of various vehicle imaging adjustment data, generating high-quality vehicle holographic images and avoiding interference with driving safety due to image quality issues.

[0039] In step 205, the processor determines the content to be displayed in the vehicle holographic image based on the valid user intent, ensuring that the image content accurately matches the user's control needs. It also improves the image quality of the vehicle holographic image based on calibration parameters. The final generated vehicle holographic image not only accurately matches the user's valid control needs, but also has a stable and clear display effect through the optimization of calibration parameters. At the same time, since the valid user intent has been adapted to the vehicle's operating conditions, the image display content does not conflict with the vehicle's current driving status, thus achieving the security of vehicle holographic interaction.

[0040] In this application, after acquiring user behavior data, current vehicle operating condition data, and current in-vehicle imaging adjustment data, a preliminary user intent is first determined based on the user behavior data. If the vehicle operating condition data meets the operating condition safety execution conditions corresponding to the preliminary user intent, it indicates that the operation to be performed by the vehicle is compatible with the current driving state, avoiding driving safety hazards caused by mismatch between user intent and vehicle operating conditions. At this point, the preliminary user intent is determined as a valid user intent. Then, calibration parameters are generated based on the in-vehicle imaging adjustment data to improve the image quality of the holographic imaging. Finally, an in-vehicle holographic image is generated based on the valid user intent and the calibration parameters. This application effectively avoids safety risks caused by operating condition mismatch by co-verifying user intent with real-time vehicle operating condition data, executing the corresponding operation only when the vehicle operating condition data meets the operating condition safety execution conditions corresponding to the preliminary user intent. Simultaneously, the calibration parameters ensure the clarity of the holographic imaging in real time, reducing user misoperation due to poor imaging quality and improving driving safety in in-vehicle holographic interaction scenarios.

[0041] As an alternative implementation method, in the practical application of in-vehicle holographic interaction, if the same safety execution conditions are applied to all user intentions without distinguishing the degree of impact of the intentions on vehicle driving safety, two problems are likely to occur: First, the verification of intentions that directly affect driving safety, such as shifting gears and switching driving modes, is too lenient, leading to safety hazards; second, the verification of intentions that do not affect driving, such as adjusting the air conditioning and controlling the volume, is too strict, frequently discarding normal operating intentions, resulting in interaction lag, invalid user operations, and a significant reduction in user experience. To solve this problem, this application classifies user intentions into safety sensitivity levels and implements differentiated safety execution conditions, thereby improving the smoothness and usability of the interaction while ensuring driving safety. The specific implementation method is as follows: Step S11: Determine the target security sensitivity level corresponding to the initial user intent based on the preset mapping relationship between user intent and security sensitivity level; Step S12: Based on the preset binding relationship between the safety sensitivity level and the safety execution conditions of the working condition, determine the target working condition safety execution conditions bound to the target safety sensitivity level; Step S13: If the vehicle speed and vehicle gear in the vehicle operating data both meet the target operating condition safety execution conditions, then the preliminary user intent is taken as a valid user intent.

[0042] In step S11, this application pre-sets multiple safety sensitivity levels to distinguish the degree of impact of user intentions on vehicle driving safety. For example, it includes high safety sensitivity levels and low safety sensitivity levels. The high safety sensitivity level corresponds to operation intentions that are directly related to the vehicle's driving status and pose a safety risk, such as gear shifting intentions or driving mode switching intentions. The low safety sensitivity level corresponds to operation intentions that adjust functions and do not affect vehicle driving safety, such as air conditioning temperature adjustment, multimedia volume control, or navigation interface switching intentions.

[0043] The processor retrieves the pre-stored mapping relationship between user intent and safety sensitivity level. This mapping relationship is pre-calibrated according to the degree of impact of various operations on driving safety. The processor determines the target safety sensitivity level corresponding to the initial user intent based on this mapping relationship. For example, when the initial user intent is to shift gears, a high safety sensitivity level is mapped, and when the initial user intent is to adjust the air conditioning, a low safety sensitivity level is mapped.

[0044] In step S12, the processor retrieves the pre-configured binding relationship between safety sensitivity levels and operating condition safety execution conditions in the vehicle system. This binding relationship is pre-set according to the safety risks of different user intentions and vehicle safety driving regulations. Different safety sensitivity levels correspond to different operating condition safety execution conditions. Operations that have a greater impact on vehicle driving safety have more stringent restrictions on vehicle speed and gear selection in their corresponding operating condition safety execution conditions. For routine operations that have no direct impact on driving safety, the corresponding operating condition safety execution conditions are more lenient. The processor matches and determines the target operating condition safety execution conditions bound to the determined target safety sensitivity level.

[0045] For example, if the initial user intent is a gear shifting operation, then this initial user intent corresponds to a high safety sensitivity level. The target operating condition safety execution conditions for the high safety sensitivity level limit the vehicle speed range to 0~90km / h and the vehicle gear range to P or N. Gear shifting can only be safely performed within this range, eliminating the safety risks of high-speed gear shifting and gear shifting without stopping. If the initial user intent is an air conditioning adjustment operation, then the initial user intent is bound to a low safety sensitivity level, and its target operating condition safety execution conditions only limit the vehicle to a normal driving speed and gear range.

[0046] In step S13, the processor compares the current vehicle speed and gear from the real-time acquired vehicle operating condition data with the vehicle speed range and gear range in the target operating condition safety execution conditions. It determines whether the current vehicle speed falls within the speed range specified by the target operating condition safety execution conditions, and simultaneously determines whether the current vehicle gear is within the gear range specified by the target operating condition safety execution conditions. If both the current vehicle speed and gear meet the target operating condition safety execution conditions, it indicates that the current vehicle driving state meets the conditions for safely executing the preliminary user intent, and the processor then formally determines the preliminary user intent as a valid user intent.

[0047] If either the current vehicle speed or the vehicle gear does not meet the safe execution conditions of the target operating condition, it means that the current vehicle operating condition cannot safely execute the initial user intent. The processor will then discard the initial user intent and not execute any subsequent related operations. At the same time, it can issue a prompt through the vehicle interaction module to inform the user that the current operation cannot be executed due to vehicle operating condition limitations.

[0048] This application first matches the initial user intent with the corresponding target safety sensitivity level, and then further matches it with the target operating condition safety execution conditions under that target safety sensitivity level that conform to the actual vehicle operation logic. By clearly defining the vehicle speed range and vehicle gear range corresponding to the safe execution of the initial user intent, it ensures strict operating condition control for high-risk intents that affect driving safety, and reasonably lenient constraints on routine operating intents, thus achieving differentiated and precise safety verification. Finally, the initial user intents that meet the target operating condition safety execution conditions are determined as valid user intents. This ensures that the in-vehicle holographic images subsequently generated based on these valid user intents avoid safety hazards such as misoperation and vehicle malfunction caused by the mismatch between user intents and vehicle operating conditions, thereby guaranteeing driving safety.

[0049] As an optional implementation, generating calibration parameters based on vehicle imaging adjustment data includes: Step S21: Input the vehicle imaging adjustment data into the environment prediction model to predict the phase offset reference value, brightness reference value and contrast reference value. The phase offset reference value is used to compensate for the imaging offset caused by vehicle vibration, the brightness reference value is used to determine the imaging brightness, and the contrast reference value is used to determine the imaging transmittance. Step S22: Obtain the actual brightness value sent by the brightness sensor; Step S23: If the deviation between the brightness reference value and the actual brightness value exceeds the preset deviation range, a preset correction mechanism is used to synchronously iteratively correct the phase offset reference value, brightness reference value, and contrast reference value. Step S24: If the detected deviation is within the preset deviation range, stop the correction, obtain the final phase offset, brightness data and contrast data, and use the phase offset, brightness data and contrast data as calibration parameters.

[0050] In step S21, the processor retrieves vehicle vibration data, illumination data, and temperature and humidity data from the vehicle imaging adjustment data, and uses this data as input parameters into the pre-trained environment prediction model. This environment prediction model has the ability to predict key imaging parameters under dynamic vehicle conditions. It can combine the multi-dimensional input data to predict the basic parameters suitable for imaging in the current scene, namely, the phase shift reference value, the brightness reference value, and the contrast reference value. Specifically, the phase shift reference value is used to initially estimate the holographic imaging shift amplitude that may be caused by vehicle vibration; the brightness reference value is used to initially determine the imaging brightness suitable for the current estimated illumination environment; and the contrast reference value is used to initially determine the imaging transmittance suitable for the current combination of illumination and temperature / humidity.

[0051] The input parameters in the environmental prediction model are normalized parameters, and the normalization process includes the following.

[0052] Illumination data = (Original brightness data - Minimum illumination) / (Maximum illumination - Minimum illumination). For example, the minimum illumination is 0 lux and the maximum illumination is 20000 lux.

[0053] Temperature data = (original temperature data - minimum temperature) / (maximum temperature - minimum temperature), for example, the minimum temperature is -40℃ and the maximum temperature is 85℃.

[0054] Humidity data = (original humidity data - minimum humidity) / (maximum humidity - minimum humidity), for example, the minimum humidity is 10%RH and the maximum humidity is 90%RH.

[0055] The acquired raw vibration time-domain signal is first windowed, then a 16-point FFT (Fast Fourier Transform) is performed to convert the time-domain signal into a frequency-domain signal. By calculating the amplitude of each frequency component of the frequency-domain signal, the frequency component with the largest amplitude is selected as the dominant vibration frequency. Finally, the dominant vibration frequency is normalized. Vehicle vibration data = (dominant vibration frequency - minimum vibration frequency) / (maximum vibration frequency - minimum vibration frequency). For example, the minimum vibration frequency is 10Hz, and the maximum vibration frequency is 2000Hz.

[0056] In step S22, the processor acquires the actual brightness value sent by the vehicle-mounted brightness sensor in real time. This brightness sensor is deployed inside the vehicle's windshield and can collect real-time ambient brightness data after the superposition of natural light and auxiliary lighting, ensuring that the collected actual brightness value is real-time and accurate. This actual brightness value will be used as a real reference standard and compared with the predicted brightness benchmark value to determine whether the prediction result of the environmental prediction model matches the current real vehicle environment, avoiding poor image quality due to prediction deviation.

[0057] In step S23, if the processor determines that the deviation between the brightness reference value and the actual brightness value exceeds a preset deviation range, it indicates that the prediction logic of the environmental prediction model is mismatched with the current real vehicle environment. Since the phase shift, brightness, and contrast of holographic imaging are not independent but strongly correlated and synergistic, jointly determining the final image quality, simply correcting the brightness reference value is insufficient to guarantee the overall imaging effect meets the standards. Therefore, a preset correction mechanism is needed to synchronously and iteratively correct the three reference values ​​to ensure that they synergistically adapt to the current real vehicle environment. This preset correction mechanism can employ a lightweight NeRF model (Neural Radiance Field).

[0058] The human eye's sensitivity to image jitter varies under different lighting conditions. For example, in bright light, even slight phase shifts are amplified visually, leading to blurred images and ghosting, severely interfering with the driver's vision. In low light, the human eye's sensitivity to jitter is reduced, but excessive jitter can still affect the recognition accuracy of in-vehicle holographic images. Furthermore, sudden changes in lighting in the in-vehicle environment are often accompanied by slight vehicle vibrations. For instance, a sudden change in lighting when entering a tunnel is accompanied by road surface undulations at the tunnel entrance. This interconnected change indirectly causes a mismatch between the phase shift reference value and the actual imaging shift caused by vibration, thus requiring simultaneous correction of the phase shift reference value.

[0059] Brightness and contrast are strongly coupled. Changes in brightness will directly cause the original contrast parameters to become invalid. For example, when the brightness increases, the original contrast will appear too low, resulting in blurred image details; when the brightness decreases, the original contrast will appear too high, resulting in distortion of brightness and glare. Therefore, the contrast reference value also needs to be corrected simultaneously.

[0060] In step S24, after the processor determines that the deviation between the brightness reference value and the actual brightness value exceeds the preset deviation range, it enters the iterative correction process. In each round of iterative correction, the phase offset reference value, brightness reference value and contrast reference value are synchronously and adaptively adjusted through the preset correction mechanism. This ensures that the correction rhythm of the three reference values ​​is consistent and mutually compatible, and avoids the imbalance of the imaging parameter system caused by adjusting a single reference value, thus providing a stable iterative basis for conforming to the real vehicle environment.

[0061] After each round of iteration and correction, the processor will detect the deviation between the brightness reference value and the actual brightness value in real time. If the deviation is detected to be within the preset deviation range, it is determined that the current three reference values ​​have been accurately adapted to the real environment. The system immediately stops correcting all reference values ​​and determines the phase offset reference value, brightness reference value and contrast reference value after this round of correction as the final phase offset amount, brightness data and contrast data, respectively. This ensures that the parameters are accurate and stable while avoiding invalid iterations and repeated corrections.

[0062] Among them, the phase offset can compensate for the impact of vehicle vibration, counteract the imaging deviation caused by vibration, and ensure that the vehicle holographic image is stable and jitter-free; the brightness data can adjust the display brightness of the vehicle holographic image to adapt to the current lighting environment, avoiding excessive brightness that is dazzling and interferes with the driver's vision, and avoiding insufficient brightness that prevents users from clearly recognizing the image content; the contrast data can optimize the brightness and darkness levels of the vehicle holographic image, making the image details clear and the overall viewing experience more comfortable.

[0063] The following section explains the correction process for the three baseline values ​​using the lightweight NeRF model.

[0064] 1. Correction of Phase Shift Reference Value: When the deviation exceeds the preset deviation range, the lightweight NeRF model first determines the real lighting environment based on the actual brightness value. If the actual brightness value is greater than the brightness reference value, it is determined to be a strong light environment. At this time, the adjustment range of the phase shift reference value is reduced to accurately compensate for slight vehicle vibrations and effectively avoid imaging distortion caused by over-correction. If the actual brightness value is less than the brightness reference value, it is determined to be a weak light environment. At this time, the adjustment range of the phase shift reference value is increased to fully compensate for the imaging shift caused by vehicle vibrations and ensure imaging accuracy. The phase shift reference value is updated synchronously with the brightness reference value in each round without setting a separate stopping condition until the deviation between the brightness reference value and the actual brightness value falls within the preset deviation range, at which point the correction stops synchronously with the brightness and contrast reference values.

[0065] 2. Correction of the Brightness Reference Value: When the deviation exceeds the preset deviation range, the lightweight NeRF model corrects the reference value towards the actual brightness value based on the comparison between the actual brightness value and the reference value. If the actual brightness value is greater than the reference value, it indicates that the light intensity predicted by the model is lower than the actual environment, and the reference value is gradually increased. If the actual brightness value is less than the reference value, it indicates that the light intensity predicted by the model is higher than the actual environment, and the reference value is gradually decreased. The reference value is updated in real time with the synchronous correction process in each round, without a separate stopping condition, until the deviation between the reference value and the actual brightness value falls within the preset deviation range, at which point the correction stops synchronously with the other two reference values.

[0066] 3. Contrast Ratio Correction: Temperature and humidity directly affect the actual contrast ratio of holographic imaging. Excessively high or low temperatures alter the transmittance, luminous efficiency, and display uniformity of the holographic projection components, disrupting the original light-dark boundaries of the image. High humidity can cause weak refraction in the imaging light path or create a slight haze on the imaging panel surface, blurring the boundaries between bright and dark areas and thus reducing the actual display contrast. Therefore, the lightweight NeRF model first uses a corrected brightness baseline value, combined with actual brightness values ​​and temperature and humidity data, to simulate the optimal contrast ratio under the current brightness, temperature, and humidity combination using light field reconstruction technology. Then, the contrast baseline value is adaptively adjusted to achieve the optimal contrast ratio. The contrast baseline value is updated synchronously with the brightness baseline value in each round, without a separate stopping condition, until the brightness deviation falls within a preset range, at which point the correction stops synchronously with the other two parameters.

[0067] This application uses an environmental prediction model to predict vehicle imaging adjustment data, obtaining phase offset, brightness, and contrast reference values. This allows for the rapid generation of initial imaging parameters adapted to the vehicle environment. Combined with actual brightness values ​​collected by a brightness sensor for deviation verification, it accurately identifies the discrepancies between predicted and actual parameters. When the deviation exceeds a preset range, a preset correction mechanism is activated to synchronously iteratively correct the three reference values, performing adaptive adjustments in each round of correction. Finally, a unified preset deviation range is used as the basis for stopping the correction. This ensures the coordinated adaptability of the three related parameters—phase offset, brightness, and contrast—avoiding overall parameter imbalance caused by adjusting a single parameter. Furthermore, iterative correction enables dynamic optimization of imaging parameters, improving the accuracy of vehicle holographic imaging parameters. This ensures that the final output of phase offset, brightness, and contrast data closely matches the real vehicle driving environment, avoiding problems such as blurring, distortion, and glare, thereby enhancing driving safety in vehicle holographic driving scenarios.

[0068] As an optional implementation, the specific process of the environmental prediction model predicting the phase shift reference value, brightness reference value, and contrast reference value includes: inputting illumination data, temperature and humidity data, and vehicle vibration data into the environmental prediction model; performing global feature extraction and fusion through the shared feature extraction layer of the environmental prediction model to obtain shared features; processing the vehicle vibration data through the first prediction branch of the environmental prediction model and performing adaptive calibration in combination with the shared features to predict the phase shift reference value; processing the illumination data through the second prediction branch of the environmental prediction model and performing adaptive calibration in combination with the shared features to predict the brightness reference value; and processing the illumination data and temperature and humidity data through the third prediction branch of the environmental prediction model and performing adaptive calibration in combination with the shared features to predict the contrast reference value.

[0069] The environmental prediction model adopts a multi-input multi-output architecture with a shared feature extraction layer and three independent prediction branches. This structure not only ensures the independence of each baseline prediction, but also improves the prediction accuracy through global environmental information. The specific prediction process is as follows.

[0070] The processor synchronously inputs the collected illumination data, temperature and humidity data, and vehicle vibration data into the environmental prediction model. The shared feature extraction layer inside the environmental prediction model performs global feature extraction and fusion on the above three heterogeneous data to explore the potential correlations between the three data. For example, high temperature will aggravate the impact of vibration on optical hardware, and strong light and high humidity will synergistically change the characteristics of optical devices. Finally, a shared feature that can comprehensively characterize the overall vehicle environment status is obtained. The role of this shared feature is to provide a global environmental context to make up for the limitations of each prediction branch when processing data separately and improve the accuracy of the baseline value prediction.

[0071] Prediction of Phase Shift Reference Value: The first prediction branch of the environmental prediction model is dedicated to predicting the phase shift reference value. It first performs feature analysis on the vehicle vibration data, extracting features such as vibration amplitude and frequency, and calculates a preliminary phase shift prediction value based on these features. When processing vibration data alone, the effects of temperature and humidity on optical hardware are not considered. For example, high temperatures cause thermal expansion and contraction of the hardware, exacerbating phase distortion caused by vibration, while low temperatures reduce hardware sensitivity, leading to deviations in phase shift prediction. Therefore, by incorporating global environmental information such as temperature, humidity, and illumination contained in the shared features, the preliminary prediction value is adaptively calibrated. The phase shift correction coefficient is adjusted according to the temperature and humidity levels in the shared features to offset the prediction deviations caused by temperature and humidity, ultimately predicting an accurate phase shift reference value to compensate for the holographic imaging phase distortion caused by vehicle vibration.

[0072] Brightness baseline prediction: The second prediction branch of the environmental prediction model is dedicated to predicting the brightness baseline. First, feature analysis is performed on the illumination data to extract features such as illumination intensity. This is then combined with the human visual comfort threshold to calculate a preliminary brightness prediction value. When processing illumination data alone, the indirect effects of vehicle vibration and temperature / humidity on the visibility of holographic imaging are not considered. For example, severe vehicle vibration can cause image jitter, and if the brightness is not appropriately increased according to the vibration intensity, image clarity will be reduced. High temperatures can cause LED light source brightness decay, resulting in a lower brightness calculated solely based on illumination. Therefore, by incorporating global environmental information such as vehicle vibration and temperature / humidity contained in the shared features, the preliminary prediction value is adaptively calibrated. Based on the vibration amplitude and temperature / humidity data in the shared features, the brightness value is fine-tuned to ensure that the vehicle-mounted holographic image is clearly visible under different environments, ultimately predicting a brightness baseline value adapted to the current environment.

[0073] Contrast baseline prediction: The third prediction branch of the environmental prediction model is specifically dedicated to predicting the contrast baseline. First, joint feature analysis is performed on the illumination and temperature / humidity data to extract features such as illumination intensity level, temperature / humidity variation trends, and their synergistic effects. This, combined with the visual hierarchy requirements of holographic imaging, yields a preliminary contrast prediction value. When jointly processing illumination and temperature / humidity data, the interference of vehicle vibration on the imaging effect of optical devices is not considered. For example, vibration can cause blurring at the edges of the vehicle-mounted holographic image. If the contrast is not properly adjusted, it will reduce image clarity. Therefore, by incorporating global environmental information such as vehicle vibration contained in the shared features, the preliminary prediction value is adaptively calibrated. Based on the vibration amplitude in the shared features, the contrast calibration ratio is adjusted to counteract the image blurring caused by vibration. Finally, a contrast baseline value that ensures the hierarchy and clarity of the vehicle-mounted holographic image is predicted.

[0074] The environmental prediction model can be a lightweight CNN (Convolutional Neural Network) model with shared feature extraction and multi-branch prediction, with MobileNetV3-Small as the backbone. The model parameters are set according to the normalized features of the vehicle imaging data. The input dimensions are 32-dimensional illumination data, temperature and humidity data, and vibration data. The shared feature extraction layer outputs 128-dimensional features. Three independent fully connected prediction branches are set to output 3-dimensional phase shift, brightness, and contrast baseline values. The optimizer is an Adam optimizer with weight decay and a learning rate of 1e-4, which is suitable for the fast prediction requirements of vehicle dynamic environment.

[0075] In this application, the environmental prediction model adopts a multi-input multi-output architecture of shared feature extraction and independent branch prediction. After inputting illumination data, temperature and humidity data, and vehicle vibration data into the model, the shared feature extraction layer first completes the extraction and fusion of global features, exploring the potential correlations and coupling effects between multi-dimensional environmental data to form shared features that can comprehensively characterize the overall vehicle environment. Then, three independent prediction branches specifically process vehicle vibration data, illumination data, and illumination and temperature and humidity fusion data to achieve personalized predictions for three types of benchmark values: phase shift, brightness, and contrast. At the same time, each branch superimposes the global shared features during the prediction process to complete adaptive calibration, allowing single-dimensional parameter prediction to incorporate the reference and correction of multi-dimensional environmental information. This approach not only ensures the specificity and independence of each benchmark value prediction, effectively avoiding the one-sidedness of single-dimensional data prediction, but also achieves deep adaptation of parameter prediction with the overall vehicle environment through the global empowerment of shared features, significantly improving the prediction accuracy of phase shift benchmark values, brightness benchmark values, and contrast benchmark values.

[0076] As an alternative implementation, since different users have personalized differences in the display style, interaction layout, functional preferences, and operating habits of in-vehicle holographic images, if the in-vehicle holographic interaction system uses standardized and uniform in-vehicle holographic images for display and interaction, it will be difficult to match the actual usage needs and operating habits of different users. Furthermore, the interaction layout may distract the driver's attention due to inconsistencies with user habits, potentially impacting driving safety. Therefore, this application, based on generating holographic images using effective user intent and calibration parameters, further introduces user profiles to participate in the generation of in-vehicle holographic images, improving the user experience through personalized adaptation. The specific implementation process is as follows.

[0077] First, the server collects user interaction data generated by multiple in-vehicle terminals during actual use. This data includes, but is not limited to, the frequency of user operations on various in-vehicle functions, frequently used preferred functions, the duration of the driver's gaze on the holographic imaging area, preferred holographic interaction response times, and habitual gestures and operation directions. After aggregating this user interaction data, the initial user profile model is jointly trained to generate a universal user profile model that can adapt to the basic interaction habits of most users. Subsequently, the universal user profile model is distributed to the local in-vehicle terminal. Combined with long-term interaction data from a single user on a single in-vehicle terminal, the universal user profile model is fine-tuned to generate a user profile for that individual user. This user profile includes personalized information such as the user's preferred holographic interface layout, the display priority of frequently used in-vehicle functions, habitual interaction gestures, preferred interaction area locations, and holographic display styles. When generating the in-vehicle holographic image, the in-vehicle terminal's processor integrates the user profile, valid user intent, and calibration parameters to generate a personalized in-vehicle holographic image adapted to the current user.

[0078] Among them, the general user profile model can be a hybrid model of collaborative filtering and lightweight MLP (Multi-Layer Perceptron). The model parameters are set according to the characteristics of in-vehicle user interaction behavior data. The input dimension is 128-dimensional user operation frequency, preferred functions and other behavioral features. The MLP is set with 3 layers and the hidden layer dimensions are 256, 128 and 64 respectively. The output is a 64-dimensional user holographic interaction preference feature vector. L2 regularization is added and dropout is set to 0.3. It is trained based on massive cross-terminal interaction data and supports local small sample fine-tuning on the in-vehicle terminal.

[0079] For example, for users who frequently use the navigation function and have a high priority for navigation needs, the system places the navigation holographic icon in an area easily visible to the driver; for users who are left-handed and use the interaction method, the system automatically adjusts the holographic interaction area to the left side of the vehicle's cockpit; in addition, the system also supports users to manually adjust the layout, size, position and other parameters of the vehicle's holographic image according to their own needs, and automatically remembers the fine-tuning parameters to update the exclusive user profile, forming a personalized interaction mode of AI intelligent adaptation and user manual fine-tuning.

[0080] This application combines a general user profile model with local personalized fine-tuning. It ensures the basic accuracy of the profile model by relying on massive amounts of data, and achieves precise personalized adaptation for individual users through local fine-tuning. At the same time, it generates in-vehicle holographic images by combining effective user intent and calibration parameters. Under the premise of meeting the user's real-time operation needs and ensuring the quality of holographic imaging, it matches the interaction habits and functional preferences of different users, reduces the user's attention distraction caused by interaction adaptation, and further enhances driving safety.

[0081] As an optional implementation, generating in-vehicle holographic images by combining user profiles, valid user intent, and calibration parameters includes: Step S31: Perform feature fusion on the user profile and effective user intent to generate holographic scene description text, wherein the holographic scene description text includes imaging layout information of user preferences and imaging content information to be displayed; Step S32: Process the holographic scene description text using the vehicle-mounted imaging model to obtain interface layout elements and holographic imaging elements, and generate a phase distribution map based on the interface layout elements and holographic imaging elements. The phase distribution map is a non-visualized optical data matrix. Step S33: Using the phase distribution map as the basis for optical modulation, adjust the pixel phase delay, pixel transmittance and driving power of the spatial light modulator according to the calibration parameters, and obtain the final vehicle holographic image after optical transmission.

[0082] In step S31, the processor determines the features of the main information to be displayed based on the valid user intent, and simultaneously extracts personalized features from the user profile, including the user's preferred holographic interface layout, interaction area adaptation habits, and icon style preferences. Subsequently, the extracted two types of features are fused, and the fused features are input into the lightweight NeRF module to transform the feature information into standardized holographic scene description text. This holographic scene description text clearly defines the imaging layout information of the in-vehicle holographic image based on the user's preferences and the imaging content information to be displayed.

[0083] For example, if the valid user intent is to open the in-vehicle navigation, the extracted intent features are navigation function and route display; if the user profile features are left-handed operation, preference for simple navigation icons, holographic interaction area located on the left side of the cockpit, and preference for cool colors for icons, then the holographic scene description text output by the NeRF module is to generate an in-vehicle holographic image of the in-vehicle navigation adapted to left-handed operation, with the interaction area located on the left side of the cockpit, the navigation icons adopting a simple cool color style, and displaying the current driving route and turn prompts for the three intersections ahead.

[0084] In step S32, the processor inputs the generated holographic scene description text into the preset vehicle imaging model. The vehicle imaging model has been trained to learn the mapping relationship between holographic scene features and standardized elements in the vehicle holographic basic element library. The vehicle holographic basic element library pre-stores standardized elements required for vehicle holographic interaction of various types, covering interface layout elements and holographic imaging elements. Among them, interface layout elements include interactive area division templates, functional area position boundary parameters, element arrangement specifications, etc., and holographic imaging elements include vehicle function icons, numerical display controls, prompt text styles, navigation route graphics, etc.

[0085] The vehicle-mounted imaging model can be a lightweight text-to-visual feature encoder-decoder model. The model parameters are set according to the specifications of the optical data matrix of the holographic scene description text features and phase distribution map. The text encoder adopts a 6-layer lightweight version of BERT, the input text length is less than 64 characters, and the output is 256-dimensional text features. The decoding layer consists of 2 fully connected layers and 1×1 convolutional layer, and the output dimension matches the phase distribution map of 256×256 pixel optical data. The activation function is ReLU (linear rectified function), and the inference latency is ≤50ms to meet the requirements of rapid holographic image generation.

[0086] The vehicle-mounted imaging model first performs semantic parsing and deep feature extraction on the holographic scene description text to accurately obtain the interface layout features, content display features, and display style features contained in the text. Then, based on the extracted multi-dimensional features, combined with the mapping relationship with standardized elements, it directly outputs the matching interface layout elements and holographic imaging elements.

[0087] For example, if the holographic scene description text is adapted for left-hand operation, the layout of the left side of the center console, and displays the current navigation route and the air conditioning temperature as 26℃, then the matched interface layout elements are: left-hand interaction adaptation, vehicle holographic images arranged in the left area of ​​the center console of the vehicle cabin, dividing the navigation display area and the air conditioning display area and the boundary range, and the holographic imaging elements are the navigation route icon, the intersection turn prompt text, the air conditioning temperature number as 26℃, and the air conditioning mode indicator.

[0088] The vehicle-mounted imaging model uses interface layout elements as the arrangement framework and holographic imaging elements as the content carrier. A phase distribution map is generated in the optical data layer. This phase distribution map is a pure optical data matrix, not a visual image. It contains pixel position and region division data corresponding to the interface layout elements, as well as the initial phase value of each pixel corresponding to the holographic imaging elements.

[0089] In step S33, the processor converts the calibration parameters into phase control parameters and contrast parameters recognizable by the SLM (Spatial Light Modulator) and brightness parameters recognizable by the LED (Light Emitting Diode) light source. The LED light source adjusts its driving power according to the brightness driving parameters, outputting light of corresponding intensity. After being collimated by the collimating optical system into uniform collimated light, it illuminates the surface of the SLM. Based on the phase distribution map as the optical modulation basis, pixel positioning is performed according to the pixel position and region division data corresponding to the interface layout elements in the phase distribution map. Combined with the initial phase value of each pixel corresponding to the holographic imaging element in the phase distribution map, the phase delay of each physical pixel is precisely calibrated and modulated by matching phase control parameters. At the same time, the transmittance of each pixel position is adapted and adjusted according to the contrast parameters, so that the transmitted beam fully carries the interface layout, element features and spatial distribution information of the vehicle holographic image given by the phase distribution map. The modulated light carrying holographic information is incident on a customized vehicle holographic prism. The prism refracts, splits and projects the modulated light in a directional manner through the internal preset optical structure, and finally forms a clear and stable final vehicle holographic image in the designated area of ​​the vehicle cabin.

[0090] Furthermore, after the in-vehicle holographic image is generated, the image resolution can be adjusted according to the vehicle speed. The relationship between vehicle speed and image resolution is inverse: if the vehicle speed is greater than a preset speed threshold, the image resolution is reduced to decrease computational load and ensure smooth interaction; if the vehicle speed is less than or equal to the preset speed threshold, the image resolution is increased to ensure image clarity. Simultaneously, the generated holographic content features anti-shake optimization by adding dynamic compensation frames to avoid image blurring caused by vehicle vibration.

[0091] This application generates holographic scene description text by fusing user profiles with valid user intent, simultaneously considering personalized user habits and real-time interaction needs, thus improving the adaptability and relevance of in-vehicle holographic images. By mapping the holographic scene description text to a preset basic element library to generate a non-visualized phase distribution map, and then adjusting the pixel phase delay, pixel transmittance, and light source driving power of the spatial light modulator based on calibration parameters, the final in-vehicle holographic image is obtained after optical transmission. The calibration parameters dynamically optimize the optical parameters of holographic imaging, effectively counteracting imaging interference caused by factors such as vibration and lighting changes in the in-vehicle environment. The final generated in-vehicle holographic image combines personalization, clarity, and stability, improving the user experience and safety of in-vehicle holographic interaction.

[0092] As an optional implementation, after generating the vehicle holographic image based on the valid user intent and calibration parameters, if the generated vehicle holographic image is not accurate enough, it will affect the user's safe driving. Therefore, it is also necessary to update the parameters of the multimodal fusion model in the processor, including the following.

[0093] Step S41: Generate vehicle control commands based on valid user intent, and control the vehicle to perform corresponding control actions based on the vehicle control commands; Step S42: Obtain user feedback data for control actions, and statistically analyze the accuracy of valid user intent recognition based on the feedback data within the preset sliding window; Step S43: If the accuracy of intent recognition is lower than the preset accuracy threshold, the parameters of the multimodal fusion model are adjusted according to the feedback data. The feedback data includes at least one of the user's touch operation, voice operation, or gesture operation on the vehicle holographic image. The multimodal fusion model is used to determine the preliminary user intent based on the user's multimodal behavior data.

[0094] In step S41, the processor parses the valid user intent and converts it into vehicle control instructions that can be executed by the vehicle control system. The processor then sends the generated vehicle control instructions to the corresponding vehicle execution units via the vehicle communication bus, such as the air conditioning controller, navigation module, window control unit, or multimedia module. After receiving the instructions, the vehicle execution units immediately execute the corresponding control actions to achieve precise response to vehicle functions.

[0095] In step S42, the processor monitors the user's feedback data through the visual acquisition device, the voice acquisition device, and the tactile acquisition device. The feedback data includes at least one of the user's touch operation feedback, voice operation feedback, or gesture operation feedback on the vehicle holographic image. For example, the feedback data includes the user clicking the interactive button in the vehicle holographic image, or the user switching control commands through voice and gesture.

[0096] This application presets the window duration of the sliding window, which can be dynamically adjusted according to the vehicle's operating conditions. For example, the window duration is 10 minutes. The processor determines whether the recognition result of each valid user intent is correct based on the feedback data related to the control action within the sliding window. If the feedback data is a user confirmation control action, such as the user touching the confirmation button of the vehicle's holographic image or not initiating any correction operation, then the intent recognition is determined to be correct. If the feedback data is a user correction control action, such as the user correcting the voice to say "heat up" instead of "cool down", then the intent recognition is determined to be incorrect.

[0097] The processor counts the number of times the intent is correctly recognized within the sliding window and the total number of recognition attempts, and calculates the intent recognition accuracy, where intent recognition accuracy = number of times the intent is correctly recognized / total number of recognition attempts × 100%.

[0098] In step S43, if the processor determines that the intention recognition accuracy is lower than a preset accuracy threshold, it adaptively adjusts the parameters of the multimodal fusion model based on the multi-dimensional interaction data of the user's feedback on vehicle control actions. As the core model for recognizing user intentions, the multimodal fusion model can perform feature extraction, weighted fusion, and intention determination on the collected multimodal user behavior data, thereby accurately determining the initial user intention. This application allows the multimodal fusion model to continuously adapt to the user's interaction habits and operation characteristics through feedback, effectively improving the intention recognition accuracy of subsequent user behavior data, reducing erroneous operations in in-vehicle holographic interaction caused by intention recognition deviations, and further ensuring driving safety in in-vehicle holographic interaction scenarios.

[0099] This application also provides a schematic flowchart of a method for generating vehicle-mounted holographic images, such as... Figure 3 As shown, the steps include the following.

[0100] Step 301: Perform multimodal fusion based on hand gestures and voice data from user behavior data to determine the user's initial intent.

[0101] The processor acquires user behavior data through a preset perception module. This data includes the three-dimensional spatial coordinates and motion trajectory of the hand, voice data, and the relative distance between the hand and the holographic interaction area. The hand's three-dimensional spatial coordinates and motion trajectory determine the hand posture. If the relative distance is greater than a preset distance threshold, or only one of the hand posture or voice data can be recognized, or the meanings of the hand posture and voice data are different, the recognition result is considered invalid, and the process is not continued until the next user behavior data acquisition. If the relative distance is less than or equal to the preset distance threshold, and the meanings of the hand posture and voice data are the same, a multimodal fusion model is used to fuse the features of the hand posture and voice data to determine the user's initial intent.

[0102] Step 302: If the vehicle operating condition data meets the safe operating condition execution conditions corresponding to the preliminary user intent, then the preliminary user intent is taken as a valid user intent.

[0103] Safety sensitivity levels are categorized based on the degree to which user intent affects driving safety, with different initial user intents corresponding to different safety sensitivity levels. The processor, based on the determined initial user intent, queries the target safety sensitivity level corresponding to that intent. Then, according to a preset binding relationship, it obtains the target operating condition safety execution conditions bound to the target safety sensitivity level. These conditions primarily include the vehicle speed range and the vehicle gear range. The processor collects vehicle operating condition data, extracting vehicle speed and gear data. It then compares this data with the speed and gear ranges in the target operating condition safety execution conditions one by one. If both vehicle speed and gear meet the target operating condition safety execution conditions, the initial user intent is determined to be a valid intent. If either vehicle speed or gear does not meet the target operating condition safety execution conditions, the intent remains invalid, and subsequent control procedures are not executed.

[0104] Step 303: Based on illumination data, temperature and humidity data, and vehicle vibration data, obtain calibration data including phase offset, brightness data, and contrast data.

[0105] The processor inputs illumination data, temperature and humidity data, and vehicle vibration data into the environmental prediction model to predict phase offset reference value, brightness reference value, and contrast reference value. If the deviation between the brightness reference value and the actual brightness value exceeds the preset deviation range, the reference value is synchronously iteratively corrected to obtain calibration data containing phase offset, brightness data, and contrast data.

[0106] Step 304: Generate holographic scene description text based on user profiles and valid user intents.

[0107] The processor extracts features from valid user intent and personalized features from user profile. The processor fuses the two types of features to form a holographic scene description text. The holographic scene description text explicitly includes imaging layout information of user preferences and imaging content information to be displayed.

[0108] Step 305: Generate a non-visualized phase distribution map from the holographic scene description text.

[0109] The processor inputs the generated holographic scene description text into a preset vehicle-mounted imaging model. The vehicle-mounted imaging model performs semantic parsing and feature extraction on the holographic scene description text, extracting features related to the imaging layout and imaging content. Then, based on the pre-learned relationships, it obtains the corresponding interface layout elements and holographic imaging elements. The processor uses the interface layout elements as the arrangement framework and the holographic imaging elements as the content carrier, and integrates and encodes them at the optical data layer to generate a phase distribution map. The phase distribution map is a non-visual optical data matrix that contains the pixel position distribution corresponding to the interface layout and the initial phase values ​​corresponding to the holographic imaging elements.

[0110] Step 306: Adjust the phase distribution map according to the calibration parameters to generate an in-vehicle holographic image.

[0111] The processor converts calibration parameters into phase control parameters, contrast parameters, and LED light source brightness parameters that the SLM can recognize. The processor controls the LED light source to adjust its driving power according to the brightness parameters, emitting uniform collimated light and illuminating the SLM surface. At the same time, the processor controls the SLM to locate pixel regions according to the pixel position distribution in the phase distribution map. For each pixel position, the processor combines the initial phase value in the phase distribution map with the phase control parameters converted from the calibration parameters to precisely adjust the phase delay of each pixel. Based on the contrast parameters, the processor controls the transmittance of each pixel, so that the transmitted light signal carries the basic optical information given by the phase distribution map and is optimized and compensated by the calibration parameters to form modulated light with 3D content spatial depth information. The processor controls the transmission of the modulated light carrying 3D information to a customized automotive holographic prism. The modulated light is refracted and reflected by the preset optical structure inside the prism, and finally a clear and stable automotive holographic image is formed.

[0112] Step 307: The processor generates vehicle control instructions and controls the vehicle to perform corresponding actions.

[0113] Step 308: Adjust the parameters of the multimodal fusion model based on the collected feedback data.

[0114] The processor collects user feedback data, counts the number of correct intent recognitions within the window and the total number of recognition attempts, and calculates the intent recognition accuracy within the sliding window. If the intent recognition accuracy is lower than a preset baseline threshold, the processor adaptively adjusts the feature weights of hand gestures and voice data in the multimodal fusion model based on the collected feedback data: if the number of intent recognition errors for a certain interaction type exceeds a preset threshold, the feature weight of that interaction type in the model fusion calculation is reduced, while the feature weight of the other interaction type is correspondingly increased; and dynamic guidance prompts for the interaction type with more errors are added to the in-vehicle holographic interface to standardize user interaction operations and reduce the probability of subsequent recognition errors. For example, for feedback of more errors in recognizing air conditioning adjustment gesture commands, the fusion weight of this type of gesture feature is adjusted from 0.5 to 0.4, while the fusion weight of air conditioning adjustment-related voice features is increased, and dynamic guidance for temperature adjustment gestures is added to the holographic interface. At the same time, the processor will also update the relevant interaction preference features in the user profile based on user feedback data. For example, when it detects that the user has repeatedly responded to the cooling function, it will identify and record the user's preference for coolness in the user profile. Ultimately, it will achieve dual adaptive optimization of multimodal fusion model parameters and user profile preference features, making intent recognition and holographic interaction more in line with the user's operating habits and usage needs.

[0115] As an optional implementation, this application provides an apparatus for generating vehicle-mounted holographic images, such as... Figure 4 As shown, the device includes: The acquisition module 401 is used to acquire user behavior data, current vehicle operating condition data, and current vehicle imaging adjustment data, wherein the user behavior data is used to indicate the user's interactive control behavior towards the vehicle. The first determining module 402 is used to determine the preliminary user intent based on user behavior data, wherein the preliminary user intent is used to indicate the user's control needs for the vehicle; The second determining module 403 is used to determine the preliminary user intent as a valid user intent if the vehicle operating condition data meets the safe operating condition execution conditions corresponding to the preliminary user intent. The safe operating condition execution conditions are used to limit the range of vehicle driving conditions corresponding to the safe execution of the preliminary user intent. The first generation module 404 is used to generate calibration parameters based on the vehicle imaging adjustment data, wherein the calibration parameters are used to adjust the quality of the vehicle imaging. The second generation module 405 is used to generate an in-vehicle holographic image based on valid user intent and calibration parameters.

[0116] Optionally, the second determining module 403 is used for: Based on the pre-defined mapping relationship between user intent and security sensitivity level, determine the target security sensitivity level corresponding to the initial user intent; Based on the pre-defined binding relationship between the safety sensitivity level and the operating condition safety execution conditions, determine the target operating condition safety execution conditions bound to the target safety sensitivity level; If the vehicle speed and gear in the vehicle operating data both meet the safe execution conditions of the target operating condition, then the preliminary user intent will be regarded as a valid user intent.

[0117] Optionally, the first generation module 404 is used for: The vehicle imaging adjustment data is input into the environmental prediction model to predict the phase offset reference value, brightness reference value and contrast reference value. The phase offset reference value is used to compensate for the imaging offset caused by vehicle vibration, the brightness reference value is used to determine the imaging brightness, and the contrast reference value is used to determine the imaging transmittance. Obtain the actual brightness value sent by the brightness sensor; If the deviation between the brightness reference value and the actual brightness value exceeds the preset deviation range, a preset correction mechanism is used to synchronously and iteratively correct the phase offset reference value, brightness reference value, and contrast reference value. If the detected deviation is within the preset deviation range, the correction is stopped, and the final phase offset, brightness data, and contrast data are obtained. The phase offset, brightness data, and contrast data are then used as calibration parameters.

[0118] Optionally, the vehicle imaging adjustment data includes illumination data, temperature and humidity data, and vehicle vibration data. The first generation module 404 is specifically used for: Input illumination data, temperature and humidity data, and vehicle vibration data into the environmental prediction model; Global feature extraction and fusion are performed through the shared feature extraction layer of the environmental prediction model to obtain shared features; The vehicle vibration data is processed by the first prediction branch of the environmental prediction model, and adaptive calibration is performed by combining shared features to predict the phase offset reference value. The illumination data is processed by the second prediction branch of the environmental prediction model, and adaptive calibration is performed by combining shared features to predict the brightness baseline value. The third prediction branch of the environmental prediction model processes the illumination and temperature / humidity data and performs adaptive calibration by combining shared features to predict the contrast baseline value.

[0119] Optionally, the second generation module 405 is used for: Obtain user profiles, which are used to characterize users' holographic interaction preferences; In-vehicle holographic images are generated by combining user profiles, valid user intent, and calibration parameters.

[0120] Optionally, the second generation module 405 is specifically used for: The user profile and effective user intent are fused to generate holographic scene description text, which includes imaging layout information of user preferences and imaging content information to be displayed. The holographic scene description text is processed by the vehicle-mounted imaging model to obtain interface layout elements and holographic imaging elements. A phase distribution map is generated based on the interface layout elements and holographic imaging elements. The phase distribution map is a non-visualized optical data matrix. Using the phase distribution map as the basis for optical modulation, the pixel phase delay, pixel transmittance, and driving power of the light source of the spatial light modulator are adjusted according to the calibration parameters, and the final vehicle-mounted holographic image is obtained after optical transmission.

[0121] Optionally, the device is also used for: Generate vehicle control commands based on valid user intent, and control the vehicle to perform corresponding control actions based on the vehicle control commands; Acquire user feedback data on control actions, and statistically analyze the accuracy of valid user intent recognition based on the feedback data within a preset sliding window; If the accuracy of intent recognition is lower than a preset accuracy threshold, the parameters of the multimodal fusion model are adjusted based on the feedback data. The feedback data includes at least one of the user's touch operation, voice operation, or gesture operation on the vehicle holographic image. The multimodal fusion model is used to determine the initial user intent based on the multimodal user behavior data.

[0122] like Figure 5 As shown, this application provides an electronic device including a processor 501, a communication interface 502, a memory 503, and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0123] Memory 503 is used to store computer programs.

[0124] In one embodiment of this application, the processor 150, when executing the program stored in the memory 503, implements the method for generating vehicle holographic images provided in any of the foregoing method embodiments.

[0125] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method for generating vehicle-mounted holographic images as provided in any of the foregoing method embodiments.

[0126] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0128] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0129] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for generating vehicle-mounted holographic images, characterized in that, The method includes: Acquire user behavior data, current vehicle operating condition data, and current vehicle imaging adjustment data, wherein the user behavior data is used to indicate the user's interactive control behavior towards the vehicle; A preliminary user intent is determined based on the user behavior data, wherein the preliminary user intent is used to indicate the user's control needs for the vehicle; If the vehicle operating condition data meets the operating condition safety execution conditions corresponding to the preliminary user intent, then the preliminary user intent is regarded as a valid user intent, wherein the operating condition safety execution conditions are used to limit the range of vehicle driving operating conditions corresponding to the safe execution of the preliminary user intent. Calibration parameters are generated based on the vehicle imaging adjustment data, wherein the calibration parameters are used to adjust the quality of the vehicle imaging. Generate an in-vehicle holographic image based on the valid user intent and the calibration parameters.

2. The method according to claim 1, characterized in that, If the vehicle operating condition data meets the operating condition safety execution conditions corresponding to the preliminary user intent, then the preliminary user intent is considered a valid user intent, including: Based on the preset mapping relationship between user intent and security sensitivity level, the target security sensitivity level corresponding to the initial user intent is determined; Based on the preset binding relationship between the safety sensitivity level and the operating condition safety execution conditions, the target operating condition safety execution conditions bound to the target safety sensitivity level are determined; If the vehicle speed and gear in the vehicle operating data both meet the target operating condition safety execution conditions, then the preliminary user intent will be considered a valid user intent.

3. The method according to claim 1, characterized in that, The calibration parameters generated based on the vehicle imaging adjustment data include: The vehicle imaging adjustment data is input into the environmental prediction model to predict the phase offset reference value, brightness reference value and contrast reference value. The phase offset reference value is used to compensate for the imaging offset caused by vehicle vibration, the brightness reference value is used to determine the imaging brightness, and the contrast reference value is used to determine the imaging transmittance. Obtain the actual brightness value sent by the brightness sensor; If the deviation between the brightness reference value and the actual brightness value exceeds a preset deviation range, a preset correction mechanism is used to synchronously and iteratively correct the phase offset reference value, the brightness reference value, and the contrast reference value. If the detected deviation is within the preset deviation range, the correction is stopped, and the final phase offset, brightness data, and contrast data are obtained. The phase offset, brightness data, and contrast data are then used as the calibration parameters.

4. The method according to claim 3, characterized in that, The vehicle imaging adjustment data includes illumination data, temperature and humidity data, and vehicle vibration data. The vehicle imaging adjustment data is input into an environmental prediction model to predict phase shift reference values, brightness reference values, and contrast reference values, including: Input the illumination data, the temperature and humidity data, and the vehicle vibration data into the environmental prediction model; Global feature extraction and fusion are performed through the shared feature extraction layer of the environmental prediction model to obtain shared features; The vehicle vibration data is processed by the first prediction branch of the environmental prediction model, and adaptive calibration is performed by combining the shared features to predict the phase offset reference value. The illumination data is processed by the second prediction branch of the environmental prediction model, and adaptive calibration is performed in conjunction with the shared features to predict the brightness reference value. The illumination data and temperature and humidity data are processed by the third prediction branch of the environmental prediction model, and adaptive calibration is performed in conjunction with the shared features to predict the contrast benchmark value.

5. The method according to claim 1, characterized in that, Generating an in-vehicle holographic image based on the valid user intent and the calibration parameters includes: Obtain a user profile, wherein the user profile is used to characterize the user's holographic interaction usage preferences; The in-vehicle holographic image is generated by combining the user profile, the valid user intent, and the calibration parameters.

6. The method according to claim 5, characterized in that, Generating an in-vehicle holographic image by combining the user profile, the valid user intent, and the calibration parameters includes: The user profile and the effective user intent are fused to generate holographic scene description text, wherein the holographic scene description text includes imaging layout information of user preferences and imaging content information to be displayed; The holographic scene description text is processed by an in-vehicle imaging model to obtain interface layout elements and holographic imaging elements, and a phase distribution map is generated based on the interface layout elements and the holographic imaging elements, wherein the phase distribution map is a non-visual optical data matrix. Using the phase distribution map as the basis for optical modulation, the pixel phase delay, pixel transmittance, and driving power of the light source of the spatial light modulator are adjusted according to the calibration parameters, and the final vehicle-mounted holographic image is obtained after optical transmission.

7. The method according to claim 1, characterized in that, After generating the vehicle-mounted holographic image based on the valid user intent and the calibration parameters, the method further includes: Generate vehicle control commands based on the valid user intent, and control the vehicle to perform corresponding control actions based on the vehicle control commands; Obtain user feedback data for the control action, and statistically analyze the accuracy of intent recognition of the valid user intent based on the feedback data within a preset sliding window; If the accuracy of intent recognition is lower than a preset accuracy threshold, the parameters of the multimodal fusion model are adjusted according to the feedback data. The feedback data includes at least one of the user's touch operation, voice operation, or gesture operation on the vehicle holographic image. The multimodal fusion model is used to determine the preliminary user intent based on the multimodal user behavior data.

8. A device for generating vehicle-mounted holographic images, characterized in that, The device includes: The acquisition module is used to acquire user behavior data, current vehicle operating condition data, and current vehicle imaging adjustment data, wherein the user behavior data is used to indicate the user's interactive control behavior towards the vehicle. The first determining module is used to determine a preliminary user intent based on the user behavior data, wherein the preliminary user intent is used to indicate the user's control needs for the vehicle; The second determining module is used to determine the preliminary user intent as a valid user intent if the vehicle operating condition data meets the operating condition safety execution conditions corresponding to the preliminary user intent, wherein the operating condition safety execution conditions are used to limit the range of vehicle driving conditions corresponding to the safe execution of the preliminary user intent. The first generation module is used to generate calibration parameters based on the vehicle imaging adjustment data, wherein the calibration parameters are used to adjust the quality of the vehicle imaging. The second generation module is used to generate an in-vehicle holographic image based on the valid user intent and the calibration parameters.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images