Method, device and system for generating virtual image display data of vehicle

Abstract

The invention relates to the field of vehicle data display, in particular to a method, a device and a system for generating vehicle virtual image display data. According to the method, the environmental elements are obtained by analyzing the environmental perception data, and the actual scene condition around the vehicle can be known in real time. Meanwhile, environment adaptation information is calculated in combination with environment sensing data and vehicle motion data, environment elements can be correspondingly adjusted according to the motion state of the vehicle, and it is ensured that the environment elements accurately correspond to the actual scene in the spatial position. Therefore, when the vehicle quickly moves or the surrounding environment changes, the virtual image display data can be updated in time, so that the virtual image display data can be synchronized with the actual situation. And the consistency of virtual image display data and an actual scene in space and time is realized.

Description

Technical Field

[0001] The present invention relates to the field of vehicle data display, and in particular to a method, device and system for generating vehicle virtual image display data. Background Art

[0002] ARHUD is an innovative technology that applies augmented reality technology to head-up display systems. HUD originated from the military aviation field. In the automotive field, it allows drivers to obtain key information within the windshield. The development of ARHUD technology is closely related to the advancement of augmented reality technology. It projects virtual information in a transparent way in front of the driver's line of sight and superimposes it with the real world, providing users with a richer and more interactive visual experience.

[0003] The current technical solutions make it difficult to ensure that the displayed virtual information accurately corresponds to the actual scene in terms of spatial position during vehicle driving, and there is a situation where the virtual information is not updated in a timely manner when the vehicle moves quickly or the surrounding environment changes. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method, device and system for generating vehicle virtual image display data to solve the problem that it is difficult to ensure that the displayed virtual information accurately corresponds to the actual scene in terms of spatial position during vehicle driving, and the virtual information is not updated in a timely manner when the vehicle moves quickly or the surrounding environment changes.

[0005] In a first aspect, an embodiment of the present invention provides a method for generating vehicle virtual image display data, the method comprising:

[0006] When the head-up display service of the vehicle is started, obtaining environmental perception data of the current environment of the vehicle, and detecting vehicle motion data and satellite navigation data of the vehicle;

[0007] Analyzing the environmental perception data to obtain environmental elements of the environment in which the vehicle is located, and calculating environmental adaptation information of the vehicle based on the environmental perception data and the vehicle motion data;

[0008] Analyzing the satellite navigation data to obtain navigation information corresponding to the vehicle;

[0009] The environmental elements, the environmental adaptation information and the navigation information are fused based on a fusion strategy corresponding to the head-up display service to obtain virtual image display data, and the virtual image display data is displayed in a head-up display area of ​​the vehicle.

[0010] Furthermore, the analyzing the environmental perception data to obtain environmental elements of the environment in which the vehicle is located includes:

[0011] Extracting video stream data from the environmental perception data, wherein the video stream data includes multiple frames of continuous perception images;

[0012] Detecting regional features of each region in the perceived image, and inferring the regional features to obtain element inference results in each of the regions, wherein the element inference results include a plurality of candidate environmental elements and confidences corresponding to the candidate environmental elements;

[0013] Get the preset confidence threshold;

[0014] The candidate environmental element whose confidence level is higher than the confidence threshold is taken as the environmental element.

[0015] Further, the calculating the environment adaptation information of the vehicle based on the environment perception data and the vehicle motion data includes:

[0016] Extracting video stream data and light detection data from the environmental perception data, wherein the video stream data includes multiple frames of continuous perception images;

[0017] Detecting a plurality of continuous frames of perceived images in the video stream data to obtain key feature points in each frame of the perceived image;

[0018] Analyze the key feature points of each two adjacent frames of perceived images to determine the relative motion information between each two adjacent frames of perceived images;

[0019] fusing the relative motion information and the vehicle motion data to obtain the position and posture information of the vehicle;

[0020] Analyzing the perceived image in the video stream data and the light detection data to obtain ambient light information;

[0021] The environment adaptation information is generated based on the posture information and the environment lighting information.

[0022] Furthermore, the fusing of the relative motion information and the vehicle motion data to obtain the position and posture information of the vehicle includes:

[0023] Acquire IMU data and wheel speed data from the vehicle motion data;

[0024] Calculating the motion change of the vehicle based on the acceleration and angular velocity included in the IMU data, and calculating the corresponding displacement change of the vehicle based on the wheel speed data;

[0025] Predicting preliminary position information of the vehicle using the motion change and the displacement change;

[0026] The preliminary position and posture information is updated based on the relative motion information to obtain the position and posture information.

[0027] Furthermore, the analyzing the perceived image in the video stream data and the light detection data to obtain the ambient light information includes:

[0028] Analyzing the image features of the perceived image according to a preset dimension to obtain a lighting feature parameter corresponding to the preset dimension, wherein the preset dimension includes one or more of color, brightness, and shadow;

[0029] The lighting detection data and the lighting characteristic parameters corresponding to the preset dimension are integrated to obtain the ambient lighting information.

[0030] Further, the analyzing the satellite navigation data to obtain navigation information corresponding to the vehicle includes:

[0031] Acquiring vehicle position data and surrounding environment data of the vehicle based on the satellite navigation data;

[0032] Obtaining the current navigation requirement of the vehicle;

[0033] The vehicle position data and the surrounding environment data are processed based on a processing strategy corresponding to the navigation requirement to obtain the navigation information.

[0034] Furthermore, the fusing of the environmental elements, the environmental adaptation information and the navigation information based on the fusion strategy corresponding to the head-up display service to obtain virtual image display data includes:

[0035] Acquire posture information and ambient lighting information from the environmental adaptation information;

[0036] Determine a display orientation corresponding to the environmental element according to the position information, and construct initial display data based on the environmental element and the display orientation;

[0037] Merging the navigation information with the initial display data to obtain target display data;

[0038] The target display data is rendered using the ambient lighting information to obtain the virtual image display data.

[0039] Further, determining the display orientation corresponding to the environmental element according to the posture information includes:

[0040] Converting the position information of the vehicle into a world coordinate system to obtain first coordinate information and angle information of the vehicle in the world coordinate system;

[0041] Convert the environmental element to the world coordinate system to obtain second coordinate information of the environmental element in the world coordinate system;

[0042] determining a display position of the environmental element according to a positional relationship between the first coordinate information and the second coordinate information, and determining a display direction of the environmental element according to the angle information;

[0043] A display orientation corresponding to the environmental element is constructed based on the display position and the display direction.

[0044] Furthermore, the merging of the navigation information with the initial display data to obtain target display data includes:

[0045] Acquire the guiding element carried in the navigation information;

[0046] Acquire a target environment element matching the guiding element from the initial display data;

[0047] Obtaining a fusion strategy corresponding to the guiding element;

[0048] The guiding element is fused with the target environment element in the initial display data according to the fusion strategy to obtain the target display data.

[0049] In a second aspect, an embodiment of the present invention provides a device for generating vehicle virtual image display data, the device comprising:

[0050] An acquisition module, used for acquiring video stream data of a current environment in which the vehicle is located when a head-up display service of the vehicle is started, and detecting vehicle motion data and satellite navigation data of the vehicle in the current environment;

[0051] A first analysis module, configured to analyze the video stream data to obtain environmental elements of the environment in which the vehicle is located, and to calculate environmental adaptation information of the vehicle based on the environmental perception data and the vehicle motion data;

[0052] A second analysis module, used for analyzing the satellite navigation data to obtain navigation information corresponding to the vehicle;

[0053] A fusion module is used to fuse the environmental elements, the environmental adaptation information and the navigation information based on a fusion strategy corresponding to the head-up display service to obtain virtual image display data, and display the virtual image display data in the head-up display area of ​​the vehicle.

[0054] In a third aspect, an embodiment of the present invention provides a system for generating vehicle virtual image display data, the system comprising: a detection component, a data processing component, a rendering display component and a computing power component, the data processing component comprising: an environment perception module, an algorithm module and a map module, the rendering display component comprising a rendering module and a HUD display module, the computing power component comprising a plurality of computing power units, wherein the computing power units associated with the environment perception module, the algorithm module, the map module and the rendering module are different;

[0055] The detection component is used to detect environmental perception data of the current environment of the vehicle, vehicle motion data and satellite navigation data, and transmit the environmental perception data, the vehicle motion data and the satellite navigation data to the data processing component;

[0056] The environment perception module in the data processing component is used to analyze the environment perception data to obtain the environmental elements of the environment where the vehicle is located, and transmit the environmental elements to the rendering and display module;

[0057] The algorithm module in the data processing component is used to calculate the environment adaptation information of the vehicle based on the environment perception data and the vehicle motion data, and transmit the environment adaptation information to the rendering and display module;

[0058] The map module in the data processing component is used to analyze the satellite navigation data to obtain navigation information corresponding to the vehicle, and transmit the navigation information to the rendering and display module;

[0059] The rendering module in the rendering display module is used to fuse the environmental elements, the environmental adaptation information and the navigation information based on the fusion strategy corresponding to the head-up display service to obtain virtual image display data, and transmit the virtual image display data to the HUD display module;

[0060] The HUD display module is used to display the virtual image display data in the head-up display area of ​​the vehicle.

[0061] In a fourth aspect, an embodiment of the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, computer instructions being stored in the memory, and the processor executing the method of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0062] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method of the first aspect or any corresponding embodiment thereof.

[0063] In a sixth aspect, an embodiment of the present invention provides a vehicle, comprising: a system for generating vehicle virtual image display data and a controller; wherein the controller stores instructions for controlling the operation of the system for generating vehicle virtual image display data, and the system for generating vehicle virtual image display data is used to execute the method for generating vehicle virtual image display data as claimed in the claim according to the instructions of the controller.

[0064] The method provided in the embodiment of the present application obtains environmental elements by analyzing environmental perception data, and can understand the actual scene conditions around the vehicle in real time. At the same time, by combining environmental perception data and vehicle motion data to calculate environmental adaptation information, the environmental elements can be adjusted accordingly according to the vehicle's motion state to ensure that they accurately correspond to the actual scene in spatial position. In this way, it is ensured that when the vehicle moves quickly or the surrounding environment changes, the virtual image display data can be updated in time to keep pace with the actual situation. The consistency of the virtual image display data and the actual scene in space and time is achieved.

[0065] The method provided in the embodiment of the present application can effectively reduce the number of environmental elements that are incorrectly identified and reduce the false alarm rate of the system through strict confidence screening. At the same time, through multi-frame analysis and regional feature reasoning, it is possible to better cope with various complex situations and improve the stability and adaptability of the system. For example, in severe weather conditions, image quality may be affected, but through multi-frame continuous analysis and confidence screening, key environmental elements can still be identified as accurately as possible.

[0066] The method provided in the embodiment of the present application extracts multiple frames of continuous perception images from environmental perception data for analysis and detects key feature points. Multi-frame images provide richer information and help to more accurately determine the key features in the image. By analyzing the key feature points of each two adjacent frames of perception images, relative motion information is determined. This method makes full use of the changes between continuous images and can more accurately reflect the motion state of the vehicle in the actual environment. Compared with the vehicle motion data at a single moment, the relative motion information can reduce errors and improve the accuracy of the posture information. The relative motion information is fused with the vehicle motion data to obtain more comprehensive and accurate vehicle posture information. The vehicle motion data may come from devices such as sensors, but there may be certain errors, and the relative motion information obtained by fusing multi-frame image analysis can be corrected and supplemented. At the same time, the perception images and lighting detection data in the video stream data can be analyzed to more accurately obtain the ambient lighting information. The perception images can provide the lighting effects in the actual scene, and the lighting detection data can provide more specific lighting parameters. Based on accurate posture information and ambient lighting information, environmental adaptation information is generated, so that the system can better adapt to different environmental conditions.

[0067] The method provided in the embodiment of the present application uses posture information to ensure that virtual elements can be accurately positioned in appropriate positions in the real environment, while the ambient lighting information makes the display of virtual elements more natural and integrated with the actual environment. The display orientation of the environmental elements is determined based on the posture information, and the initial display data is constructed in combination with the environmental elements. This enables the virtual elements to be presented to the user in the correct direction and position, improving the readability and usability of the information. The navigation information is merged with the initial display data to obtain the target display data. This enables the navigation instructions to be closely integrated with the actual environmental elements, providing users with accurate navigation guidance. Finally, the target display data is rendered using the ambient lighting information, further enhancing the realism of the virtual image display data. The rendered virtual elements can better adapt to different lighting conditions and are consistent with the actual scene.

[0068] The system provided in the embodiment of the present application allocates different modules in the vehicle virtual image display data generation system to different hardware computing power units, which can give full play to the advantages of each unit, improve system performance and efficiency, balance the computing power load, adapt to different task characteristics, and have better scalability and flexibility, thereby fully utilizing and giving play to the various software and hardware resources of the platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0070] Figure 1 is a flow chart of a method for generating vehicle virtual image display data according to some embodiments of the present invention;

[0071] Figure 2 is a schematic diagram of a processing framework of video stream data according to some embodiments of the present invention;

[0072] Figure 3 is a schematic diagram of an environment perception module according to some embodiments of the present invention;

[0073] Figure 4 is a schematic diagram of a processing framework of sensor data according to some embodiments of the present invention;

[0074] Figure 5 is a schematic diagram of an algorithm module according to some embodiments of the present invention;

[0075] Figure 6 is a schematic diagram of a map module according to some embodiments of the present invention;

[0076] Figure 7 is a schematic diagram of a system for generating vehicle virtual image display data according to some embodiments of the present invention;

[0077] Figure 8 is a schematic diagram of a device for generating vehicle virtual image display data according to some embodiments of the present invention;

[0078] Fig. 9 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0080] According to an embodiment of the present invention, a method, device and system for generating vehicle virtual image display data are provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0081] In this embodiment, a method for generating vehicle virtual image display data is provided. Figure 1 is a flow chart of a method for generating vehicle virtual image display data according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0082] Step S101, when the head-up display service of the vehicle is started, the environmental perception data of the current environment of the vehicle is obtained, and the vehicle motion data and satellite navigation data of the vehicle are detected.

[0083] In an embodiment of the present application, the head-up display service system of the vehicle is started through the electronic control system of the vehicle or a specific software startup program. After the head-up display service is started, the sensors or cameras deployed in the vehicle are called to detect the current environment and driving conditions of the vehicle, and the detected data is passed to the corresponding data processing module for processing. The data processing module includes: an environmental perception module, an algorithm module, a map module and a rendering module. Among them, the environmental perception module is used to receive video stream data, and after processing the video stream data, the perception algorithm perceives the environmental elements around the vehicle. The algorithm module is used to receive a variety of data, output six degrees of freedom (6dof) data for 3D rendering through a posture algorithm, and can estimate ambient light information to enhance the effect of virtual image fusion with reality. The map module is used to receive satellite navigation data and locate the vehicle, and provide path planning and navigation guidance and other functions based on this.

[0084] Specifically, environmental perception data can be obtained in the following ways: First, start the sensor devices such as cameras and environmental sensors on the vehicle. These sensors can capture environmental information around the vehicle, such as road conditions, other vehicles, pedestrians, obstacles, etc. Make sure that the sensor parameters are set correctly, such as the resolution and frame rate of the camera, the detection range and accuracy of the radar, etc. The sensor starts to collect environmental data and transmits the raw data to the data processing unit. For example, the video stream data captured by the camera, and the environmental sensor collects environmental data, such as light, weather, etc. Then pre-process the collected raw data to remove noise, correct the image, etc. Filtering algorithms, image enhancement algorithms, etc. can be used to process the data to improve the quality of the data. Finally, the data from different sensors are fused to obtain more comprehensive and accurate environmental perception information.

[0085] Vehicle motion data can be obtained in the following ways: connect the speed sensor, acceleration sensor, gyroscope and other sensors deployed on the vehicle to read the vehicle motion data from the sensor, and perform analysis and conversion to obtain the vehicle motion data. For example, convert the original signal output by the sensor into actual speed, acceleration, rotation angle, wheel speed and other values.

[0086] The satellite navigation data of the vehicle can be obtained in the following ways: First, determine the device status of the satellite navigation device on the vehicle, the satellite navigation device can be such as a GPS receiver. If the device status is turned on, it is determined that the receiver can normally receive satellite signals and establish a connection with the satellite navigation system. The receiver receives navigation data from the satellite, including: position data (such as longitude, latitude, altitude, etc.), time information, speed data (such as horizontal speed, vertical speed, etc.).

[0087] Step S102: Analyze the environmental perception data to obtain environmental elements of the environment in which the vehicle is located, and calculate the environmental adaptation information of the vehicle based on the environmental perception data and the vehicle motion data.

[0088] In the embodiment of the present application, analyzing the environmental perception data to obtain the environmental elements of the environment where the vehicle is located includes the following steps A1-A4:

[0089] Step A1, extracting video stream data from environmental perception data, wherein the video stream data includes multiple frames of continuous perception images.

[0090] Specifically, after receiving the environmental perception data, the environmental perception module extracts video stream data from the environmental perception data. The video stream data consists of multiple frames of continuous perception images, each of which contains various information about the vehicle's surroundings, such as the shape, texture and color of the road, the position, size and shape of other vehicles, the appearance and movement of pedestrians, the style and position of traffic signs and markings, etc. The details in the image can provide rich clues for environmental perception and analysis, such as cracks on the road, the license plate number of the vehicle, the expression of the pedestrian, etc., and this information may have different importance for different application scenarios.

[0091] Step A2, detecting the regional features of each region in the perceived image, and inferring the regional features to obtain element inference results in each region, wherein the element inference results include multiple candidate environmental elements and confidences corresponding to the candidate environmental elements.

[0092] Specifically, for each perceived image frame in the video stream, the environmental perception module divides the perceived image into multiple regions. The division method can be regular grid division, or adaptive division based on the content and structure of the image. Then, feature detection is performed on each region. These regional features can include color features, texture features, shape features, etc. For example, color features can be the average color and color distribution of the region; texture features can be the roughness and directionality of the region; shape features can be the outline shape and geometric shape of the region.

[0093] Based on the detected regional features, the environmental perception module uses a pre-trained inference model to infer each area. This inference model can be a neural network model based on deep learning, such as a convolutional neural network (CNN), a recurrent neural network (RNN), etc. The result of the inference is the candidate environmental elements that may exist in each area and the confidence level corresponding to each candidate environmental element. Candidate environmental elements may include lane lines, road boundaries, traffic lights, ground signs, pedestrians, vehicles, etc. The confidence level indicates the possibility that the candidate environmental element exists in the area, and is a value between 0 and 1.

[0094] Step A3: obtaining a preset confidence threshold.

[0095] Specifically, in environmental perception, due to the influence of various factors, the reasoning results of the environmental perception module for candidate environmental elements may have certain uncertainties. The purpose of setting the confidence threshold is to screen out those candidate environmental elements with higher reliability to ensure that the final environmental elements are more accurate and reliable. For example, if all candidate environmental elements are regarded as real environmental elements, erroneous detection results may be introduced, thereby affecting subsequent decision-making and control. By setting the confidence threshold, the probability of false detection can be reduced and the accuracy of environmental perception can be improved.

[0096] The confidence threshold can be determined by analyzing the confidence distribution of candidate environmental elements in different scenarios and determining the confidence threshold based on the statistical results. For example, the average confidence and confidence variance of different candidate environmental elements in different scenarios can be calculated, and then the confidence threshold can be determined based on these statistics.

[0097] Step A4: taking the candidate environmental elements whose confidence is higher than the confidence threshold as environmental elements.

[0098] Specifically, for each candidate environmental element, the corresponding confidence is compared with a preset confidence threshold. If the confidence of the candidate environmental element is higher than the confidence threshold, the candidate environmental element is used as a determined environmental element. For example, if the confidence threshold is 0.8 and the confidence of a candidate environmental element is 0.85, then the candidate environmental element is determined as an environmental element. The screened environmental elements are output to a subsequent rendering module.

[0099] Based on this, by obtaining a preset confidence threshold and taking candidate environmental elements with confidence higher than the threshold as environmental elements, the accuracy and reliability of environmental perception can be improved.

[0100] The method provided in the embodiment of the present application can effectively reduce the number of environmental elements that are incorrectly identified and reduce the false alarm rate of the system through strict confidence screening. At the same time, through multi-frame analysis and regional feature reasoning, it is possible to better cope with various complex situations and improve the stability and adaptability of the system. For example, in severe weather conditions, image quality may be affected, but through multi-frame continuous analysis and confidence screening, key environmental elements can still be identified as accurately as possible.

[0101] It should be noted that the processing framework of video stream data is as follows Figure 2As shown in the figure, the Camera device on the vehicle is responsible for capturing video stream data. Cameras are installed in different positions. Camera devices can be ordinary visible light cameras, or they may include infrared cameras, depth cameras, etc., for different application scenarios. The Image Signal Processor (ISP) processes the captured raw images. ISP processing includes operations such as denoising, contrast enhancement, color balance adjustment, and automatic exposure control to improve image quality. The processed images are clearer and more accurate in color, which is suitable for subsequent processing and analysis.

[0102] The video stream data passes through the underlying driver layer for hardware interface management and data transmission control. This layer is responsible for communicating with the Camera device, reading the video stream data from the hardware device, and providing an interface for the upper software. The underlying driver layer ensures that the video stream data can be correctly transmitted from the Camera device to other parts of the Android system. HAL (Hardware Abstraction Layer) is a layer in the Android system that is used to shield the underlying hardware differences. It provides a unified interface for the upper Android Framework layer, so that different hardware devices can be accessed in the same way in the Android system. In this process, the video stream data passes through the Android HAL layer, so that the Android system can interact with a variety of different Camera devices without relying on specific hardware implementations. The Electronic Video Stabilization manager (EVS manager) may play a role in video stabilization in this process. It can process the video stream data to reduce image jitter caused by vehicle vibration or other factors. EVS manager ensures that the video stream data provided to the upper layer is more stable, improving the visual effect and the accuracy of subsequent processing. The video stream data finally reaches the Android Framework layer, which is the core part of the Android system and provides a framework for various services and functions.

[0103] In the actual acquisition process, the environment perception module registers the Camera image data callback interface, and the environment perception module can be notified when there is new image frame data. At the same time, the Camera data capture interface is called to start actively acquiring video stream data to ensure that the latest perception image can be obtained in time. When the Android HAL (Hardware Abstraction Layer) layer has image frame data, the EVS Manager service notifies the environment perception module in the head-up display service system through the HAL interface. This can achieve efficient data transmission and processing, ensuring that image data can be quickly passed to the perception module for subsequent processing.

[0104] like Figure 3 As shown in the figure, after the environment perception module reads the image data, it performs data processing such as image format conversion and distortion correction processing. Image format conversion is to convert the original image format into a format suitable for subsequent processing, such as converting the format output by a specific camera into a universal image format. Distortion correction processing is to solve the image distortion problem that may be caused by the camera lens and ensure that the shape and position of the object in the image are more accurate. The processed image data is sent to the perception algorithm unit, which includes: lane line perception algorithm, object perception algorithm / dynamic object tracking algorithm, etc. The environmental elements are perceived through the above algorithms. Then, based on the inference acceleration unit (such as TNN, SNPE, SIDL, etc.), the hardware computing unit (CPU, GPU, NPU, etc.) is used for inference. The reasoning efficiency is improved by the cooperation between the inference acceleration unit and the hardware computing unit. They can use deep learning algorithms to analyze and process image data and identify various environmental elements around the vehicle.

[0105] In an embodiment of the present application, calculating the environment adaptation information of the vehicle based on the environment perception data and the vehicle motion data includes the following steps B1-B6:

[0106] Step B1, extracting video stream data and light detection data from the environmental perception data, wherein the video stream data includes multiple frames of continuous perception images.

[0107] Specifically, the video stream data consists of multiple frames of continuous perception images, each of which contains various information about the vehicle's surroundings, such as the shape, texture and color of the road, the position, size and shape of other vehicles, the appearance and movement of pedestrians, the style and position of traffic signs and markings, etc. Light detection data is collected by a dedicated light sensor. The light sensor can measure parameters such as the light intensity and color temperature of the vehicle's surroundings. These data are used to adjust the image display effect.

[0108] Step B2, detecting multiple frames of continuous perceived images in the video stream data, and obtaining key feature points in each frame of the perceived image.

[0109] Specifically, the algorithm module can use various image processing algorithms and techniques to detect key feature points in each frame of the image. Common feature point detection algorithms include scale-invariant feature transform (SIFT), speeded up robust features (SURF), oriented FAST, and rotation BRIEF (ORB). These algorithms usually look for unique and stable points in the image, such as corner points, edge points, spots, etc. These points have good invariance under different image transformations (such as rotation, scaling, translation, etc.), and can effectively represent the content and structure of the image.

[0110] The algorithm module selects every two adjacent frames of perception images in the video stream for analysis. Since the video stream is continuous, the time interval between two adjacent frames is usually very short, so the changes in the vehicle's surroundings recorded by them are relatively small.

[0111] For each two adjacent frames, their key feature points need to be matched first. This can be achieved by comparing the descriptors of the feature points. The descriptor is a mathematical description of the area around the feature point, which can reflect the local characteristics of the feature point. Feature point matching methods can be Brute-Force Matching and Fast Approximate Nearest Neighbor Search (FLANN). These methods can quickly find similar feature points in two frames of images, thereby establishing a correspondence between them.

[0112] Step B3: Analyze the key feature points of each two adjacent frames of perceived images to determine the relative motion information between each two adjacent frames of perceived images.

[0113] Specifically, the relative motion information between each two adjacent frames can be calculated by matching key feature points. That is, the relative motion information is determined by tracking the position changes of key feature points in continuous images. For example, algorithms such as Kalman filters can be used to predict and update the positions of feature points to obtain their motion trajectories.

[0114] Step B4, fusing the relative motion information and the vehicle motion data to obtain the vehicle's position and posture information.

[0115] Specifically, the relative motion information and vehicle motion data are integrated to obtain the vehicle's posture information, including: obtaining IMU data and wheel speed data from the vehicle motion data; calculating the vehicle's motion changes based on the acceleration and angular velocity contained in the IMU data, and calculating the corresponding displacement changes of the vehicle based on the wheel speed data; predicting the vehicle's preliminary posture information using the motion changes and displacement changes; and updating the preliminary posture information based on the relative motion information to obtain posture information.

[0116] It should be noted that acceleration can be used to calculate the change in the vehicle's linear acceleration. By integrating the acceleration over time, the change in the vehicle's speed can be obtained. For example, if the acceleration remains constant over a period of time, the change in speed is equal to the acceleration multiplied by time. Angular velocity can be used to calculate the change in the vehicle's angular acceleration. By integrating the angular velocity over time, the change in the vehicle's angle can be obtained. For example, if the angular velocity remains constant over a period of time, the change in angle is equal to the angular velocity multiplied by time. The wheel speed data can be used to calculate the displacement change of the vehicle through the circumference and rotation speed of the wheel. Assuming the circumference of the wheel is C and the wheel speed is v, the displacement change of the vehicle over a period of time is equal to C×v×t, where t is time.

[0117] First, the calculated motion change and displacement change are combined to predict the preliminary position and posture information of the vehicle. Position and posture information usually includes the position and attitude of the vehicle. The position can be represented by three-dimensional coordinates (x, y, z), and the attitude can be represented by Euler angles (roll, pitch, yaw) or quaternions. By integrating the velocity and displacement, the position change of the vehicle can be obtained, thereby updating the position coordinates of the vehicle. By integrating the angle change, the attitude change of the vehicle can be obtained, thereby updating the attitude parameters of the vehicle.

[0118] The predicted preliminary pose information will have certain uncertainty and errors. This is because the IMU data and wheel speed data may contain noise and errors, and the integration process will also introduce cumulative errors. Therefore, the preliminary pose information needs to be further updated and corrected.

[0119] Secondly, relative motion information is obtained by analyzing the environmental perception data around the vehicle. For example, the motion of the vehicle relative to the surrounding environment can be determined by analyzing two adjacent frames of video stream data. This relative motion information can provide additional information about the displacement and rotation changes of the vehicle.

[0120] Then, the preliminary pose information is updated and corrected using the relative motion information. Various algorithms, such as Kalman filtering, extended Kalman filtering, or particle filtering, can be used to fuse the relative motion information with the preliminary pose information. These filtering algorithms can perform weighted fusion of the relative motion information and preliminary pose information based on their uncertainties to obtain more accurate pose information. At the same time, the filtering algorithms can also estimate and correct errors to improve the accuracy and reliability of pose information.

[0121] The final pose information is usually expressed in the form of six degrees of freedom (6dof), which includes three translational degrees of freedom (displacement in the x, y, z directions) and three rotational degrees of freedom (rotation in the roll, pitch, yaw directions).

[0122] Step B5, analyzing the perceived image and light detection data in the video stream data to obtain ambient light information.

[0123] Specifically, the perceived image and the light detection data in the video stream data are analyzed to obtain the ambient light information, including: analyzing the image features of the perceived image according to a preset dimension to obtain the light feature parameters corresponding to the preset dimension; fusing the light detection data and the light feature parameters corresponding to the preset dimension to obtain the ambient light information.

[0124] It should be noted that the preset dimensions include one or more of color, brightness, and shadow. Among them, the analysis of the color in the perceived image can be: determining the color characteristics by statistically analyzing the distribution of different color channels in the image. For example, the mean, variance and other statistics of the red, green, and blue channels in the image are calculated to understand the overall color tendency of the image. Objects or areas of specific colors can also be identified, such as the color of traffic lights. The analysis of brightness can be: calculating the average brightness value of the image, or dividing the image into different areas and calculating the brightness value of each area separately. The analysis of shadows can be: detecting the shadow area in the image. By analyzing the unevenly illuminated parts of the image, the position, shape, and size of the shadow can be determined.

[0125] By analyzing the preset dimensions, the corresponding illumination characteristic parameters are obtained. For example, for the color dimension, the color histogram, main color and other parameters may be obtained; for the brightness dimension, the average brightness value, brightness range and other parameters may be obtained; for the shadow dimension, the shadow area, shadow direction and other parameters may be obtained.

[0126] The light detection data is then fused with the light feature parameters analyzed from the perceived image. A variety of methods can be used for fusion, such as weighted averaging, rule-based fusion, etc. For example: different weights are assigned to the light detection data and light feature parameters according to the reliability and importance of different data, and then weighted averaging is performed to obtain the final ambient light information. For example, if the data from the light sensor is relatively accurate, a higher weight can be given; while the image feature parameters may be affected by image quality and environmental factors, and the weight can be relatively low.

[0127] After fusion, comprehensive ambient lighting information is obtained. Ambient lighting information can include multiple aspects such as light intensity, color temperature, light direction, shadow distribution, etc., which can more comprehensively and accurately reflect the lighting conditions of the vehicle's surrounding environment.

[0128] Step B6, generating environment adaptation information based on the posture information and the environment lighting information.

[0129] It should be noted that the above-mentioned light detection data and vehicle motion data are all detected by sensors on the vehicle. The processing framework of the sensor is as follows Figure 4 As shown in the figure, the sensor data first passes through the Sensor Core of the driver layer. This part is responsible for managing and controlling the underlying driver of the sensor hardware, and preliminarily processing and encapsulating the raw data of the sensor. The data is transmitted from the driver layer to the Android HAL (Hardware Abstraction Layer). The function of the HAL layer is to provide a unified interface for the upper-layer Android system to shield the differences between different sensor hardware. In this way, regardless of the specific model and manufacturer of the sensor, the Android system can access the sensor data in the same way, improving the scalability and compatibility of the system. Finally, the sensor data is managed by the Sensor Service. SensorService is a service component in the Android system that is specifically responsible for processing sensor-related tasks. ARHUDService (Augmented Reality Head-Up Display Service) requires sensor data to implement its functions. It obtains sensor data by calling the SensorService service. ARHUD Service processes the sensor data obtained from the Sensor Service. The processed sensor data is sent to the algorithm module.

[0130] Specifically, Figure 5As shown in the figure, the algorithm module is deployed with SLAM (Simultaneous Localization and Mapping) and VIO (Visual Inertial Odometry) algorithms. The algorithm module uses the input video stream data, IMU and wheel speed data to calculate the position and posture of the vehicle in three-dimensional space in real time (6dof, i.e. six degrees of freedom, including three translational degrees of freedom and three rotational degrees of freedom). Through the analysis and feature point matching of continuous image frames, as well as the motion estimation combined with IMU data and wheel speed data, the vehicle's posture information can be continuously updated to achieve accurate tracking of the vehicle's position and posture. At the same time, the DSP algorithm acceleration unit is used to improve the calculation speed and efficiency of the posture algorithm, ensuring that the posture data can be output quickly and accurately in real-time applications.

[0131] The method provided in the embodiment of the present application extracts multiple frames of continuous perception images from environmental perception data for analysis and detects key feature points. Multi-frame images provide richer information and help to more accurately determine the key features in the image. By analyzing the key feature points of each two adjacent frames of perception images, relative motion information is determined. This method makes full use of the changes between continuous images and can more accurately reflect the motion state of the vehicle in the actual environment. Compared with the vehicle motion data at a single moment, the relative motion information can reduce errors and improve the accuracy of the posture information. The relative motion information is fused with the vehicle motion data to obtain more comprehensive and accurate vehicle posture information. The vehicle motion data may come from devices such as sensors, but there may be certain errors, and the relative motion information obtained by fusing multi-frame image analysis can be corrected and supplemented. At the same time, the perception images and lighting detection data in the video stream data can be analyzed to more accurately obtain the ambient lighting information. The perception images can provide the lighting effects in the actual scene, and the lighting detection data can provide more specific lighting parameters. Based on accurate posture information and ambient lighting information, environmental adaptation information is generated, so that the system can better adapt to different environmental conditions.

[0132] Step S103, analyzing the satellite navigation data to obtain navigation information corresponding to the vehicle.

[0133] In the embodiment of the present application, analyzing the satellite navigation data to obtain the navigation information corresponding to the vehicle includes the following steps C1-C3:

[0134] Step C1, obtaining vehicle position data and surrounding environment data of the vehicle based on satellite navigation data.

[0135] Specifically, satellite navigation systems (such as GPS, BeiDou, etc.) can determine the precise location of a vehicle on the earth by receiving signals from multiple satellites. Location data is usually expressed in the form of longitude and latitude coordinates, and can also include information such as altitude. In addition to the vehicle's location, satellite navigation systems can also provide some surrounding environment data. For example, some satellite navigation map software can display information such as nearby roads, buildings, landmarks, etc.

[0136] Step C2, obtaining the current navigation requirements of the vehicle.

[0137] Specifically, navigation requirements may include route planning, navigation guidance, real-time positioning, POI search, etc.

[0138] Step C3, processing the vehicle position data and surrounding environment data based on a processing strategy corresponding to the navigation requirement to obtain navigation information.

[0139] Specifically, corresponding processing strategies are pre-set for different navigation requirements. For example, if the navigation requirement is the fastest route, the processing strategy may be to give priority to highways and main roads while avoiding traffic congestion. If it is the shortest route, the processing strategy may be to choose the shortest road, even if these roads may be narrow or in poor condition.

[0140] According to the selected processing strategy, the vehicle position data and surrounding environment data are processed to obtain navigation information. For example: For path planning: map data and algorithms can be used to calculate the best path from the vehicle's current location to the destination. For real-time navigation: during the vehicle's driving process, navigation information can be updated in real time according to changes in the vehicle's location and surrounding environment. For providing surrounding information: in addition to path navigation, relevant information about the surrounding environment can also be provided, such as the location and distance of gas stations, restaurants, parking lots, etc. This information can help the driver make decisions during driving.

[0141] like Figure 6 As shown in the figure, the input of the map module is satellite navigation data, which includes GNSS data and network data. The main functional modules of the map module include path planning, navigation guidance, real-time positioning, POI, traffic information prompts, inertial navigation and other functions.

[0142] Step S104: fuse the environmental elements, the environmental adaptation information and the navigation information based on the fusion strategy corresponding to the head-up display service to obtain virtual image display data, and display the virtual image display data in the head-up display area of ​​the vehicle.

[0143] In the embodiment of the present application, the environmental elements, the environmental adaptation information and the navigation information are fused based on the fusion strategy corresponding to the head-up display service to obtain the virtual image display data, including the following steps D1-D4:

[0144] Step D1, obtaining posture information and ambient lighting information from the environment adaptation information.

[0145] Step D2, determining the display orientation corresponding to the environmental element according to the position information, and constructing initial display data based on the environmental element and the display orientation;

[0146] Specifically, determining the display orientation corresponding to the environmental element according to the posture information includes: converting the posture information of the vehicle into the world coordinate system to obtain the first coordinate information and angle information of the vehicle in the world coordinate system; converting the environmental element into the world coordinate system to obtain the second coordinate information of the environmental element in the world coordinate system; determining the display position of the environmental element according to the positional relationship between the first coordinate information and the second coordinate information, and determining the display direction of the environmental element according to the angle information; and constructing the display orientation corresponding to the environmental element based on the display position and the display direction.

[0147] First, select a suitable world coordinate system as a reference, which can be the initial position of the vehicle or a fixed point on the map. Convert the vehicle's posture data from its original coordinate system to the world coordinate system, and convert the environmental element data to the world coordinate system to obtain the first coordinate information and angle information of the vehicle in the world coordinate system, and the second coordinate information of the environmental element in the world coordinate system.

[0148] After obtaining the first coordinate information of the vehicle in the world coordinate system and the second coordinate information of the environmental element in the world coordinate system, the positional relationship between them is calculated to determine the position of the environmental element in the display screen. For example, if the vehicle is located at the coordinates (x1, y1, z1) and a certain environmental element is located at the coordinates (x2, y2, z2), the relative position of the environmental element in the display screen can be determined based on the difference between the two in the three directions of x, y, and z, as well as the size and ratio of the display screen. For example, if the environmental element is in front of the vehicle and the distance is relatively close, it will be displayed in the center of the display screen; if it is on the left side of the vehicle and the distance is relatively far, it will be displayed at the left edge of the screen.

[0149] In addition, the angle information of the vehicle in the world coordinate system indicates the orientation of the vehicle. This angle information can be used to determine the display direction of the environmental element in the display screen. For example, if the vehicle is facing east and a certain environmental element is on the north side of the vehicle, then in the display screen, this environmental element should be displayed above the vehicle (assuming that the top of the display screen corresponds to the north). By matching the angle information of the vehicle with the relative direction of the environmental element, the display direction of the environmental element in the display screen can be accurately determined.

[0150] By comprehensively considering the display position and display direction of environmental elements, the display orientation corresponding to the environmental elements can be constructed. The display orientation determines the specific presentation of environmental elements in the augmented reality display screen. For example, if it is a traffic light, the display orientation can determine its position and orientation in the screen so that the driver can intuitively see the status of the traffic light and its relative position to the vehicle. In addition, in the process of constructing the display orientation, other factors can also be considered, such as the importance of environmental elements, distance, display priority, etc., to optimize the display effect.

[0151] Step D3, fusing the navigation information with the initial display data to obtain target display data.

[0152] Specifically, the navigation information is fused with the initial display data to obtain the target display data, including: obtaining the guiding element carried in the navigation information; obtaining the target environment element matching the guiding element from the initial display data; obtaining the fusion strategy corresponding to the guiding element; and fusing the guiding element with the target environment element in the initial display data according to the fusion strategy to obtain the target display data.

[0153] First, obtain guiding elements from the navigation information. Guiding elements can be arrow indications, route markings, distance prompts, etc., where arrow indications are used to indicate the direction of the vehicle. Arrows can be simple straight arrows or arrows with turning angles to more accurately prompt users to turn in the direction at intersections or forks. Route markings usually display the route of the vehicle on the map in different colors or line styles. Route markings can help users intuitively understand which road they should take. Distance prompts are used to show the distance between the vehicle and the next key location (such as an intersection, a turn, a destination, etc.).

[0154] Secondly, find the target environmental elements that match the guiding elements from the initial display data. The initial display data needs to be analyzed and identified to determine which environmental elements are related to the guiding elements in the navigation information. For example, if there is an arrow in the navigation information indicating a right turn, then it is necessary to find the corresponding environmental elements such as intersections or forks in the initial display data as target environmental elements. The matching of environmental elements can be achieved through image recognition technology, sensor data fusion or other methods.

[0155] Then, a fusion strategy corresponding to the guiding element is obtained. The fusion strategy refers to the specific method and rules for fusing the guiding element with the target environmental element. The selection of the fusion strategy depends on many factors, including the type of guiding element, the characteristics of the target environmental element, the performance of the display device, and the needs of the user. For example, for arrow indication, the arrow can be superimposed on the road in the video stream so that it looks like it is pointing directly to the direction the vehicle should travel. For distance prompts, they can be displayed in the form of numbers or text near the target environmental element in the display screen so that the user can clearly see the distance information.

[0156] For example, the transparency of the guiding element can be adjusted so that it does not completely block the target environment element when it is integrated with the target environment element, while clearly displaying the guiding information. For example, the transparency of the arrow indication can be set to a certain degree so that it is integrated with the image of the road, which does not affect the user's observation of the road, while clearly indicating the driving direction.

[0157] Alternatively, choose a color that coordinates with the target environment element to display the guiding element to improve the visual effect of integration. For example, if the target environment element is a green road, you can choose to use green arrows or route markings to match the color of the road to enhance the overall visual consistency.

[0158] Step D4, rendering the target display data using the ambient lighting information to obtain virtual image display data.

[0159] Specifically, the brightness and contrast of the target display data are adjusted according to the ambient light intensity. In a strong light environment, the brightness of the virtual image needs to be reduced to avoid excessive glare; in a weak light environment, the brightness can be appropriately increased to ensure clear visibility. The adjustment of contrast can make different elements in the virtual image more prominent.

[0160] At the same time, the color temperature of the environment can also be taken into account to correct the color of the virtual image. For example, if the ambient light is warm, the color of the virtual image needs to be adjusted to make it look more natural under this lighting. This can be achieved by adjusting parameters such as the color balance and saturation of the image. Then, shadows and lighting effects can be simulated in the virtual image using the lighting direction in the ambient lighting information. The realism of the virtual image can be enhanced to better integrate it into the actual environment.

[0161] After the rendering process, the virtual image display data obtained is the target display data adjusted by the ambient lighting information. The virtual image display data contains all the information that needs to be presented in the vehicle head-up display area, and has been optimized according to the actual ambient lighting to ensure the best visual effect.

[0162] Finally, the virtual image display data is sent to the rendering engine. The rendering engine is a key component responsible for converting input data into visual images. After receiving the virtual image display data, it processes the data according to the preset rendering rules and algorithms to generate an image suitable for display in the head-up display area. The rendered image is projected to the head-up display area of ​​the vehicle. In the head-up display area, the driver can see the virtual image display data processed by the rendering engine. In addition, as the vehicle travels and the environment changes, the virtual image display data needs to be updated in real time.

[0163] The method provided in the embodiment of the present application uses posture information to ensure that virtual elements can be accurately positioned in appropriate positions in the real environment, while the ambient lighting information makes the display of virtual elements more natural and integrated with the actual environment. The display orientation of the environmental elements is determined based on the posture information, and the initial display data is constructed in combination with the environmental elements. This enables the virtual elements to be presented to the user in the correct direction and position, improving the readability and usability of the information. The navigation information is merged with the initial display data to obtain the target display data. This enables the navigation instructions to be closely integrated with the actual environmental elements, providing users with accurate navigation guidance. Finally, the target display data is rendered using the ambient lighting information, further enhancing the realism of the virtual image display data. The rendered virtual elements can better adapt to different lighting conditions and are consistent with the actual scene.

[0164] In this embodiment, a system for generating vehicle virtual image display data is also provided. Figure 7 As shown, the system includes: a detection component, a data processing component, a rendering and display component and a computing power component, the data processing component includes: an environment perception module, an algorithm module and a map module, the rendering and display component includes a rendering module and a HUD display module, and the computing power component includes multiple computing power units, wherein the computing power units associated with the environment perception module, the algorithm module, the map module and the rendering module are different;

[0165] A detection component, used to detect environmental perception data of the current environment of the vehicle, vehicle motion data and satellite navigation data, and transmit the environmental perception data, vehicle motion data and satellite navigation data to a data processing component;

[0166] The environment perception module in the data processing component is used to analyze the environment perception data to obtain the environmental elements of the environment where the vehicle is located, and transmit the environmental elements to the rendering display module;

[0167] The algorithm module in the data processing component is used to calculate the vehicle's environmental adaptation information based on the environmental perception data and the vehicle motion data, and transmit the environmental adaptation information to the rendering display module;

[0168] The map module in the data processing component is used to analyze the satellite navigation data to obtain the navigation information corresponding to the vehicle, and transmit the navigation information to the rendering display module;

[0169] The rendering module in the rendering display module is used to fuse the environmental elements, environmental adaptation information and navigation information based on the fusion strategy corresponding to the head-up display service to obtain virtual image display data, and transmit the virtual image display data to the HUD display module;

[0170] The HUD display module is used to display virtual image display data in the head-up display area of ​​the vehicle.

[0171] It should be noted that the system architecture of the embodiment of the present application is built on the Android system and the QNX (Linux) system, and has high flexibility and compatibility. It can run on the same hardware platform to achieve centralized resource utilization and efficient collaboration; it can also run on different hardware platforms to meet different application scenarios and requirements and adapt to the hardware configuration of various vehicles.

[0172] The embodiment of the present application deploys high computing power requirements such as artificial intelligence perception and simultaneous positioning and mapping (SLAM, VIO) algorithms in computing units such as APU (accelerated processing unit), NPU (neural network processing unit), DSP (digital signal processor), GPU (graphics processing unit). In this way, the computing power load of the CPU (central processing unit) unit is effectively reduced, and the CPU is prevented from having performance bottlenecks due to taking on too many computing tasks. By allocating different algorithm modules and functions to different hardware computing units, the various software and hardware resources of the platform can be fully utilized and utilized.

[0173] For example, the main computing power of the environment perception module is based on NPU / APU, which can use the powerful parallel computing capabilities of these units to handle complex artificial intelligence perception tasks; the main computing power of the algorithm module is based on DSP / CPU, combining the advantages of both for real-time environment perception and posture calculation. The main computing power of the map module is based on CPU, which uses its capabilities in data processing and logical operations to process navigation data; the main computing power of the rendering module is based on GPU, which gives full play to its efficient performance in graphics processing. By sharing their respective functions with different computing units, the system freeze problem caused by insufficient computing power of a certain computing unit is effectively reduced.

[0174] The distributed architecture provided by the embodiment of the present application improves the balance of system computing resource utilization, allowing each module to operate efficiently, thereby achieving a smooth experience of functional effects and providing users with smooth and stable augmented reality display. It effectively solves some key problems faced by ARHUD and improves system performance and user experience.

[0175] In this embodiment, a device for generating vehicle virtual image display data is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0176] This embodiment provides a device for generating vehicle virtual image display data, such as Figure 8 As shown, including:

[0177] The acquisition module 801 is used to acquire the video stream data of the current environment of the vehicle when the head-up display service of the vehicle is started, and detect the vehicle motion data and satellite navigation data of the vehicle in the current environment;

[0178] A first analysis module 802 is used to analyze the video stream data to obtain environmental elements of the environment where the vehicle is located, and calculate the environmental adaptation information of the vehicle based on the environmental perception data and the vehicle motion data;

[0179] The second analysis module 803 is used to analyze the satellite navigation data to obtain navigation information corresponding to the vehicle;

[0180] The fusion module 804 is used to fuse the environmental elements, the environmental adaptation information and the navigation information based on the fusion strategy corresponding to the head-up display service to obtain virtual image display data, and display the virtual image display data in the head-up display area of ​​the vehicle.

[0181] In the embodiment of the present application, the first analysis module 802 includes:

[0182] The first extraction submodule is used to extract video stream data from the environmental perception data, wherein the video stream data includes multiple frames of continuous perception images.

[0183] The detection submodule is used to detect the regional features of each area in the perceived image and infer the regional features to obtain the element inference results in each area, wherein the element inference results include multiple candidate environmental elements and the confidence levels corresponding to the candidate environmental elements.

[0184] The acquisition submodule is used to obtain a preset confidence threshold.

[0185] The comparison submodule is used to take the candidate environmental elements whose confidence is higher than the confidence threshold as environmental elements.

[0186] In the embodiment of the present application, the first analysis module 802 includes:

[0187] The second extraction submodule is used to extract video stream data and light detection data from the environmental perception data, wherein the video stream data includes multiple frames of continuous perception images; detect multiple frames of continuous perception images in the video stream data to obtain key feature points in each frame of perception images.

[0188] The first analysis submodule is used to analyze key feature points of each two adjacent frames of perceived images to determine relative motion information between each two adjacent frames of perceived images;

[0189] A fusion submodule is used to fuse the relative motion information and the vehicle motion data to obtain the vehicle's position and posture information;

[0190] The second analysis submodule is used to analyze the perceived image and the light detection data in the video stream data to obtain the ambient light information;

[0191] The generation submodule is used to generate environment adaptation information based on posture information and ambient lighting information.

[0192] In an embodiment of the present application, a fusion submodule is used to obtain IMU data and wheel speed data from vehicle motion data; calculate the vehicle's motion changes based on the acceleration and angular velocity contained in the IMU data, and calculate the corresponding displacement changes of the vehicle based on the wheel speed data; predict the vehicle's preliminary posture information using the motion changes and displacement changes; and update the preliminary posture information based on the relative motion information to obtain posture information.

[0193] In an embodiment of the present application, the second analysis submodule is used to analyze the image features of the perceived image according to a preset dimension to obtain lighting feature parameters corresponding to the preset dimension, wherein the preset dimension includes one or more of color, brightness and shadow; and to fuse the lighting detection data and the lighting feature parameters corresponding to the preset dimension to obtain ambient lighting information.

[0194] In an embodiment of the present application, the second analysis module 803 is used to obtain the vehicle position data and surrounding environment data of the vehicle based on satellite navigation data; obtain the current navigation requirements of the vehicle; and process the vehicle position data and surrounding environment data based on a processing strategy corresponding to the navigation requirements to obtain navigation information.

[0195] In an embodiment of the present application, the fusion module 804 is used to obtain posture information and ambient lighting information from the environmental adaptation information; determine the display orientation corresponding to the environmental elements according to the posture information, and construct initial display data based on the environmental elements and the display orientation; fuse the navigation information with the initial display data to obtain target display data; and render the target display data using the ambient lighting information to obtain virtual image display data.

[0196] In an embodiment of the present application, the fusion module 804 is used to convert the vehicle's posture information into a world coordinate system to obtain first coordinate information and angle information of the vehicle in the world coordinate system; convert environmental elements into the world coordinate system to obtain second coordinate information of the environmental elements in the world coordinate system; determine the display position of the environmental elements according to the positional relationship between the first coordinate information and the second coordinate information, and determine the display direction of the environmental elements according to the angle information; and construct the display orientation corresponding to the environmental elements based on the display position and the display direction.

[0197] In an embodiment of the present application, the fusion module 804 is used to obtain the guiding element carried in the navigation information; obtain the target environmental element matching the guiding element from the initial display data; obtain the fusion strategy corresponding to the guiding element; and fuse the guiding element with the target environmental element in the initial display data according to the fusion strategy to obtain the target display data.

[0198] An embodiment of the present invention provides a vehicle, which includes: a system for generating vehicle virtual image display data and a controller; wherein the controller stores instructions for controlling the operation of the system for generating vehicle virtual image display data, and the system for generating vehicle virtual image display data is used to execute the method for generating vehicle virtual image display data in the above embodiment according to the instructions of the controller.

[0199] See also Fig. 9 , Fig. 9 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig. 9 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system).

[0200] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0201] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0202] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a computer device based on the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0203] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0204] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0205] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0206] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for generating vehicle virtual image display data, characterized in that: The method comprises: When the head-up display service of the vehicle is started, obtaining environmental perception data of the current environment of the vehicle, and detecting vehicle motion data and satellite navigation data of the vehicle; Analyzing the environmental perception data to obtain environmental elements of the environment where the vehicle is located, and calculating environmental adaptation information of the vehicle based on the environmental perception data and the vehicle motion data; Analyzing the satellite navigation data to obtain navigation information corresponding to the vehicle; The environmental elements, the environmental adaptation information and the navigation information are fused based on a fusion strategy corresponding to the head-up display service to obtain virtual image display data, and the virtual image display data is displayed in a head-up display area of ​​the vehicle.

2. The method according to claim 1, characterized in that The step of analyzing the environmental perception data to obtain environmental elements of the environment where the vehicle is located includes: Extracting video stream data from the environmental perception data, wherein the video stream data includes multiple frames of continuous perception images; Detecting regional features of each region in the perceived image, and inferring the regional features to obtain element inference results in each of the regions, wherein the element inference results include a plurality of candidate environmental elements and confidences corresponding to the candidate environmental elements; Get the preset confidence threshold; The candidate environmental element whose confidence level is higher than the confidence threshold is taken as the environmental element.

3. The method according to claim 1, characterized in that The calculating the environment adaptation information of the vehicle based on the environment perception data and the vehicle motion data includes: Extracting video stream data and light detection data from the environmental perception data, wherein the video stream data includes multiple frames of continuous perception images; Detecting a plurality of continuous frames of perceived images in the video stream data to obtain key feature points in each frame of the perceived image; Analyze the key feature points of each two adjacent frames of perceived images to determine the relative motion information between each two adjacent frames of perceived images; fusing the relative motion information and the vehicle motion data to obtain the position and posture information of the vehicle; Analyzing the perceived image in the video stream data and the light detection data to obtain ambient light information; The environment adaptation information is generated based on the posture information and the environment lighting information.

4. The method according to claim 3, characterized in that The fusing the relative motion information and the vehicle motion data to obtain the position and posture information of the vehicle includes: Acquire IMU data and wheel speed data from the vehicle motion data; Calculating the motion change of the vehicle based on the acceleration and angular velocity included in the IMU data, and calculating the corresponding displacement change of the vehicle based on the wheel speed data; Predicting preliminary position information of the vehicle using the motion change and the displacement change; The preliminary position and posture information is updated based on the relative motion information to obtain the position and posture information.

5. The method according to claim 3, characterized in that: The analyzing the perceived image in the video stream data and the light detection data to obtain the ambient light information includes: Analyzing the image features of the perceived image according to a preset dimension to obtain a lighting feature parameter corresponding to the preset dimension, wherein the preset dimension includes one or more of color, brightness, and shadow; The lighting detection data and the lighting characteristic parameters corresponding to the preset dimension are integrated to obtain the ambient lighting information.

6. The method according to claim 1, characterized in that The analyzing the satellite navigation data to obtain navigation information corresponding to the vehicle includes: Acquiring vehicle position data and surrounding environment data of the vehicle based on the satellite navigation data; Obtaining the current navigation requirement of the vehicle; The vehicle position data and the surrounding environment data are processed based on a processing strategy corresponding to the navigation requirement to obtain the navigation information.

7. The method according to claim 1, characterized in that The step of fusing the environmental elements, the environmental adaptation information and the navigation information based on the fusion strategy corresponding to the head-up display service to obtain virtual image display data includes: Acquire posture information and ambient lighting information from the environmental adaptation information; Determine a display orientation corresponding to the environmental element according to the position information, and construct initial display data based on the environmental element and the display orientation; Merging the navigation information with the initial display data to obtain target display data; The target display data is rendered using the ambient lighting information to obtain the virtual image display data.

8. The method according to claim 7, characterized in that The determining the display orientation corresponding to the environmental element according to the posture information includes: Converting the position information of the vehicle into a world coordinate system to obtain first coordinate information and angle information of the vehicle in the world coordinate system; Convert the environmental element to the world coordinate system to obtain second coordinate information of the environmental element in the world coordinate system; determining a display position of the environmental element according to a positional relationship between the first coordinate information and the second coordinate information, and determining a display direction of the environmental element according to the angle information; A display orientation corresponding to the environmental element is constructed based on the display position and the display direction.

9. The method according to claim 7, characterized in that: The step of fusing the navigation information with the initial display data to obtain target display data includes: Acquire the guiding element carried in the navigation information; Acquire a target environment element matching the guiding element from the initial display data; Obtaining a fusion strategy corresponding to the guiding element; The guiding element is fused with the target environment element in the initial display data according to the fusion strategy to obtain the target display data.

10. A device for generating vehicle virtual image display data, characterized in that: The device comprises: An acquisition module, used for acquiring video stream data of a current environment in which the vehicle is located when a head-up display service of the vehicle is started, and detecting vehicle motion data and satellite navigation data of the vehicle in the current environment; A first analysis module, configured to analyze the video stream data to obtain environmental elements of the environment in which the vehicle is located, and to calculate environmental adaptation information of the vehicle based on the environmental perception data and the vehicle motion data; A second analysis module, used for analyzing the satellite navigation data to obtain navigation information corresponding to the vehicle; A fusion module is used to fuse the environmental elements, the environmental adaptation information and the navigation information based on a fusion strategy corresponding to the head-up display service to obtain virtual image display data, and display the virtual image display data in the head-up display area of ​​the vehicle.

11. A system for generating vehicle virtual image display data, characterized in that: The system comprises: a detection component, a data processing component, a rendering and display component and a computing power component, wherein the data processing component comprises: an environment perception module, an algorithm module and a map module, the rendering and display component comprises a rendering module and a HUD display module, and the computing power component comprises a plurality of computing power units, wherein the computing power units associated with the environment perception module, the algorithm module, the map module and the rendering module are different; The detection component is used to detect environmental perception data of the current environment of the vehicle, vehicle motion data and satellite navigation data, and transmit the environmental perception data, the vehicle motion data and the satellite navigation data to the data processing component; The environment perception module in the data processing component is used to analyze the environment perception data to obtain the environmental elements of the environment where the vehicle is located, and transmit the environmental elements to the rendering and display module; The algorithm module in the data processing component is used to calculate the environment adaptation information of the vehicle based on the environment perception data and the vehicle motion data, and transmit the environment adaptation information to the rendering and display module; The map module in the data processing component is used to analyze the satellite navigation data to obtain navigation information corresponding to the vehicle, and transmit the navigation information to the rendering and display module; The rendering module in the rendering display module is used to fuse the environmental elements, the environmental adaptation information and the navigation information based on the fusion strategy corresponding to the head-up display service to obtain virtual image display data, and transmit the virtual image display data to the HUD display module; The HUD display module is used to display the virtual image display data in the head-up display area of ​​the vehicle.

12. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for generating vehicle virtual image display data according to any one of claims 1 to 9 by executing the computer instructions.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method for generating vehicle virtual image display data according to any one of claims 1 to 9.

14. A vehicle, characterized in that: The vehicle comprises: a system for generating vehicle virtual image display data and a controller; wherein the controller stores instructions for controlling the operation of the system for generating vehicle virtual image display data, and the system for generating vehicle virtual image display data is used to execute the method for generating vehicle virtual image display data according to any one of claims 1 to 9 according to the instructions of the controller.

Images