Vehicle driving environment information display method, controller, medium, product and vehicle

Abstract

The invention relates to a vehicle driving environment information display method, a controller, a medium, a product and a vehicle, and the method comprises the steps: displaying projection information based on the driving environment information under the condition that the driving environment of the vehicle meets a preset environment condition; the projection information comprises mapping of an entity object in the driving environment in a virtual space. According to the method, effective auxiliary information can be provided for the user in the specific driving environment, the user can perceive the entity objects in the surrounding driving environment, and the driving safety is effectively improved.

Description

Technical Field

[0001] This application relates to the field of vehicle display technology, and in particular to a method, controller, medium, product, and vehicle for displaying vehicle driving environment information. Background Technology

[0002] Advanced driver assistance systems (ADAS) are a set of systems designed to improve driving safety. These systems typically utilize various sensors, cameras, and radar, as well as advanced software algorithms, to enhance the driver's perception of the surrounding environment.

[0003] However, existing driver assistance systems are limited in their effectiveness in specific environments, failing to provide users with effective assistance information and thus impacting driving safety. Summary of the Invention

[0004] This application provides a method, controller, medium, product, and vehicle for displaying vehicle driving environment information, aiming to provide effective auxiliary information to improve driving safety in specific environments.

[0005] This application provides a method for displaying vehicle driving environment information, including:

[0006] When the vehicle's driving environment meets preset environmental conditions, projection information is displayed based on the driving environment information; the projection information includes the mapping of entity objects in the driving environment in virtual space.

[0007] As a feasible embodiment of this application, the method further includes:

[0008] Based on the data collected by the vehicle's sensors, it is determined whether the vehicle's driving environment meets the preset environmental conditions.

[0009] As a feasible embodiment of this application, determining whether the vehicle's driving environment meets preset environmental conditions based on data collected by the vehicle's sensors includes:

[0010] Based on a Bayesian network and data collected by multiple sensors of the vehicle, the probability of the vehicle being in a preset environmental condition is determined.

[0011] Based on the probability that the vehicle is in a preset environmental condition, it is determined whether the vehicle's driving environment meets the preset environmental condition.

[0012] As a feasible embodiment of this application, determining whether the driving environment of the vehicle meets the preset environmental conditions based on the probability that the vehicle is in preset environmental conditions includes:

[0013] If the probability that the vehicle is in a preset environmental condition is greater than a preset probability threshold, then the vehicle's driving environment is determined to meet the preset environmental condition.

[0014] As a feasible embodiment of this application, determining whether the driving environment of the vehicle meets the preset environmental conditions based on the probability that the vehicle is in preset environmental conditions includes:

[0015] If the probability that the vehicle is in a preset environmental condition is less than or equal to a preset probability threshold, it is determined that the vehicle's driving environment does not meet the preset environmental conditions.

[0016] As a possible embodiment of this application, the vehicle's sensors include at least two of the following: a light sensor, a temperature sensor, and a humidity sensor.

[0017] As a feasible embodiment of this application, the preset environmental condition is a low visibility environmental condition.

[0018] As a feasible embodiment of this application, the display of projection information based on the driving environment information includes:

[0019] The driving environment information is processed to obtain entity information in the driving environment;

[0020] The projection information is displayed based on the entity information.

[0021] As a possible embodiment of this application, the driving environment information includes three-dimensional point cloud data collected by radar;

[0022] The process of processing the driving environment information to obtain entity information in the driving environment includes:

[0023] The three-dimensional point cloud data is converted into a two-dimensional plane data representation to obtain two-dimensional data.

[0024] The two-dimensional data is processed to obtain entity information in the driving environment.

[0025] As a feasible embodiment of this application, the step of converting the three-dimensional point cloud data into a data representation on a two-dimensional plane to obtain two-dimensional data includes:

[0026] The three-dimensional point cloud data is projected onto a preset plane using a bird's-eye view algorithm to obtain two-dimensional data.

[0027] As a feasible embodiment of this application, the step of processing the two-dimensional data to obtain entity information in the driving environment includes:

[0028] The two-dimensional data is input into a trained neural network model for processing to obtain entity information in the driving environment.

[0029] As one possible embodiment of this application, the two-dimensional data is two-dimensional image data.

[0030] As a feasible embodiment of this application, the entity information includes at least one of entity type, entity location, entity speed, and entity motion trajectory.

[0031] As one possible embodiment of this application, the entity type includes at least one of vehicles, pedestrians, and road signs.

[0032] As a feasible embodiment of this application, displaying the projection information based on the entity information includes:

[0033] The projection information corresponding to the entity information is displayed by projecting it using an augmented reality device.

[0034] As a feasible embodiment of this application, the step of projecting and displaying the projection information corresponding to the entity information through an augmented reality device includes:

[0035] The entity position in the entity information is transformed to a preset world coordinate system by a preset coordinate transformation matrix to obtain the entity position in the preset world coordinate system.

[0036] Based on the preset user view matrix and projection matrix, the entity position under the preset world coordinates is transformed to obtain the projected position of the entity information on the windshield.

[0037] The augmented reality device projects the projection information corresponding to the entity information onto the projection position.

[0038] As one possible embodiment of this application, the user view matrix is ​​determined based on the driver's head pose.

[0039] As a feasible embodiment of this application, the step of projecting the projection information corresponding to the entity information onto the projection position using an augmented reality device includes:

[0040] Based on the entity information, projection information containing the virtual identifier corresponding to the entity type is generated;

[0041] The projection information is projected onto the projection location using an augmented reality device.

[0042] Furthermore, this application also provides a controller, including one or more processors and a memory, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the vehicle driving environment information display method provided in this application.

[0043] Furthermore, this application embodiment also provides a storage medium storing a computer program. When the computer program is run on a controller, the computer program is used to cause the controller to execute any of the vehicle driving environment information display methods provided in this application embodiment.

[0044] Furthermore, this application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement any of the vehicle driving environment information display methods provided in this application.

[0045] Furthermore, this application also provides a vehicle that includes the controller described above.

[0046] In this embodiment of the application, when the vehicle's driving environment meets the preset environmental conditions, projection information is displayed based on the vehicle's driving environment information. The projection information includes the mapping of entity objects in the driving environment in the virtual space, thereby providing users with effective auxiliary information in a specific driving environment, enabling users to perceive entity objects in the surrounding driving environment, and effectively improving driving safety. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart illustrating the steps of a method for displaying vehicle driving environment information provided in an embodiment of this application;

[0049] Figure 2 This is a schematic diagram of the structure of a lidar provided in an embodiment of this application;

[0050] Figure 3 This application provides a flowchart illustrating the steps for obtaining entity information based on point cloud data processing in an embodiment of the present application.

[0051] Figure 4 This is a schematic diagram of the structure of an augmented reality head-up display unit provided in an embodiment of this application;

[0052] Figure 5 A schematic flowchart illustrating steps for processing entity information for display via an augmented reality device, as provided in an embodiment of this application.

[0053] Figure 6a This is a schematic diagram of the complete process of displaying a vehicle driving environment provided in an embodiment of this application;

[0054] Figure 6b This is a schematic diagram of a complete data processing flow provided in an embodiment of this application;

[0055] Figure 7 This is a schematic diagram of the structure of a vehicle driving environment information display device provided in an embodiment of this application;

[0056] Figure 8 This is a schematic diagram of the structure of a controller provided in an embodiment of this application;

[0057] Figure 9 This is a schematic diagram of the structure of a vehicle provided in an embodiment of this application. Detailed Implementation

[0058] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] Furthermore, in the embodiments of this application, "multiple" refers to two or more. The terms "first" and "second," etc., in the embodiments of this application are used for distinguishing descriptions and should not be construed as implying relative importance.

[0060] This application provides a method for displaying vehicle driving environment information, a controller, a medium, a product, and a vehicle. The method for displaying vehicle driving environment information can run in a controller, which can be a server or a terminal, or other similar device.

[0061] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms.

[0062] The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and the server can be connected directly or indirectly through wired or wireless communication, which is not limited herein.

[0063] To clearly understand the method for displaying vehicle driving environment information provided in this application, the relevant implementation background of the method is first explained. Specifically, a vehicle-assisted driving system refers to an advanced driver assistance system (ADAS) designed to improve driving safety. These systems typically utilize various sensors, cameras, and radars, as well as advanced software algorithms, to enhance the driver's perception of the surrounding environment. However, existing driver assistance systems have limited effectiveness in specific environments, such as in low-visibility conditions like severe weather, and cannot provide effective assistance information, thus affecting driving safety.

[0064] To address the aforementioned issues, this application provides a method, controller, medium, product, and vehicle for displaying vehicle driving environment information. By displaying projection information based on driving environment information when the vehicle's driving environment meets preset environmental conditions, the projection information includes the mapping of entity objects in the driving environment to virtual space. This effectively provides users with perception information about their surroundings and improves driving safety.

[0065] For details, please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a method for displaying vehicle driving environment information according to an embodiment of this application. The display method may include the following step S110:

[0066] S110, when the vehicle's driving environment meets preset environmental conditions, display projection information based on the driving environment information; the projection information includes the mapping of entity objects in the driving environment in virtual space.

[0067] In this embodiment, the vehicle's driving environment meeting preset environmental conditions generally refers to environmental conditions where the vehicle's driver assistance system cannot provide effective assistance information to the user. This inability to provide effective assistance information may mean that the driver assistance system cannot provide perception information about the surrounding environment, or that the perception information provided by the driver assistance system is unclear and potentially erroneous. Therefore, as a feasible solution, the vehicle's driving environment can be determined based on the perception information provided by the driver assistance system to determine whether the vehicle's driving environment meets the preset environmental conditions.

[0068] Furthermore, considering that driver assistance systems typically use sensors and cameras to visually perceive information about the surrounding environment, and that the data collected by the sensors can reflect the vehicle's driving environment to a certain extent, such as determining whether the vehicle is driving in specific adverse weather conditions, as another feasible implementation of this application, it is also possible to determine whether the vehicle's driving environment meets preset environmental conditions by using data collected by the vehicle's existing sensors, thereby avoiding the introduction of additional costs.

[0069] In other words, in one embodiment, the method for displaying the vehicle driving environment information further includes:

[0070] Based on the data collected by the vehicle's sensors, it is determined whether the vehicle's driving environment meets the preset environmental conditions.

[0071] The vehicle's sensors typically include multiple sensors used to collect environmental information from multiple dimensions to detect the vehicle's driving environment. For example, in a feasible implementation, the vehicle's sensors typically include two of the following: a light sensor, a temperature sensor, and a humidity sensor. Of course, based on the vehicle's actual configuration information, the vehicle's sensors may also include other sensors besides those mentioned above, such as a barometric pressure sensor, a wind speed sensor, etc. This application does not impose any limitations on these embodiments.

[0072] Of course, considering that each sensor reflects different environmental information, and that different environmental information has different impacts on environmental visibility, in order to accurately determine whether the vehicle's driving environment meets preset conditions based on the data collected by the vehicle's sensors, a feasible implementation of this application can also be considered: using a mathematical model to automatically and accurately predict whether the vehicle's driving environment meets preset environmental conditions based on the data collected by the vehicle's sensors. For example, in a feasible embodiment, the probability of the vehicle being in a driving environment with preset environmental conditions can be predicted based on a Bayesian network, thereby determining whether the vehicle's driving environment meets the preset environmental conditions. Specifically, that is, determining whether the vehicle's driving environment meets preset environmental conditions based on the data collected by the vehicle's sensors includes:

[0073] Based on a Bayesian network and data collected by multiple sensors of the vehicle, the probability of the vehicle being in a preset environmental condition is determined.

[0074] Based on the probability that the vehicle is in a preset environmental condition, it is determined whether the vehicle's driving environment meets the preset environmental condition.

[0075] Specifically, to facilitate understanding of the above, the probability model of a vehicle driving in a low-visibility environment can be expressed as:

[0076]

[0077] Wherein, P[LowVis|S1, S2, ..., Sn) is the probability that the current environment in which the vehicle is located is under preset environmental conditions, given sensor readings S1, S2, ..., Sn. P[S1, S2, ..., Sn|LowVis) is the probability of a specific sensor reading occurring under preset environmental conditions, P[LowVis) is the prior probability of the preset environmental conditions occurring under normal circumstances, and P[s1, s2, ..., Sn) is the probability of these specific sensor readings occurring. P[S1, S2, ..., Sn|LowVis), P[LowVis), and P[S1, S2, ..., Sn) can be calculated in advance using applied statistics, which will not be elaborated further in this embodiment.

[0078] The probability of a vehicle being in a preset environmental condition, obtained through the aforementioned method, can be used to determine whether the vehicle's driving environment meets the preset environmental conditions. For example, as a common feasible implementation, when the probability of the vehicle being in the preset environmental conditions is high, the driver assistance system is more likely to fail to provide effective environmental perception information. Conversely, when the probability of the vehicle being in the preset environmental conditions is low, the driver assistance system can often provide accurate and effective environmental perception information to the user, such as the driver. Therefore, as a feasible embodiment of this application, the determination of whether the vehicle's driving environment meets the preset environmental conditions can be based on the probability of the vehicle being in the preset environmental conditions and a preset probability threshold. That is, determining whether the vehicle's driving environment meets the preset environmental conditions based on the probability of the vehicle being in the preset environmental conditions includes:

[0079] If the probability that the vehicle is in a preset environmental condition is greater than a preset probability threshold, then the vehicle's driving environment is determined to meet the preset environmental conditions; and / or

[0080] If the probability that the vehicle is in a preset environmental condition is less than or equal to a preset probability threshold, it is determined that the vehicle's driving environment does not meet the preset environmental conditions.

[0081] Of course, as a feasible implementation, the preset environmental conditions here can specifically be low visibility environments, that is, driving in extreme weather conditions such as nighttime, fog, heavy rain, or blizzards. Of course, choosing other preset environmental conditions that affect the driver assistance system is also feasible. This application does not impose any restrictions on this.

[0082] In the case where the driving environment of the vehicle meets preset environmental conditions, i.e., the vehicle's driver assistance system cannot provide effective environmental perception information to the user, the solution provided in this application displays projection information based on the collected driving environment information. This projection information includes the mapping of entity objects in the driving environment to virtual space. Specifically, in order to display the mapping of entity objects in the driving environment to virtual space based on the collected driving environment information, thereby displaying projection information, in one embodiment, it is typically necessary to process the driving environment information to obtain entity information in the driving environment, and then project and display the corresponding projection information based on the obtained entity information. That is, displaying projection information based on the driving environment information includes:

[0083] The driving environment information is processed to obtain entity information in the driving environment;

[0084] The projection information is displayed based on the entity information.

[0085] Specifically, it can be understood that in order to process the collected driving environment information to obtain the entity information in the driving environment information, the driving environment information here is usually information collected independently of environmental conditions.

[0086] For ease of understanding, in one embodiment, taking the aforementioned preset environmental conditions as a low-visibility environment as an example, the collected driving environment information is typically visual environment information acquired without relying on cameras or other image acquisition devices. For example, in a feasible implementation, the driving environment information includes three-dimensional point cloud data acquired by radar. Here, the radar can be a lidar system. For details, please refer to... Figure 2 , Figure 2 The following is a detailed structural diagram of a lidar provided in an embodiment of this application.

[0087] In this embodiment, the lidar includes units such as a scanning mirror, a rotary motor, a laser, and a detector. The rotary motor drives the scanning mirror to rotate, causing the pulsed laser light generated by the laser to be emitted optically and receiving laser signals reflected back from objects. The captured laser signals contain important information such as the position, shape, and distance of the objects and are converted into electronic signals. These signals are then converted into three-dimensional point cloud data through signal processing algorithms to represent the surrounding environment. By processing the three-dimensional point cloud data containing important information such as the position, shape, and distance of objects, the entity information in the driving environment can be effectively reconstructed.

[0088] Furthermore, to ensure the real-time effectiveness of the displayed projection information, it is often necessary to quickly identify entity information in the driving environment. Considering that the point cloud data collected by LiDAR is typically 3D point cloud data, involving a large amount of data computation, a feasible implementation scheme in this application is to quickly and accurately identify entity information in the driving environment to display the corresponding projection information. This can be achieved by converting the 3D point cloud data into a 2D plane representation, thereby quickly identifying entity information in the driving environment. For details, please refer to... Figure 3 , Figure 3 This application provides a flowchart illustrating steps for obtaining entity information based on point cloud data processing, specifically including steps S310 to S320:

[0089] S310, the three-dimensional point cloud data is converted into a data representation on a two-dimensional plane to obtain two-dimensional data.

[0090] In this embodiment, by processing the 3D point cloud data, such as projecting it onto a preset plane, the 3D point cloud data can be converted into a 2D plane data representation, thus obtaining 2D data. Of course, in order to intuitively display entity information in the driving environment, such as entity location and shape, on a 2D plane, the obtained 2D data can typically be 2D image data. For example, as a feasible implementation, the 3D point cloud data can be processed using a Bird's Eye View (BEV) algorithm. Specifically, converting the 3D point cloud data into a 2D plane data representation to obtain 2D data includes:

[0091] The three-dimensional point cloud data is projected onto a preset plane using a bird's-eye view algorithm to obtain two-dimensional data.

[0092] In this embodiment, by using a bird's-eye view algorithm, information in three-dimensional space can be converted into a two-dimensional representation. This conversion allows the position and shape information of objects to be intuitively displayed on a two-dimensional plane, facilitating analysis and processing. The conversion process can be represented by the following formula:

[0093] BEV(x,y)=∫ z Lidar(x,y,z)dz

[0094] Here, Lidar(x,y,z) represents the original point cloud data at coordinate point (x,y,z).

[0095] S320, perform data processing on the two-dimensional data to obtain entity information in the driving environment.

[0096] After the aforementioned conversion of 3D point cloud data, further data processing of the converted 2D data can effectively and quickly obtain entity information in the driving environment while reducing the amount of data processing required. This data processing typically includes data clustering, data analysis, and can also incorporate machine learning principles. For example, 2D data can be input into a trained neural network model to directly obtain entity information in the driving environment.

[0097] Of course, when the two-dimensional data is two-dimensional image data, the neural network model used can usually be a neural network model that involves image recognition processing, such as a convolutional neural network (CNN) or other neural networks to process the two-dimensional image data. This application does not limit the neural network model used. Any neural network that can effectively process images and obtain entity information in the image after training is within the scope of protection claimed in this application.

[0098] Furthermore, in order to ensure the accuracy of the mapping of entity objects in the driving environment in the virtual space in the projection information, the aforementioned entity information may include, in addition to the specific entity object, at least one of the entity type, entity position, entity speed, and entity motion trajectory of the entity object. For example, the instance type of the entity object usually includes at least one of vehicles, pedestrians, road signs, and other environmental obstacles.

[0099] After identifying the entity information of the driving environment around the vehicle through the aforementioned solution, in order to enable users to understand the entity information of the driving environment around the vehicle more intuitively, the technical solution provided in this application will further display the projection information based on the entity information, wherein the projection information includes the mapping of entity objects in the driving environment in virtual space.

[0100] Specifically, in one embodiment, displaying the projection information based on the entity information can be achieved using an augmented reality device. That is, displaying the projection information based on the entity information includes:

[0101] The projection information corresponding to the entity information is displayed by projecting it using an augmented reality device.

[0102] Specifically, in one feasible implementation, the augmented reality device can employ an augmented reality head-up display (AR-HUD). For further details, please refer to [link to relevant documentation]. Figure 4 , Figure 4A schematic diagram of an augmented reality head-up display unit provided in an embodiment of this application is described in detail below.

[0103] In this embodiment, the augmented reality head-up display unit includes a correction mirror, a concave mirror, and an image generation unit. It converts processed data into a visual image and projects it onto the windshield in the driver's field of vision. Specifically, the image generation unit is mainly used to receive and analyze data from the aforementioned sources, such as 3D point cloud data provided by LiDAR or further calculated entity information, and convert it into projected visual information.

[0104] Of course, it should be noted that, in order to ensure the consistency between the information displayed by the augmented reality head-up display unit and the driver's actual field of vision, the solution provided in this application embodiment also provides a solution for processing the processed data to project and display the corresponding virtual image through the augmented reality display device.

[0105] For details, please refer to Figure 5 , Figure 5 This application provides a flowchart illustrating steps for processing entity information for display via an augmented reality display device, specifically including steps S510 to S530:

[0106] S510, the entity position in the entity information is transformed to a preset world coordinate system using a preset coordinate transformation matrix to obtain the entity position in the preset world coordinate system.

[0107] In this embodiment of the application, considering that the entity information identified based on point cloud data usually refers to the entity information in the local coordinate system where the lidar is located, in order to ensure that the augmented reality display information corresponds accurately with the actual environment, it is often necessary to use a preset coordinate transformation matrix to transform the entity position in the entity information to a preset world coordinate system, thereby obtaining the entity position in the preset world coordinate system.

[0108] Specifically, this transformation includes translation and rotation steps. That is, the preset coordinate transformation matrix typically includes a translation vector and a rotation matrix. The specific transformation formula is as follows:

[0109] P world =R·(P local +T)

[0110] Among them, P local Let T be the position of the entity in the local coordinate system where the LiDAR is located, T be the preset translation vector, R be the rotation matrix, and P be the position of the entity in the local coordinate system where the LiDAR is located. world This represents the position of the entity in world coordinates after the transformation.

[0111] S520, transform the entity position under the preset world coordinates based on the preset user view matrix and projection matrix to obtain the projection position of the entity information on the windshield.

[0112] Building upon the steps described above, in order to ensure that the content displayed on the augmented reality device is consistent with the driver's actual line of sight and to provide accurate navigation and warning signals, it is necessary to calculate the intersection of the ray from the driver's head posture to the location of the identified object and the vehicle's windshield. This involves transforming the entity's position using a user view matrix, which is often associated with the user's head posture.

[0113] Furthermore, in order to further calculate the correct display position of the object on the augmented reality display device, it is usually necessary to convert the object's three-dimensional spatial coordinates into two-dimensional screen (windshield) coordinates in this embodiment of the application. Therefore, taking head tracking information into account, the following formula can be used for calculation:

[0114] P screen =M projection ·M view ·P world

[0115] Among them, P world M represents the position of the transformed entity in world coordinates. view It is a user view matrix adjusted according to the driver's head posture, and M projection It is a projection matrix that converts three-dimensional coordinates into a two-dimensional projection on the windshield. Therefore, the obtained P screen This refers to the projection position of the entity information onto the windshield, which is the two-dimensional coordinate ultimately displayed on the AR-HUD. The above calculation process ensures that the information displayed on the AR-HUD is consistent with the driver's field of vision.

[0116] S530, the projection information corresponding to the entity information is projected onto the projection position using an augmented reality device.

[0117] In this embodiment of the application, after determining the projection position of the entity information on the windshield, the corresponding projection information of the entity information is further projected onto the projection position using an augmented reality device, such as AR-HUD. This allows the corresponding entity object to be displayed at the corresponding position on the windshield, so that the driver can more intuitively determine the entity objects in the vehicle's surrounding environment.

[0118] Furthermore, as another feasible implementation of this application, the augmented reality device can also use virtual images and symbols to represent the entity type of the identified entities. These images and symbols are designed to be both intuitive and non-disruptive to the driver's normal field of vision, such as using different colors and icons to represent different types of objects, while simultaneously displaying distance and speed information. Specifically, the principles of computer graphics, particularly shader programming and ray tracing technology, are considered to achieve more realistic virtual images. For example, ray tracing algorithms can be used to simulate the interaction between light and virtual objects; the formula can be expressed as:

[0119] C=∫ Ω L(p,ω)·fr(p,ω i ,ω)·cos(θ)·dω

[0120] Where C is the color observed at the observation point, L(p,ω) is the incident light from direction ω to point p, fr is the reflectivity function, and Ω is all possible incident directions on the hemisphere.

[0121] In other words, projecting the projection information corresponding to the entity information onto the projection position using an augmented reality display device includes:

[0122] Based on the entity type in the entity information, projection information containing the virtual identifier corresponding to the entity type is generated;

[0123] The projection information is projected onto the projection location using an augmented reality display device.

[0124] The augmented reality device projects the aforementioned information onto the windshield, blending it seamlessly with the driver's actual view of the environment. This augmented reality technology not only enhances the driver's perception of their surroundings but also helps improve their reaction time to potential hazards.

[0125] In this embodiment of the application, when the vehicle's driving environment meets the preset environmental conditions, projection information is displayed based on the vehicle's driving environment information. The projection information includes the mapping of entity objects in the driving environment in the virtual space, thereby providing users with effective auxiliary information in a specific driving environment, enabling users to perceive entity objects in the surrounding driving environment, and effectively improving driving safety.

[0126] To clearly understand the vehicle driving environment information display method provided in this application embodiment, the following will be combined with the foregoing. Figures 1-5 The provided embodiment is a complete flowchart illustrating a method for displaying vehicle driving environment information. For details, please refer to [link / reference]. Figure 6a It includes the following steps:

[0127] (1) Environmental monitoring to determine if it is a low visibility scene.

[0128] This step is a prerequisite for the entire system to start. The system first uses sensors and algorithms to determine whether the current environment is low-visibility, such as fog, heavy rain, or nighttime. This determination is based on multiple data sources, including environmental sensors inside and outside the vehicle (such as light sensors and humidity sensors) as well as real-time weather information. If a low-visibility environment is confirmed, the system will proceed to the next step; otherwise, to conserve energy and processing power, the system will not start.

[0129] Specifically, in the environmental monitoring process, statistical and machine learning theories can be applied to more accurately determine low visibility conditions. For example, a Bayesian network can be used, combining data from multiple sensors to calculate the probability of the current environmental state. Specific methods can be found in the descriptions of the foregoing embodiments.

[0130] (2) Radar sensing

[0131] Once the environment is deemed suitable, the LiDAR (Light Detection and Ranging) system activates and scans the surroundings. LiDAR detects objects by emitting laser pulses and capturing the reflected signals. This process is effective in low-visibility conditions, unaffected by light or weather, providing the system with high-precision spatial data.

[0132] In the lidar sensing phase, the Fast Fourier Transform (FFT) can also be used to process the radar signal to improve the signal-to-noise ratio. The FFT formula can be expressed as:

[0133]

[0134] Where x[n] is the original time series data, and x[k] is the frequency domain representation after FFT transformation.

[0135] (3) Data processing

[0136] The collected radar data is processed by the perception algorithm module. This module identifies and classifies surrounding objects, such as other vehicles, pedestrians, road signs, etc., and calculates their position, speed, and trajectory. This process involves complex data analysis, including machine learning algorithms, to ensure fast and accurate target identification in dynamic environments. For details, please refer to [link to relevant documentation]. Figure 6b , Figure 6b This illustrates a complete data processing flow, specifically including the following steps:

[0137] ① Environmental monitoring

[0138] At this stage, the system first uses light and humidity sensors to assess the current environmental conditions. The data from these sensors is used to determine whether the system is in a low visibility mode, such as in foggy weather, at night, or other conditions that impair vision.

[0139] ② Determine low visibility

[0140] The system determines whether to activate the low visibility mode based on the data collected in the previous step. If the detected environment meets the low visibility conditions, the system will enter the low visibility processing mode; otherwise, the system will maintain the normal mode.

[0141] ③ Raw data preprocessing

[0142] In low-visibility mode, the raw point cloud data collected by the LiDAR first undergoes preprocessing. This step includes data cleaning and preliminary analysis to prepare for subsequent advanced processing.

[0143] ④BEV conversion

[0144] Next, this point cloud data is processed using the Bird's Eye View (BEV) algorithm to convert the information in three-dimensional space into a two-dimensional representation. The core of the BEV algorithm is to project the three-dimensional point cloud data onto a horizontal plane, facilitating subsequent object detection and tracking. This conversion allows the position and shape information of objects to be intuitively displayed on a two-dimensional plane, simplifying analysis and processing. The specific conversion formulas can be found in the descriptions of the aforementioned embodiments.

[0145] ⑤ Object detection and classification

[0146] The data converted from BEV (Browser-Electronic Vehicle) data enters the machine learning module for object detection and classification. In this stage, the system uses advanced algorithms such as convolutional neural networks (CNNs) to identify and classify various objects, such as vehicles, pedestrians, and road signs, and calculate their positions, speeds, and trajectories.

[0147] (4) Position transformation

[0148] After completing the obstacle identification stage described above and outputting entity information such as object category and location, the system proceeds to the projection rendering stage. In this step, the system transforms the object positions detected by the LiDAR from the local coordinate system to the vehicle's world coordinate system. The specific position transformation formula can be found in the relevant descriptions of the aforementioned embodiments.

[0149] (5) Projection Calculation

[0150] The calculation calculates the intersection of a ray from the driver's head position to the location of the identified object and the vehicle's windshield. This calculation takes into account the driver's field of vision and head position to ensure that the display on the AR-HUD aligns with the driver's actual line of sight, providing accurate navigation and warning information.

[0151] Furthermore, to ensure the correct display position of objects on the AR-HUD, it is necessary to convert the object's three-dimensional spatial coordinates into two-dimensional screen coordinates. That is, the corresponding projection position needs to be determined using the user's viewpoint matrix associated with the user's head pose and the projection matrix that converts the three-dimensional coordinates into a two-dimensional projection onto the windshield.

[0152] (6) AR-HUD rendering

[0153] In AR-HUD, the system uses virtual avatars and symbols to represent identified objects. These avatars and symbols are designed to be both intuitive and non-distracting to the driver's normal field of vision, such as using different colors and icons to represent different types of objects, while simultaneously displaying distance and speed information. We are considering leveraging principles of computer graphics, particularly shader programming and ray tracing techniques, to achieve even more realistic virtual images.

[0154] (7) Virtual Avatar Display

[0155] The system projects the virtual image onto the windshield, seamlessly integrating it with the driver's actual view of the environment. This augmented reality technology not only enhances the driver's perception of their surroundings but also helps improve their reaction time to potential hazards.

[0156] (8) End

[0157] Once an operation cycle is completed, such as when the driver leaves a low-visibility environment or shuts down the vehicle, the system will end the current cycle and return to its initial state. The system remains in standby mode, ready to be restarted whenever needed.

[0158] To better understand the method for displaying vehicle driving environment information provided in this application, please refer to [link / reference]. Figure 7 , Figure 7 This is a schematic diagram of a vehicle driving environment information display device provided in an embodiment of this application. Specifically, the display device may include:

[0159] The display module 710 displays projection information based on the driving environment information when the vehicle's driving environment meets preset environmental conditions; the projection information includes the mapping of entity objects in the driving environment in virtual space.

[0160] In one feasible implementation, the display module 710 is further configured to determine whether the vehicle's driving environment meets preset environmental conditions based on data collected by the vehicle's sensors.

[0161] In one feasible implementation, the display module 710 is further configured to determine the probability that the vehicle is in a low visibility environment based on a Bayesian network and data collected by multiple sensors of the vehicle.

[0162] Based on the probability that the vehicle is in a low-visibility environment, it is determined whether the vehicle's driving environment meets the preset environmental conditions.

[0163] In one feasible implementation, the display module 710 is further configured to determine that the vehicle's driving environment meets preset environmental conditions when the probability that the vehicle is in a low visibility environment is greater than a preset probability threshold.

[0164] In one feasible implementation, the display module 710 is further configured to determine that the vehicle's driving environment does not meet the preset environmental conditions when the probability that the vehicle is in a low visibility environment is less than or equal to a preset probability threshold.

[0165] In one feasible implementation, the display module 710 is further configured to process the driving environment information to obtain entity information in the driving environment;

[0166] The projection information is displayed based on the entity information.

[0167] In one feasible implementation, the display module 710 is further configured to convert the three-dimensional point cloud data into a data representation on a two-dimensional plane to obtain two-dimensional data;

[0168] The two-dimensional data is processed to obtain entity information in the driving environment.

[0169] In one feasible implementation, the display module 710 is further used to project the three-dimensional point cloud data onto a preset plane using a bird's-eye view algorithm to obtain two-dimensional data.

[0170] In one feasible implementation, the display module 710 is further configured to input the two-dimensional data into a trained neural network model for processing to obtain entity information in the driving environment.

[0171] In one feasible implementation, the display module 710 is further configured to project and display the projection information corresponding to the entity information via an augmented reality device.

[0172] In one feasible implementation, the display module 710 is further configured to transform the entity position in the entity information to a preset world coordinate system using a preset coordinate transformation matrix, thereby obtaining the entity position in the preset world coordinate system.

[0173] Based on the preset user view matrix and projection matrix, the entity position under the preset world coordinates is transformed to obtain the projected position of the entity information on the windshield.

[0174] The augmented reality device projects the projection information corresponding to the entity information onto the projection position.

[0175] In one feasible implementation, the display module 710 is further configured to generate projection information containing a virtual identifier corresponding to the entity type based on the entity information;

[0176] The projection information is projected onto the projection location using an augmented reality device.

[0177] Therefore, when the vehicle's driving environment meets the preset environmental conditions, projection information is displayed based on the vehicle's driving environment information. The projection information includes the mapping of physical objects in the driving environment in the virtual space. This can provide users with effective auxiliary information in specific driving environments, enabling users to perceive physical objects in the surrounding driving environment and effectively improving driving safety.

[0178] In practice, each of the above modules can be implemented as an independent entity or can be combined arbitrarily to be implemented as the same or several entities. For the specific implementation methods and corresponding beneficial effects of each of the above modules, please refer to the previous method embodiments, which will not be repeated here.

[0179] This application also provides a controller, such as... Figure 8 As shown, it illustrates a schematic diagram of the controller involved in an embodiment of this application. Specifically:

[0180] The controller may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more storage media, a power supply 303, and an input unit 304. Those skilled in the art will understand that... Figure 8 The controller structure shown does not constitute a limitation on the controller and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0181] The processor 301 is the control center of the controller, connecting various parts of the controller via various interfaces and lines. It executes various functions and processes data by running or executing computer programs and / or modules stored in the memory 302, and by calling data stored in the memory 302. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 301.

[0182] The memory 302 can be used to store computer programs and modules. The processor 301 executes various functional applications and parking space recognition by running the computer programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, computer programs required for at least one function (such as audio-visual prompts, anti-pinch functions, etc.), etc.; the data storage area may store data created according to the use of the controller, etc. In addition, the memory 302 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0183] The controller also includes a power supply 303 that supplies power to the various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 303 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0184] The controller may also include an input unit 304, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0185] Although not shown, the controller may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 301 in the controller loads the executable files corresponding to the processes of one or more computer programs into the memory 302 according to the following instructions, and the processor 301 runs the computer programs stored in the memory 302 to realize various functions, such as:

[0186] When the vehicle's driving environment meets preset environmental conditions, projection information is displayed based on the driving environment information; the projection information includes the mapping of entity objects in the driving environment in virtual space.

[0187] Therefore, when the vehicle's driving environment meets the preset environmental conditions, projection information is displayed based on the vehicle's driving environment information. The projection information includes the mapping of physical objects in the driving environment in the virtual space. This can provide users with effective auxiliary information in specific driving environments, enabling users to perceive physical objects in the surrounding driving environment and effectively improving driving safety.

[0188] For details on the specific implementation methods and corresponding beneficial effects of the above operations, please refer to the detailed description of the enhanced display method for the vehicle driving environment above, which will not be repeated here.

[0189] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by a computer program, or by a computer program controlling related hardware. The computer program can be stored in a storage medium and loaded and executed by a processor.

[0190] Therefore, embodiments of this application provide a storage medium storing a computer program that can be loaded by a processor to execute the steps in any of the vehicle driving environment information display methods provided in embodiments of this application. For example, the computer program can execute the following steps:

[0191] When the vehicle's driving environment meets preset environmental conditions, projection information is displayed based on the driving environment information; the projection information includes the mapping of entity objects in the driving environment in virtual space.

[0192] Therefore, the storage medium provided in this application embodiment can display projection information based on the vehicle's driving environment information when the vehicle's driving environment meets preset environmental conditions. The projection information includes the mapping of entity objects in the driving environment in the virtual space, thereby providing users with effective auxiliary information in specific driving environments so that users can perceive entity objects in the surrounding driving environment and effectively improve driving safety.

[0193] For details on the specific implementation methods and corresponding beneficial effects of the above operations, please refer to the previous embodiments, which will not be repeated here.

[0194] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0195] Since the computer program stored in the storage medium can execute the steps in any of the vehicle driving environment information display methods provided in the embodiments of this application, the beneficial effects that any of the vehicle driving environment information display methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0196] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a storage medium. A processor of a computer device reads the computer instructions from the storage medium and executes the computer instructions, causing the computer device to perform the aforementioned method for displaying vehicle driving environment information.

[0197] This application also provides a vehicle; please refer to [link / reference]. Figure 9 The vehicle includes a lidar, an onboard computing module, an AR-HUD unit, and other collaborative structures, which enable the display of vehicle driving environment information provided in this application.

[0198] This application does not limit the specific structure of the vehicle. The specific implementation methods and corresponding beneficial effects of the above-described operations of the controller are also applicable to this vehicle. For details, please refer to the detailed description of the enhanced display method for the vehicle's driving environment above, which will not be repeated here.

[0199] The foregoing has provided a detailed description of a method for displaying vehicle driving environment information, a controller, a storage medium, a computer program product, and a vehicle provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A display method of vehicle running environment information, characterized by, The method comprises: In a case where a driving environment of a vehicle satisfies a preset environmental condition, displaying projection information based on the driving environment information, the projection information comprising a mapping of a real object in the driving environment in a virtual space.

2. The method of claim 1, wherein, The method further comprises: Determining whether the driving environment of the vehicle satisfies the preset environmental condition based on data collected by sensors of the vehicle.

3. The method of claim 2, wherein, The determining whether the driving environment of the vehicle satisfies the preset environmental condition based on data collected by sensors of the vehicle comprises: Determining a probability that the vehicle is in the preset environmental condition based on a Bayesian network and data collected by multiple sensors of the vehicle; Determining whether the driving environment of the vehicle satisfies the preset environmental condition based on the probability that the vehicle is in the preset environmental condition.

4. The method of claim 3, wherein, The determining whether the driving environment of the vehicle satisfies the preset environmental condition based on the probability that the vehicle is in the preset environmental condition comprises: In a case where the probability that the vehicle is in the preset environmental condition is greater than a preset probability threshold, determining that the driving environment of the vehicle satisfies the preset environmental condition.

5. The method of claim 3, wherein, The determining whether the driving environment of the vehicle satisfies the preset environmental condition based on the probability that the vehicle is in the preset environmental condition comprises: In a case where the probability that the vehicle is in the preset environmental condition is less than or equal to the preset probability threshold, determining that the driving environment of the vehicle does not satisfy the preset environmental condition.

6. The method of claim 2, wherein, The sensors of the vehicle at least include two of a light sensor, a temperature sensor, and a humidity sensor.

7. The method according to any one of claims 1 to 6, characterized in that, The preset environmental condition is a low-visibility environmental condition.

8. The method of claim 1, wherein, The displaying projection information based on the driving environment information comprises: Processing the driving environment information to obtain entity information in the driving environment; Displaying the projection information based on the entity information.

9. The method of claim 8, wherein, The driving environment information comprises three-dimensional point cloud data collected by a radar; The processing the driving environment information to obtain entity information in the driving environment comprises: Converting the three-dimensional point cloud data into a data representation on a two-dimensional plane to obtain two-dimensional data; Processing the two-dimensional data to obtain the entity information in the driving environment.

10. The method of claim 9, wherein, The converting the three-dimensional point cloud data into a data representation on a two-dimensional plane to obtain two-dimensional data comprises: Projecting the three-dimensional point cloud data into a preset plane by a bird's-eye view algorithm to obtain two-dimensional data.

11. The method of claim 9, wherein, The processing the two-dimensional data to obtain the entity information in the driving environment comprises: Inputting the two-dimensional data into a trained neural network model for processing to obtain the entity information in the driving environment.

12. The method of claim 9, wherein, The two-dimensional data is two-dimensional image data.

13. The method of claim 8, wherein, The entity information comprises at least one of an entity type, an entity position, an entity speed, and an entity motion trajectory.

14. The method of claim 13, wherein, The entity type comprises at least one of a vehicle, a pedestrian, and a road sign.

15. The method of claim 8, wherein, The displaying the projection information based on the entity information comprises: Projecting and displaying the projection information corresponding to the entity information by an augmented reality device.

16. The method of claim 15, wherein, The projecting and displaying the projection information corresponding to the entity information by the augmented reality device comprises: transforming an entity position in the entity information to a preset world coordinate through a preset coordinate transformation matrix to obtain an entity position in the preset world coordinate; transforming the entity position in the preset world coordinate based on a preset user view matrix and a projection matrix to obtain a projection position of the entity information on the windshield; projecting, through an augmented reality device, projection information corresponding to the entity information to the projection position.

17. The method of claim 16, wherein, The user view matrix is determined based on a head pose of the driver.

18. The method of claim 17, wherein, The projecting, through the augmented reality device, the projection information corresponding to the entity information to the projection position comprises: generating, based on the entity information, projection information containing a virtual identifier corresponding to the entity type; projecting, through the augmented reality device, the projection information to the projection position.

19. A controller characterized by comprising: A computer program product comprising one or more processors and a memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method of any one of claims 1 to 18.

20. A storage medium, characterized by A computer program product comprising a computer program that, when executed on a controller, causes the controller to perform the steps of the method of any one of claims 1 to 18.

21. A computer program product, characterised in that, A computer program product comprising a computer program or instructions that, when executed by a processor, implement the steps of the method of any one of claims 1 to 18.

22. A vehicle characterized by The vehicle comprises the controller of claim 19.

Images