Method, apparatus and device for visualizing a physical environment

Abstract

The present disclosure relates to a physical environment visualization method, device and equipment, and the method belongs to the technical field of vehicles. The method comprises: receiving data information of a height map representing physical environment occupancy information and data information of a discrete grid; performing layering processing along the height direction, obtaining a plurality of height layer slice maps for expressing sampling information of each height layer based on the data information of the height map; performing surface reconstruction on the discrete grid based on the data information of the discrete grid to obtain a reconstructed grid; obtaining a reconstructed height map for expressing geometric relief information of the physical environment based on the reconstructed grid; performing contour fusion on the plurality of height layer slice maps and the reconstructed height map to obtain a completed contour information height map; and rendering a visualization display result based on the contour information height map.

Description

Technical Field

[0001] This disclosure relates to the field of vehicle technology, and in particular to a method, apparatus, and device for visualizing a physical environment. Background Technology

[0002] Perceptual visualization is a technology that transforms the perceptions of the surrounding environment by intelligent driving systems into visual graphics that humans can intuitively understand. However, the rendering output of related technologies often suffers from excessive layering and noticeable pixelation. Therefore, achieving a smooth and unified rendering output is a pressing issue that needs to be addressed. Summary of the Invention

[0003] This disclosure provides a method, apparatus, and device for visualizing a physical environment, the technical solution of which includes at least: According to one aspect of the present disclosure, a method for visualizing a physical environment is provided, the method comprising: Receive data information from height maps representing physical environment occupancy information and data information from discrete grids; Layering is performed along the height direction; based on the data information of the height map, multiple height layer slices are obtained to represent the sampling information of each height layer; Based on the data information of the discrete mesh, a surface reconstruction is performed on the discrete mesh to obtain the reconstructed mesh; based on the reconstructed mesh, a reconstructed height map is obtained to represent the geometric undulation information of the physical environment; Contour fusion is performed on multiple height layer slices and reconstructed height maps to obtain a complete contour information height map; The visualization results are rendered based on the contour information height map.

[0004] In some optional designs, contour fusion is performed on multiple height layer slices and reconstructed height maps to obtain a complete contour information height map, including: For each pixel location in multiple height layer slices and the reconstructed height map, if valid contour information exists at the pixel location in any input image, the valid contour information is retained; based on the processing results of each pixel location, a contour information height map with continuous and complete contours is determined.

[0005] In some optional designs, a visualization is rendered based on a contour information heightmap, including: The contour information height map is rendered as a texture onto the reconstructed mesh; Rendering is performed based on the reconstructed mesh, generating image data corresponding to the contour information height map, which is then used as the visualization result.

[0006] In some optional designs, layering is performed along the height direction, including: Preset the height range for layering along the height direction; The height range is divided into various sampling heights according to a fixed or adaptive step size; Image sampling is performed on the height map at each height layer corresponding to each sampling height.

[0007] In some optional designs, based on the heightmap data, multiple height-level slice maps are obtained to represent the sampling information of each height level, including: Based on the data information of the height map, determine whether the physical environment is occupied by one or more objects at each horizontal sampling position of each height layer, and determine the sampling information of each height layer; The sampling information of each height layer is mapped onto each two-dimensional pixel image. The two-dimensional pixel image is a collection of pixels arranged in a two-dimensional grid. The coordinates of each pixel on each two-dimensional pixel image correspond to the coordinates of each horizontal sampling position of each height layer. Construct two-dimensional planar grids corresponding to each height layer. The horizontal range of each two-dimensional planar grid is consistent with the horizontal sampling range of each height layer, and the height of each two-dimensional planar grid is the same as the sampling height corresponding to each height layer. Each two-dimensional pixel image is attached to the surface of each two-dimensional planar grid to generate multiple textured planar grids that carry the sampling information of each height layer. Render multiple textured planar meshes to obtain multiple height layer slices that represent the sampling information of each height layer.

[0008] In some optional designs, the discrete mesh is reconstructed based on the data information of the discrete mesh to obtain the reconstructed mesh, including: determining the data information of multiple three-dimensional coordinate points and multiple facets for generating the discrete mesh based on the data information of the discrete mesh; A preset algorithm is used to determine the connection relationship between multiple three-dimensional coordinate points and the belonging relationship between multiple three-dimensional coordinate points and multiple patches; Based on the connection relationship between multiple 3D coordinate points and the belonging relationship between multiple 3D coordinate points and multiple patches, multiple adjacent patches are constructed with multiple 3D coordinate points as vertices. The multiple patches are continuously connected to form a closed 3D surface, thus obtaining the reconstructed mesh.

[0009] In some optional designs, a reconstructed height map is obtained based on the reconstructed mesh to represent the geometric undulations of the physical environment, including: setting a downward-looking observation camera above the reconstructed mesh; The height values ​​of the grid are reconstructed by observing the camera at various horizontal sampling positions on the horizontal reference plane. Based on the height values ​​of the reconstructed mesh, a reconstructed height map is obtained to represent the geometric undulation information of the physical environment; The height value of the reconstructed mesh is used to characterize the geometric undulation information of the physical environment.

[0010] In some optional designs, before receiving the heightmap data and discrete mesh data, at least one data buffer for data exchange with the graphics processor is configured to receive the heightmap data and discrete mesh data, and initialize the texture size and image format of the heightmap according to preset parameters.

[0011] In some alternative designs, this method is applied to in-vehicle display terminals for real-time rendering and presentation of the vehicle's surrounding environment.

[0012] According to another aspect of the present disclosure, a physical environment visualization device is provided, the device comprising: The receiving module is used to receive data information of the height map representing the physical environment occupancy information and data information of the discrete grid; The layering module is used to perform layered processing along the height direction. Based on the data information of the height map, it obtains multiple height layer slices to represent the sampling information of each height layer. The reconstruction module is used to reconstruct the surface of the discrete mesh based on the data information of the discrete mesh, and obtain the reconstructed mesh; based on the reconstructed mesh, a reconstructed height map is obtained to express the geometric undulation information of the physical environment; The fusion module is used to fuse multiple height layer slices and reconstructed height maps to obtain a complete height map with contour information. The rendering module is used to render a visual display result based on the contour information height map.

[0013] In some optional designs, the fusion module is also used to retain valid contour information for each pixel location in multiple height layer slices and reconstructed height maps if valid contour information exists at the pixel location in any input map; and to determine a contour information height map with continuous and complete contour information based on the processing results of each pixel location.

[0014] In some optional designs, the rendering module is also used to render the contour information height map as a texture onto the reconstructed mesh; based on the reconstructed mesh, rendering is performed to generate image data corresponding to the contour information height map, and the image data is used as the visualization result.

[0015] In some optional designs, the layering module is also used to preset the height range for layering along the height direction; divide the height range into various sampling heights according to a fixed or adaptive step size; and perform image sampling on the height map at each height layer corresponding to each sampling height.

[0016] In some optional designs, the layering module is also used to determine, based on the height map data, whether the physical environment is occupied by one or more objects at each horizontal sampling position of each height layer, and to determine the sampling information of each height layer; map the sampling information of each height layer onto each two-dimensional pixel image, which is a set of pixels arranged in the form of a two-dimensional grid, and the coordinates of each pixel on each two-dimensional pixel image correspond to the coordinates of each horizontal sampling position of each height layer; construct each two-dimensional planar grid corresponding to each height layer, the horizontal range of each two-dimensional planar grid is consistent with the horizontal sampling range of each height layer, and the height of each two-dimensional planar grid is the same as the sampling height of each height layer; attach each two-dimensional pixel image to the surface of each two-dimensional planar grid to generate multiple textured planar grids carrying the sampling information of each height layer; render the multiple textured planar grids to obtain multiple height layer slices used to express the sampling information of each height layer.

[0017] In some optional designs, the reconstruction module is also used to determine the data information of multiple three-dimensional coordinate points and multiple patches for generating the discrete mesh based on the data information of the discrete mesh; A preset algorithm is used to determine the connection relationship between multiple three-dimensional coordinate points and the belonging relationship between multiple three-dimensional coordinate points and multiple patches; Based on the connection relationship between multiple 3D coordinate points and the belonging relationship between multiple 3D coordinate points and multiple patches, multiple adjacent patches are constructed with multiple 3D coordinate points as vertices. The multiple patches are continuously connected to form a closed 3D surface, thus obtaining the reconstructed mesh.

[0018] In some optional designs, the reconstruction module is also used to set up a downward-looking observation camera above the reconstruction mesh; to acquire the height value of the reconstruction mesh at each horizontal sampling position on the horizontal reference plane through the observation camera; and to obtain a reconstruction height map based on the height value of the reconstruction mesh to express the geometric undulation information of the physical environment; wherein the height value of the reconstruction mesh is used to characterize the geometric undulation information of the physical environment.

[0019] In some optional designs, the device also includes a processing module for configuring at least one data buffer for data exchange with a graphics processor before receiving the heightmap data and discrete mesh data, and for initializing the texture size and image format of the heightmap according to preset parameters.

[0020] According to another aspect of the present disclosure, a vehicle is provided, the vehicle including: an environmental perception sensor, an on-board processor, and an on-board memory; the environmental perception sensor is used to collect image data of the vehicle's surrounding environment, and the on-board memory stores a computer program; the computer program is loaded and executed by the on-board processor to realize the physical environment visualization method of the above aspects.

[0021] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising: A processor; a transceiver connected to the processor; a memory for storing executable instructions of the processor; wherein the processor is configured to load and execute the executable instructions to implement a visualization method of the physical environment as described above.

[0022] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, which stores at least one program that is loaded and executed by a processor to implement a method for visualizing a physical environment as described in the various aspects above.

[0023] According to another aspect of the present disclosure, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium, a processor retrieving the computer instructions from the computer-readable storage medium, and the processor executing the computer instructions to implement the physical environment visualization method as described in the various aspects above.

[0024] The technical solutions provided in this disclosure can include the following beneficial effects: In this disclosure, by fusing the outlines of multiple height layer slices and reconstructed height maps for integrated rendering, it can present the internal details and geometric accuracy of a specified horizontal section. At the same time, through the surface reconstruction of discrete meshes, it preserves the overall spatial structure and vertical continuity of the physical environment, achieving a smooth transition between different height layers. This avoids the discontinuities or holes that occur when rendering multiple height layer slices separately, and improves the coherence of the visualization. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A schematic diagram of an in-vehicle display provided in an exemplary embodiment of the present disclosure is shown; Figure 2This illustration shows a schematic diagram of the visualization result of a first occupancy network rendering scheme provided in an exemplary embodiment of the present disclosure; Figure 3 This illustration shows a schematic diagram of the visualization result of a second occupancy network rendering scheme provided in an exemplary embodiment of the present disclosure; Figure 4 A schematic diagram illustrating a method for visualizing a physical environment provided in an exemplary embodiment of this disclosure is shown; Figure 5 This illustration shows a schematic diagram of contour fusion of multiple height layer slice maps and reconstructed height maps provided in an exemplary embodiment of the present disclosure; Figure 6 A flowchart illustrating a method for visualizing a physical environment provided by an exemplary embodiment of this disclosure is shown; Figure 7 This illustration shows a flowchart of contour fusion of multiple height layer slice maps and reconstructed height maps provided in an exemplary embodiment of this disclosure; Figure 8 This invention illustrates a flowchart of a visualization display result rendered based on a contour information height map, provided by an exemplary embodiment of the present disclosure. Figure 9 This illustration shows a flowchart of a layered processing of a height map to obtain multiple height layer slice maps, provided by an exemplary embodiment of the present disclosure. Figure 10 A flowchart illustrating surface reconstruction of a discrete mesh provided in an exemplary embodiment of this disclosure is shown; Figure 11 This disclosure illustrates a flowchart of obtaining a reconstructed height map based on a reconstructed mesh, provided by an exemplary embodiment. Figure 12 This illustration shows a schematic diagram of performing layered processing on a height map to obtain multiple height layer slice maps, according to an exemplary embodiment of the present disclosure; Figure 13 A schematic diagram illustrating a method for visualizing a physical environment provided in an exemplary embodiment of this disclosure is shown; Figure 14 This diagram illustrates the effect of a visualization of an exemplary embodiment of the present disclosure. Figure 15 A flowchart illustrating a method for visualizing a physical environment provided by an exemplary embodiment of this disclosure is shown; Figure 16 A block diagram of a visualization apparatus for a physical environment provided in an exemplary embodiment of this disclosure is shown; Figure 17 A schematic diagram of the structure of a visualization device for a physical environment provided in an exemplary embodiment of this disclosure is shown. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this disclosure clearer, embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings. Exemplary embodiments will be described in detail here, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0028] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0029] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0030] In this embodiment, the vehicle collects environmental perception data such as visible light images, depth maps, and obstacle point clouds around the vehicle using environmental perception sensors such as cameras and LiDAR. The entire process of data collection, storage, processing, real-time rendering, and local use strictly complies with the laws and regulations of relevant countries or regions. All perception data is limited to use only for vehicle safety functions such as real-time rendering of the physical environment around the vehicle, obstacle warning, and parking assistance in this solution, fully protecting the data-related legal rights of the information subject.

[0031] Figure 1 This illustration shows an architectural diagram of an in-vehicle display screen 100 provided in an exemplary embodiment of the present disclosure. The display content of the in-vehicle display screen 100 includes: Occupancy Rendering (OCC rendering) result 110.

[0032] In some embodiments, the OCC rendering result 110 is a spatial representation result generated by the vehicle-mounted perception system through real-time detection and understanding of the surrounding physical environment, processed by an algorithm. The real-time detection and understanding of the surrounding physical environment includes obstacles in the physical environment and the space they occupy, and can also be understood as the real-time position, three-dimensional size, and motion state of one or more types of targets, including vehicles, pedestrians, and static objects. The OCC rendering result 110 can also be presented as any of the following: environmental situational awareness display, perception visualization interface, three-dimensional panoramic image system, or dynamic occupancy grid display; this disclosure does not limit this.

[0033] Among them, obstacles refer to physical objects detected by the vehicle perception system that occupy space on the current or potential driving path, mainly including one or more of the following: dynamic obstacles (such as other vehicles, pedestrians, and non-motorized vehicles), static obstacles (such as curbs, guardrails, construction cones, and medians), low obstacles (such as speed bumps and fallen goods), and suspended obstacles (such as traffic sign poles and height restriction poles).

[0034] The OCC rendering result 110 is presented as a simplified 3D geometry or a mesh with transparency, and uses color coding to distinguish obstacle type, height, or risk level. Optionally, the OCC rendering result 110 can be overlaid on a vehicle display image, bird's-eye view, or high-definition map, and displayed in a semi-transparent or wireframe style.

[0035] The OCC rendering result 110 is automatically activated when the vehicle is in intelligent driving mode (such as adaptive cruise control, assisted parking, and city navigation) to explain the environmental state perceived by the onboard perception system to the driver. It is highlighted when the onboard perception system detects potential risks (such as close-range cut-ins or blind spot obstacles) or when the driver actively queries the environmental state (such as via the panoramic view button). In one or more low-speed complex scenarios, including parking and navigating narrow lanes, the OCC rendering result 110 will be continuously displayed to provide continuous environmental perception assistance.

[0036] On the vehicle display screen 100, the OCC rendering result 110 is presented at a scaling ratio adapted to the vehicle display screen 100, and its size is dynamically adjusted according to the scaling level. The visual size of the obstacle has a certain proportional relationship with its actual physical size and distance. The overall display area of ​​the OCC rendering result 110 occupies a part or the entire screen of the vehicle display screen 100, and the specific proportion is automatically adjusted according to the driving mode and safety requirements.

[0037] In some embodiments, the in-vehicle display screen 100 is mounted on the dashboard in the center of the cockpit, in the area to the right of the steering wheel and between the driver and passenger seats. Optionally, the in-vehicle display screen 100 is mounted behind the steering wheel to display vehicle status, navigation information, and driver assistance information to the driver; and / or the in-vehicle display screen 100 is mounted directly in front of the passenger seat, on the back of the front seats, behind the center armrest, or tilted down from the roof to provide passengers with audio-visual entertainment, independent navigation, games, and vehicle environmental control functions; and / or the in-vehicle display screen 100 is mounted via a robotic arm or sliding rail, and its height and angle can be adjusted within a certain range to adapt to different driver seating positions or usage scenarios. This disclosure does not limit this aspect.

[0038] In some embodiments, the vehicle display 100 supports various auxiliary operation methods, including touch interaction, voice control, shortcut keys or physical controls, to control the viewing angle rotation of the OCC rendering result 110.

[0039] Optionally, in touch interaction mode, users can control the rotation of the OCC rendering result 110 on the display area of ​​the OCC rendering result 110 on the in-vehicle display screen 100 by using a single-finger swipe or two-finger twist gesture. Optionally, the in-vehicle display screen 100 supports rotation and reset operations via voice commands. Optionally, the vehicle is equipped with one or more hardware controls, including multi-function buttons on the steering wheel, a central control knob, or a touchpad, allowing users to indirectly control the rotation of the OCC rendering result 110 by using directional keys, knob rotation, or touchpad swipe gestures. Optionally, in some in-vehicle display screens 100 that support gesture recognition, users can trigger the corresponding rotation of the OCC rendering result 110 by using a specified gesture within the sensing range of the in-vehicle camera. The above interaction methods can be flexibly selected or combined according to the driving scenario and system configuration to ensure safe, efficient, and intuitive control of the viewing angle of the OCC rendering result in a driving environment.

[0040] In some embodiments, the vehicle display 100 is connected to the vehicle control system via an internal vehicle communication bus (such as an Ethernet vehicle bus or a Controller Area Network (CAN) bus) and other vehicle displays or other terminals in the vehicle; the OCC rendering result 110 in the vehicle display 100 is implemented through the corresponding service function module in the screen interface.

[0041] First, a brief introduction to one or more terms used in the embodiments of this disclosure: Screens: In the field of intelligent vehicles, screens include one or more of the following: central control screen, passenger entertainment screen, rear entertainment screen, sliding screen, magic control screen, and central touch screen. The central control screen, as a crucial hub for cockpit interaction, is typically located in the center of the dashboard. It offers comprehensive functionality, enabling navigation planning, multimedia playback, air conditioning adjustment, and seat settings, and some models also support vehicle-to-everything (V2X) connectivity and app downloads. The passenger entertainment screen serves as both a basic control function and can interact with other screens. The front passenger entertainment screen is a dedicated screen designed for the front passenger, allowing for independent video playback, content browsing, or game playing, avoiding interference with the driver. It also supports various functions, including smartphone screen mirroring, enhancing the front passenger experience. Rear entertainment screens are typically installed in the rear seat headrests or roof, are usually large, and offer one or more functions such as playing high-definition videos, connecting gaming devices, and mobile office capabilities. Some also support adjusting rear air conditioning and / or seats. The sliding screen is a movable interactive screen that allows for position adjustment by sliding it across different areas of the vehicle and / or angle adjustment. The smart control screen is a small interactive screen that integrates multiple convenient control functions, allowing users to control cabin functions from any location within the vehicle. The central touchscreen integrates specific vehicle control functions, centrally managing various detailed operations. As a supplement to the central control screen, it clarifies function categories and reduces the operational layers of a single screen.

[0042] Autonomous Driving Domain (ADD Domain): The ADD domain is the functional domain within the vehicle's electronic and electrical architecture responsible for intelligent driving functions. It is a functional integration and computing control unit within the vehicle. The ADD domain is responsible for centrally processing raw data from one or more sensors, including cameras, LiDAR, and millimeter-wave radar, and performing one or more computing tasks, including perception fusion, localization, decision planning, and control command generation.

[0043] In this disclosure, the ADD domain provides real-time structured data and a unified world coordinate system for perception visualization, ensuring that the data information of the height map and the data information of the discrete mesh are expressed in the same coordinate system centered on the vehicle, so that subsequent modeling, rendering and display have geometric consistency and real-time performance.

[0044] The Digital Cockpit Domain (DCD domain) is a functional domain within the vehicle's electronic and electrical architecture responsible for managing in-cabin interactions and experiences. It serves as the integration and execution hub for in-vehicle infotainment, human-machine interaction, and display systems. The DCD domain is responsible for driving and managing one or more functions, including the instrument panel, central control screen, head-up display, and in-vehicle systems. It processes user input commands and provides one or more interactive feedback methods, including graphical interfaces, audio, and video.

[0045] In this disclosure, the DCD domain receives data information from the height map and discrete grid from the ADD domain, and transforms them into graphics and images suitable for display on an in-vehicle display screen.

[0046] Discrete grid: A discrete grid is a data structure that uses regularly arranged cells to discretize and quantize a continuous space.

[0047] In this disclosure, the continuous physical environment around the vehicle is divided into a large number of regularly arranged cubic cells, thereby transforming the continuous and complex physical environment into a discrete data structure that can be efficiently processed by a computer.

[0048] Height map: A height map is a data structure that uses a two-dimensional array to represent three-dimensional height information. In this two-dimensional array, each pixel position corresponds to a horizontal position in the physical environment, and the value of each pixel position corresponds to the vertical height of that horizontal position.

[0049] Occupancy Rendering (OCC Rendering): OCC rendering is a computer graphics rendering technique used to draw voxelized scenes. It processes 3D data, represented as discrete meshes or continuums, and converts it into pixelated images on a 2D screen.

[0050] In this disclosure, OCC rendering performs graphical processing and visual presentation on the triangular mesh surface generated by the preset algorithm. By using different colors, one or more visual effects can be achieved. For example, different colors represent the probability of occupancy, distinguish object categories, represent distance, perform lighting calculations, generate shadows, and generate mixed transparency.

[0051] Secondly, the relevant content involved in this disclosure will be introduced.

[0052] As intelligent driving technology advances to higher levels, driving tasks and responsibilities are gradually shifting from the driver to the system. The importance of the human-machine interface in ensuring driving safety and building user trust is becoming increasingly prominent. To meet users' needs for high-quality information on system status, decision-making intentions, and environmental risks, and to address the standardized requirements of relevant domestic and international regulations and standards for intelligent driving human-machine interaction, perception and visualization systems have become a key component of intelligent driving vehicles.

[0053] Vehicles utilize data from one or more multi-source sensors, including LiDAR, cameras, and millimeter-wave radar, to generate and render real-time OCC (Optical Character Class) results of the physical environment, encompassing one or more elements such as traffic participants, obstacles, and lane markings. Mainstream advanced driver assistance systems (ADAS) commonly provide real-time perception and planning information displays on the central control or instrument panel. Furthermore, augmented reality head-up display (HUD) technology is gradually being commercialized, integrating navigation, warnings, and obstacle alerts in front of the driver's field of vision, further reducing line-of-sight deviation and improving interactive intuitiveness and driving safety.

[0054] In related technologies, the visualization results of the first network rendering scheme are as follows: Figure 2 As shown, the implementation method is to use discrete polyhedral primitives to represent each element of the network. Different colors are assigned to the discrete polyhedral primitives according to the type of each element to represent different types of obstacle information in the physical environment. The rendering of the polyhedral mesh is accelerated by instantiating the graphics processing unit (GPU).

[0055] The occupancy network rendering style presented by the first occupancy network rendering scheme is too pixelated, resulting in an overly engineered and unattractive display effect. Furthermore, the use of a fixed-size primitive grid cannot accurately express a precise grid occupancy rate. For a primitive grid, there are only two expressions: rendering or not rendering, which cannot accurately reflect the surrounding obstacle information.

[0056] The visualization results of the second network rendering scheme are as follows: Figure 3 As shown, the implementation involves discretizing the occupancy network according to the recognition height and abstracting the obstacle information in each height layer into multiple height map images. These multiple height map images within a certain height range are then overlaid and rendered to describe the shape and contour information of obstacles in the surrounding physical environment.

[0057] The second occupancy network rendering scheme can only be used in low-speed, parking, or certain specific scenarios. It cannot be provided in one or more scenarios, including driving. Moreover, since the second occupancy network rendering scheme uses multiple height maps within a certain range around the vehicle for superimposed rendering, when users use gestures to rotate the view to certain angles on the in-vehicle display, they will see obvious layering, resulting in a lack of refinement.

[0058] Based on the problems encountered in the aforementioned related technologies, this disclosure proposes a method for visualizing the physical environment, such as... Figure 4 As shown, it includes at least one of the following steps: Step 401: Configure at least one data buffer and initialize the texture size and image format of the heightmap.

[0059] In some embodiments, before receiving the heightmap data and discrete mesh data, at least one data buffer for data exchange with the graphics processor is configured to receive the heightmap data and discrete mesh data, and initialize the texture size and image format of the heightmap according to preset parameters.

[0060] In some embodiments, the height map data includes the height values ​​of the physical environment at each horizontal sampling position on the horizontal reference plane within a preset horizontal sampling range. The height values ​​are used to characterize whether the physical environment at each horizontal sampling position on the horizontal reference plane is occupied by one or more objects, and when it is occupied by one or more objects, the height of one or more objects.

[0061] The horizontal reference plane is a preset horizontal reference plane that serves as a reference for determining the height of one or more objects. Each horizontal sampling position is a coordinate position obtained by dividing the horizontal sampling range according to the preset sampling accuracy of the height map.

[0062] In some embodiments, the physical environment is occupied by one or more objects at at least one horizontal sampling position among various horizontal sampling positions on the horizontal reference plane; the data information of the discrete mesh includes data information of multiple three-dimensional coordinate points and multiple patches used to generate the discrete mesh, and the data information of the three-dimensional coordinate points and patches of the discrete mesh is used to characterize the spatial position and geometry of one or more objects.

[0063] Among them, multiple three-dimensional coordinate points are multiple spatial points obtained by discrete sampling of the surface of one or more objects, and patches are geometric units obtained by connecting multiple three-dimensional coordinate points according to preset connection rules.

[0064] Optionally, the discrete mesh data information includes environmental state quantization parameters contained in each voxel unit within the three-dimensional space after the physical environment is divided into regular meshes. These environmental state quantization parameters include at least one of the following: the probability that each voxel unit is occupied by an obstacle, the object category corresponding to each voxel unit, and the inter-frame changes and motion trends of the state of each voxel unit.

[0065] In this context, a voxel is the smallest indivisible basic cube unit within a regular grid that divides three-dimensional space, and it has unique index coordinates in three-dimensional space. All voxel units are closely arranged in three-dimensional space, without overlap or gaps, collectively filling the entire three-dimensional space.

[0066] Step 402: The ADD domain and DCD domain establish a link through the vehicle Ethernet. The ADD domain obtains the height map data and discrete mesh data through sensing technology and transmits them to the DCD domain through the vehicle Ethernet. After receiving the height map data and discrete mesh data, the assisted driving rendering service on the DCD domain system processes the data.

[0067] Within the DCD domain system, the Assisted Driving Rendering Service is a software module used to generate real-time visualizations of intelligent driving. This service receives heightmap data and discrete mesh data from the ADD domain, transforms this data into a visual scene containing one or more elements, including 3D obstacle models, terrain undulations, lane lines, and navigation guidance, through the graphics rendering pipeline. It then applies one or more visual processing techniques, such as color coding and lighting, and finally renders the visualization based on the completed contour heightmap, outputting it to the in-vehicle display screen.

[0068] Step 403: After receiving the height map data, perform layer processing at each sampling height. Each sampling height corresponds to a height layer. Perform image sampling processing on the height map at each sampling height to obtain multiple height layer slices that express the sampling information of each height layer. Then, stack and combine the multiple height layer slices.

[0069] Step 404: After receiving the data information of the discrete mesh, perform surface reconstruction on the discrete mesh based on the data information of the discrete mesh to obtain the reconstructed mesh; based on the reconstructed mesh, obtain the reconstructed height map used to express the geometric undulation information of the physical environment.

[0070] Step 405: Perform contour fusion on multiple height layer slice images and reconstructed height maps to obtain a complete contour information height map, and render the visualization display result based on the contour information height map.

[0071] Step 406: Present the visualization results on the in-vehicle display screen to provide users with rendering and feedback.

[0072] In summary, the OCC rendering results generated by this disclosure not only solve the coarseness of OCC discrete mesh rendering, but also solve the sense of layering in OCC heightmap rendering.

[0073] The numbers in each step of the above flowchart are used to uniquely identify and reference each step, and do not represent the order of time or importance of the steps. This disclosure does not impose any restrictions on this.

[0074] This disclosure does not limit the data used in the height layer slices, the perceived range of the physical environment represented, or the algorithm for surface reconstruction of the discrete mesh. This disclosure uses 19 height layer slices as an example.

[0075] For example, multiple height layer slices are used to represent the general outline information of obstacles in the physical environment. A reconstructed height map is obtained based on the data information of the preset algorithm and discrete grid. The multiple height layer slices are fused with the reconstructed height map to obtain a complete outline information height map. The multiple height layer slices represent a range centered on the rear axle of the target vehicle, with a length of 20 meters, a width of 20 meters, and a height of 2 meters.

[0076] For example, the preset algorithm includes a preset isosurface extraction algorithm, wherein the isosurface extraction algorithm includes one or more algorithms such as the Marching Cubes Algorithm, the Moving Tetrahedron Algorithm, the Poisson Surface Reconstruction Algorithm, the Surface Network Algorithm, and the Neural Implicit Representation.

[0077] Among them, the Marching Cubes algorithm is an algorithm for extracting isosurfaces from a three-dimensional scalar field. By processing cube elements in a discrete mesh one by one, and comparing the scalar values ​​of the three-dimensional coordinates of each cube element with a preset threshold, a preset topology table is searched, thereby generating one or more triangular patches within the cube element.

[0078] For example, the Marching Cubes algorithm takes a discrete grid containing discrete grid data as input three-dimensional scalar field. By traversing each discrete grid, it constructs smooth, continuous triangular mesh surfaces at the boundaries where the physical environment is occupied by one or more objects and where the physical environment is not occupied by one or more objects. In this way, the discrete grid data is converted into geometric primitives that can be processed by the graphics rendering pipeline.

[0079] In summary, this disclosure integrates the contours of multiple height layer slices and reconstructed height maps for unified rendering. This not only presents the internal details and geometric accuracy of a specified horizontal section, but also preserves the overall spatial structure and vertical continuity of the physical environment through the surface reconstruction of discrete meshes, achieving a smooth transition between different height layers. This avoids the discontinuities or holes that occur when rendering multiple height layer slices separately, thus improving the coherence of the visualization.

[0080] Figure 5 This illustration shows a schematic diagram of contour fusion of multiple height layer slices and reconstructed height maps provided in an exemplary embodiment of the present disclosure.

[0081] For example, such as Figure 5 As shown, the contour blending process mainly includes the following: The input contains two sets of contour data from different sources. The upper left shows multiple height layer slices obtained by layered sampling along the height direction. Each slice is presented in a stacked form, corresponding to different spatial height levels (layer 1 to layer 5), used to express the cross-sectional contour information after vertical discretization. The lower left shows the reconstructed height map obtained based on discrete mesh surface reconstruction, which expresses the overall contour information of the continuous surface in the form of a grayscale pixel array.

[0082] The intermediate process illustrates the core steps of pixel-by-pixel comparison. After spatially aligning the multiple height layer slices with the reconstructed height map, each pixel position is compared one by one: if any input image contains valid contour information at that pixel position, it is retained; when contour information from different sources conflicts at that pixel position, a decision is made based on geometric priors. The figure uses different gray levels to distinguish contourless regions, valid contour regions, and conflicting pixel regions.

[0083] The right side of the output shows the completed contour information height map obtained after contour fusion. This contour information height map ensures the continuity and integrity of the contour through closure detection and interpolation completion, preserving the cross-sectional details in the height layer slice map while integrating the overall spatial structure of the reconstructed height map.

[0084] Figure 6 The flowchart illustrates a method for visualizing a physical environment provided in an exemplary embodiment of this disclosure. This method is executed by a physical environment visualization device, with the physical environment visualization device serving as the... Figure 1 Taking the vehicle-mounted display screen 100 shown as an example, the method includes at least one of the following steps: Step 610: Receive the data information of the height map representing the physical environment occupancy information and the data information of the discrete grid.

[0085] In some embodiments, the height map data includes the height values ​​of the physical environment at each horizontal sampling position on the horizontal reference plane within a preset horizontal sampling range. The height values ​​are used to characterize whether the physical environment at each horizontal sampling position on the horizontal reference plane is occupied by one or more objects, and when it is occupied by one or more objects, the height of one or more objects.

[0086] In some embodiments, the data information of the discrete mesh includes data information of multiple three-dimensional coordinate points and multiple facets used to generate the discrete mesh, and the data information of the three-dimensional coordinate points and facets of the discrete mesh is used to characterize the spatial position and geometry of one or more objects.

[0087] Step 620: Perform layered processing along the height direction. Based on the data information of the height map, obtain multiple height layer slice maps to express the sampling information of each height layer.

[0088] In some embodiments, the sampling information of each height layer is used to characterize whether the physical environment is occupied by one or more objects at each horizontal sampling position of each height layer.

[0089] In some embodiments, the height direction refers to the vertical axis based on the direction of gravity, represented as the Z-axis of the world coordinate system or vehicle coordinate system. When digitally representing the physical environment in three-dimensional space, the height direction is the direction perpendicular to the ground, used to measure the vertical distance of one or more objects occupying the physical environment from the ground.

[0090] In some embodiments, hierarchical processing refers to dividing a continuous height range into multiple discrete and continuous intervals, and processing and analyzing the data within each interval independently. Exemplarily, hierarchical processing includes at least one of the following steps: Define layering parameters, including one or more parameters such as height range, layer thickness, and number of layers; based on the heightmap data, classify each 3D coordinate point in the 3D space of the physical environment into its corresponding height layer according to its Z-axis coordinate value; independently statistically analyze or visualize the point cloud or raster data within each height layer. By reducing the complex 3D spatial analysis to multiple 2D layers, computational complexity is reduced while preserving the structured information of the vertical dimension.

[0091] In some embodiments, obtaining multiple height layer slices is the output of layered processing, which can also be understood as generating a series of two-dimensional images, each corresponding to environmental information sampling at each height layer. First, based on the layering parameters, data corresponding to each height layer is extracted, including extracting the sampling information of each height layer, which can also be understood as extracting each point cloud and its attributes or occupancy raster state on each height layer; the sampling information of each height layer is vertically projected onto a horizontal reference plane to form height layer slices for each height layer. For example, the projected sampling information of each height layer is rasterized and visualized, and the density, average height, or occupancy probability of each three-dimensional coordinate point is represented by the color of each pixel, generating height layer slices that can intuitively display the obstacle distribution, terrain features, or semantic information of each height layer.

[0092] Each point cloud at each altitude level represents a 3D coordinate point within that altitude level. The attributes of each 3D coordinate point within each altitude level are analyzed to identify the material composition, category distribution, and surface morphology details of obstacles within that altitude level. The attributes of each point cloud include at least one of the following: reflection intensity, object category information, local surface normal vector, and curvature.

[0093] In this context, "occupied grid state" refers to discretizing the three-dimensional space corresponding to each height layer into a regular two-dimensional grid array. Grid cells within this array are assigned status labels, including: occupied, idle, and unknown. An occupied state indicates that the three-dimensional space corresponding to the grid cell is filled by one or more objects; an idle state indicates that the three-dimensional space corresponding to the grid cell is freely passable; and an unknown state indicates that the three-dimensional space corresponding to the grid cell has not been effectively detected by the sensor.

[0094] Optionally, multiple height layer switching maps can be replaced with multiple depth maps.

[0095] Step 630: Reconstruct the surface of the discrete mesh based on the data information of the discrete mesh to obtain the reconstructed mesh; based on the reconstructed mesh, obtain the reconstructed height map to express the geometric undulation information of the physical environment.

[0096] In some embodiments, geometric undulation information of the physical environment is used to characterize the height undulations and surface boundaries of one or more objects.

[0097] Optionally, geometric undulation information is a set of geometric parameters used to comprehensively characterize the physical environment, including the continuous undulation features of the roadbed and the envelope morphology of one or more object surfaces. For example, geometric undulation information characterizes one or more terrain attributes, including road surface slope and curvature, and / or abrupt changes in the surface curvature and relative height of one or more objects, including vehicles and obstacles.

[0098] In some embodiments, surface reconstruction refers to constructing a smooth geometric surface at the boundary between occupied and idle regions based on the environmental state quantization parameters contained in each voxel unit and using an isosurface extraction algorithm, thereby converting the discrete mesh into a continuous geometric surface that can be rendered by the graphics pipeline, and realizing the reconstruction of the outer surface of one or more objects.

[0099] In some embodiments, the reconstructed mesh includes at least one of the following features: describing the topology of the object surface using triangular facets as basic units, and each vertex corresponding to a three-dimensional coordinate point in three-dimensional space.

[0100] Step 640: Perform contour fusion on multiple height layer slices and reconstructed height maps to obtain a complete contour information height map.

[0101] In some embodiments, contour fusion of multiple height layer slice images and reconstructed height images involves combining the vertically discretized cross-sectional contours provided by the multiple height layer slice images with the continuous overall surface contours provided by the reconstructed height image, and generating a contour information height image that can reflect the complete outer edge of the object and retain the internal structural details through geometric reasoning and interpolation.

[0102] For example, contour lines or significant edges are extracted from multiple height layer slices and reconstructed height maps as contour lines, respectively, and spatial correspondence between contour lines from different sources is established. Based on the geometric consistency principle including smoothness and continuity, contour lines from different sources are merged, conflict is resolved, and missing parts are interpolated and filled.

[0103] In some embodiments, height slices are cross-sections of the physical environment on a series of discrete horizontal planes. Therefore, it is difficult to directly represent the continuous, smooth, and complete outer surface of an object in the vertical direction. For inclined surfaces or complex curved surfaces, the object's contour exhibits positional jumps across different height slices, failing to form smooth, continuous boundaries. In densely populated areas, multiple height slices may provide repetitive edge information; while in simple or open areas, the sampling information provided by multiple height slices for each height level appears redundant; and when the distance between sampling heights is large, important contour features may be missed between two height levels.

[0104] In some embodiments, the reconstructed height map is obtained by sampling after surface reconstruction of a discrete mesh. While removing noise, there is a situation where some sharp, local geometric features are over-smoothed out. For example, it is impossible to accurately display one or more local features, including the edges of a vehicle or the fine contours of a thin rod-like object.

[0105] This disclosure fuses multiple height layer slices and a reconstructed height map into a contour. In areas where the overall contour is smooth on the reconstructed height map, the cross-sectional contour extracted from the corresponding height layer slice is introduced as a constraint. One or more local features, including sharp corners and small protrusions that were smoothed out during the surface reconstruction process, are reintroduced into the final contour information height map, improving the geometric fidelity of the contour.

[0106] For example, using the continuous and smooth surface model provided by the reconstructed height map as the skeleton, the contour segments extracted from multiple height layer slices that belong to the same object in three-dimensional space but are discrete in position are reasonably connected and interpolated to construct a complete and coherent three-dimensional boundary representation of the object, thus solving the problem of vertical discontinuity in the height layer slices.

[0107] For example, for the same edge of the same object, the fusion algorithm simultaneously examines the overall edge given by the reconstructed heightmap and the local edges given by multiple height layer slice images. When the overall edge and local edges are highly consistent in 3D space, the reliability of the contour is enhanced; when there are differences between the overall edge and local edges, the fusion algorithm makes judgments and optimizations based on geometric priors, thereby effectively suppressing noise and errors in single-source data and outputting a more reliable and accurate object contour. For vertical walls, combining the cross-sectional edges of each height layer provided by the height layer slice images more accurately locates its true boundary, mitigating the edge blurring problem when relying solely on heightmaps.

[0108] The fusion algorithms include optimization-based methods, probability and statistics-based methods, and data-driven methods.

[0109] Optionally, optimization-based methods include constructing and solving an energy function that minimizes contour differences and violates geometric priors; probabilistic and statistical methods include Bayesian inference that treats different sources as noisy observations of the true contour; and data-driven methods include using deep learning models to learn an end-to-end mapping from multi-source inputs to the optimized contour.

[0110] Geometric priors refer to the inherent laws and reasonable constraints regarding the geometric structure and spatial relationships of the physical environment introduced during multi-source data fusion. Geometric prior knowledge originates from general observations and inductions of real-world scenarios. For example, geometric prior knowledge includes one or more inherent laws or reasonable constraints, such as the continuity of object surfaces, the vertical or parallel alignment characteristics of major structural edges, the approximate size range of common obstacles, and the support relationships that require objects to contact the ground.

[0111] For example, when there are conflicts, ambiguities or missing contour information from multiple height layer slices and reconstructed height maps, the fusion algorithm infers the most likely true contour shape based on geometric priors, thereby avoiding the generation of geometric results that violate physical common sense, are broken or distorted, and ensuring that the output completed contour not only conforms to the input data, but also follows the spatial logic of the actual physical environment.

[0112] Step 650: Render the visualization result based on the contour information height map.

[0113] For example, the visualization displayed on the in-vehicle display is a real-time generated OCC rendering result. This OCC rendering result is centered on the vehicle itself, using a bird's-eye view or a tilted view, with a reference surface in the background that clearly expresses the road surface slope and undulation with color gradients or grid textures.

[0114] In this disclosure, by fusing the outlines of multiple height layer slices and reconstructed height maps for integrated rendering, it can present the internal details and geometric accuracy of a specified horizontal section. At the same time, through the surface reconstruction of discrete meshes, it preserves the overall spatial structure and vertical continuity of the physical environment, achieving a smooth transition between different height layers. This avoids the discontinuities or holes that occur when rendering multiple height layer slices separately, and improves the coherence of the visualization.

[0115] The following examples will provide a detailed description of the physical environment visualization method.

[0116] Contour fusion of multiple height layer slices and reconstructed height maps For step 640, such as Figure 7 As shown, contour fusion is performed on multiple height layer slices and reconstructed height maps to obtain a complete contour information height map, including: Step 641: For each pixel location in multiple height layer slices and the reconstructed height map, if valid contour information exists at the pixel location in any input map, then retain the valid contour information.

[0117] For example, multiple height layer slices and reconstructed height maps are overlaid, and then the numerical differences between the multiple height layer slices and reconstructed height maps at exactly the same pixel locations are checked one by one.

[0118] Before comparison, it is ensured that multiple height layer slices and reconstructed height maps are absolutely aligned in space. This can also be understood as multiple height layer slices and reconstructed height maps having the exact same resolution, i.e., the same width and height, containing the same number of pixels, and each pixel corresponding to the exact same 3D coordinate point in the 3D space of the physical environment.

[0119] For example, starting from the top left corner of the image, the first pixel is extracted. For example, the grayscale value of this location in the height layer slice image is 50, while the grayscale value of this location in the reconstructed height map is 55. The difference between these two grayscale values ​​is recorded. Subsequently, following the order from left to right and from top to bottom, all pixels of the complete image are traversed without omission, and their differences are recorded one by one. The differences are used to control the blending weight of each pixel location. In areas with large differences, more of the overall contour of the continuous surface carried by the reconstructed height map is displayed.

[0120] Optionally, multiple height layer slices and reconstructed height maps store the vertical height data for each pixel location. When there are abrupt changes in height along the object's contour, the height difference between the current pixel location and its neighboring pixel locations is calculated. If this height difference exceeds a system-preset gradient threshold, the current pixel location is marked as having valid contour information. For example, the height of the edge of a zebra crossing on a road or the side skirt of a vehicle changes drastically; the pixel location corresponding to this change is marked as having valid contour information.

[0121] Optionally, the height change rate is greatest at the contour, and the acceleration of the height change is zero or undergoes a sudden change in sign. Pixel locations where the second derivative crosses zero in multiple height layer slices and the reconstructed height map, or pixel locations with large absolute values ​​of local second derivatives, are marked as pixel locations containing valid contour information.

[0122] Optionally, for a pixel location and a small neighborhood around it, the height values ​​of each pixel are extracted, and the variance or standard deviation of the height values ​​of each pixel is calculated. If the variance or standard deviation is greater than a set threshold, it indicates that there are obvious undulations in the height of a pixel location and a small neighborhood around it, that is, there is an object edge or a valid contour. The pixel location is then marked as having valid contour information.

[0123] Optionally, in multiple height layer slices, different height layer slices represent different spatial levels. By subtracting adjacent height layer slices, when the numerical difference between adjacent height layer slices at a pixel location is greater than a first threshold, it is indicated that the pixel location is the boundary between upper and lower levels, and the pixel location is marked as a pixel location with valid contour information.

[0124] Step 642: Based on the processing results of each pixel position, determine a contour information height map that is continuous and complete.

[0125] For example, after pixel-by-pixel comparison of multiple height layer slices and reconstructed height maps, the comparison results are output in the form of a multidimensional data matrix, which contains at least one of the following three dimensions: • Spatial coordinate mapping table: records the coordinates of each pixel position in the three-dimensional space of the physical environment.

[0126] • Validity Marker Matrix: The validity marker matrix is ​​a matrix consisting of Boolean values ​​or 0s and 1s, used to mark whether any input image provides valid contour information at each pixel location.

[0127] • Error and Fusion Weight Matrix: The error and fusion weight matrix calculates the height difference at the same pixel location in different input images. It is used to determine the proportion of multiple height layer slices and the reconstructed height map when generating the final contour information height map.

[0128] Based on the above three dimensions, at least one of the following steps is used to obtain a continuous and complete contour information height map: Traverse each pixel position; if the validity marker matrix of the pixel position is true (i.e., any input image has a valid contour at that pixel position), retain the contour data of that pixel position in the final contour information height map; search for closely connected valid pixel groups in the physical environment; if one or more valid pixels are found to be isolated or extremely short (less than the preset minimum physical size), determine that the one or more valid pixels are sensor noise or flying points, and remove them from the final contour information height map; check whether the retained contours form one or more complete closed loops. By tracing the direction of the contour lines, if a small height difference or spatial gap is found between the start and end points of a contour line, interpolation calculation is performed to close it.

[0129] Visual display results are rendered based on the contour information height map. In some embodiments, for step 650, such as Figure 8 As shown, the visualization results are rendered based on the contour information heightmap, including: Step 651: Render the contour information height map as a texture onto the reconstructed mesh.

[0130] Step 652: Render based on the reconstructed mesh to generate image data corresponding to the contour information height map, which will be used as the visualization result.

[0131] Optionally, the contour information height map is rendered as a texture onto the reconstructed mesh, rendering is performed based on the reconstructed mesh to generate image data corresponding to the contour information height map, and the image data is used as the visualization result, including at least one of the following steps: • Establish the spatial correspondence between the vertices of the reconstructed mesh and the two-dimensional pixel image.

[0132] Optionally, a set of coordinates is calculated for each vertex of the reconstructed mesh, defining the sampling position of each vertex in the contour information height map. These coordinates are generated using a parametric mapping algorithm to ensure that the geometric continuity of important contours and minimize distortion are maintained when the 3D reconstructed mesh is unfolded into 2D texture space.

[0133] Configure texture sampling and filtering parameters.

[0134] Optionally, the sampling state and filtering rules of the texture can be configured in the graphics rendering pipeline. The default contour information height map is a single-channel grayscale texture, and an appropriate sampler filtering mode can be configured to balance accuracy and performance.

[0135] • Write and bind shader programs.

[0136] Optionally, a shader program is developed to implement texture sampling and visual mapping. The shader reads the coordinates of each pixel position, samples the corresponding grayscale value from the contour information height map, and converts the height information corresponding to each pixel position into visual attributes through a preset transfer function.

[0137] The transfer function is implemented as a color lookup table, which maps the height information corresponding to each pixel position to the color value of the specified color band.

[0138] • Perform real-time rendering and compositing.

[0139] Optionally, the configured texture, shader, and reconstructed mesh data are submitted to the GPU rendering pipeline. The GPU processes each pixel in parallel, obtains the height information corresponding to each pixel position through texture sampling, applies the lighting and material models defined in the shader for calculation, and finally generates image data corresponding to the contour information height map.

[0140] For example, other visual elements in the physical environment include, but are not limited to, at least one of the following: icons and real-time distance labels floating above the outlines of vehicles and pedestrians for quick identification of traffic participants; colored light strips, lane-changing guide arrows, and electronic fences in the center of the road to provide clear route planning and traffic restriction prompts; giant navigation arrows projected onto the real road surface and Point of Interest (POI) labels registered in the physical environment; and system start / stop, takeover requests, and sensor status conveyed through surrounding halos and flashing icons.

[0141] Points of Interest (POIs) are virtual, two-dimensional graphical interface elements used to represent locations useful for driving or travel. POIs include one or more locations such as gas stations, parking lots, charging stations, restaurants, hotels, and attractions.

[0142] For example, the POI label may take one or more of the following forms: a "P" icon, a gas station nozzle icon, a coffee cup icon, or a text label with a name.

[0143] The height map is layered to obtain multiple height layer slices. In some embodiments, for step 620, such as Figure 9As shown, layering is performed along the height direction. Based on the data information of the height map, multiple height layer slices are obtained to represent the sampling information of each height layer, including: Step 621: Preset the height range for layering along the height direction.

[0144] For example, based on the effective detection capability of the sensor and the driving scenario, a specific height range is preset, for example, the height range is from the ground to the top of the largest obstacle that the vehicle can detect.

[0145] The driving scenarios include one or more scenarios such as parking, narrow space passage, complex terrain and indoor scenarios, and off-road obstacle passage.

[0146] Step 622: Divide the height range into various sampling heights according to a fixed or adaptive step size.

[0147] For example, a series of horizontal cross-section positions are determined within a preset height range. Optionally, the division method is a uniform fixed step size, for example, taking a height layer every 0.2 meters; alternatively, the division method is adaptive, using a smaller step size in areas with dense obstacles or complex geometry to capture details, and a larger step size in open areas to save computing power.

[0148] Step 623: Perform image sampling on the height map at each height layer corresponding to each sampling height.

[0149] For example, the height map is a two-dimensional matrix, where each pixel stores the height value of the object surface at the corresponding horizontal sampling position. By traversing each pixel of the height map at each sampling height, the relationship between the height value stored in each pixel and each sampling height is determined, thereby determining whether each horizontal sampling position at each height layer intersects with the object surface.

[0150] Step 624: Based on the data information of the height map, determine whether the physical environment is occupied by one or more objects at each horizontal sampling position of each height layer, and determine the sampling information of each height layer.

[0151] For example, for each horizontal sampling position on each height layer, binarization or probabilistic judgment is performed based on the data information of the height map. The height value of the object at each horizontal sampling position on each height layer is compared with the corresponding sampling height of each height layer.

[0152] Optionally, if the sampling height corresponding to each height layer is less than or equal to the height value of the object at each horizontal sampling position, it indicates that the voxel corresponding to each horizontal sampling position is located on or inside the surface of one or more objects, thereby determining that the physical environment is occupied by one or more objects at each horizontal sampling position.

[0153] Optionally, if the sampling height corresponding to each height layer is greater than the height value of the object at each horizontal sampling position, it indicates that the voxel corresponding to each horizontal sampling position is located on the surface of one or more objects, thus determining that the physical environment is not occupied by one or more objects at each horizontal sampling position. By traversing each horizontal sampling position of each height layer, a two-dimensional occupancy state matrix is ​​generated for each height layer. This matrix is ​​of Boolean or probability type and is used to represent the sampling information of each height layer.

[0154] Step 625: Map the sampling information of each height layer onto each two-dimensional pixel image.

[0155] The two-dimensional pixel image is a collection of pixels arranged in a two-dimensional grid. The coordinates of each pixel in each two-dimensional pixel image correspond to the coordinates of each horizontal sampling position in each height layer.

[0156] For example, a two-dimensional pixel image is created for each height layer, and each pixel position in the two-dimensional pixel image uniquely corresponds to each horizontal sampling position. The sampling information of each height layer is encoded into grayscale values ​​or color channel values ​​corresponding to each pixel position, and the sampling information of each height layer is converted into intuitively visualized two-dimensional pixel images.

[0157] Step 626: Construct the individual two-dimensional planar meshes corresponding to each height layer.

[0158] The horizontal range of each two-dimensional planar grid is consistent with the horizontal sampling range of each height layer, and the height of each two-dimensional planar grid is the same as the sampling height of each height layer.

[0159] Step 627: Attach each two-dimensional pixel image to the surface of each two-dimensional planar grid to generate multiple textured planar grids that carry the sampling information of each height layer.

[0160] For example, each generated two-dimensional pixel image is used as a texture image and attached to the corresponding two-dimensional planar mesh. By establishing a mapping relationship between the coordinates of each pixel position in each two-dimensional pixel image and the vertex coordinates of the two-dimensional planar mesh, it is ensured that each pixel in each two-dimensional pixel image is correctly displayed in the corresponding position of the two-dimensional planar mesh, thereby generating multiple textured planar meshes that carry the sampling information of each height layer.

[0161] Step 628: Render multiple textured planar meshes to obtain multiple height layer slices that represent the sampling information of each height layer.

[0162] For example, multiple textured planar meshes are received, and the color, brightness, and transparency of the multiple textured planar meshes are calculated based on the camera view, lighting model, and material settings, thereby rendering the multiple textured planar meshes to obtain multiple height layer slice images.

[0163] Among them, the multiple height layer slices include multiple semi-transparent image layers arranged in order of height. On each of the multiple semi-transparent image layers, the horizontal distribution of one or more objects occupying the physical environment at each sampling height is shown in clear graphics (such as highlighted dots, lines or regions).

[0164] Surface reconstruction of discrete mesh In some embodiments, for step 630, such as Figure 10 As shown, surface reconstruction is performed on the discrete mesh based on the data information of the discrete mesh to obtain the reconstructed mesh, including: Step 631: Based on the data information of the discrete mesh, determine the data information of multiple three-dimensional coordinate points and multiple facets used to generate the discrete mesh.

[0165] Step 632: Use a preset algorithm to determine the connection relationship between multiple three-dimensional coordinate points and the belonging relationship between multiple three-dimensional coordinate points and multiple facets.

[0166] For example, the three-dimensional space of the physical environment is divided into cubic units with a fixed step size, and each cubic unit contains 8 voxel corner points; the occupancy or vacancy information of the 8 voxel corner points of each cubic unit is determined to form an 8-bit binary status code; based on the 8-bit binary status code, the voxel edges traversed by the isosurface in each cubic unit are determined by a preset algorithm; for the voxel edges with occupancy and vacancy at the two end voxel corner points respectively, the surface vertices of the isosurface are calculated by linear interpolation; according to the triangular facet topological relationship in the lookup table, the surface vertices are connected to form triangular facets.

[0167] Step 633: Construct multiple adjacent facets using multiple 3D coordinate points as vertices. The facets are continuously connected to form a closed 3D surface, resulting in a reconstructed mesh.

[0168] The construction of adjacent patches is determined based on the connection relationship between multiple three-dimensional coordinate points and the attribution relationship between multiple three-dimensional coordinate points and multiple patches.

[0169] For example, multiple three-dimensional coordinate points are used as vertices, and then, according to the connection order specified by a preset algorithm, every three vertices belonging to the same facet are connected to form a triangular facet. These triangular facets are spatially adjacent and seamlessly connected, ultimately combining to form a complete and closed reconstructed mesh, thereby ensuring the topological correctness and geometric consistency from the discrete mesh to the continuous surface.

[0170] Based on the reconstructed mesh, a reconstructed height map is obtained. In some embodiments, for step 630, such as Figure 11 As shown, based on the reconstructed mesh, a reconstructed height map is obtained to represent the geometric undulation information of the physical environment, including: Step 634: Set up a downward-looking observation camera above the reconstructed mesh.

[0171] For example, an observation camera using orthographic projection is created directly above the reconstructed mesh. The camera's position and orientation are calibrated to ensure its optical axis is parallel to the direction of gravity and completely covers the reconstructed mesh and its corresponding horizontal reference plane in a vertically downward direction. This configuration eliminates distortion caused by perspective projection, ensuring geometric consistency when acquiring the height values ​​of the reconstructed mesh.

[0172] Step 635: Reconstruct the height values ​​of the grid at each horizontal sampling position on the horizontal reference plane by observing the camera.

[0173] For example, within a preset horizontal reference plane, a regularly distributed array of sampling points is generated at a preset resolution. A detection ray is emitted vertically upwards along each horizontal sampling position, and the intersection point with the triangular facet of the reconstructed mesh is calculated. The three-dimensional spatial coordinates of the first intersection point are calculated, and the Z-axis component of the three-dimensional spatial coordinates is extracted as the height value of each horizontal sampling position, thereby obtaining the height value of the reconstructed mesh.

[0174] Step 636: Based on the height values ​​of the reconstructed mesh, obtain a reconstructed height map to represent the geometric undulation information of the physical environment.

[0175] The height value of the reconstructed mesh is used to characterize the geometric undulation information of the physical environment.

[0176] For example, the height values ​​of the acquired reconstructed grid are normalized and converted into standardized grayscale or pseudo-color values ​​according to a preset numerical mapping relationship. The converted grayscale or pseudo-color values ​​are then arranged in spatial order according to the horizontal sampling positions to form a two-dimensional raster image, i.e., a reconstructed height map. This reconstructed height map not only contains the height information of terrain and obstacles but also inherits the geometric integrity and smoothness characteristics of the reconstructed grid, resulting in higher data quality and stronger noise resistance, reflecting the macroscopic undulation characteristics and microscopic structural details of the physical environment.

[0177] Program initialization In some embodiments, before receiving the heightmap data and discrete mesh data, at least one data buffer for data exchange with the graphics processor is configured to receive the heightmap data and discrete mesh data, and initialize the texture size and image format of the heightmap according to preset parameters.

[0178] Optionally, before receiving the heightmap data and discrete mesh data, a dedicated data exchange channel with the graphics processor is established. This is achieved by configuring at least one data buffer, which can also be understood as forming a structured storage area in graphics memory, serving as a high-speed data transmission bridge between the central processing unit and the graphics processor. The at least one data buffer must meet both the textured storage requirements of the heightmap and the structured storage requirements of the discrete mesh.

[0179] For example, at least one configured data buffer is used to receive heightmap data and discrete mesh data. A texture buffer is configured for the heightmap data, allowing the heightmap data to be sampled and accessed in the shader program as an image texture. Vertex buffers and index buffers are configured for the discrete mesh data, storing the geometric attribute data of 3D coordinate points and the topological connectivity between points, respectively.

[0180] For example, after at least one data buffer is configured, the texture size and image format of the heightmap are initialized according to preset parameters. The texture size of the heightmap matches the spatial resolution of the heightmap's data information. The image format of the heightmap needs to balance accuracy requirements with memory bandwidth; for example, the image format of the heightmap is a single-channel floating-point format or a compressed integer format.

[0181] In some embodiments, the method is applied to an in-vehicle display terminal for real-time rendering and presentation of the vehicle's surrounding environment. The in-vehicle display terminal includes, but is not limited to, one or more of the following: an in-vehicle central control screen, an instrument display screen, and a streaming media rearview mirror display screen.

[0182] The above-mentioned "contour fusion of multiple height layer slices and reconstructed height maps", and / or "rendering visualization results based on contour information height maps", and / or "layering of height maps to obtain multiple height layer slices", and / or "surface reconstruction of discrete meshes", and / or "obtaining reconstructed height maps based on reconstructed meshes", and / or "program initialization" can be implemented individually as an embodiment of a physical environment visualization method; or, the above embodiments can be combined to jointly implement an embodiment of a physical environment visualization method, and this disclosure does not limit this embodiment.

[0183] Next, the visualization method of the physical environment disclosed herein will be explained through specific implementation scenarios.

[0184] In some embodiments, program initialization is performed, including pre-setting a data buffer for receiving discrete mesh data and GPU data exchange, and initializing the heightmap texture size and image format for subsequent reception of heightmap data.

[0185] In some embodiments, height map data and discrete grid data are received from the server of the ADD domain at a certain frequency and processed, including data decompression, data coordinate transformation and image position correction.

[0186] For example, a communication connection is established with the server at a fixed frequency of once every 100 milliseconds to continuously receive compressed data packets containing heightmap data and discrete grid data. Upon receiving the compressed data packets, the underlying protocol header is stripped to extract the payload. Subsequently, a highly efficient decompression algorithm is used to process the payload, restoring it to a structured original data matrix.

[0187] For example, the data information of the decompressed heightmap and the data information of the discrete mesh are based on the global coordinate system or the projection coordinate system specified by the server, which cannot be directly matched with the physical environment and coordinate system that are dynamically changing in real time from the vehicle itself. Through a spatial rigid body transformation algorithm, each pixel in the decompressed heightmap and each 3D coordinate point in the discrete mesh are mapped from the global coordinate system or the projection coordinate system specified by the server to the relative coordinate system of the vehicle itself, thus completing spatial alignment.

[0188] For example, after the coordinate transformation is completed, in order to eliminate the errors caused by differences in sensor installation positions and high-frequency chassis vibration during vehicle operation, the coordinate data is multiplied by a pre-calibrated projection matrix, and the three-dimensional coordinate points in the three-dimensional space where the physical environment is located are projected onto the pixel positions of the height map and the reconstructed height map.

[0189] For example, such as Figure 12 As shown, the image of the parsed and corrected height map is sampled from low to high along the height direction. After sampling, the obtained sampling information is mapped onto a single planar grid to obtain the contour information of the physical environment around the OCC, which is expressed in the form of multiple height layer slices.

[0190] exist Figure 12The leftmost image shows two color mapping rules. White squares correspond to RGB values ​​of 255 / 255 / 255, and black squares correspond to 0 / 0 / 0, representing the occupancy status of the physical environment. White indicates that the physical environment is occupied by one or more objects, while black indicates that the physical environment is not occupied by one or more objects. The image slightly to the left of the middle shows the process of discretizing the 3D space where the physical environment is located along the vertical height direction. The preset height range is 1.8 meters, divided into 19 height layers. The different gray values ​​of the gradient curve represent the height characteristics of the physical environment occupied by objects. The black and white image on the far right is a slice image of the 19 height layers generated by projecting the sampling information of each height layer onto a 2D plane, used to represent the surface texture and edge contour features of the physical environment. The height layer slice image at the sampling height of the 19th layer does not have a white outline because at each horizontal sampling position at the sampling height of the 19th layer, the physical environment is not occupied by one or more objects.

[0191] The original height layer slice image contains noise or jagged edges. By introducing a Gaussian blur image processing algorithm, the original height layer slice image is smoothed to generate multiple height layer slice images, thereby improving the overall robustness and smoothness of the final generated OCC rendering result while ensuring computational efficiency.

[0192] In some embodiments, the received discrete grid data information is used to obtain the correlation between discrete grids through a preset algorithm and generate a combined grid. A reconstructed height map is obtained by observing this combined grid through a top-down observation camera. This reconstructed height map is used to express the associated contour information of the surrounding physical environment.

[0193] In some embodiments, the union of multiple height layer slice images and reconstructed height images is calculated to obtain a completed contour information height image. Finally, the completed contour information height image is rendered as a texture onto the corresponding reconstructed mesh, thereby expressing the complete contour information of the physical environment.

[0194] like Figure 13 As shown in (a), for example, within a local perception area of ​​20m × 20m centered on a vehicle, the three-dimensional coordinates of one or more objects in the physical environment are identified by clustering, edge extraction, and geometric algorithm processing of raw point cloud data collected by multiple sensors. The geometric outlines of one or more objects are delineated by luminous lines and dot matrix highlights.

[0195] like Figure 13As shown in (b), for example, after obtaining the three-dimensional coordinates of one or more objects, the three-dimensional coordinates of the one or more objects are sorted in layers along the vertical height direction. This can also be understood as mapping the three-dimensional coordinates of one or more objects to each height layer and converting the three-dimensional coordinates of one or more objects into an ordered list of coordinates on each height layer.

[0196] like Figure 13 As shown in (c), for example, an ordered list of coordinates for each height layer is plotted in a 200×200 pixel, four-channel two-dimensional image matrix. This can also be understood as mapping the sampling information of each height layer onto individual two-dimensional pixel images. Each two-dimensional pixel image is then overlaid onto the surface of a two-dimensional planar grid, generating multiple textured planar grids carrying the sampling information of each height layer. White pixels represent grid cells occupied by obstacles at each height layer, while large areas of black background represent free passage space. This operation of converting the coordinates of the physical environment's three-dimensional space into pixel coordinates reduces the complexity of data processing.

[0197] like Figure 13 As shown in (d), for example, the two-dimensional pixel images corresponding to each height layer are stacked in the channel dimension and fused to form a three-dimensional feature tensor. The spatial resolution (width × height) of this tensor is 4096 × 4096 pixels, and its channel dimension sequentially carries the sampling information of each height layer expressed by multiple height layer slices.

[0198] like Figure 13 As shown in (e), contour fusion is performed on multiple height layer slices and reconstructed height maps to obtain a complete contour information height map; the visualization display result is rendered based on the contour information height map.

[0199] In some embodiments, a reconstructed mesh carrying a heightmap texture with contour information is rendered onto a hardware screen via a hardware-level graphics application programming interface (API).

[0200] like Figure 14As shown, for example, the rendered patch on the hardware screen uses a bird's-eye view or orthographic projection perspective to fully display the physical environment in which the vehicle is located. This minimizes perspective distortion and provides a global view for the driver or autonomous driving system. The vehicle, located in the center of the OCC rendering result, is rendered with high brightness and clear structural details, creating self-distinguishability from its surroundings. A black and white striped area extends in front of the vehicle to represent a zebra crossing or stop line; a dark gray rectangular plane connects to the bottom of the vehicle. These elements represent the geometry of the current driving path, lane occupancy, and the specified road conditions ahead.

[0201] The vehicles adjacent to the left and right of the vehicle are rendered as three-dimensional cuboid models with thickness through continuous grayscale gradients. One or more objects or pedestrians in the upper left corner are rendered as smooth, realistic geometric shapes. Rendering based on the contour information height map transforms the originally flat detection boxes into solid entities with a sense of actual volume, enhancing the intuitiveness of obstacle avoidance warnings.

[0202] In addition, objects that are closer to the vehicle and have a direct impact on driving safety are brighter and have clearer edges; while objects that are farther away or at the edge of the field of vision are relatively blurry or in shadow, which meets the needs of the vehicle system for efficient and interference-free environmental perception.

[0203] In this disclosure, by fusing the outlines of multiple height layer slices and reconstructed height maps for integrated rendering, it can present the internal details and geometric accuracy of a specified horizontal section. At the same time, through the surface reconstruction of discrete meshes, it preserves the overall spatial structure and vertical continuity of the physical environment, achieving a smooth transition between different height layers. This avoids the discontinuities or holes that occur when rendering multiple height layer slices separately, and improves the coherence of the visualization.

[0204] The visualization method of the physical environment provided in this disclosure is illustrated by way of example while the vehicle is in motion. For example, when the vehicle is in motion, the program initialization operation is first performed, including pre-setting two types of data buffers. One type of buffer is used to continuously receive data information from the discrete mesh, and the other type is a dedicated buffer for GPU data exchange, completing the initial configuration of height map texture size and four-channel image format.

[0205] For example, the vehicle controller establishes a long connection with the ADD domain server at a fixed communication frequency of once every 100 milliseconds, and receives two types of data packets: height map data and discrete grid data. The controller then performs preprocessing, which includes one or more processing steps such as data packet decompression, multi-dimensional coordinate transformation, and image position deviation correction.

[0206] For example, a data packet sent by the ADD domain is received every 100 milliseconds. Based on the data packet, the payloads corresponding to the height map and the discrete grid are extracted. The payloads corresponding to the height map and the discrete grid are processed respectively to restore the original data including the original pixel matrix of the height map and / or the coordinate matrix of the three-dimensional discrete grid.

[0207] For example, the coordinate system corresponding to the original data output from the ADD domain is mapped one-to-one with the vehicle's own relative coordinate system to achieve alignment between the perceived data and the physical space where the vehicle is located.

[0208] For example, after the coordinate mapping is completed, since the vehicle is in motion, there is a spatial error caused by the continuous shaking of the vehicle during driving. The coordinate points of the obstacles in the three-dimensional space are projected to the corresponding pixel positions of the height map, thereby eliminating the image misalignment problem caused by shaking and outputting the height map after parsing and correction.

[0209] For example, to address the need for real-time performance when the vehicle is in motion, there are many dynamic obstacles, and real-time performance is a priority, the obtained height map image is sampled in layers from low to high along the vertical height direction from the ground.

[0210] For example, the total height range is preset to 1.8 meters, and it is evenly divided into 19 independent height layers. The obstacle occupancy status of each pixel position is determined layer by layer. The sampling results of each layer are individually mapped to an independent planar grid to generate 19 height layer slice images. In this way, the OCC three-dimensional contour of the driving conditions around the vehicle is characterized layer by layer.

[0211] Optionally, a Gaussian blur image processing algorithm can be used to smooth the height layer slice image, thereby improving the stability and smoothness of the OCC rendered image when the vehicle is in motion.

[0212] For example, the contours are extracted from 19 height layer slice images and the contours are combined to obtain the completed contour information height map.

[0213] For example, a 20m × 20m perception range is defined with the vehicle as the center to obtain a three-dimensional coordinate matrix of one or more obstacles, including vehicles, pedestrians, and guardrails, within the road conditions. The three-dimensional coordinate matrix is ​​then sorted in layers along the height direction, and the three-dimensional points in the matrix are mapped to 19 height layers to generate a coordinate list corresponding to the three-dimensional points in each layer. The coordinate list is filled into a two-dimensional image matrix, and the two-dimensional images are stacked along the channels to form a three-dimensional feature tensor. By fusing the contours in the height layer slices, a complete contour information height map of the vehicle in motion is generated.

[0214] For example, the GPU hardware's underlying graphics API is called to input a multi-layer planar mesh carrying contour texture into the rendering pipeline and output the visual rendering result to the vehicle's central control hardware screen.

[0215] Optionally, the visualization rendering result adopts an orthogonal bird's-eye view. The vehicle model is in the center of the screen, and black and white stripes are drawn to represent lane stop lines and zebra crossings in the direction of vehicle movement. The dark gray plane at the bottom represents the road surface where the vehicle is driving. The adjacent vehicles on the left and right are rendered as grayscale three-dimensional cuboids, and the distance is distinguished by the rendering brightness, thereby weakening the interference.

[0216] An exemplary description is provided of the method for visualizing the physical environment provided in this disclosure while the vehicle is parked. For example, when a vehicle enters a parking state, a discrete mesh receiving buffer and a GPU data exchange buffer are pre-allocated, and the height map texture size and four-channel image format are initialized to meet the data storage requirements for obtaining height layer slices and 3D mesh reconstruction.

[0217] For example, the vehicle controller establishes a long connection with the ADD domain server at a fixed communication frequency of once every 100 milliseconds, and receives two types of data packets: height map data and discrete grid data. The controller then performs preprocessing, which includes one or more processing steps such as data packet decompression, multi-dimensional coordinate transformation, and image position deviation correction.

[0218] For example, a data packet sent by the ADD domain is received every 100 milliseconds. Based on the data packet, the payloads corresponding to the height map and the discrete grid are extracted. The payloads corresponding to the height map and the discrete grid are processed respectively to restore the original data including the original pixel matrix of the height map and / or the coordinate matrix of the three-dimensional discrete grid.

[0219] For example, the original parking space perception data output by the ADD domain is based on the global geodetic coordinate system, which cannot match the dynamic local coordinate system caused by the frequent movement of the vehicle body during parking. By uniformly mapping the pixels of the height map and the three-dimensional coordinate points of the discrete grid to the vehicle body relative coordinate system, the spatial alignment of one or more elements including parking spaces, ground locks, walls, and pillars is completed.

[0220] For example, after completing the coordinate mapping, the calibration projection matrix is ​​retrieved to compensate for the positioning error caused by one or more factors, including the slight tilt of the vehicle body and the distortion of the field of view of the close-range sensor when moving the vehicle at low speed. The three-dimensional coordinates of the obstacles in the parking space are projected to the corresponding pixel positions of the height map, and the parsed and corrected height map is output.

[0221] For example, the system slices the corrected height map, samples and maps layer by layer to generate multiple height layer slice maps, and enables the discrete mesh reconstruction process as the main link for parking rendering.

[0222] For example, to meet the requirement of complete and continuous obstacle surfaces in parking scenarios, the discrete mesh data of the parking space is input into a preset algorithm to generate complete triangular patches and merge them into a continuous combined mesh; a top-down observation camera is set up to render a single reconstructed height map of the continuous combined mesh in a single rendering. This reconstructed height map fully represents the continuous and smooth three-dimensional contour of one or more obstacles, including parking space walls, parking locks, and pillars.

[0223] For example, the contour union of the obtained multiple height layer slice images and the reconstructed height map output by discrete mesh reconstruction is obtained, and the contour features of the two types of data are fused to obtain the contour information height map of the completed parking space. The fused contour has the ability to distinguish the layer heights and the details of the continuous surface.

[0224] For example, a 20m x 20m parking perception range is defined with the vehicle as the center, and the three-dimensional coordinates of one or more obstacles, including parking space boundaries, low-profile parking locks, and pillars, are identified. On the one hand, the three-dimensional coordinates are sorted in layers to generate an ordered list of coordinates for each height layer, which is then filled into a 200x200 pixel two-dimensional image matrix and stacked into a 4096x4096 three-dimensional feature tensor to obtain multi-layer height slice contours. On the other hand, a continuous combined mesh is generated based on the discrete point matrix using a preset algorithm, and the reconstructed height map is rendered. Finally, the two types of image contours are fused to output a height map of parking space contour information with complete details.

[0225] For example, the combined mesh carrying the completed contour texture is input into the rendering pipeline through the GPU's underlying graphics API, generating a visual rendering result and outputting it to the vehicle's central control screen.

[0226] Optionally, the visualization rendering result adopts an orthogonal projection bird's-eye view to show the parking space around the vehicle, with a 3D model of the vehicle displayed in the center of the screen; the dark gray plane at the bottom of the vehicle represents the parking reference ground; the black and white stripe textures on the front and sides mark the parking space stop line and / or parking space boundary; one or more obstacles, including the walls on both sides of the parking space, the parking lock, and adjacent parked vehicles, are rendered as 3D geometry; and the rendering brightness is used to distinguish the distance, thereby weakening the interference.

[0227] It should be noted that the various parameters, algorithms, and visualization rendering effects, including various sizes, durations, number of layers, resolutions, and distances, in the above embodiments are all exemplary adjustable configurations used to clearly illustrate the complete implementation logic of this disclosure, and can be flexibly changed according to vehicle model, sensing hardware specifications, and driving conditions.

[0228] Figure 15A flowchart illustrating a method for visualizing a physical environment provided by an exemplary embodiment of this disclosure is shown. First, the general outline information of obstacles in the surrounding physical environment is expressed using multiple height layer slice images. Then, using a preset algorithm and discrete grid data information, the correlation between discrete grids is obtained. A reconstructed height map is generated using the correlation between discrete grids. The multiple height layer slice images and the reconstructed height map are then fused to obtain a complete outline information height map.

[0229] like Figure 15 As shown, the steps of this method are as follows: Step 1501: Receive height map data.

[0230] In some embodiments, the height map data includes the height values ​​of the physical environment at each horizontal sampling position on the horizontal reference plane within a preset horizontal sampling range. The height values ​​are used to characterize whether the physical environment at each horizontal sampling position on the horizontal reference plane is occupied by one or more objects, and when it is occupied by one or more objects, the height of one or more objects.

[0231] The horizontal reference plane is a preset horizontal reference plane that serves as a reference for determining the height of one or more objects. Each horizontal sampling position is a coordinate position obtained by dividing the horizontal sampling range according to the preset sampling accuracy of the height map.

[0232] Step 1502: Perform layering processing on the height map.

[0233] For example, based on the effective detection capability of the sensor and the driving scenario, a specific height range is pre-defined. Within the pre-defined height range, a series of horizontal cross-sectional positions are determined to define each height layer.

[0234] Step 1503: Determine the sampling information for each height layer.

[0235] For example, by traversing each pixel of the height map at each sampling height, the relationship between the height value stored in each pixel and each sampling height is determined, thereby determining whether each horizontal sampling position at each height layer intersects with the object surface.

[0236] Step 1504: Attach the two-dimensional pixel image carrying the sampling information of each height layer to the surface of each two-dimensional planar grid.

[0237] For example, a two-dimensional pixel image is created for each height layer, where each pixel position uniquely corresponds to each horizontal sampling position. Then, the sampling information of each height layer is encoded into the grayscale value or color channel value of each pixel, transforming the sampling information of each height layer into intuitively visualized two-dimensional pixel images.

[0238] For example, each two-dimensional planar grid corresponding to each height layer is constructed, the horizontal range of each two-dimensional planar grid is consistent with the horizontal sampling range of each height layer, and the height of each two-dimensional planar grid is the same as the sampling height corresponding to each height layer.

[0239] For example, each two-dimensional pixel image is attached to the surface of each two-dimensional planar grid to generate multiple textured planar grids that carry sampling information of each height layer.

[0240] Step 1505: Render each textured planar mesh into multiple height layer slices.

[0241] For example, multiple textured planar meshes are received, and the color, brightness, and transparency of the multiple textured planar meshes are calculated based on the camera view, lighting model, and material settings, thereby rendering the multiple textured planar meshes to obtain multiple height layer slice images.

[0242] Step 1506: Obtain data information from the discrete grid.

[0243] In some embodiments, the data information of the discrete mesh includes data information of multiple three-dimensional coordinate points and multiple facets used to generate the discrete mesh, and the data information of the three-dimensional coordinate points and facets of the discrete mesh is used to characterize the spatial position and geometry of one or more objects.

[0244] Step 1507: Use a preset algorithm to reconstruct the surface of the discrete mesh to obtain the reconstructed mesh.

[0245] In some embodiments, surface reconstruction refers to converting a discrete mesh into a continuous geometric surface that can be rendered by the graphics pipeline, based on the environmental state quantization parameters contained within each voxel unit. Smooth geometric surfaces are constructed at the boundaries of occupied and idle regions using an isosurface extraction algorithm to reconstruct the outer surface of one or more objects.

[0246] Step 1508: Render the reconstructed mesh as a reconstructed heightmap.

[0247] For example, within a preset horizontal reference plane, a regularly distributed array of sampling points is generated at a preset resolution. A detection ray is emitted vertically upwards along each horizontal sampling position, and the intersection point with the triangular facet of the reconstructed mesh is calculated. The three-dimensional spatial coordinates of the first intersection point are calculated, and its Z component is extracted as the height value of each horizontal sampling position, thereby obtaining the height value of the reconstructed mesh.

[0248] For example, the height values ​​of the acquired reconstructed grid are normalized and converted into standardized grayscale or pseudo-color values ​​according to a preset numerical mapping relationship. The converted grayscale or pseudo-color values ​​are then arranged in spatial order according to the horizontal sampling positions to form a two-dimensional raster image, i.e., a reconstructed height map.

[0249] Step 1509: Determine a contour information height map that is continuous and complete.

[0250] For example, for each pixel location in multiple height layer slice images and the reconstructed height map, if valid contour information exists at the pixel location in any input image, the valid contour information is retained; based on the processing result of each pixel location, a contour information height map with continuous and complete contours is determined.

[0251] For example, for each pixel location in multiple height layer slices and the reconstructed height map, the corresponding valid contour information in each input map is obtained; a dynamic confidence score is assigned to the valid contour information of each pixel location, and the dynamic confidence score is determined based on at least one of the following: the preset source weights of each input map, the consistency between the valid contour information corresponding to each pixel location in each input map, and the reasonableness of the height gradient formed by the valid contour information of each pixel location and its neighboring high-confidence pixel locations; the valid contour information is weighted and fused based on the dynamic confidence score to obtain the preliminary contour information of each pixel location and its associated comprehensive confidence.

[0252] Among them, the height gradient rationality refers to judging whether the height change between the effective contour information of each pixel position and the high-confidence pixel position in the neighborhood is within a preset reasonable threshold range based on the spatial continuity constraints of the physical environment; if the height change exceeds the reasonable threshold, the gradient rationality of the effective contour information is judged to be low.

[0253] For example, using preliminary contour information as data constraints and comprehensive confidence level as the basis for constraint strength, an energy function containing data terms and smoothing terms is constructed; through numerical optimization algorithms, a set of pixel height values ​​that make the energy function achieve a global or local minimum is found, thereby obtaining a spatially continuous and edge-preserving optimized contour information height map.

[0254] The data term is used to make the optimized contour information approximate the preliminary contour information, and the degree of approximation is positively correlated with the overall confidence level. The smoothing term is used to constrain the height difference between adjacent pixel positions, and the strength of the constraint varies with the size of the height difference. When the height difference is less than a preset significance threshold, the smoothing term applies a stronger constraint to eliminate noise and ensure spatial continuity. When the height difference is greater than or equal to the significance threshold, the constraint applied by the smoothing term is weakened to characterize the significant height changes of the edges of the physical environment.

[0255] For example, pixel regions with a comprehensive confidence level below a preset threshold in the optimized contour information height map are identified as regions to be repaired. From the boundary neighborhood of the region to be repaired, pixels with a comprehensive confidence level above a preset reliability threshold are selected, and their effective contour information is used as geometric constraint points. Based on the spatial coordinates and height values ​​of the selected geometric constraint points, the interpolated height corresponding to each pixel position in the region to be repaired is calculated. A continuous surface covering the region to be repaired is generated through surface fitting. The region to be repaired is resampled on the continuous surface, and the sampled values ​​are used as the final contour information of the corresponding pixel positions, thereby generating a contour information height map with continuous and complete contours.

[0256] Step 1510: Render the contour information height map onto the reconstructed mesh to generate a visual display result.

[0257] The numbers in each step of the above flowchart are used to uniquely identify and reference each step, and do not represent the order of time or importance of the steps. This disclosure does not impose any restrictions on this.

[0258] Alternatively, this disclosure also includes other implementations of methods for visualizing the physical environment: Render multiple height layer slices to generate a visual display result. Optionally, when rendering each of the multiple height-layer slices, a virtual observation camera is set in front of the vehicle. This camera defines the viewing angle and projection relationship of the three-dimensional space in which the physical environment is located. For each pixel in the three-dimensional space, a virtual line of sight originating from the observation camera and entering the three-dimensional space is calculated in reverse. When this virtual line of sight first intersects with the visualization result reconstructed from the multiple height-layer slices, the spatial height of the intersection point is the line-of-sight intersection height.

[0259] For example, by comparing the vertical distance between the height of the line-of-sight intersection and the sampling heights corresponding to multiple height layer slices, the transparency weight of each height layer in the OCC rendering result is dynamically calculated. Height layers with greater differences are displayed more opaquely at the pixel where the intersection is located, and have a higher weight; height layers with smaller differences are displayed more transparently at the pixel where the intersection is located, and have a lower weight, until they are completely transparent to the next layer or background. Through pixel-by-pixel transparency blending calculation based on physical line-of-sight depth differences, a visually continuous and smoothly transitioning OCC rendering result is synthesized.

[0260] Optionally, all height layer slices are input into the GPU as multiple 2D texture images. In the vertex shader or fragment shader, bilinear or trilinear interpolation algorithms are used to continuously sample each coordinate in 3D space, calculate smooth height values, and then render.

[0261] Figure 16 A block diagram of a physical environment visualization device 1600 provided in an exemplary embodiment of the present disclosure is shown. The physical environment visualization device 1600 includes: The receiving module 1610 is used to receive data information of the height map representing the physical environment occupancy information and data information of the discrete grid; The layering module 1620 is used to perform layering processing along the height direction; based on the data information of the height map, multiple height layer slice maps are obtained to represent the sampling information of each height layer; The reconstruction module 1630 is used to perform surface reconstruction on the discrete mesh based on the data information of the discrete mesh to obtain the reconstructed mesh; based on the reconstructed mesh, a reconstructed height map is obtained to express the geometric undulation information of the physical environment. The fusion module 1640 is used to perform contour fusion on multiple height layer slice images and reconstructed height images to obtain a complete contour information height image; The rendering module 1650 is used to render the visualization results based on the contour information height map.

[0262] In some embodiments, the fusion module 1640 is further configured to retain valid contour information for each pixel location in the multiple height layer slice maps and the reconstructed height map if valid contour information exists at the pixel location in any input map.

[0263] In some embodiments, the rendering module 1650 is further configured to render the contour information height map as a texture onto the reconstructed mesh; and to perform rendering based on the reconstructed mesh to generate image data corresponding to the contour information height map as a visualization display result.

[0264] In some embodiments, the layering module 1620 is further configured to preset a height range for layering processing along the height direction; divide the height range into various sampling heights according to a fixed or adaptive step size; and perform image sampling on the height map at each height layer corresponding to each sampling height.

[0265] In some embodiments, the layering module 1620 is further configured to determine, based on the data information of the height map, whether the physical environment is occupied by one or more objects at each horizontal sampling position of each height layer, and to determine the sampling information of each height layer; map the sampling information of each height layer onto each two-dimensional pixel image, wherein the two-dimensional pixel image is a set of pixels arranged in the form of a two-dimensional grid, and the coordinates of each pixel on each two-dimensional pixel image correspond to the coordinates of each horizontal sampling position of each height layer; construct each two-dimensional planar grid corresponding to each height layer, wherein the horizontal range of each two-dimensional planar grid is consistent with the horizontal sampling range of each height layer, and the height of each two-dimensional planar grid is the same as the sampling height corresponding to each height layer; attach each two-dimensional pixel image to the surface of each two-dimensional planar grid to generate multiple textured planar grids carrying the sampling information of each height layer; and render the multiple textured planar grids to obtain multiple height layer slice images used to express the sampling information of each height layer.

[0266] In some embodiments, the reconstruction module 1630 is further configured to determine, based on the data information of the discrete mesh, multiple three-dimensional coordinate points and multiple facets for generating the discrete mesh; A preset algorithm is used to determine the connection relationship between multiple three-dimensional coordinate points and the belonging relationship between multiple three-dimensional coordinate points and multiple patches; Based on the connection relationship between multiple 3D coordinate points and the belonging relationship between multiple 3D coordinate points and multiple patches, multiple adjacent patches are constructed with multiple 3D coordinate points as vertices. The multiple patches are continuously connected to form a closed 3D surface, thus obtaining the reconstructed mesh.

[0267] In some embodiments, the reconstruction module 1630 is further configured to set a downward-looking observation camera above the reconstruction grid; acquire the height value of the reconstruction grid at each horizontal sampling position on the horizontal reference plane through the observation camera; and obtain a reconstruction height map based on the height value of the reconstruction grid to express the geometric undulation information of the physical environment; wherein the height value of the reconstruction grid is used to characterize the geometric undulation information of the physical environment.

[0268] In some embodiments, the device 1600 further includes a processing module 1660, configured to, before receiving the height map data information and the discrete mesh data information, configure at least one data buffer for data exchange with the graphics processor, and initialize the texture size and image format of the height map according to preset parameters.

[0269] In some embodiments, the device 1600 is applied to an in-vehicle display terminal for real-time rendering and presentation of the vehicle's surrounding environment.

[0270] In some embodiments, the specific physical environment visualization method refers to the following in the above method side embodiments: "contour fusion of multiple height layer slices and reconstructed height maps", and / or, "rendering visualization results based on contour information height maps", and / or, "layering the height map to obtain multiple height layer slices", and / or, "surface reconstruction of discrete meshes", and / or, "obtaining reconstructed height maps based on reconstructed meshes", and / or, "program initialization", and / or, Figures 2 to 15 The content in [the document / article].

[0271] In this disclosure, by fusing the outlines of multiple height layer slices and reconstructed height maps for integrated rendering, it can present the internal details and geometric accuracy of a specified horizontal section. At the same time, through the surface reconstruction of discrete meshes, it preserves the overall spatial structure and vertical continuity of the physical environment, achieving a smooth transition between different height layers. This avoids the discontinuities or holes that occur when rendering multiple height layer slices separately, and improves the coherence of the visualization.

[0272] It should be noted that the specific limitations of the embodiments of the physical environment visualization device provided above can be found in the limitations of the physical environment visualization method above, and will not be repeated here. Each module of the above device can be implemented entirely or partially by software, hardware, or a combination thereof. Each module can be embedded in the processor of the computer device in hardware form or independent of the processor, or it can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0273] Figure 17 A schematic diagram of a physical environment visualization device 1700 provided in an exemplary embodiment of this disclosure is shown. This physical environment visualization device 1700 can be used to perform the method steps in the above embodiments. The physical environment visualization device 1700 includes: a processor 1701, an input / output device 1702, a memory 1703, and a touchscreen 1704.

[0274] The processor 1701 includes one or more processing cores, and the processor 1701 executes various functional applications and information processing by running software programs and modules.

[0275] The memory 1703 can be connected to the processor 1701 and the input / output device 1702.

[0276] The memory 1703 can be used to store a computer program executed by the processor, and the processor 1701 is used to execute the computer program to implement the various steps in the above method embodiments.

[0277] Furthermore, the memory 1703 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, including but not limited to: magnetic disks or optical disks, electrically erasable programmable read-only memory, erasable programmable read-only memory, static on-demand memory, read-only memory, magnetic memory, flash memory, and programmable read-only memory.

[0278] For details not described in this embodiment, please refer to the method-side embodiment above, which will not be repeated here.

[0279] This disclosure also provides a non-transitory computer-readable storage medium storing a computer program for execution by a processor to implement a visualization method of the physical environment executed by the aforementioned electronic device. In some embodiments, the non-transitory computer-readable storage medium may include ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0280] This disclosure also provides a chip, which includes programmable logic circuits and / or program instructions, for implementing the visualization method of the physical environment described above when the chip is running.

[0281] This disclosure also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor reads from the computer-readable storage medium and executes the computer program to implement the above-described method for visualizing the physical environment.

[0282] It should be understood that the term "instruction" mentioned in the embodiments of this disclosure can be a direct instruction, an indirect instruction, or an indication of a relationship. For example, A instructing B can mean that A directly instructs B, such as B being able to obtain information through A; it can also mean that A indirectly instructs B, such as A instructing C, so B can obtain information through C; or it can mean that there is a relationship between A and B.

[0283] In the description of the embodiments of this disclosure, the term "correspondence" may indicate a direct or indirect correspondence between two things, or an association between two things, or a relationship of instruction and being instructed, configuration and being configured, etc.

[0284] In some embodiments of this disclosure, "predefined" can be achieved by pre-storing corresponding codes, tables, or other means of indicating relevant information in the device (e.g., including sensing signal receiving nodes and sensing signal transmitting nodes). This disclosure does not limit the illustrative implementation methods. For example, "predefined" can refer to what is defined in a protocol.

[0285] In some embodiments of this disclosure, "protocol" may refer to standard protocols in the field of communications, such as LTE protocols, NR protocols, and related protocols applied to future communication systems, and this disclosure does not limit it.

[0286] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0287] In this article, "greater than or equal to" can mean greater than or equal to, and "less than or equal to" can mean less than or equal to.

[0288] Furthermore, the step numbers described herein are merely illustrative of one possible order of execution between the steps. In some other embodiments, the steps may not be executed in the order of their numbers, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the figures. This disclosure does not limit this practice.

[0289] Those skilled in the art will recognize that the functions described in the embodiments of this disclosure in one or more of the foregoing examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0290] The above are merely exemplary embodiments of this disclosure and are not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A method for visualizing a physical environment, characterized in that, The method includes: Receive data information of a height map representing the physical environment occupancy information and data information of a discrete grid; Layering is performed along the height direction; based on the data information of the height map, multiple height layer slice maps are obtained to express the sampling information of each height layer; Based on the data information of the discrete mesh, surface reconstruction is performed on the discrete mesh to obtain a reconstructed mesh; based on the reconstructed mesh, a reconstructed height map is obtained to express the geometric undulation information of the physical environment; Contour fusion is performed on the multiple height layer slices and the reconstructed height map to obtain a complete contour information height map; The visualization result is rendered based on the contour information height map.

2. The method according to claim 1, characterized in that, The step of contour fusion of the multiple height layer slices and the reconstructed height map to obtain a completed contour information height map includes: For each pixel position in the plurality of height layer slices and the reconstructed height map, if there is valid contour information at the pixel position in any input map, then the valid contour information is retained; Based on the processing results of each pixel position, a contour information height map that is continuous and complete is determined.

3. The method according to claim 1, characterized in that, The rendering of the visualization result based on the contour information height map includes: The contour information height map is rendered as a texture onto the reconstructed mesh; Rendering is performed based on the reconstructed mesh to generate image data corresponding to the contour information height map, which is then used as the visualization result.

4. The method according to claim 1, characterized in that, The layering process along the height direction includes: A preset height range for the layering process along the height direction; The height range is divided into various sampling heights according to a fixed or adaptive step size; Image sampling is performed on the height map at each height layer corresponding to each sampling height.

5. The method according to claim 1, characterized in that, Based on the data information of the height map, multiple height layer slice maps are obtained to represent the sampling information of each height layer, including: Based on the data information of the height map, determine whether the physical environment is occupied by one or more objects at each horizontal sampling position of each height layer, and determine the sampling information of each height layer; The sampling information of each height layer is mapped onto each two-dimensional pixel image. The two-dimensional pixel image is a set of pixels arranged in a two-dimensional grid. The coordinates of each pixel on each two-dimensional pixel image correspond to the coordinates of each horizontal sampling position of each height layer. Construct two-dimensional planar grids corresponding to each height layer, wherein the horizontal range of each two-dimensional planar grid is consistent with the horizontal sampling range of each height layer, and the height of each two-dimensional planar grid is the same as the sampling height corresponding to each height layer; Each two-dimensional pixel image is attached to the surface of each two-dimensional planar grid to generate multiple textured planar grids that carry the sampling information of each height layer; The multiple textured planar meshes are rendered to obtain multiple height layer slice images used to express the sampling information of each height layer.

6. The method according to claim 1, characterized in that, The process of reconstructing a surface from the discrete mesh based on the data information of the discrete mesh to obtain a reconstructed mesh includes: Based on the data information of the discrete mesh, data information for multiple three-dimensional coordinate points and multiple facets used to generate the discrete mesh is determined; A preset algorithm is used to determine the connection relationship between the plurality of three-dimensional coordinate points and the attribution relationship between the plurality of three-dimensional coordinate points and the plurality of facets; Based on the connection relationship between the multiple three-dimensional coordinate points and the belonging relationship between the multiple three-dimensional coordinate points and the multiple facets, the multiple facets that are adjacent to each other are constructed with the multiple three-dimensional coordinate points as vertices. The multiple facets are continuously connected to form a closed three-dimensional surface, thus obtaining the reconstructed mesh.

7. The method according to claim 1, characterized in that, The process of obtaining a reconstructed height map based on the reconstructed mesh to represent the geometric undulation information of the physical environment includes: A downward-looking observation camera is positioned above the reconstructed grid; The observation camera acquires the height values ​​of the reconstructed grid at various horizontal sampling positions on the horizontal reference plane. Based on the height values ​​of the reconstructed mesh, a reconstructed height map is obtained to represent the geometric undulation information of the physical environment.

8. The method according to claim 1, characterized in that, The method further includes: Before receiving the height map data information and the discrete grid data information, at least one data buffer for data exchange with the graphics processor is configured to receive the height map data information and the discrete grid data information; The texture size and image format of the height map are initialized according to preset parameters.

9. The method according to any one of claims 1 to 8, characterized in that, The method is applied to in-vehicle display terminals to render and present the vehicle's surrounding environment in real time.

10. A visualization device for a physical environment, characterized in that, The device includes: The receiving module is used to receive data information of the height map representing the physical environment occupancy information and data information of the discrete grid; The layering module is used to perform layering processing along the height direction, and based on the data information of the height map, obtain multiple height layer slice maps to express the sampling information of each height layer; The reconstruction module is used to perform surface reconstruction on the discrete mesh based on the data information of the discrete mesh to obtain a reconstructed mesh; and based on the reconstructed mesh, to obtain a reconstructed height map to express the geometric undulation information of the physical environment. The fusion module is used to perform contour fusion on the multiple height layer slice images and the reconstructed height map to obtain a complete contour information height map; The rendering module is used to render a visual display result based on the contour information height map.

11. The apparatus according to claim 10, characterized in that, The fusion module is further configured to retain the valid contour information if, for each pixel position in the plurality of height layer slice images and the reconstructed height image, there is valid contour information in any input image at the pixel position; Based on the processing results of each pixel position, a contour information height map that is continuous and complete is determined.

12. A vehicle, characterized in that, The vehicle includes: an environmental perception sensor, an on-board processor, and an on-board memory; the environmental perception sensor is used to collect image data of the vehicle's surrounding environment, and the on-board memory stores a computer program; the computer program is loaded and executed by the on-board processor to implement the physical environment visualization method as described in any one of claims 1 to 9.

13. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the method for visualizing the physical environment as described in any one of claims 1 to 9.

14. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores a computer program that is loaded and executed by a processor to implement the method for visualizing the physical environment as described in any one of claims 1 to 9.

15. A computer program product, characterized in that, The computer program product stores a computer program that is loaded and executed by a processor to implement the method for visualizing the physical environment as described in any one of claims 1 to 9.

Images