Point cloud frame and depth map generation method and device, electronic equipment and storage medium

Abstract

A point cloud frame and depth map generation method and device, electronic equipment and storage medium are disclosed. The method comprises: obtaining map point cloud data and preset point cloud interception parameters, processing the map point cloud data according to the preset point cloud interception parameters to obtain an intermediate matrix, the intermediate matrix comprising at least the following attribute information: point cloud three-dimensional coordinates, point cloud depth values, generating a virtual laser radar point cloud frame and a virtual laser radar depth map according to the point cloud three-dimensional coordinates and the point cloud depth values in the intermediate matrix. The embodiment of the present application obtains an intermediate matrix by processing map point cloud data according to preset point cloud interception parameters, and simultaneously generates a virtual laser radar point cloud frame and its corresponding virtual laser radar depth map according to the intermediate matrix, thereby simplifying the generation process of the virtual laser radar point cloud frame and the depth map, and greatly improving the generation efficiency and real-time performance.

Description

Technical Field

[0001] This invention relates to the field of lidar technology, and in particular to a method, apparatus, electronic device, and storage medium for generating point cloud frames and depth maps. Background Technology

[0002] LiDAR is a crucial sensing device in technologies and products for autonomous driving and unmanned operations. Virtual LiDAR point cloud frames and corresponding depth maps generated from point cloud maps provide important references for applications such as drivable area perception, obstacle perception, object perception, and localization in autonomous driving or unmanned operations. Currently, the generation of virtual LiDAR point cloud frames and the generation of depth maps from virtual LiDAR point cloud frames are processed separately, resulting in poor generation efficiency and real-time performance. Summary of the Invention

[0003] This invention provides a method, apparatus, electronic device, and storage medium for generating point cloud frames and depth maps. The method processes map point cloud data according to preset point cloud extraction parameters to obtain an intermediate matrix, and simultaneously generates virtual LiDAR point cloud frames and their corresponding virtual LiDAR depth maps based on the intermediate matrix. This simplifies the generation process of virtual LiDAR point cloud frames and depth maps, and significantly improves generation efficiency and real-time performance.

[0004] According to one aspect of the present invention, a method for generating point cloud frames and depth maps is provided, the method comprising:

[0005] Acquire map point cloud data and preset point cloud capture parameters;

[0006] The map point cloud data is processed according to the preset point cloud extraction parameters to obtain an intermediate matrix. The intermediate matrix includes at least the following attribute information: point cloud 3D coordinates and point cloud depth value.

[0007] Virtual LiDAR point cloud frames and virtual LiDAR depth maps are generated based on the 3D coordinates and depth values ​​of the point cloud in the intermediate matrix.

[0008] According to another aspect of the present invention, a point cloud frame and depth map generation apparatus is provided, the apparatus comprising:

[0009] The data acquisition module is used to acquire map point cloud data and preset point cloud capture parameters;

[0010] The intermediate matrix determination module is used to process map point cloud data according to preset point cloud truncation parameters to obtain an intermediate matrix. The intermediate matrix includes at least the following attribute information: point cloud 3D coordinates and point cloud depth values.

[0011] The point cloud frame and depth map generation module is used to generate virtual LiDAR point cloud frames and virtual LiDAR depth maps according to the point cloud 3D coordinates and point cloud depth values ​​in the intermediate matrix.

[0012] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0013] At least one processor; and

[0014] A memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the point cloud frame and depth map generation method according to any embodiment of the present invention.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the point cloud frame and depth map generation method according to any embodiment of the present invention.

[0017] The technical solution of this invention involves acquiring map point cloud data and preset point cloud truncation parameters, processing the map point cloud data according to the preset point cloud truncation parameters to obtain an intermediate matrix. The intermediate matrix includes at least the following attribute information: point cloud 3D coordinates and point cloud depth values. Virtual LiDAR point cloud frames and virtual LiDAR depth maps are generated based on the point cloud 3D coordinates and point cloud depth values ​​in the intermediate matrix. This invention simplifies the generation process of virtual LiDAR point cloud frames and depth maps by processing map point cloud data according to preset point cloud truncation parameters and simultaneously generating virtual LiDAR point cloud frames and their corresponding virtual LiDAR depth maps based on the intermediate matrix, thus significantly improving generation efficiency and real-time performance.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1This is a flowchart of a point cloud frame and depth map generation method provided in Embodiment 1 of the present invention;

[0021] Figure 2 This is a flowchart of a point cloud frame and depth map generation method according to Embodiment 2 of the present invention;

[0022] Figure 3 This is a flowchart of a point cloud frame and depth map generation method provided in Embodiment 3 of the present invention;

[0023] Figure 4 This is an example diagram of a depth map generated from a real lidar point cloud frame according to Embodiment 3 of the present invention;

[0024] Figure 5 This is an example diagram of a depth map generated from a virtual lidar point cloud frame according to Embodiment 3 of the present invention;

[0025] Figure 6 This is an example diagram of another depth map generated from a virtual lidar point cloud frame according to Embodiment 3 of the present invention;

[0026] Figure 7 This is an example diagram of another depth map generated from a real lidar point cloud frame according to Embodiment 3 of the present invention;

[0027] Figure 8 This is an example diagram of another depth map generated from a virtual lidar point cloud frame according to Embodiment 3 of the present invention;

[0028] Figure 9 This is a schematic diagram of a point cloud frame and depth map generation device according to Embodiment 4 of the present invention;

[0029] Figure 10 This is a schematic diagram of the structure of an electronic device that implements the point cloud frame and depth map generation method of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Example 1

[0033] Figure 1 This is a flowchart illustrating a method for generating point cloud frames and depth maps according to Embodiment 1 of the present invention. This embodiment is applicable to situations where virtual LiDAR point cloud frames and virtual LiDAR depth maps are generated using map point cloud data. This method can be executed by a point cloud frame and depth map generation device, which can be implemented in hardware and / or software. Figure 1 As shown in the figure, the point cloud frame and depth map generation method provided in this embodiment includes the following steps:

[0034] S110. Obtain map point cloud data and preset point cloud capture parameters.

[0035] Map point cloud data refers to the data source used to generate virtual LiDAR point cloud frames and their corresponding virtual LiDAR depth maps. This data can originate from LiDAR point cloud data in autonomous driving, unmanned operations, or other scenarios. Preset point cloud extraction parameters can be understood as pre-configured parameters used to extract point cloud data within a certain range from the map point cloud data. These parameters may include: the virtual LiDAR's horizontal field of view, vertical field of view, orientation of the field of view's central axis, nearest point distance limit, and farthest point distance limit, etc.

[0036] In this embodiment of the invention, map point cloud data for generating virtual LiDAR point cloud frames and their corresponding virtual LiDAR depth maps can first be obtained from a map point cloud database stored locally or on a cloud server. Additionally, preset point cloud truncating parameters for generating virtual LiDAR point cloud frames and their corresponding virtual LiDAR depth maps can be obtained from a preset configuration file or a preset configuration table. This allows for the subsequent generation of virtual LiDAR point cloud frames and their corresponding virtual LiDAR depth maps based on the map point cloud data and the preset point cloud truncating parameters. It is understood that this embodiment of the invention does not limit the storage location of the map point cloud data and the preset point cloud truncating parameters. Furthermore, the preset point cloud truncating parameters can be configured according to the type of virtual LiDAR. For example, the parameter indicating the orientation of the field of view's central axis is only required for solid-state LiDAR types of virtual LiDAR, while other types of virtual LiDAR do not require this parameter. The above examples are merely illustrative, and this embodiment of the invention does not impose specific limitations.

[0037] S120. Process map point cloud data according to preset point cloud extraction parameters to obtain an intermediate matrix. The intermediate matrix shall include at least the following attribute information: point cloud 3D coordinates and point cloud depth value.

[0038] The intermediate matrix can refer to an intermediate data structure designed in this embodiment of the invention to simultaneously generate virtual lidar point cloud frames and their corresponding virtual lidar depth maps. The intermediate matrix can include at least the following attribute information: point cloud three-dimensional coordinates and point cloud depth values.

[0039] In this embodiment of the invention, in order to simultaneously generate virtual LiDAR point cloud frames and their corresponding virtual LiDAR depth maps using map point cloud data, an intermediate data structure, namely an intermediate matrix, is designed. The intermediate matrix can include at least the following attribute information: point cloud 3D coordinates and point cloud depth values. The methods for processing map point cloud data according to preset point cloud truncation parameters to obtain the intermediate matrix can include, but are not limited to, the following: based on the field of view of the virtual LiDAR, all point clouds within a certain spatial range can be truncated from the map point cloud data using preset point cloud truncation parameters. All truncated point clouds are then first converted to the virtual LiDAR coordinate system and then mapped to the already created intermediate matrix. The corresponding position; alternatively, preset point cloud extraction parameters can be used to filter the field of view in the map point cloud data, and then the point cloud after the field of view filtering is converted to a pre-configured virtual LiDAR coordinate system. The coordinate-transformed point cloud is then mapped to the corresponding position of the already created intermediate matrix. Considering that the coordinate-transformed point cloud lacks occlusion relationships between multiple objects, the visibility of the above point cloud can be further filtered during the element filling process of the intermediate matrix. That is, when multiple point clouds are mapped to the same position in the intermediate matrix, only the point cloud data with the smallest point cloud depth value is retained. After all point clouds in the virtual LiDAR coordinate system have undergone mapping operations, the required intermediate matrix can be obtained. It should be understood that the above implementation method is only an example. For example, the type of virtual LiDAR coordinate system used, the mapping rules for mapping point clouds to the intermediate matrix, etc., can all be configured according to the actual situation. It is only necessary to ensure that the generated intermediate matrix includes at least the attribute information of the point cloud's three-dimensional coordinates and point cloud depth value. Of course, the intermediate matrix can also include attribute information such as the point cloud's reflection intensity, color, and normal vector. This embodiment of the invention does not limit this.

[0040] S130. Generate virtual lidar point cloud frames and virtual lidar depth maps according to the point cloud 3D coordinates and point cloud depth values ​​in the intermediate matrix.

[0041] In this context, a virtual LiDAR point cloud frame refers to a set of point clouds obtained from map point cloud data. Generally, a virtual LiDAR point cloud frame can be a subset of the map point cloud data. A virtual LiDAR depth map refers to a two-dimensional depth map generated from map point cloud data. The pixel value of each pixel in the virtual LiDAR depth map represents the depth value of the corresponding point cloud. The size of the virtual LiDAR depth map can be the same as the size of the intermediate matrix.

[0042] In this embodiment of the invention, to facilitate the direct extraction of the corresponding virtual LiDAR depth map from the intermediate matrix, after obtaining the intermediate matrix, a virtual LiDAR depth map of the same size can be created according to the number of rows and columns of the intermediate matrix. Then, the 3D coordinates of the point cloud corresponding to each element in the intermediate matrix are stored in the virtual LiDAR point cloud frame. Finally, the point cloud depth values ​​corresponding to each element in the intermediate matrix are filled into the corresponding positions of the virtual LiDAR depth map, thus obtaining the required virtual LiDAR point cloud frame and the corresponding virtual LiDAR depth map. Furthermore, since point cloud data is discrete and sparse, when mapping the point cloud to the intermediate matrix and then projecting it onto the virtual LiDAR depth map, the depth map will inevitably contain many empty spaces. Therefore, a hole-filling operation can be performed on the obtained virtual LiDAR depth map to further increase its visibility and realism. The hole-filling method is not limited to: filling using the average depth value of the nearest neighbor pixels, or filling using the minimum depth value of the nearest neighbor pixels, etc.

[0043] The technical solution of this invention involves acquiring map point cloud data and preset point cloud truncation parameters, processing the map point cloud data according to the preset point cloud truncation parameters to obtain an intermediate matrix. The intermediate matrix includes at least the following attribute information: point cloud 3D coordinates and point cloud depth values. Virtual LiDAR point cloud frames and virtual LiDAR depth maps are generated based on the point cloud 3D coordinates and point cloud depth values ​​in the intermediate matrix. This invention simplifies the generation process of virtual LiDAR point cloud frames and depth maps by processing map point cloud data according to preset point cloud truncation parameters and simultaneously generating virtual LiDAR point cloud frames and their corresponding virtual LiDAR depth maps based on the intermediate matrix, thus significantly improving generation efficiency and real-time performance.

[0044] Example 2

[0045] Figure 2 This is a flowchart of a point cloud frame and depth map generation method provided in Embodiment 2 of the present invention. It is further optimized and extended based on the above embodiments and can be combined with various optional technical solutions in the above embodiments. For example... Figure 2 As shown in the figure, the point cloud frame and depth map generation method provided in this embodiment includes the following steps:

[0046] S210. Obtain map point cloud data and preset point cloud capture parameters.

[0047] In this embodiment of the invention, the preset point cloud capture parameters may include at least one of the following: preset coordinate origin, preset virtual lidar orientation, virtual lidar horizontal field of view, virtual lidar vertical field of view, virtual lidar horizontal angular resolution, virtual lidar vertical angular resolution, virtual lidar maximum visible distance, and virtual lidar minimum visible distance.

[0048] S220. Use the preset coordinate origin in the preset point cloud cropping parameters as the coordinate origin of the virtual lidar coordinate system, and use the preset virtual lidar orientation in the preset point cloud cropping parameters as the positive X-axis direction of the virtual lidar coordinate system to establish the virtual lidar coordinate system.

[0049] The preset coordinate origin can refer to the origin of a pre-configured virtual LiDAR coordinate system. It can be any point cloud location in the map point cloud data, chosen as the preset origin based on actual needs. The preset virtual LiDAR orientation can be understood as the orientation of the central axis of the pre-configured virtual LiDAR field of view. It can also be any central axis orientation selected from the map point cloud data, chosen as the preset virtual LiDAR orientation. The virtual LiDAR coordinate system can be a three-dimensional coordinate system established based on the desired preset coordinate origin and preset virtual LiDAR orientation.

[0050] In this embodiment of the invention, a preset coordinate origin and a preset virtual lidar orientation related to the virtual lidar coordinate system can be extracted from preset point cloud extraction parameters. Then, the preset coordinate origin and the preset virtual lidar orientation are used as the coordinate origin and positive X-axis direction of the virtual lidar coordinate system, respectively, to establish the corresponding virtual lidar coordinate system. The selection of the preset coordinate origin and the preset virtual lidar orientation can be based on actual needs, selecting a point cloud position and the orientation of the central axis of a field of view in the map point cloud data as the preset coordinate origin and the preset virtual lidar orientation.

[0051] S230. Transform all point clouds in the map point cloud data to the virtual lidar coordinate system to obtain the first virtual point cloud set.

[0052] The first virtual point cloud set can refer to the virtual point cloud set obtained after coordinate system transformation of all point clouds in the map point cloud data.

[0053] In this embodiment of the invention, after establishing the virtual lidar coordinate system, coordinate system transformation can be performed on all point clouds in the map point cloud data, and all point clouds can be expressed in the virtual lidar coordinate system and denoted as the first virtual point cloud set.

[0054] S240. According to the virtual lidar horizontal field of view, virtual lidar vertical field of view, virtual lidar horizontal angular resolution, virtual lidar vertical angular resolution, virtual lidar maximum visible distance, and virtual lidar minimum visible distance in the preset point cloud extraction parameters, the field of view of the point cloud in the first virtual point cloud set is filtered to obtain the second virtual point cloud set.

[0055] The second virtual point cloud set can refer to the virtual point cloud set obtained after filtering the point clouds in the first virtual point cloud set by the field of view.

[0056] In this embodiment of the invention, the virtual lidar horizontal field of view, virtual lidar vertical field of view, virtual lidar horizontal angular resolution, virtual lidar vertical angular resolution, virtual lidar maximum visible distance, and virtual lidar minimum visible distance related to the virtual lidar can be extracted from the preset point cloud truncation parameters first. Then, the field of view of all point clouds in the first virtual point cloud set is filtered using the above preset point cloud truncation parameters, and point clouds that exceed the field of view are removed. The point cloud set after the field of view filtering is used as the second virtual point cloud set.

[0057] Furthermore, based on the above embodiments of the invention, S240 specifically includes the following steps:

[0058] S2401. Determine the distance from each point cloud in the first virtual point cloud set to the origin of the coordinate system.

[0059] S2402. If the distance is greater than the farthest visible distance of the virtual lidar, or if the distance is less than the nearest visible distance of the virtual lidar, then the corresponding point cloud in the first virtual point cloud set is removed.

[0060] S2403. If the angle between the vector from each point cloud to the origin and the OXZ coordinate plane of the virtual lidar coordinate system is greater than half of the horizontal field of view of the virtual lidar, or if the angle between the vector from each point cloud to the origin and the OXZ coordinate plane of the virtual lidar coordinate system is less than the negative of half of the horizontal field of view of the virtual lidar, then the corresponding point cloud in the first set of virtual point clouds shall be removed.

[0061] S2404. If the angle between the vector from each point cloud to the origin and the OXY coordinate plane of the virtual lidar coordinate system is greater than half of the vertical field of view of the virtual lidar, or if the angle between the vector from each point cloud to the origin and the OXY coordinate plane of the virtual lidar coordinate system is less than the negative of half of the vertical field of view of the virtual lidar, then the corresponding point cloud in the first virtual point cloud set is removed.

[0062] S2405. The first virtual point cloud set after elimination is used as the second virtual point cloud set.

[0063] Specifically, S2401 to S2404 can be executed sequentially on all point clouds in the first virtual point cloud set to filter out point clouds located within a certain field of view, and the point cloud set after being filtered by the field of view can be used as the second virtual point cloud set.

[0064] S250. Use the preset depth map coordinate mapping formula to determine the depth map row coordinates and depth map column coordinates corresponding to each point cloud in the second virtual point cloud set.

[0065] The preset depth map coordinate mapping formula can refer to a pre-configured formula used to determine the depth map row coordinates and depth map column coordinates corresponding to the point cloud.

[0066] In this embodiment of the invention, a preset depth map coordinate mapping formula can be pre-configured on the electronic device to determine the depth map row coordinates and depth map column coordinates corresponding to each point cloud in the second virtual point cloud set. For example, the preset depth map coordinate mapping formula can be expressed as follows:

[0067]

[0068]

[0069] Where, r i α represents the depth map coordinates corresponding to the i-th point cloud; i represents the point cloud number; i RES represents the angle between the vector from the i-th point cloud to the origin and the OXZ coordinate plane of the virtual lidar coordinate system; H Indicates the horizontal angular resolution of the virtual lidar; Half_FOV H This represents half of the horizontal field of view of the virtual lidar; c i β represents the depth map coordinates corresponding to the i-th point cloud; i RES represents the angle between the vector from the i-th point cloud to the origin and the OXY coordinate plane of the virtual lidar coordinate system; V Indicates the vertical angular resolution of the virtual lidar; Half_FOV V This represents half of the vertical field of view of the virtual lidar.

[0070] S260. Use the ratio of the virtual lidar's horizontal field of view to its horizontal angular resolution as the number of rows in the intermediate matrix, and the ratio of the virtual lidar's vertical field of view to its vertical angular resolution as the number of columns in the intermediate matrix. Create the intermediate matrix based on the number of rows and columns.

[0071] In this embodiment of the invention, the number of rows and columns of the intermediate matrix can be determined using the virtual lidar horizontal field of view, virtual lidar vertical field of view, virtual lidar horizontal angular resolution, and virtual lidar vertical angular resolution in the preset point cloud cropping parameters. Specifically, the ratio of the virtual lidar horizontal field of view to the virtual lidar horizontal angular resolution is used as the number of rows in the intermediate matrix, and the ratio of the virtual lidar vertical field of view to the virtual lidar vertical angular resolution is used as the number of columns in the intermediate matrix. The corresponding intermediate matrix is ​​then created based on the determined number of rows and columns.

[0072] S270. Using preset field-of-view occlusion conditions and depth map row coordinates and depth map column coordinates, map the corresponding point cloud in the second virtual point cloud set to the intermediate matrix.

[0073] The preset view occlusion condition can be understood as a pre-configured judgment condition for determining whether there is a view occlusion relationship between each point cloud in the second virtual point cloud set. The preset view occlusion condition can be used to filter the visibility of the point clouds in the second virtual point cloud set. For example, the preset view occlusion condition may include whether the point cloud depth value exceeds a preset distance threshold, or whether the point cloud depth value is the minimum value, etc.

[0074] In this embodiment of the invention, since the point clouds in the second virtual point cloud set after the field of view screening lack the occlusion relationship between multiple objects, the generated virtual LiDAR point cloud frame and its corresponding virtual LiDAR depth map also lack the corresponding occlusion relationship between multiple objects, which seriously reduces the visibility and realism of the virtual LiDAR point cloud frame and its corresponding virtual LiDAR depth map. The technical solution of this embodiment of the invention can use the preset field of view occlusion conditions and the point cloud depth value of each point cloud to perform visibility screening on all point clouds in the second virtual point cloud set, remove point clouds that do not meet the preset field of view occlusion conditions, and map the corresponding point cloud to the corresponding position of the intermediate matrix according to the depth map row coordinates and depth map column coordinates of each point cloud determined in S250.

[0075] Furthermore, based on the above embodiments of the invention, S270 specifically includes the following steps:

[0076] S2701. The distance from each point cloud in the second virtual point cloud set to the origin of the coordinate system is taken as the point cloud depth value of the corresponding point cloud.

[0077] S2702. When the point cloud depth value meets the preset field-of-view occlusion condition, the depth map row coordinates and depth map column coordinates are used as the position index of the intermediate matrix, and the point cloud 3D coordinates and point cloud depth values ​​corresponding to each point cloud are filled into the corresponding positions of the intermediate matrix according to the corresponding position index.

[0078] In this embodiment of the invention, considering the occlusion relationship between multiple objects and the possibility that multiple point clouds may be mapped to the same position in the intermediate matrix, the visibility of all point clouds in the second virtual point cloud set can be filtered using preset view occlusion conditions and the point cloud depth values ​​of each point cloud. The filtering strategy is to retain only the corresponding point cloud with the smallest point cloud depth value, and then use the depth map row coordinates and depth map column coordinates corresponding to that point cloud as the position index of the intermediate matrix. The point cloud 3D coordinates and point cloud depth values ​​corresponding to that point cloud are filled into the corresponding positions of the intermediate matrix according to the corresponding position index.

[0079] It's important to understand that point cloud maps are created by stitching together a large number of LiDAR point cloud frames collected from various locations and orientations. This provides a near-complete representation of the scene's spatial structure, eliminating the issue of missing point cloud data for objects behind them due to occlusion from multiple objects in a single field of view. In existing technologies, when generating virtual LiDAR point cloud frames and their corresponding virtual LiDAR depth maps from map point clouds, the point cloud is typically cropped directly based on the field of view, without considering occlusion relationships between objects. Therefore, objects behind objects occluded by objects in the actual scene will also appear in the generated virtual LiDAR point cloud frames and their corresponding depth maps, resulting in "distortion" and severely reducing visibility and realism. The technical solution of this invention, after filtering all point clouds in the map point cloud data based on their field of view, adds a visibility filtering step for the cropped point clouds, significantly increasing the visibility and realism of the subsequently generated virtual LiDAR point cloud frames and their corresponding depth maps.

[0080] S280. Create a virtual LiDAR depth map of the same size as the number of rows and columns of the intermediate matrix.

[0081] In this embodiment of the invention, in order to facilitate the direct extraction of the corresponding virtual LiDAR depth map from the intermediate matrix, virtual LiDAR depth maps of the same size can be created according to the number of rows and columns of the intermediate matrix.

[0082] S290. Store the three-dimensional coordinates of the point cloud corresponding to each element in the intermediate matrix into the virtual LiDAR point cloud frame, fill the corresponding position of the virtual LiDAR depth map with the point cloud depth value corresponding to each element in the intermediate matrix, and perform a hole filling operation on the virtual LiDAR depth map.

[0083] In this embodiment of the invention, each element in the intermediate matrix can be separated into two dimensions: point cloud 3D coordinates and point cloud depth values. Specifically, when an element in the intermediate matrix is ​​non-empty, the point cloud 3D coordinates corresponding to that element are stored in the virtual LiDAR point cloud frame. Furthermore, the point cloud depth values ​​corresponding to that element are filled into the corresponding positions of the virtual LiDAR depth map according to the corresponding depth map row coordinates and depth map column coordinates. In addition, after filling the virtual LiDAR depth map with point cloud depth values, a hole-filling operation can be performed on it to further increase the visibility and realism of the virtual LiDAR depth map.

[0084] Furthermore, based on the above embodiments of the invention, a hole-filling operation is performed on the virtual lidar depth map, specifically including the following steps:

[0085] A. Obtain the nearest neighbor pixels of each pixel in the virtual LiDAR depth map, and determine the minimum pixel value among the nearest neighbor pixels;

[0086] B. If the pixel value of a pixel in the virtual LiDAR depth map is empty, then fill the corresponding pixel value with the minimum pixel value.

[0087] C. If the pixel value of a pixel in the virtual LiDAR depth map is less than or equal to the minimum pixel value, then the pixel value of the corresponding pixel will not be modified.

[0088] D. If the pixel value of a pixel in the virtual LiDAR depth map is greater than the minimum pixel value, then the pixel value of the corresponding pixel will be filled with the minimum pixel value.

[0089] It's important to understand that because point cloud data is discrete and sparse, it inevitably has many gaps when projected onto a 2D depth map. Furthermore, during the visibility filtering of the point cloud in S270, there might be instances where, although background objects are occluded, their long field of view and the sparse point cloud mean that a few background object points might still be missed on the foreground objects. Therefore, in S290, after filling the virtual LiDAR depth map with point cloud depth values, a gap-filling operation can be performed, i.e., obtaining the nearest neighbor pixels for each pixel in the virtual LiDAR depth map. For example, using a four-neighbor or eight-neighbor domain, the minimum pixel value (i.e., the minimum point cloud depth value) among each nearest neighbor pixel is calculated. The pixel value of this nearest neighbor pixel is compared with the minimum pixel value. If the pixel value is null, it is filled with the minimum pixel value. If the pixel value is less than or equal to the minimum pixel value, it is left unchanged. If the pixel value is greater than the minimum pixel value, it is filled with the minimum pixel value. After hole filling, the final virtual LiDAR depth map is obtained. This invention's solution utilizes hole operations to fill in gaps in the virtual LiDAR depth map and filter out a small number of background object points that may be missed on foreground objects, resulting in a more visually appealing and realistic virtual LiDAR depth map.

[0090] The technical solution of this invention involves acquiring map point cloud data and preset point cloud cropping parameters. A preset coordinate origin in the preset point cloud cropping parameters is used as the origin of a virtual LiDAR coordinate system. A preset virtual LiDAR orientation in the preset point cloud cropping parameters is used as the positive X-axis direction of the virtual LiDAR coordinate system to establish the virtual LiDAR coordinate system. All point clouds in the map point cloud data are transformed into the virtual LiDAR coordinate system to obtain a first virtual point cloud set. Based on the virtual LiDAR horizontal field of view, virtual LiDAR vertical field of view, virtual LiDAR horizontal angular resolution, virtual LiDAR vertical angular resolution, virtual LiDAR maximum visible distance, and virtual LiDAR minimum visible distance in the preset point cloud cropping parameters, the point clouds in the first virtual point cloud set are filtered by field of view to obtain a second virtual point cloud set. A preset depth map coordinate mapping is then used. The formula determines the row and column coordinates of the depth map corresponding to each point cloud in the second virtual point cloud set. The ratio of the virtual LiDAR's horizontal field of view to its horizontal angular resolution is used as the number of rows in the intermediate matrix, and the ratio of the virtual LiDAR's vertical field of view to its vertical angular resolution is used as the number of columns in the intermediate matrix. An intermediate matrix is ​​created based on the number of rows and columns. Using preset field-of-view occlusion conditions and the row and column coordinates of the depth map, the corresponding point clouds in the second virtual point cloud set are mapped to the intermediate matrix. A virtual LiDAR depth map of the same size is created according to the number of rows and columns of the intermediate matrix. The 3D coordinates of the point clouds corresponding to each element in the intermediate matrix are stored in the virtual LiDAR point cloud frame. The depth values ​​of the point clouds corresponding to each element in the intermediate matrix are filled into the corresponding positions of the virtual LiDAR depth map, and a hole-filling operation is performed on the virtual LiDAR depth map. This invention generates an intermediate matrix by sequentially filtering map point cloud data based on field of view and visibility. Then, based on the point cloud 3D coordinates and depth values ​​of the elements in this intermediate matrix, it simultaneously generates virtual LiDAR point cloud frames and their corresponding virtual LiDAR depth maps, significantly improving generation efficiency and real-time performance. Furthermore, during the generation of the intermediate matrix, occlusion relationships between multiple objects are assessed, greatly increasing the visibility and realism of the generated virtual LiDAR point cloud frames and their corresponding virtual LiDAR depth maps. In addition, after generating the virtual LiDAR depth map, a hole-filling operation is performed, fully considering the impenetrability of LiDAR imaging light. Hole filling and filtering out a small number of background object points that might be missed on foreground objects are performed in a way that more closely conforms to physical characteristics, further enhancing the visibility and realism of the virtual LiDAR depth map.

[0091] Example 3

[0092] Figure 3This is a flowchart of a point cloud frame and depth map generation method provided in Embodiment 3 of the present invention. Based on the above embodiments, this embodiment provides an implementation method for generating point cloud frames and depth maps. It can simultaneously generate virtual LiDAR point cloud frames and their corresponding virtual LiDAR depth maps using map point cloud data by cleverly designing an intermediate data structure—an intermediate matrix. It also considers the occlusion relationships between multiple objects and more reasonable hole-filling operations, resulting in higher visibility and realism of the final generated virtual LiDAR point cloud frames and their corresponding virtual LiDAR depth maps. Figure 3 As shown, the point cloud frame and depth map generation method provided in Embodiment 3 of the present invention specifically includes the following steps:

[0093] S310: Read in map point cloud data and input preset point cloud capture parameters.

[0094] Specifically, map point cloud data can be read in and stored in set M. Then, the map location where virtual LiDAR point cloud frames and virtual LiDAR depth maps need to be generated, i.e., the preset coordinate origin P0(x0,y0,z0), and the preset virtual LiDAR orientation θ, can be input.

[0095] S320. Establish a virtual lidar coordinate system and transform all point clouds in the map point cloud data to the virtual lidar coordinate system to obtain the first virtual point cloud set.

[0096] Specifically, a virtual lidar coordinate system OLxyz is established with the map location, i.e., the preset coordinate origin P0(x0,y0,z0), as the coordinate origin and the preset virtual lidar orientation θ as the positive X-axis. The coordinate system of all point clouds in set M is then transformed to express the first virtual point cloud set ML in the virtual lidar coordinate system OLxyz.

[0097] S330. Perform field of view and visibility filtering on the point clouds in the first virtual point cloud set to generate an intermediate matrix.

[0098] In this embodiment of the invention, the horizontal field of view (FOV) of the virtual lidar can be preset. H Virtual LiDAR Vertical Field of View (FOV) V Horizontal angular resolution (RES) of virtual lidar H RES (Vertical Angular Resolution) of Virtual LiDAR V The maximum visible distance of the virtual lidar is D. max And the closest line-of-sight distance D of the virtual lidar min And set the half-width range: Half_FOV H =FOV H / 2, Half_FOV V =FOVV / 2. Meanwhile, the intermediate matrix T can be designed as follows:

[0099]

[0100] Here, the intermediate matrix T is a three-dimensional data structure, and its elements t ij = (x, y, z, range) represents the three-dimensional coordinates (x, y, z) of the point cloud and the depth value range of the point cloud, respectively.

[0101] Specifically, performing field-of-view and visibility filtering on the point clouds in the first virtual point cloud set ML to generate an intermediate matrix may include the following steps:

[0102] A1. Calculate Pi(x) for all point clouds. i ,y i ,z i The distance Di from the origin P0 of the virtual lidar coordinate system OLxyz is given by the formula: If Di > D max Or, Di <D min Then the corresponding point cloud in the first virtual point cloud set ML will be removed, and the distance Di can also represent the point cloud depth value corresponding to point cloud Pi.

[0103] A2. Calculate Pi(x) for all point clouds. i ,y i ,z i The angle α between the vector OPi (O is the origin of the OLxyz coordinate system) to the origin and the OXZ coordinate plane is... i If α i Half_FOV H , or α i <-Half_FOV H If so, the corresponding point cloud in the first virtual point cloud set ML will be removed.

[0104] A3. Calculate Pi(x) for all point clouds. i ,y i ,z i The angle β between the vector OPi (O is the origin of the OLxyz coordinate system) to the origin and the OXY coordinate plane is... i If β i Half_FOV V , or β i <-Half_FOV V If so, the corresponding point cloud in the first virtual point cloud set ML will be removed.

[0105] A4. Determine the depth map row coordinates and depth map column coordinates corresponding to each point cloud in the second virtual point cloud set using a preset depth map coordinate mapping formula. The preset depth map coordinate mapping formula can be expressed as follows:

[0106]

[0107]

[0108] A5. Update the elements of the intermediate matrix according to the depth map row coordinates and depth map column coordinates corresponding to each point cloud.

[0109] The object to be updated is the element t in the intermediate matrix T rc , that is, map the point cloud Pi(x i , y i , z i ) to the corresponding position in the intermediate matrix T. At the same time, check whether there is an element value stored or updated before for the element t rc . The specific steps are as follows:

[0110] A51. When there is no old value at the corresponding position of the intermediate matrix T, fill the four components of the vector t rc : x = x i , y = y i , z = z i , range = Di;

[0111] A52. When there is an old value at the corresponding position of the intermediate matrix T, judge the size relationship between the point cloud depth value range in the further old value and the point cloud depth value Di of the current point cloud. If range < Di, it means that the previously stored point cloud data is closer to the coordinate origin than the current point cloud, and the current point cloud should be blocked by the previous old point cloud. Therefore, keep the old value at the corresponding position of the intermediate matrix T unchanged; if range > Di, it means that the previously stored point cloud data is farther from the coordinate origin than the current point cloud, and the previous old point cloud should be blocked by the current point cloud. Therefore, use the current point cloud Pi to update the four components of the vector t rc : x = x i , y = y i , z = z i , range = Di.

[0112] In summary, the point clouds in the first virtual point cloud set can be screened for the field of view and visibility to generate an intermediate matrix. The prior art solutions do not consider the occlusion relationship between different front and rear scenes. The technical solution of the embodiment of the present invention uses the designed intermediate matrix T to judge the occlusion relationship, effectively solving the problem of invisibility of the occluded objects in the background.

[0113] S340. Separate the point cloud depth values ​​of the three-dimensional coordinates of each element in the intermediate matrix into virtual lidar point cloud frames and their corresponding virtual lidar depth maps.

[0114] Specifically, if the element t in the intermediate matrix ij If the value is empty, skip that element; if the element t in the intermediate matrix is ​​empty... ij If the value is non-empty, then element t will be... ij The x, y, z components are stored in a virtual lidar point cloud frame L = {l0, l1, ..., l k}, where l k =(x k ,y k ,z k ) represents the k-th spatial point in the point cloud frame L, and element t is... ij The range component is stored in the corresponding position R of the virtual LiDAR depth map R. ij .

[0115] S350, Perform hole filling operation on the virtual lidar depth map.

[0116] In this embodiment of the invention, the pixel R of each point in the virtual LiDAR depth map can be obtained. ij Find the nearest neighbor pixels, such as the four-neighbor or eight-neighbor areas, and calculate the minimum pixel value n among the nearest neighbor pixels. min (i.e., minimum point cloud depth value), the pixel value r of this pixel. ij Compared with the above minimum pixel value n min Compare the pixel values, if the pixel value r of that pixel is... ij If the value is empty, then the pixel value r of the corresponding pixel point will be... ij Fill with the minimum pixel value n min If the pixel value r of this pixel ij Less than or equal to the minimum pixel value n min If the pixel value is r, then the pixel value of the corresponding pixel will not be modified; if the pixel value of the pixel is r ij Greater than the minimum pixel value n min Then the pixel value r of the corresponding pixel point ij Fill with the minimum pixel value n min After filling the holes, the final virtual LiDAR depth map R is obtained. The technical solution of this embodiment of the invention fully considers the impenetrability of light in LiDAR imaging during post-processing of the virtual LiDAR depth map, performs hole filling in a way that is more in line with physical characteristics, and filters out a small number of background object points that may be missed on the foreground object, resulting in a higher visibility and realism of the final virtual LiDAR depth map.

[0117] Figure 4This is an example diagram of a depth map generated from a real LiDAR point cloud frame, provided in Embodiment 3 of the present invention. Figure 4 As can be seen, there are multiple walls in this map scene, and each wall has a strict occlusion relationship with the others. Figure 5 This is an example diagram of a depth map generated from a virtual lidar point cloud frame, provided in Embodiment 3 of the present invention. Figure 5 As shown, when using existing technology to directly extract the corresponding point cloud from map point cloud data to generate a virtual LiDAR depth map, it is obvious that the occlusion relationship between the multiple walls is not correctly represented in the depth map. The point cloud of the back wall will be seen through the front wall, making the virtual LiDAR depth map look messy, which seriously affects its visibility and realism. Figure 6 This is an example diagram of another depth map generated from a virtual LiDAR point cloud frame, provided in Embodiment 3 of the present invention. Figure 6 It can be seen that, when using the technical solution of this invention and without performing a hole-filling operation, the generated virtual lidar depth map exhibits higher visibility and realism compared to... Figure 5 There has been a significant improvement. Figure 7 This is an example diagram of another depth map generated from a real lidar point cloud frame, provided in Embodiment 3 of the present invention. Figure 8 This is an example diagram of another depth map generated from a virtual lidar point cloud frame, provided in Embodiment 3 of the present invention. Figure 7 and Figure 8 The image shows the real LiDAR depth map and the virtual LiDAR depth map obtained after performing the hole filling operation using the complete technical solution of the embodiment of the present invention. It is easy to see that the two are very close. Therefore, it is shown that the virtual LiDAR depth map generated by the technical solution of the embodiment of the present invention has significantly increased visibility and realism compared with the prior art.

[0118] This invention, as an innovative point cloud processing method, belongs to the underlying core technology and can be applied to the perception, positioning, and simulation of autonomous driving and unmanned operations. Furthermore, it can be extended to a series of unmanned vehicle products, such as various unmanned engineering machinery products that require unmanned operation. This invention does not limit these applications.

[0119] For example, the technical solutions of the embodiments of the present invention can have the following application scenarios:

[0120] (1) Application in scenarios of autonomous driving and unmanned operation perception. The virtual LiDAR point cloud frame and virtual LiDAR depth map generated by the embodiments of the present invention can be compared with the real LiDAR point cloud frame and depth map in real time, so as to perceive which targets are inherent targets in the scene map and which are dynamically appearing targets. For example, after comparing the virtual LiDAR point cloud frame and depth map with the real LiDAR point cloud frame and depth map using the perception algorithm, it can be known which targets have been added to the current dynamic scene compared with the high-precision point cloud map (static inherent scene). These additional targets are very likely to be obstacles or newly added operation objects. On the other hand, it can also be clearly known which objects are inherent objects in the scene, such as walls and pillars, and therefore cannot be regarded as operation objects and a safe distance must be maintained.

[0121] (2) Applied to unmanned driving and unmanned operation simulation scenarios. In existing simulation technologies, there is a new technology direction that uses a large amount of real sensor (LiDAR, camera, etc.) data to construct virtual scenes. This invention also provides a method for generating visualization and output data of LiDAR field of view and depth map field of view in such virtual scenes;

[0122] (3) Application in autonomous driving positioning scenarios. In a positioning system based on a high-precision point cloud map, the technical solution of this invention can be used to generate a virtual lidar point cloud field of view and a depth map field of view. By comparing it with the real lidar point cloud field of view and depth map field of view at the current moment, the position and heading angle of the unmanned vehicle on the map at the current moment can be deduced, thereby realizing the positioning of the unmanned vehicle.

[0123] It should be understood that the above application scenarios are merely examples. The technical solutions of this invention, as a point cloud processing method belonging to the underlying core technology, can be applied to all related technical fields, and this invention does not limit them.

[0124] The technical solution of this invention involves reading map point cloud data and inputting preset point cloud extraction parameters to establish a virtual LiDAR coordinate system. All point clouds in the map point cloud data are then converted to the virtual LiDAR coordinate system. The point clouds in the first virtual point cloud set are filtered for field of view and visibility to generate an intermediate matrix. The point cloud depth values ​​of the three-dimensional coordinates of each element in the intermediate matrix are separated into virtual LiDAR point cloud frames and their corresponding virtual LiDAR depth maps. Hole filling operations are then performed on the virtual LiDAR depth maps. The technical solution of this invention, starting from the essence of physical imaging, simplifies the generation of virtual LiDAR point cloud frames and virtual LiDAR depth maps into a single technical processing step using a designed intermediate matrix, significantly improving the efficiency and real-time performance of point cloud processing. Simultaneously, during the generation of the intermediate matrix, occlusion relationships between multiple objects are determined, effectively resolving the issue of the invisibility of occluded objects in the background, greatly increasing the visibility and realism of the generated virtual LiDAR point cloud frames and their corresponding virtual LiDAR depth maps. Furthermore, after generating the virtual LiDAR depth map, a hole-filling operation is performed, fully considering the impermeability of LiDAR imaging light, and using a more physically consistent method to fill holes and filter out a small number of background object points that might be missed on foreground objects, further enhancing the visibility and realism of the virtual LiDAR depth map.

[0125] Example 4

[0126] Figure 9 This is a schematic diagram of a point cloud frame and depth map generation device provided in Embodiment 4 of the present invention. Figure 9 As shown, the device includes:

[0127] The data acquisition module 41 is used to acquire map point cloud data and preset point cloud capture parameters.

[0128] The intermediate matrix determination module 42 is used to process map point cloud data according to preset point cloud truncation parameters to obtain an intermediate matrix. The intermediate matrix includes at least the following attribute information: point cloud three-dimensional coordinates and point cloud depth values.

[0129] The point cloud frame and depth map generation module 43 is used to generate virtual lidar point cloud frames and virtual lidar depth maps according to the point cloud 3D coordinates and point cloud depth values ​​in the intermediate matrix.

[0130] The technical solution of this invention involves acquiring map point cloud data and preset point cloud truncating parameters through a data acquisition module, and processing the map point cloud data according to the preset point cloud truncating parameters to obtain an intermediate matrix. The intermediate matrix includes at least the following attribute information: point cloud 3D coordinates and point cloud depth values. A point cloud frame and depth map generation module generates virtual LiDAR point cloud frames and virtual LiDAR depth maps based on the point cloud 3D coordinates and depth values ​​in the intermediate matrix. This invention simplifies the generation process of virtual LiDAR point cloud frames and depth maps by processing map point cloud data according to preset point cloud truncating parameters to obtain an intermediate matrix, and simultaneously generating virtual LiDAR point cloud frames and their corresponding virtual LiDAR depth maps based on this intermediate matrix, significantly improving generation efficiency and real-time performance.

[0131] Furthermore, based on the above embodiments of the invention, the intermediate matrix determination module 42 includes:

[0132] The coordinate system establishment unit is used to take the preset coordinate origin in the preset point cloud cropping parameters as the coordinate origin of the virtual lidar coordinate system, and to take the preset virtual lidar orientation in the preset point cloud cropping parameters as the positive X-axis direction of the virtual lidar coordinate system to establish the virtual lidar coordinate system.

[0133] The first point cloud set acquisition unit is used to transform all point clouds in the map point cloud data into the virtual lidar coordinate system to obtain the first virtual point cloud set.

[0134] The second point cloud set acquisition unit is used to filter the point cloud in the first virtual point cloud set according to the virtual lidar horizontal field of view, virtual lidar vertical field of view, virtual lidar horizontal angular resolution, virtual lidar vertical angular resolution, virtual lidar maximum visible distance, and virtual lidar nearest visible distance in the preset point cloud extraction parameters to obtain the second virtual point cloud set.

[0135] The coordinate determination unit is used to determine the depth map row coordinates and depth map column coordinates corresponding to each point cloud in the second virtual point cloud set using a preset depth map coordinate mapping formula.

[0136] The intermediate matrix creation unit is used to take the ratio of the virtual lidar's horizontal field of view to its horizontal angular resolution as the number of rows in the intermediate matrix, and the ratio of the virtual lidar's vertical field of view to its vertical angular resolution as the number of columns in the intermediate matrix, and creates the intermediate matrix based on the number of rows and columns.

[0137] The intermediate matrix determination unit is used to map the corresponding point cloud in the second virtual point cloud set to the intermediate matrix using preset field-of-view occlusion conditions and depth map row coordinates and depth map column coordinates.

[0138] Furthermore, based on the above embodiments of the invention, the second cloud collection acquisition unit is specifically used for:

[0139] Determine the distance from the origin of each point cloud in the first virtual point cloud set;

[0140] If the distance is greater than the farthest visible distance of the virtual lidar, or if the distance is less than the nearest visible distance of the virtual lidar, then the corresponding point cloud in the first virtual point cloud set will be removed.

[0141] If the angle between the vector from each point cloud to the origin and the OXZ coordinate plane of the virtual lidar coordinate system is greater than half of the horizontal field of view of the virtual lidar, or if the angle between the vector from each point cloud to the origin and the OXZ coordinate plane of the virtual lidar coordinate system is less than the negative of half of the horizontal field of view of the virtual lidar, then the corresponding point cloud in the first set of virtual point clouds will be removed.

[0142] If the angle between the vector from each point cloud to the origin and the OXY coordinate plane of the virtual lidar coordinate system is greater than half of the vertical field of view of the virtual lidar, or if the angle between the vector from each point cloud to the origin and the OXY coordinate plane of the virtual lidar coordinate system is less than the negative of half of the vertical field of view of the virtual lidar, then the corresponding point cloud in the first set of virtual point clouds will be removed.

[0143] The first set of virtual point clouds after the removal is used as the second set of virtual point clouds.

[0144] Furthermore, based on the above embodiments of the invention, the intermediate matrix determining unit is specifically used for:

[0145] The distance from each point cloud in the second virtual point cloud set to the origin of the coordinate system is taken as the point cloud depth value of the corresponding point cloud.

[0146] When the point cloud depth value meets the preset field-of-view occlusion condition, the depth map row coordinates and depth map column coordinates are used as the position index of the intermediate matrix, and the point cloud 3D coordinates and point cloud depth values ​​corresponding to each point cloud are filled into the corresponding positions of the intermediate matrix according to the corresponding position index.

[0147] Furthermore, based on the above embodiments of the invention, the point cloud frame and depth map generation module 43 includes:

[0148] The depth map creation unit is used to create virtual LiDAR depth maps of the same size according to the number of rows and columns of the intermediate matrix.

[0149] The point cloud frame and depth map generation unit is used to store the three-dimensional coordinates of the point cloud corresponding to each element in the intermediate matrix into the virtual LiDAR point cloud frame, fill the corresponding position of the point cloud depth value corresponding to each element in the intermediate matrix into the virtual LiDAR depth map, and perform hole filling operation on the virtual LiDAR depth map.

[0150] Furthermore, based on the above embodiments of the invention, the point cloud frame and depth map generation unit is also used for:

[0151] Obtain the nearest neighbor pixels of each pixel in the virtual LiDAR depth map, and determine the minimum pixel value among the nearest neighbor pixels;

[0152] If the pixel value of a pixel in the virtual LiDAR depth map is empty, then the pixel value of the corresponding pixel will be filled with the minimum pixel value.

[0153] If the pixel value of a pixel in the virtual LiDAR depth map is less than or equal to the minimum pixel value, the pixel value of the corresponding pixel will not be modified.

[0154] If the pixel value of a pixel in the virtual LiDAR depth map is greater than the minimum pixel value, then the pixel value of the corresponding pixel will be filled with the minimum pixel value.

[0155] Furthermore, based on the above embodiments of the invention, the preset point cloud capture parameters include at least one of the following: preset coordinate origin, preset virtual lidar orientation, virtual lidar horizontal field of view, virtual lidar vertical field of view, virtual lidar horizontal angular resolution, virtual lidar vertical angular resolution, virtual lidar maximum visible distance, and virtual lidar minimum visible distance.

[0156] The point cloud frame and depth map generation device provided in the embodiments of the present invention can execute the point cloud frame and depth map generation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0157] Example 5

[0158] Figure 10 A schematic diagram of an electronic device 50 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0159] like Figure 10As shown, the electronic device 50 includes at least one processor 51 and a memory, such as a read-only memory (ROM) 52 and a random access memory (RAM) 53, communicatively connected to the at least one processor 51. The memory stores computer programs executable by the at least one processor. The processor 51 can perform various appropriate actions and processes based on the computer program stored in the ROM 52 or loaded into the RAM 53 from storage unit 58. The RAM 53 can also store various programs and data required for the operation of the electronic device 50. The processor 51, ROM 52, and RAM 53 are interconnected via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0160] Multiple components in electronic device 50 are connected to I / O interface 55, including: input unit 56, such as keyboard, mouse, etc.; output unit 57, such as various types of monitors, speakers, etc.; storage unit 58, such as disk, optical disk, etc.; and communication unit 59, such as network card, modem, wireless transceiver, etc. Communication unit 59 allows electronic device 50 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0161] Processor 51 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 51 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 51 performs the various methods and processes described above, such as point cloud frame and depth map generation methods.

[0162] In some embodiments, the point cloud frame and depth map generation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 58. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 50 via ROM 52 and / or communication unit 59. When the computer program is loaded into RAM 53 and executed by processor 51, one or more steps of the point cloud frame and depth map generation method described above may be performed. Alternatively, in other embodiments, processor 51 may be configured to perform the point cloud frame and depth map generation method by any other suitable means (e.g., by means of firmware).

[0163] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0164] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0165] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0166] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0167] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0168] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0169] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and no limitation is imposed herein.

[0170] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for generating point cloud frames and depth maps, characterized in that, The method includes: Acquire map point cloud data and preset point cloud capture parameters; The map point cloud data is processed according to the preset point cloud extraction parameters to obtain an intermediate matrix. The intermediate matrix includes at least the following attribute information: point cloud three-dimensional coordinates and point cloud depth value. A virtual lidar point cloud frame and a virtual lidar depth map are generated based on the point cloud 3D coordinates and point cloud depth values ​​in the intermediate matrix. The step of processing the map point cloud data according to the preset point cloud extraction parameters to obtain an intermediate matrix includes: The virtual lidar coordinate system is established by taking the preset coordinate origin in the preset point cloud cropping parameters as the coordinate origin of the virtual lidar coordinate system and taking the preset virtual lidar orientation in the preset point cloud cropping parameters as the positive X-axis direction of the virtual lidar coordinate system. Transform all point clouds in the map point cloud data to the virtual lidar coordinate system to obtain the first virtual point cloud set; According to the virtual lidar horizontal field of view, virtual lidar vertical field of view, virtual lidar horizontal angular resolution, virtual lidar vertical angular resolution, virtual lidar maximum visible distance, and virtual lidar minimum visible distance in the preset point cloud extraction parameters, the point cloud in the first virtual point cloud set is filtered by field of view to obtain the second virtual point cloud set. The depth map row coordinates and depth map column coordinates corresponding to each point cloud in the second virtual point cloud set are determined using a preset depth map coordinate mapping formula. The ratio of the horizontal field of view of the virtual lidar to the horizontal angular resolution of the virtual lidar is used as the number of rows in the intermediate matrix, and the ratio of the vertical field of view of the virtual lidar to the vertical angular resolution of the virtual lidar is used as the number of columns in the intermediate matrix. The intermediate matrix is ​​then created based on the number of rows and the number of columns. Using preset field-of-view occlusion conditions and the depth map row coordinates and depth map column coordinates, the point cloud corresponding to the second virtual point cloud set is mapped to the intermediate matrix.

2. The method according to claim 1, characterized in that, The preset point cloud capture parameters include at least one of the following: preset coordinate origin, preset virtual lidar orientation, virtual lidar horizontal field of view, virtual lidar vertical field of view, virtual lidar horizontal angular resolution, virtual lidar vertical angular resolution, virtual lidar maximum visible distance, and virtual lidar minimum visible distance.

3. The method according to claim 1, characterized in that, The step of filtering the point clouds in the first virtual point cloud set according to the preset point cloud cropping parameters—virtual lidar horizontal field of view, virtual lidar vertical field of view, virtual lidar horizontal angular resolution, virtual lidar vertical angular resolution, virtual lidar maximum visible distance, and virtual lidar minimum visible distance—to obtain a second virtual point cloud set includes: Determine the distance from each point cloud in the first virtual point cloud set to the origin of the coordinate system; If the distance is greater than the farthest visible distance of the virtual lidar, or if the distance is less than the nearest visible distance of the virtual lidar, then the corresponding point cloud in the first virtual point cloud set is removed. If the angle between the vector from each point cloud to the origin and the OXZ coordinate plane of the virtual lidar coordinate system is greater than half of the horizontal field of view of the virtual lidar, or if the angle between the vector from each point cloud to the origin and the OXZ coordinate plane of the virtual lidar coordinate system is less than the opposite of half of the horizontal field of view of the virtual lidar, then the corresponding point cloud in the first set of virtual point clouds is removed. If the angle between the vector from each point cloud to the origin and the OXY coordinate plane of the virtual lidar coordinate system is greater than half of the vertical field of view of the virtual lidar, or if the angle between the vector from each point cloud to the origin and the OXY coordinate plane of the virtual lidar coordinate system is less than the opposite of half of the vertical field of view of the virtual lidar, then the corresponding point cloud in the first set of virtual point clouds is removed. The first set of virtual point clouds after the removal is used as the second set of virtual point clouds.

4. The method according to claim 1, characterized in that, The step of mapping the point cloud corresponding to the second virtual point cloud set to the intermediate matrix using preset view occlusion conditions and the depth map row coordinates and the depth map column coordinates includes: The distance from each point cloud in the second virtual point cloud set to the origin of the coordinate system is taken as the point cloud depth value of the corresponding point cloud; When the point cloud depth value satisfies the preset field-of-view occlusion condition, the depth map row coordinates and the depth map column coordinates are used as the position index of the intermediate matrix, and the point cloud 3D coordinates and point cloud depth values ​​corresponding to each point cloud are filled into the corresponding positions of the intermediate matrix according to the corresponding position index.

5. The method according to claim 1, characterized in that, The step of generating a virtual LiDAR point cloud frame and a virtual LiDAR depth map according to the point cloud 3D coordinates and point cloud depth values ​​in the intermediate matrix includes: Create a virtual LiDAR depth map of the same size according to the number of rows and columns of the intermediate matrix; The three-dimensional coordinates of the point cloud corresponding to each element in the intermediate matrix are stored in the virtual lidar point cloud frame. The point cloud depth values ​​corresponding to each element in the intermediate matrix are filled into the corresponding positions of the virtual lidar depth map, and a hole filling operation is performed on the virtual lidar depth map.

6. The method according to claim 5, characterized in that, The hole-filling operation on the virtual lidar depth map includes: Obtain the nearest neighbor pixel of each pixel in the virtual lidar depth map, and determine the minimum pixel value among the nearest neighbor pixels; If the pixel value of a pixel in the virtual LiDAR depth map is empty, then the pixel value of the corresponding pixel will be filled with the minimum pixel value; If the pixel value of a pixel in the virtual LiDAR depth map is less than or equal to the minimum pixel value, then the pixel value of the corresponding pixel is not modified. If the pixel value of a pixel in the virtual LiDAR depth map is greater than the minimum pixel value, then the pixel value of the corresponding pixel will be filled with the minimum pixel value.

7. A point cloud frame and depth map generation device, characterized in that, The device includes: The data acquisition module is used to acquire map point cloud data and preset point cloud capture parameters; The intermediate matrix determination module is used to process the map point cloud data according to the preset point cloud truncation parameters to obtain an intermediate matrix. The intermediate matrix includes at least the following attribute information: point cloud three-dimensional coordinates and point cloud depth values. The point cloud frame and depth map generation module is used to generate a virtual lidar point cloud frame and a virtual lidar depth map according to the point cloud three-dimensional coordinates and point cloud depth values ​​in the intermediate matrix. The intermediate matrix determination module includes: The coordinate system establishment unit is used to take the preset coordinate origin in the preset point cloud truncation parameters as the coordinate origin of the virtual lidar coordinate system, and to take the preset virtual lidar orientation in the preset point cloud truncation parameters as the positive X-axis direction of the virtual lidar coordinate system to establish the virtual lidar coordinate system. The first point cloud set acquisition unit is used to convert all point clouds in the map point cloud data to the virtual lidar coordinate system to obtain the first virtual point cloud set. The second point cloud set acquisition unit is used to filter the point cloud in the first virtual point cloud set according to the virtual lidar horizontal field of view, virtual lidar vertical field of view, virtual lidar horizontal angular resolution, virtual lidar vertical angular resolution, virtual lidar maximum visible distance and virtual lidar nearest visible distance in the preset point cloud truncation parameters to obtain the second virtual point cloud set. The coordinate determination unit is used to determine the depth map row coordinates and depth map column coordinates corresponding to each point cloud in the second virtual point cloud set using a preset depth map coordinate mapping formula; An intermediate matrix creation unit is used to take the ratio of the horizontal field of view of the virtual lidar to the horizontal angular resolution of the virtual lidar as the number of rows of the intermediate matrix, and the ratio of the vertical field of view of the virtual lidar to the vertical angular resolution of the virtual lidar as the number of columns of the intermediate matrix, and create the intermediate matrix based on the number of rows and the number of columns. The intermediate matrix determination unit is used to map the point cloud corresponding to the second virtual point cloud set to the intermediate matrix using preset field-of-view occlusion conditions and the depth map row coordinates and the depth map column coordinates.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the point cloud frame and depth map generation method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the point cloud frame and depth map generation method according to any one of claims 1-6.

Images