Point cloud rendering method based on laser radar and camera data fusion

Abstract

The point cloud rendering method based on the fusion of laser radar and camera data provided in this application sets corresponding layered partitioning and blocking templates for different types of rendering objects under the constraints of the same camera field of view depth, the same camera angle range, and the same distance between the rendering object and the center of the camera field of view. By directly obtaining the layered partitioning and blocking template that can accommodate the point cloud space body and has the maximum accommodation intersection ratio from the template library for the point cloud space body constructed for the rendering object by the multi-eye vision method, in the subsequent point cloud rendering, it is no longer necessary to perform layering, partitioning, and blocking calculations for the camera field of view area of ​​each frame, which is conducive to improving the point cloud rendering speed. The type and coordinate position of the point cloud incorrectly constructed by the multi-eye vision method are corrected by laser radar, which reduces the error rate of judging whether each block of the subsequent frame relative to the previous frame needs to be re-rendered in steps L3-L4, and the rendering effect of details is increased through two strategies.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a point cloud rendering method based on laser radar and camera data fusion. Background Art

[0002] At present, most point cloud construction methods use vision-based camera construction methods or laser radar-based construction methods. Taking the point cloud construction based on multi-view vision as an example, the principle of the camera-based point cloud construction method is: multiple cameras collect multiple views of the same object, and then calculate the three-dimensional coordinates of each feature point through the geometric relationship of multiple views, and use existing technologies such as curvature analysis, eigenvalue analysis, and normal vector estimation to identify the type of feature points. The principle of laser radar point cloud construction is: through laser ranging and the emission angle of the laser beam, the coordinate position of the feature point in three-dimensional space is calculated, and the type of feature point is also identified by eigenvalue estimation and other methods. Camera-based or radar-based point cloud construction methods have their own advantages and disadvantages. In summary, laser radar has higher penetration and can find feature points that are difficult to find due to visual occlusion and other reasons. Therefore, it has higher point cloud construction accuracy, but the speed of laser acquiring point cloud data is slow, and the richness of the acquired data features is poor, such as it is difficult to capture the texture, color and other details of the feature points. Therefore, the fusion of feature points discovered by laser radar and camera is a hot research direction in this field to improve the accuracy of point cloud construction.

[0003] In the prior art, there are many methods for fusing feature points discovered by lidar and cameras. After completing the point cloud construction, this field focuses on how to improve the rendering speed of the fused point cloud and increase the rendering effect. However, there are many factors that affect the rendering effect of the point cloud, such as the depth of camera field of view, camera angle, the distance between the rendering object and the center of the camera field of view, different requirements for rendering accuracy of different types of rendering objects, and different requirements for rendering accuracy of the scenes where the rendering objects are located. In some large-scale online games, online modeling and other application scenarios that require real-time rendering processing of large amounts of data, in the prior art, these factors that affect the rendering effect, as well as the real-time data of the rendering objects and rendering scenes are usually input into the pre-trained deep learning model, and the model outputs the rendering solution after a series of data analysis through the built-in algorithm. However, the existing method considers too many factors that affect the rendering effect, and the amount of data processed is too large, which greatly sacrifices the rendering speed, which will seriously affect the user experience in scenes with high requirements for rendering response speed.

[0004] In conventional point cloud rendering scenarios, the camera field of view depth, camera angle range, and the distance between the rendered object and the center of the camera field of view are usually fixed. Therefore, changes in the rendering accuracy requirements for the rendered object usually occur when the type of the rendered object or the scene (environment) in which the rendered object is located changes, or when the camera rotates. Therefore, people in this field are looking forward to how to ensure the accuracy of point cloud rendering in this application scenario while avoiding the use of the above-mentioned deep learning model through complex calculation processes for point cloud rendering. And they are looking forward to seeking breakthroughs in the following two technical directions to solve this technical problem.

[0005] 1. Find technical breakthroughs in the process of fusing lidar and camera data to help improve the speed and accuracy of subsequent rendering of the fused point cloud data.

[0006] 2. While avoiding the rendering process that uses deep learning models for complex calculations, find technological breakthroughs in the rendering process to capture more feature details that need to be rendered to improve rendering effects. Summary of the invention

[0007] The present invention aims to find technical breakthroughs in the process of fusing lidar and camera data under constrained application scenarios, and to find technical breakthroughs in the rendering process that avoids using deep learning models, so as to capture more detailed features that need to be rendered and improve rendering effect and rendering speed. A point cloud rendering method based on the fusion of lidar and camera data is provided.

[0008] To achieve this object, the present invention adopts the following technical solutions:

[0009] A point cloud rendering method based on laser radar and camera data fusion is provided, which is applied to a point cloud rendering scene under the constraints of fixed camera field of view depth, camera angle range, and distance between the rendering object and the center of the camera field of view, and the camera is rotated, and / or the type of the rendering object is changed, and / or the scene where the rendering object is located is changed, comprising the steps of:

[0010] L1, construct point cloud based on multi-eye vision method, and obtain the hierarchical partitioning and blocking template with the maximum accommodation intersection ratio from the template library to accommodate the constructed point cloud space volume and place it in The axis divides the three-dimensional space into eight quadrants, and the center point of the internal space of the hierarchical partitioning template is located at The origin of the coordinate system;

[0011] L2, the laser radar verifies the type and coordinate position of the feature points of each point cloud in each block within the hierarchical partition separation template, and corrects the type and coordinate position of the point cloud incorrectly constructed based on the multi-eye vision method;

[0012] L3, real-time monitoring and extraction of singular blocks of the latter frame relative to the former frame in two consecutive frames verified and corrected by the laser radar;

[0013] L4, for the singular blocks of type "the point cloud is newly added but the point cloud position is not offset", the first strategy is used to render the point cloud data; for the singular blocks of type "the point cloud is not newly added but the point cloud position is offset", the second strategy is used to render the point cloud data.

[0014] Preferably, the same hierarchical partitioning and blocking template is used to accommodate the point cloud space volumes respectively constructed by the camera for the rear frame and the front frame in two consecutive frames, each block in the hierarchical partitioning and blocking template has a unique block number, and the hierarchical partitioning and blocking template is used to accommodate the first point cloud space volume generated in the front frame and the second point cloud space volume generated in the rear frame. The position of the hierarchical partitioning and blocking template in the three-dimensional space is fixed and unchanged, and the first point cloud space volume and the second point cloud space volume are used to represent the same rendering object;

[0015] In step L3, the method for identifying the singular block comprises the steps of:

[0016] A1, from the hierarchical partition block template for accommodating the first point cloud spatial volume, obtain a second block having the same number as the first block in the same hierarchical partition block template for accommodating the second point cloud spatial volume, and then further obtain adjacent blocks that are in the same layer as the first block and are adjacent to the first block above, below, left, right, upper left, lower left, upper right, and lower right;

[0017] A2, determining whether the point cloud formed in the second block is offset to at least one of the adjacent blocks;

[0018] If yes, extract the adjacent block that generates the offset, and then determine whether the first block is the singular block according to the offset singular block identification strategy;

[0019] If not, go to step A3;

[0020] A3, determining whether the number of newly added point clouds of the first block compared with the second block exceeds a first threshold, and if so, determining that the first block is the singular block.

[0021] Preferably, in step A2, the method for judging whether the first block is the singular block according to the offset singular block identification strategy is:

[0022] Determine whether the number of the adjacent blocks that generate the offset is greater than a preset second threshold, and if so, determine that the first block is the singular block;

[0023] and / or determining whether a ratio of the number of first point clouds offset in each of the adjacent blocks to the number of second point clouds not offset in the second block is greater than a preset third threshold, and if so, determining that the first block is the singular block;

[0024] And / or determine whether the offset of at least one point cloud in at least one of the adjacent blocks is greater than a preset offset threshold; if so, determine that the first block is the singular block.

[0025] Preferably, in step L4, the method for rendering point cloud data using the first strategy comprises the following steps:

[0026] B1, in the layer where the first block determined to be a singular block is located, extract the blocks arranged adjacent to each other at the upper, lower, left, right, upper left, lower left, upper right, and lower right positions of the first block to form a same-layer block combination with the first block, and in the adjacent layers adjacent to the same-layer block combination, extract the adjacent-layer block combination that overlaps with the same-layer block combination in terms of hierarchical depth;

[0027] B2, using the parameter changes of each block in the same-layer block combination and the adjacent-layer block combination generated in the same subsequent frame as the screening condition, screening out a candidate set from the database;

[0028] B3, selecting from the candidate set a historical block set having the maximum point cloud change similarity with the block set consisting of the same-layer block combination and the adjacent-layer block combination, and then performing block expansion on the historical block set to obtain an expanded set, wherein each block in the block set has an expanded block with the same number in the expanded set;

[0029] B4, from the expanded blocks in the expanded set having the same block number as the first block, further identify a second rendering basis point cloud that has a spatial position corresponding to the newly added point cloud in the first block and has the same point cloud type, and then assign the historical rendering method of the second rendering basis point cloud to the newly added point cloud corresponding to the position.

[0030] Preferably, in step B2, the method for screening out the candidate set comprises the steps of:

[0031] B21, identifying the changed blocks in the same-layer block combination and the adjacent-layer block combination generated in the same subsequent frame, including newly added point cloud blocks, and / or disappeared point cloud blocks, and / or blocks where point clouds are added and disappeared simultaneously, and then obtaining the change parameters of each of the changed blocks, including the spatial coordinates and point cloud type of the newly added point cloud and / or the disappeared point cloud in the changed block, the number of the changed block, and the number of the block where the newly added point cloud and / or the disappeared point cloud landed in the previous frame, and the spatial coordinates of the landing point in the landing block;

[0032] B22, using the change parameter of the change block as a matching condition, matching a second historical block having point cloud change similarity with the change block from the same historical frame in the database, wherein the second historical block and the change block having point cloud change similarity have the same level; each of the second historical blocks at the same level constitutes a historical block combination at the same level, or constitutes a historical block combination at an adjacent level that is adjacent to the historical block combination at the same level;

[0033] B23, adding the historical block set associated with the historical frame consisting of the same-layer historical block combination and the adjacent-layer historical block combination generated in the same historical frame into the candidate set.

[0034] Preferably, the method for determining whether the newly added point cloud in the change block has point cloud change similarity with the historical point cloud in the second historical block is:

[0035] When the first spatial coordinate of the newly added point cloud in the change block is the same as the second spatial coordinate of the historical point cloud in the second historical block or the coordinate deviation distance is less than the preset distance threshold, and the newly added point cloud is of the same type as the historical point cloud, and the number of the change block where the newly added point cloud lands is the same as the number of the second historical block where the historical point cloud lands, and the third spatial coordinate of the newly added point cloud landing point in the previous frame block is the same as the fourth spatial coordinate of the historical point cloud landing point in the historical previous frame block of the historical previous frame or the coordinate deviation distance is less than the preset distance threshold, and the previous frame block has the same number as the historical previous frame block, then it is determined that the newly added point cloud has point cloud change similarity with the historical point cloud;

[0036] If each change point cloud in the change block is matched to a historical point cloud with point cloud change similarity in the second history block, it is determined that the change block has point cloud change similarity with the second history block;

[0037] The number of block pairs formed by the change blocks and the second historical blocks having a point cloud change similarity relationship in the block set and the historical block set is counted as the point cloud change similarity between the block set and the historical block set.

[0038] Preferably, in step L4, the method for rendering point cloud data using the second strategy comprises the steps of:

[0039] C1, extracting rendering reference blocks from the layer where the first block determined as a singular block is located and the adjacent layers adjacent to the layer where the first block is located, and obtaining a unique block number corresponding to each of the rendering reference blocks;

[0040] C2, identifying each offset point cloud in each of the extracted rendering reference blocks, and obtaining offset parameters of each offset point cloud, including: the coordinate position of the offset point cloud after being offset to the rendering reference block, the distance from the initial coordinate position in the previous frame, and the offset point cloud type;

[0041] C3, for each of the rendering reference blocks at the same level, using the offset parameter as a matching condition, matching the first historical blocks respectively corresponding to the point cloud offset similarity in the same historical layer of the same historical frame from the database, and then obtaining the first block numbers respectively corresponding to each of the matched first historical blocks, and obtaining the second block numbers respectively corresponding to each of the rendering reference blocks having the point cloud offset similarity with each of the first historical blocks;

[0042] C4, determining whether the first block number of the first historical block having a point cloud offset similarity relationship is the same as the second block number of the rendering reference block,

[0043] If yes, extract the first historical block having the same block number as the first block from the historical frame as the rendering basis block, and then proceed to step C5;

[0044] If not, it is determined that the rendering basis block cannot be extracted from the historical frame;

[0045] C5, for each remaining point cloud that has not undergone positional offset in the first block, identify a first rendering basis point cloud of the same type and having a spatial position corresponding relationship with each of the remaining point clouds from the rendering basis block, and then assign the rendering method of the first rendering basis point cloud to the remaining point cloud corresponding to the position.

[0046] Preferably, in step C3, the method for judging the point cloud offset similarity between the rendering reference block and the first historical block is:

[0047] When the first offset point cloud in the rendering reference block is of the same type as the second offset point cloud in the first historical block; and the first coordinate position of the first offset point cloud in the rendering reference block is the same as the second coordinate position of the second offset point cloud in the first historical block or the offset distance is less than a preset offset distance threshold; and the absolute value of the difference between the first distance between the first offset point cloud at the first coordinate position and the third coordinate position in the previous frame and the second distance between the second offset point cloud at the second coordinate position and the fourth coordinate position in the previous historical frame is less than a preset difference absolute value threshold, then it is determined that the first offset point cloud and the second offset point cloud have point cloud offset similarity;

[0048] If each first-shifted point cloud in the rendering reference block finds a second-shifted point cloud with point cloud offset similarity in the first historical block, it is determined that the rendering reference block has point cloud offset similarity with the first historical block.

[0049] The present invention has the following beneficial effects:

[0050] 1. Under the constraints of the same camera field of view depth, the same camera angle range, and the same distance between the rendering object and the center of the camera field of view, corresponding layered partitioning and blocking templates are set for different types of rendering objects. By directly obtaining the layered partitioning and blocking template that can accommodate the point cloud space volume and has the maximum accommodation intersection and union ratio from the template library for the point cloud space volume constructed for the rendering object based on the multi-eye vision method, in the point cloud rendering of steps L3-L4, it is no longer necessary to perform layering, partitioning, and blocking calculations for the camera field of view area of ​​each frame, which is conducive to improving the point cloud rendering speed in steps L3-L4. In addition, the acquisition of the layered partitioning and blocking template is based on the constraint condition that the layered partitioning and blocking template can accommodate the point cloud space volume constructed for the corresponding type of rendering object and has the maximum accommodation intersection and union ratio, so that the obtained layered partitioning and blocking template has the least number of blocks when each block size is the same. This reduces the number of judgments on whether each block of the subsequent frame needs to be re-rendered relative to the previous frame for the subsequent rendering process based on blocks.

[0051] 2. In step L2, the type and coordinate position of the point cloud incorrectly constructed by the multi-eye vision method are corrected by laser radar, which reduces the error rate of judging whether each block of the subsequent frame relative to the previous frame needs to be re-rendered in steps L3-L4, thereby improving the rendering accuracy. At the same time, it avoids unnecessary judgments on whether some blocks need to be re-rendered, thereby improving rendering efficiency.

[0052] 3. By layering, partitioning, and blocking the point cloud data, and then relying on the point cloud data generated in the previous frame and the next frame, identify the singular blocks of the type "point cloud is newly added but the point cloud position is not offset" and / or the type "point cloud is not newly added but the point cloud position is offset" from the blocks of the next frame, and then use different strategies for the two different types of singular blocks to render the point cloud data. The technical core of the two strategies is that, based on the unique block number bound to the block, through the block parameter change or the point cloud offset similarity matching method, the rendering basis block with block similarity to the singular block is matched from the database, and then the singular block is directly rendered with the rendering scheme of the rendering basis block, without the need for a complex deep learning model, thereby ensuring the rendering speed of the partition details under the conditional constraint scenario of the present application and improving the rendering effect of the details.

[0053] 4. Different singular block identification methods are provided for blocks where point clouds are added and / or disappear, and for blocks where point clouds are offset. The layered-partitioned-blocked method provided in the present application can quickly identify singular blocks as point cloud recognition objects in the front and back frames in each block divided by the layered-partitioned-blocked template, find the areas that need to be rendered in detail in the back frame, and the system subsequently focuses on rendering these detail areas, balancing the speed and accuracy of re-rendering in real-time rendering scenarios under the constraints of fixed camera field of view depth, camera angle range, and the distance between the rendering object and the center of the camera field of view. The rendering requirements change due to camera rotation, change in the type of the rendering object, or change in the scene in which the rendering object is located.

[0054] 5. A layered-partitioned-blocked method is provided for the constraint condition of "camera field of view depth, camera angle range, and fixed distance between the rendering object and the center of the camera field of view". Under this method, it is possible to identify singular blocks of the type "point cloud is added but the point cloud position is not offset", and it is also possible to identify singular blocks of the type "point cloud is not added but the point cloud position is offset". This makes it possible to balance the rendering accuracy and rendering speed of the detail area in the real-time rendering scene under the above constraints.

[0055] 6. Through the provided layering-partitioning-blocking method, through the frame number constraints of the previous and next frames, and the judgment of whether to shift to the adjacent block in step A2, the recognition of the rendering object for the next frame as the basis for judging whether the details need to be rendered each time is limited to each adjacent block, which greatly reduces the amount of data for judging the singular blocks, which is conducive to improving the subsequent rendering speed of the singular blocks.

[0056] 7. For different singular blocks of type "newly added point cloud but not shifted point cloud position" and type "not newly added point cloud but shifted point cloud position", the first strategy and the second strategy are used to render point cloud data respectively. In the first strategy, the screening conditions of the candidate set in step B2 are limited to the parameter changes of each block in the same layer block combination and adjacent layer block combination of the associated subsequent frame, which narrows the search range of the rendering basis block (expanded block) for the singular block of type "newly added point cloud but not shifted point cloud position", instead of searching for the rendering basis block within the point cloud space range constructed by the overall historical frame, which greatly reduces the rendering time using the first strategy. And with the set change parameters as the search feature, the expanded block as the basis for rendering the singular block can be quickly found. In the second strategy, the search range of the rendering reference block is expanded from the same layer as the layer where the first block is located in the historical frame to the adjacent layer, so as to prevent the problem of poor rendering effect of the first block due to excessive number of point clouds offset to adjacent layers when rendering the point cloud of a singular block of type "no new point cloud is added but the point cloud position is offset", without considering the impact of the point clouds offset to adjacent layers on the rendering effect of the remaining point clouds in the first block that have not been offset.

[0057] 8. The first strategy and the second strategy used for point cloud rendering of different types of singular blocks have complementary rendering effects. The first strategy is to perform point cloud rendering on singular blocks of type "newly added point cloud but not offset point cloud position", and the second strategy is to perform point cloud rendering on singular blocks of type "not new point cloud but offset point cloud position". When the point cloud in a block is offset to other blocks, new point clouds are added to other blocks. At this time, the first strategy is used to render the point clouds of other blocks, while the second strategy is used to render the point cloud of this block, that is, the point cloud that disappears from the block (the point cloud offset to other blocks) is rendered in supplementary fashion using the first strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 is a diagram of implementation steps of a point cloud rendering method based on laser radar and camera data fusion provided by an embodiment of the present invention;

[0060] Figure 2 It is a schematic diagram of numbering the blocks in the cube from left to right and from top to bottom layer by layer. DETAILED DESCRIPTION

[0061] The technical solution of the present invention is further described below with reference to the accompanying drawings and through specific implementation methods.

[0062] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on this patent; in order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0063] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "inner", "outer", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as a limitation on this patent. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0064] In the description of the present invention, unless otherwise clearly specified and limited, if the term "connection" or the like appears to indicate the connection relationship between components, the term should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two components or the interaction relationship between two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0065] First of all, it should be noted that the point cloud rendering method based on the fusion of laser radar and camera data provided in this embodiment is applied in the case where the camera field of view depth, camera angle range, and the distance between the rendering object and the center of the camera field of view are fixed, and the camera rotates, and / or the type of the rendering object changes, and / or the scene in which the rendering object is located changes. The meaning of the camera field of view depth and camera angle range will be specifically explained in the subsequent method of layering, partitioning, and blocking point cloud data, and will not be explained here. The fixed distance between the rendering object and the center of the camera field of view means that the camera beam emission point is the center of the camera field of view, and the rendering object is the focus point of the camera, that is, the distance between the focus point of the camera and the camera beam emission point is fixed. Camera rotation includes left and right rotation or up and down rotation or spherical rotation of the camera; the change of the type of the rendering object means that the focus object of the camera in the previous frame is assumed to be object O1, and the focus object in the next frame is assumed to be O2, or part of it is O1, which means that the type of the rendering object has changed, or the type of the object specified to be rendered in the previous frame and the next frame has changed. The scene where the rendering object is located changes, for example, the rendering object O1 is in the grass in the previous frame and enters the river in the next frame, and the scene types where the rendering object O1 is located in the previous frame and the next frame change.

[0066] The point cloud rendering method based on laser radar and camera data fusion provided in this embodiment is as follows: Figure 1 As shown, the steps include:

[0067] L1, construct point cloud based on multi-eye vision method, and obtain the hierarchical partitioning and blocking template with the maximum accommodation intersection ratio from the template library to accommodate the constructed point cloud space volume and place it in The axis divides the three-dimensional space into eight quadrants, and the center point of the internal space of the hierarchical partitioning template is located at The origin of the coordinate system;

[0068] The method based on multi-eye vision to construct point cloud is an existing method and will not be explained here. The layered, partitioned and block templates obtained from the template library are as follows: Figure 2 The point cloud volume is preferably centered in the hierarchical partitioning and blocking template. There are many ways to center the point cloud volume, such as first calculating the center point of the point cloud volume, and then comparing the center point to the The origins of the coordinate systems coincide, thereby centering the point cloud volume in the hierarchical partitioning block template.

[0069] L2, the laser radar verifies the type and coordinate position of the feature points of each point cloud in each block within the layered partitioning and blocking template, and corrects the type and coordinate position of the point cloud constructed based on multi-view errors;

[0070] It should be noted that in step L2, the laser radar does not reconstruct the point cloud data of the rendered object, but verifies the feature point type and coordinate position of each point cloud in the point cloud space constructed by the camera. The verification method is as follows:

[0071] The laser radar emits a laser beam to a point cloud in the point cloud space volume to perform laser ranging, and calculates the coordinate position of the point cloud in three-dimensional space based on the laser emission angle, and then identifies the type of the point cloud using a point cloud type recognition method different from that used by the camera. If the point cloud type is inconsistent or the point cloud type of the location cannot be determined, a laser point cloud of the same type as the point cloud recognized by the camera is searched in the neighborhood of the location (such as a rectangular box that selects the location, and the size of the rectangular box is set according to the rendering accuracy requirements), and the coordinate position of the laser point cloud in three-dimensional space is calculated to correct the coordinate position of the point cloud recognized by the camera.

[0072] L3, real-time monitoring and extraction of singular blocks of the latter frame relative to the former frame in two consecutive frames verified and corrected by LiDAR;

[0073] L4, for the singular blocks of type "the point cloud is added but the point cloud position is not offset", the first strategy is used to render the point cloud data; for the singular blocks of type "the point cloud is not added but the point cloud position is offset", the second strategy is used to render the point cloud data.

[0074] "Newly added point clouds but unchanged point cloud positions" in a block means: for example, the point cloud data generated in the previous frame is divided into several cubic blocks in three-dimensional space, and in one of the blocks, there are three point clouds, p1, p2, and p3, whose positions in the block are assumed to be c1, c2, and c3 respectively. In the next frame, assuming that in the same block, there are four point clouds, p1, p2, p3, and p4, and the positions of the three point clouds p1, p2, and p3 in the previous and next frames remain unchanged, or the position change amplitude is less than the preset change amplitude threshold, then it is determined that "newly added point clouds but unchanged point cloud positions" have occurred in the block.

[0075] "No new point cloud is added but the point cloud position is offset" in the block means: for example, the point cloud p1 in the above-mentioned block is offset to other blocks in the subsequent frame, and the latter's position change amplitude is greater than or equal to the preset change amplitude threshold, and no new point cloud is added to the block in the subsequent frame, then it is determined that "no new point cloud is added but the point cloud position is offset" occurs in the block in the subsequent frame.

[0076] The following is a description of the method for identifying the singular blocks of the subsequent frame relative to the previous frame in step L3:

[0077] In this embodiment, the same hierarchical partitioning block template is used to accommodate the point cloud space volumes constructed by the camera for the rear frame and the front frame in two consecutive shots. Each block in the hierarchical partitioning block template has a unique block number. When the hierarchical partitioning block template is used to accommodate the first point cloud space volume generated in the front frame and the second point cloud space volume generated in the rear frame, the position of the hierarchical partitioning block template in the three-dimensional space remains fixed. The first point cloud space volume and the second point cloud space volume are used to represent the same rendering object.

[0078] For example, a method of uniquely numbering each block in a layered, partitioned, and block template is to arrange 16 regular cubes (blocks) of exactly the same shape and size in "4 rows and 4 columns" to form a cube matrix unit. Such 4 identical cube matrix units overlap and overlap each other to form a cube formed by 64 regular cubes. Each regular cube in the cube is numbered from left to right and from top to bottom layer by layer to obtain a unique number corresponding to each regular cube. Figure 2 As shown in , in the cube matrix unit facing the cube, each regular cube is numbered 1, 2, ..., 16 from left to right and from top to bottom. In another cube matrix unit adjacent to the cube matrix unit, each regular cube is numbered 17, 18, ..., 32 from left to right and from top to bottom.

[0079] After the point cloud data verified and corrected by the laser radar falls on the corresponding block in the layered partitioning and blocking template, the method for identifying the singular block of the subsequent frame relative to the previous frame includes the steps of:

[0080] A1, obtain a second block having the same number as the first block generated in the next frame from the block set generated in the previous frame, and then further obtain adjacent blocks that are in the same layer as the first block and are adjacent to the first block above, below, left, right, upper left, lower left, upper right, and lower right;

[0081] For example, suppose Figure 2 The block 6 in is the first block generated in the subsequent frame that needs to be judged as a singular block. The unique number of the first block is "number 6". Then, from the previous frame continuous with the subsequent frame, the block numbered "6" is obtained as the second block. Then obtain Figure 2 The adjacent blocks that are in the same layer and adjacent to block 6 include block 1, block 2, block 3, block 5, block 7, block 9, block 10, and block 11.

[0082] A2, determining whether the point cloud formed in the second block is offset to at least one adjacent block,

[0083] If yes, extract the adjacent blocks that cause the offset, and then determine whether the first block is a singular block according to the offset singular block identification strategy;

[0084] If not, go to step A3;

[0085] The method for determining whether the point cloud formed in the second block is offset to the adjacent block is an existing method. For example, the type of each point cloud in the second block can be firstly identified. For example, for a face model, each key point constituting the face has a corresponding unique key point type. Then, when the same point cloud that appears in the second block in the previous frame is identified in the adjacent block generated in the subsequent frame, it is determined that the point cloud has offset to the adjacent block.

[0086] When step A2 judges "if no" and turns to step A3, it means that the system's judgment on whether the first block is a singular block turns to the judgment process of whether the first block is a singular block of the type "point cloud is added but the point cloud position is not offset". When step A2 judges "if yes", it means that the system's judgment on whether the first block is a singular block turns to the judgment process of whether the first block is a singular block of the type "point cloud is not added but the point cloud position is offset".

[0087] In step A3, the method for determining whether the first block is a singular block of the type "point cloud is not added but point cloud position is offset" is:

[0088] It is determined whether the number of newly added point clouds of the first block compared with the second block exceeds a first threshold. If so, the first block is determined to be a singular block.

[0089] In step A2, the method for determining whether the first block is a singular block according to the offset singular block identification strategy is:

[0090] Determine whether the number of adjacent blocks that produce offsets is greater than a preset second threshold value, and if so, determine that the first block is a singular block; and / or determine whether the ratio of the number of first point clouds offset in each adjacent block to the number of second point clouds that are not offset in the second block is greater than a preset third threshold value, and if so, determine that the first block is a singular block; and / or determine whether the offset of at least one point cloud offset in at least one adjacent block is greater than a preset offset threshold value, and if so, determine that the first block is a singular block.

[0091] For example, point clouds p1, p2, p3, p4, and p5 are formed in the second block of the previous frame, where only point cloud p1 is offset to Figure 2In the adjacent block numbered "2" shown, assuming that the second threshold is set to "2", the number of adjacent blocks that produce offsets is "1", and the condition of being greater than the preset second threshold is not met, so the first block is determined not to be a singular block. When the offset singular block identification strategy is set to: at least one of the three judgment conditions is met, that is, when the first block is determined to be a singular block, it is necessary to continue to judge whether the ratio of the number of first point clouds offset in each adjacent block to the number of second point clouds that have not been offset in the second block is greater than the preset third threshold. Continuing with the above example, only point cloud p1 is offset in the adjacent block numbered "2", then the number of first point clouds is "1", and the number of second point clouds that have not been offset in the second block is "4", then the ratio of the two is one-fourth. Assuming that the third threshold is set to one-third, the judgment condition is not met, and it is necessary to continue to judge whether the offset of at least one point cloud offset in at least one adjacent block is greater than the preset offset threshold. Assuming that the judgment condition is met, the first block is also determined to be a singular block.

[0092] Through steps A1-A3, after finding the singular blocks of the type "point cloud is added but point cloud position is not offset" and / or the type "point cloud is not added but point cloud position is offset" of the latter frame relative to the previous frame in two consecutive frames, this embodiment uses two different strategies in step L4 to render point cloud data for different types of singular blocks.

[0093] In step L4, the method of rendering point cloud data for the singular blocks of the type "newly added point cloud but not offset point cloud position" by using the first strategy includes the following steps:

[0094] B1, in the layer where the first block determined as a singular block is located, extract the blocks arranged adjacent to the first block at the upper, lower, left, right, upper left, lower left, upper right, and lower right positions to form a same-layer block combination with the first block, and in the adjacent layer adjacent to the same-layer block combination, extract the adjacent layer block combination that overlaps with the same-layer block combination in terms of hierarchical depth;

[0095] For example, suppose Figure 2 The first block numbered "6" in the is determined to be a singular block of the type "newly added point cloud but not offset point cloud position". After the point cloud data is layered, partitioned, and blocked in the three-dimensional space in this embodiment, the level where the block No. 6 is located is assumed to be the first level, and 8 blocks arranged at the top, bottom, left, right, upper left, lower left, upper right, and lower right positions of the block No. 6 are extracted, that is, Figure 2The blocks 1, 2, 3, 5, 7, 9, 10, and 11 in the image are combined with block 6 to form a block combination on the same layer. In the adjacent layer adjacent to the block combination on the same layer, that is, in the second layer where the layers are sequentially deepened along the depth direction of the visual field, the adjacent layer block combination that overlaps with the block combination on the same layer in terms of layer depth is extracted, that is, the adjacent layer block combination that overlaps with the block combination on the same layer in terms of layer depth is extracted. Figure 2 Blocks No. 17, No. 18, No. 19, No. 21, No. 22, No. 23, No. 25, No. 26, and No. 27 in the structure constitute an adjacent layer block combination that overlaps in level depth with the same layer block combination consisting of block No. 1, No. 2, No. 3, No. 5, No. 6, No. 7, No. 9, No. 10, and No. 11. It should be noted here that when the same layer block combination is at the second level, the adjacent layer block combination includes two combinations at the first level and the third level.

[0096] B2, using the parameter changes of each block in the same layer block combination and the adjacent layer block combination generated in the same subsequent frame as the screening condition, screening out the candidate set from the database, the specific method includes the steps of:

[0097] B21, identify the changed blocks in the same layer block combination and adjacent layer block combination generated in the same subsequent frame, including newly added blocks of point cloud, and / or disappeared blocks of point cloud, and / or blocks where point cloud is added and disappeared at the same time, and then obtain the change parameters of each changed block, including the spatial coordinates and point cloud type of the newly added point cloud and / or disappeared point cloud in the changed block. The point cloud type is used to represent the key point features of the rendered object or the key point features of the rendered scene where the rendered object is located. For example, a certain point cloud is assumed to be the zygomatic key point in the key point of the human face. The zygomatic key point falls on block a in the previous frame and on block b in the subsequent frame. The zygomatic key point is added to block b, and the zygomatic key point disappears in block a. Then the spatial coordinates and point cloud type of the zygomatic key point in block b are obtained, and the point cloud type is "zygomatic key point".

[0098] B22, using the change parameter of the change block as a matching condition, matching a second historical block having point cloud change similarity with the change block from the same historical frame in the database, the second historical block and the change block having point cloud change similarity have the same level; each second historical block at the same level constitutes a historical block combination at the same level or an adjacent historical block adjacent to the historical block combination at the same level;

[0099] For example, assume that the change blocks identified in step B21 from the same-layer block combination generated in the same post-frame and the adjacent-layer block combination hierarchically adjacent to the same-layer block combination include change blocks BH1, BH2, BH3, BH4, and BH5, where change blocks BH1 and BH2 are blocks in the same-layer block combination, and BH3, BH4, and BH5 are blocks in the adjacent-layer block combination. Also assume that point cloud p1 is added to change block BH1, point cloud p2 disappears from change block BH2, point cloud p3 is added to change block BH3, point cloud p4 is added to change block BH4 but point cloud p5 disappears, and point cloud p6 disappears from change block BH5.

[0100] Assume that the second historical block LSFK1 generated in a certain historical frame has point cloud change similarity with the change block BH1, that is, a historical point cloud lsp1 in the second historical block LSFK1 has point cloud change similarity with the newly added point cloud p1 in the change block BH1. The point cloud change similarity is calculated by the following method:

[0101] The newly added point cloud p1 has corresponding spatial coordinates in the change block BH1, and the change block BH1 has a unique block number. In the front frame that constitutes two consecutive frames with the rear frame, the newly added point cloud p1 may appear in a block in the front frame, or may not appear in the front frame. If it appears in a block in the front frame, the number of the previous frame block and the spatial coordinates of the newly added point cloud p1 in the previous frame block are obtained. If it does not appear in the previous frame, the original change parameters of the newly added point cloud p1 in the previous frame are set to "0". The spatial coordinates of the newly added point cloud p1 in the change block BH1, the type of point cloud p1, the number of the change block BH1, the spatial coordinates of the newly added point cloud p1 in the previous frame block, and the number of the previous frame block constitute the change parameters of the newly added point cloud p1 after it lands on the change block BH1.

[0102] The change parameters bound to each change point cloud in the change block BH1 are matched with the change parameters bound to each change point cloud in the second historical block LSFK1 in terms of parameter similarity (i.e., point cloud change similarity). If the parameter similarities of all change point clouds (each feature point) in the two blocks are matched successfully, it is determined that the change block BH1 has point cloud change similarity with the second historical block LSFK1.

[0103] The newly added point cloud p1 and the historical point cloud lsp1 have point cloud change similarity: when the first spatial coordinate of the newly added point cloud p1 in the change block BH1 is the same as the second spatial coordinate of the historical point cloud lsp1 in the second historical block LSFK1, or the coordinate deviation distance (that is, the straight line length of the first spatial coordinate deviating from the second spatial coordinate) is less than the preset distance threshold, and the newly added point cloud p1 and the historical point cloud lsp1 are of the same type (for example, both are zygomatic key points), and the change block BH1 and the second historical block LSFK1 have the same number, and the spatial coordinate of the newly added point cloud p1 in the previous frame block is the same as the spatial coordinate of the historical point cloud lsp1 in the historical previous frame block in the historical previous frame that constitutes two consecutive frames with the second historical block LSFK1, or the coordinate distance is less than the preset distance threshold, and the previous frame block and the historical previous frame block have the same number, it is determined that the newly added point cloud p1 and the historical point cloud lsp1 have point cloud change similarity.

[0104] The principle of the method for performing point cloud change similarity matching on the disappeared blocks where point clouds disappear, and on the blocks where point clouds are added and disappeared at the same time, is the same as that for the newly added blocks, and will not be described in detail.

[0105] After finding the corresponding second historical blocks having point cloud change similarity with each change block identified in step B21 from the same historical frame, the second historical blocks generated at the same level of the same historical frame constitute a same-layer historical block combination, or constitute an adjacent-layer historical block combination adjacent to the same-layer historical block combination.

[0106] Continuing with the above example, it is assumed that, through step B22, the corresponding second historical blocks with point cloud change similarity found for the change blocks BH1-BH5 are LSFK1-LSFK5, and the second historical blocks LSFK1-LSFK2 are at the same level, then the second historical blocks LSFK1 and LSFK2 constitute a same-level historical block combination. The second historical blocks LSFK3-LSFK5 are at the same level, then the second historical blocks LSFK3-LSFK5 constitute an adjacent layer historical block combination adjacent to the same layer historical block combination.

[0107] Then proceed to step:

[0108] B23, adding the historical block set associated with the historical frame consisting of the same-layer historical block combination and the adjacent-layer historical block combination generated in the same historical frame to the candidate set.

[0109] After step B2, after selecting the candidate set from the database, in step L4, the method of rendering point cloud data using the first strategy proceeds to step:

[0110] B3, select from the candidate set the historical block set that has the maximum point cloud change similarity with the block set composed of the same layer block combination and the adjacent layer block combination, and then expand the selected historical block set by blocks to obtain an expanded set. Each block in the block set has an expanded block with the same number in the expanded set.

[0111] The method of selecting the historical block set with the maximum point cloud change similarity with the block set from the candidate set is briefly described as follows:

[0112] In step B22, a method for matching a second historical block having point cloud change similarity with a change block from a database has been given. When calculating the point cloud change similarity between a block set and a historical block set, in this embodiment, the number of block pairs consisting of a change block and a second historical block having a point cloud change similarity in the block set and the historical block set is counted as the point cloud change similarity between the block set and the historical block set.

[0113] After screening out the historical block set with the maximum point cloud change similarity, the method of expanding the historical block set into blocks to obtain the expanded set is as follows: each block in the block set has an expanded block with the same number in the expanded set. That is, the expanded set has the same spatial matrix form as the spatial matrix presented by the block set composed of the same-layer block combination and the adjacent-layer block combination. It should be explained here that each second historical block in the historical block set obtained by calculating the point cloud change similarity is a changed block, and the block set includes changed blocks and / or unchanged blocks. Therefore, it is necessary to expand the historical block set here to ensure that the expanded block (rendering based on the block) with the same block number as the first block can be extracted in step B4.

[0114] It should also be noted that if the candidate set screened out after step B2 is an "empty set", it means that the point cloud data rendering using the first strategy has failed, and the user is prompted to perform manual rendering or use other algorithms for rendering, such as using a conventional deep learning model for rendering.

[0115] B4, from the expanded blocks in the expanded set that have the same block number as the first block determined to be of the type "newly added point cloud but the point cloud position has not been offset", further identify the second rendering basis point cloud that has a spatial position corresponding relationship with the newly added point cloud in the first block (the spatial position is the same or the offset distance is less than a preset distance threshold) and the same point cloud type, and then assign the historical rendering method of the second rendering basis point cloud to the newly added point cloud corresponding to the position.

[0116] In step L4, the method of rendering point cloud data for the singular blocks of the type "point cloud is not added but the point cloud position is offset" by using the second strategy includes the following steps:

[0117] C1, in the layer where the first block whose singularity type is determined to be "no new point cloud is added but the point cloud position is offset" is located and the adjacent layers adjacent to the layer where the first block is located, extract the rendering reference blocks, and obtain the unique block number corresponding to each rendering reference block; the rendering reference blocks in the same layer as the first block include: blocks adjacent to the top, bottom, left, right, upper left, lower left, upper right, and lower right of the first block; the rendering reference blocks in the adjacent layers of the first block include: each rendering reference block in the same layer as the first block and the first block together constitute a same-layer block combination, and in the adjacent layers adjacent to the same-layer block combination, extract the adjacent layer block combination that overlaps with the same-layer block combination in terms of hierarchical depth;

[0118] C2, identifying each offset point cloud in each extracted rendering reference block, and obtaining the offset parameters of each offset point cloud, including: the coordinate position of the offset point cloud after being offset to the rendering reference block, the offset point cloud type, and the distance from the initial coordinate position in the previous frame;

[0119] How to identify the offset point cloud from the blocks has been explained in the first strategy mentioned above and will not be repeated here.

[0120] C3, for each rendering reference block at the same level, using the offset parameter as a matching condition, matching the first historical blocks respectively corresponding to the point cloud offset similarity in the same historical layer of the same historical frame from the database, and then obtaining the first block numbers respectively corresponding to the matched first historical blocks, and obtaining the second block numbers respectively corresponding to the rendering reference blocks respectively corresponding to the point cloud offset similarity with each first historical block;

[0121] The method for calculating the point cloud offset similarity between the rendering reference block and the first historical block is briefly described as follows by taking an example:

[0122] Assume that the rendering reference blocks at the same level as the first block judged as the singularity type of "no new point cloud is added but the point cloud position is offset" and adjacent to the first block include XRCZFK1-XRCZFK8. A certain historical layer of a certain historical frame includes 9 first historical blocks DYLSFK1-DYLSFK9, and DYLSFK1-DYLSFK5 and DYLSFK7-DYLSFK9 are respectively set at the top, bottom, left, right, upper left, lower left, upper right, and lower right positions of DYLSFK6. Assume that there are two offset point clouds in the rendering reference block XRCZFK1, which are respectively recorded as XRCZFK1-p1 and XRCZFK1-p2, and there are also two offset point clouds in the first historical block DYLSFK1, which are respectively recorded as DYLSFK1-p1 and DYLSFK1-p2. Assume that XRCZFK1-p1 is of the same type as DYLSFK1-p1, and the coordinate position of XRCZFK1-p1 in the rendering reference block XRCZFK1 corresponds to the coordinate position of DYLSFK1-p1 in the first historical block DYLSFK1 (they have the same coordinate position in the same three-dimensional space coordinate system, or the deviation of the coordinate positions of the two is less than a preset deviation threshold), and the distance between the coordinate position of XRCZFK1-p1 in XRCZFK1 and the coordinate position of XRCZFK1-p1 in the previous frame (defined as the first distance) is the same as D If the absolute value of the difference between the coordinate position of YLSFK1-p1 in the first historical block DYLSFK1 and the distance (defined as the second distance) between the coordinate position of DYLSFK1-p1 in the historical previous frame that constitutes two consecutive frames with the historical frame is less than the preset absolute value threshold of the difference, it is determined that XRCZFK1-p1 and DYLSFK1-p1 have point cloud offset similarity, and when XRCZFK1-p1 and DYLSFK1-p1 also have point cloud offset similarity, it is determined that the rendering reference block XRCZFK1 and the first historical block DYLSFK1 have point cloud offset similarity.

[0123] Then, if the first historical blocks DYLSFK2-DYLSFK5 and DYLSFK7-DYLSFK9 that generate point cloud offsets in the same layer of the same historical frame as the first historical block DYLSFK1 are matched to the corresponding rendering reference blocks in XRCZFK2-XRCZFK8 through point cloud offset similarity matching, the unique block numbers corresponding to the first historical blocks that generate point cloud offsets are obtained. For example, assuming that DYLSFK1, DYLSFK2, DYLSFK5, and DYLSFK8 generate point cloud offsets, the first block numbers corresponding to DYLSFK1, DYLSFK2, DYLSFK5, and DYLSFK8 are obtained. For example, the first block numbers corresponding to DYLSFK1, DYLSFK2, DYLSFK5, and DYLSFK8 are "1", "2", "5", and "8", respectively. Assuming that the rendering reference blocks that have point cloud offset similarity with DYLSFK1, DYLSFK2, DYLSFK5, and DYLSFK8 are XRCZFK1, XRCZFK3, XRCZFK7, and XRCZFK5, respectively, the second block numbers corresponding to XRCZFK1, XRCZFK3, XRCZFK7, and XRCZFK5 are further obtained, and they are assumed to be "1", "2", "5", and "8" respectively.

[0124] It should be noted here that in step C3, if for each rendering reference block at the same level, it is impossible to find the first historical blocks corresponding to each rendering reference block having point cloud offset similarity in a certain historical layer of a certain historical frame in the database, then the second strategy is determined to have failed, and the manual rendering process is entered or the conventional point cloud rendering process is performed using a deep learning model.

[0125] C4, determining whether the first block number of the first historical block having a point cloud offset similarity relationship is the same as the second block number of the rendering reference block,

[0126] If yes, extract the first historical block having the same block number as the first block in the historical frame as the rendering basis block, and then proceed to step C5;

[0127] If not, it is determined that no rendering basis block can be extracted from the historical frame;

[0128] For example, the first blocks corresponding to DYLSFK1, DYLSFK2, DYLSFK5, and DYLSFK8 are numbered "1", "2", "5", and "8", respectively. Assuming that the second blocks corresponding to XRCZFK1, XRCZFK3, XRCZFK7, and XRCZFK5, which have point cloud offset similarity relationships with DYLSFK1, DYLSFK2, DYLSFK5, and DYLSFK8, are numbered "1", "2", "8", and "5", respectively, it is determined that the rendering basis block cannot be extracted in the historical frame.

[0129] Under the "if" judgment condition of step C4, an example of a method for extracting a first historical block having the same block number as the first block from the historical frame is as follows: assuming that the block number of the first block is "6", and the block number of DYLSFK6 at the same level as DYLSFK1, DYLSFK2, DYLSFK5, and DYLSFK8 is also "6", then DYLSFK6 is extracted from the historical frame as the rendering basis block.

[0130] After extracting the rendering basis block for the first block, the second strategy proceeds to step:

[0131] C5, for each remaining point cloud that has not undergone positional offset in the first block, identify a first rendering basis point cloud of the same type and having a spatial position correspondence with each remaining point cloud from the rendering basis block, and then assign the rendering method of the first rendering basis point cloud to the remaining point cloud corresponding to the position.

[0132] For example, assuming that the remaining point clouds in the first block that have not undergone positional offset include SYDY-p1, then the rendering basis point cloud DYXRYJDY-p1 (defined as the first rendering basis point cloud) that has a spatial positional correspondence or the same type as SYDY-p1 is identified from the rendering basis block. The meaning of SYDY-p1 and DYXRYJDY-p1 having a spatial positional relationship has been explained in the above content and will not be repeated. The same point cloud type means that, for example, SYDY-p1 and DYXRYJDY-p1 both represent cheekbone key points in the face, which indicates that the point cloud types of the two are the same.

[0133] The following describes the method for performing layering, partitioning and blocking on the layered, partitioned and blocked template in this embodiment.

[0134] The method to layer the camera field of view is:

[0135] Let the camera's field of view depth be Axis orientation Axis, depth of field, point cloud The maximum coordinate value in the axis direction is recorded as , the minimum value is recorded as , if the depth of field is divided into layer, then the thickness of each layer for , point cloud exist The axis coordinates are labeled , then the point cloud Layer ;

[0136] The method of partitioning the camera field of view is: point cloud exist The angle between the horizontal plane formed by the axis and the center direction of the camera's horizontal viewing angle , Representing point clouds exist The coordinates in the axis direction; the horizontal viewing angle width of the camera is , then When the axis is the center direction of the camera's horizontal viewing angle, the camera's horizontal viewing angle range is ; If Divide into partitions, the angle range of each partition , then the point cloud The partition .

[0137] The method of dividing the camera field of view into blocks is:

[0138] The point cloud data is divided into 8 spatial quadrants through the same 3D spatial coordinate system, and each layered-partitioned spatial volume in each spatial quadrant is divided into The division method is:

[0139] In the hierarchical-partitioned space volume, the point cloud is axis, axis, The coordinate ranges of the axis directions are , , , then each block in the layered-partitioned space is The length in the axial direction is ,exist The width in the axial direction is ,exist The height in the axial direction is ;

[0140] The spatial coordinates are Point Cloud The unique spatial position index of the block that falls into is expressed as .

[0141] In order to further increase the rendering speed of point cloud data by the point cloud rendering method based on lidar and camera data fusion provided in this embodiment, it is more preferred that the layered partitioned blocks constructed for the same frame of point cloud data are distributed in 8 spatial quadrants divided by the same three-dimensional space coordinate system, and steps L1-L4 are executed in parallel for the point cloud data in each spatial quadrant.

[0142] In summary, the point cloud rendering method based on the fusion of laser radar and camera data provided by the present application sets corresponding layered, partitioned and block templates for different types of rendering objects under the constraints of the same camera field of view depth, the same camera angle range, and the same distance between the rendering object and the center of the camera field of view. By directly obtaining the layered, partitioned and block template that can accommodate the point cloud space body and has the maximum accommodation intersection ratio from the template library for the point cloud space body constructed for the rendering object by the multi-eye vision method, in the subsequent point cloud rendering, it is no longer necessary to perform layering, partitioning and block calculations for the camera field of view area of ​​each frame, which is conducive to improving the point cloud rendering speed. The type and coordinate position of the point cloud incorrectly constructed by the multi-eye vision method are corrected by laser radar, which reduces the error rate of judging whether each block of the subsequent frame relative to the previous frame needs to be re-rendered in steps L3-L4, and the rendering effect of details is increased through two strategies.

[0143] It should be noted that the above specific implementations are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art should understand that various modifications, equivalent substitutions, changes, etc. can be made to the present invention. However, as long as these changes do not deviate from the spirit of the present invention, they should be within the scope of protection of the present invention. In addition, some terms used in the specification and claims of this application are not restrictive, but are only for the convenience of description.

Claims

1. A point cloud rendering method based on laser radar and camera data fusion, characterized in that: Includes steps: L1, construct point cloud based on multi-eye vision method, and obtain the hierarchical partitioning and blocking template with the maximum accommodation intersection ratio from the template library to accommodate the constructed point cloud space volume and place it in The axis divides the three-dimensional space into eight quadrants, and the center point of the internal space of the hierarchical partitioning template is located at The origin of the coordinate system; L2, the laser radar verifies the type and coordinate position of the feature points of each point cloud in each block within the hierarchical partition separation template, and corrects the type and coordinate position of the point cloud incorrectly constructed based on the multi-eye vision method; L3, real-time monitoring and extraction of singular blocks of the latter frame relative to the former frame in two consecutive frames verified and corrected by the laser radar; L4, for the singular block of type "point cloud is added but point cloud position is not offset", the first strategy is used to render the point cloud data; for the singular block of type "point cloud is not added but point cloud position is offset", the second strategy is used to render the point cloud data; The same hierarchical partitioning and blocking template is used to accommodate the point cloud space volumes respectively constructed by the camera for the rear frame and the front frame in two consecutive frames, each block in the hierarchical partitioning and blocking template has a unique block number, and the hierarchical partitioning and blocking template is used to accommodate the first point cloud space volume generated in the front frame and the second point cloud space volume generated in the rear frame. The position of the hierarchical partitioning and blocking template in the three-dimensional space is fixed and unchanged, and the first point cloud space volume and the second point cloud space volume are used to represent the same rendering object; In step L3, the method for identifying the singular block comprises the steps of: A1, from the hierarchical partition block template for accommodating the first point cloud spatial volume, obtain a second block having the same number as the first block in the same hierarchical partition block template for accommodating the second point cloud spatial volume, and then further obtain adjacent blocks that are in the same layer as the first block and are adjacent to the first block above, below, left, right, upper left, lower left, upper right, and lower right; A2, determining whether the point cloud formed in the second block is offset to at least one of the adjacent blocks; If yes, extract the adjacent block that generates the offset, and then determine whether the first block is the singular block according to the offset singular block identification strategy; If not, go to step A3; A3, determining whether the number of newly added point clouds of the first block compared with the second block exceeds a first threshold, and if so, determining that the first block is the singular block.

2. The point cloud rendering method based on laser radar and camera data fusion according to claim 1, characterized in that: In step A2, the method for determining whether the first block is the singular block according to the offset singular block identification strategy is: Determine whether the number of the adjacent blocks that generate the offset is greater than a preset second threshold, and if so, determine that the first block is the singular block; and / or determining whether a ratio of the number of first point clouds offset in each of the adjacent blocks to the number of second point clouds not offset in the second block is greater than a preset third threshold, and if so, determining that the first block is the singular block; And / or determine whether the offset of at least one point cloud in at least one of the adjacent blocks is greater than a preset offset threshold; if so, determine that the first block is the singular block.

3. The point cloud rendering method based on laser radar and camera data fusion according to claim 1, characterized in that: In step L4, the method for rendering point cloud data using the first strategy includes the following steps: B1, in the layer where the first block determined to be a singular block is located, extract the blocks arranged adjacent to each other at the upper, lower, left, right, upper left, lower left, upper right, and lower right positions of the first block to form a same-layer block combination with the first block, and in the adjacent layers adjacent to the same-layer block combination, extract the adjacent-layer block combination that overlaps with the same-layer block combination in terms of hierarchical depth; B2, using the parameter changes of each block in the same-layer block combination and the adjacent-layer block combination generated in the same subsequent frame as the screening condition, screening out a candidate set from the database; B3, selecting from the candidate set a historical block set having the maximum point cloud change similarity with the block set consisting of the same-layer block combination and the adjacent-layer block combination, and then performing block expansion on the historical block set to obtain an expanded set, wherein each block in the block set has an expanded block with the same number in the expanded set; B4, from the expanded blocks in the expanded set having the same block number as the first block, further identify a second rendering basis point cloud that has a spatial position corresponding to the newly added point cloud in the first block and has the same point cloud type, and then assign the historical rendering method of the second rendering basis point cloud to the newly added point cloud corresponding to the position.

4. The point cloud rendering method based on laser radar and camera data fusion according to claim 3 is characterized in that: In step B2, the method for screening out the candidate set includes the steps of: B21, identifying the changed blocks in the same-layer block combination and the adjacent-layer block combination generated in the same subsequent frame, including newly added point cloud blocks, and / or disappeared point cloud blocks, and / or blocks where point clouds are added and disappeared simultaneously, and then obtaining the change parameters of each of the changed blocks, including the spatial coordinates and point cloud type of the newly added point cloud and / or the disappeared point cloud in the changed block, the number of the changed block, and the number of the block where the newly added point cloud and / or the disappeared point cloud landed in the previous frame, and the spatial coordinates of the landing point in the landing block; B22, using the change parameter of the change block as a matching condition, matching a second historical block having point cloud change similarity with the change block from the same historical frame in the database, wherein the second historical block has the same level as the change block having point cloud change similarity; The second historical blocks at the same level constitute a historical block combination at the same level, or constitute a historical block combination at an adjacent level that is adjacent to the historical block combination at the same level; B23, adding the historical block set associated with the historical frame consisting of the same-layer historical block combination and the adjacent-layer historical block combination generated in the same historical frame into the candidate set.

5. The point cloud rendering method based on laser radar and camera data fusion according to claim 3 or 4, characterized in that: The method for determining whether the newly added point cloud in the change block has point cloud change similarity with the historical point cloud in the second historical block is: When the first spatial coordinate of the newly added point cloud in the change block is the same as the second spatial coordinate of the historical point cloud in the second historical block or the coordinate deviation distance is less than the preset distance threshold, and the newly added point cloud is of the same type as the historical point cloud, and the number of the change block where the newly added point cloud lands is the same as the number of the second historical block where the historical point cloud lands, and the third spatial coordinate of the newly added point cloud landing point in the previous frame block is the same as the fourth spatial coordinate of the historical point cloud landing point in the historical previous frame block of the historical previous frame or the coordinate deviation distance is less than the preset distance threshold, and the previous frame block has the same number as the historical previous frame block, then it is determined that the newly added point cloud has point cloud change similarity with the historical point cloud; If each change point cloud in the change block is matched to a historical point cloud with point cloud change similarity in the second history block, it is determined that the change block has point cloud change similarity with the second history block; The number of block pairs formed by the change blocks and the second historical blocks having a point cloud change similarity relationship in the block set and the historical block set is counted as the point cloud change similarity between the block set and the historical block set.

6. The point cloud rendering method based on laser radar and camera data fusion according to claim 2 or 3, characterized in that: In step L4, the method for rendering point cloud data using the second strategy includes the following steps: C1, extracting rendering reference blocks from the layer where the first block determined as a singular block is located and the adjacent layers adjacent to the layer where the first block is located, and obtaining a unique block number corresponding to each of the rendering reference blocks; C2, identifying each offset point cloud in each of the extracted rendering reference blocks, and obtaining offset parameters of each offset point cloud, including: the coordinate position of the offset point cloud after being offset to the rendering reference block, the distance from the initial coordinate position in the previous frame, and the offset point cloud type; C3, for each of the rendering reference blocks at the same level, using the offset parameter as a matching condition, matching the first historical blocks respectively corresponding to the point cloud offset similarity in the same historical layer of the same historical frame from the database, and then obtaining the first block numbers respectively corresponding to each of the matched first historical blocks, and obtaining the second block numbers respectively corresponding to each of the rendering reference blocks having the point cloud offset similarity with each of the first historical blocks; C4, determining whether the first block number of the first historical block having a point cloud offset similarity relationship is the same as the second block number of the rendering reference block, If yes, extract the first historical block having the same block number as the first block from the historical frame as the rendering basis block, and then proceed to step C5; If not, it is determined that the rendering basis block cannot be extracted from the historical frame; C5, for each remaining point cloud that has not undergone positional offset in the first block, identify a first rendering basis point cloud of the same type and having a spatial position corresponding relationship with each of the remaining point clouds from the rendering basis block, and then assign the rendering method of the first rendering basis point cloud to the remaining point cloud corresponding to the position.

7. The point cloud rendering method based on laser radar and camera data fusion according to claim 6, characterized in that: In step C3, the method for judging the point cloud offset similarity between the rendering reference block and the first historical block is: When the first offset point cloud in the rendering reference block is of the same type as the second offset point cloud in the first historical block; and the first coordinate position of the first offset point cloud in the rendering reference block is the same as the second coordinate position of the second offset point cloud in the first historical block or the offset distance is less than a preset offset distance threshold; and the absolute value of the difference between the first distance between the first offset point cloud at the first coordinate position and the third coordinate position in the previous frame and the second distance between the second offset point cloud at the second coordinate position and the fourth coordinate position in the previous historical frame is less than a preset difference absolute value threshold, then it is determined that the first offset point cloud and the second offset point cloud have point cloud offset similarity; If each first-shifted point cloud in the rendering reference block finds a second-shifted point cloud with point cloud offset similarity in the first historical block, it is determined that the rendering reference block has point cloud offset similarity with the first historical block.

Images