Scene model reconstruction method, device, electronic device and storage medium

Abstract

The present application relates to a scene model reconstruction method, device, electronic device and storage medium. The method comprises: respectively obtaining point normals corresponding to multiple scanning points in a multi-frame radar point cloud corresponding to the scene to be reconstructed, splicing the multi-frame radar point cloud to obtain a global radar point cloud, that is, a complete point cloud corresponding to the scene to be reconstructed. The global radar point cloud is split into multiple sub-point clouds, and a scene model corresponding to the scene to be reconstructed is generated based on the multiple sub-point clouds and the point normals corresponding to the scanning points. Compared with the model generated by directly reconstructing the model with dense three-dimensional point clouds, reconstructing the model with sub-point clouds as units can improve the resolution of the scene model and ensure that the details of the objects displayed by the scene model are clear.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a scene model reconstruction method, device, electronic device and storage medium. Background Art

[0002] In applications such as autonomous driving and AR navigation, high-precision reconstruction of environmental scenes is required to achieve functions such as positioning, navigation, and path planning. Generally, laser radar is used to scan and reconstruct outdoor or indoor scenes. The point cloud model or CAD mesh model obtained based on the existing scene reconstruction method has problems such as high noise and blurred scene details that are not conducive to display. Summary of the invention

[0003] In order to solve the above technical problems, the present application provides a scene model reconstruction method, device, electronic device and storage medium.

[0004] In a first aspect, the present application provides a scene model reconstruction method, comprising:

[0005] Respectively obtaining point normals corresponding to a plurality of scanning points in a multi-frame radar point cloud corresponding to the scene to be reconstructed, wherein each frame of radar point cloud in the multi-frame radar point cloud is obtained by a laser radar by scanning a local scene of the scene to be reconstructed;

[0006] Splicing the multiple frames of radar point cloud to obtain a global radar point cloud;

[0007] Splitting the global radar point cloud into a plurality of sub-point clouds according to a preset reconstruction scene resolution;

[0008] Based on the plurality of sub-point clouds and point normals corresponding to the scanning points, a scene model corresponding to the scene to be reconstructed is generated.

[0009] In a second aspect, the present application provides a scene model reconstruction device, comprising:

[0010] An acquisition module, used to respectively acquire point normals corresponding to multiple scanning points in a multi-frame radar point cloud corresponding to a scene to be reconstructed;

[0011] A stitching module, used for stitching the multiple frames of radar point cloud to obtain a global radar point cloud;

[0012] A splitting module, used for splitting the global radar point cloud into multiple sub-point clouds;

[0013] The model reconstruction module is used to generate a scene model corresponding to the scene to be reconstructed based on the plurality of sub-point clouds and the point normals corresponding to the scanning points.

[0014] In a third aspect, the present application provides a laser radar, including a collection device, a point cloud conversion device and the above-mentioned scene model reconstruction device, wherein:

[0015] The acquisition device is used to acquire multiple frames of images of the scene to be reconstructed;

[0016] The point cloud conversion device is used to convert the multiple frames of images of the scene to be reconstructed into multiple frames of radar point clouds;

[0017] The scene reconstruction device generates a scene model corresponding to the scene to be reconstructed using the multiple frames of radar point clouds.

[0018] In a fourth aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0019] Respectively obtain point normals corresponding to a plurality of scanning points in a multi-frame radar point cloud corresponding to the scene to be reconstructed, wherein each frame of radar point cloud in the multi-frame radar point cloud is obtained by a laser radar by scanning a local scene of the scene to be reconstructed;

[0020] Splicing the multiple frames of radar point cloud to obtain a global radar point cloud;

[0021] Splitting the global radar point cloud into a plurality of sub-point clouds according to a preset reconstruction scene resolution;

[0022] Based on the plurality of sub-point clouds and point normals corresponding to the scanning points, a scene model corresponding to the scene to be reconstructed is generated.

[0023] In a fifth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0024] Respectively obtain point normals corresponding to multiple scanning points in the multi-frame radar point cloud corresponding to the scene to be reconstructed;

[0025] Splicing the multiple frames of radar point cloud to obtain a global radar point cloud;

[0026] Splitting the global radar point cloud into a plurality of sub-point clouds according to a preset reconstruction scene resolution;

[0027] Based on the plurality of sub-point clouds and point normals corresponding to the scanning points, a scene model corresponding to the scene to be reconstructed is generated.

[0028] Based on the above scene model reconstruction method, since each frame of radar point cloud can only represent the local scene of the scene to be reconstructed, it is necessary to reconstruct the scene model corresponding to the scene to be reconstructed through multiple frames of radar point clouds, respectively obtain the point normals corresponding to multiple scanning points in the multi-frame radar point cloud corresponding to the scene to be reconstructed, and splice the multi-frame radar point cloud to obtain the global radar point cloud, that is, the complete point cloud corresponding to the scene to be reconstructed. Since the laser radar device is in a real-time scanning state, there may be the same point cloud in two adjacent radar point clouds per frame. In order to reduce the amount of calculation data in the subsequent model reconstruction process, the partially overlapping point clouds are eliminated by point cloud splicing, and all the point clouds corresponding to the scene to be reconstructed are also collected. The global radar point cloud is split into multiple sub-point clouds, and the scene model corresponding to the scene to be reconstructed is generated based on the multiple sub-point clouds and the point normals corresponding to the scanning points. Compared with the model directly reconstructed by dense three-dimensional point clouds, the model reconstruction based on sub-point clouds improves the resolution of the scene model and ensures that the details of the objects displayed by the scene model are clear. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0031] Figure 1 A diagram of an application environment of a scene model reconstruction method in an embodiment;

[0032] Figure 2 A schematic diagram of a flow chart of a scene model reconstruction method in one embodiment;

[0033] Figure 3 This is a schematic diagram of the effect of each frame of radar point cloud in one embodiment;

[0034] Figure 4 A schematic diagram of the effect of gridding each frame of radar point cloud in one embodiment;

[0035] Figure 5 A schematic diagram of the effect of gridding each frame of radar point cloud in one embodiment;

[0036] Figure 6 A schematic diagram of a program module of a scene model reconstruction device in one embodiment;

[0037] Figure 7 is a structural diagram of a laser radar in one embodiment;

[0038] Figure 8 FIG. 4 is a structural diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0040] Figure 1 FIG. 1 is an application environment diagram of a scene model reconstruction method in an embodiment. Figure 1 , the scene model reconstruction method is applied to a scene model reconstruction system. The scene model reconstruction system includes a laser radar device 110 and a processor 120. The scene model reconstruction system can be applied to any product that needs to realize functions such as positioning, navigation, and path planning, such as outdoor vehicles that need to perform positioning and navigation or indoor mobile devices that need to perform positioning and navigation. The indoor mobile devices that need to perform positioning and navigation can specifically be sweeping robots, customer service robots, or portable dehumidifiers.

[0041] The laser radar device 110 can be a mobile laser radar scanning device in different forms, specifically a vehicle-mounted device, a backpack-type device, or a cart-type device, etc. The form of the laser radar device 110 is related to the form of the product it carries. The laser radar device 110 transmits the collected laser data to the processor 120 for data processing, and further realizes positioning, navigation, path planning and other functions according to the processing result of the laser data by the processor 120.

[0042] The processor 120 can be implemented using an independent processor 120 or a processor cluster composed of multiple processors 120, or it can be set in the laser radar device 110 and integrated with the laser radar device 110, which is not limited here.

[0043] In one embodiment, Figure 2 FIG. 1 is a flow chart of a scene model reconstruction method in an embodiment, referring to FIG. Figure 2 , provides a scene model reconstruction method. This embodiment mainly applies this method to the above Figure 1 Taking the processor 120 in the example as an example, the scene model reconstruction method specifically includes the following steps:

[0044] Step S210, respectively obtain point normals corresponding to multiple scanning points in a multi-frame radar point cloud corresponding to the scene to be reconstructed, wherein each frame of radar point cloud in the multi-frame radar point cloud is obtained by a laser radar by scanning a local scene of the scene to be reconstructed.

[0045] In this embodiment, the scene to be reconstructed may specifically be an indoor scene or an outdoor scene. During the dynamic scanning process of the laser radar device 110, each frame of radar point cloud refers to a set of scanning points obtained by scanning a preset coverage range at a position of the laser radar device 110. The preset coverage range is used to indicate the maximum scanning angle that the laser radar device 110 can cover by rotating in a horizontal plane, such as 60°, 180°, 360°, etc. For the laser radar device 110 scanning at 360°, each frame of radar point cloud refers to a set of scanning points obtained by the laser radar by scanning a local scene of the scene to be reconstructed, that is, a set of scanning points collected by the laser radar device 110 by scanning a circle at a position through horizontal rotation. The scanning points are used to represent the surface characteristics of the scanned object. A multi-frame radar point cloud refers to a set of scanning points collected by the laser radar device 110 scanning a circle at different times. Each frame of radar point cloud includes multiple scanning lines. Usually, the number of scanning lines is 16, 32, 64, etc. Each scanning line includes multiple scanning points. In the radar coordinate system, the scanning points will form multiple horizontal continuous scanning lines, such as Figure 3 As shown, since each frame of radar point cloud refers to a set of scanning points obtained by the laser radar device 110 scanning a position once, it is impossible to accurately restore the complete scene to be reconstructed. Therefore, each frame of radar point cloud only indicates a local scene, and multiple frames of radar point cloud need to be spliced ​​to reconstruct the complete scene.

[0046] The point cloud is sampled on the surface of the object, and the normal of the object surface is the point normal. Therefore, the geometry of the object surface can be estimated first, and then the point normal of the scanning point can be calculated. Generally, a low-order polynomial surface can be used for local fitting, or a plane can be used for local fitting, so as to determine the point normal of the scanning point through the plane normal or the plane normal.

[0047] Step S220: stitching multiple frames of radar point cloud to obtain a global radar point cloud.

[0048] In this embodiment, since the laser radar device 110 is in a real-time scanning state, there may be the same point clouds in two adjacent frames of radar point clouds. The splicing relationship between the two adjacent frames of radar point clouds can be determined based on the same point clouds. After the splicing is completed, the overlapping point clouds in the two adjacent frames of radar point clouds can be removed. In order to reduce the amount of calculation data in the subsequent model reconstruction process, since each frame of radar point cloud indicates a local scene, all point clouds corresponding to the scene to be reconstructed are gathered by splicing multiple frames of radar point clouds.

[0049] Step S230: split the global radar point cloud into multiple sub-point clouds according to a preset reconstruction scene resolution.

[0050] In this embodiment, the global radar point cloud is split according to the bounding box algorithm. The bounding box size set in the bounding box algorithm is used to affect the resolution of the reconstructed scene model. The smaller the bounding box size, the higher the resolution of the scene model. The bounding box size can be customized according to actual needs (i.e., the preset reconstructed scene resolution), which is specifically determined by the computer memory size and the target resolution. In this embodiment, each frame of radar point cloud is split according to the bounding box size of 20m to obtain multiple split sub-point clouds. At the same time, the resolution of the scene model can be as high as 2cm, which is smaller than the nominal error of general laser radar (about 3cm).

[0051] Step S240: generating a scene model corresponding to the scene to be reconstructed based on the multiple sub-point clouds and the point normals corresponding to the scanning points.

[0052] In this embodiment, compared with the model generated by directly reconstructing the model with a dense three-dimensional point cloud, reconstructing the model with sub-point clouds as units improves the resolution of the scene model and ensures that the details of the objects displayed in the scene model are clear. In addition, the amount of data calculated by the processor 120 when reconstructing the model for the sub-point cloud is less than the amount of data calculated by the processor 120 when reconstructing the model for the dense three-dimensional point cloud. This further reduces the data processing pressure of the processor 120 for model reconstruction while ensuring the high resolution of the model.

[0053] In one embodiment, point normals corresponding to multiple scanning points in a multi-frame radar point cloud corresponding to a scene to be reconstructed are respectively obtained, including: gridding each frame of radar point cloud to obtain a preprocessed grid; extracting a first feature based on all the scanning points in the preprocessed grid; filtering the first feature to obtain a second feature including feature points, wherein the feature points are a set of scanning points that can constitute the preprocessed grid and are filtered; and determining point normals corresponding to the feature points.

[0054] Specifically, since the point cloud data collected by the laser radar device 110 is measured at a long distance compared to the depth map collected by the depth camera, the point cloud distribution is sparse, and the absolute noise level is large, the accuracy of the features extracted directly from the relatively sparse point cloud is low. Therefore, each frame of the sparsely distributed radar point cloud is gridded and each frame of the sparsely distributed radar point cloud is converted into a dense preprocessing grid to improve the accuracy of subsequent feature extraction. The feature here is the first feature mentioned above, and the first feature includes scanning points that can constitute a preprocessing grid, a plane formed by the scanning points, and the intersection line between any two planes.

[0055] Each frame of radar point cloud is meshed, and specifically, triangular meshing or quadrilateral meshing can be used. In this embodiment, each frame of radar point cloud is meshed. Figure 3 As shown in the figure, the points inside each scan line are densely distributed and the resolution is high, but the gaps between the scan lines are large and the data is sparse. Therefore, it is necessary to triangulate the gaps between every two scan lines so that each frame of radar point cloud forms a triangular mesh. Figure 4 This is the front view after meshing each frame of radar point cloud. Figure 5 The side view after the frame radar point cloud is meshed, such as Figure 4 and Figure 5 As shown in the figure, the sparsely distributed point cloud is converted into a dense triangular mesh, and the pre-processed mesh based on the triangular mesh can extract more accurate local features, thereby improving the accuracy of model reconstruction based on accurate local features. It should be noted that each frame of radar point cloud after gridding reduces the amount of data compared to each frame of radar point cloud that has not been gridded, which naturally reduces the amount of calculation data involved in subsequent model reconstruction, speeds up the subsequent model reconstruction process, and improves the efficiency of model reconstruction.

[0056] Furthermore, based on extracting the second feature from the preprocessed grid after gridding, model reconstruction is performed to generate a grid model. Compared with the point cloud model represented by a dense three-dimensional point cloud, the grid model can be constructed with only a few feature points and the lines between the feature points. The data requirement is relatively small and the amount of computational data in the model reconstruction process is reduced.

[0057] In one embodiment, the first feature is filtered, that is, the extracted plane is Laplace filtered to filter out some noise points before point cloud stitching and model reconstruction to obtain the second feature. The second feature includes the filtered plane. The noise of the plane can be reduced by filtering the plane. Based on the plane with reduced noise, more reliable intersection lines and scanning points can be obtained naturally. That is, the second feature also includes a set of filtered scanning points and intersection lines. The scanning points here are the above-mentioned feature points, so as to improve the accuracy of subsequent point cloud stitching, and then improve the accuracy of model reconstruction. Furthermore, by gridding each frame of radar point cloud, not only the effect of the sparsity of lidar data on the reconstruction of fine grid models is solved, but also the lidar data has a strong anti-noise ability.

[0058] It should be understood that the traditional method is to use PCA fitting to extract point normals on the stitched point cloud, which is easily affected by noise and stitching errors, making the point normals inaccurate and resulting in poor Poisson reconstruction. Based on the filtered second feature, the point normals determined subsequently are more accurate and reliable.

[0059] In one embodiment, multiple frames of radar point clouds are stitched together to obtain a global radar point cloud, including: determining a transformation relationship between each frame of radar point cloud and a corresponding preceding radar point cloud; and stitching multiple frames of radar point clouds together to generate a global radar point cloud based on the transformation relationship between each frame of radar point cloud and a corresponding preceding radar point cloud.

[0060] Specifically, the preceding radar point cloud is the preceding radar point cloud of each frame of radar point cloud. The transformation relationship between each frame of radar point cloud and the preceding radar point cloud is used to indicate the position and posture of each frame of radar point cloud relative to the preceding radar point cloud, so that the change of each frame of radar point cloud relative to the preceding radar point cloud can be understood. According to the position and posture between each frame of radar point cloud and its corresponding preceding radar point cloud, the coordinate system of each frame of radar point cloud is converted to the same coordinate system, thereby realizing the splicing of the global radar point cloud. It should be noted that, in this embodiment, the global radar point cloud can also be filtered to remove noise points in the global radar point cloud. Usually, the local density of the point cloud in the global radar point cloud can be calculated by a statistical method (statistical outlier removal), and the points whose local density is lower than the density threshold are filtered out, so as to realize point cloud denoising.

[0061] In one embodiment, determining the transformation relationship between each frame of radar point cloud and the corresponding preceding radar point cloud includes: determining the Euclidean distance between a second feature in each frame of radar point cloud and a third feature in the preceding radar point cloud; determining a matching pair consisting of the second feature and the third feature based on the Euclidean distance; substituting a representation vector of the matching pair into an error function to obtain a function result; and determining the transformation relationship between each frame of radar point cloud and the preceding radar point cloud according to the function result.

[0062] Specifically, the third feature is used to indicate the feature extracted from the preceding radar point cloud. Similarly, the third feature includes scanning points, surfaces formed by scanning points, and intersection lines between surfaces. In this embodiment, the Euclidean distance between the scanning points in the second feature and the scanning points in the third feature is determined for explanation. The Euclidean distance between the second feature and the third feature is calculated to determine the matching degree between the two features. The smaller the Euclidean distance, the higher the matching degree between the two features. The matching degree between the two features can also be determined based on the difference between the normal of the second feature and the normal of the third feature. The higher the difference in normals, the lower the matching degree between the two features. Each second feature in each frame of radar point cloud will calculate the Euclidean distance with each third feature in the preceding radar point cloud, and the second feature with the smallest Euclidean distance will be selected to form a matching pair with the corresponding third feature, that is, each second feature will find a third feature with the smallest Euclidean distance, and multiple matching pairs will be obtained. The representation vector of the matching pair includes the feature vector of the second feature and the feature vector of the third feature. The representation vector of the matching pair is substituted into the error function, where the minimization of mean square error function is adopted. The minimization of mean square error function is iteratively calculated using the representation vectors of multiple pairs of matching pairs until the function result of minimizing the mean square error function is the minimum value, and the position and posture of each frame of radar point cloud relative to the preceding radar point cloud is obtained, thereby realizing the transformation relationship between each frame of radar point cloud and the preceding radar point cloud based on the SLAM algorithm.

[0063] In one embodiment, the transformation relationship between each frame of radar point cloud and the preceding radar point cloud is determined according to the function result, including: determining the point cloud pose according to the function result; determining the point cloud registration accuracy error estimate according to the point cloud pose; integrating positioning information according to the point cloud registration accuracy error estimate to determine the transformation relationship between each frame of radar point cloud and the preceding radar point cloud.

[0064] Specifically, in the previous embodiment, the transformation relationship between each frame of radar point cloud and the preceding radar point cloud is determined only based on the radar point cloud, but the accuracy of the transformation relationship obtained in this way is not high. Therefore, in this embodiment, the radar point cloud is combined with positioning information to determine the transformation relationship between each frame of radar point cloud and the preceding radar point cloud, thereby improving the reliability of the obtained transformation relationship.

[0065] Furthermore, the point cloud pose determined according to the function result is used to indicate the transformation relationship between each frame of radar point cloud and the previous adjacent radar point cloud by relying solely on the radar point cloud; the point cloud registration accuracy error estimate determined according to the point cloud pose is represented by the first covariance matrix; the positioning information is used for INS / GNSS (satellite / inertial navigation system) information indication, that is, used to indicate the position of the lidar device 110.

[0066] According to the first covariance matrix, a second covariance matrix for representing the integration accuracy estimation of point cloud pose and positioning information is calculated; the point cloud pose credibility is determined according to the second covariance matrix and the third covariance matrix generated by the point cloud pose; the positioning information credibility is determined according to the second covariance matrix and the fourth covariance matrix generated by the positioning information; the target pose is generated according to the point cloud pose, the point cloud pose credibility, the positioning information and the positioning information credibility integration, that is, the target pose represents the transformation relationship between each frame of radar point cloud and the previous adjacent radar point cloud, and the trajectory drift of the point cloud pose is guaranteed to be within a controllable range by integrating the positioning information, thereby improving the reliability of the transformation relationship between each frame of radar point cloud and the previous adjacent radar point cloud, and improving the point cloud stitching accuracy based on the accurate target pose, thereby optimizing the reconstruction effect after model reconstruction.

[0067] In one embodiment, a scene model corresponding to the scene to be reconstructed is generated based on multiple sub-point clouds and point normals corresponding to the scanning points, including: reconstructing the sub-point clouds based on the point normals corresponding to the feature points in the sub-point clouds to generate a local mesh model; filtering the local mesh model to obtain a filtered local mesh model; performing feature-preserving simplification processing on the filtered local mesh model to obtain a local optimized model; and superimposing the local optimized mesh models corresponding to each sub-point cloud to generate the scene model corresponding to the scene to be reconstructed.

[0068] Specifically, reconstructing the model based on sub-point clouds can not only ensure the high resolution of the model, but also reduce the data processing pressure of the processor 120 when reconstructing the model. In addition, the local grid model obtained by reconstructing the model based on the gridded sub-point clouds reduces the data volume of the model compared to the point cloud model, and is more convenient for mobile storage and transmission.

[0069] The local mesh model is filtered to remove noise points in the local mesh model. Usually, the Laplace filter method can be used to remove noise. The feature-preserving simplification process is used to further simplify the data volume of the local mesh model to obtain a local optimization model. The local optimization models corresponding to each sub-point cloud are superimposed to obtain the most lightweight scene model. This can be achieved through the QEM (Quadric Error Metric) mesh simplification method.

[0070] In one embodiment, after the local mesh model after filtering is simplified with preserved features to obtain a local optimized model, the method further includes: when an image sequence is acquired, texture mapping is performed on the local optimized model according to the image sequence to obtain a local optimized model with mapping; the local optimized mesh models corresponding to each sub-point cloud are superimposed to generate a scene model corresponding to the scene to be reconstructed, that is, the local optimized models with mapping corresponding to each sub-point cloud are superimposed to generate a scene model corresponding to the scene to be reconstructed.

[0071] Specifically, in order to achieve better reconstruction effect, the acquired image sequence is mapped to the local optimization model to obtain a local optimization model with color mapping, and the local optimization models with color mapping are superimposed to obtain a scene model corresponding to the scene to be reconstructed. Combining the above embodiments, the scene model finally obtained not only solves the burden of large data volume of large scene point cloud models, but also provides a relatively lightweight and detailed scene model for scene modeling of applications such as autonomous driving and AR navigation.

[0072] Figure 2 FIG. 1 is a flow chart of a scene model reconstruction method in one embodiment. It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0073] In one embodiment, Figure 6 As shown, a scene model reconstruction device is provided, comprising:

[0074] An acquisition module 310 is used to respectively acquire point normals corresponding to a plurality of scanning points in a multi-frame radar point cloud corresponding to the scene to be reconstructed, wherein each frame of the radar point cloud in the multi-frame radar point cloud is obtained by a laser radar scanning a local scene of the scene to be reconstructed;

[0075] A stitching module 320 is used to stitch multiple frames of radar point cloud to obtain a global radar point cloud;

[0076] A splitting module 330 is used to split the global radar point cloud into multiple sub-point clouds according to a preset reconstruction scene resolution;

[0077] The model reconstruction module 340 is used to generate a scene model corresponding to the scene to be reconstructed based on the multiple sub-point clouds and the point normals corresponding to the scanning points.

[0078] In one embodiment, the acquisition module 310 is further configured to:

[0079] Grid each frame of radar point cloud to obtain a pre-processed grid;

[0080] Extracting the first feature based on all scan points in the preprocessing network;

[0081] Performing filtering on the first feature to obtain a second feature; wherein the second feature includes a feature point;

[0082] Determine the point normal corresponding to the feature point.

[0083] In one embodiment, the splicing module 320 is further used for:

[0084] Determine a transformation relationship between each frame of radar point cloud and a corresponding preceding radar point cloud; wherein the preceding radar point cloud is a previous frame of radar point cloud of each frame of radar point cloud;

[0085] Based on the transformation relationship between each frame of radar point cloud and its corresponding preceding radar point cloud, each frame of radar point cloud is spliced ​​to generate a global radar point cloud.

[0086] In one embodiment, the splicing module 320 is further configured to:

[0087] Determine the Euclidean distance between the second feature in each frame of radar point cloud and the third feature in the previous adjacent radar point cloud;

[0088] Based on the Euclidean distance, determining a matching pair consisting of the second feature and the third feature;

[0089] Substituting the representation vector of the matching pair into the error function to obtain a function result; wherein the representation vector of the matching pair includes the feature vector of the second feature and the feature vector of the third feature;

[0090] The transformation relationship between each frame of radar point cloud and the previous adjacent radar point cloud is determined according to the function result.

[0091] In one embodiment, the splicing module 320 is further used for:

[0092] Determine the point cloud pose based on the function result;

[0093] Determine the point cloud registration accuracy error estimate based on the point cloud pose;

[0094] The positioning information is integrated according to the point cloud registration accuracy error estimation to determine the transformation relationship between each frame of radar point cloud and the previous adjacent radar point cloud.

[0095] In one embodiment, the model reconstruction module 340 is further configured to:

[0096] Reconstruct the sub-point cloud model based on the point normals corresponding to the feature points in the sub-point cloud to generate a local mesh model;

[0097] Performing filtering processing on the local grid model to obtain the filtered local grid model;

[0098] Perform feature-preserving simplification on the filtered local mesh model to obtain a local optimized model;

[0099] The local optimized mesh models corresponding to each sub-point cloud are superimposed to generate a scene model corresponding to the scene to be reconstructed.

[0100] In one embodiment, the model reconstruction module 340 is further configured to:

[0101] When the image sequence is acquired, texture mapping is performed on the local optimization model according to the image sequence to obtain the local optimization model with the mapping;

[0102] The local optimized mesh models corresponding to each sub-point cloud are superimposed to generate a scene model corresponding to the scene to be reconstructed, including:

[0103] The local optimized models with maps corresponding to the sub-point clouds are superimposed to generate the scene model corresponding to the scene to be reconstructed.

[0104] Figure 7 The internal structure of a laser radar in one embodiment is shown. The laser radar may be Figure 1 The laser radar device 110 in the embodiment of the present invention comprises a collection device 401, a point cloud conversion device 402 and Figure 6 The scene model reconstruction device 403 shown in the figure, wherein the acquisition device 401 includes any one of a depth camera, a 3D scanner, etc., for acquiring multiple frames of images of the scene to be reconstructed; the point cloud conversion device 402 is used to convert the multiple frames of images of the scene to be reconstructed into multiple frames of radar point clouds; the scene model reconstruction device 403 is used to use any of the above methods to realize the generation of a scene model corresponding to the scene to be reconstructed from the multiple frames of radar point clouds. It should be noted that the point cloud conversion device 402 and the scene model reconstruction device 403 can be set independently or as an integrated device, which is not limited here.

[0105] Figure 8 The internal structure diagram of an electronic device in one embodiment is shown. The electronic device may be Figure 1 The processor 120 in FIG. Figure 7 As shown, the electronic device includes a processor 120 and a memory 130 connected via a system bus. Among them, the memory 130 includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the electronic device stores an operating system and may also store a computer program, which, when executed by the processor 120, enables the processor 120 to implement the scene model reconstruction method. The internal memory may also store a computer program, which, when executed by the processor 120, enables the processor 120 to execute the scene model reconstruction method.

[0106] Those skilled in the art will understand that Figure 8 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0107] In one embodiment, the scene model reconstruction device provided by the present application can be implemented in the form of a computer program. The computer program can be Figure 8 The memory of the electronic device may store various program modules constituting the scene model reconstruction device, for example, Figure 6 The acquisition module 310, the splicing module 320, the splitting module 330 and the model reconstruction module shown in the figure. The computer program composed of each program module enables the processor 120 to execute the steps in the scene model reconstruction method of each embodiment of the present application described in this specification.

[0108] Figure 8 The electronic device shown can be Figure 6 The acquisition module 310 in the scene model reconstruction device shown in the figure is used to respectively acquire point normals corresponding to multiple scanning points in the multi-frame radar point cloud corresponding to the scene to be reconstructed. The electronic device can execute splicing of the multi-frame radar point cloud through the splicing module 320 to obtain a global radar point cloud. The electronic device can execute splitting of the global radar point cloud into multiple sub-point clouds through the splitting module 330. The electronic device can execute generation of a scene model corresponding to the scene to be reconstructed based on the multiple sub-point clouds and the point normals corresponding to the scanning points through the model reconstruction module.

[0109] In one embodiment, an electronic device is provided, including a memory, a processor 120, and a computer program stored in the memory and executable on the processor 120, wherein the processor 120 implements the method described in any of the above embodiments when executing the computer program.

[0110] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor 120, the method described in any of the above embodiments is implemented.

[0111] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0112] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0113] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A scene model reconstruction method, characterized in that: The method comprises: Respectively obtain point normals corresponding to a plurality of scanning points in a multi-frame radar point cloud corresponding to the scene to be reconstructed, wherein each frame of radar point cloud in the multi-frame radar point cloud is obtained by a laser radar by scanning a local scene of the scene to be reconstructed; Splicing the multiple frames of radar point clouds to obtain a global radar point cloud of the scene to be reconstructed; Splitting the global radar point cloud into a plurality of sub-point clouds according to a preset reconstruction scene resolution; Based on the plurality of sub-point clouds and point normals corresponding to the scanning points, a scene model corresponding to the scene to be reconstructed is generated.

2. The method according to claim 1, characterized in that The step of respectively obtaining point normals corresponding to a plurality of scanning points in a multi-frame radar point cloud corresponding to the scene to be reconstructed comprises: Grid each frame of radar point cloud to obtain a pre-processed grid; Extracting a first feature based on all the scanning points in the preprocessing grid; wherein the first feature includes scanning points that can constitute the preprocessing grid, a plane formed by the scanning points, and an intersection line between any two planes; Performing filtering processing on the first feature to obtain a second feature including feature points, wherein the feature points are a set of scanning points that can constitute the preprocessed grid and have been filtered, and the second feature includes a filtered plane, a filtered scanning point, and a set of intersection lines; Determine the point normal corresponding to the feature point.

3. The method according to claim 2, characterized in that The step of stitching the multiple frames of radar point clouds to obtain a global radar point cloud includes: Determine a transformation relationship between each frame of radar point cloud and a corresponding preceding radar point cloud; wherein the preceding radar point cloud is a previous frame of radar point cloud of each frame of radar point cloud; Based on the transformation relationship between each frame of radar point cloud and the corresponding preceding radar point cloud, the multiple frames of radar point clouds are spliced ​​to generate the global radar point cloud.

4. The method according to claim 3, characterized in that The determining of the transformation relationship between each frame of radar point cloud and the corresponding preceding radar point cloud comprises: Determine the Euclidean distance between the second feature in each frame of the radar point cloud and the third feature in the preceding radar point cloud; wherein the third feature is a feature extracted from the preceding radar point cloud, and the third feature includes scanning points, a surface formed by scanning points, and intersection lines between surfaces; Based on the Euclidean distance, determining a matching pair consisting of the second feature and the third feature; Substituting the representation vector of the matching pair into the error function to obtain a function result; wherein the representation vector of the matching pair includes the feature vector of the second feature and the feature vector of the third feature; The transformation relationship between each frame of radar point cloud and the preceding radar point cloud is determined according to the function result.

5. The method according to claim 4, characterized in that The step of determining the transformation relationship between each frame of radar point cloud and the preceding radar point cloud according to the function result includes: Determine the point cloud pose according to the function result; Determining a point cloud registration accuracy error estimate based on the point cloud pose; The positioning information is integrated according to the point cloud registration accuracy error estimation to determine the transformation relationship between each frame of radar point cloud and the preceding radar point cloud.

6. The method according to claim 5, characterized in that The step of generating a scene model corresponding to the scene to be reconstructed based on the plurality of sub-point clouds and the point normals corresponding to the scanning points comprises: Reconstructing the sub-point cloud based on the point normals corresponding to the feature points in the sub-point cloud to generate a local mesh model; Performing filtering processing on the local grid model to obtain the filtered local grid model; Performing feature-preserving simplification processing on the filtered local grid model to obtain a local optimized model; The local optimization models corresponding to the sub-point clouds are superimposed to generate the scene model corresponding to the scene to be reconstructed.

7. The method according to claim 6, characterized in that After performing feature-preserving simplification processing on the filtered local mesh model to obtain a local optimized model, the method further includes: When the image sequence is acquired, texture mapping is performed on the local optimization model according to the image sequence to obtain the local optimization model with the texture; The step of superimposing the local optimization models corresponding to the sub-point clouds to generate the scene model corresponding to the scene to be reconstructed includes: The local optimization models with maps corresponding to the sub-point clouds are superimposed to generate the scene model corresponding to the scene to be reconstructed.

8. A scene model reconstruction device, characterized in that: The device comprises: An acquisition module, used to respectively acquire point normals corresponding to a plurality of scanning points in a multi-frame radar point cloud corresponding to a scene to be reconstructed, wherein each frame of radar point cloud in the multi-frame radar point cloud is obtained by a laser radar by scanning a local scene of the scene to be reconstructed; A stitching module, used for stitching the multiple frames of radar point cloud to obtain a global radar point cloud; A splitting module, used for splitting the global radar point cloud into multiple sub-point clouds according to a preset reconstruction scene resolution; The model reconstruction module is used to generate a scene model corresponding to the scene to be reconstructed based on the plurality of sub-point clouds and the point normals corresponding to the scanning points.

9. A laser radar, characterized in that: It comprises a collection device, a point cloud conversion device and the scene model reconstruction device according to claim 8, wherein: The acquisition device is used to acquire multiple frames of images of the scene to be reconstructed; The point cloud conversion device is used to convert the multiple frames of images of the scene to be reconstructed into multiple frames of radar point clouds; The scene reconstruction device generates a scene model corresponding to the scene to be reconstructed using the multiple frames of radar point clouds.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images