Method and device for completing sparse map point cloud based on remote sensing image assistance

Abstract

The application relates to a sparse map point cloud defect completion method and device based on remote sensing image assistance, wherein the method comprises the following steps: acquiring a defective ground point cloud to be completed, and extracting a remote sensing image covering a current area; a projection image with a ground center point as a perspective is generated; corresponding point-by-point elevation information of the remote sensing image is extracted, and the point-by-point elevation information is geometrically converted to generate a remote sensing re-projection image; the density and defects of the superimposed image are monitored to obtain a point cloud defect distribution mask; three-dimensional point cloud data of a region to be completed is generated, and the three-dimensional point cloud data of the region to be completed and the defective ground point cloud to be completed are fused to generate a complete map point cloud. Therefore, the problems in the prior art that the real ground terrain distribution is complex, it is difficult to have a specific symmetry, the point cloud completion method based on geometric priori is difficult to be applied to a real world environment, and a ground point cloud with higher precision and completeness cannot be obtained are solved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and apparatus for completing sparse map point clouds based on remote sensing image assistance. Background Technology

[0002] Point cloud completion is mainly used to process and reconstruct sparse or incomplete point cloud data, generating a complete and accurate 3D model based on existing point cloud data.

[0003] In related technologies, geometric prior methods are usually used, which require the target point cloud to meet a certain geometric pairing or geometric distribution. Then, the extracted point cloud features are copied and expanded, and the point cloud data is restored to make up for the defects and holes in the map point cloud.

[0004] However, in related technologies, the completed point cloud portion has low realism, and the distribution of real ground terrain is complex and difficult to have specific symmetry. Point cloud completion methods based on geometric priors are difficult to apply to real-world environments and cannot obtain more accurate and complete ground point clouds, which urgently need to be improved. Summary of the Invention

[0005] This application provides a method and apparatus for sparse map point cloud incompleteness completion based on remote sensing image assistance, in order to solve the problems in related technologies, such as the low realism of the completed point cloud, the complex distribution of real ground terrain, the difficulty in having specific symmetry, the difficulty in applying point cloud completion methods based on geometric priors to real-world environments, and the inability to obtain more accurate and complete ground point clouds.

[0006] The first aspect of this application provides a method for sparse map point cloud incompleteness completion based on remote sensing image assistance, comprising the following steps: acquiring the incomplete ground point cloud to be completed and extracting the remote sensing image covering the current area; reprojecting the panoramic image corresponding to the incomplete ground point cloud to be completed to generate a projection map with the ground center point as the viewpoint; extracting the point-by-point elevation information corresponding to the remote sensing image and performing geometric transformation on the point-by-point elevation information to generate a remote sensing reprojection image; superimposing the projection map and the remote sensing reprojection image to obtain a superimposed image, and monitoring the density and defects of the superimposed image to obtain a point cloud defect distribution mask; and based on the point cloud defect distribution mask, reconstructing the image information in the remote sensing reprojection image through point cloud 3D reconstruction to generate 3D point cloud data of the area to be completed, and fusing the 3D point cloud data of the area to be completed and the incomplete ground point cloud to be completed to generate a complete map point cloud.

[0007] Optionally, in one embodiment of this application, the step of generating a projection map with the ground center point as the viewpoint by reprojecting the panoramic image corresponding to the incomplete ground point cloud to be completed includes: taking the center point of the incomplete ground point cloud to be completed as the origin of the world coordinate system and taking the preset coordinates as the target point in the world coordinate system; reprojecting the incomplete ground point cloud to be completed onto the cylindrical panoramic image plane according to at least one of the camera parameters, namely the rotation matrix and translation vector of the panoramic camera intrinsic parameters and panoramic camera extrinsic parameters; and extracting the homogeneous coordinates of the reprojected point cloud to obtain the reprojected cylindrical panoramic image, so as to generate the projection map.

[0008] Optionally, in one embodiment of this application, the step of geometrically transforming the point-by-point elevation information to generate a remote sensing reprojection image includes: converting the point-by-point elevation information and the RGB pixel information in the remote sensing image to obtain a three-dimensional geographic model within the coverage area of ​​the remote sensing image; using the point in the three-dimensional geographic model corresponding to the center point of the remote sensing image as the origin, and using the panoramic camera intrinsic parameters, the extrinsic parameter rotation matrix, and the translation vector as camera parameters to reproject the three-dimensional geographic model to obtain a reprojected three-dimensional geographic model; extracting the homogeneous coordinates of the reprojected three-dimensional geographic model, and obtaining the panoramic image corresponding to the reprojected elevation remote sensing image to generate the remote sensing reprojection image.

[0009] Optionally, in one embodiment of this application, the step of superimposing the projection map and the remote sensing reprojection image to obtain a superimposed image, and monitoring the density and defects of the superimposed image to obtain a point cloud defect distribution mask, includes: superimposing the projection map of the point cloud reprojection and the remote sensing reprojection image to generate a first parameter matrix; performing channel conversion on the first parameter matrix to convert the first parameter matrix into a second parameter matrix, and using any element in the second parameter matrix as an attention coefficient to generate an attention coefficient matrix; generating a defect difference matrix based on the projection map and the remote sensing reprojection image, and averaging the defect difference matrix and the remote sensing reprojection image to generate a relative defect difference matrix and an extreme value distribution matrix; constructing a correlation coefficient matrix between the relative defect interpolation and the extreme value distribution matrix, multiplying the attention coefficient matrix and the correlation coefficient matrix to generate an attention-weighted first defect distribution, and performing channel conversion on the first defect distribution to generate a second defect distribution.

[0010] Optionally, in one embodiment of this application, the step of generating 3D point cloud data of the region to be completed by reconstructing image information from the remote sensing reprojection image based on the point cloud defect distribution mask, and fusing the 3D point cloud data of the region to be completed with the incomplete ground point cloud to generate a complete map point cloud includes: constructing a completion target binary mask based on a defect threshold and the second defect distribution; obtaining a point cloud completion source matrix according to the completion target binary mask and the remote sensing reprojection image, and calculating the point cloud completion source matrix according to the inverse matrix of the camera parameter matrix to obtain the 3D point cloud data of the region to be completed; and superimposing the 3D point cloud data of the region to be completed with the incomplete ground point cloud to generate the completed map point cloud.

[0011] A second aspect of this application provides a sparse map point cloud incompleteness completion device based on remote sensing image assistance, comprising: an acquisition module for acquiring incomplete ground point cloud to be completed and extracting remote sensing images covering the current area; a first generation module for reprojecting a panoramic image corresponding to the incomplete ground point cloud to be completed, generating a projection map with the ground center point as the viewpoint; a conversion module for extracting point-by-point elevation information corresponding to the remote sensing image and performing geometric transformation on the point-by-point elevation information to generate a remote sensing reprojection image; a monitoring module for superimposing the projection map and the remote sensing reprojection image to obtain a superimposed image, and monitoring the density and defects of the superimposed image to obtain a point cloud defect distribution mask; and a second generation module for generating three-dimensional point cloud data of the area to be completed by reconstructing image information in the remote sensing reprojection image through three-dimensional reconstruction of the point cloud based on the point cloud defect distribution mask, and fusing the three-dimensional point cloud data of the area to be completed and the incomplete ground point cloud to be completed to generate a complete map point cloud.

[0012] Optionally, in one embodiment of this application, the first generation module includes: a first generation unit, configured to take the center point of the incomplete ground point cloud to be completed as the origin of the world coordinate system, and take the preset coordinates as the target point in the world coordinate system, reproject the incomplete ground point cloud to be completed onto the cylindrical panoramic image plane according to at least one of the camera parameters, namely the rotation matrix and translation vector of the panoramic camera intrinsic parameters and panoramic camera extrinsic parameters, and extract the homogeneous coordinates of the reprojected point cloud to obtain the reprojected cylindrical panoramic image, so as to generate the projection map.

[0013] Optionally, in one embodiment of this application, the conversion module includes: a conversion unit, configured to convert the point-by-point elevation information and the RGB pixel information in the remote sensing image to obtain a three-dimensional geographic model within the coverage area of ​​the remote sensing image; an acquisition unit, configured to take the point in the three-dimensional geographic model corresponding to the center point of the remote sensing image as the origin, and take the panoramic camera intrinsic parameters, the extrinsic parameter rotation matrix, and the translation vector as camera parameters to reproject the three-dimensional geographic model to obtain a reprojected three-dimensional geographic model; and a second generation unit, configured to extract the homogeneous coordinates of the reprojected three-dimensional geographic model and acquire the panoramic image corresponding to the reprojected elevation remote sensing image to generate the remote sensing reprojection image.

[0014] Optionally, in one embodiment of this application, the monitoring module includes: an overlay unit, used to overlay the projection map of the point cloud reprojection and the remote sensing reprojection image to generate a first parameter matrix; a third generation unit, used to perform channel conversion on the first parameter matrix, converting the first parameter matrix into a second parameter matrix, and using any element in the second parameter matrix as an attention coefficient to generate an attention coefficient matrix; a fourth generation unit, used to generate a defect difference matrix based on the projection map and the remote sensing reprojection image, and to calculate the mean of the defect difference matrix and the remote sensing reprojection image to generate a relative defect difference matrix and an extreme value distribution matrix; and a construction unit, used to construct a correlation coefficient matrix between the relative defect interpolation and the extreme value distribution matrix, multiply the attention coefficient matrix and the correlation coefficient matrix to generate an attention-weighted first defect distribution, and perform channel conversion on the first defect distribution to generate a second defect distribution.

[0015] Optionally, in one embodiment of this application, the second generation module includes: a construction unit, configured to construct a binary mask for the completion target based on a defect threshold and the second defect distribution; a calculation unit, configured to obtain a point cloud completion source matrix based on the binary mask for the completion target and the remote sensing reprojection image, and calculate the point cloud completion source matrix based on the inverse matrix of the camera parameter matrix to obtain the three-dimensional point cloud data of the area to be completed; and a fifth generation unit, configured to superimpose the three-dimensional point cloud data of the area to be completed and the incomplete ground point cloud to be completed to generate the completed map point cloud.

[0016] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the remote sensing image-assisted sparse map point cloud incompleteness completion method as described in the above embodiments.

[0017] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for completing sparse map point clouds based on remote sensing image assistance.

[0018] This application's embodiments utilize information provided by remote sensing images to complete incomplete and hollow ground map point clouds. The point cloud and remote sensing image are reprojected onto a cylindrical ground projection. The remote sensing image is used to obtain the true information of the missing parts of the ground point cloud, and geometric transformation is performed to obtain a realistically accurate incomplete point cloud. Ultimately, a more accurate and complete ground point cloud is obtained. This solves the problems in related technologies, such as the low realism of the completed point cloud, the complexity of real-world terrain distribution, the difficulty in achieving specific symmetry, and the difficulty in applying geometrically prior point cloud completion methods to real-world environments, thus failing to obtain more accurate and complete ground point clouds.

[0019] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0020] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0021] Figure 1 This is a flowchart of a sparse map point cloud incompleteness completion method based on remote sensing image assistance provided in an embodiment of this application;

[0022] Figure 2 This is a flowchart of a sparse map point cloud incompleteness completion method based on remote sensing image assistance according to an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of a sparse map point cloud incompleteness completion device based on remote sensing image assistance provided in an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0025] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0026] The following describes, with reference to the accompanying drawings, a method and apparatus for sparse map point cloud incompleteness completion based on remote sensing image assistance, according to embodiments of this application. Addressing the issues raised in the background art, where the completed point cloud portion exhibits low realism, and given the complex distribution of real-world terrain and the difficulty in achieving specific symmetry, point cloud incompleteness methods based on geometric priors are difficult to apply to real-world environments, failing to obtain more accurate and complete ground point clouds. This application provides a method for sparse map point cloud incompleteness completion based on remote sensing image assistance. In this method, incomplete and hollow ground map point clouds can be completed using information provided by remote sensing images. The point cloud and remote sensing image are reprojected onto a ground cylindrical projection, utilizing the remote sensing image to obtain the true information of the incomplete ground point cloud portion, and performing geometric transformations to obtain a realistically accurate incomplete point cloud, ultimately achieving a more accurate and complete ground point cloud. This solves the problems in related technologies, such as the low realism of the completed point cloud, the complexity of the real ground terrain distribution, the difficulty in achieving specific symmetry, the difficulty of applying point cloud completion methods based on geometric priors to real-world environments, and the inability to obtain more accurate and complete ground point clouds.

[0027] Specifically, Figure 1 This is a flowchart illustrating a method for completing sparse map point clouds based on remote sensing image assistance, provided in an embodiment of this application.

[0028] like Figure 1 As shown, the method for completing sparse map point clouds based on remote sensing image assistance includes the following steps:

[0029] In step S101, the incomplete ground point cloud to be completed is obtained, and the remote sensing image covering the current area is extracted.

[0030] In actual implementation, the embodiments of this application can obtain the incomplete ground point cloud to be completed, denoted as P, and the size of the incomplete ground point cloud data to be completed is n×3, where n is the number of points in the ground point cloud; the embodiments of this application can extract remote sensing images that can cover the entire point cloud of the current area from a public remote sensing database, denoted as S, and extract the corresponding point-by-point elevation data Z (in image form) from the public database.

[0031] The embodiments of this application can acquire incomplete ground point clouds to be completed and extract remote sensing images covering the current area, thereby facilitating the completion of incomplete and hollow ground map point clouds by using information provided by remote sensing images, and obtaining complete and uniform map point clouds.

[0032] In step S102, a projection map with the ground center point as the viewpoint is generated based on the panoramic image reprojection corresponding to the incomplete ground point cloud to be completed.

[0033] In actual implementation, the embodiments of this application can perform panoramic image reprojection on the incomplete ground point cloud to be completed. Based on the panoramic image reprojection corresponding to the incomplete ground point cloud to be completed, a projection map with the ground center point as the viewpoint can be generated, which provides support for obtaining a more accurate and complete ground point cloud in the future.

[0034] Optionally, in one embodiment of this application, generating a projection map with the ground center point as the viewpoint based on the panoramic image reprojection corresponding to the incomplete ground point cloud to be completed includes: taking the center point of the incomplete ground point cloud to be completed as the origin of the world coordinate system and taking the preset coordinates as the target point in the world coordinate system; reprojecting the incomplete ground point cloud to be completed onto the cylindrical panoramic image plane according to at least one of the camera parameters in the rotation matrix and translation vector of the panoramic camera intrinsic parameters and panoramic camera extrinsic parameters; and extracting the homogeneous coordinates of the reprojected point cloud to obtain the reprojected cylindrical panoramic image to generate the projection map.

[0035] It is understood that the preset coordinates in the embodiments of this application may be, but are not limited to, n×3 coordinates.

[0036] As one possible implementation, this embodiment of the application can use the center point of the incomplete ground point cloud to be completed as the origin of the world coordinate system, and use the n×3 coordinates as the target point in the world coordinate system. The panoramic camera's intrinsic parameter is set to K, the rotation matrix of the panoramic camera's extrinsic parameter is R, and the translation vector is T. Based on the camera parameters such as the panoramic camera's intrinsic parameter K, the rotation matrix R, and the translation vector T, the incomplete ground point cloud to be completed is reprojected onto the cylindrical panoramic image plane. The projected point cloud is denoted as P. RP =K(RP+T), the embodiments of this application can extract the homogeneous coordinates of the reprojected point cloud to obtain the reprojected panoramic image I of the cylinder. RP This generates projection maps, which further supports the subsequent acquisition of more accurate and complete ground point clouds.

[0037] It should be noted that the preset coordinates can be set by those skilled in the art according to the actual situation, and no specific restrictions are imposed here.

[0038] In step S103, point-by-point elevation information corresponding to the remote sensing image is extracted, and the point-by-point elevation information is geometrically transformed to generate a remote sensing reprojection image.

[0039] In actual implementation, the embodiments of this application can obtain the elevation information corresponding to each point of the remote sensing image from the database, obtain the point-by-point elevation information, perform geometric transformation on the point-by-point elevation information, obtain the reprojection of the ground panoramic image of the corresponding location, and generate a remote sensing reprojection image, thereby facilitating the acquisition of the true information of the incomplete parts of the ground point cloud.

[0040] Optionally, in one embodiment of this application, the point-by-point elevation information is geometrically transformed to generate a remote sensing reprojection image, including: converting the point-by-point elevation information and the RGB pixel information in the remote sensing image to obtain a three-dimensional geographic model within the coverage area of ​​the remote sensing image; using the point in the three-dimensional geographic model corresponding to the center point of the remote sensing image as the origin, and using the rotation matrix and translation vector of the panoramic camera's intrinsic and extrinsic parameters as camera parameters, the three-dimensional geographic model is reprojected to obtain a reprojected three-dimensional geographic model; the homogeneous coordinates of the reprojected three-dimensional geographic model are extracted to obtain a panoramic image corresponding to the reprojected elevation remote sensing image, thereby generating a remote sensing reprojection image.

[0041] In actual implementation, this embodiment can convert the point-by-point elevation data Z and the corresponding RGB (Red, Green, Blue) pixel information in the remote sensing image S to obtain a 3D geographic model M in the form of a triangular mesh within the coverage area of ​​the remote sensing image. This embodiment can use the point in the 3D geographic model corresponding to the center point of the remote sensing image as the origin, and use the same panoramic camera intrinsic parameters K, extrinsic parameters R, and translation vector T as the camera parameters in step S102 above, to reproject the 3D geographic model, obtaining the reprojected 3D geographic model. The reprojected 3D geographic model in the form of a triangular mesh is denoted as M. RP =K(RM+T), extract the homogeneous coordinates of the 3D geographic model in the form of a triangular mesh from the reprojection, and generate the panoramic image I corresponding to the reprojected elevation remote sensing image. RS This refers to remote sensing reprojection images, which further facilitates the acquisition of true information about incomplete parts of the ground point cloud, and helps to complete incomplete and hollow ground map point clouds, resulting in a complete and uniform map point cloud.

[0042] In step S104, the projection map and the remote sensing reprojection image are superimposed to obtain a superimposed image, and the density and defects of the superimposed image are monitored to obtain a point cloud defect distribution mask.

[0043] In actual implementation, the embodiments of this application can overlay the projection map and the remote sensing reprojection image to obtain an overlay image, and perform density and defect detection based on the overlay image to obtain a point cloud defect distribution mask, which is beneficial for completing the incomplete and hollow ground map point cloud.

[0044] Optionally, in one embodiment of this application, a point cloud defect distribution mask is obtained by overlaying a projection map and a remote sensing reprojection image, and monitoring the density and defects of the overlay image. This includes: overlaying the projection map of the point cloud reprojection and the remote sensing reprojection image to generate a first parameter matrix; performing channel transformation on the first parameter matrix to convert it into a second parameter matrix, and using any element in the second parameter matrix as an attention coefficient to generate an attention coefficient matrix; generating a defect difference matrix based on the projection map and the remote sensing reprojection image, and averaging the defect difference matrix and the remote sensing reprojection image to generate a relative defect difference matrix and an extreme value distribution matrix; constructing a correlation coefficient matrix between the relative defect interpolation and the extreme value distribution matrix, multiplying the attention coefficient matrix and the correlation coefficient matrix to generate an attention-weighted first defect distribution, and performing channel transformation on the first defect distribution to generate a second defect distribution.

[0045] It is understood that the projection image in the embodiments of this application can be a reprojected panoramic image of a cylinder; the remote sensing reprojection image in the embodiments of this application can be a panoramic image corresponding to an elevation remote sensing image.

[0046] As one possible implementation, embodiments of this application can use the reprojected panoramic image I of the cylinder obtained from the point cloud reprojection in step S102 above. RP That is, the projection image and the panoramic image I corresponding to the elevation remote sensing image in step S103 above. RS That is, the remote sensing reprojection images are superimposed, where I RP and I RS The image size is H×W×3, and the images are superimposed to obtain a first parameter matrix with a size of H×W×6, denoted as F0. In this embodiment, the first parameter matrix F0 can be transformed based on the image generation network to convert the first parameter matrix with a size of H×W×6 into a second parameter matrix with a size of H×W×1, and any element in the second parameter matrix can be used as an attention coefficient to generate an attention coefficient matrix.

[0047] Furthermore, in this embodiment of the application, the reprojected panoramic image I of the cylinder obtained from the point cloud reprojection in step S102 above can be further processed. RP That is, the projection image and the panoramic image I corresponding to the elevation remote sensing image in step S103 above. RS That is, by subtracting the remote sensing reprojection images, a defect difference matrix F2 = I with size H×W×3 is obtained. RS -I RP The defect difference matrix F2 and the remote sensing reprojection image I are used to... RS The average of the three values ​​in the third dimension is calculated to obtain two parameter matrices of size H×W×1, which represent the relative defect difference matrix F3 and the extreme value distribution matrix F4, respectively.

[0048] In this embodiment, a correlation coefficient matrix between the relative defect difference matrix F3 and the extreme value distribution matrix F4 can be constructed as follows:

[0049]

[0050] Where Cov(F3,F4) represents the covariance of F3 and F4, Var(F3) represents the variance of F3, and Var(F4) represents the variance of F4;

[0051] In this embodiment, the attention coefficient matrix F1 can be multiplied by the correlation coefficient matrix r(F3,F4) to obtain the attention-weighted first defect distribution:

[0052] F5 = F1 × r(F3, F4),

[0053] The dimensions of F5 are H×W×1;

[0054] Furthermore, in this embodiment, another image generation network can be constructed to perform channel transformation on the defect distribution F5 through mapping optimization to obtain a second defect distribution F6 with the same number of channels as the input F5, and the size is also H×W×1, which is beneficial for further completing the incomplete and hollow ground map point cloud.

[0055] In step S105, based on the point cloud defect distribution mask, the three-dimensional point cloud data of the area to be completed is generated by using the image information in the remote sensing reprojection image of the three-dimensional reconstruction of the point cloud, and the three-dimensional point cloud data of the area to be completed and the incomplete ground point cloud to be completed are fused to generate a complete map point cloud.

[0056] In actual implementation, the embodiments of this application can use a point cloud defect distribution mask to perform three-dimensional point cloud reconstruction on areas with high attention values ​​based on image information in remote sensing reprojection images, generate three-dimensional point cloud data of the area to be completed, and fuse the three-dimensional point cloud data of the area to be completed with the incomplete ground point cloud to obtain a complete map point cloud, thereby ensuring that the map point cloud is obtained with higher accuracy and more completeness.

[0057] Optionally, in one embodiment of this application, based on a point cloud defect distribution mask, three-dimensional point cloud data of the region to be completed is generated by using image information from the remote sensing reprojection image reconstructed from the point cloud three-dimensional reconstruction. The three-dimensional point cloud data of the region to be completed and the incomplete ground point cloud to be completed are then fused to generate a complete map point cloud. This includes: constructing a binary mask for the completion target based on a defect threshold and a second defect distribution; obtaining a point cloud completion source matrix based on the binary mask for the completion target and the remote sensing reprojection image, and calculating the point cloud completion source matrix based on the inverse matrix of the camera parameter matrix to obtain the three-dimensional point cloud data of the region to be completed; and superimposing the three-dimensional point cloud data of the region to be completed and the incomplete ground point cloud to be completed to generate the completed map point cloud.

[0058] It is understood that the defect threshold in the embodiments of this application can be set to δ.

[0059] In actual implementation, this embodiment of the application can set a defect threshold δ, and construct a binary mask for the completion target based on δ and the defect distribution F6 in step S104 above. The construction method is as follows:

[0060] Mask = F6 < δ,

[0061] Mask is also an H×W×1 parameter matrix, where a value of 0 indicates that the corresponding pixel does not need to be padded, and a value of 1 indicates that the corresponding pixel needs to be padded.

[0062] This application embodiment can complete the target binary mask Mask and the panoramic image I corresponding to the elevation remote sensing image in step S103 above. RS That is, the remote sensing reprojection images are superimposed and multiplied to obtain the point cloud completion source matrix Mask·I. RS The 3D point cloud data P of the region to be filled is obtained by calculating the inverse matrix of the camera parameter matrix used for reprojection. lost The specific calculation method is as follows:

[0063] P lost =(Mask·I RS ·K -1 -T)R -1 ,

[0064] This application embodiment can obtain the 3D point cloud data P of the region to be completed. lost The incomplete ground point cloud P to be completed in step S101 is overlaid to obtain the completed point cloud data P. com =P l0st +P.

[0065] The embodiments of this application can overlay the three-dimensional point cloud data of the area to be completed and the incomplete ground point cloud to be completed, so as to complete the incomplete and hollow ground map point cloud and obtain a complete and uniform map point cloud.

[0066] Specifically, it can be combined with Figure 2 As shown, a specific embodiment is used to elaborate in detail on the working principle of the sparse map point cloud incompleteness completion method based on remote sensing image assistance in the embodiments of this application.

[0067] like Figure 2 As shown, embodiments of this application may include the following steps:

[0068] Step S201: Input the incomplete ground point cloud to be completed, and extract remote sensing images that can cover the area from the remote sensing database.

[0069] Step S202: Perform panoramic image reprojection on the ground point cloud to obtain a projection map with the ground center point as the viewpoint.

[0070] Step S203: Obtain the elevation information corresponding to each point of the remote sensing image from the database, and perform geometric transformation to obtain the reprojection of the ground panoramic image of the corresponding location.

[0071] Step S204: Overlay the point cloud reprojection and remote sensing reprojection images, and perform density and defect detection based on the overlaid images to obtain a point cloud defect attention distribution mask.

[0072] Step S205: Based on the point cloud defect attention distribution mask, for areas with high attention values, perform point cloud 3D reconstruction based on image information in the remote sensing reprojection image, and fuse it with the original incomplete ground point cloud to obtain a complete point cloud map.

[0073] The sparse map point cloud incompleteness completion method based on remote sensing image assistance proposed in this application can complete incomplete and hollow ground map point clouds by using information provided by remote sensing images. The point cloud and remote sensing image are reprojected onto a cylindrical ground projection, and the real information of the incomplete parts of the ground point cloud is obtained using the remote sensing image. Geometric transformation is then performed to obtain a ground point cloud that conforms to reality, ultimately resulting in a more accurate and complete ground point cloud. This solves the problem in related technologies where the completed point cloud has low realism, and the complex distribution of real ground terrain makes it difficult to achieve specific symmetry. Point cloud completion methods based on geometric priors are difficult to apply to real-world environments and cannot obtain more accurate and complete ground point clouds.

[0074] Next, referring to the accompanying drawings, a sparse map point cloud incompleteness completion device based on remote sensing image assistance proposed in the embodiments of this application is described.

[0075] Figure 3This is a schematic diagram of the sparse map point cloud incompleteness completion device based on remote sensing image assistance according to an embodiment of this application.

[0076] like Figure 3 As shown, the sparse map point cloud incompleteness completion device 10 based on remote sensing image assistance includes: acquisition module 100, first generation module 200, conversion module 300, monitoring module 400, and second generation module 500.

[0077] Specifically, the acquisition module 100 is used to acquire the incomplete ground point cloud to be completed and extract the remote sensing image covering the current area.

[0078] The first generation module 200 is used to generate a projection map with the ground center point as the viewpoint by reprojecting the panoramic image corresponding to the incomplete ground point cloud to be completed.

[0079] The conversion module 300 is used to extract point-by-point elevation information corresponding to the remote sensing image and perform geometric transformation on the point-by-point elevation information to generate a remote sensing reprojection image.

[0080] The monitoring module 400 is used to overlay the projection map and the remote sensing reprojection image to obtain the overlay image, and to monitor the density and defects of the overlay image to obtain a point cloud defect distribution mask.

[0081] The second generation module 500 is used to generate three-dimensional point cloud data of the area to be completed based on the point cloud defect distribution mask and the image information in the remote sensing reprojection image reconstructed from the point cloud three-dimensional reconstruction, and to fuse the three-dimensional point cloud data of the area to be completed and the incomplete ground point cloud to be completed to generate a complete map point cloud.

[0082] Optionally, in one embodiment of this application, the first generation module 200 includes: a first generation unit.

[0083] The first generation unit is used to take the center point of the incomplete ground point cloud to be completed as the origin of the world coordinate system and the preset coordinates as the target point in the world coordinate system. Based on at least one of the camera parameters, namely the rotation matrix and translation vector of the panoramic camera intrinsic parameters and panoramic camera extrinsic parameters, the incomplete ground point cloud to be completed is reprojected onto the cylindrical panoramic image plane, and the homogeneous coordinates of the reprojected point cloud are extracted to obtain the reprojected cylindrical panoramic image, so as to generate the projection map.

[0084] Optionally, in one embodiment of this application, the conversion module 300 includes: a conversion unit, an acquisition unit, and a second generation unit.

[0085] The conversion unit is used to convert point-by-point elevation information and RGB pixel information in remote sensing images to obtain a three-dimensional geographic model within the coverage area of ​​the remote sensing image.

[0086] The acquisition unit is used to take the point in the 3D geographic model corresponding to the center point of the remote sensing image as the origin, and use the rotation matrix and translation vector of the panoramic camera's intrinsic and extrinsic parameters as camera parameters to reproject the 3D geographic model, thereby obtaining the reprojected 3D geographic model.

[0087] The second generation unit is used to extract the homogeneous coordinates of the reprojected 3D geographic model and obtain the panoramic image corresponding to the reprojected elevation remote sensing image in order to generate a remote sensing reprojection image.

[0088] Optionally, in one embodiment of this application, the monitoring module 400 includes: an overlay unit, a third generation unit, a fourth generation unit, and a construction unit.

[0089] The overlay unit is used to overlay the projection map of the point cloud reprojection and the remote sensing reprojection image to generate the first parameter matrix.

[0090] The third generation unit is used to perform channel transformation on the first parameter matrix, converting the first parameter matrix into a second parameter matrix, and using any element in the second parameter matrix as an attention coefficient to generate an attention coefficient matrix.

[0091] The fourth generation unit is used to generate a defect difference matrix based on the projection map and the remote sensing reprojection image, and to calculate the mean of the defect difference matrix and the remote sensing reprojection image to generate a relative defect difference matrix and an extreme value distribution matrix.

[0092] The construction unit is used to construct the correlation coefficient matrix between the relative defect interpolation and the extreme value distribution matrix, multiply the attention coefficient matrix and the correlation coefficient matrix to generate the attention-weighted first defect distribution, and perform channel transformation on the first defect distribution to generate the second defect distribution.

[0093] Optionally, in one embodiment of this application, the second generation module 500 includes: a construction unit, a calculation unit, and a fifth generation unit.

[0094] The construction unit is used to construct a binary mask for completing the target based on the defect threshold and the second defect distribution.

[0095] The calculation unit is used to obtain the point cloud completion source matrix based on the binary mask of the completion target and the remote sensing reprojection image, and to calculate the point cloud completion source matrix based on the inverse matrix of the camera parameter matrix, so as to obtain the three-dimensional point cloud data of the region to be completed.

[0096] The fifth generation unit is used to overlay the 3D point cloud data of the area to be completed and the incomplete ground point cloud to be completed, and generate the complete map point cloud.

[0097] It should be noted that the foregoing explanation of the embodiment of the sparse map point cloud incompleteness completion method based on remote sensing image assistance also applies to the sparse map point cloud incompleteness completion device based on remote sensing image assistance in this embodiment, and will not be repeated here.

[0098] The sparse map point cloud incompleteness completion device based on remote sensing image assistance proposed in this application can complete incomplete and hollow ground map point clouds by using information provided by remote sensing images. The point cloud and remote sensing image are reprojected onto a cylindrical ground projection. The remote sensing image is used to obtain the true information of the incomplete parts of the ground point cloud, and geometric transformation is performed to obtain a realistic incomplete point cloud, ultimately resulting in a more accurate and complete ground point cloud. This solves the problem in related technologies where the completed point cloud has low realism, and the complex distribution of real ground terrain makes it difficult to achieve specific symmetry. Point cloud completion methods based on geometric priors are difficult to apply to real-world environments, thus failing to obtain more accurate and complete ground point clouds.

[0099] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0100] The memory 401, the processor 402, and the computer program stored on the memory 401 and capable of running on the processor 402.

[0101] When the processor 402 executes the program, it implements the sparse map point cloud incompleteness completion method based on remote sensing image assistance provided in the above embodiments.

[0102] Furthermore, electronic devices also include:

[0103] Communication interface 403 is used for communication between memory 401 and processor 402.

[0104] The memory 401 is used to store computer programs that can run on the processor 402.

[0105] The memory 401 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0106] If the memory 401, processor 402, and communication interface 403 are implemented independently, then the communication interface 403, memory 401, and processor 402 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0107] Optionally, in a specific implementation, if the memory 401, processor 402, and communication interface 403 are integrated on a single chip, then the memory 401, processor 402, and communication interface 403 can communicate with each other through an internal interface.

[0108] Processor 402 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0109] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for completing sparse map point clouds based on remote sensing image assistance.

[0110] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0111] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0112] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0113] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0114] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0115] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0116] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0117] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for completing incomplete sparse map point clouds based on remote sensing image assistance, characterized in that, Includes the following steps: Acquire the incomplete ground point cloud to be completed and extract the remote sensing image covering the current area; Based on the panoramic image reprojection corresponding to the incomplete ground point cloud to be completed, a projection map with the ground center point as the viewpoint is generated. Extract the point-by-point elevation information corresponding to the remote sensing image, and perform geometric transformation on the point-by-point elevation information to generate a remote sensing reprojection image; The projection image and the remote sensing reprojection image are superimposed to obtain a superimposed image, and the density and defects of the superimposed image are monitored to obtain a point cloud defect distribution mask; as well as Based on the point cloud defect distribution mask, image information in the remote sensing reprojection image is extracted through point cloud 3D reconstruction to generate 3D point cloud data of the area to be completed, and the 3D point cloud data of the area to be completed and the incomplete ground point cloud to be completed are fused to generate a complete map point cloud. The process of superimposing the projection image and the remote sensing reprojection image to obtain a superimposed image, and monitoring the density and defects of the superimposed image to obtain a point cloud defect distribution mask, includes: By overlaying the point cloud reprojection image and the remote sensing reprojection image, a first parameter matrix is ​​generated. This first parameter matrix includes the following: the image size of both the point cloud reprojection image and the remote sensing reprojection image is [missing information]. The size obtained by superposition is The first parameter matrix; The first parameter matrix is ​​channel-transformed into a second parameter matrix, and any element in the second parameter matrix is ​​used as an attention coefficient to generate an attention coefficient matrix. The transformation of the first parameter matrix into the second parameter matrix includes: converting the first parameter matrix into a matrix of size [missing information]. The first parameter matrix is ​​converted to a size of The second parameter matrix; A defect difference matrix is ​​generated based on the projected image and the remotely sensed reprojected image. The mean of the defect difference matrix and the remotely sensed reprojected image is then calculated to generate a relative defect difference matrix and an extreme value distribution matrix. The step of generating the defect difference matrix based on the projected image and the remotely sensed reprojected image, and then averaging the defect difference matrix and the remotely sensed reprojected image to generate the relative defect difference matrix and the extreme value distribution matrix, includes: subtracting the projected image from the remotely sensed reprojected image to obtain a matrix of size [value missing]. The defect difference matrix is ​​obtained by averaging the three values ​​in the third dimension of the defect difference matrix and the remote sensing reprojection image to obtain two values ​​of size . The parameter matrices represent the relative defect difference matrix and the extreme value distribution matrix, respectively; Construct a correlation coefficient matrix between the relative defect difference and the extreme value distribution matrix. Multiply the attention coefficient matrix and the correlation coefficient matrix to generate an attention-weighted first defect distribution. Perform channel transformation on the first defect distribution to generate a second defect distribution. The size of the first defect distribution is... The size of the second defect distribution is ; The process involves extracting image information from the remote sensing reprojection image based on the point cloud defect distribution mask through 3D point cloud reconstruction, generating 3D point cloud data for the area to be completed, and fusing the 3D point cloud data of the area to be completed with the incomplete ground point cloud to generate a complete map point cloud, including: Based on the defect threshold and the second defect distribution, a binary mask for completing the target is constructed; The point cloud completion source matrix is ​​obtained by multiplying the binary mask of the completion target with the remote sensing reprojection image, and the point cloud completion source matrix is ​​calculated based on the inverse matrix of the camera parameter matrix to obtain the three-dimensional point cloud data of the region to be completed. The three-dimensional point cloud data of the area to be completed and the incomplete ground point cloud to be completed are superimposed to generate the completed map point cloud.

2. The method according to claim 1, characterized in that, The step of reprojecting the panoramic image corresponding to the incomplete ground point cloud to be completed, generating a projection map with the ground center point as the viewpoint, includes: The center point of the incomplete ground point cloud to be completed is taken as the origin of the world coordinate system, and the preset coordinates are taken as the target point in the world coordinate system. The incomplete ground point cloud to be completed is reprojected onto the cylindrical panoramic image plane according to at least one of the camera parameters, namely the rotation matrix and translation vector of the panoramic camera intrinsic parameters and panoramic camera extrinsic parameters. The homogeneous coordinates of the reprojected point cloud are extracted to obtain the reprojected cylindrical panoramic image, so as to generate the projection map.

3. The method according to claim 2, characterized in that, The step of geometrically transforming the point-by-point elevation information to generate a remote sensing reprojection image includes: The point-by-point elevation information and the RGB pixel information in the remote sensing image are converted to obtain a three-dimensional geographic model within the coverage area of ​​the remote sensing image; Using the point in the 3D geographic model corresponding to the center point of the remote sensing image as the origin, and taking the rotation matrix and translation vector of the panoramic camera intrinsic parameters and extrinsic parameters as camera parameters, the 3D geographic model is reprojected to obtain the reprojected 3D geographic model. Extract the homogeneous coordinates of the reprojected 3D geographic model, obtain the panoramic image corresponding to the reprojected elevation remote sensing image, and generate the remote sensing reprojection image.

4. A sparse map point cloud incompleteness completion device based on remote sensing image assistance, employing the sparse map point cloud incompleteness completion method based on remote sensing image assistance as described in any one of claims 1-3, characterized in that, include: The acquisition module is used to acquire the incomplete ground point cloud to be completed and extract the remote sensing image covering the current area; The first generation module is used to generate a projection map with the ground center point as the viewpoint by reprojecting the panoramic image corresponding to the incomplete ground point cloud to be completed. The conversion module is used to extract the point-by-point elevation information corresponding to the remote sensing image and perform geometric transformation on the point-by-point elevation information to generate a remote sensing reprojection image. The monitoring module is used to overlay the projection image and the remote sensing reprojection image to obtain an overlay image, and to monitor the density and defects of the overlay image to obtain a point cloud defect distribution mask. as well as The second generation module is used to extract image information from the remote sensing reprojection image through point cloud 3D reconstruction based on the point cloud defect distribution mask, generate 3D point cloud data of the area to be completed, and fuse the 3D point cloud data of the area to be completed and the incomplete ground point cloud to be completed to generate a complete map point cloud.

5. The apparatus according to claim 4, characterized in that, The first generation module includes: The first generation unit is used to take the center point of the incomplete ground point cloud to be completed as the origin of the world coordinate system and the preset coordinates as the target point in the world coordinate system. Based on at least one camera parameter among the rotation matrix and translation vector of the panoramic camera intrinsic parameters and panoramic camera extrinsic parameters, the incomplete ground point cloud to be completed is reprojected onto the cylindrical panoramic image plane, and the homogeneous coordinates of the reprojected point cloud are extracted to obtain the reprojected cylindrical panoramic image, so as to generate the projection map.

6. The apparatus according to claim 5, characterized in that, The conversion module includes: The conversion unit is used to convert the point-by-point elevation information and the RGB pixel information in the remote sensing image to obtain a three-dimensional geographic model within the coverage area of ​​the remote sensing image. The acquisition unit is used to take the point in the three-dimensional geographic model corresponding to the center point of the remote sensing image as the origin, and take the rotation matrix of the panoramic camera intrinsic parameters and the translation vector as camera parameters to reproject the three-dimensional geographic model to obtain the reprojected three-dimensional geographic model. The second generation unit is used to extract the homogeneous coordinates of the reprojected three-dimensional geographic model and obtain the panoramic image corresponding to the reprojected elevation remote sensing image to generate the remote sensing reprojection image.

7. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the remote sensing image-assisted sparse map point cloud incompleteness completion method as described in any one of claims 1-3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the remote sensing image-assisted sparse map point cloud incompleteness completion method as described in any one of claims 1-3.

Images