Heterogeneous point cloud-based point cloud hole repairing method, device, medium and equipment

Abstract

Embodiments of the present application provide a point cloud hole repairing method based on heterogeneous point clouds, comprising: inputting the to-be-repaired point cloud data and the first heterogeneous point cloud data into a machine learning model to output a corresponding first pose transformation matrix, and performing pose transformation processing on the first heterogeneous point cloud data to obtain second heterogeneous point cloud data; using an iterative closest point algorithm to match the to-be-repaired point cloud data and the second heterogeneous point cloud data, and performing pose transformation processing on the second heterogeneous point cloud data to obtain target heterogeneous point cloud data; identifying a to-be-repaired hole according to the position information of the points in the to-be-repaired point cloud data and the position information of the points in the target heterogeneous point cloud data; and repairing the to-be-repaired point cloud data according to the points in the target heterogeneous point cloud data corresponding to the to-be-repaired hole to obtain repaired target point cloud data. The technical solution of the embodiments of the present application improves the accuracy of hole repairing in the point cloud model, thereby ensuring the accuracy of the repaired point cloud model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a point cloud hole repairing method and device based on heterogeneous point clouds, a medium and an equipment. BACKGROUND

[0002] In the process of establishing a point cloud model, due to various reasons (such as sensor limitations, device operation processes or environmental complexity, etc.), the generated point cloud model is often incomplete and has various holes, which seriously affects the quality of the established point cloud model. In the current technical solution, the holes are generally repaired based on the assumption of interpolation, however, this kind of repair method has low robustness, and a single type of point cloud data cannot represent the rich local features in the environment due to the single description scale. Therefore, how to improve the accuracy of hole repair in the point cloud model and further ensure the accuracy of the repaired point cloud model has become a technical problem to be solved. SUMMARY

[0003] Embodiments of the present application provide a point cloud hole repairing method and device based on heterogeneous point clouds, a computer readable medium and an electronic device, which can at least improve the accuracy of hole repair in the point cloud model and further ensure the accuracy of the repaired point cloud model.

[0004] Other characteristics and advantages of the present application will become apparent from the following detailed description, or will be learned by practice of the present application.

[0005] According to an aspect of an embodiment of the present application, a point cloud hole repairing method based on heterogeneous point clouds is provided, comprising:

[0006] inputting the to-be-repaired point cloud data and the corresponding first heterogeneous point cloud data into a pre-trained machine learning model, so that the machine learning model outputs a corresponding first pose transformation matrix;

[0007] performing pose transformation processing on the first heterogeneous point cloud data according to the first pose transformation matrix to obtain second heterogeneous point cloud data, the second heterogeneous point cloud data being in the same coordinate system as the to-be-repaired point cloud data;

[0008] performing matching on the to-be-repaired point cloud data and the second heterogeneous point cloud data by using an iterative closest point algorithm to obtain a corresponding second pose transformation matrix;

[0009] performing pose transformation processing on the second heterogeneous point cloud data according to the second pose transformation matrix to obtain target heterogeneous point cloud data;

[0010] According to the position information of the points in the to-be-repaired point cloud data and the position information of the points in the target heterogeneous point cloud data, a to-be-repaired hole existing in the to-be-repaired point cloud data is identified.

[0011] According to the points in the target heterogeneous point cloud data corresponding to the to-be-repaired hole, the to-be-repaired point cloud data is repaired to obtain repaired target point cloud data.

[0012] According to an aspect of an embodiment of the present application, a point cloud hole repairing device based on heterogeneous point clouds is provided, comprising:

[0013] A first calculation module is configured to input to-be-repaired point cloud data and first heterogeneous point cloud data corresponding thereto into a machine learning model trained in advance, so that the machine learning model outputs a corresponding first pose conversion matrix.

[0014] A first pose conversion module is configured to perform pose conversion processing on the first heterogeneous point cloud data according to the first pose conversion matrix to obtain second heterogeneous point cloud data, wherein the second heterogeneous point cloud data is in the same coordinate system as the to-be-repaired point cloud data.

[0015] A second calculation module is configured to match the to-be-repaired point cloud data and the second heterogeneous point cloud data using an iterative closest point algorithm to obtain a corresponding second pose conversion matrix.

[0016] A second pose conversion module is configured to perform pose conversion processing on the second heterogeneous point cloud data according to the second pose conversion matrix to obtain target heterogeneous point cloud data.

[0017] A hole identification module is configured to identify a to-be-repaired hole existing in the to-be-repaired point cloud data according to the position information of the points in the to-be-repaired point cloud data and the position information of the points in the target heterogeneous point cloud data.

[0018] A repairing module is configured to repair the to-be-repaired point cloud data according to the points in the target heterogeneous point cloud data corresponding to the to-be-repaired hole to obtain repaired target point cloud data.

[0019] According to an aspect of an embodiment of the present application, a computer readable medium having a computer program stored thereon is provided, wherein the computer program is executed by a processor to implement the point cloud hole repairing method based on heterogeneous point clouds as described in the above embodiments.

[0020] According to an aspect of some embodiments of the present application, an electronic device is provided, including: one or more processors; a storage device configured to store one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the point cloud hole repairing method based on heterogeneous point clouds as described in the above embodiments.

[0021] According to an aspect of some embodiments of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the point cloud hole repairing method based on heterogeneous point clouds provided in the above embodiments.

[0022] In the technical solutions provided in some embodiments of the present application, the point cloud data to be repaired and the first heterogeneous point cloud data corresponding thereto are input into a machine learning model that has been pre-trained, so that the machine learning model outputs a first pose transformation matrix, and the first heterogeneous point cloud data is subjected to pose transformation processing according to the first pose transformation matrix, to obtain second heterogeneous point cloud data in the same coordinate system as the point cloud data to be repaired. Then, the iterative closest point algorithm is used to match the point cloud data to be repaired and the second heterogeneous point cloud data, to obtain a second pose transformation matrix, and the second heterogeneous point cloud data is subjected to pose transformation processing according to the second pose transformation matrix, to obtain target heterogeneous point cloud data. It should be understood that the matching degree of the target heterogeneous point cloud data obtained after two times of pose transformation and the point cloud data to be repaired is higher, and thus the accuracy of the hole to be repaired identified from the position information of the points in the point cloud data to be repaired and the position information of the points in the target heterogeneous point cloud data is higher, which ensures the accuracy of subsequent hole repairing, and further improves the accuracy of the repaired point cloud data.

[0023] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0024] The drawings incorporated in the specification and constituting a part of the specification illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings. In the drawings:

[0025] Figure 1Fig. 1 shows a flowchart of a point cloud hole repairing method based on heterogeneous point clouds according to an example embodiment of the present application;

[0026] Figure 2 Fig. 2 shows a flowchart of a hole identification for building plane included in the point cloud hole repairing method based on heterogeneous point clouds according to an example embodiment of the present application;

[0027] Figure 3 Fig. 3 shows a flowchart of a point cloud hole repairing method based on heterogeneous point clouds according to an example embodiment of the present application; Figure 1 Fig. 4 shows a flowchart of constructing a first heterogeneous point cloud data included in the point cloud hole repairing method based on heterogeneous point clouds according to an example embodiment of the present application;

[0028] Figure 4 Fig. 5 shows a block diagram of a point cloud hole repairing apparatus based on heterogeneous point clouds according to an example embodiment of the present application;

[0029] Figure 5 Fig. 6 shows a structural diagram of a computer system of an electronic device suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0030] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations can be implemented in any

[0031] Moreover, described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the

[0032] The block diagrams in the drawings show only the functionality of the example embodiments and do not imply any particular physical or architectural arrangement of the example embodiments. No inference should be made regarding the architecture (i.e., software, firmware, hardware, circuitry, processor, etc.) of actual, yet-to-be-developed, implementations, products, devices, systems, or methods of the example embodiments based on the content of the drawings alone or based on the contents of any related documents that can have been filed with the present application. One skilled in the art will recognize that the functionality and layout of the example embodiments can be carried out in other ways not depicted, such as in different sequences of operations, using other structures, interchanges, combinations, and / or parallel operations.

[0033] The flowchart shown in the drawing is only an exemplary illustration, and is not necessarily required to include all contents and operations / steps, nor is it necessarily required to be executed in the order described. For example, some operations / steps can be further decomposed, and some operations / steps can be combined or partially combined, so the actual execution order can be changed according to actual conditions.

[0034] Figure 1 A flowchart of a point cloud hole repairing method based on heterogeneous point clouds according to an exemplary embodiment of the present application is shown. It should be noted that the method can be applied in a terminal device or a server, wherein the terminal device can be one or more of a smartphone, a tablet computer, a portable computer, or other electronic devices with data processing functions, and the server can be a physical server or a cloud server, etc., which are not particularly limited in the present application.

[0035] As shown in Figure 1 The point cloud hole repairing method based on heterogeneous point clouds at least includes steps S110 to S160, which are described in detail as follows:

[0036] In step S110, the point cloud data to be repaired and the corresponding first heterogeneous point cloud data are input into a machine learning model that has been pre-trained to output a corresponding first pose transformation matrix.

[0037] The point cloud data to be repaired can be point cloud data established by a laser radar or other means. Due to the complexity of the scene environment, the accuracy of the sensor, or the operation of the device, etc., holes often exist in the point cloud data, so it needs to be repaired.

[0038] The heterogeneous point cloud data can be point cloud data established in a different way or at a different angle than the point cloud data to be repaired. It should be understood that the point cloud data to be repaired and the heterogeneous point cloud data correspond to the same scene, such as the same room, etc., so that the point cloud data to be repaired can be repaired according to the heterogeneous point cloud data.

[0039] The pose transformation matrix can be a matrix used to adjust the pose of the heterogeneous point cloud data. It should be understood that the point cloud data to be repaired and the heterogeneous point cloud data can be established based on different coordinate systems, so the pose of the heterogeneous point cloud data needs to be adjusted by the pose transformation matrix to register the heterogeneous point cloud data with the point cloud data to be repaired. Specifically, the pose transformation matrix can include a rotation matrix and a translation vector. Thus, after adjustment by the pose transformation matrix, the matching degree of the heterogeneous point cloud data and the point cloud data to be repaired can be higher, avoiding the influence of subsequent repair processing due to different poses.

[0040] In an example embodiment of the present application, a machine learning model can be constructed and trained by a person skilled in the art, so that the machine learning model can output a corresponding pose transformation matrix according to the point cloud data of the same scene, to register two point cloud data. After obtaining the to-be-repaired point cloud data and the corresponding first heterogeneous point cloud data, they can be input into the trained machine learning model, so that the machine learning model can output a corresponding first pose transformation matrix to register the to-be-repaired point cloud data and the first heterogeneous point cloud data.

[0041] In an example, a person skilled in the art can construct a corresponding machine learning model based on "SpinNet: Learning a General Surface Descriptor for 3D Point Cloud Registration" for training; in other examples, a person skilled in the art can also establish a machine learning model based on other machine learning algorithms. A person skilled in the art can construct a corresponding machine learning model according to actual implementation needs, which is not specially limited in the present application.

[0042] In step S120, the first heterogeneous point cloud data is subjected to pose transformation processing according to the first pose transformation matrix, to obtain second heterogeneous point cloud data, which is in the same coordinate system as the to-be-repaired point cloud data.

[0043] In this step, based on the first pose transformation matrix, the three-dimensional coordinate information of each point in the first heterogeneous point cloud data is transformed, to obtain the second heterogeneous point cloud data. It should be understood that the second heterogeneous point cloud data after pose transformation processing can be in the same coordinate system as the to-be-repaired point cloud data, so as to realize registration of the to-be-repaired point cloud data and the first heterogeneous point cloud data.

[0044] In step S130, an iterative closest point algorithm is used to match the to-be-repaired point cloud data and the second heterogeneous point cloud data, to obtain a corresponding second pose transformation matrix.

[0045] The iterative closest point algorithm (ICP) can be used for point cloud registration in three-dimensional space, which is not only suitable for planar scenes, but also suitable for curved and curved surface scenes. By continuously reducing errors in iteration, an ideal accuracy value can be achieved. After the initial rough registration of the above step S120, a good initial pose is provided, and the iterative closest point algorithm can further reduce the error between the two, thereby improving the accuracy of subsequent processing.

[0046] In this step, the second pose transformation matrix corresponding to the to-be-repaired point cloud data and the second heterogeneous point cloud data can be obtained after the iterative closest point algorithm. It should be understood that, compared with the first pose transformation matrix, the second pose transformation matrix can realize more refined and accurate registration between the two point cloud data, so as to provide a better initial pose for subsequent processing.

[0047] Therefore, after registration by the machine learning model and the iterative closest point algorithm (coarse registration + fine registration), a better registration result can be obtained between the to-be-repaired point cloud data and the heterogeneous point cloud data, thereby providing a good foundation for subsequent hole identification and point cloud repair.

[0048] In step S140, the second heterogeneous point cloud data is subjected to pose transformation processing according to the second pose transformation matrix, to obtain target heterogeneous point cloud data.

[0049] In this step, the position information of each point in the second heterogeneous point cloud data is subjected to pose transformation processing based on the second pose transformation matrix, thereby obtaining the target heterogeneous point cloud data. It should be understood that the registration degree between the target heterogeneous point cloud data obtained after the second pose transformation processing and the to-be-repaired point cloud data is high, thereby improving the accuracy of subsequent hole identification processing and hole repair, and ensuring the accuracy of the repair result of the point cloud data.

[0050] In step S150, the to-be-repaired hole existing in the to-be-repaired point cloud data is identified according to the position information of the points in the to-be-repaired point cloud data and the position information of the points in the target heterogeneous point cloud data.

[0051] In this step, it should be understood that the registration degree between the target heterogeneous point cloud data and the to-be-repaired point cloud data is high, and the position information of the corresponding points is the same or similar. Therefore, the to-be-repaired hole existing in the to-be-repaired point cloud data can be identified according to the position information of the points in the to-be-repaired point cloud data and the position information of the points in the target heterogeneous point cloud data. For example, a region exists in the points of the heterogeneous point cloud data but does not exist in the points of the to-be-repaired point cloud data, and the region can be confirmed as the to-be-repaired hole.

[0052] Therefore, by comparing the to-be-repaired point cloud data and the target heterogeneous point cloud data after the second registration, the to-be-repaired hole existing in the to-be-repaired point cloud data can be accurately identified, thereby providing a good foundation for subsequent repair.

[0053] In step S160, the to-be-repaired point cloud data is repaired according to the points in the target heterogeneous point cloud data corresponding to the to-be-repaired hole, to obtain repaired target point cloud data.

[0054] In this step, since the identified hole to be repaired is a range of a region, after identifying the hole to be repaired contained in the point cloud data to be repaired, the points in the target heterogeneous point cloud data in the region information corresponding to the hole to be repaired can be determined as the points corresponding to the hole to be repaired according to the region information corresponding to the hole to be repaired. In an example, the point cloud data to be repaired can be repaired according to the position information of all points in the target heterogeneous point cloud data corresponding to the hole to be repaired, that is, the point cloud data to be repaired is interpolated according to the position information of the corresponding points in the target heterogeneous point cloud data, so as to repair the hole to be repaired, so as to perfect the point cloud data to be repaired, and obtain the repaired target point cloud data.

[0055] Therefore, based on the point cloud hole repair method based on heterogeneous point clouds shown in the figure, the registration degree of the target heterogeneous point cloud data obtained after two times of pose conversion and the point cloud data to be repaired is high, so that the accuracy of the hole to be repaired existing in the point cloud data to be repaired identified according to the position information of the points in the point cloud data to be repaired and the position information of the points in the target heterogeneous point cloud data is improved, the rationality of subsequent hole repair is ensured, and the accuracy of the repaired point cloud data is ensured. Figure 1 Therefore, based on the point cloud hole repair method based on heterogeneous point clouds shown in the figure, the registration degree of the target heterogeneous point cloud data obtained after two times of pose conversion and the point cloud data to be repaired is high, so that the accuracy of the hole to be repaired existing in the point cloud data to be repaired identified according to the position information of the points in the point cloud data to be repaired and the position information of the points in the target heterogeneous point cloud data is improved, the rationality of subsequent hole repair is ensured, and the accuracy of the repaired point cloud data is ensured.

[0056] Figure 1 Based on the point cloud hole repair method based on heterogeneous point clouds shown in the figure, in an example embodiment of the present application, the hole to be repaired existing in the point cloud data to be repaired is identified according to the position information of the points in the point cloud data to be repaired and the position information of the points in the target heterogeneous point cloud data, comprising:

[0057] According to the position information of the points in the point cloud data to be repaired and the position information of the points in the target heterogeneous point cloud data, if there are only points in the target heterogeneous point cloud data in a certain region with a certain size, and there are points in the point cloud data to be repaired and points in the target heterogeneous point cloud data in other regions adjacent to the certain region, or there are only points in the point cloud data to be repaired, the certain region is identified as the hole to be repaired existing in the point cloud data to be repaired.

[0058] ​In this embodiment, it should be understood that in the point cloud data, the point cloud data corresponding to each object should be continuous. Thus, the holes existing in the point cloud data to be repaired can be determined by comparing the position information of the points in the point cloud data to be repaired with the position information of the points in the target foreign point cloud data. Specifically, when identifying, if there are only points in the target foreign point cloud data in a certain region with a certain size, and there are both points in the point cloud data to be repaired and points in the target foreign point cloud data in the other regions adjacent to the region, or there are only points in the point cloud data to be repaired in the other regions adjacent to the region, it indicates that the region is a region where the point cloud data to be repaired is interrupted, i.e., a hole to be repaired.

[0059] Specifically, for the point cloud data to be repaired and the target foreign point cloud data, the three-dimensional space can be uniformly divided into a plurality of three-dimensional grids with a predetermined edge length, and it is detected whether each three-dimensional grid contains the two kinds of point cloud data. If the three-dimensional grid contains neither the point cloud data to be repaired nor the target foreign point cloud data, the three-dimensional grid is regarded as an invalid grid. If the three-dimensional grid contains the point cloud data to be repaired, the grid is marked as 0. If the three-dimensional grid contains no point cloud data to be repaired but contains foreign point cloud data, the three-dimensional grid is marked as 1.

[0060] After the marking of each three-dimensional grid is completed, since the point cloud data corresponding to each object is continuous, each three-dimensional grid marked as 1 should be adjacent to at least one three-dimensional grid marked as 0. Any three-dimensional grid marked as 1 can be taken as a starting point, and a three-dimensional region growing algorithm is used to search for all three-dimensional grids marked as 1 in the adjacent region until the region is surrounded by the three-dimensional grids marked as 0 and the invalid grids. The marking of the peripheral three-dimensional grids marked as 0 and the invalid grids is updated to 2. After the search is completed, all three-dimensional grids marked as 1 in the region can be regarded as a three-dimensional hole in which the point cloud data to be repaired exists, i.e., a hole to be repaired. The three-dimensional grids marked as 2 form the boundary of the three-dimensional hole.

[0061] Thus, the hole to be repaired existing in the point cloud data to be repaired can be accurately identified for subsequent hole repair.

[0062] It should be noted that if there is no three-dimensional grid marked as 0 in the adjacent region of a certain region, i.e., the region contains only the target foreign point cloud data, and the adjacent region contains no point cloud data to be repaired, the region is identified as a redundant part of the target foreign point cloud data relative to the point cloud data to be repaired, and is not processed.

[0063] Based on the foregoing embodiment, Figure 2 A flowchart for hole identification for a building plane included in a point cloud hole repair method based on foreign point clouds according to an example embodiment of the present application is shown.

[0064] Referring to Figure 2 As shown, the hole recognition for the building plane includes at least steps S210 to S220, which are described in detail as follows:

[0065] In step S210, the building plane is identified from the to-be-repaired point cloud data.

[0066] The building plane can be a wall surface, a floor, a ceiling, or the like. It should be understood that the building plane usually has a high flatness and a large area, and therefore, based on the position information of each point in the to-be-repaired point cloud data, the points corresponding to the building plane in the to-be-repaired point cloud data can be identified.

[0067] In step S220, in the building plane, if there is no point in the to-be-repaired point cloud data in a certain region with a certain size, the certain region is identified as a to-be-repaired hole existing in the to-be-repaired point cloud data.

[0068] In this step, because the points corresponding to the building plane are close to a plane in spatial distribution, it can be directly judged whether the points corresponding to the building plane in the to-be-repaired point cloud data appear to be interrupted in a certain region, without the need for comparison with the target heterogeneous point cloud data. That is, in the building plane, if there is a region without points of the to-be-repaired point cloud data, the region is identified as a to-be-repaired hole existing in the to-be-repaired point cloud data.

[0069] In an example, each building plane can be rotated to be parallel to the X-Y plane, and the points corresponding to the building plane can be projected onto the X-Y plane. The building plane can be divided into a plurality of two-dimensional grids, and if there is no point of the to-be-repaired point cloud data in a certain two-dimensional grid, the region corresponding to the two-dimensional grid is identified as a to-be-repaired hole existing in the building plane.

[0070] Thus, when the building plane is recognized, the hole recognition method shown in Figure 2 can improve the recognition efficiency of the hole and ensure the accuracy of the hole recognition, and reduce the computing resources occupied by the hole recognition.

[0071] In an example embodiment of the present application, when the hole recognition is performed, a deep learning algorithm can be used to divide the to-be-repaired point cloud data and the target heterogeneous point cloud data into a building body and a non-building body. For the building body, a method such as Figure 2The shown recognition method performs hole recognition, and for non-building bodies, the aforementioned three-dimensional grid division method can be used for recognition, thereby ensuring the accuracy of hole recognition and improving the efficiency of hole recognition. In an example, a corresponding deep learning algorithm can be established based on "PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds", or other algorithms can be used, which are not specially limited in the present application.

[0072] Further, when identifying a building body, the maximum and minimum points on the X axis and the maximum and minimum points on the Y axis can be determined based on the position information of the points corresponding to the building body, and then a minimum enclosing plane is determined, and the minimum enclosing plane is divided into a plurality of two-dimensional grids for hole recognition, thereby avoiding excessive useless calculation and reducing resource occupation.

[0073] Similarly, when identifying a non-building body, the maximum and minimum values on the three coordinate axes X, Y and Z can be determined to construct a minimum enclosing space, and then the minimum enclosing space is used to divide the three-dimensional grid for hole recognition.

[0074] It can be understood that by pre-determining the minimum enclosing plane and the minimum enclosing space, the range of hole recognition in the specified area is reduced, unnecessary calculation is avoided, the occupation of computing resources is greatly reduced, and the accuracy of hole recognition is improved.

[0075] Based on the foregoing embodiments, Figure 3 shows a point cloud hole repairing method based on heterogeneous point clouds according to an example embodiment of the present application. Figure 1 The point cloud hole repairing method based on heterogeneous point clouds according to an example embodiment of the present application further includes a flowchart for constructing first heterogeneous point cloud data.

[0076] Referring to Figure 3 As shown, constructing first heterogeneous point cloud data includes at least steps S310 to S320, which are described in detail as follows:

[0077] In step S310, image data corresponding to the same scene as the point cloud data to be repaired is obtained.

[0078] The image data can be an image or an image sequence composed of multiple images. The image data and the point cloud data to be repaired correspond to the same scene, for example, the point cloud data to be repaired corresponds to a conference room, and the image data is obtained by image acquisition of the conference room, etc.

[0079] In step S320, according to the image data, first heterogeneous point cloud data corresponding to the to-be-repaired point cloud data is constructed.

[0080] In this embodiment, according to the obtained image data, three-dimensional reconstruction can be performed based on “HSfM: Hybrid Structure-from-Motion”, thereby obtaining the first heterogeneous point cloud data which can be used to repair the to-be-repaired point cloud data.

[0081] In an example embodiment of the present application, since the three-dimensional reconstruction result can have outliers and noise points, which can greatly affect the accuracy of subsequent registration. Therefore, after constructing the first heterogeneous point cloud data corresponding to the to-be-repaired point cloud data, the first heterogeneous point cloud data can also be filtered and denoised. Specifically, a statistical filter can be used to denoise the first heterogeneous point cloud data, which can determine whether a point in a certain region is a noise point according to the size of the local point cloud density. It should be understood that the more dense the point, the more useful the point contained in the region, and vice versa, the more sparse the point, the more likely it contains noise points. Thus, by calculating the average distance S i of each point to its nearest k points, the average value and the standard deviation of all S i are obtained, and finally the noise points are removed according to the set threshold.

[0082] In an example embodiment of the present application, when identifying the to-be-repaired hole, the points contained in the to-be-repaired point cloud data and the target heterogeneous point cloud data can be overlapped to obtain a “large point cloud”. Before identifying the to-be-repaired hole for the “large point cloud”, the “large point cloud” can be smoothed by using the Moving Least Squares (MLS) method. Specifically, the point cloud can be mapped by MLS to make the “large point cloud” smoother. Especially after filtering and denoising, the surface fitting accuracy of the smoothed point cloud will be higher.

[0083] In an example, after identifying the to-be-repaired hole, the points corresponding to the to-be-repaired hole extracted from the target heterogeneous point cloud data can be subjected to density detection to detect whether the density is consistent with the size of the edge of the to-be-repaired hole, and the corresponding encryption or decimation method is adopted for adjustment. The extracted points in the target heterogeneous point cloud data can also be smoothed, thereby improving the quality of hole repair.

[0084] Based on the foregoing embodiments, in an example embodiment of the present application, before inputting the to-be-repaired point cloud data and the first heterogeneous point cloud data corresponding thereto into the machine learning model which has been pre-trained, the processing method further comprises:

[0085] According to the position information of the points in the to-be-repaired point cloud data and the position information of the points in the first heterogeneous point cloud data, a scaling multiple of the first heterogeneous point cloud data is calculated;

[0086] The first heterogeneous point cloud data is scaled according to the scaling multiple, to obtain scaled first heterogeneous point cloud data.

[0087] In this embodiment, it should be understood that, since the monocular camera-based HSfM method lacks absolute physical scale information, the first heterogeneous point cloud data constructed will be scaled to a certain extent compared with the real physical scale, which will cause the subsequent registration to fail. Therefore, the scaling multiple corresponding to the first heterogeneous point cloud data can be calculated according to the position information of the points in the to-be-repaired point cloud data and the position information of the points in the first heterogeneous point cloud data.

[0088] Specifically, the feature points corresponding to the two can be selected, such as corner points, to calculate the scaling multiple between the two. The calculation formula is as follows:

[0089]

[0090] Where (X1, Y1, Z1) and (X2, Y2, Z2) are the three-dimensional coordinates of two feature points P1 and P2 in the to-be-repaired point cloud data, and (X’1, Y’1, Z’1) and (X’2, Y’2, Z’2) are the three-dimensional coordinates of P’1 and P’2 corresponding to P1 and P2 in the first heterogeneous point cloud data. According to the formula, the scaling multiple S is:

[0091]

[0092] The physical meaning of the above method is to a set of matched line segments, and the scaling multiple of the set of point cloud data is obtained according to the ratio of the lengths of the two line segments. Since the to-be-repaired point cloud data itself has a real physical scale, the first heterogeneous point cloud data corresponding to the three-dimensional reconstruction is scaled by S times, and the real physical scale is obtained, so that the subsequent registration can be implemented.

[0093] Based on the foregoing embodiments, in an example embodiment of the present application, before the to-be-repaired point cloud data and the first heterogeneous point cloud data corresponding thereto are input into the machine learning model trained in advance, the method further includes:

[0094] According to the density information of the points in the to-be-repaired point cloud data, the density of the points in the first heterogeneous point cloud data corresponding thereto is adjusted, to obtain the first heterogeneous point cloud data after density adjustment.

[0095] In this embodiment, because the density of the point cloud data generated by the laser radar mapping algorithm is higher than that of the point cloud data generated by the three-dimensional reconstruction of the image data. Therefore, in order to improve the quality of the hole repair in the point cloud data to be repaired, the density of the points in the first heterogeneous point cloud data corresponding to the points in the point cloud data to be repaired can be adjusted based on the density information of the points in the point cloud data to be repaired, so that the density information of the two is matched.

[0096] Further, the first heterogeneous point cloud data and the point cloud data to be repaired after density adjustment are input into the machine learning model trained in advance, so that the machine learning model can output the first pose transformation matrix corresponding to the two. Therefore, because of the density adjustment, the density of the points in the first heterogeneous point cloud data and the point cloud data to be repaired is similar, which facilitates the identification of the machine learning model, thereby improving the accuracy of the output results of the machine learning model.

[0097] In an example, a linear triangular net interpolation method can be used to interpolate and encrypt the first heterogeneous point cloud data, so that the density of the points in the first heterogeneous point cloud data is consistent with the density of the points in the point cloud data to be repaired. The principle of this encryption method is based on the best Delaunay triangulation, connecting the points on the original data to form triangles, and making all the edges of the triangles not intersect with other triangles, thereby forming a triangular net composed of triangles. Each triangular plane defines an inner grid point plane, and uniform interpolation encryption is performed in the inner grid point plane.

[0098] Based on Figure 1 In an example embodiment of the present application, the point cloud hole repair method based on heterogeneous point clouds further includes:

[0099] Obtaining at least two training point cloud models of the same scene and different sources, and splitting the at least two training point cloud models into a plurality of point cloud segments of a predetermined size;

[0100] Selecting a plurality of pairs of training point cloud segments from the plurality of point cloud segments, the two training point cloud segments in each pair of training point cloud segments having an overlapping part, and the number of training point cloud segments corresponding to different training point cloud models in the plurality of pairs of training point cloud segments being the same;

[0101] Randomly transforming the training point cloud segments in each of the pairs of training point cloud segments, and inputting the pairs of training point cloud segments into a pre-constructed machine learning model for training, so that the machine learning model outputs a real pose transformation matrix corresponding to the two training point cloud segments in the pair of training point cloud segments.

[0102] In this embodiment, at least two training point cloud models corresponding to the same scene and different sources can be acquired in advance to train the pre-constructed machine learning model according to the at least two training point cloud models. Specifically, the at least two training point cloud models are first split into a plurality of point cloud segments of a predetermined size, for example, 2m*2m*2m point cloud segments, and the like.

[0103] Then, a plurality of training point cloud segment pairs are selected from the plurality of point cloud segments, each training point cloud segment pair including two training point cloud segments, and the two training point cloud segments have an overlapping part, for example, there is an overlapping part of 30%, or an overlapping part of 40%, and the like. It should be noted that the two training point cloud segments in a training point cloud segment pair can belong to the same training point cloud model, or can belong to different training point cloud models, which is not specially limited.

[0104] In addition, in order to ensure the rationality of the training data, the number of training point cloud segments corresponding to different training point cloud models in the plurality of training point cloud segment pairs is the same, for example, the training point cloud segments corresponding to the training point cloud model A and the training point cloud segments corresponding to the training point cloud model B each account for 50%, and the like. Thus, the rationality of subsequent training of the machine learning model can be improved.

[0105] After determining the plurality of training point cloud segment pairs, the training point cloud segments in each training point cloud segment pair can be randomly pose-transformed, and each training point cloud segment pair after the random pose transformation is input into the pre-constructed machine learning model for training, and by adjusting the parameters of the machine learning model, the machine learning model can output the real pose transformation matrix corresponding to the two training point cloud segments in the training point cloud segment pair.

[0106] It should be understood that by pre-transforming the training point cloud segments in each training point cloud segment pair, the training can truly simulate the actual use scenario, and the accuracy of the training result of the machine learning model is ensured. In addition, the training data can be enriched by splitting the training point cloud model into point cloud segments to construct the training data, which reduces the acquisition cost of the training data. At the same time, through the training of the rich data, the accuracy of the subsequent training result is also ensured.

[0107] The device embodiment of the present application is introduced below, which can be used to execute the point cloud hole repairing method based on different source point clouds in the above-mentioned embodiments of the present application. For details not disclosed in the device embodiment of the present application, please refer to the above-mentioned embodiments of the point cloud hole repairing method based on different source point clouds.

[0108] Figure 4 A block diagram of a point cloud hole repairing device based on different source point clouds according to an embodiment of the present application is shown.

[0109] Referring toFigure 4 As shown, the point cloud hole repairing device based on heterogeneous point clouds according to one embodiment of the present application comprises:

[0110] The first computing module 410 is configured to input the point cloud data to be repaired and the first heterogeneous point cloud data corresponding thereto into a machine learning model that has been pre-trained to obtain a corresponding first pose conversion matrix.

[0111] The first pose conversion module 420 is configured to perform pose conversion processing on the first heterogeneous point cloud data according to the first pose conversion matrix to obtain second heterogeneous point cloud data, wherein the second heterogeneous point cloud data is in the same coordinate system as the point cloud data to be repaired.

[0112] The second computing module 430 is configured to match the point cloud data to be repaired and the second heterogeneous point cloud data by using an iterative closest point algorithm to obtain a corresponding second pose conversion matrix.

[0113] The second pose conversion module 440 is configured to perform pose conversion processing on the second heterogeneous point cloud data according to the second pose conversion matrix to obtain target heterogeneous point cloud data.

[0114] The hole identification module 450 is configured to identify a hole to be repaired in the point cloud data to be repaired according to position information of points in the point cloud data to be repaired and position information of points in the target heterogeneous point cloud data.

[0115] The repairing module 460 is configured to repair the point cloud data to be repaired according to points in the target heterogeneous point cloud data corresponding to the hole to be repaired to obtain target point cloud data after repair.

[0116] In one embodiment of the present application, the hole identification module 450 is configured to: according to the position information of the points in the point cloud data to be repaired and the position information of the points in the target heterogeneous point cloud data, if there are only points in the target heterogeneous point cloud data in a certain region with a certain size, and there are both points in the point cloud data to be repaired and points in the target heterogeneous point cloud data in other regions adjacent to the certain region, or there are only points in the point cloud data to be repaired, the certain region is identified as a hole to be repaired in the point cloud data to be repaired.

[0117] In one embodiment of the present application, the hole identification module 450 is further configured to: identify a building plane from the point cloud data to be repaired; and in the building plane, if there are no points in the point cloud data to be repaired in a certain region with a certain size, the certain region is identified as a hole to be repaired in the point cloud data to be repaired.

[0118] In an embodiment of the present application, the first computing module 410 is further configured to: acquire image data corresponding to the same scene as the point cloud data to be repaired; and construct first multi-source point cloud data corresponding to the point cloud data to be repaired according to the image data.

[0119] In an embodiment of the present application, the first computing module 410 is further configured to: calculate a scaling factor of the first multi-source point cloud data according to position information of points in the point cloud data to be repaired and position information of points in the first multi-source point cloud data; and scale the first multi-source point cloud data according to the scaling factor to obtain scaled first multi-source point cloud data.

[0120] In an embodiment of the present application, the first computing module 410 is further configured to: adjust the density of points in the first multi-source point cloud data corresponding to the point cloud data to be repaired according to density information of points in the point cloud data to be repaired to obtain first multi-source point cloud data after density adjustment.

[0121] In an embodiment of the present application, the first computing module 410 is further configured to: acquire at least two training point cloud models of different sources in the same scene, and split the at least two training point cloud models into a plurality of point cloud segments of a predetermined size; select a plurality of pairs of training point cloud segments from the plurality of point cloud segments, the two training point cloud segments in each pair of training point cloud segments having an overlapping part, and the number of training point cloud segments corresponding to different training point cloud models in the plurality of pairs of training point cloud segments being the same; perform random pose transformation on the training point cloud segments in each pair of training point cloud segments, and input the pairs of training point cloud segments into a pre-constructed machine learning model for training, so that the machine learning model outputs a real pose transformation matrix corresponding to the two training point cloud segments in each pair of training point cloud segments.

[0122] Figure 5 A structural schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the present application is shown.

[0123] It should be noted that, Figure 5 The computer system of the electronic device shown is only an example and should not impose any limitation on the functions and use range of embodiments of the present application.

[0124] As Figure 5As shown, the computer system includes a central processing unit (CPU) 501 which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 502 or loaded into a random access memory (RAM) 503 from a storage section 508, for example, and execute the methods described in the above embodiments. In the RAM 503, various programs and data required for the operation of the system are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0125] Connected to the I / O interface 505 are an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a display device such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as necessary. A removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 510 as necessary, so that a computer program read therefrom is installed into the storage section 508 as necessary.

[0126] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing a computer program for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 509, and / or installed from the removable recording medium 511. When the computer program is executed by the central processing unit (CPU) 501, various functions defined in the system of the present application are performed.

[0127] It should be noted that the computer-readable medium in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present application, the computer-readable signal medium can include a data signal carrying computer-readable computer programs in a baseband or as a part of a carrier wave. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit programs for use by or in conjunction with an instruction execution system, device or apparatus. The computer programs contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination thereof.

[0128] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In the flowcharts or block diagrams, each block can represent a module, a program segment or a part of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders than that shown in the drawings. For example, two blocks represented in succession can actually be executed substantially in parallel, and sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0129] The units described in the embodiments of the present application can be implemented by software, or by hardware, or by a combination of software and hardware. The units described can also be located in a single processor. In some cases, the names of the units do not limit the units themselves.

[0130] As another aspect, the present application provides a computer readable medium, which can be included in the electronic device described in the above embodiments, or can exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the method described in the above embodiments.

[0131] It should be noted that although several modules or units for performing actions are mentioned in the above detailed description, the division into the modules or units is not mandatory. In fact, according to the embodiments of the present application, features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, features and functions of one module or unit described above can be further divided into a plurality of modules or units.

[0132] From the above description of the embodiments, those skilled in the art will readily appreciate that the example embodiments described herein can be implemented by software and / or by hardware coupled with software. Accordingly, the technical solutions of the embodiments of the present application can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, or the like) or on a network, and includes a number of instructions for causing a computing device (which can be a personal computer, a server, a terminal, or a network device, etc.) to perform the methods according to the embodiments of the present application.

[0133] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the embodiments disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such

[0134] It should be understood that the present application is not limited to the precise construction that has been described above and illustrated in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the present application. The scope of the present application is limited only by the appended claims.

Claims

1. A method for hole filling of point clouds based on heterogeneous point clouds, characterized in that, The method comprises the following steps: inputting the to-be-repaired point cloud data and the first heterogeneous point cloud data corresponding thereto into a pre-trained machine learning model, so that the machine learning model outputs a corresponding first pose transformation matrix; performing pose transformation processing on the first heterogeneous point cloud data according to the first pose transformation matrix to obtain second heterogeneous point cloud data, wherein the second heterogeneous point cloud data is in the same coordinate system as the to-be-repaired point cloud data; performing matching on the to-be-repaired point cloud data and the second heterogeneous point cloud data by using an iterative closest point algorithm to obtain a corresponding second pose transformation matrix; performing pose transformation processing on the second heterogeneous point cloud data according to the second pose transformation matrix to obtain target heterogeneous point cloud data; identifying a to-be-repaired hole existing in the to-be-repaired point cloud data according to position information of points in the to-be-repaired point cloud data and position information of points in the target heterogeneous point cloud data; performing repair on the to-be-repaired point cloud data according to points in the target heterogeneous point cloud data corresponding to the to-be-repaired hole to obtain repaired target point cloud data; wherein, identifying a to-be-repaired hole existing in the to-be-repaired point cloud data according to position information of points in the to-be-repaired point cloud data and position information of points in the target heterogeneous point cloud data comprises: if there are only points in the target heterogeneous point cloud data in a certain region with a certain size, and there are both points in the to-be-repaired point cloud data and points in the target heterogeneous point cloud data in other regions adjacent to the certain region, or there are only points in the to-be-repaired point cloud data, the certain region is identified as the to-be-repaired hole existing in the to-be-repaired point cloud data according to position information of points in the to-be-repaired point cloud data and position information of points in the target heterogeneous point cloud data.

2. The method of claim 1, wherein, Before inputting the to-be-repaired point cloud data and the first heterogeneous point cloud data corresponding thereto into the pre-trained machine learning model, so that the machine learning model outputs a corresponding first pose transformation matrix, the method further comprises: obtaining image data corresponding to the same scene as the to-be-repaired point cloud data; constructing the first heterogeneous point cloud data corresponding to the to-be-repaired point cloud data according to the image data.

3. The method of claim 2, wherein, Before inputting the to-be-repaired point cloud data and the first heterogeneous point cloud data corresponding thereto into the pre-trained machine learning model, so that the machine learning model outputs a corresponding first pose transformation matrix, the method further comprises: calculating a scaling multiple of the first heterogeneous point cloud data according to position information of points in the to-be-repaired point cloud data and position information of points in the first heterogeneous point cloud data; scaling the first heterogeneous point cloud data according to the scaling multiple to obtain scaled first heterogeneous point cloud data.

4. The method according to any one of claims 1 to 3, characterized in that, Before inputting the to-be-repaired point cloud data and the first heterogeneous point cloud data corresponding thereto into the pre-trained machine learning model, so that the machine learning model outputs a corresponding first pose transformation matrix, the method further comprises: According to the density information of the points in the to-be-repaired point cloud data, the points in the first heterogeneous point cloud data corresponding to the points are density-adjusted to obtain the first heterogeneous point cloud data after density adjustment.

5. The method of claim 1, wherein, The method further comprises: acquiring at least two training point cloud models under the same scene and being heterogeneous, and splitting the at least two training point cloud models into a plurality of point cloud segments of a predetermined size; selecting a plurality of pairs of training point cloud segments from the plurality of point cloud segments, the two training point cloud segments in each pair of training point cloud segments having an overlapping part, and the number of training point cloud segments corresponding to different training point cloud models in the plurality of pairs of training point cloud segments being the same; performing random pose transformation on the training point cloud segments in each pair of training point cloud segments, and inputting the pair of training point cloud segments into a pre-constructed machine learning model for training, so that the machine learning model outputs a real pose transformation matrix corresponding to the two training point cloud segments in the pair of training point cloud segments.

6. A point cloud hole repairing device based on heterogeneous point clouds, characterized in that, comprise: a first calculation module configured to input to-be-repaired point cloud data and first heterogeneous point cloud data corresponding to the to-be-repaired point cloud data into a pre-trained machine learning model, so that the machine learning model outputs a corresponding first pose transformation matrix; a first pose transformation module configured to perform pose transformation processing on the first heterogeneous point cloud data according to the first pose transformation matrix to obtain second heterogeneous point cloud data, the second heterogeneous point cloud data being in the same coordinate system as the to-be-repaired point cloud data; a second calculation module configured to perform matching on the to-be-repaired point cloud data and the second heterogeneous point cloud data using an iterative closest point algorithm to obtain a corresponding second pose transformation matrix; a second pose transformation module configured to perform pose transformation processing on the second heterogeneous point cloud data according to the second pose transformation matrix to obtain target heterogeneous point cloud data; a hole identification module configured to identify a to-be-repaired hole existing in the to-be-repaired point cloud data according to position information of points in the to-be-repaired point cloud data and position information of points in the target heterogeneous point cloud data; a repairing module configured to repair the to-be-repaired point cloud data according to points in the target heterogeneous point cloud data corresponding to the to-be-repaired hole to obtain repaired target point cloud data; wherein the to-be-repaired hole existing in the to-be-repaired point cloud data is identified according to the position information of the points in the to-be-repaired point cloud data and the position information of the points in the target heterogeneous point cloud data, comprising: if, according to the position information of the points in the to-be-repaired point cloud data and the position information of the points in the target heterogeneous point cloud data, there are only points in the target heterogeneous point cloud data in a certain region of a certain size, and there are both points in the to-be-repaired point cloud data and points in the target heterogeneous point cloud data in other regions adjacent to the certain region, or there are only points in the to-be-repaired point cloud data, the certain region is identified as the to-be-repaired hole existing in the to-be-repaired point cloud data.

7. A computer readable medium having stored thereon a computer program, characterized in that The computer program is executed by a processor to implement the point cloud hole repairing method based on heterogeneous point clouds according to any one of claims 1 to 5.

8. An electronic device, comprising: comprise: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the point cloud hole filling method based on heterogeneous point clouds according to any one of claims 1 to 5.

Images