Airborne point cloud automatic color assignment method and system

Abstract

The invention relates to the technical field of image processing, and discloses an airborne point cloud automatic color assignment method and system, and the method comprises the steps: obtaining image data of a target region through an image collection device, and obtaining point cloud data and airborne attitude data; performing feature extraction on the image data, and generating a feature descriptor; matching the feature descriptors of the adjacent images according to the timestamps of the image data and the airborne attitude data to generate an image feature matching result; calculating a rigid body transformation matrix of the image acquisition device and the airborne carrier according to an image feature matching result; according to the rigid body transformation matrix, space coordinates of the point cloud data are converted into an image coordinate system, and an image pixel position corresponding to each point cloud is positioned; and extracting color information of the image data, and endowing the color information to the point cloud data according to the image pixel position corresponding to the point cloud. According to the invention, manual intervention is reduced, full automation of airborne point cloud color assignment and reusable results are realized, and the color assignment efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically, to an airborne point cloud automatic coloring method and system. Background Art

[0002] In recent years, with the rapid development of UAV technology and the significant progress of 3D laser scanning hardware in terms of stability, accuracy, and ease of operation, new surveying and mapping equipment and technologies have emerged one after another. Among them, the airborne 3D laser scanning equipment based on UAVs is particularly prominent. It uses the fixed-route flight and terrain-imitating flight characteristics of UAVs, equipped with 3D laser scanners and color cameras, to quickly and automatically obtain the 3D spatial data of the terrain. However, 3D laser scanners can only collect information such as 3D spatial point data and reflection intensity of the terrain. In the process of point cloud visualization, there is a lack of support for color information, and it cannot provide a more intuitive and detailed display effect. Therefore, it becomes a necessary process to give each 3D point cloud corresponding color information.

[0003] The current airborne point cloud coloring method usually relies on oblique photography technology. First, the image data is posture solved, and the image is orthorectified based on the solution result to generate an orthophoto. Since the coordinate system of the orthophoto is not consistent with the coordinate system of the three-dimensional laser scanning point cloud, manual intervention is required to establish the correspondence between the two coordinate systems by selecting feature points in the two coordinate systems for point puncture operations. After the corresponding relationship is established, the coordinate transformation is performed to map the color information to the point cloud. However, this method has the following defects: First, manual participation in point puncture and coordinate system alignment is required, and the steps are complicated and time-consuming; second, the coordinate transformation relationship for each task is not fixed, resulting in the inability to reuse; third, the entire process cannot be fully automated, the efficiency is low, and it is difficult to meet the needs of real-time point cloud coloring. Summary of the invention

[0004] In order to overcome the defects of the existing airborne point cloud coloring technology, such as low efficiency, inability to be fully automated, and non-reusable results, the present invention proposes the following technical solutions:

[0005] In a first aspect, the present invention provides an airborne point cloud automatic coloring method, comprising:

[0006] Acquire image data and point cloud data of the target area, as well as airborne attitude data;

[0007] Extract features from image data and generate feature descriptors;

[0008] According to the timestamp of the image data and the airborne attitude data, the feature descriptors of adjacent images are matched to generate image feature matching results;

[0009] According to the image feature matching results, the rigid body transformation matrix of the image acquisition device and the airborne carrier is calculated;

[0010] According to the rigid body transformation matrix, the spatial coordinates of the point cloud data are converted into the image coordinate system to locate the image pixel position corresponding to each point cloud;

[0011] The color information of the image data is extracted, and the color information is assigned to the point cloud data according to the image pixel position corresponding to the point cloud.

[0012] As a preferred technical solution, before extracting features from the image data, the method further includes:

[0013] It is determined whether the timestamp of the image data is within the timestamp coverage range of the airborne attitude data. If so, the image data is retained as valid image data; if not, the image data is discarded.

[0014] As a preferred technical solution, feature extraction is performed on the image data and feature descriptors are generated, including:

[0015] Grayscale the image data;

[0016] Use feature point detection algorithm to detect the grayscale processed image data, and use the detected local extreme points as feature points;

[0017] Calculate the pixel gradient direction and amplitude distribution in the neighborhood of the feature point;

[0018] Based on the pixel gradient direction and amplitude distribution in the neighborhood of the feature point, the feature point is quantized to generate a feature descriptor.

[0019] As a preferred technical solution, feature descriptors of adjacent images are matched according to the timestamp of the image data and the airborne attitude data to generate image feature matching results, including:

[0020] According to the timestamp of the image data, the spatial position coordinates corresponding to each image data are interpolated from the airborne attitude data;

[0021] According to the spatial position coordinates of the image data, adjacent image data whose spatial distance is within a preset range are screened out;

[0022] The feature similarity between feature descriptors of adjacent image data is calculated, and the feature descriptors are matched according to the phase feature similarity to generate image feature matching results.

[0023] As a preferred technical solution, according to the image feature matching results, the rigid body transformation matrix of the image acquisition device and the airborne carrier is calculated, including:

[0024] According to the image feature matching results, the SFM incremental reconstruction method is used to calculate the posture data and camera intrinsic parameters of all image data;

[0025] According to the image feature matching results, as well as the posture data of all image data and the camera internal parameters, the rigid body transformation matrix of the image acquisition device and the airborne carrier is calculated.

[0026] As a preferred technical solution, based on the image feature matching results, the SFM incremental reconstruction method is used to calculate the posture data and camera intrinsic parameters of all image data, including:

[0027] Select two images from the image feature matching results and extract feature point matching pairs of the two images;

[0028] Calculate the posture data of the two images based on the feature point matching pairs;

[0029] Introduce new images one by one, extract feature point matching pairs between the new images and the images whose pose data has been calculated, and calculate the pose data and camera intrinsic parameters of the new images through the PnP algorithm based on the feature point matching pairs;

[0030] The bundle adjustment optimization algorithm is used to optimize and output the posture data and camera intrinsic parameters of all images.

[0031] As a preferred technical solution, the rigid body transformation matrix of the image acquisition device and the airborne carrier is calculated based on the image feature matching results, the posture data of all image data and the camera internal parameter parameters, including:

[0032] Extracting feature point matching pairs of image data from image feature matching results;

[0033] According to the feature point matching pair and the camera internal parameter parameters, the relative rotation matrix R between the image data is calculated. c,ij and the relative translation vector t c,ij ;

[0034] According to the posture data of the image data, as well as the relative rotation matrix and relative translation vector between the image data, the rotation matrix R between the image acquisition device and the airborne carrier is calculated. bc and the translation vector t bc , whose expression is as follows:

[0035]

[0036] Where b represents the airborne carrier coordinate system, c represents the image acquisition device coordinate system, ω represents the global coordinate system, and R bc represents the rotation matrix between the optimized image acquisition device and the airborne carrier, R ω,birepresents the rotation matrix of the airborne coordinate system corresponding to the i-th image relative to the global coordinate system, R c,ij represents the relative rotation matrix between the i-th image and the j-th image in the image acquisition device; t bc represents the translation vector between the optimized image acquisition device and the airborne carrier, t ω,bi represents the translation vector of the airborne coordinate system corresponding to the i-th image relative to the global coordinate system, t c,ij represents the relative translation vector between the i-th image and the j-th image in the image acquisition device;

[0037] According to the rotation matrix R of the image acquisition device and the airborne carrier bc and the translation vector t bc , construct the rigid body transformation matrix T, whose expression is as follows:

[0038]

[0039] As a preferred technical solution, after calculating the rotation matrix R between the image acquisition device and the airborne carrier, bc and the translation vector t bc Afterwards, the method further comprises:

[0040] Extract the best feature point matching pair p from the image feature matching results c,k,i and p c,k,j , construct the rotation matrix R bc and the translation vector t bc The error objective function; where p c,k,i represents the coordinates of the i-th feature point in the k-th image in the image acquisition device coordinate system, p c,k,j represents the coordinates of the jth feature point in the kth image in the coordinate system of the image acquisition device;

[0041] Iteratively optimize the rotation matrix R using nonlinear least squares method bc and the translation vector t bc , so that the error function converges to the minimum value, and outputs the optimized rotation matrix R bc and the translation vector t bc Construct the rigid body transformation matrix of the image acquisition device and the airborne carrier.

[0042] As a preferred technical solution, the rotation matrix R bc and the translation vector t bc The expression of the error objective function is as follows:

[0043]

[0044] In a second aspect, the present invention further provides an airborne point cloud automatic coloring system, which is applied to the airborne point cloud automatic coloring method as described in any solution of the first aspect, comprising:

[0045] An acquisition module is used to acquire image data and point cloud data of the target area, as well as airborne attitude data;

[0046] An extraction module, used for extracting features from image data and generating feature descriptors;

[0047] A matching module, used to match feature descriptors of adjacent images according to the timestamp of the image data and the airborne attitude data, and generate an image feature matching result;

[0048] A calculation module, used to calculate the rigid body transformation matrix of the image acquisition device and the airborne carrier according to the image feature matching result;

[0049] The positioning module is used to convert the spatial coordinates of the point cloud data into the image coordinate system according to the rigid body transformation matrix, and locate the image pixel position corresponding to each point cloud;

[0050] The color assignment module is used to extract the color information of the image data and assign the color information to the point cloud data according to the image pixel position corresponding to the point cloud.

[0051] The beneficial effects of the present invention include at least:

[0052] The present invention obtains image data and point cloud data of the target area, combines them with airborne attitude data, generates feature descriptors based on feature extraction, matches images using timestamps and generates image feature matching results, thereby calculating the rigid body transformation matrix between the image acquisition device and the airborne carrier. Through the rigid body transformation matrix, the spatial coordinates of the point cloud data are accurately converted to the image coordinate system, achieving an accurate correspondence between the point cloud and the image pixels, and based on this, the color information of the image is extracted and assigned to the point cloud data, achieving full automation of airborne point cloud coloring, reducing manual intervention and improving work efficiency. At the same time, due to the fixedness of the rigid body transformation matrix, the results are reusable and can be calculated in advance, thereby further reducing unnecessary calculation time, achieving efficient real-time color point cloud coloring, and significantly improving overall work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic flow chart of an airborne point cloud automatic coloring method provided in an embodiment of the present invention.

[0054] Figure 2 This is an example of the coloring effect provided by the embodiment of the present invention. Figure I .

[0055] Figure 3This is an example of the coloring effect provided by the embodiment of the present invention. Figure II .

[0056] Figure 4 An architectural diagram of an airborne point cloud automatic coloring system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred technical solutions. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred technical solutions are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0058] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0059] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0060] Example 1

[0061] This embodiment proposes an automatic coloring method for airborne point clouds, such as Figure 1 As shown, Figure 1 A schematic diagram of a flow chart of an automatic coloring method for airborne point clouds provided by an embodiment of the present invention, the method comprising the following steps:

[0062] S1: Obtain image data of the target area through an image acquisition device, as well as point cloud data and airborne attitude data;

[0063] S2: Extract features from image data and generate feature descriptors;

[0064] S3: Match the feature descriptors of adjacent images according to the timestamp of the image data and the airborne attitude data to generate an image feature matching result;

[0065] S4: Calculate the rigid body transformation matrix of the image acquisition device and the airborne carrier according to the image feature matching result;

[0066] S5: According to the rigid body transformation matrix, the spatial coordinates of the point cloud data are converted into the image coordinate system, and the image pixel position corresponding to each point cloud is located;

[0067] S6: Extract color information of the image data, and assign the color information to the point cloud data according to the image pixel position corresponding to the point cloud.

[0068] As an exemplary explanation, in this embodiment, the image data and point cloud data of the target area are obtained by installing an image acquisition device and a three-dimensional laser scanner on the drone, and the airborne attitude data is obtained by the airborne sensor. In the specific implementation process, the image data is firstly subjected to feature extraction, and the SIFT algorithm is used to detect the feature points in the image and generate feature descriptors to characterize the local information of the feature points. Then, according to the timestamp of the image data and the airborne attitude data, the feature points of the images adjacent in time and space are matched, and the image feature matching results are generated in combination with the matching feature descriptors.

[0069] Furthermore, the image feature matching results are used to calculate the rigid body transformation matrix between the image acquisition device and the UAV airborne carrier using the PnP algorithm, including the rotation matrix and the translation vector, which are used to describe the fixed spatial relationship between the two coordinate systems. Based on the calculated rigid body transformation matrix, the spatial coordinates of the point cloud data are converted to the image coordinate system, and the spatial position of each point cloud point is corresponded to the image pixel position. Finally, the color information is extracted from the image data, and the corresponding color information is assigned to the point cloud data according to the position of the point cloud point in the image pixel, so as to realize the automatic generation of the color point cloud.

[0070] It can be understood that by acquiring the image data and point cloud data of the target area, and combining the airborne attitude data, generating feature descriptors based on feature extraction, matching the images using timestamps and generating image feature matching results, the rigid body transformation matrix between the image acquisition device and the airborne carrier is calculated. Through the rigid body transformation matrix, the spatial coordinates of the point cloud data are accurately converted to the image coordinate system, realizing the accurate correspondence between the point cloud and the image pixels, and based on this, the color information of the image is extracted and assigned to the point cloud data, realizing the full automation of airborne point cloud coloring, reducing manual intervention, and improving work efficiency. At the same time, due to the fixed nature of the rigid body transformation matrix, the results are reusable and can be calculated in advance, thereby further reducing unnecessary calculation time, realizing efficient real-time color point cloud coloring, and significantly improving overall work efficiency.

[0071] Example 2

[0072] This embodiment makes improvements based on the airborne point cloud automatic coloring method proposed in Embodiment 1.

[0073] In this embodiment, before extracting features from the image data, it is determined whether the timestamp of the image data is within the timestamp coverage of the airborne attitude data. If so, the image data is retained as valid image data; if not, the image data is discarded.

[0074] In the specific implementation process, after the image data of the target area is acquired through the image acquisition device, the timestamp information recorded in the image data is read and compared with the timestamp coverage range in the airborne attitude data. If the timestamp of the image data falls within the time range of the airborne attitude data, the image data is marked as valid image data and retained; if the timestamp of the image data exceeds the coverage range of the airborne attitude data, the image data is considered invalid and directly discarded. The entire judgment process is automatically executed by the system without manual intervention.

[0075] It is understandable that by effectively screening the image data, the subsequent processed image data and the airborne attitude data are ensured to have temporal consistency, avoiding image and point cloud mapping errors caused by time mismatch. Through this screening operation, the efficiency of image data processing can be improved, the waste of computing resources caused by invalid data can be reduced, and the accuracy and stability of the coloring results can be guaranteed.

[0076] In this embodiment, feature extraction is performed on the image data and a feature descriptor is generated, including:

[0077] Grayscale the image data;

[0078] Use feature point detection algorithm to detect the grayscale processed image data, and use the detected local extreme points as feature points;

[0079] Calculate the pixel gradient direction and amplitude distribution in the neighborhood of the feature point;

[0080] Based on the pixel gradient direction and amplitude distribution in the neighborhood of the feature point, the feature point is quantized to generate a feature descriptor.

[0081] In the specific implementation process, the image data of the target area acquired by the image acquisition device is grayed, and the color image is converted into a single-channel gray image to reduce the interference of external environmental factors such as lighting on subsequent feature extraction. Subsequently, the SIFT feature detection algorithm is used to detect feature points on the grayed image data, locate local extreme points in the image, and use these local extreme points as feature points. Then, based on the neighborhood information of each feature point, the gradient direction and amplitude distribution of the neighborhood pixels are calculated. Finally, according to the preset feature descriptor generation rules, the feature points are quantified using the calculated gradient direction and amplitude distribution to generate feature descriptors as the unique representation of the feature points.

[0082] It can be understood that by accurately extracting the feature points of the image data and generating feature descriptors that represent its local information, it has high robustness and stability. The generation of feature descriptors effectively improves the accuracy and efficiency of feature matching between images, and provides a reliable data basis for subsequent matching between images and coloring of point clouds. At the same time, grayscale processing reduces the interference of external factors on feature extraction, making the feature point detection results more accurate and consistent.

[0083] In this embodiment, the feature descriptors of adjacent images are matched according to the timestamp of the image data and the airborne attitude data to generate an image feature matching result, including:

[0084] According to the timestamp of the image data, the spatial position coordinates corresponding to each image data are interpolated from the airborne attitude data;

[0085] According to the spatial position coordinates of the image data, adjacent image data whose spatial distance is within a preset range are screened out;

[0086] The feature similarity between feature descriptors of adjacent image data is calculated, and the feature descriptors are matched according to the phase feature similarity to generate image feature matching results.

[0087] In the specific implementation process, the timestamp information recorded in the image data is extracted and compared with the timestamp of the airborne attitude data. During the comparison process, a linear interpolation algorithm is used to calculate the spatial position coordinates of the image acquisition system at each time point according to the timestamp of the image data. Furthermore, based on the calculated spatial position coordinates, adjacent image data whose spatial distance is within the set threshold range are screened out from all image data. Then, for the screened adjacent image data, a feature similarity algorithm is used to compare the feature descriptors and calculate the feature similarity between the feature descriptors. Based on the feature similarity, the feature point pairs between the images are matched to generate image feature matching results for subsequent processing.

[0088] Feature data screening improves the screening accuracy of adjacent image data and reduces the interference of invalid data. The similarity calculation of feature descriptors accurately matches the feature points between images and generates high-precision image feature matching results, providing data reliability for subsequent processing.

[0089] In this embodiment, according to the image feature matching result, the rigid body transformation matrix of the image acquisition device and the airborne carrier is calculated, including:

[0090] Select two images from the image feature matching results and extract feature point matching pairs of the two images;

[0091] Calculate the posture data of the two images based on the feature point matching pairs;

[0092] Introduce new images one by one, extract feature point matching pairs between the new images and the images whose pose data has been calculated, and calculate the pose data and camera intrinsic parameters of the new images through the PnP algorithm based on the feature point matching pairs;

[0093] Use the bundle adjustment optimization algorithm to optimize and output the posture data and camera intrinsic parameters of all images;

[0094] According to the image feature matching results, as well as the posture data of all image data and the camera internal parameters, the rigid body transformation matrix of the image acquisition device and the airborne carrier is calculated.

[0095] In the specific implementation process, the image pair with the highest matching accuracy is selected from the image feature matching results, and its feature point matching pairs are extracted. The matching results are input into the PnP algorithm to calculate the posture data of the two images, including the rotation matrix and translation vector, and the camera intrinsic parameters of the image are combined. Subsequently, the newly added image is introduced and matched with the image with the calculated posture data, and the feature point matching pairs between the newly added image and the calculated image are extracted. The posture data and camera intrinsic parameters of the newly added image are calculated again using the PnP algorithm. To ensure the accuracy of the overall posture data, the posture data and camera intrinsic parameters of all images are input into the bundle adjustment optimization algorithm (Bundle Adjustment, BA), and the error is reduced through iterative optimization. The optimized rotation matrix and translation vector are output as the image posture data, and combined with the camera intrinsic parameters, it provides high-precision basic data for the subsequent calculation of the rigid body transformation matrix of the image acquisition device and the airborne carrier.

[0096] It is understandable that by introducing new images and calculating their postures, the comprehensiveness and accuracy of image data processing can be significantly improved; the bundle adjustment optimization algorithm improves the consistency of image posture data and camera intrinsic parameters, laying the foundation for the subsequent high-precision calculation of rigid body transformation matrices, thereby achieving efficient and automated support for the point cloud colorization process.

[0097] In this embodiment, the rigid body transformation matrix of the image acquisition device and the airborne carrier is calculated based on the image feature matching results, the posture data of all image data and the camera internal parameter parameters, including:

[0098] Extracting feature point matching pairs of image data from image feature matching results;

[0099] According to the feature point matching pair and the camera internal parameter parameters, the relative rotation matrix R between the image data is calculated. c,ij and the relative translation vector t c,ij ;

[0100] According to the posture data of the image data, as well as the relative rotation matrix and relative translation vector between the image data, the rotation matrix R between the image acquisition device and the airborne carrier is calculated. bc and the translation vector t bc , whose expression is as follows:

[0101]

[0102] Where b represents the airborne carrier coordinate system, c represents the image acquisition device coordinate system, ω represents the global coordinate system, and R bc represents the rotation matrix between the optimized image acquisition device and the airborne carrier, R ω,bi represents the rotation matrix of the airborne coordinate system corresponding to the i-th image relative to the global coordinate system, R c,ij represents the relative rotation matrix between the i-th image and the j-th image in the image acquisition device; t bc represents the translation vector between the optimized image acquisition device and the airborne carrier, t ω,bi represents the translation vector of the airborne coordinate system corresponding to the i-th image relative to the global coordinate system, t c,ij Represents the relative translation vector between the i-th image and the j-th image in the image acquisition device.

[0103] According to the rotation matrix R of the image acquisition device and the airborne carrier bc and the translation vector t bc , construct the rigid body transformation matrix T, whose expression is as follows:

[0104]

[0105] It can be understood that this embodiment can quickly and accurately solve the rigid body transformation matrix between the image acquisition device and the airborne carrier, avoiding the low efficiency and error accumulation problems caused by manual calibration in traditional methods. At the same time, the optimized rigid body transformation matrix has global consistency and high reusability, and can be directly used for coloring real-time point cloud data, significantly improving the efficiency of point cloud coloring, reducing manual participation, ensuring the full automation of the system, and further improving the overall efficiency and accuracy of drone mapping.

[0106] In this embodiment, after calculating the rotation matrix R between the image acquisition device and the airborne carrier, bc and the translation vector t bc Afterwards, the best feature point matching pair p is extracted from the image feature matching results. c,k,i and p c,k,j , construct the rotation matrix R bc and the translation vector t bc The error objective function is:

[0107]

[0108] Among them, p c,k,i represents the coordinates of the i-th feature point in the k-th image in the image acquisition device coordinate system, p c,k,j represents the coordinates of the jth feature point in the kth image in the coordinate system of the image acquisition device;

[0109] Iteratively optimize the rotation matrix R using nonlinear least squares method bc and the translation vector t bc , so that the error function converges to the minimum value, and outputs the optimized rotation matrix R bc and the translation vector t bc Construct the rigid body transformation matrix of the image acquisition device and the airborne carrier.

[0110] In the specific implementation process, by acquiring the image data collected by multiple image acquisition devices and their corresponding posture information, the feature point matching pair p with the smallest error in the image feature matching result is selected. c,k,i and p c,k,j , construct the error objective function, and transform the rotation matrix R bc and the translation vector t bc Perform nonlinear least squares iterative optimization and iteratively adjust the parameter values ​​to make the error of the objective function reach the global minimum. The rotation matrix R obtained after optimization bc and the translation vector t bc It can accurately characterize the rigid body transformation relationship between the image acquisition device and the airborne carrier.

[0111] It can be understood that this embodiment can accurately calculate the rotation matrix and translation vector between the image acquisition device and the airborne carrier to form a stable rigid body transformation relationship. The optimized result significantly reduces the error of feature point matching, ensures the conversion accuracy between image posture and point cloud coordinates, and thus realizes an efficient and automated point cloud coloring process. At the same time, due to the long-term stability of the rigid body transformation relationship, the obtained results can be reused, further improving the work efficiency and reducing the resource consumption of repeated calculations.

[0112] like Figure 2 and Figure 3 As shown, Figure 2 This is an example of the coloring effect provided by the embodiment of the present invention. Figure I , Figure 3 This is an example of the coloring effect provided by the embodiment of the present invention. Figure II . Figure 2 The effect of the present invention on the coloring of airborne point clouds is demonstrated, which is specifically manifested in that after the spatial coordinates of the point cloud data and the image data are matched, the color information of the image is successfully assigned to the point cloud. Figure 2 The color point cloud display effects of a large range of terrain, roads and vegetation areas can be clearly seen, indicating that the invention has efficient point cloud coloring capabilities in wide-area scenes. Figure 3 The coloring effect of the present invention in local scenes is further demonstrated, especially the coloring of point clouds of details such as village buildings and fields. After coloring, the building outlines, field textures, and color features of vegetation in the point cloud are accurately presented, indicating that the present invention can perform high-precision coloring on complex scenes and demonstrate excellent color reproduction capabilities.

[0113] Figure 2 and Figure 3 The comparison illustrates the applicability of the present invention in large-scale and local detail scenes, and verifies the technical advantages of the present invention in improving point cloud visualization effects, including achieving fully automated, efficient and reusable coloring results.

[0114] Example 3

[0115] like Figure 4 As shown, this embodiment proposes an airborne point cloud automatic coloring system, which is applied to the airborne point cloud automatic coloring method as described in the above embodiment, including: an acquisition module 100, an extraction module 200, a first generation module 300, a second generation module 400 and a conversion module 500.

[0116] Among them, the acquisition module 100 is used to acquire the image data and point cloud data of the target area, and to acquire the airborne attitude data; the extraction module 200 is used to extract features from the image data and generate feature descriptors; the matching module 300 is used to match the feature descriptors of adjacent images according to the timestamp of the image data and the airborne attitude data, and generate image feature matching results; the calculation module 400 is used to calculate the rigid body transformation matrix of the image acquisition device and the airborne carrier according to the image feature matching results; the positioning module 500 is used to convert the spatial coordinates of the point cloud data into the image coordinate system according to the rigid body transformation matrix, and locate the image pixel position corresponding to each point cloud; the color assignment module 600 is used to extract the color information of the image data, and assign the color information to the point cloud data according to the image pixel position corresponding to the point cloud.

[0117] It should be noted that the aforementioned explanation of the embodiment of the airborne point cloud automatic coloring method is also applicable to the airborne point cloud automatic coloring system of this embodiment, and will not be repeated here.

[0118] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0119] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0120] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0121] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0122] A person skilled in the art may understand that all or part of the steps in the above-mentioned embodiment method may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0123] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. An automatic coloring method for airborne point clouds, characterized in that: include: Acquire image data of the target area through an image acquisition device, as well as point cloud data and airborne attitude data; Extract features from image data and generate feature descriptors; According to the timestamp of the image data and the airborne attitude data, the feature descriptors of adjacent images are matched to generate image feature matching results; According to the image feature matching results, the rigid body transformation matrix of the image acquisition device and the airborne carrier is calculated; According to the rigid body transformation matrix, the spatial coordinates of the point cloud data are converted into the image coordinate system to locate the image pixel position corresponding to each point cloud; The color information of the image data is extracted, and the color information is assigned to the point cloud data according to the image pixel position corresponding to the point cloud.

2. The method for automatic coloring of airborne point clouds according to claim 1, characterized in that: Before extracting features from the image data, the method further includes: It is determined whether the timestamp of the image data is within the timestamp coverage range of the airborne attitude data. If so, the image data is retained as valid image data; if not, the image data is discarded.

3. The method for automatic coloring of airborne point clouds according to claim 1, characterized in that: Extract features from image data and generate feature descriptors, including: Grayscale the image data; Use feature point detection algorithm to detect the grayscale processed image data, and use the detected local extreme points as feature points; Calculate the pixel gradient direction and amplitude distribution in the neighborhood of the feature point; Based on the pixel gradient direction and amplitude distribution in the neighborhood of the feature point, the feature point is quantified to generate a feature descriptor.

4. The method for automatically coloring airborne point clouds according to claim 1, characterized in that: According to the timestamp of the image data and the airborne attitude data, the feature descriptors of adjacent images are matched to generate image feature matching results, including: According to the timestamp of the image data, the spatial position coordinates corresponding to each image data are interpolated from the airborne attitude data; According to the spatial position coordinates of the image data, adjacent image data whose spatial distance is within a preset range are screened out; The feature similarity between feature descriptors of adjacent image data is calculated, and the feature descriptors are matched according to the phase feature similarity to generate image feature matching results.

5. The method for automatic coloring of airborne point clouds according to claim 1, characterized in that: According to the image feature matching results, the rigid body transformation matrix of the image acquisition device and the airborne carrier is calculated, including: According to the image feature matching results, the SFM incremental reconstruction method is used to calculate the posture data and camera intrinsic parameters of all image data; According to the image feature matching results, as well as the posture data of all image data and the camera internal parameters, the rigid body transformation matrix of the image acquisition device and the airborne carrier is calculated.

6. The method for automatic coloring of airborne point clouds according to claim 5, characterized in that: According to the image feature matching results, the SFM incremental reconstruction method is used to calculate the posture data and camera intrinsic parameters of all image data, including: Select two images from the image feature matching results and extract feature point matching pairs of the two images; Calculate the posture data of the two images based on the feature point matching pairs; Introduce new images one by one, extract feature point matching pairs between the new images and the images whose pose data has been calculated, and calculate the pose data and camera intrinsic parameters of the new images through the PnP algorithm based on the feature point matching pairs; The bundle adjustment optimization algorithm is used to optimize and output the posture data and camera intrinsic parameters of all images.

7. The method for automatically coloring airborne point clouds according to claim 5, characterized in that: According to the image feature matching results, as well as the posture data of all image data and the camera internal parameters, the rigid body transformation matrix of the image acquisition device and the airborne carrier is calculated, including: Extracting feature point matching pairs of image data from image feature matching results; According to the feature point matching pair and the camera internal parameter parameters, the relative rotation matrix R between the image data is calculated. c,ij and the relative translation vector t c,ij ; According to the posture data of the image data, as well as the relative rotation matrix and relative translation vector between the image data, the rotation matrix R between the image acquisition device and the airborne carrier is calculated. bc and the translation vector t bc , whose expression is as follows: Where b represents the airborne carrier coordinate system, c represents the image acquisition device coordinate system, ω represents the global coordinate system, and R bc represents the rotation matrix between the optimized image acquisition device and the airborne carrier, R ω,bi represents the rotation matrix of the airborne coordinate system corresponding to the i-th image relative to the global coordinate system, R c,ij represents the relative rotation matrix between the i-th image and the j-th image in the image acquisition device; t bc represents the translation vector between the optimized image acquisition device and the airborne carrier, t ω,bi represents the translation vector of the airborne coordinate system corresponding to the i-th image relative to the global coordinate system, t c,ij represents the relative translation vector between the i-th image and the j-th image in the image acquisition device; According to the rotation matrix R of the image acquisition device and the airborne carrier bc and the translation vector t bc , construct the rigid body transformation matrix T, whose expression is as follows:

8. The method for automatically coloring airborne point clouds according to claim 7, characterized in that: After calculating the rotation matrix R between the image acquisition device and the airborne carrier bc and the translation vector t bc Afterwards, the method further comprises: Extract the best feature point matching pair p from the image feature matching results c,k,i and p c,k,j , construct the rotation matrix R bc and the translation vector t bc The error objective function; where p c,k,i represents the coordinates of the i-th feature point in the k-th image in the image acquisition device coordinate system, p c,k,j represents the coordinates of the jth feature point in the kth image in the coordinate system of the image acquisition device; Iteratively optimize the rotation matrix R using nonlinear least squares method bc and the translation vector t bc , so that the error function converges to the minimum value, and outputs the optimized rotation matrix R bc and the translation vector t bc Construct the rigid body transformation matrix of the image acquisition device and the airborne carrier.

9. The method for automatic coloring of airborne point clouds according to claim 8, characterized in that: Rotation matrix R bc and the translation vector t bc The expression of the error objective function is as follows:

10. An airborne point cloud automatic coloring system, characterized in that: include: An acquisition module is used to acquire image data and point cloud data of the target area, as well as airborne attitude data; An extraction module, used for extracting features from image data and generating feature descriptors; A matching module, used to match feature descriptors of adjacent images according to the timestamp of the image data and the airborne attitude data, and generate an image feature matching result; A calculation module, used to calculate the rigid body transformation matrix of the image acquisition device and the airborne carrier according to the image feature matching result; The positioning module is used to convert the spatial coordinates of the point cloud data into the image coordinate system according to the rigid body transformation matrix, and locate the image pixel position corresponding to each point cloud; The color assignment module is used to extract the color information of the image data and assign the color information to the point cloud data according to the image pixel position corresponding to the point cloud.

Images