Unmanned aerial vehicle-mounted real-time color point cloud generation method
Through drone-mounted LiDAR and camera, combined with the principle of small hole imaging, the unified processing of laser point clouds and image data is realized, and the color point clouds are generated, which solves the problem of laser point clouds lacking texture information and improves the realism of the three-dimensional model and data processing efficiency, suitable for high-precision three-dimensional modeling and real-time disaster monitoring.
Patent Information
- Application Number
- CN202411001404.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
Smart Images

Figure CN119064950A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of laser point cloud data processing, and in particular relates to a method for generating real-time color point clouds carried by unmanned aerial vehicles. Background Art
[0002] Laser scanning measurement technology (Light Detection And Ranging), referred to as LiDAR, is also known as "real scene replication technology". Airborne LiDAR technology uses LiDAR technology on aircraft to achieve ground observation. It integrates three technologies: global positioning system, inertial navigation system and laser. It can quickly and efficiently obtain the precise three-dimensional spatial coordinates of each sampling point on the surface of the object. Through a large number of discrete sampling points to form a point cloud set, the spatial shape of the target object is accurately restored. It has unique advantages in obtaining vertical structural information of topography, buildings and vegetation. It is widely used in digital cities, disaster monitoring, coastal engineering, forestry applications, etc. However, a simple laser point cloud cannot provide information such as the texture and spectrum of the object surface, which is not conducive to the filtering, classification and interpretation of the point cloud.
[0003] Nowadays, in the fields of digital earth and digital city, more and more photo-realistic 3D models need to be established. The development of 3D laser scanning technology, the progress of modeling theory, and the widespread application of modeling tools have made it possible to reconstruct high-precision 3D geometric models, but this is only the first step in the reconstruction of true 3D models. Only when the framework model or entity model matches the real and fine texture information can a 3D model with a strong sense of immersion be truly generated. Therefore, color texture is an important factor affecting the visualization effect of model reconstruction. There are rich texture details on the surface of real scenes, and people distinguish scenes of the same shape based on these details. Therefore, in order to make the model visually realistic, the texture information of the scene is often projected onto the 3D surface through texture mapping technology. Color texture mapping often defines a texture pattern in a texture space, establishes the corresponding relationship between the texture space coordinates (u, v) and the object space coordinates (x, y, z) through the mapping algorithm, and covers the texture on the surface of the 3D object. At the same time, for some special scenes such as emergency rescue, the requirement of real-time acquisition of color point clouds is proposed. Based on the above background, the present invention proposes a real-time color point cloud generation method for unmanned aerial vehicles. Summary of the invention
[0004] The purpose of the present invention is to provide a method for generating real-time color point clouds onboard an unmanned aerial vehicle, aiming to solve the problems raised in the above-mentioned background technology.
[0005] To achieve the above object, the present invention provides the following technical solutions:
[0006] A method for generating a real-time color point cloud carried by an unmanned aerial vehicle comprises the following steps:
[0007] Step 1: Data collection and recording: The drone is equipped with LiDAR to continuously scan the ground, and the drone is equipped with a camera to take pictures of the ground. At the same time, the POS system continuously records the location and attitude information of the equipment, including GNSS positioning and speed data and IMU attitude data; through the wireless data real-time transmission system, the ground receiving equipment obtains GNSS positioning and speed data, IMU attitude data, LiDAR scanned ground point cloud data and camera images;
[0008] Step 2: Coordinate system unification: Use satellite and inertial combined navigation positioning and attitude determination technology to process POS posture data to obtain the real-time absolute geographic coordinates and flight attitude of the drone; calculate the absolute geographic coordinates of the point cloud data obtained by LiDAR scanning and the absolute geographic coordinates and exterior orientation elements of each camera image through interpolation algorithms, and unify the coordinate systems between different sensors;
[0009] Step 3: Camera imaging and coordinate conversion: Based on the pinhole imaging principle and combined with the calibration results, the conversion relationship between the world coordinate system, camera coordinate system, image coordinate system and pixel coordinate system is established; the coordinates of the LiDAR point cloud data and the camera image data are unified into the same reference coordinate system;
[0010] Step 4: Determine the range of a single photo: Calculate the coverage of each photo based on the properties of similar triangles;
[0011] Step 5, determine the coordinates of the upper right corner of a single photo in the e-coordinate system: determine the geographic coordinates of the upper right corner of each photo in a unified coordinate system;
[0012] Step 6, judging whether the ground laser point is within the coverage area of the photo: comparing the coordinates of the ground laser point with the rectangular area covered by the photo to judge whether the ground laser point is within the coverage area of the specified photo;
[0013] Step 7, back-projection calculation and color assignment: For the ground laser points within the area covered by the photo, back-projection calculation is performed using the pinhole imaging principle to obtain the coordinates of the ground laser points in the camera coordinate system, and then the pixel coordinates are calculated; the RGB information in the color table is decoded from the pixel and assigned to the RGB field of the point cloud to generate color point cloud data.
[0014] Furthermore, the principle formula of LiDAR positioning is as follows:
[0015] Formula 1:
[0016] in: is the coordinate of the ground laser point P in the geocentric rectangular coordinate system, i.e., the e system; is the coordinate of the IMU center in the e system; It is the IMU coordinate system, i.e. the rotation matrix from the I system to the e system; It is the component expression of the bias from the Lidar scanning center to the IMU navigation center in the I system; is the rotation matrix from the Lidar coordinate system, i.e., the S system to the I system; is the coordinate of the ground laser point P in the S system; the coordinate (X, Y, Z) of the ground laser point in the e system is obtained by formula 1.
[0017] Furthermore, the world coordinate system is denoted as OX w Y w Z w , used to describe the specific position of any object in three-dimensional space, using (X w ,Y w ,Z w ) represents the coordinate value;
[0018] The camera coordinate system is denoted as SX c Y c Z c , the coordinate origin S is the optical center of the camera, the X-axis and Y-axis are parallel to the X-axis and Y-axis of the image coordinate system respectively, the Z-axis is the optical axis of the camera, and satisfies the right-hand coordinate system with the X-axis and Y-axis. c ,Y c ,Z c ) represents the coordinate value; there is a point P in the world coordinate system w (X w ,Y w ,Z w ), and obtain its coordinates in the camera coordinate system (X c ,Y c ,Z c ):
[0019] Formula 2:
[0020] Among them, R 3×3 is the rotation matrix, T 3×1 is the translation matrix, M is called the extrinsic parameter matrix;
[0021] The image coordinate system is denoted as O-XY, the coordinate origin is the center of the image plane, the X-axis and the Y-axis are parallel to the two vertical sides of the image plane respectively, and (x, y) is used to represent the coordinate value of the pixel point on the image plane;
[0022] The pixel coordinate system is denoted as O-UV, the coordinate origin is the upper left vertex of the image plane, the U axis and the V axis are parallel to the X axis and the Y axis of the image coordinate system respectively, and (u, v) is used to represent the coordinate value of the pixel point on the image plane; the image is stored in the computer using an M×N matrix, and each coordinate point in the image with M rows and N columns is called a pixel point, or pixel, and each pixel point contains the gray value of the image;
[0023] There is a point (x, y) in the image coordinate system, and its corresponding pixel coordinate system is represented by (u, v). The conversion relationship is:
[0024] Formula 3:
[0025] Among them, (u 0 ,v 0 ) represents the coordinates of the origin of the image coordinate system in the pixel coordinate system, d x and d y Respectively represent the physical size of each pixel on the x-axis and y-axis.
[0026] Furthermore, in the step of determining the range of a single photo, the coordinates of the upper right corner of the photo in the camera coordinate system are:
[0027] Formula 4:
[0028] Among them, PixelSize is the size of a single pixel; PixLenth and PixWidth are the number of pixels in the length and width directions of the photo respectively; H is the average altitude of the drone; and f is the focal length of the camera.
[0029] Furthermore, in the step of determining the coordinates of the upper right corner of a single photo in the e-coordinate system, there is the following formula:
[0030] Formula 5:
[0031] in, The coordinates of the ground corner points in the photo coverage area in the e system; is the coordinate of the IMU center in the e system; is the rotation matrix from I system to e system; It is the component expression of the offset from the camera photography center to the IMU navigation center in the I system; is the rotation matrix from the photo coordinate system to the I system; The coordinates of the ground corner points of the photo coverage in the camera coordinate system.
[0032] Furthermore, in the step of judging whether the ground laser point is within the photo coverage area, the rectangle and the ground laser point are projected into the photo coordinate system, that is, the Z coordinate is reset to zero, and vector dot multiplication is used for judgment. The four points of the projected rectangular area are set as P1, P2, P3, and P4, and the ground laser point is P. If the following formula is satisfied, it is preliminarily judged that the projection of the ground laser point P is within the photo:
[0033] Formula 6:
[0034] Furthermore, in the back-projection calculation and coloring step, the pixel coordinates of the ground laser point corresponding to the photo are calculated using the back-projection formula, and it is determined whether the pixel coordinates exceed the maximum pixel size of the photo. If so, it is determined that the photo is not in the specified photo.
[0035] The back-projection calculation formula for pixel coordinates is as follows:
[0036] Formula 7:
[0037] Where u and v are the pixel coordinates of the back-projection of the ground laser point on the photo; PixLenth and PixWidth are the length and width of the photo pixel size; f is the focal length of the camera; (X w ,Y w ,Z w ) is the ground laser point P w The coordinates of (X c ,Y c ,Z c ) is the coordinate of the camera photography center in the world coordinate system; PixelSize is the actual size of a single pixel.
[0038] Furthermore, when calibrating the camera, a group of image control points are obtained by a camera calibration method, a group of compensation values are obtained by the group of image control points and the exterior orientation elements obtained by POS, and the group compensation values are averaged; the formula is as follows:
[0039] Formula 8:
[0040] Among them, R ex The rotation matrix generated for the angle compensation value; R pix4D It is the rotation matrix generated by the software pix4D; is the rotation matrix generated by the POS attitude angle; Δt is the delay between the camera shutter signal being sent and the shutter being pressed; X pix4D is the coordinate of the camera photography center obtained by the software pix4D; X pos is the coordinate of the camera photography center obtained from POS data; pos is the speed of the drone calculated from the POS coordinates.
[0041] Compared with the prior art, the present invention has the following beneficial effects:
[0042] 1. Improve the realism of 3D models: By combining laser scanning data and camera optical images, a 3D point cloud model with real color texture can be generated. This model can not only accurately reflect the geometric shape of the object, but also show the color and texture details of the object's surface, greatly enhancing the realism and immersion of the 3D model, making the model more realistically reflect the actual scene.
[0043] 2. Real-time data processing capability: The use of drone-based platforms can acquire and process data in real time during flight, which is particularly important for scenarios that require rapid response, such as emergency rescue and disaster monitoring. The generation of real-time color point clouds can help rescuers quickly understand the situation in the disaster area and improve rescue efficiency.
[0044] 3. Promote multi-source data fusion: Through multi-sensor data fusion technology, laser scanning data, camera image data and posture data are unified into the same reference coordinate system, achieving accurate registration of texture information and position information. This method not only improves the accuracy of the data, but also enhances the complementarity of the data, making the final generated 3D model more comprehensive and accurate.
[0045] 4. Simplify the 3D modeling process: Since the method provided by the present invention can directly generate 3D point cloud data with accurate texture information, it can greatly simplify the workflow of later 3D modeling, reduce work difficulty and cost, and improve overall work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 It is a flow chart of the present invention.
[0047] Figure 2 This is a schematic diagram of the LiDAR positioning principle in the present invention.
[0048] Figure 3 This is a relationship diagram between the pixel coordinate system and the image coordinate system in the present invention.
[0049] Figure 4 It is the ideal imaging model of the camera in the present invention.
[0050] Figure 5 It is a schematic diagram of judging that a point is within a rectangular coverage area in the present invention.
[0051] Figure 6 Schematic diagram of colorless point cloud.
[0052] Figure 7 Schematic diagram after coloring the point cloud. DETAILED DESCRIPTION
[0053] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0054] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.
[0055] like Figure 1 As shown, a method for generating a real-time color point cloud on an unmanned aerial vehicle provided by an embodiment of the present invention includes the following steps:
[0056] Step 1: Data collection and recording: The drone is equipped with LiDAR to continuously scan the ground, and the drone is equipped with a camera to take pictures of the ground. At the same time, the POS system continuously records the position and attitude information of the device (hereinafter referred to as posture), including GNSS positioning and speed data and IMU attitude data; through the wireless data real-time transmission system built into the device, the ground receiving device can obtain GNSS positioning and speed data, IMU attitude data, LiDAR scanned ground point cloud data and camera images in real time;
[0057] Step 2: Coordinate system unification: The POS posture data is processed by the leading satellite and inertial combined navigation positioning and attitude determination technology at home and abroad mastered by Wuhan Jishang Navigation Company to obtain the real-time absolute geographic coordinates and flight attitude of the drone; the absolute geographic coordinates of the point cloud data obtained by LiDAR scanning and the absolute geographic coordinates and external orientation elements of each camera image are calculated through the interpolation algorithm to unify the coordinate systems between different sensors;
[0058] Step 3: Camera imaging and coordinate conversion: Based on the pinhole imaging principle and combined with the calibration results, the conversion relationship between the world coordinate system, camera coordinate system, image coordinate system and pixel coordinate system is established; the coordinates of the LiDAR point cloud data and the camera image data are unified into the same reference coordinate system;
[0059] Step 4: Determine the range of a single photo: In the unified coordinate system, the coverage of each photo can be calculated by the properties of similar triangles. Since the coordinates of the ground laser point are unchanged, it can be calculated whether the point is within the specified photo.
[0060] Step 5, determine the coordinates of the upper right corner of a single photo in the e-coordinate system: determine the geographic coordinates of the upper right corner of each photo in a unified coordinate system;
[0061] Step 6, judging whether the ground laser point is within the coverage area of the photo: comparing the coordinates of the ground laser point with the rectangular area covered by the photo to judge whether the ground laser point is within the coverage area of the specified photo;
[0062] Step 7, back-projection calculation and color assignment: For the ground laser points within the area covered by the photo, back-projection calculation is performed using the pinhole imaging principle to obtain the coordinates of the ground laser points in the camera coordinate system, and then the pixel coordinates are calculated; the RGB information in the color table is decoded from the pixel and assigned to the RGB field of the point cloud to generate color point cloud data.
[0063] As a preferred embodiment of the present invention, see Figure 2 , the principle formula of LiDAR positioning is as follows:
[0064] Formula 1:
[0065] in: is the coordinate of the ground laser point P in the geocentric rectangular coordinate system (abbreviated as e system); is the coordinate of the IMU center in the e system; It is the rotation matrix from the IMU coordinate system (I system for short) to the e system; It is the component expression of the bias from the Lidar scanning center to the IMU navigation center in the I system; is the rotation matrix from the Lidar coordinate system (S system for short) to the I system; is the coordinate of the ground laser point P in the S system; the coordinate (X, Y, Z) of the ground laser point in the e system is obtained by the above formula.
[0066] As a preferred embodiment of the present invention, the world coordinate system is denoted as OX w Y w Z w , used to describe the specific position of any object in three-dimensional space, using (X w ,Y w ,Z w ) represents the coordinate value;
[0067] The camera coordinate system is denoted as SX c Y c Z c , the coordinate origin S is the optical center of the camera, the X-axis and Y-axis are parallel to the X-axis and Y-axis of the image coordinate system respectively, the Z-axis is the optical axis of the camera, and satisfies the right-hand coordinate system with the X-axis and Y-axis. c ,Y c ,Z c ) represents the coordinate value; there is a point P in the world coordinate system w (X w ,Y w ,Z w ), and obtain its coordinates in the camera coordinate system (X c ,Y c ,Z c ):
[0068] Formula 2:
[0069] Among them, R 3×3 is the rotation matrix, T 3×1 is the translation matrix, M is called the extrinsic parameter matrix;
[0070] The image coordinate system is denoted as O-XY, the coordinate origin is the center of the image plane, the X-axis and the Y-axis are parallel to the two vertical sides of the image plane respectively, and (x, y) is used to represent the coordinate value of the pixel point on the image plane;
[0071] The pixel coordinate system is denoted as O-UV, the coordinate origin is the upper left corner vertex of the image plane, the U axis and the V axis are parallel to the X axis and the Y axis of the image coordinate system respectively, and (u, v) is used to represent the coordinate value of the pixel point on the image plane; the image is stored in the computer using an M×N matrix, and each coordinate point in the image of M rows and N columns is called a pixel point (pixel), and each pixel point contains the gray value of the image; the relationship between the pixel coordinate system and the image coordinate system is as follows Figure 3 shown.
[0072] There is a point (x, y) in the image coordinate system, and its corresponding pixel coordinate system is represented by (u, v). The conversion relationship is:
[0073] Formula 3:
[0074] Among them, (u 0 ,v 0 ) represents the coordinates of the origin of the image coordinate system in the pixel coordinate system, d x and d y Represent the physical size of each pixel on the x-axis and y-axis respectively. The ideal imaging model of the camera is as follows Figure 4 shown.
[0075] As a preferred embodiment of the present invention, in the step of determining the range of a single photo, the coordinates of the upper right corner of the photo in the camera coordinate system are:
[0076] Formula 4:
[0077] Among them, PixelSize is the size of a single pixel, unit: millimeter (mm); PixLenth and PixWidth are the number of pixels in the length and width directions of the photo respectively; H is the average altitude of the drone flight, unit: meter (m); f is the focal length of the camera, unit: millimeter (mm).
[0078] As a preferred embodiment of the present invention, in the step of determining the coordinates of the upper right corner point of a single photo in the e-coordinate system, there is the following formula:
[0079] Formula 5:
[0080] in, The coordinates of the ground corner points of the photo coverage area in the geocentric rectangular coordinate system (e system); is the coordinate of the IMU center in the e system; is the rotation matrix from the IMU coordinate system (I system) to the e system; It is the component expression of the offset from the camera photography center to the IMU navigation center in the I system; is the rotation matrix from the photo coordinate system to the I system; The coordinates of the ground corner points of the photo coverage in the camera coordinate system.
[0081] As a preferred embodiment of the present invention, in the step of determining whether the ground laser point is within the photo coverage area, the rectangle and the ground laser point are projected into the photo coordinate system, that is, the Z coordinate is reset to zero, and vector dot multiplication is used for determination. The four points of the projected rectangular area are set as P1, P2, P3, and P4, and the ground laser point is set as P, as shown in the attached figure. Figure 5 As shown, if the following formula is satisfied, it is preliminarily determined that the projection of the ground laser point P is within the photo:
[0082] Formula 6:
[0083] As a preferred embodiment of the present invention, in the back-projection calculation and coloring step, the pixel coordinates of the ground laser point corresponding to the photo are calculated using the back-projection formula, and it is determined whether the pixel coordinates exceed the maximum pixel size of the photo. If so, it is determined that the photo is not in the specified photo.
[0084] The back-projection calculation formula for pixel coordinates is as follows:
[0085] Formula 7:
[0086] Where u and v are the pixel coordinates of the back-projection of the ground laser point on the photo; PixLenth and PixWidth are the length and width of the photo pixel size; f is the focal length of the camera, in millimeters (mm); (X w ,Y w ,Z w ) is the ground laser point P w The coordinates of (X c ,Y c ,Z c ) is the coordinate of the camera photography center in the world coordinate system; PixelSize is the actual size of a single pixel, unit: millimeter (mm).
[0087] After the above steps, the ground points and the pixels of the image can be matched, and then the RGB information in the color table decoded from the pixels is assigned to the RGB field of the point cloud, so as to integrate the color texture information into the point cloud.
[0088] As a preferred embodiment of the present invention, since the above theoretical derivations are all based on ideal conditions, in actual operation, due to installation errors of laser radars or cameras, that is, the laser radar to IMU and the camera photography center to IMU cannot guarantee 100% coincidence of coordinate axes during coordinate conversion. At the same time, the camera also has a shutter delay problem, that is, the time delay between the photo signal and pressing the shutter. As a result, the two point clouds that should have overlapping parts will have a large deflection, and the coloring of the point clouds will also have a large offset. Therefore, it is necessary to calibrate a compensation value, and multiply the compensation value when converting the coordinate system to achieve a satisfactory effect. For the calibration of the laser radar, a preset automatic calibration software is used to achieve an extremely high degree of point cloud coincidence. When calibrating the camera, a set of image control points is obtained through the traditional camera calibration method, and a set of compensation values are obtained through the group of image control points and the external orientation elements obtained by POS, and the group compensation values are averaged; the formula is as follows:
[0089] Formula 8:
[0090] Among them, R ex The rotation matrix generated for the angle compensation value; R pix4D It is the rotation matrix generated by the software pix4D; is the rotation matrix generated by the POS attitude angle; Δt is the delay between the camera shutter signal being sent and the shutter being pressed; X pix4D is the coordinate of the camera photography center obtained by the software pix4D; X pos is the coordinate of the camera photography center obtained from POS data; pos is the speed of the drone calculated from the POS coordinates.
[0091] The effect of point cloud coloring is as follows Figure 6 and Figure 7 As shown, the color texture is real and natural.
[0092] The above are only preferred embodiments of the present invention. It should be pointed out that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These should also be regarded as the protection scope of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.
Claims
1. A method for generating real-time color point cloud on an unmanned aerial vehicle, characterized in that: The following steps are involved: Step 1: Data collection and recording: The drone is equipped with LiDAR to continuously scan the ground, and the drone is equipped with a camera to take pictures of the ground. At the same time, the POS system continuously records the location and attitude information of the equipment, including GNSS positioning and speed data and IMU attitude data; through the wireless data real-time transmission system, the ground receiving equipment obtains GNSS positioning and speed data, IMU attitude data, LiDAR scanned ground point cloud data and camera images; Step 2: Coordinate system unification: Use satellite and inertial combined navigation positioning and attitude determination technology to process POS posture data to obtain the real-time absolute geographic coordinates and flight attitude of the drone; calculate the absolute geographic coordinates of the point cloud data obtained by LiDAR scanning and the absolute geographic coordinates and exterior orientation elements of each camera image through interpolation algorithms, and unify the coordinate systems between different sensors; Step 3: Camera imaging and coordinate conversion: Based on the pinhole imaging principle and combined with the calibration results, the conversion relationship between the world coordinate system, camera coordinate system, image coordinate system and pixel coordinate system is established; the coordinates of the LiDAR point cloud data and the camera image data are unified into the same reference coordinate system; Step 4: Determine the range of a single photo: Calculate the coverage of each photo based on the properties of similar triangles; Step 5, determine the coordinates of the upper right corner of a single photo in the e-coordinate system: determine the geographic coordinates of the upper right corner of each photo in a unified coordinate system; Step 6, judging whether the ground laser point is within the coverage area of the photo: comparing the coordinates of the ground laser point with the rectangular area covered by the photo to judge whether the ground laser point is within the coverage area of the specified photo; Step 7, back-projection calculation and color assignment: For the ground laser points within the area covered by the photo, back-projection calculation is performed using the pinhole imaging principle to obtain the coordinates of the ground laser points in the camera coordinate system, and then the pixel coordinates are calculated; the RGB information in the color table is decoded from the pixel and assigned to the RGB field of the point cloud to generate color point cloud data.
2. The method for generating real-time color point cloud onboard an unmanned aerial vehicle according to claim 1, characterized in that: The principle formula of LiDAR positioning is as follows: Formula 1: in: is the coordinate of the ground laser point P in the geocentric rectangular coordinate system, i.e., the e system; is the coordinate of the IMU center in the e system; It is the IMU coordinate system, i.e. the rotation matrix from the I system to the e system; It is the component expression of the bias from the Lidar scanning center to the IMU navigation center in the I system; is the rotation matrix from the Lidar coordinate system, i.e., the S system to the I system; is the coordinate of the ground laser point P in the S system; the coordinate (X, Y, Z) of the ground laser point in the e system is obtained by formula 1.
3. The method for generating real-time color point cloud onboard an unmanned aerial vehicle according to claim 1, characterized in that: The world coordinate system is denoted as OX w Y w Z w , used to describe the specific position of any object in three-dimensional space, using (X w ,Y w ,Z w ) represents the coordinate value; The camera coordinate system is denoted as SX c Y c Z c , the coordinate origin S is the optical center of the camera, the X-axis and Y-axis are parallel to the X-axis and Y-axis of the image coordinate system respectively, the Z-axis is the optical axis of the camera, and satisfies the right-hand coordinate system with the X-axis and Y-axis. c ,Y c ,Z c ) represents the coordinate value; there is a point P in the world coordinate system w (X w ,Y w ,Z w ), and obtain its coordinates in the camera coordinate system (X c ,Y c ,Z c ): Formula 2: Among them, R 3×3 is the rotation matrix, T 3×1 is the translation matrix, M is called the extrinsic parameter matrix; The image coordinate system is denoted as O-XY, the coordinate origin is the center of the image plane, the X-axis and the Y-axis are parallel to the two vertical sides of the image plane respectively, and (x, y) is used to represent the coordinate value of the pixel point on the image plane; The pixel coordinate system is denoted as O-UV, the coordinate origin is the upper left vertex of the image plane, the U axis and the V axis are parallel to the X axis and the Y axis of the image coordinate system respectively, and (u, v) is used to represent the coordinate value of the pixel point on the image plane; the image is stored in the computer using an M×N matrix, and each coordinate point in the image with M rows and N columns is called a pixel point, or pixel, and each pixel point contains the gray value of the image; There is a point (x, y) in the image coordinate system, and its corresponding pixel coordinate system is represented by (u, v). The conversion relationship is: Formula 3: Among them, (u0, v0) represents the coordinates of the origin of the image coordinate system in the pixel coordinate system, d x and d y Respectively represent the physical size of each pixel on the x-axis and y-axis.
4. The method for generating real-time color point cloud onboard an unmanned aerial vehicle according to claim 1, characterized in that: In the step of determining the range of a single photo, the coordinates of the upper right corner of the photo in the camera coordinate system are: Formula 4: Among them, PixelSize is the size of a single pixel; PixLenth and PixWidth are the number of pixels in the length and width directions of the photo respectively; H is the average altitude of the drone; and f is the focal length of the camera.
5. The method for generating real-time color point cloud onboard an unmanned aerial vehicle according to claim 1, characterized in that: In the step of determining the coordinates of the upper right corner of a single photo in the e-coordinate system, there is the following formula: Formula 5: in, The coordinates of the ground corner points in the photo coverage area in the e system; is the coordinate of the IMU center in the e system; is the rotation matrix from I system to e system; It is the component expression of the offset from the camera photography center to the IMU navigation center in the I system; is the rotation matrix from the photo coordinate system to the I system; The coordinates of the ground corner points of the photo coverage in the camera coordinate system.
6. The method for generating real-time color point cloud onboard an unmanned aerial vehicle according to claim 1, characterized in that: In the step of judging whether the ground laser point is within the area covered by the photo, the rectangle and the ground laser point are projected into the photo coordinate system, that is, the Z coordinate is reset to zero, and vector dot multiplication is used for judgment. The four points of the projected rectangular area are set as P1, P2, P3, and P4, and the ground laser point is P. If the following formula is satisfied, it is preliminarily judged that the projection of the ground laser point P is within the photo: Formula 6:
7. The method for generating real-time color point cloud onboard an unmanned aerial vehicle according to claim 1, characterized in that: In the back-projection calculation and coloring step, the pixel coordinates of the ground laser point corresponding to the photo are calculated using the back-projection formula, and it is determined whether the pixel coordinates exceed the maximum pixel size of the photo. If so, it is determined that the photo is not in the specified photo. The back-projection calculation formula for pixel coordinates is as follows: Formula 7: Where u and v are the pixel coordinates of the back-projection of the ground laser point on the photo; PixLenth and PixWidth are the length and width of the photo pixel size; f is the focal length of the camera; (X w ,Y w ,Z w ) is the ground laser point P w The coordinates of (X c ,Y c ,Z c ) is the coordinate of the camera photography center in the world coordinate system; PixelSize is the actual size of a single pixel.
8. The method for generating real-time color point cloud onboard an unmanned aerial vehicle according to claim 1, characterized in that: When calibrating the camera, a group of image control points are obtained by the camera calibration method, a group of compensation values are obtained by the group of image control points and the exterior orientation elements obtained by POS, and the average value of the group compensation values is calculated; the formula is as follows: Formula 8: Among them, R ex The rotation matrix generated for the angle compensation value; R pix4D It is the rotation matrix generated by the software pix4D; is the rotation matrix generated by the POS attitude angle; Δt is the delay between the camera shutter signal being sent and the shutter being pressed; X pix4D is the coordinate of the camera photography center obtained by the software pix4D; X pos is the coordinate of the camera photography center obtained from POS data; pos is the speed of the drone calculated from the POS coordinates.
