Three-dimensional point cloud map generation method and device based on pose information

Abstract

The invention discloses a three-dimensional point cloud map generation method and device based on pose information. The method comprises the following steps: acquiring multi-source data acquired by an unmanned aerial vehicle in a flight process, and performing time synchronization and fusion processing on the multi-source data to obtain target multi-source data; according to the target multi-source data, determining an image frame of which the pose change quantity is greater than a pose change threshold and / or the image feature change quantity is greater than an image change threshold in the flight process of the unmanned aerial vehicle as a key frame; performing feature matching on the plurality of key frames to obtain a matching relationship among the plurality of key frames; determining a plurality of key frames for displaying the same scene from different visual angles as a key frame group according to the matching relationship, and obtaining a plurality of key frame groups; and generating a three-dimensional point cloud map of the target area according to the plurality of key frame groups. According to the method, the technical problems of high calculation complexity, insufficient system real-time performance, strong dependence on GNSS signals and high difficulty in multi-source data fusion when an existing drawing method is used for drawing are solved.

Description

Technical Field

[0001] The present invention relates to the field of mapping technology, and in particular to a method and device for generating a three-dimensional point cloud map based on posture information. Background Art

[0002] Traditional UAV mapping methods usually rely on post-processing mode, that is, after the UAV completes the flight mission, the collected image data is transmitted to the ground station or other computing devices, and the image is stitched, corrected and mapped through software. Although this method has high accuracy, it has poor real-time performance and cannot meet the application scenarios that require rapid acquisition of map information, such as emergency rescue and dynamic environmental monitoring. This problem can be solved by using real-time mapping technology based on UAV posture information. The posture information includes the posture (including pitch angle, roll angle, heading angle) and position (latitude, longitude and altitude data obtained by GNSS) obtained in real time during the flight of the UAV. Combined with the mapping algorithm of posture information, the image map can be generated in real time while the UAV is flying, which significantly improves the timeliness of data processing.

[0003] The existing real-time mapping technology based on the posture information of unmanned aerial vehicles has shown some limitations in application, which are mainly reflected in the following aspects: First, the visual SLAM (Simultaneous Localization and Mapping) algorithm widely used in the prior art can realize the positioning and mapping of unmanned aerial vehicles in environments without GNSS signals or with poor signals, but its computational complexity is high and the hardware requirements of unmanned aerial vehicles are high. SLAM algorithms usually require high-performance processors and large-capacity storage devices to process image data and complex mathematical operations in real time. This significantly increases the manufacturing cost of unmanned aerial vehicles. At the same time, due to the occupation of computing resources, the reaction speed of the system may be affected, and it is difficult to meet the application scenarios with high real-time requirements. Secondly, the direct mapping method based on the combination of GNSS and images can provide high geographic accuracy in outdoor environments, but in complex terrain, urban canyons and other environments, GNSS signals are easily affected by occlusion and multipath effects, resulting in a decrease in positioning accuracy, which in turn affects the accuracy of mapping. In addition, this method relies on accurate GNSS positioning, and its mapping effect will be greatly reduced in the case of poor signal quality or signal interruption. At the same time, this method has high requirements for the real-time registration of images and GNSS data, and the algorithm implementation process is complex, which increases the difficulty of system design and implementation. In addition, existing real-time mapping technologies also face certain challenges when dealing with multi-source data fusion. Data from different sensors usually have spatiotemporal synchronization errors, and how to efficiently and accurately fuse these data is a difficulty. In traditional methods, complex calibration and filtering techniques are usually used to reduce errors, which not only increases the complexity of the system, but also puts higher requirements on the real-time processing capabilities of drones. In addition, the accumulation of errors may lead to a gradual decrease in mapping accuracy during long-term flight, which is a major limitation for some application scenarios that require high-precision and long-term monitoring. Finally, existing mapping algorithms usually require a lot of pre-preparation and post-processing, including sensor calibration, data post-processing and other steps. The complexity of these processes not only increases the difficulty of operation, but also limits the versatility and ease of use of drone systems to a certain extent.

[0004] In summary, although the existing technical solutions have solved the problem of real-time mapping of drones to a certain extent, they still have disadvantages such as high hardware requirements, high algorithm complexity, accuracy is greatly affected by the environment, and high difficulty in data fusion. That is, the following problems still exist: First, the real-time mapping algorithm based on visual SLAM has high requirements on computing resources, usually requiring a powerful processor and a large-capacity storage device, which puts high demands on the load capacity of the drone; secondly, the method of combining GNSS with images is difficult to guarantee accuracy and stability in an environment with weak signals or significant multipath effects.

[0005] There are usually problems such as high computational complexity, insufficient system real-time performance, strong dependence on GNSS signals, and difficulty in fusing multi-source data when using the above-mentioned existing mapping methods for mapping. No effective solution has been proposed yet. Summary of the invention

[0006] The embodiments of the present invention provide a method and device for generating a three-dimensional point cloud map based on posture information, so as to at least solve the technical problems that are usually present when using existing mapping methods for mapping, such as high computational complexity, insufficient system real-time performance, strong dependence on GNSS signals, and difficulty in multi-source data fusion.

[0007] According to one aspect of an embodiment of the present invention, a method for generating a three-dimensional point cloud map based on posture information is provided, comprising: acquiring multi-source data collected by a drone during flight, and performing time synchronization and fusion processing on the multi-source data to obtain target multi-source data, wherein the multi-source data refers to a plurality of sensor data collected by multiple sensors installed in the drone during flight of the drone; determining, according to the target multi-source data, an image frame in which a posture change amount of the drone during flight is greater than a posture change threshold and / or an image feature change amount is greater than an image change threshold as a key frame, wherein the posture change amount represents the position of the drone in the image frame. The image feature change refers to the change of the first posture of the drone in the previous image frame compared to the second posture of the drone in the previous image frame, and the image feature change refers to the change of the first image feature of the image frame compared to the second influencing feature of the previous image frame; feature matching is performed on multiple key frames to obtain a matching relationship between the multiple key frames; based on the matching relationship, multiple key frames that display the same scene from different perspectives are determined to be a key frame group to obtain multiple key frame groups; a three-dimensional point cloud map of the target area is generated based on the multiple key frame groups, wherein the target area is the area monitored by the drone, and the three-dimensional point cloud map is used to display the three-dimensional spatial structure of the target area.

[0008] Optionally, multi-source data collected by the drone during flight is obtained, and the multi-source data is time-synchronized and fused to obtain target multi-source data, including: obtaining the multi-source data collected by the drone using multiple sensors during flight, wherein the multi-source data includes at least visible light image data, inertial measurement unit IMU data, and global navigation satellite system GNSS positioning data; determining a first timestamp when the drone collects the visible light image data, a second timestamp when the IMU data is collected, and a third timestamp when the GNSS positioning data is collected; based on the second timestamp and the third timestamp, the first timestamp is corrected by interpolation or synchronization signal to achieve time synchronization processing of the multi-source data; after completing the time synchronization processing of the multi-source data, the multi-source data is fused to obtain the target multi-source data.

[0009] Optionally, the multi-source data are fused to obtain the target multi-source data, including: pre-processing the multi-source data by pre-integrating the IMU data, performing elevation filtering on the GNSS positioning data, and extracting and tracking features on the visible light image data; obtaining a pre-integration result of the pre-integration processing on the IMU data; processing the visible light image data using a structured motion algorithm SFM to obtain an SFM result; after aligning the pre-integration result with the SFM result, determining the posture information of the UAV according to the IMU data and the GNSS positioning data, wherein the IMU data is used to determine the attitude and speed of the UAV, and the GNSS positioning data is used to determine the position information of the UAV; matching and fusing the posture information with the image frames in the visible light image data to obtain the target multi-source data.

[0010] Optionally, determining, according to the target multi-source data, that an image frame in which a change in the posture of the UAV during flight is greater than a posture change threshold and / or an image feature change is greater than an image change threshold as a key frame comprises: determining, according to the posture information of the UAV, a posture corresponding to each image frame, wherein the posture information is information determined according to IMU data and GNSS positioning data in the multi-source data; determining, according to the multi-source data, using a first formula, the change in the posture of the UAV during flight, wherein the first formula is: ΔP i =‖P i -P i-1 ‖, i represents the number of the image frame, P i represents the first posture of the image frame of the i-th frame, P i-1 represents the second posture of the image frame of the i-1th frame, ΔP irepresents the change in posture of the image frame of the i-th frame compared to the image frame of the i-1-th frame, ‖·‖ represents the Euclidean distance, and the image frame of the i-1-th frame is the image frame before the image frame of the i-th frame; the change in posture is compared with the posture change threshold to obtain a first comparison result; feature extraction is performed on each of the image frames to obtain a plurality of first feature points of each of the image frames; the change in the image feature of the UAV during flight is determined by using a second formula according to the multi-source data, wherein the second formula is: ΔF i =∑ j ‖f i,j -f i-1,j ‖, j represents the label of the first feature point in any of the image frames, f i,j represents the jth first feature point in the i-th image frame, f i-1,j represents the jth first feature point in the i-1th image frame, ΔF i Representing the change in image features of the i-th image frame compared to the i-1-th image frame; comparing the image feature change with the image change threshold to obtain a second comparison result; when the first comparison result indicates that the posture change is greater than the posture change threshold, and / or the second comparison result indicates that the image feature change is greater than the image change threshold, determining that the image frame is the key frame.

[0011] Optionally, feature matching is performed on multiple key frames to obtain a matching relationship between the multiple key frames, including: performing feature extraction on each key frame to obtain multiple second feature points of each key frame; processing each second feature point using a descriptor algorithm to generate a feature descriptor for each second feature point, wherein the feature descriptor is a vector of local image features of the feature point; calculating a descriptor distance between the feature descriptors in any two adjacent key frames, wherein the descriptor distance is used to quantify the similarity between the two feature descriptors; when the descriptor distance is less than a distance threshold, the second feature points corresponding to the two feature descriptors are respectively regarded as a matching point group; correcting the second feature points in each group of the matching point groups through geometric constraints to obtain a correction result; when the correction result indicates that the second feature points in the matching point group meet the accuracy requirement, determining that the key frames corresponding to each second feature point in the matching point group have the matching relationship.

[0012] Optionally, generating a three-dimensional point cloud map of a target area according to a plurality of the key frame groups includes: obtaining the disparity between the key frames having the matching relationship in each of the key frame groups, wherein the disparity refers to the difference in pixel positions of the same scene imaged at different viewing angles; and calculating the depth value of the pixel point in the target sub-area using a third formula according to the disparity of each of the key frame groups to obtain a depth map of the target sub-area, wherein the third formula is: z represents the depth value, f represents the focal length of the image acquisition device, d represents the parallax, B represents the baseline length between the key frames having the matching relationship in the key frame group, the target sub-region refers to the sub-region displayed by each key frame group in the target region, the depth value represents the distance between the pixel point and the image acquisition device, and the depth map is an image constructed according to the depth value of each pixel point; the three-dimensional point cloud of each pixel point in the depth map corresponding to each target sub-region is restored by using the triangulation method to generate an initial three-dimensional point cloud map of the target region; the color information of each pixel point in each depth map is obtained; the initial three-dimensional point cloud map is filled with color according to the color information to obtain the three-dimensional point cloud map of the target region.

[0013] Optionally, after generating a three-dimensional point cloud map of the target area according to the plurality of key frame groups, the method for generating a three-dimensional point cloud map based on posture information further comprises: calculating the reprojection error of the second feature point in all the matching point groups using a fourth formula, wherein the fourth formula is: E reproj =∑ a ∑ b ‖Q a,b -π(R a X b +r a )‖ 2 , E reproj represents the reprojection error, a represents the number of the key frame, b represents the number of the second feature point, Q a,b represents the bth second feature point in the ath key frame, R a represents the rotation matrix of the key frame of frame a, X b represents the real coordinates corresponding to the bth second feature point in the ath key frame, r arepresents the translation vector of the key frame of the ath frame, and π() represents the projection function; with the goal of minimizing the reprojection error, the three-dimensional point cloud map is iteratively optimized using the optimization factor until the iteration termination condition is reached, and a first target three-dimensional point cloud map after the three-dimensional point cloud map is optimized is obtained, wherein the optimization factor at least includes: system state, prior factor, posture factor composed of posture information, GNSS observation factor, and visual factor, and the iteration termination condition at least includes: the number of iterations of iterative optimization of the three-dimensional point cloud map reaches a preset number of iterations, and the reprojection error is not greater than the error threshold; the GNSS coordinate system corresponding to the first target three-dimensional point cloud map is converted into an absolute coordinate system with actual physical meaning by using the fifth formula, so as to perform scale recovery processing on the first target three-dimensional point cloud map to obtain a second target three-dimensional point cloud map, wherein the fifth formula is: map =λ·Y fusion , Y map represents the first target three-dimensional point cloud map, Y fusion represents the second target three-dimensional point cloud map, and λ represents the scale factor.

[0014] According to another aspect of an embodiment of the present invention, a three-dimensional point cloud map generation device based on posture information is also provided, including: a first acquisition unit, used to acquire multi-source data collected by a drone during flight, and perform time synchronization and fusion processing on the multi-source data to obtain target multi-source data, wherein the multi-source data refers to a variety of sensor data collected during the flight of the drone using multiple sensors installed in the drone; a determination unit, used to determine, based on the target multi-source data, that an image frame in which a posture change amount of the drone during flight is greater than a posture change threshold and / or an image feature change amount is greater than an image change threshold is a key frame, wherein the posture change amount represents the first position of the drone in the image frame. The image feature change refers to the change of the first image feature of the image frame compared to the second influencing feature of the previous image frame; the second acquisition unit is used to perform feature matching on multiple key frames to obtain a matching relationship between the multiple key frames; the third acquisition unit is used to determine, based on the matching relationship, that multiple key frames showing the same scene from different perspectives are a key frame group to obtain multiple key frame groups; the generation unit is used to generate a three-dimensional point cloud map of the target area based on the multiple key frame groups, wherein the target area is the area monitored by the drone, and the three-dimensional point cloud map is used to display the three-dimensional spatial structure of the target area.

[0015] Optionally, the first acquisition unit includes: a first acquisition module, used to acquire the multi-source data collected by the UAV using multiple sensors during flight, wherein the multi-source data includes at least visible light image data, inertial measurement unit IMU data, and global navigation satellite system GNSS positioning data; a first determination module, used to determine the first timestamp when the UAV collects the visible light image data, the second timestamp when the IMU data is collected, and the third timestamp when the GNSS positioning data is collected; a correction module, used to correct the first timestamp based on the second timestamp and the third timestamp through interpolation or synchronization signal to achieve time synchronization processing of the multi-source data; a second acquisition module, used to fuse the multi-source data after completing the time synchronization processing of the multi-source data to obtain the target multi-source data.

[0016] Optionally, the second acquisition module includes: a preprocessing submodule, used to complete the preprocessing of the multi-source data by pre-integrating the IMU data, performing elevation filtering on the GNSS positioning data, and extracting and tracking features on the visible light image data; a first acquisition submodule, used to obtain the pre-integration result of the pre-integration processing of the IMU data; a second acquisition submodule, used to process the visible light image data using the structure from motion algorithm SFM to obtain an SFM result; a determination submodule, used to determine the posture information of the UAV according to the IMU data and the GNSS positioning data after aligning the pre-integration result with the SFM result, wherein the IMU data is used to determine the attitude and speed of the UAV, and the GNSS positioning data is used to determine the position information of the UAV; a third acquisition submodule, used to match and fuse the posture information with the image frame in the visible light image data to obtain the target multi-source data.

[0017] Optionally, the determination unit includes: a second determination module, used to determine the posture corresponding to each of the image frames according to the posture information of the drone, wherein the posture information is information determined according to the IMU data and GNSS positioning data in the multi-source data; a third determination module, used to determine the posture change amount of the drone during flight according to the multi-source data using a first formula, wherein the first formula is: ΔP i =‖P i -P i-1 ‖, i represents the number of the image frame, P i represents the first posture of the image frame of the i-th frame, P i-1 represents the second posture of the image frame of the i-1th frame, ΔP irepresents the posture change of the i-th image frame compared to the i-1-th image frame, ‖·‖ represents the Euclidean distance, and the i-1-th image frame is the previous image frame of the i-th image frame; a third acquisition module is used to compare the posture change with the posture change threshold to obtain a first comparison result; a fourth acquisition module is used to extract features from each of the image frames to obtain multiple first feature points of each of the image frames; a fourth determination module is used to determine the image feature change of the drone during flight using a second formula based on the multi-source data, wherein the second formula is: ΔF i =∑ j ‖f i,j -f i-1,j ‖, j represents the label of the first feature point in any of the image frames, f i,j represents the jth first feature point in the i-th image frame, f i-1,j represents the jth first feature point in the i-1th image frame, ΔF i Represents the change in image features of the i-th image frame compared to the i-1-th image frame; a fifth acquisition module, used to compare the image feature change with the image change threshold to obtain a second comparison result; a fifth determination module, used to determine that the image frame is the key frame when the first comparison result indicates that the posture change is greater than the posture change threshold, and / or the second comparison result indicates that the image feature change is greater than the image change threshold.

[0018] Optionally, the second acquisition unit includes: a sixth acquisition module, used to perform feature extraction on each of the key frames to obtain multiple second feature points of each of the key frames; a first generation module, used to process each of the second feature points using a descriptor algorithm to generate a feature descriptor for each of the second feature points, wherein the feature descriptor is a vector of local image features of the feature point; a calculation module, used to calculate the descriptor distance between the feature descriptors in any two adjacent key frames, wherein the descriptor distance is used to quantify the similarity of the two feature descriptors; a sixth determination module, used to treat the second feature points corresponding to the two feature descriptors as a matching point group when the descriptor distance is less than a distance threshold; a seventh acquisition module, used to correct the second feature points in each group of the matching point groups through geometric constraints to obtain a correction result; and a seventh determination module, used to determine that the key frames corresponding to the second feature points in the matching point group have the matching relationship when the correction result indicates that the second feature points in the matching point group meet the accuracy requirement.

[0019] Optionally, the generating unit comprises: an eighth acquisition module, configured to acquire the disparity between the key frames having the matching relationship in each of the key frame groups, wherein the disparity refers to the difference in pixel positions of the same scene imaged at different viewing angles; and a ninth acquisition module, configured to calculate the depth value of the pixel point in the target sub-region using a third formula according to the disparity of each of the key frame groups to obtain a depth map of the target sub-region, wherein the third formula is: z represents the depth value, f represents the focal length of the image acquisition device, d represents the parallax, B represents the baseline length between the key frames having the matching relationship in the key frame group, the target sub-region refers to the sub-region displayed by each key frame group in the target region, the depth value represents the distance between the pixel point and the image acquisition device, and the depth map is an image constructed according to the depth value of each pixel point; the second generation module is used to restore the three-dimensional point cloud of each pixel point in the depth map corresponding to each target sub-region by using the triangulation method to generate an initial three-dimensional point cloud map of the target region; the tenth acquisition module is used to obtain the color information of each pixel point in each depth map; the eleventh acquisition module is used to fill the initial three-dimensional point cloud map according to the color information to obtain the three-dimensional point cloud map of the target area.

[0020] Optionally, the three-dimensional point cloud map generation device based on posture information further includes: a calculation unit, which is used to calculate the reprojection error of the second feature point in all the matching point groups using a fourth formula after generating the three-dimensional point cloud map of the target area according to the plurality of key frame groups, wherein the fourth formula is: E reproj =∑ a ∑ b ‖Q a,b -π(R a X b +r a )‖ 2 , E reproj represents the reprojection error, a represents the number of the key frame, b represents the number of the second feature point, Q a,b represents the bth second feature point in the ath key frame, R a represents the rotation matrix of the key frame of frame a, X b represents the real coordinates corresponding to the bth second feature point in the ath key frame, r arepresents the translation vector of the key frame of the ath frame, and π() represents the projection function; a fourth acquisition unit, used to iteratively optimize the three-dimensional point cloud map using the optimization factor with the goal of minimizing the reprojection error, until the iteration termination condition is reached, and obtain the first target three-dimensional point cloud map after the three-dimensional point cloud map is optimized, wherein the optimization factor at least includes: system state, prior factor, posture factor composed of posture information, GNSS observation factor, and visual factor, and the iteration termination condition at least includes: the number of iterations of iterative optimization of the three-dimensional point cloud map reaches a preset number of iterations, and the reprojection error is not greater than the error threshold; a fifth acquisition unit, used to convert the GNSS coordinate system corresponding to the first target three-dimensional point cloud map into an absolute coordinate system with actual physical meaning using the fifth formula, so as to perform scale recovery processing on the first target three-dimensional point cloud map to obtain a second target three-dimensional point cloud map, wherein the fifth formula is: map =λ·Y fusion , Y map represents the first target three-dimensional point cloud map, Y fusion represents the second target three-dimensional point cloud map, and λ represents the scale factor.

[0021] According to another aspect of an embodiment of the present invention, a three-dimensional point cloud map generation system based on posture information is also provided. The three-dimensional point cloud map generation system based on posture information uses any of the three-dimensional point cloud map generation methods based on posture information described above.

[0022] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored program, wherein the program executes any one of the above-mentioned methods for generating a three-dimensional point cloud map based on posture information.

[0023] According to another aspect of an embodiment of the present invention, a processor is further provided, wherein the processor is used to run a program, wherein when the program is run, any one of the above-mentioned methods for generating a three-dimensional point cloud map based on posture information is executed.

[0024] According to another aspect of an embodiment of the present invention, a computer program product is also provided, comprising computer instructions, which, when executed by a processor, execute any one of the above-mentioned methods for generating a three-dimensional point cloud map based on posture information.

[0025] In an embodiment of the present invention, multi-source data collected by a drone during flight is obtained, and time synchronization and fusion processing are performed on the multi-source data to obtain target multi-source data, wherein the multi-source data refers to multiple sensor data collected by multiple sensors installed in the drone during flight; according to the target multi-source data, an image frame whose posture change amount of the drone during flight is greater than a posture change threshold and / or whose image feature change amount is greater than an image change threshold is determined as a key frame, wherein the posture change amount represents the change amount of the first posture of the drone in the image frame compared to the second posture of the drone in the previous image frame, and the image feature change amount refers to the change amount of the first image feature of the image frame compared to the second influencing feature of the previous image frame; feature matching is performed on multiple key frames to obtain a matching relationship between the multiple key frames; according to the matching relationship, multiple key frames showing the same scene from different perspectives are determined as a key frame group to obtain multiple key frame groups; a three-dimensional point cloud map of the target area is generated according to the multiple key frame groups, wherein the target area is an area monitored by the drone, and wherein the three-dimensional point cloud map is used to display the three-dimensional spatial structure of the target area. Through the above technical scheme, the purpose of time synchronization and fusion of multi-source data collected during the flight of the UAV is achieved, and the target area monitored by the UAV is mapped in real time by selecting key frames, and the technical effect of generating a three-dimensional point cloud map of the target area by analyzing and processing multi-source data is achieved, which improves the real-time and robustness of the system, simplifies the complexity of the system, and improves the efficiency and accuracy of mapping, thereby solving the technical problems that are usually present when using existing mapping methods for mapping, such as high computational complexity, insufficient system real-time performance, strong dependence on GNSS signals, and difficulty in multi-source data fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0027] Figure 1 It is a hardware structure block diagram of a mobile terminal of a method for generating a three-dimensional point cloud map based on posture information according to an embodiment of the present invention;

[0028] Figure 2 is a flow chart of a method for generating a three-dimensional point cloud map based on posture information according to an embodiment of the present invention;

[0029] Figure 3 is a flowchart of an optional method for generating a three-dimensional point cloud map based on posture information according to an embodiment of the present invention;

[0030] Figure 4 is a flowchart of multi-source data time synchronization and fusion according to an embodiment of the present invention;

[0031] Figure 5 is a schematic diagram of selecting key frames according to an embodiment of the present invention;

[0032] Figure 6 is a schematic diagram of constructing a three-dimensional point cloud map according to an embodiment of the present invention;

[0033] Figure 7 is a schematic diagram of optimizing a three-dimensional point cloud map according to an embodiment of the present invention;

[0034] Figure 8 is a schematic diagram of a three-dimensional point cloud map generating device based on posture information according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] As introduced in the background technology, when using existing mapping methods for mapping, there are usually problems such as high computational complexity, insufficient system real-time performance, strong dependence on GNSS signals, and difficulty in multi-source data fusion. In view of the above shortcomings, a method and device for generating a three-dimensional point cloud map based on pose information are provided in an embodiment of the present invention.

[0038] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0039] The method embodiments provided in the embodiments of the present invention can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of a method for generating a three-dimensional point cloud map based on position information according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0040] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the three-dimensional point cloud map generation method based on posture information in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks and combinations thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific network examples may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0041] According to an embodiment of the present invention, a method embodiment of a method for generating a three-dimensional point cloud map based on posture information is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0042] Figure 2 is a flow chart of a method for generating a three-dimensional point cloud map based on pose information according to an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0043] Step S202, obtaining multi-source data collected by the drone during flight, and performing time synchronization and fusion processing on the multi-source data to obtain target multi-source data, wherein the multi-source data refers to a variety of sensor data collected during the flight of the drone using multiple sensors installed in the drone.

[0044] Combine the following Figure 3 The above embodiments of the present invention are described in detail. Figure 3 is a flowchart of an optional method for generating a three-dimensional point cloud map based on posture information according to an embodiment of the present invention; Figure 3 As shown, when the method provided by the embodiment of the present invention is used to map the three-dimensional point cloud map of the target area, the following four steps are generally included: 1) time synchronization and fusion of multi-source data; 2) key frame determination and matching; 3) dense reconstruction and color projection; 3) pose graph optimization and scale recovery.

[0045] In this embodiment, since the drone collects multiple sensor data during flight, including visible light images, inertial measurement unit (IMU) data, global navigation satellite system GNSS positioning data, etc. (i.e., multi-source data), these data usually have different sampling frequencies and timestamps, and are generated in different hardware modules, so these multi-source data must undergo time synchronization and data fusion in order to be effectively processed in the same space-time framework.

[0046] Below and above Figure 3 The time synchronization and fusion data of the multi-source data shown are explained in detail.

[0047] According to the above embodiment of the present invention, in the above step S202, multi-source data collected by the drone during flight is obtained, and the multi-source data is time-synchronized and fused to obtain target multi-source data, including: obtaining multi-source data collected by the drone using multiple sensors during flight, wherein the multi-source data at least includes: visible light image data, inertial measurement unit IMU data, and global navigation satellite system GNSS positioning data; determining a first timestamp for the drone to collect visible light image data, a second timestamp for collecting IMU data, and a third timestamp for collecting GNSS positioning data; based on the second timestamp and the third timestamp, the first timestamp is corrected by interpolation or synchronization signal to achieve time synchronization processing of the multi-source data; after completing the time synchronization processing of the multi-source data, the multi-source data is fused to obtain the target multi-source data.

[0048] Combine the following Figure 4 The above embodiments of the present invention are described in detail. Figure 4 is a flowchart of multi-source data time synchronization and fusion according to an embodiment of the present invention. Figure 4 As shown in Figure 1, in order to ensure the time synchronization of different data sources, the timestamps of the sensor data must first be processed uniformly. Since GNSS data and IMU data usually have high time accuracy and can be used as time references, the timestamps of visible light image data need to be corrected through interpolation or synchronization signals to ensure that the time points corresponding to all data are consistent. Common methods include linear interpolation based on timestamps, or using hardware-triggered synchronization to ensure that each frame of image is synchronized with the corresponding pose data. In the process of time synchronization and data fusion, it is assumed that the timestamp of the IMU data is t IMU , the timestamp of GNSS data is t GNSS , the timestamp of the image data is t image , after interpolation or synchronization signal correction, the unified time stamp of all data is t sync , the time synchronization formula is:

[0049] In the above embodiment of the present invention, multi-source data are fused to obtain target multi-source data, including: pre-processing the multi-source data by pre-integrating the IMU data, performing elevation filtering on the GNSS positioning data, and extracting and tracking features on the visible light image data; obtaining the pre-integration result of the pre-integration processing of the IMU data; processing the visible light image data using the structured motion algorithm SFM to obtain the SFM result; after aligning the pre-integration result with the SFM result, determining the posture information of the drone according to the IMU data and the GNSS positioning data, wherein the IMU data is used to determine the posture and speed of the drone, and the GNSS positioning data is used to determine the position information of the drone; matching and fusing the posture information with the image frames in the visible light image data to obtain the target multi-source data.

[0050] As above Figure 4 As shown in the figure, when fusing multi-source data after time synchronization, first pre-process the IMU data, GNSS data and visible light image data, pre-integrate the IMU data, perform elevation filtering on the GNSS data to obtain stable data, and extract and track feature points on the visible light image; then initialize the pre-processed data, introduce a coarse-to-fine method to gradually optimize the positioning information of the GNSS raw data, and align the results of the IMU pre-integration with the results after the visible light image SFM. After all data are initialized, enter the factor graph optimization process, and use the factor graph to optimize the multi-source data to obtain more accurate pose information. Finally, pair each frame of the image with its corresponding pose information to form an image sequence with spatiotemporal information (i.e., target multi-source data). Data fusion can provide a reliable basis for subsequent key frame extraction and matching. After data fusion, the pose P generated by IMU and GNSS is fusion (t) is expressed as: P fusion (t) = P GNSS (t)+R IMU (t)×P IMU (t), where R IMU (t) represents the rotation matrix provided by the IMU, which is used to transform the pose in the IMU coordinate system to the global coordinate system.

[0051] Step S204, determining, based on the target multi-source data, an image frame in which the change in the drone's posture during flight is greater than a posture change threshold and / or the change in image features is greater than an image change threshold as a key frame, wherein the posture change represents the change in the first posture of the drone in the image frame compared to the second posture of the drone in the previous image frame, and the image feature change refers to the change in the first image feature of the image frame compared to the second influencing feature of the previous image frame.

[0052] In this embodiment, in order to achieve efficient map reconstruction under limited computing resources, key frames can be used to process the image sequence. Key frames refer to a group of representative image frames selected during the flight process for map generation and optimization. The selection criteria of key frames directly affect the accuracy and computing efficiency of the map. Through a reasonable key frame selection strategy, redundant calculations can be reduced, a high map update speed can be maintained, and the accuracy of the map can be ensured. For example, representative image frames can be dynamically selected as key frames based on the two indicators of the drone's posture change and image feature change during flight.

[0053] Below and above Figure 3 The determination of the key frames shown is described in detail with the determination of the key frames in the matching.

[0054] According to the above embodiment of the present invention, in the above step S204, determining the image frame whose posture change amount of the drone during flight is greater than the posture change threshold and / or whose image feature change amount is greater than the image change threshold as a key frame according to the target multi-source data includes: determining the posture corresponding to each image frame according to the posture information of the drone, wherein the posture information is information determined according to the IMU data and the GNSS positioning data in the multi-source data; determining the posture change amount of the drone during flight according to the multi-source data using the first formula, wherein the first formula is: ΔP i =‖P i -P i-1 ‖, i represents the number of the image frame, P i represents the first pose of the i-th image frame, P i-1 represents the second pose of the i-1th frame, ΔP i represents the change in posture of the i-th image frame compared to the i-1-th image frame, ‖·‖ represents the Euclidean distance, and the i-1-th image frame is the previous image frame of the i-th image frame; the change in posture is compared with the posture change threshold to obtain a first comparison result; feature extraction is performed on each image frame to obtain multiple first feature points of each image frame; the image feature change of the drone during flight is determined using the second formula based on multi-source data, where the second formula is: ΔF i =∑ j ‖f i,j -f i-1,j ‖, j represents the number of the first feature point in any image frame, f i,j represents the jth first feature point in the i-th image frame, f i-1,j represents the jth first feature point in the i-1th image frame, ΔF iRepresents the change in image features between the i-th image frame and the i-1-th image frame; compares the image feature change with the image change threshold to obtain a second comparison result; when the first comparison result indicates that the posture change is greater than the posture change threshold, and / or the second comparison result indicates that the image feature change is greater than the image change threshold, determines that the image frame is a key frame.

[0055] Combine the following Figure 5 The above embodiments of the present invention are described in detail. Figure 5 is a schematic diagram of selecting key frames according to an embodiment of the present invention, such as Figure 5 As shown in FIG. 1 , the selection of key frames is usually based on the motion state of the drone and the amount of change in the image content. Common selection criteria include: when the posture of the drone changes significantly (such as a large change in the flight direction or speed), or the image content changes dramatically (such as a significant change in environmental characteristics), the current image frame can be selected as a key frame; in addition, a threshold can be set according to the time interval or distance interval. When the drone flies a certain distance or time, the key frame is automatically selected. These selection methods can effectively reduce repeated information and reduce the computational burden. In this embodiment, a moving distance of 3m is initially selected as the threshold for key frame selection. For the selection of key frames, the posture change ΔP between image frames can be calculated. i and image feature change ΔF i To determine whether to select a key frame; the formula for the change in posture is: ΔP i =‖P i -P i-1 ‖, where P i represents the pose of the i-th image frame, P i-1 represents the pose of the i-1th image frame, ΔP i represents the change in position of the i-th image frame compared to the i-1-th image frame, the i-1-th image frame is the previous image frame of the i-th image frame, ‖·‖ represents the Euclidean distance, and the change in position ΔP i It indicates the motion change of the drone between adjacent frames. When the posture change of the drone exceeds a certain threshold, it indicates that the current image frame (i.e., the i-th image frame) may contain important new information and can be selected as a key frame. The change in image features is measured by the feature descriptor distance after the first feature point is matched. The Euclidean distance ΔF i :ΔF i =∑ j ‖f i,j -f i-1,j ‖, where f i-1,j represents the jth feature point in the i-1th image frame, ΔF i The image feature change ΔF represents the change in image feature between the i-th image frame and the i-1-th image frame. iIt reflects the change in image content between adjacent frames. When the change in image features is greater than the set threshold, it means that the scene has changed significantly, and the current image frame can be selected as a key frame.

[0056] By combining these two indicators, representative image frames can be dynamically selected as key frames. Reasonable selection of key frames can not only reduce the amount of data processing, but also ensure that the selected image frames have sufficient information representativeness. i or ΔF i When setting the threshold, select the current image frame as the key frame.

[0057] Step S206: performing feature matching on the multiple key frames to obtain matching relationships between the multiple key frames.

[0058] In this embodiment, after the key frames are selected, feature matching is required for adjacent key frames to establish spatial associations between them. The purpose of feature matching is to find the same scene points that appear in different key frames for subsequent reconstruction and optimization.

[0059] Below and above Figure 3 The determination of the key frames shown and the matching of the key frames in the matching are described in detail.

[0060] According to the above embodiment of the present invention, in the above step S206, feature matching is performed on multiple key frames to obtain a matching relationship between the multiple key frames, including: feature extraction is performed on each key frame to obtain multiple second feature points of each key frame; each second feature point is processed by a descriptor algorithm to generate a feature descriptor for each second feature point, wherein the feature descriptor is a vector of local image features of the feature point; the descriptor distance between the feature descriptors in any two adjacent key frames is calculated, wherein the descriptor distance is used to quantify the similarity between the two feature descriptors; when the descriptor distance is less than a distance threshold, the second feature points corresponding to the two feature descriptors are respectively regarded as a matching point group; the second feature points in each group of matching point groups are corrected by geometric constraints to obtain a correction result; when the correction result indicates that the second feature points in the matching point group meet the accuracy requirement, it is determined that the key frames corresponding to the second feature points in the matching point group have a matching relationship.

[0061] Specifically, a method based on image feature points can be used for feature matching; first, the second feature points in the key frame image can be extracted by the GFTT algorithm. These second feature points are rotation and scale invariant and can remain consistent under different viewing angles; next, the second feature points can be matched using descriptors, and matching pairs can be determined by calculating the descriptor distance between the second feature points. For example, two second feature points whose descriptor distance is less than a distance threshold can be considered as a matching pair (i.e., a matching point group), and geometric constraint correction can be performed through the essential matrix or the basic matrix to ensure the correctness of the matching. For example, the following formula can be used: (Here a and c both represent keyframe numbers, used to represent two different keyframes) to perform geometric constraint correction, where p a and p c are the corresponding matching points in the two key frames (i.e., the two second feature points in the matching point group), and A is the essential matrix. Through this geometric constraint, the wrong matching points can be eliminated, and the matching points with correct geometric relationships can be retained. It can also be determined that there is a matching relationship between the key frames corresponding to the second feature points in each group of matching point groups.

[0062] In addition, after completing feature matching, further pose optimization can be performed. Through the bundle adjustment algorithm, the reprojection error of all matching points is minimized, thereby optimizing the pose of all key frames and the three-dimensional position of feature points. The reprojection error formula is: E reproj =∑ a ∑ b ‖Q a,b -π(R a X b +r a )‖ 2 , E reproj represents the reprojection error, a represents the number of the key frame, b represents the number of the second feature point, Q a,b represents the bth second feature point in the ath key frame, R a represents the rotation matrix of the a-th key frame, X b represents the real coordinates corresponding to the bth second feature point in the ath key frame, r a Represents the translation vector of the a-th key frame, and π() represents the projection function; by minimizing the reprojection error, the accuracy of pose estimation is improved, and accurate input data is provided for subsequent dense reconstruction.

[0063] Step S208: Determine, based on the matching relationship, that a plurality of key frames showing the same scene from different viewing angles are a key frame group, thereby obtaining a plurality of key frame groups.

[0064] In this embodiment, multiple key frames showing the same scene from different viewing angles may be determined as a key frame group according to the obtained matching relationship, thereby obtaining multiple key frame groups.

[0065] Step S210, generating a three-dimensional point cloud map of the target area according to the multiple key frame groups, wherein the target area is the area monitored by the drone, and wherein the three-dimensional point cloud map is used to display the three-dimensional spatial structure of the target area.

[0066] In this embodiment, based on the key frame determination and matching, dense reconstruction and color projection can be further performed. Dense reconstruction means generating a high-density point cloud starting from sparse feature points, and assigning color information to each point to form a colored three-dimensional point cloud map. The core goal of dense reconstruction is to generate a high-resolution three-dimensional point cloud. Compared with the sparse feature point matching results, dense reconstruction requires depth estimation on each pixel of the image to form a dense three-dimensional point set. This process is usually divided into two stages: disparity calculation and triangulation.

[0067] Below and above Figure 3 The dense reconstruction is shown with color projection for detailed illustration.

[0068] According to the above embodiment of the present invention, in the above step S210, a three-dimensional point cloud map of the target area is generated according to multiple key frame groups, including: obtaining the disparity between key frames having a matching relationship in each key frame group, wherein the disparity refers to the difference in pixel positions of the same scene imaged at different viewing angles; and calculating the depth value of the pixel point in the target sub-area using the third formula according to the disparity of each key frame group to obtain a depth map of the target sub-area, wherein the third formula is: z represents the depth value, f represents the focal length of the image acquisition device, d represents the parallax, B represents the baseline length between the key frames with matching relationship in the key frame group, the target sub-region refers to the sub-region displayed by each key frame group in the target area, the depth value represents the distance between the pixel point and the image acquisition device, and the depth map is an image composed of the depth values ​​of each pixel point; the three-dimensional point cloud of each pixel point in the depth map corresponding to each target sub-region is restored by triangulation method to generate an initial three-dimensional point cloud map of the target area; the color information of each pixel point in each depth map is obtained; the initial three-dimensional point cloud map is filled with color according to the color information to obtain a three-dimensional point cloud map of the target area.

[0069] Combine the following Figure 6 The above embodiments of the present invention are described in detail. Figure 6 It is a schematic diagram of constructing a three-dimensional point cloud map according to an embodiment of the present invention.

[0070] like Figure 6As shown in the figure, dense reconstruction refers to imaging the same object on the ground from images from different perspectives. First, the depth map is obtained by parallax calculation. Parallax refers to the displacement difference of the same scene point under different perspectives. Assuming that the baseline length between the two image frames is B and the focal length of the camera is f, the relationship between the parallax d and the depth value z is: In actual operation, the multi-view projection method calculates the disparity through feature point matching and epipolar geometry constraints to generate a depth map. The depth map represents the distance information of each pixel in the scene. This distance information can be used to restore the 3D point cloud. Then, the spatial coordinates of the 3D point are restored from the depth map using triangulation. The 3D point is The corresponding projection point on the plane is (u, v), and the projection model can be expressed as: Among them, K is the camera's intrinsic parameter matrix, including parameters such as focal length and principal point offset; through this projection model, the corresponding 3D point cloud can be restored from the 2D pixel coordinates of the image. In the dense reconstruction process of multi-view projection, images from multiple perspectives are usually used to improve the accuracy and density of reconstruction. By integrating information from multiple perspectives, occlusion problems can be effectively reduced and high-quality 3D reconstruction results can be generated. In short, the depth map provides depth information from 2D pixels to 3D space coordinates, and the triangulation algorithm uses this depth information and multi-perspective information to calculate the spatial coordinates of 3D points, thereby achieving 3D reconstruction, which is equivalent to the correspondence between 3D points in 3D space coordinates and 2D pixels.

[0071] After dense reconstruction is completed, the generated 3D point cloud usually only contains geometric position information. In order to enhance the visualization effect of the map, it is necessary to project the color information of the image onto the point cloud to generate a color point cloud map. Specifically, for each 3D point X, its projection point (u, v) on the image plane corresponds to a pixel position in the image. By finding the RGB value (R, G, B) of the pixel position, the color information can be assigned to the corresponding 3D point. This process needs to consider the internal and external parameters of the camera and the geometric relationship of multi-view images to ensure the accuracy of color projection. Since each 3D point may be observed by multiple image frames, it needs to be processed during color projection to avoid color conflicts. The weighted averaging method or the angle-based selective projection strategy can be used to ensure that the color of each 3D point is as close to the actual situation as possible. Through color projection, the generated color point cloud map not only has a high-precision spatial structure, but also has rich color information, which is convenient for subsequent analysis and application.

[0072] In an optional embodiment of the present invention, after generating a three-dimensional point cloud map of the target area according to a plurality of key frame groups, the method for generating a three-dimensional point cloud map based on posture information further includes: calculating the reprojection error of the second feature point in all matching point groups using a fourth formula, wherein the fourth formula is: E reproj =∑ a ∑ b ‖Q a,b -π(R a X b +r a )‖ 2 , E reproj represents the reprojection error, a represents the number of the key frame, b represents the number of the second feature point, Q a,b represents the bth second feature point in the ath key frame, R a represents the rotation matrix of the a-th key frame, X b represents the real coordinates corresponding to the bth second feature point in the ath key frame, r a represents the translation vector of the a-th frame key frame, π() represents the projection function; with the goal of minimizing the reprojection error, the three-dimensional point cloud map is iteratively optimized using the optimization factor until the iteration termination condition is reached, and the first target three-dimensional point cloud map after the three-dimensional point cloud map is optimized is obtained, wherein the optimization factor at least includes: system state, prior factor, posture factor composed of posture information, GNSS observation factor, and visual factor, and the iteration termination condition at least includes: the number of iterations of iterative optimization of the three-dimensional point cloud map reaches the preset number of iterations, and the reprojection error is not greater than the error threshold; the GNSS coordinate system corresponding to the first target three-dimensional point cloud map is converted into an absolute coordinate system with actual physical meaning using the fifth formula, so as to perform scale recovery processing on the first target three-dimensional point cloud map to obtain a second target three-dimensional point cloud map, wherein the fifth formula is: map =λ·Y fusion , Y map represents the first target 3D point cloud map, Y fusion represents the second target 3D point cloud map, and λ represents the scale factor.

[0073] In this embodiment, after completing dense reconstruction and color projection, in order to further improve the accuracy and consistency of the map, global pose graph optimization and scale recovery can be performed to correct the map distortion caused by sensor noise or accumulated errors, and ensure the accuracy of the generated color point cloud map in the global coordinate system; Figure 7 The above embodiments of the present invention and Figure 3 The pose graph optimization and scale recovery shown in the figure are explained in detail. Figure 7 It is a schematic diagram of optimizing a three-dimensional point cloud map according to an embodiment of the present invention.

[0074] Specifically, pose graph optimization is achieved by minimizing the reprojection error between all keyframes. During flight, the pose of the drone will be disturbed by various external factors, such as vibration, wind speed changes, etc., which will cause cumulative errors in pose estimation, thereby affecting the accuracy of the map. Global pose graph optimization methods can be used to correct these errors; for example, an optimization problem that includes the transition edges between all keyframes and the constraints of the original GNSS measurement data can be constructed. The goal of the optimization is to adjust the pose of the keyframes by minimizing the reprojection error, thereby improving the consistency and accuracy of the overall map, such as Figure 7 As shown in the figure, the pose graph in the pose graph optimization includes system status, prior factors, pose factors composed of pose information, GNSS observation factors (clock factors and pseudorange factors), visual factors, etc. The joint optimization of multiple factors can significantly improve the accuracy of the constructed map. Through pose graph optimization, the pose of each key frame and the position of the corresponding three-dimensional point can be optimized globally, thereby eliminating cumulative errors and ensuring the overall accuracy and consistency of the map.

[0075] After the pose graph is optimized, since a monocular visible light vision camera is used, the generated map (here refers to the first target 3D point cloud map) is usually presented in relative scale, that is, the distance and position on the map do not have absolute physical meaning. In order to convert the relative pose into an absolute coordinate system with actual physical meaning, scale recovery is required. Scale recovery is achieved by combining the relative pose information of the visual system with the absolute position information provided by GNSS. Specifically, by calculating the scale factor λ between the GNSS coordinates of the key frame and the corresponding relative pose, the scale of the visual pose can be adjusted to the actual physical scale. The specific calculation formula of the scale factor λ is: in, and denote the GNSS positions of the ath and bth keyframes respectively, and They represent the pose estimated by the visual system. By introducing the scale factor, the entire point cloud map can be adjusted to an object coordinate system with an actual physical scale. The calculation formula is: map =λ·Y fusion The generated color point cloud map (here refers to the second target three-dimensional point cloud map) not only has a high-precision spatial structure and rich color information, but also contains an absolute geographic location scale.

[0076] It can be seen from the above that through the technical solution provided by the above embodiment of the present invention, multi-source data collected by the drone during flight can be obtained, and the multi-source data can be time-synchronized and fused to obtain target multi-source data, wherein the multi-source data refers to a variety of sensor data collected during the flight of the drone using multiple sensors installed in the drone; according to the target multi-source data, the image frames in which the change in the posture of the drone during flight is greater than the posture change threshold and / or the change in image features is greater than the image change threshold are determined as key frames, wherein the posture change represents the change in the first posture of the drone in the image frame compared to the second posture of the drone in the previous image frame, and the image feature change refers to the change in the first image feature of the image frame compared to the second influencing feature of the previous image frame; for multiple key frames Perform feature matching to obtain a matching relationship between multiple key frames; determine multiple key frames that display the same scene from different perspectives as a key frame group based on the matching relationship, and obtain multiple key frame groups; generate a three-dimensional point cloud map of the target area based on the multiple key frame groups, wherein the target area is the area monitored by the drone, wherein the three-dimensional point cloud map is used to display the three-dimensional spatial structure of the target area, thereby achieving the purpose of time synchronization and fusion of multi-source data collected during the flight of the drone, and real-time mapping of the target area monitored by the drone by selecting key frames, realizing the technical effect of generating a three-dimensional point cloud map of the target area by analyzing and processing multi-source data, improving the real-time and robustness of the system, simplifying the complexity of the system, and improving the efficiency and accuracy of mapping.

[0077] Therefore, the technical solutions provided by the above embodiments of the present invention solve the technical problems that are usually present when using existing mapping methods for mapping, such as high computational complexity, insufficient system real-time performance, strong dependence on GNSS signals, and difficulty in multi-source data fusion.

[0078] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0079] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0080] According to an embodiment of the present invention, there is also provided a three-dimensional point cloud map generation device based on posture information for implementing the above-mentioned three-dimensional point cloud map generation method based on posture information. Figure 8 is a schematic diagram of a three-dimensional point cloud map generating device based on posture information according to an embodiment of the present invention. Figure 8 As shown, the device includes: a first acquisition unit 81, a determination unit 83, a second acquisition unit 85, a third acquisition unit 87 and a generation unit 89. The device for generating a three-dimensional point cloud map based on position and posture information is described in detail below.

[0081] The first acquisition unit 81 is used to acquire multi-source data collected by the UAV during flight, and perform time synchronization and fusion processing on the multi-source data to obtain target multi-source data, wherein the multi-source data refers to multiple sensor data collected during the flight of the UAV using multiple sensors installed in the UAV.

[0082] The determination unit 83 is used to determine, based on the target multi-source data, that an image frame in which a change in the posture of the UAV during flight is greater than a posture change threshold and / or an change in image features is greater than an image change threshold is a key frame, wherein the posture change represents a change in the first posture of the UAV in the image frame compared to the second posture of the UAV in the previous image frame, and the image feature change refers to a change in the first image feature of the image frame compared to the second influencing feature of the previous image frame.

[0083] The second acquisition unit 85 is used to perform feature matching on multiple key frames to obtain matching relationships between the multiple key frames.

[0084] The third acquisition unit 87 is used to determine, according to the matching relationship, that a plurality of key frames showing the same scene from different viewing angles are a key frame group, and obtain a plurality of key frame groups.

[0085] The generating unit 89 is used to generate a three-dimensional point cloud map of the target area according to the multiple key frame groups, wherein the target area is the area monitored by the drone, and the three-dimensional point cloud map is used to display the three-dimensional spatial structure of the target area.

[0086] It should be noted here that the above-mentioned first acquisition unit 81, determination unit 83, second acquisition unit 85, third acquisition unit 87 and generation unit 89 correspond to steps S202 to S210 in the above-mentioned embodiments, and the five units are the same as the instances and application scenarios implemented by the corresponding steps, but are not limited to the contents disclosed in the above-mentioned embodiments.

[0087] As can be seen from the above, in the scheme recorded in the above-mentioned embodiment of the present invention, a first acquisition unit can be used to acquire multi-source data collected by the drone during flight, and the multi-source data can be time-synchronized and fused to obtain target multi-source data, wherein the multi-source data refers to a variety of sensor data collected by multiple sensors installed in the drone during flight; then a determination unit is used to determine, based on the target multi-source data, an image frame in which the change in the drone's posture during flight is greater than a posture change threshold and / or the change in image features is greater than an image change threshold as a key frame, wherein the posture change represents the change in the first posture of the drone in the image frame compared to the second posture of the drone in the previous image frame, and the image feature change refers to the change in the first image feature of the image frame compared to the second influencing feature of the previous image frame; then a second acquisition unit is used to acquire multiple The three-dimensional point cloud map is a kind of map of the image processing method of the present invention, and the three-dimensional point cloud map is a kind of map of the image processing method of the present invention. The three-dimensional point cloud map is a kind of map of the image processing method of the present invention. The three-dimensional point cloud map is a kind of map of the image processing method of the present invention. The three-dimensional point cloud map is a kind of map of the image processing method of the present invention. The three-dimensional point cloud map is a kind of map of the image processing method of the present invention. The three-dimensional point cloud map is a kind of map of the image processing method of the present invention. The three-dimensional point cloud map is a kind of map of the image processing method of the present invention. The three-dimensional point cloud map is a kind of map of the image processing method of the present invention. The three-dimensional point cloud map is a kind of map of the image processing method of the present invention.

[0088] Therefore, the technical solutions provided by the above embodiments of the present invention solve the technical problems that are usually present when using existing mapping methods for mapping, such as high computational complexity, insufficient system real-time performance, strong dependence on GNSS signals, and difficulty in multi-source data fusion.

[0089] Optionally, the first acquisition unit includes: a first acquisition module, used to acquire multi-source data collected by the UAV using multiple sensors during flight, wherein the multi-source data includes at least: visible light image data, inertial measurement unit IMU data, and global navigation satellite system GNSS positioning data; a first determination module, used to determine a first timestamp for the UAV to collect visible light image data, a second timestamp for collecting IMU data, and a third timestamp for collecting GNSS positioning data; a correction module, used to correct the first timestamp based on the second timestamp and the third timestamp through interpolation or synchronization signal to achieve time synchronization processing of the multi-source data; a second acquisition module, used to fuse the multi-source data after completing the time synchronization processing of the multi-source data to obtain target multi-source data.

[0090] Optionally, the second acquisition module includes: a preprocessing submodule, which is used to complete the preprocessing of multi-source data by pre-integrating the IMU data, performing elevation filtering on the GNSS positioning data, and extracting and tracking features on the visible light image data; a first acquisition submodule, which is used to obtain the pre-integration result of the pre-integration processing of the IMU data; a second acquisition submodule, which is used to process the visible light image data using the structured motion algorithm SFM to obtain the SFM result; a determination submodule, which is used to determine the posture information of the UAV according to the IMU data and the GNSS positioning data after aligning the pre-integration result with the SFM result, wherein the IMU data is used to determine the attitude and speed of the UAV, and the GNSS positioning data is used to determine the position information of the UAV; a third acquisition submodule, which is used to match and fuse the posture information with the image frames in the visible light image data to obtain the target multi-source data.

[0091] Optionally, the determination unit includes: a second determination module, used to determine the posture corresponding to each image frame according to the posture information of the drone, wherein the posture information is information determined according to the IMU data and the GNSS positioning data in the multi-source data; a third determination module, used to determine the posture change amount of the drone during the flight according to the multi-source data using the first formula, wherein the first formula is: ΔP i =‖P i -P i-1 ‖, i represents the number of the image frame, P i represents the first pose of the i-th image frame, P i-1 represents the second pose of the i-1th frame, ΔP irepresents the change in posture of the i-th image frame compared to the i-1-th image frame, ‖·‖ represents the Euclidean distance, and the i-1-th image frame is the previous image frame of the i-th image frame; the third acquisition module is used to compare the change in posture with the posture change threshold to obtain a first comparison result; the fourth acquisition module is used to extract features from each image frame to obtain multiple first feature points of each image frame; the fourth determination module is used to determine the change in image features of the drone during flight using the second formula based on multi-source data, wherein the second formula is: ΔF i =∑ j ‖f i,j -f i-1,j ‖, j represents the number of the first feature point in any image frame, f i,j represents the jth first feature point in the i-th image frame, f i-1,j represents the jth first feature point in the i-1th image frame, ΔF i Represents the change in image features between the i-th image frame and the i-1-th image frame; a fifth acquisition module, used to compare the image feature change with the image change threshold to obtain a second comparison result; a fifth determination module, used to determine that the image frame is a key frame when the first comparison result indicates that the posture change is greater than the posture change threshold, and / or the second comparison result indicates that the image feature change is greater than the image change threshold.

[0092] Optionally, the second acquisition unit includes: a sixth acquisition module, which is used to extract features from each key frame to obtain multiple second feature points for each key frame; a first generation module, which is used to process each second feature point using a descriptor algorithm to generate a feature descriptor for each second feature point, wherein the feature descriptor is a vector of local image features of the feature point; a calculation module, which is used to calculate the descriptor distance between feature descriptors in any two adjacent key frames, wherein the descriptor distance is used to quantify the similarity between two feature descriptors; a sixth determination module, which is used to treat the second feature points corresponding to the two feature descriptors as a matching point group when the descriptor distance is less than a distance threshold; a seventh acquisition module, which is used to correct the second feature points in each group of matching point groups through geometric constraints to obtain a correction result; and a seventh determination module, which is used to determine that the key frames corresponding to the second feature points in the matching point group have a matching relationship when the correction result indicates that the second feature points in the matching point group meet the accuracy requirement.

[0093] Optionally, the generating unit includes: an eighth acquisition module, used to acquire the disparity between key frames having a matching relationship in each key frame group, wherein the disparity refers to the difference in pixel positions of the same scene imaged at different viewing angles; and a ninth acquisition module, used to calculate the depth value of the pixel point in the target sub-region according to the disparity of each key frame group using a third formula to obtain a depth map of the target sub-region, wherein the third formula is: z represents the depth value, f represents the focal length of the image acquisition device, d represents the parallax, B represents the baseline length between the key frames with matching relationship in the key frame group, the target sub-region refers to the sub-region displayed by each key frame group in the target area, the depth value represents the distance between the pixel point and the image acquisition device, and the depth map is an image constructed according to the depth value of each pixel point; the second generation module is used to restore the three-dimensional point cloud of each pixel point in the depth map corresponding to each target sub-region by using the triangulation method to generate an initial three-dimensional point cloud map of the target area; the tenth acquisition module is used to obtain the color information of each pixel point in each depth map; the eleventh acquisition module is used to fill the initial three-dimensional point cloud map according to the color information to obtain the three-dimensional point cloud map of the target area.

[0094] Optionally, the three-dimensional point cloud map generation device based on posture information further includes: a calculation unit, which is used to calculate the reprojection error of the second feature point in all matching point groups using a fourth formula after generating a three-dimensional point cloud map of the target area according to multiple key frame groups, wherein the fourth formula is: E reproj =∑ a ∑ b ‖Q a,b -π(R a X b +r a )‖ 2 , E reproj represents the reprojection error, a represents the number of the key frame, b represents the number of the second feature point, Q a,b represents the bth second feature point in the ath key frame, R a represents the rotation matrix of the a-th key frame, X b represents the real coordinates corresponding to the bth second feature point in the ath key frame, r arepresents the translation vector of the a-th frame key frame, π() represents the projection function; a fourth acquisition unit is used to iteratively optimize the three-dimensional point cloud map using the optimization factor with the goal of minimizing the reprojection error, until the iteration termination condition is reached, and obtain the first target three-dimensional point cloud map after the three-dimensional point cloud map is optimized, wherein the optimization factor at least includes: system state, prior factor, posture factor composed of posture information, GNSS observation factor, and visual factor, and the iteration termination condition at least includes: the number of iterations of iterative optimization of the three-dimensional point cloud map reaches a preset number of iterations, and the reprojection error is not greater than the error threshold; a fifth acquisition unit is used to convert the GNSS coordinate system corresponding to the first target three-dimensional point cloud map into an absolute coordinate system with actual physical meaning using the fifth formula, so as to perform scale recovery processing on the first target three-dimensional point cloud map to obtain a second target three-dimensional point cloud map, wherein the fifth formula is: map =λ·Y fusion , Y map represents the first target 3D point cloud map, Y fusion represents the second target 3D point cloud map, and λ represents the scale factor.

[0095] According to another aspect of an embodiment of the present invention, a three-dimensional point cloud map generation system based on posture information is also provided. The three-dimensional point cloud map generation system based on posture information uses any of the above-mentioned three-dimensional point cloud map generation methods based on posture information.

[0096] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, the computer-readable storage medium comprising a stored program, wherein the program executes any one of the above-mentioned methods for generating a three-dimensional point cloud map based on posture information.

[0097] Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the communication devices in a communication device group.

[0098] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: obtaining multi-source data collected by the drone during flight, and performing time synchronization and fusion processing on the multi-source data to obtain target multi-source data, wherein the multi-source data refers to multiple sensor data collected by multiple sensors installed in the drone during flight; determining, based on the target multi-source data, an image frame in which the change in the posture of the drone during flight is greater than a posture change threshold and / or the change in image features is greater than an image change threshold as a key frame, wherein the change in posture represents the change in the first posture of the drone in the image frame compared to the second posture of the drone in the previous image frame, and the change in image features refers to the change in the first image feature of the image frame compared to the second influencing feature of the previous image frame; performing feature matching on multiple key frames to obtain a matching relationship between the multiple key frames; determining, based on the matching relationship, multiple key frames that display the same scene from different perspectives as a key frame group to obtain multiple key frame groups; generating a three-dimensional point cloud map of the target area based on the multiple key frame groups, wherein the target area is an area monitored by the drone, and the three-dimensional point cloud map is used to display the three-dimensional spatial structure of the target area.

[0099] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: obtaining multi-source data collected by the drone using multiple sensors during flight, wherein the multi-source data includes at least: visible light image data, inertial measurement unit IMU data, and global navigation satellite system GNSS positioning data; determining a first timestamp for the drone to collect visible light image data, a second timestamp for collecting IMU data, and a third timestamp for collecting GNSS positioning data; based on the second timestamp and the third timestamp, correcting the first timestamp by interpolation or synchronization signal to achieve time synchronization processing of the multi-source data; after completing the time synchronization processing of the multi-source data, fusing the multi-source data to obtain target multi-source data.

[0100] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: pre-processing the multi-source data by pre-integrating the IMU data, performing elevation filtering on the GNSS positioning data, and performing feature extraction and tracking on the visible light image data; obtaining the pre-integration results of the pre-integration processing of the IMU data; processing the visible light image data using the structured motion algorithm SFM to obtain SFM results; after aligning the pre-integration results with the SFM results, determining the posture information of the UAV according to the IMU data and the GNSS positioning data, wherein the IMU data is used to determine the attitude and speed of the UAV, and the GNSS positioning data is used to determine the position information of the UAV; matching and fusing the posture information with the image frames in the visible light image data to obtain target multi-source data.

[0101] Optionally, in this embodiment, the computer-readable storage medium is configured to store a program code for executing the following steps: determining the posture corresponding to each image frame according to the posture information of the drone, wherein the posture information is information determined according to the IMU data and the GNSS positioning data in the multi-source data; determining the posture change amount of the drone during the flight according to the multi-source data using a first formula, wherein the first formula is: ΔP i =‖P i -P i-1 ‖, i represents the number of the image frame, P i represents the first pose of the i-th image frame, P i-1 represents the second pose of the i-1th frame, ΔP i represents the change in posture of the i-th image frame compared to the i-1-th image frame, ‖·‖ represents the Euclidean distance, and the i-1-th image frame is the previous image frame of the i-th image frame; the change in posture is compared with the posture change threshold to obtain a first comparison result; feature extraction is performed on each image frame to obtain multiple first feature points of each image frame; the image feature change of the drone during flight is determined using the second formula based on multi-source data, where the second formula is: ΔF i =∑ j ‖f i,j -f i-1,j ‖, j represents the number of the first feature point in any image frame, f i,j represents the jth first feature point in the i-th image frame, f i-1,j represents the jth first feature point in the i-1th image frame, ΔF i Represents the change in image features between the i-th image frame and the i-1-th image frame; compares the image feature change with the image change threshold to obtain a second comparison result; when the first comparison result indicates that the posture change is greater than the posture change threshold, and / or the second comparison result indicates that the image feature change is greater than the image change threshold, determines that the image frame is a key frame.

[0102] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: performing feature extraction on each key frame to obtain multiple second feature points for each key frame; processing each second feature point using a descriptor algorithm to generate a feature descriptor for each second feature point, wherein the feature descriptor is a vector of local image features of the feature point; calculating a descriptor distance between feature descriptors in any two adjacent key frames, wherein the descriptor distance is used to quantify the similarity between the two feature descriptors; when the descriptor distance is less than a distance threshold, treating the second feature points corresponding to the two feature descriptors as a matching point group; correcting the second feature points in each group of matching point groups through geometric constraints to obtain a correction result; when the correction result indicates that the second feature points in the matching point group meet the accuracy requirements, determining that there is a matching relationship between the key frames corresponding to the second feature points in the matching point group.

[0103] Optionally, in this embodiment, the computer-readable storage medium is configured to store a program code for executing the following steps: obtaining the disparity between key frames having a matching relationship in each key frame group, wherein the disparity refers to the difference in pixel positions of the same scene imaged at different viewing angles; and calculating the depth value of the pixel point in the target sub-region using a third formula according to the disparity of each key frame group to obtain a depth map of the target sub-region, wherein the third formula is: z represents the depth value, f represents the focal length of the image acquisition device, d represents the parallax, B represents the baseline length between the key frames with matching relationship in the key frame group, the target sub-region refers to the sub-region displayed by each key frame group in the target area, the depth value represents the distance between the pixel point and the image acquisition device, and the depth map is an image composed of the depth values ​​of each pixel point; the three-dimensional point cloud of each pixel point in the depth map corresponding to each target sub-region is restored by triangulation method to generate an initial three-dimensional point cloud map of the target area; the color information of each pixel point in each depth map is obtained; the initial three-dimensional point cloud map is filled with color according to the color information to obtain a three-dimensional point cloud map of the target area.

[0104] Optionally, in this embodiment, the computer-readable storage medium is configured to store a program code for executing the following steps: calculating the reprojection error of the second feature point in all matching point groups using the fourth formula, wherein the fourth formula is: E reproj =∑ a ∑ b ‖Q a,b -π(R a X b +r a )‖ 2 , E reproj represents the reprojection error, a represents the number of the key frame, b represents the number of the second feature point, Q a,brepresents the bth second feature point in the ath key frame, R a represents the rotation matrix of the a-th key frame, X b represents the real coordinates corresponding to the bth second feature point in the ath key frame, r a represents the translation vector of the a-th frame key frame, π() represents the projection function; with the goal of minimizing the reprojection error, the three-dimensional point cloud map is iteratively optimized using the optimization factor until the iteration termination condition is reached, and the first target three-dimensional point cloud map after the three-dimensional point cloud map is optimized is obtained, wherein the optimization factor at least includes: system state, prior factor, posture factor composed of posture information, GNSS observation factor, and visual factor, and the iteration termination condition at least includes: the number of iterations of iterative optimization of the three-dimensional point cloud map reaches the preset number of iterations, and the reprojection error is not greater than the error threshold; the GNSS coordinate system corresponding to the first target three-dimensional point cloud map is converted into an absolute coordinate system with actual physical meaning using the fifth formula, so as to perform scale recovery processing on the first target three-dimensional point cloud map to obtain a second target three-dimensional point cloud map, wherein the fifth formula is: map =λ·Y fusion , Y map Represents the first target 3D point cloud map, Y fusion represents the second target 3D point cloud map, and λ represents the scale factor.

[0105] According to another aspect of an embodiment of the present invention, a processor is further provided, and the processor is used to run a program, wherein when the program is run, any one of the above-mentioned three-dimensional point cloud map generation methods based on posture information is executed.

[0106] According to another aspect of an embodiment of the present invention, a computer program product is also provided, including computer instructions, which, when executed by a processor, execute any one of the above-mentioned methods for generating a three-dimensional point cloud map based on posture information.

[0107] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0108] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0109] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0110] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0111] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0112] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0113] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for generating a three-dimensional point cloud map based on pose information, characterized in that: include: Acquire multi-source data collected by the drone during flight, and perform time synchronization and fusion processing on the multi-source data to obtain target multi-source data, wherein the multi-source data refers to multiple sensor data collected by multiple sensors installed in the drone during flight of the drone; Determine, according to the target multi-source data, an image frame in which a posture change amount of the UAV during flight is greater than a posture change threshold and / or an image feature change amount is greater than an image change threshold as a key frame, wherein the posture change amount represents a change amount of the first posture of the UAV in the image frame compared to the second posture of the UAV in the previous image frame, and the image feature change amount refers to a change amount of the first image feature of the image frame compared to the second influencing feature of the previous image frame; Performing feature matching on the multiple key frames to obtain a matching relationship between the multiple key frames; Determine, according to the matching relationship, that a plurality of key frames showing the same scene from different viewing angles are a key frame group, thereby obtaining a plurality of key frame groups; A three-dimensional point cloud map of a target area is generated according to a plurality of the key frame groups, wherein the target area is an area monitored by the drone, and the three-dimensional point cloud map is used to display a three-dimensional spatial structure of the target area.

2. The method for generating a three-dimensional point cloud map based on posture information according to claim 1, characterized in that: Acquire multi-source data collected by the drone during flight, and perform time synchronization and fusion processing on the multi-source data to obtain target multi-source data, including: Acquire the multi-source data collected by the drone using the multiple sensors during flight, wherein the multi-source data at least includes: visible light image data, inertial measurement unit IMU data, and global navigation satellite system GNSS positioning data; Determine a first timestamp for the drone to collect the visible light image data, a second timestamp for collecting the IMU data, and a third timestamp for collecting the GNSS positioning data; Taking the second timestamp and the third timestamp as references, correcting the first timestamp by interpolation or synchronization signal to achieve time synchronization processing of the multi-source data; After completing the time synchronization processing on the multi-source data, the multi-source data is fused to obtain the target multi-source data.

3. The method for generating a three-dimensional point cloud map based on posture information according to claim 2, characterized in that: The multi-source data is subjected to fusion processing to obtain the target multi-source data, including: Preprocessing the multi-source data is completed by pre-integrating the IMU data, performing elevation filtering on the GNSS positioning data, and extracting and tracking features on the visible light image data; Obtaining a pre-integration result of pre-integration processing on the IMU data; Processing the visible light image data using a structure from motion algorithm (SFM) to obtain an SFM result; After aligning the pre-integration result with the SFM result, determining the posture information of the UAV according to the IMU data and the GNSS positioning data, wherein the IMU data is used to determine the attitude and speed of the UAV, and the GNSS positioning data is used to determine the position information of the UAV; The position information is matched and fused with the image frame in the visible light image data to obtain the target multi-source data.

4. The method for generating a three-dimensional point cloud map based on posture information according to claim 1, characterized in that: Determining, according to the target multi-source data, an image frame in which a posture change amount of the UAV during flight is greater than a posture change threshold and / or an image feature change amount is greater than an image change threshold as a key frame, comprising: Determine the posture corresponding to each of the image frames according to the posture information of the drone, wherein the posture information is information determined according to the IMU data and the GNSS positioning data in the multi-source data; The position change of the UAV during flight is determined using a first formula according to the multi-source data, wherein the first formula is: ΔP i =‖P i -P i-1 ‖, i represents the number of the image frame, P i represents the first posture of the image frame of the i-th frame, P i-1 represents the second posture of the image frame of the i-1th frame, ΔP i represents the position change of the i-th image frame compared to the i-1-th image frame, ‖·‖ represents the Euclidean distance, and the i-1-th image frame is the previous image frame of the i-th image frame; Compare the posture change amount with the posture change threshold to obtain a first comparison result; Performing feature extraction on each of the image frames to obtain a plurality of first feature points of each of the image frames; The image feature change amount of the UAV during flight is determined using a second formula based on the multi-source data, wherein the second formula is: ΔF i =∑ j ‖f i,j -f i-1,j ‖, j represents the label of the first feature point in any of the image frames, f i,j represents the jth first feature point in the i-th image frame, f i-1,j represents the jth first feature point in the i-1th image frame, ΔF i represents the change amount of the image feature of the i-th image frame compared with the i-1-th image frame; Comparing the image feature change amount with the image change threshold to obtain a second comparison result; When the first comparison result indicates that the posture change is greater than the posture change threshold, and / or the second comparison result indicates that the image feature change is greater than the image change threshold, the image frame is determined to be the key frame.

5. The method for generating a three-dimensional point cloud map based on posture information according to claim 1, characterized in that: Performing feature matching on the multiple key frames to obtain a matching relationship between the multiple key frames includes: Performing feature extraction on each of the key frames to obtain a plurality of second feature points of each of the key frames; Processing each of the second feature points using a descriptor algorithm to generate a feature descriptor for each of the second feature points, wherein the feature descriptor is a vector of a local image feature of the feature point; Calculating a descriptor distance between the feature descriptors in any two adjacent key frames, wherein the descriptor distance is used to quantify the similarity between the two feature descriptors; When the descriptor distance is less than a distance threshold, taking the second feature points respectively corresponding to the two feature descriptors as a matching point group; Correcting the second feature points in each group of matching points by geometric constraints to obtain a correction result; When the correction result indicates that the second feature points in the matching point group meet the accuracy requirement, it is determined that the key frames corresponding to the second feature points in the matching point group have the matching relationship.

6. The method for generating a three-dimensional point cloud map based on posture information according to claim 1, characterized in that: Generating a three-dimensional point cloud map of a target area according to a plurality of the key frame groups includes: Obtaining the disparity between the key frames having the matching relationship in each of the key frame groups, wherein the disparity refers to the difference in pixel positions of the same scene imaged at different viewing angles; The depth value of the pixel point in the target sub-region is calculated using a third formula according to the disparity of each key frame group to obtain a depth map of the target sub-region, wherein the third formula is: z represents the depth value, f represents the focal length of the image acquisition device, d represents the parallax, B represents the baseline length between the key frames having the matching relationship in the key frame group, the target sub-region refers to the sub-region displayed by each key frame group in the target region, the depth value represents the distance between the pixel point and the image acquisition device, and the depth map is an image formed according to the depth value of each pixel point; Using triangulation to restore the three-dimensional point cloud of each pixel point in the depth map corresponding to each target sub-area, to generate an initial three-dimensional point cloud map of the target area; Obtaining color information of each pixel in each depth map; The initial three-dimensional point cloud map is filled with color according to the color information to obtain the three-dimensional point cloud map of the target area.

7. The method for generating a three-dimensional point cloud map based on position information according to claim 5, characterized in that: After generating a three-dimensional point cloud map of the target area according to the plurality of key frame groups, the method further includes: The reprojection error of the second feature point in all the matching point groups is calculated using the fourth formula, wherein the fourth formula is: reproj =∑ a ∑ b ‖Q a,b -π(R a X b +r a )‖ 2 , E reproj represents the reprojection error, a represents the number of the key frame, b represents the number of the second feature point, Q a,b represents the bth second feature point in the ath key frame, R a represents the rotation matrix of the key frame of frame a, X b represents the real coordinates corresponding to the bth second feature point in the ath key frame, r a represents the translation vector of the key frame of the a-th frame, and π() represents the projection function; With the goal of minimizing the reprojection error, the three-dimensional point cloud map is iteratively optimized using an optimization factor until an iterative termination condition is reached, thereby obtaining a first target three-dimensional point cloud map after the three-dimensional point cloud map is optimized, wherein the optimization factor at least includes: a system state, a priori factors, a posture factor composed of posture information, a GNSS observation factor, and a visual factor, and the iterative termination condition at least includes: the number of iterations of iterative optimization of the three-dimensional point cloud map reaches a preset number of iterations, and the reprojection error is not greater than an error threshold; The GNSS coordinate system corresponding to the first target three-dimensional point cloud map is converted into an absolute coordinate system with actual physical meaning by using the fifth formula, so as to perform scale recovery processing on the first target three-dimensional point cloud map to obtain a second target three-dimensional point cloud map, wherein the fifth formula is: map =λ·Y fusion , Y map represents the first target three-dimensional point cloud map, Y fusion represents the second target three-dimensional point cloud map, and λ represents the scale factor.

8. A three-dimensional point cloud map generation device based on posture information, characterized in that: include: A first acquisition unit is used to acquire multi-source data collected by the UAV during flight, and perform time synchronization and fusion processing on the multi-source data to obtain target multi-source data, wherein the multi-source data refers to multiple sensor data collected by multiple sensors installed in the UAV during flight of the UAV; A determination unit, configured to determine, based on the target multi-source data, an image frame in which a change in the posture of the drone during flight is greater than a posture change threshold and / or an change in the image feature is greater than an image change threshold as a key frame, wherein the change in posture represents a change in the first posture of the drone in the image frame compared to the second posture of the drone in the previous image frame, and the change in image feature refers to a change in the first image feature of the image frame compared to the second influencing feature of the previous image frame; A second acquisition unit is used to perform feature matching on the multiple key frames to obtain a matching relationship between the multiple key frames; A third acquisition unit is used to determine, according to the matching relationship, that a plurality of key frames showing the same scene from different viewing angles are a key frame group, thereby obtaining a plurality of key frame groups; A generating unit is used to generate a three-dimensional point cloud map of a target area according to a plurality of the key frame groups, wherein the target area is an area monitored by the drone, and the three-dimensional point cloud map is used to display the three-dimensional spatial structure of the target area.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the three-dimensional point cloud map generation method based on posture information as described in any one of claims 1 to 7.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by the processor, the method for generating a three-dimensional point cloud map based on posture information as described in any one of claims 1 to 7 is performed.

Images