Aerial thermal infrared image positioning method, device and equipment based on view synthesis

Abstract

The present application relates to a method, apparatus, and device for positioning aerial thermal infrared images based on view synthesis. The method comprises: constructing a query image set, a reference map set, and a ground control point set; the query image set includes multiple query images; calculating the initial pose of the query image using prior information acquired by the device sensor, rendering and synthesizing a pre-constructed three-dimensional reference model based on the initial pose and rendering software to obtain a reference image and a depth image of the reference image; matching feature points of the query image and the reference image based on a deep learning algorithm to construct a depth-based 2D-3D correspondence relationship, and using an n-point perspective algorithm to solve the motion between the three-dimensional point set and the two-dimensional point set with a 2D-3D correspondence relationship in the query image to obtain the final pose. This method can improve the accuracy of thermal infrared image positioning.

Description

Technical Field

[0001] The present application relates to the field of image positioning technology, and in particular to a method, device and equipment for positioning aerial thermal infrared images based on view synthesis. Background Art

[0002] Global Navigation Satellite Systems (GNSS), such as the BeiDou Navigation Satellite System (BDS) and the Global Positioning System (GPS), utilize satellites, ground control systems, and terminal receivers to provide positioning services for both military and civilian applications. However, various conditions can lead to inaccuracies in these systems, with errors ranging from a few meters to tens or even hundreds of meters. Modern positioning technologies, characterized by their simple structure, low cost, rich information content, and strong anti-interference capabilities, offer promising prospects for both military and civilian applications, making them a promising alternative to complementing GNSS. However, most existing visual positioning algorithms are based on visible light imagery and require sufficient lighting conditions, making them difficult to use for military operations or for civilian applications that span weather, seasons, and day and night, such as nighttime rescue operations. Thermal imaging captures temperature information, making them more adaptable to various environments and offering a longer imaging range. Compared to visible light images, they are less affected by glare and weather conditions, enabling clear object perception even in darkness, fog, and smoke. Furthermore, thermal infrared cameras have become increasingly miniaturized in recent years, making them less cost-prohibitive. This makes it possible to use thermal imagery to assist in the six-degree-of-freedom positioning of drones or aerial vehicles.

[0003] However, currently available thermal infrared localization datasets either lack annotations or focus on relative visual localization in indoor scenes. This data scarcity has severely hindered the development of this field, resulting in low localization accuracy in thermal infrared images. Summary of the Invention

[0004] Based on this, it is necessary to provide an aerial thermal infrared image positioning method, device and equipment based on view synthesis that can improve the accuracy of thermal infrared image positioning in response to the above technical problems.

[0005] An aerial thermal infrared image positioning method based on view synthesis, the method comprising:

[0006] Constructing a query image set, a reference map set, and a ground control point set; the query image set includes multiple query images;

[0007] The initial pose of the query image is calculated using the prior information obtained by the device sensor. The pre-built 3D reference model is rendered and synthesized based on the initial pose and the rendering software to obtain the reference image and the depth image of the reference image.

[0008] According to the deep learning algorithm, the feature points of the query image and the reference image are matched to construct a depth-based 2D-3D correspondence relationship. The n-point perspective algorithm is used to solve the motion between the three-dimensional point set and the two-dimensional point set with 2D-3D correspondence in the query image to obtain the final pose.

[0009] In one embodiment, the process of constructing a query image set, a reference map set, and a ground control point set includes:

[0010] All flight paths are planned into a grid, and the DJI M300 flight control system is used to automatically build a reference map set;

[0011] The flight platform is manually operated to freely navigate in the scene. Two shooting strategies, continuous shooting with a 2-second interval and random manual shooting, are used to construct a query image set from images obtained at different times and under different weather conditions from the reference map set.

[0012] Eight object edges in the reference map scene are selected as ground control points to construct a ground control point set.

[0013] In one embodiment, calculating the initial pose of the query image using prior information obtained from a device sensor includes:

[0014] The initial pose of the query image is calculated using the prior information obtained from the device sensor:

[0015]

[0016] Among them, R represents the rotation matrix of the image orientation, t represents the position coordinate of the query image in three-dimensional space, 0 T Indicates a zero-filling operation on the secondary coordinates.

[0017] In one embodiment, the pre-built 3D reference model includes a visible model, a thermal model, a super-resolution thermal model, and a geometrically refined thermal model; and the process of building the 3D reference model includes:

[0018] Visible light and thermal imaging images are collected to perform super-resolution of thermal imaging images and fusion of visible light and thermal images to obtain visible models, thermal models, super-resolution thermal models and geometrically refined thermal models.

[0019] In one embodiment, performing feature point matching on a query image and a reference image according to a deep learning algorithm to construct a depth-based 2D-3D correspondence includes:

[0020] The depth of the matching point pr in the reference image is back-projected into the reference map to obtain the coordinates P3D of the corresponding 3D point in the world coordinate system;

[0021] The coordinate P3D is mapped to the matching point pq in the query, thus completing the 2D-3D correspondence.

[0022] In one embodiment, back-projecting the depth of the matching point in the reference image into the reference map to obtain the coordinates of the corresponding 3D point in the world coordinate system includes:

[0023] The depth of the matching point in the reference image is back-projected onto the reference map, and the coordinates of the corresponding 3D point in the world coordinate system are obtained as follows:

[0024]

[0025] Among them, T r is the reference pose above, is the homogeneous coordinate form of the three-dimensional coordinate point, is the homogeneous form of the pixel coordinates of the matching point in the reference image, d is the depth value corresponding to the point, and K is the camera intrinsic parameter matrix.

[0026] In one embodiment, the motion between a 3D point set and a 2D point set having a 2D-3D correspondence in a query image is solved using an n-point perspective algorithm to obtain a final pose, including:

[0027] The n-point perspective algorithm is used to solve the motion between the three-dimensional point set and the two-dimensional point set with 2D-3D correspondence in the query image, and the probability density function is obtained as follows:

[0028]

[0029] Where p(X|y) represents the probability density function of the 3D point cloud X projected onto the image plane given the camera pose y, and f i (y) represents the error between the projection position of the i-th feature point on the image plane and the coordinates of the corresponding 3D point cloud in the camera coordinate system, Represents the sum of squared errors of all feature points;

[0030] The probability density function is minimized to obtain the final pose.

[0031] An aerial thermal infrared image positioning device based on view synthesis, the device comprising:

[0032] A data set construction module is used to construct a query image set, a reference map set and a ground control point set; the query image set includes multiple query images;

[0033] The rendering and synthesis module is used to calculate the initial pose of the query image using the prior information obtained by the device sensor, and perform rendering synthesis on the pre-built 3D reference model based on the initial pose and the rendering software to obtain the reference image and the depth image of the reference image;

[0034] The feature point matching and pose solving module is used to match the feature points of the query image and the reference image based on the deep learning algorithm to construct a depth-based 2D-3D correspondence relationship. The n-point perspective algorithm is used to solve the motion between the three-dimensional point set and the two-dimensional point set with a 2D-3D correspondence relationship in the query image to obtain the final pose.

[0035] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0036] Constructing a query image set, a reference map set, and a ground control point set; the query image set includes multiple query images;

[0037] The initial pose of the query image is calculated using the prior information obtained by the device sensor. The pre-built 3D reference model is rendered and synthesized based on the initial pose and the rendering software to obtain the reference image and the depth image of the reference image.

[0038] According to the deep learning algorithm, the feature points of the query image and the reference image are matched to construct a depth-based 2D-3D correspondence relationship. The n-point perspective algorithm is used to solve the motion between the three-dimensional point set and the two-dimensional point set with 2D-3D correspondence in the query image to obtain the final pose.

[0039] The above-mentioned aerial thermal infrared image positioning method, device and equipment based on view synthesis, this application provides the accurate posture of the query image by constructing four three-dimensional reference models as the aerial thermal image positioning data set, and records information from multiple sensors. The data set covers different weather conditions to better simulate the real situation, and provides 6-DoF postures of visible images and all images, thereby improving the accuracy of aerial thermal infrared image positioning in subsequent image positioning, and rendering the reference view according to the initial posture and rendering software to generate a depth map of the reference image. The depth map contains the depth values ​​of all pixels in the reference image, which can be more convenient for establishing 2D-3D correspondences in subsequent pose solution, thereby improving positioning efficiency. Finally, the query image and the reference image are matched with feature points through a deep learning algorithm to construct a depth-based 2D-3D correspondence. The n-point perspective algorithm is used to solve the movement between the three-dimensional point set and the two-dimensional point set with a 2D-3D correspondence in the query image to obtain the final pose, which can achieve meter-level positioning accuracy, greatly improving the accuracy of aerial thermal infrared image positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 1 is a flow chart of a method for positioning aerial thermal infrared images based on view synthesis in one embodiment;

[0041] Figure 2 1 is a schematic diagram of a framework of an aerial thermal infrared image positioning device based on view synthesis in one embodiment;

[0042] Figure 3 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0044] In one embodiment, Figure 1 As shown, a method for positioning aerial thermal infrared images based on view synthesis is provided, comprising the following steps:

[0045] Step 102: construct a query image set, a reference map set, and a ground control point set; the query image set includes multiple query images.

[0046] This application planned all flight paths into a grid, automated using the DJI M300's flight control system to ensure full and uniform coverage of the survey area. A reference map set was constructed. A DJI H20T camera was then used to acquire all query images. Considering the effects of temperature and weather on thermal imaging, several sets of query test images were collected under different conditions. To enhance data diversity and practicality, the flight platform was manually operated to freely navigate the scene while acquiring query images. Two shooting strategies, continuous shooting with a 2-second interval and random manual shooting, were used to construct the query image set. Ground control points are often required in photogrammetry to ensure a more accurate geographical alignment of the model with the real world. Because infrared cameras cannot capture color and texture information, eight edges of objects in the scene were selected as ground control points, and their geographic coordinates were measured using a handheld RTK device to construct the ground control point set.

[0047] This is achieved by constructing a query image set, a reference map set, and a ground control point set

[0048] Step 104 , using the prior information obtained by the device sensor to calculate the initial pose of the query image, and performing rendering synthesis on the pre-built 3D reference model according to the initial pose and rendering software to obtain a reference image and a depth image of the reference image.

[0049] Different types of drones typically have various built-in sensors, such as inertial sensors (e.g., accelerometers, gyroscopes, gravimeters, compasses). Given an initial position and velocity, the current position and velocity are continuously updated by calculating and integrating information from these sensors. However, the error of inertial sensors increases over time. Therefore, the given angle can only be used as prior information to determine the initial pose for subsequent perspective rendering. Onboard GPS is typically unstable, and altitude errors are often large. Therefore, this application uses a barometer to obtain the initial altitude as the z value.

[0050] Various 3D reference models are constructed from the collected visible light and thermal imaging images. These models are divided into two types: visible light models and thermal models. Due to the low resolution of thermal images, the geometric accuracy of this type of 3D models is low. In order to improve this accuracy, this application experiments with two methods: thermal image super-resolution and visible light geometric model thermal image fusion, and obtains four 3D reference models: visible model, thermal model, super-resolution thermal model and geometric refinement thermal model. Modern 3D reconstruction technology is used to build a textured mesh model and align it with the real geographic world through built-in RTK measurement and GCPs.

[0051] Visible reference model: A visible reference model is reconstructed from the high-overlap and high-resolution visible aerial images collected above. This model serves as the baseline for our dataset and has a ground sampling distance (GSD) of 1 cm.

[0052] Thermal Reference Model: Thermal images are collected for 3D reference model construction. This method avoids cross-modal matching issues and provides a contrast with the visible model. It also provides aligned multimodal data for subsequent research. However, due to limitations in the acquisition equipment, the obtained thermal images have a low resolution. This results in a 3D model constructed under the same setup with a ground-sizer of only 10 cm. Therefore, this application improves the accuracy of the thermal model from two aspects:

[0053] Super-resolution thermal model: Before 3D reconstruction, image super-resolution technology is used to improve the spatial resolution of the thermal 3D map. To this end, a super-resolution thermal image network is used to upscale the original thermal image and then construct a super-resolution thermal model.

[0054] Geometric Refinement of Thermal Model: Another way to improve the spatial resolution of the thermal model is to directly adjust its geometry, as the geometry determines the spatial resolution. During the texture mapping phase, all models are geo-aligned using RTK and GCPs, and the geometry in the thermal model is replaced with the corresponding geometry extracted from the visible model, which is a high-precision, texture-free model.

[0055] The four 3D reference models described above serve as a dataset for aerial thermal image localization. They provide the accurate pose of the query image and capture information from multiple sensors. Furthermore, the dataset covers diverse weather conditions to better simulate real-world conditions. This is the first dataset enabling the study of 6-DOF localization in aerial thermal imagery.

[0056] Pre-built 3D reference models are used for position-oriented viewpoint synthesis according to different scene requirements. Various off-the-shelf renderers can be used to render images from 3D meshes, and the open-source 3D rendering software Blender offers the advantage of an extensible API with high rendering quality. Therefore, based on the initial pose obtained above, this application uses Blender to render a reference view and generate a depth map of the reference image. This depth map contains the depth values ​​of all pixels in the reference image, which is used to establish 2D-3D correspondences in the subsequent pose solution.

[0057] In step 106, feature point matching is performed on the query image and the reference image according to a deep learning algorithm to construct a depth-based 2D-3D correspondence relationship, and the motion between the three-dimensional point set and the two-dimensional point set with a 2D-3D correspondence relationship in the query image is solved using an n-point perspective algorithm to obtain the final pose.

[0058] Given feature point matches between a query image and a synthesized reference image, a 2D-3D correspondence between the query and reference images can be established. First, the matching point pr in the rendered image is depth-backprojected onto the reference map to obtain the corresponding 3D point's coordinates P3D in the world coordinate system. P3D corresponds to the matching point pq in the query, thus completing the 2D-3D correspondence. The PnP algorithm is then used to solve the motion between the corresponding 3D and 2D point sets, yielding the final pose.

[0059] The above-mentioned aerial thermal infrared image positioning method based on view synthesis, this application provides the accurate posture of the query image by constructing four three-dimensional reference models as the aerial thermal image positioning data set, and records information from multiple sensors. The data set covers different weather conditions to better simulate the real situation, and provides 6-DoF postures of visible images and all images, thereby improving the accuracy of aerial thermal infrared image positioning in subsequent image positioning, and rendering the reference view according to the initial posture and rendering software to generate a depth map of the reference image. The depth map contains the depth values ​​of all pixels in the reference image, which can be more convenient for establishing 2D-3D correspondences in subsequent pose solution, thereby improving positioning efficiency. Finally, the query image and the reference image are matched with feature points through a deep learning algorithm to construct a depth-based 2D-3D correspondence. The n-point perspective algorithm is used to solve the movement between the three-dimensional point set and the two-dimensional point set with a 2D-3D correspondence in the query image to obtain the final pose, which can achieve meter-level positioning accuracy, greatly improving the accuracy of aerial thermal infrared image positioning.

[0060] In one embodiment, the process of constructing a query image set, a reference map set, and a ground control point set includes:

[0061] All flight paths are planned into a grid, and the DJI M300 flight control system is used to automatically build a reference map set;

[0062] The flight platform is manually operated to freely navigate in the scene. Two shooting strategies, continuous shooting with a 2-second interval and random manual shooting, are used to construct a query image set from images obtained at different times and under different weather conditions from a reference map set.

[0063] Eight object edges in the reference map scene are selected as ground control points to construct a ground control point set.

[0064] In one embodiment, calculating the initial pose of the query image using prior information obtained from a device sensor includes:

[0065] The initial pose of the query image is calculated using the prior information obtained from the device sensor:

[0066]

[0067] Among them, R represents the rotation matrix of the image orientation, t represents the position coordinate of the query image in three-dimensional space, 0 T Indicates a zero-filling operation on the secondary coordinates.

[0068] In one embodiment, the pre-built 3D reference model includes a visible model, a thermal model, a super-resolution thermal model, and a geometrically refined thermal model; and the process of building the 3D reference model includes:

[0069] Visible light and thermal imaging images are collected to perform super-resolution of thermal imaging images and fusion of visible light and thermal images to obtain visible models, thermal models, super-resolution thermal models and geometrically refined thermal models.

[0070] In one embodiment, performing feature point matching on a query image and a reference image according to a deep learning algorithm to construct a depth-based 2D-3D correspondence includes:

[0071] The depth of the matching point pr in the reference image is back-projected into the reference map to obtain the coordinates P3D of the corresponding 3D point in the world coordinate system;

[0072] The coordinate P3D is mapped to the matching point pq in the query, thus completing the 2D-3D correspondence.

[0073] In one embodiment, back-projecting the depth of the matching point in the reference image into the reference map to obtain the coordinates of the corresponding 3D point in the world coordinate system includes:

[0074] The depth of the matching point in the reference image is back-projected onto the reference map, and the coordinates of the corresponding 3D point in the world coordinate system are obtained as follows:

[0075]

[0076] Among them, T r is the reference pose above, is the homogeneous coordinate form of the three-dimensional coordinate point, is the homogeneous form of the pixel coordinates of the matching point in the reference image, d is the depth value corresponding to the point, and K is the camera intrinsic parameter matrix.

[0077] In one embodiment, the motion between a 3D point set and a 2D point set having a 2D-3D correspondence in a query image is solved using an n-point perspective algorithm to obtain a final pose, including:

[0078] The n-point perspective algorithm is used to solve the motion between the three-dimensional point set and the two-dimensional point set with 2D-3D correspondence in the query image, and the probability density function is obtained as follows:

[0079]

[0080] Where p(X|y) represents the probability density function of the 3D point cloud X projected onto the image plane given the camera pose y, and f i (y) represents the error between the projection position of the i-th feature point on the image plane and the coordinates of the corresponding 3D point cloud in the camera coordinate system, Represents the sum of squared errors of all feature points;

[0081] The probability density function is minimized to obtain the final pose.

[0082] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0083] In one embodiment, Figure 2 As shown, an aerial thermal infrared image positioning device based on view synthesis is provided, comprising: a data set construction module 202, a rendering synthesis module 204, and a feature point matching and pose solving module 206, wherein:

[0084] The data set construction module 202 is used to construct a query image set, a reference map set and a ground control point set; the query image set includes multiple query images;

[0085] A rendering and synthesis module 204 is configured to calculate an initial pose of the query image using prior information acquired by the device sensor, and perform rendering and synthesis on a pre-built 3D reference model based on the initial pose and the rendering software to obtain a reference image and a depth image of the reference image;

[0086] The feature point matching and pose solving module 206 is used to perform feature point matching on the query image and the reference image according to the deep learning algorithm to construct a depth-based 2D-3D correspondence relationship, and use the n-point perspective algorithm to solve the movement between the three-dimensional point set and the two-dimensional point set with a 2D-3D correspondence relationship in the query image to obtain the final pose.

[0087] In one embodiment, the dataset construction module 202 is further configured to construct the query image set, the reference map set, and the ground control point set, including:

[0088] All flight paths are planned into a grid, and the DJI M300 flight control system is used to automatically build a reference map set;

[0089] The flight platform is manually operated to freely navigate in the scene. Two shooting strategies, continuous shooting with a 2-second interval and random manual shooting, are used to construct a query image set from images obtained at different times and under different weather conditions from the reference map set.

[0090] Eight object edges in the reference map scene are selected as ground control points to construct a ground control point set.

[0091] In one embodiment, the rendering and synthesis module 204 is further configured to calculate an initial pose of the query image using prior information acquired by a device sensor, including:

[0092] The initial pose of the query image is calculated using the prior information obtained from the device sensor:

[0093]

[0094] Among them, R represents the rotation matrix of the image orientation, t represents the position coordinate of the query image in three-dimensional space, 0 T Indicates a zero-filling operation on the secondary coordinates.

[0095] In one embodiment, the feature point matching and pose solving module 206 is further configured to perform feature point matching on the query image and the reference image according to a deep learning algorithm to construct a depth-based 2D-3D correspondence relationship, including:

[0096] The depth of the matching point pr in the reference image is back-projected into the reference map to obtain the coordinates P3D of the corresponding 3D point in the world coordinate system;

[0097] The coordinate P3D is mapped to the matching point pq in the query, thus completing the 2D-3D correspondence.

[0098] In one embodiment, the feature point matching and pose solving module 206 is further configured to reversely project the depth of the matching points in the reference image into the reference map to obtain the coordinates of the corresponding 3D points in the world coordinate system, including:

[0099] The depth of the matching point in the reference image is back-projected onto the reference map, and the coordinates of the corresponding 3D point in the world coordinate system are obtained as follows:

[0100]

[0101] Among them, T r is the reference pose above, is the homogeneous coordinate form of the three-dimensional coordinate point, is the homogeneous form of the pixel coordinates of the matching point in the reference image, d is the depth value corresponding to the point, and K is the camera intrinsic parameter matrix.

[0102] In one embodiment, the feature point matching and pose solving module 206 is further configured to solve the motion between the three-dimensional point set and the two-dimensional point set having a 2D-3D correspondence relationship in the query image using an n-point perspective algorithm to obtain a final pose, including:

[0103] The n-point perspective algorithm is used to solve the motion between the three-dimensional point set and the two-dimensional point set with 2D-3D correspondence in the query image, and the probability density function is obtained as follows:

[0104]

[0105] Where p(X|y) represents the probability density function of the 3D point cloud X projected onto the image plane given the camera pose y, and f i (y) represents the error between the projection position of the i-th feature point on the image plane and the coordinates of the corresponding 3D point cloud in the camera coordinate system, Represents the sum of squared errors of all feature points;

[0106] The probability density function is minimized to obtain the final pose.

[0107] Regarding the specific limitations of the aerial thermal infrared image positioning device based on view synthesis, please refer to the limitations of the aerial thermal infrared image positioning method based on view synthesis above, and will not be repeated here. The various modules in the above-mentioned aerial thermal infrared image positioning device based on view synthesis can be implemented in whole or in part through software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of the above-mentioned modules.

[0108] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 3As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for positioning aerial thermal infrared images based on view synthesis is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0109] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0110] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0111] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for positioning aerial thermal infrared images based on view synthesis, characterized in that: The method comprises: Constructing a query image set, a reference map set, and a ground control point set; the query image set includes a plurality of query images; Calculating an initial pose of the query image using prior information acquired by a device sensor, and performing rendering synthesis on a pre-built three-dimensional reference model based on the initial pose and rendering software to obtain a reference image and a depth image of the reference image; Performing feature point matching on the query image and the reference image according to a deep learning algorithm to construct a depth-based 2D-3D correspondence relationship, solving the motion between a three-dimensional point set and a two-dimensional point set in the query image having a 2D-3D correspondence relationship using an n-point perspective algorithm to obtain a final pose; performing feature point matching on the query image and the reference image according to a deep learning algorithm to construct a depth-based 2D-3D correspondence relationship, including: Back-project the depth of the matching point pr in the reference image into the reference map to obtain the coordinates P3D of the corresponding 3D point in the world coordinate system; Map the coordinate P3D to the matching point pq in the query, thus completing the 2D-3D correspondence; Back-projecting the depth of the matching point in the reference image onto the reference map to obtain the coordinates of the corresponding 3D point in the world coordinate system, including: The depth of the matching point in the reference image is back-projected onto the reference map to obtain the coordinates of the corresponding 3D point in the world coordinate system: Among them, T r is the reference pose, is the homogeneous coordinate form of the three-dimensional coordinate point, is the homogeneous form of the pixel coordinates of the matching point in the reference image, d is the depth value corresponding to the point, and K is the camera intrinsic parameter matrix.

2. The method according to claim 1, characterized in that The process of constructing the query image set, reference map set, and ground control point set includes: All flight paths are planned into a grid, and the DJI M300 flight control system is used to automatically build a reference map set; The flight platform is manually operated to freely navigate in the scene. Two shooting strategies, continuous shooting with a 2-second interval and random manual shooting, are used to construct a query image set from images obtained at different times and under different weather conditions from the reference map set. Eight object edges in the reference map scene are selected as ground control points to construct a ground control point set.

3. The method according to claim 1, characterized in that Calculating the initial pose of the query image using prior information obtained from the device sensor, including: The initial pose of the query image is calculated using the prior information obtained by the device sensor: Among them, R represents the rotation matrix of the image orientation, t represents the position coordinate of the query image in three-dimensional space, 0 T Indicates a zero-filling operation on the secondary coordinates.

4. The method according to any one of claims 1 to 3, characterized in that The pre-built three-dimensional reference model includes a visible model, a thermal model, a super-resolution thermal model, and a geometrically refined thermal model; and the process of building the three-dimensional reference model includes: Visible light and thermal imaging images are collected to perform super-resolution of thermal imaging images and fusion of visible light and thermal images to obtain visible models, thermal models, super-resolution thermal models and geometrically refined thermal models.

5. The method according to claim 1, characterized in that The n-point perspective algorithm is used to solve the motion between the three-dimensional point set and the two-dimensional point set with a 2D-3D correspondence relationship in the query image to obtain the final pose, including: The n-point perspective algorithm is used to solve the motion between the three-dimensional point set and the two-dimensional point set with 2D-3D correspondence in the query image, and the probability density function is obtained as follows: Where p(X|y) represents the probability density function of the 3D point cloud X projected onto the image plane given the camera pose y, and f i (y) represents the error between the projection position of the i-th feature point on the image plane and the coordinates of the corresponding 3D point cloud in the camera coordinate system, ∑ i＝1 N||f i (y)||2 represents the sum of squared errors of all feature points; The probability density function is minimized to obtain the final pose.

6. An aerial thermal infrared image positioning device based on view synthesis, characterized in that: The device comprises: A data set construction module is used to construct a query image set, a reference map set and a ground control point set; the query image set includes multiple query images; a rendering and synthesis module, configured to calculate an initial pose of the query image using prior information acquired by a device sensor, and perform rendering and synthesis on a pre-built 3D reference model based on the initial pose and rendering software to obtain a reference image and a depth image of the reference image; A feature point matching and pose solving module is used to perform feature point matching on the query image and the reference image according to a deep learning algorithm to construct a depth-based 2D-3D correspondence relationship, including: Back-project the depth of the matching point pr in the reference image into the reference map to obtain the coordinates P3D of the corresponding 3D point in the world coordinate system; Map the coordinate P3D to the matching point pq in the query, thus completing the 2D-3D correspondence; Back-projecting the depth of the matching point in the reference image to the reference map to obtain the coordinates of the corresponding 3D point in the world coordinate system, including: back-projecting the depth of the matching point in the reference image to the reference map to obtain the coordinates of the corresponding 3D point in the world coordinate system as Among them, T r is the reference pose, is the homogeneous coordinate form of the three-dimensional coordinate point, is the homogeneous form of the pixel coordinates of the matching point in the reference image, d is the depth value corresponding to the point, and K is the camera intrinsic parameter matrix; the n-point perspective algorithm is used to solve the motion between the three-dimensional point set and the two-dimensional point set with a 2D-3D correspondence in the query image to obtain the final pose.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

Images