A method and system for constructing a true three-dimensional temperature field of buildings from airborne thermal infrared images

Abstract

The present invention belongs to the technical field of thermal infrared photogrammetry, and relates to a method and system for constructing a true three-dimensional temperature field of airborne thermal infrared images of buildings, including: obtaining a visible light image sequence and a thermal infrared image sequence through oblique aerial photography; respectively generating a visible light image point cloud and a thermal infrared image point cloud from the visible light image sequence and the thermal infrared image sequence, and completing the rough registration of the visible light image point cloud and the thermal infrared image point cloud; based on the rough registration result, realizing the fine registration of the visible light image point cloud and the thermal infrared image point cloud through a conjugate four-plane matching set; determining a candidate thermal infrared texture set for each three-dimensional ground point, and finely adjusting the external orientation elements through global image pose optimization, so as to realize the three-dimensional construction of a large-range temperature field. The present invention reduces the tiny geometric registration error and radiation temperature difference between overlapping images, and can realize the three-dimensional construction of a high-precision building temperature field.

Description

Technical Field

[0001] The present invention relates to a method and system for constructing a true three-dimensional temperature field of buildings from airborne thermal infrared images, and belongs to the technical field of thermal infrared photogrammetry. Background Art

[0002] Benefiting from the rapid development of uncooled microbolometers, uncooled thermal cameras are widely used in building airtightness detection, forest fire protection, fluid flow velocity measurement, night vision, precision agriculture and other fields. Considering that the energy consumption of large-scale building areas accounts for one-third of the global total energy consumption, reducing the energy consumption of buildings in all regions has become the top priority of energy conservation and emission reduction in various countries. In order to improve the effective energy utilization rate of buildings, reducing heat loss caused by aging and damage of building components has become the key. Thermal infrared imaging cameras can capture and visualize the thermal anomalies of different parts of buildings, and accurately locate defects such as building damage and cracks. Due to the limited spatial resolution and narrow field of view of thermal infrared cameras, each thermal infrared image can only cover a part of a large-scale observation scene, and directly applying two-dimensional thermal infrared images will lose the detailed features of objects in three-dimensional space, making it difficult to establish the connection between the two-dimensional projection of objects and the true three-dimensional model. Therefore, the current research in this field mainly focuses on fusing thermal infrared image sequences with large-scale three-dimensional models to construct three-dimensional temperature fields. However, most of the existing fusion methods for thermal infrared image sequences and three-dimensional models are mainly applied to the airtightness analysis of small-scale building areas, such as the detection of thermal cracks in single buildings, and the thermal infrared three-dimensional modeling based on large-scale building group scenes is still lacking. Summary of the Invention

[0003] Aiming at the above problems, the purpose of the present invention is to provide a high-precision true three-dimensional temperature field reconstruction method and system that can adapt to large-scale building groups.

[0004] To achieve the above purpose, the present invention proposes the following technical solutions: A method for reconstructing a true three-dimensional temperature field of buildings, comprising: obtaining a visible light image sequence and a thermal infrared image sequence through oblique aerial photography; respectively generating a visible light image point cloud and a thermal infrared image point cloud from the visible light image sequence and the thermal infrared image sequence, and completing the rough registration of the visible light image point cloud and the thermal infrared image point cloud; based on the rough registration result, achieving the fine registration of the visible light image point cloud and the thermal infrared image point cloud through a conjugate four-plane matching set; determining the candidate thermal infrared texture set of each three-dimensional ground point, and fine-tuning the external orientation elements through global image pose optimization, so as to achieve the three-dimensional reconstruction of a large-scale temperature field.

[0005] Further, the visible light image point cloud and the thermal infrared image point cloud are used to perform three-dimensional reconstruction on the visible light image sequence and the thermal infrared image sequence through the Structure from Motion (SfM) technology, respectively generating the visible light image point cloud and the thermal infrared image point cloud, and the rough registration of the visible light image point cloud and the thermal infrared image point cloud is realized by using the auxiliary positioning data.

[0006] Further, the method for fine registration is as follows: Resample the visible light image point cloud and the thermal infrared image point cloud by using the octree-based point cloud compression method to realize the voxelization of the visible light image point cloud and the thermal infrared image point cloud; Calculate the significant features of each voxel of the visible light image point cloud and the thermal infrared image point cloud, and classify the voxels into planar regions and non-planar regions. Determine the set of voxels belonging to the planar region through the significant features, and merge the voxels within adjacent planar regions, so as to realize the extraction of planar patches of the visible light image point cloud and the thermal infrared image point cloud; According to the result of the planar patch extraction, with the rough registration result as the prior condition, realize the fine registration of the visible light image point cloud and the thermal infrared image point cloud through the conjugate four-plane set.

[0007] Further, the calculation method for the significant feature is as follows: Calculate the covariance matrix according to the discrete point distribution within the voxel of the visible light image point cloud and the thermal infrared image point cloud; Calculate the eigenvalues and eigenvectors of the covariance matrix through principal component analysis; Calculate the significant feature of the voxel through the eigenvalues and eigenvectors.

[0008] Further, the method for voxel classification is as follows: Compare the magnitudes of the three significant features. If the second significant feature is greater than the first significant feature and greater than the third significant feature, the voxel is a planar region, otherwise it is a non-planar region.

[0009] Further, the method for the conjugate four-plane matching set is as follows: Taking the visible light image point cloud as the reference model, find four planes {P a , P b , P c , P d} with a large spatial distance and a large difference in normal vector angles as the reference four planes; Taking the thermal infrared image point cloud as the target model, find four planes {P' a , P' b , P' c , P' d} with a large spatial distance and a large difference in normal vector angles as the target four planes; Taking the invariance of the angle ratio between the normal vectors and the intersection vectors of the four planes of the reference model and the target model as the matching condition, use the Random Sample Consensus (RANSAC) algorithm to find the inlier matching set that satisfies the most matching conditions, and use the least squares method to realize the accurate calculation of the affine transformation parameters, so as to realize the fine registration of the visible light image point cloud and the thermal infrared image point cloud.

[0010] Furthermore, the method for determining the candidate thermal infrared texture set is as follows: removing the occluded and incorrect textures through the distance buffer visibility test method to determine the candidate thermal infrared texture set for each three-dimensional ground point; screening the diffuse reflection ground points as the targets for global image pose optimization through the maximum temperature difference of the overlapping images; taking each diffuse reflection three-dimensional ground point as the basic unit, aiming to minimize the temperature difference between the overlapping images, using the least squares method to optimize the pose of the thermal infrared image sequence, calculating the equal-weighted temperature average value of the overlapping images as the thermal infrared texture mapping result, and completing the reconstruction of the true three-dimensional temperature field model.

[0011] Furthermore, the method for optimizing the pose of the thermal infrared image sequence is as follows: calculating the average temperature value of all candidate thermal infrared texture sets and using it as the reference temperature value; calculating the temperature difference between the temperature value of each overlapping image and the reference temperature value; taking the sum of the temperature differences as the global objective function; expressing the global objective function in the least squares form, and iteratively optimizing the exterior orientation elements of the overlapping images until the average temperature residual of all overlapping images is less than the threshold or the number of iterations exceeds the threshold; calculating the equal-weighted temperature average value of all overlapping images as the thermal infrared texture mapping result to generate the true three-dimensional temperature field of the building.

[0012] Furthermore, the method for iteratively optimizing the exterior orientation elements of the overlapping images is as follows: Let x represent the target optimization vector, the parameters to be optimized include the exterior orientation elements of all overlapping images, accumulate the difference between each overlapping image and the average temperature value of the overlapping images, and update through the following formula:

[0013] x k+1 = x k + Δx

[0014] where k is the number of iterations, Δx is the update vector, and the iteration continues until the average temperature residual of all overlapping images is less than the threshold or the number of iterations exceeds the threshold.

[0015] The present invention also discloses a system for reconstructing the true three-dimensional temperature field of a building, including: an image acquisition module for acquiring a visible light image sequence and a thermal infrared image sequence through an oblique aerial photography system; a rough registration module for generating a visible light image point cloud and a thermal infrared image point cloud based on the visible light image sequence and the thermal infrared image sequence respectively, and completing the rough registration of the visible light image point cloud and the thermal infrared image point cloud; a fine registration module for performing fine registration of the visible light image point cloud and the thermal infrared image point cloud based on the rough registration result through a conjugate four-plane matching set; a texture mapping module for determining the candidate thermal infrared texture set for each three-dimensional ground point, and finely adjusting the exterior orientation elements through global image pose optimization, so as to achieve large-scale three-dimensional reconstruction of the temperature field.

[0016] Due to the above technical solutions adopted by the present invention, it has the following advantages:

[0017] 1. The existing Structure from Motion (SfM) technology based on auxiliary positioning data (Global Navigation Satellite System / Inertial Measurement Unit, GNSS / IMU) cannot generate heterologous image point clouds with high-precision registration relationships. The present invention utilizes the principle that the ratio of the included angles between the normal vector and the intersection vector of the conjugate four-plane matching set remains unchanged, realizing the high-precision registration of visible light image point clouds and thermal infrared image point clouds, laying a solid foundation for thermal infrared texture mapping and the reconstruction of the true three-dimensional temperature field model of buildings.

[0018] 2. After realizing the registration of the two-dimensional thermal infrared image sequence to the three-dimensional point cloud model, there are still small-scale geometric mismatches, a certain degree of radiance differences, and a large amount of redundant information between the overlapping images. The present invention assumes that the temperature values of the same diffuse object remain unchanged on the overlapping images, and proposes a method for iteratively optimizing the exterior orientation elements of candidate images during the texture mapping process, realizing the high-precision reconstruction of the true three-dimensional temperature field of buildings.

[0019] 3. The existing methods mainly conduct research on single buildings or small-scale areas. The method provided by the present invention can quickly and roughly detect potential thermal anomalies in large-scale building groups, providing effective support for the screening of thermal anomalies in large-scale scenarios. It should be noted that in practical applications, in order to improve the accuracy of thermal anomaly detection results, it is also necessary to conduct on-site inspections of potential thermal anomaly areas to exclude misjudgments caused by differences in the emissivity of different building materials.

[0020] 4. The thermal infrared image sequence based on an oblique airborne platform can not only provide the roof information of the building group, but also describe the facade information of the building group, thus realizing the reconstruction of the true three-dimensional temperature field covering the roof and facade of the building group. This achievement can not only be used for the detection of thermal anomalies in building groups, but also assist in the solar energy siting of buildings. Through the analysis of the true three-dimensional temperature field, it is possible to conveniently find the positions on the building that receive the most sunlight, and then realize the intelligent siting of solar energy. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flowchart of the method for reconstructing the true three-dimensional temperature field of buildings in an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0022] To enable those skilled in the art to better understand the technical direction of the present invention, the present invention is described in detail through specific embodiments. However, it should be understood that the provision of specific embodiments is only for better understanding of the present invention, and they should not be construed as limitations on the present invention. In the description of the present invention, it should be understood that the terms used are only for the purpose of description and cannot be construed as indicating or implying relative importance.

[0023] The present invention relates to a method for reconstructing the true three-dimensional temperature field of a building based on an oblique airborne thermal infrared image sequence, which mainly includes 4 visible light cameras and 4 uncooled thermal infrared cameras. The visible light cameras and the thermal infrared cameras simultaneously acquire images at the same frame rate (5 Hz). The camera system is carried on a fixed-wing unmanned aerial vehicle to acquire image data at an altitude of 150 meters. Considering that the spatial resolution of visible light images is higher and the texture information is richer, the present invention uses the visible light image point cloud as a three-dimensional reference model and uses the thermal infrared image sequence to provide temperature information. Therefore, in order to fuse the thermal infrared image sequence with the visible light image point cloud, it is first necessary to realize the registration of the two-dimensional thermal infrared image sequence to the three-dimensional visible light image point cloud model; after registration, there are still certain degrees of radiance differences and information redundancy between the overlapping images. The present invention adopts a method of global image pose optimization to reduce the stitching effect in the texture mapping result and realize the reconstruction of a high-precision true three-dimensional temperature field model.

[0024] Embodiment 1

[0025] This embodiment discloses a method for reconstructing the true three-dimensional temperature field of a building based on an oblique airborne thermal infrared image sequence, as Figure 1 shown, including:

[0026] S1 Obtain visible light image sequences and thermal infrared image sequences through oblique airborne aerial photography. The oblique airborne aerial photography device includes 4 visible light cameras and 4 uncooled thermal infrared cameras.

[0027] S2 Generate visible light image point clouds and thermal infrared image point clouds according to the visible light image sequences and thermal infrared image sequences respectively, and complete the rough registration of the visible light image point clouds and the thermal infrared image point clouds.

[0028] The visible light image point clouds and the thermal infrared image point clouds are reconstructed from the visible light image sequences and the thermal infrared image sequences through the structure from motion technology SfM to generate visible light image point clouds and thermal infrared image point clouds respectively, and the rough registration of the visible light image point clouds and the thermal infrared image point clouds is realized by using auxiliary positioning data (GNSS / IMU).

[0029] S3 Based on the rough registration result, realize the fine registration of the visible light image point cloud and the thermal infrared image point cloud through the conjugate four-plane matching set.

[0030] S3.1 Resample the visible light image point cloud and the thermal infrared image point cloud using the octree-based point cloud compression method to achieve the voxelization of the visible light image point cloud and the thermal infrared image point cloud;

[0031] S3.2 Calculate the saliency features of each voxel, classify the voxels into planar regions and non-planar regions, determine the set of voxels belonging to the planar region through the saliency features, and merge the voxels within adjacent planar regions, thereby realizing the extraction of planar patches from the visible light image point cloud and the thermal infrared image point cloud.

[0032] S3.3 According to the results of planar patch extraction, taking the results of rough registration as prior conditions, achieve the fine registration of the visible light image point cloud and the thermal infrared image point cloud through the conjugate four-plane set.

[0033] The calculation method of the saliency feature is as follows: Statistically analyze the discrete point distribution within the voxels of the visible light image point cloud and the thermal infrared image point cloud, and calculate the covariance matrix; Calculate the eigenvalues (λ 1 >λ 2 >λ 3 ) and eigenvectors (e 1 ,e 2 ,e 3 ) of the covariance matrix through principal component analysis, where the eigenvector e 3 is the normal vector of the target voxel s; Calculate the saliency feature of the voxel through the eigenvalues and eigenvectors. The calculation formula is as follows:

[0034]

[0035] The method of voxel classification is as follows: Compare the magnitudes of the three saliency features. If the second saliency feature is greater than the first saliency feature and greater than the third saliency feature, the voxel is a planar region; otherwise, it is a non-planar region.

[0036] The voxel classification result can be expressed by the following formula:

[0037]

[0038] where h s is the classification result; s is the voxel, and (m 1 ,m 2 ,m 3 ) are the saliency features of the voxel s. Merge adjacent planar voxels into planar patches, and use formula (2) to determine whether the merged voxels meet the planar condition. If they meet the condition, merge them into planar patches; if they do not meet the condition, do not merge. After multiple iterations until no further merging is possible.

[0039] The method for the conjugate four-plane matching set is as follows: taking the visible light image point cloud as the reference model, find four planes {P a , P b , P c , P d} with a large spatial distance and a large difference in normal vector angles as the reference four planes. {n a , n b , n c , n d} represents the normal vectors of the four planes {P a , P b , P c , P d}. C ab represents the plane formed by the vector n a and the vector n b . C cd represents the plane formed by the vector n c and the vector n d . v represents the intersection vector between the plane C ab and the plane C cd . The angle ratios r a , P b} and the intersection vector v are: a , n b}, the normal vectors {n c , P d}, the normal vectors {n c , n d} of the double plane {P ab and r cd are:

[0040]

[0041]

[0042] Taking the thermal infrared image point cloud as the target model, find four planes {P′ a , P′ b , P′ c , P′ d} with a large spatial distance and a large difference in normal vector angles as the target four planes. {n′ a , n′ b , n′ c , n′ d} represents the normal vectors of the four planes {P′ a , P′ b , P′ c , P′ d}. C′ ab represents the plane formed by the vector n′ a and the vector n′ bThe plane formed, C′ cd denotes the plane formed by the vector n′ c and the vector n′ d The plane formed, v′ represents the intersection vector between the plane C′ ab and the plane C′ cd For the bi - plane {P′ a , P′ b}, the normal vectors {n′ a , n′ b}, for the bi - plane {P′ c , P′ d}, the normal vectors {n′ c , n′ d} and the included - angle ratio r′ between the intersection vector v′ ab and r′ cd are:

[0043]

[0044]

[0045] There is still a rigid - body affine transformation including rotation and translation between the visible - light image point cloud and the thermal - infrared image point cloud under rough registration. Under the condition of affine transformation, taking the conjugate four - plane as the matching criterion, the fine registration of the visible - light image point cloud and the thermal - infrared image point cloud can be realized. The matching condition is that the included - angle ratio between the normal vectors and the intersection vector of the reference four - plane and the target four - plane remains unchanged, that is, it satisfies the following conditions:

[0046]

[0047] Therefore, theoretically, only a pair of conjugate four - plane matching sets in the visible - light image point cloud and the thermal - infrared image point cloud need to be found to determine the affine - transformation parameters (i.e., rotation - transformation parameters and translation - transformation parameters) between them. However, in the actual solution process, the visible - light image point cloud and the thermal - infrared image point cloud generally contain hundreds of pairs of conjugate four - planes. In this embodiment, the Random Sample Consensus (RANSAC) algorithm is used to find the inlier matching set that satisfies the matching condition, that is, the condition in formula (7), with the largest number, and the least - squares method is used to realize the accurate calculation of the affine - transformation parameters, so as to realize the fine registration of the visible - light image point cloud and the thermal - infrared image point cloud.

[0048] S4 Determine the candidate thermal - infrared texture set for each three - dimensional ground point, and finely adjust the exterior orientation elements through global image pose optimization, so as to realize the three - dimensional reconstruction of the large - range temperature field.

[0049] After realizing the registration of a two-dimensional image sequence to a three-dimensional point cloud, there are still small geometric mismatches, a certain degree of radiance differences, and a large amount of information redundancy between the overlapping images. Considering that the acquisition time of airborne images is short, assuming that the temperature values of the same ground object remain unchanged on adjacent overlapping images, this embodiment proposes a method for continuously optimizing the exterior orientation elements of candidate images during the texture mapping process, that is, global image pose optimization is achieved by minimizing the temperature differences of overlapping images. The specific process is as follows:

[0050] S4.1 Remove the occluded incorrect textures through the distance buffer visibility test method, and determine the candidate thermal infrared texture set for each three-dimensional ground point.

[0051] Before thermal infrared texture mapping, for each point on the visible light image point cloud, establish a visible candidate image list. Considering that each thermal infrared image can only cover a small part of a large-scale building group, in this embodiment, the distance buffer algorithm based on spatial subdivision is used to detect occlusion. The specific processing method is as follows:

[0052] S4.1.1 Assign all three-dimensional ground points to different storage boxes according to the main direction of the building facade, and at the same time, assign the projection center of each thermal infrared image to the corresponding storage box according to its main direction coordinates;

[0053] S4.1.2 Take each thermal infrared image as a basic unit. The three-dimensional ground points corresponding to each unit determine the unit position by searching the storage box where the image is located and its surrounding storage boxes instead of the entire stereo point cloud, which helps to improve the efficiency of the distance buffer algorithm.

[0054] S4.1.3 For each pixel in the thermal infrared image, only the three-dimensional ground point closest to it can be defined as visible, and other three-dimensional ground points are removed. Through the above method, the candidate visible texture set corresponding to each three-dimensional ground point can be determined.

[0055] S4.2 Use the maximum temperature difference of overlapping images to screen diffuse reflection ground points as the target for global image pose optimization. Since the acquisition time of images is short, it is correct to assume that the temperature of the same diffuse reflection ground object remains unchanged on thermal infrared overlapping images at different shooting angles. However, for specular reflection targets (such as metals and glasses), this assumption does not hold because specular reflection objects always reflect the temperature values of nearby other objects, and the measurement results are closely related to the incident angle. Therefore, the present invention proposes a method for identifying specular reflection ground objects to exclude specular reflection ground points from participating in global image pose optimization.

[0056] S4.2.1 For ground point p, its corresponding candidate thermal infrared overlapping image set is I p , and the number of candidate overlapping images is np , (R i , t i ) represents the exterior orientation elements of the image I i , and T i (p, R i , t i ) represents the temperature value of the ground point p on a certain overlapping image I i (I i ∈ I p ). Calculate the average temperature value of all overlapping images as the standard value as shown in formula (8).

[0057]

[0058] S4.2.2 Calculate the temperature values T i (p, R i , t i ) of all candidate overlapping images and the absolute value of the difference from the standard value . Statistically calculate the maximum temperature difference ΔT(p) among all overlapping images. If the value of ΔT(p) is greater than the set threshold θ T , then the ground point p is a specular reflection point; if the value of ΔT(p) is less than or equal to the set threshold θ T , then the ground point p is a diffuse reflection point. The present invention uses the above method to eliminate the influence of specular reflection points, that is, only diffuse reflection ground points (such as: concrete, wood) are included in the global objective function to participate in the pose optimization of the thermal infrared image.

[0059] S4.3 Take each diffuse reflection three-dimensional ground point as the basic unit, aim at minimizing the temperature difference between overlapping images, use the least squares method to realize the pose optimization of the thermal infrared image sequence, calculate the equal-weight temperature average value of the overlapping images as the thermal infrared texture mapping result, and complete the reconstruction of the true three-dimensional temperature field model.

[0060] S4.3.1 For the diffuse reflection ground point p j , use the average temperature value of all candidate images as the standard value to calculate the difference between the temperature value of each candidate image and the standard value, and accumulate the sum of the temperature differences of all candidate overlapping images to form the global objective function. For the image I i , the objective function e(R i , t i ) represents the temperature difference of all ground points it contains, and its formula is:[[]]

[0061]

[0062] Among them, N i represents the number of ground points contained in the image I i , and N Prepresents the number of all diffuse ground points in the visible light image point cloud, N i <<N P , m i,j represents the ground point p j projected onto the overlapping image I i of the temperature residual. Let (R, t) = {(R i , t i )} represent the set of exterior orientation elements of all texture-mapped images. Therefore, the global objective function e(R, t) is:

[0063]

[0064] where n I represents the number of images in the thermal infrared image sequence, n p <<n I , in order to minimize the objective function shown in formula (10) and further optimize the exterior orientation elements of all candidate images, the calculation method of T(p, R, t) needs to be further refined. T(p, R, t) can be expressed as a combination of the collinearity condition equation and the temperature image interpolation function, that is, T(g(p, R, t)). Specifically, g(p, R, t) represents the collinearity condition equation, as shown in formula (11). T(g x , g y ) represents using bilinear interpolation to calculate the temperature value at the coordinates (g x , g y ) on the temperature image.

[0065]

[0066] where, (X S , Y S , Z S ) T represents the projection center of the image, r r,c represents the elements of the rotation matrix R, and the rotation matrix R can be expressed as the product of three rotation angle matrices (X, Y, Z) T represents the ground point p, f represents the camera principal distance, (x 0 , y 0 ) T represents the camera principal point, (Δx, Δy) T represents the distortion of the camera. During the iterative optimization of texture mapping, the interior orientation elements of the camera obtained by geometric calibration remain unchanged, and only the exterior orientation elements of the image sequence are optimized. The interior orientation elements of the camera include: principal distance, principal point, distortion parameters, etc.

[0067] S4.3.2 Express the global objective function in the least - squares form, and iteratively optimize the exterior orientation elements of the overlapping images until the average temperature residual of all overlapping images is less than the threshold or the number of iterations exceeds the threshold. The method for iteratively optimizing the exterior orientation elements of the overlapping images is as follows: Let \(x\) represent the target optimization vector. The parameters to be optimized include the exterior orientation elements \((R, t)\) of all overlapping images. The initial values of the exterior orientation elements \((R 0 , t 0 ) are obtained through the precise registration of the visible - light image point cloud and the thermal - infrared image point cloud. Calculate the temperature difference between each overlapping image and the average temperature value, and calculate the update vector \(\Delta x\) through the following formula:

[0068] J(x k ) T J(x k )\(\Delta x=-J(x k ) T m(x k ) (12)

[0069] where \(J(x k )\) and \(m(x k )\) represent the Jacobian vector and the residual vector calculated at \(x k \) respectively. In this embodiment, a six - dimensional vector is used to represent the exterior orientation elements of the image, and \(k\) is the number of iterations.

[0070] Calculate the partial derivatives of the exterior orientation elements \((R, t)\) through the chain rule. The calculation formula is as follows:

[0071]

[0072] where, represents the gradient of the temperature image, which can be achieved by applying the Sobel operator on the thermal - infrared image. \(J g (\zeta)\) represents the Jacobian vector of the collinearity condition equation, that is, the partial derivative of the exterior orientation elements. After obtaining the update vector \(\Delta x\), the exterior orientation elements can be updated through the following formula:

[0073] x k+1 =x k +\(\Delta x

[0074] where \(k\) is the number of iterations. Continue the iteration until the average temperature residual of all overlapping images is less than the threshold or the number of iterations exceeds the threshold.

[0075] S4.3.3 Calculate the equal - weight temperature average value of all overlapping images as the thermal - infrared texture mapping result, and generate the true three - dimensional temperature field of the building.

[0076] Embodiment 2

[0077] Based on the same inventive concept, this embodiment discloses a true three-dimensional temperature field reconstruction system for buildings, including:

[0078] An image acquisition module for obtaining a visible light image sequence and a thermal infrared image sequence through oblique aerial photography;

[0079] A rough registration module for generating a visible light image point cloud and a thermal infrared image point cloud respectively according to the visible light image sequence and the thermal infrared image sequence, and completing the rough registration of the visible light image point cloud and the thermal infrared image point cloud;

[0080] A fine registration module for realizing the fine registration of the visible light image point cloud and the thermal infrared image point cloud through a conjugate four-plane matching set according to the rough registration result;

[0081] A texture mapping module for determining a candidate thermal infrared texture set for each three-dimensional ground point, and finely adjusting the exterior orientation elements through global image pose optimization, so as to realize the three-dimensional reconstruction of a large-range temperature field.

[0082] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0083] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for realizing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0084] These computer program instructions can also be stored in a computer-readable memory capable of guiding a computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means realizes the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0085] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable apparatus to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable apparatus provide steps for realizing the functions specified in one process or a plurality of processes and / or one block or a plurality of blocks. Figure 1 one process or a plurality of processes and / or Figure 1 steps of the functions specified in one block or a plurality of blocks.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: it is still possible to modify the specific implementation manners of the present invention or make equivalent replacements. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention shall be covered by the protection scope of the claims of the present invention. The above content is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, and all of them should be covered by the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for constructing a true three-dimensional temperature field of buildings from airborne thermal infrared images, characterized in that, it includes: Obtaining a visible light image sequence and a thermal infrared image sequence through oblique aerial photography; Generating a visible light image point cloud and a thermal infrared image point cloud respectively according to the visible light image sequence and the thermal infrared image sequence, and completing the rough registration of the visible light image point cloud and the thermal infrared image point cloud; Based on the rough registration result, the fine registration of the visible light image point cloud and the thermal infrared image point cloud is realized through the conjugate four-plane matching set; taking the visible light image point cloud as the reference model, four planes with a large spatial distance and a large difference in normal vector angles are found as the reference four planes; Using the thermal infrared image point cloud as the target model, find four planes with a large spatial distance and a large difference in normal vector angles as the target four planes; Taking the unchanged ratio of the angles between the normal vectors and the intersection vectors of the four planes of the reference model and the target model as the matching condition, using the Random Sample Consensus algorithm RANSAC to find the inlier matching set that satisfies the matching condition the most, and using the least squares method to achieve the precise calculation of the affine transformation parameters, so as to realize the fine registration of the visible light image point cloud and the thermal infrared image point cloud; Determining the candidate thermal infrared texture set of each three-dimensional ground point, and fine-tuning the external orientation elements through global image pose optimization, so as to realize the three-dimensional construction of the large-scale temperature field.

2. The method for constructing a true three-dimensional temperature field of buildings according to claim 1, characterized in that, The visible light image point cloud and the thermal infrared image point cloud are constructed from the visible light image sequence and the thermal infrared image sequence through the Structure from Motion technology SfM, and the visible light image point cloud and the thermal infrared image point cloud are generated respectively, and the rough registration of the visible light image point cloud and the thermal infrared image point cloud is realized by using the auxiliary positioning data.

3. The method for constructing a true three-dimensional temperature field of buildings according to claim 1, characterized in that, The method for fine registration is: Resampling the visible light image point cloud and the thermal infrared image point cloud by using the point cloud compression method based on octree to realize the voxelization of the visible light image point cloud and the thermal infrared image point cloud; Calculating the significant features of the visible light image point cloud and the thermal infrared image point cloud after voxelization, classifying the voxels into plane regions and non-plane regions, determining the voxel set belonging to the plane region through the significant features, and merging the voxels in adjacent plane regions, so as to realize the extraction of plane patches of the visible light image point cloud and the thermal infrared image point cloud; According to the result of the plane patch extraction, taking the result of the rough registration as the prior condition, realizing the fine registration of the visible light image point cloud and the thermal infrared image point cloud through the conjugate four-plane set.

4. The method for constructing a true three-dimensional temperature field of buildings according to claim 3, characterized in that, The calculation method of the significant feature is: Calculating the covariance matrix through all discrete points in the voxel of the visible light image point cloud and the thermal infrared image point cloud; Calculating the eigenvalues and eigenvectors of the covariance matrix through principal component analysis; Calculating the significant feature of the voxel through the eigenvalues and eigenvectors.

5. The method for constructing a true three-dimensional temperature field of buildings according to claim 4, characterized in that, The method for voxel classification is: obtaining three significant features, if the second significant feature is greater than the first significant feature and greater than the third significant feature, then the voxel is a plane region, otherwise it is a non-plane region.

6. The method for constructing a true three-dimensional temperature field of buildings according to any one of claims 1-5, characterized in that, The method for determining the candidate thermal infrared texture set is: Remove the occluded false textures through the distance buffer visibility test method, and determine the candidate thermal infrared texture set for each three-dimensional ground point; taking each diffuse three-dimensional ground point as the basic unit, aiming at minimizing the temperature difference between overlapping images, use the least squares method to optimize the pose of the thermal infrared image sequence, calculate the equal-weighted temperature average value of the overlapping images as the thermal infrared texture mapping result, and complete the construction of the true three-dimensional temperature field model.

7. The method for constructing the true three-dimensional temperature field of a building according to claim 6, characterized in that the method for optimizing the pose of the thermal infrared image sequence is: taking the diffuse ground points as the basic unit, calculating the average temperature value of all the candidate thermal infrared texture sets, and using it as the reference temperature value; calculating the temperature difference between the temperature value of each overlapping image and the reference temperature value; taking the sum of the temperature differences as the global objective function; expressing the global objective function in the least squares form, and iteratively optimizing the exterior orientation elements of the overlapping images until the average temperature residual of all overlapping images is less than the threshold or the number of iterations exceeds the threshold; calculating the equal-weighted temperature average value of all overlapping images as the thermal infrared texture mapping result, and generating the true three-dimensional temperature field of the building.

8. The method for constructing the true three-dimensional temperature field of a building according to claim 7, characterized in that The method for iteratively optimizing the exterior orientation elements of overlapping images is as follows: Let represent the target optimization vector. The parameters to be optimized include the exterior orientation elements of all overlapping images. The difference between each overlapping image and the average temperature value of the overlapping images is accumulated and updated through the following formula: where k is the number of iterations, is the update vector, and the iteration continues until the average temperature residual of all overlapping images is less than the threshold or the number of iterations exceeds the threshold.

9. An airborne thermal infrared image true three-dimensional temperature field construction system, characterized in that comprising: an image acquisition module, configured to obtain a visible light image sequence and a thermal infrared image sequence through oblique aerial photography; a rough registration module, configured to generate a visible light image point cloud and a thermal infrared image point cloud according to the visible light image sequence and the thermal infrared image sequence respectively, and complete the rough registration of the visible light image point cloud and the thermal infrared image point cloud; A fine registration module, which is used to implement fine registration of the visible light image point cloud and the thermal infrared image point cloud through a conjugate four-plane matching set according to the rough registration result; taking the visible light image point cloud as a reference model, four planes with a large spatial distance and a large difference in normal vector angles are found As the reference four planes; Taking the thermal infrared image point cloud as the target model, find four planes with a large spatial distance and a large difference in normal vector angles As the target four planes; Taking the invariance of the angle ratio between the normal vectors and the intersection vectors of the four planes of the reference model and the target model as the matching condition, using the random sample consensus algorithm RANSAC to find the inlier matching set that satisfies the most matching conditions, and using the least squares method to achieve the precise solution of the affine transformation parameters, so as to achieve the precise registration of the visible light image point cloud and the thermal infrared image point cloud; a texture mapping module, configured to determine the candidate thermal infrared texture set for each three-dimensional ground point, and fine-tune the exterior orientation elements through global image pose optimization, so as to realize the three-dimensional construction of a large-scale temperature field.

Images