A method and system for constructing a three-dimensional model of the temperature field of a building using a handheld dual camera

Abstract

The present invention belongs to the technical field of thermal infrared image photogrammetry, and relates to a method and system for constructing a three-dimensional model of a temperature field of a handheld dual-camera without auxiliary positioning data, including: obtaining a visible light image sequence and a thermal infrared image sequence; generating a three-dimensional point cloud of visible light images according to the visible light image sequence; performing feature extraction and precise matching on visible light-thermal infrared image pairs; calculating the external orientation elements of the thermal infrared image sequence relative to the visible light image point cloud; using a visibility test method based on a distance buffer to remove occluded wrong textures and determining a candidate set of visible texture features; achieving the optimal selection of thermal infrared image textures with the goal of minimizing the temperature difference between overlapping images and completing the three-dimensional reconstruction of the temperature field. The present invention reduces the radiation temperature difference between overlapping images and can achieve high-precision three-dimensional reconstruction of the building facade temperature field.

Description

Technical Field

[0001] The present invention relates to a method and a system for constructing a three-dimensional model of a building temperature field by using a handheld dual camera, and belongs to the technical field of thermal infrared photogrammetry. Background Art

[0002] In order to maintain the indoor temperature of a building within a range comfortable for humans, the building has become a major contributor to energy consumption. The energy consumption related to buildings accounts for about 36% of the global total energy consumption, and the carbon dioxide emissions account for 39% of the global total emissions. In order to improve the effective energy utilization rate of buildings, reducing heat loss caused by aging and damage of building components has become the key. Thermal infrared cameras have the ability to capture and visualize heat loss in buildings. Therefore, constructing a three-dimensional temperature field of a building by using thermal infrared image sequences and realizing automatic identification of building thermal cracks have become research hotspots.

[0003] Considering that the spatial resolution of a single thermal infrared image is relatively low and the field of view is limited, each image can only cover a part of the three-dimensional building scene, and directly applying two-dimensional images lacks three-dimensional spatial position information. Therefore, existing research mainly adopts the method of registering and texture mapping thermal infrared image sequences to an existing three-dimensional model. A three-dimensional model with correct geometric relationships and accurate temperature information can accurately locate, identify, and visualize the heat loss of a building.

[0004] Most of the existing registration methods from two-dimensional image sequences to three-dimensional models rely on auxiliary positioning data, such as GNSS / IMU (Global Navigation Satellite System / Inertial Measurement Unit) data. However, currently commonly used handheld thermal infrared cameras usually lack prior auxiliary positioning data. Therefore, existing methods are not applicable to handheld thermal infrared cameras. In addition, existing texture mapping methods are mainly used for visible light image sequences. In order to improve the visual effect of three-dimensional models, the stitching seam effect caused by uneven illumination conditions is mainly smoothed through models such as conditional random fields. In the texture mapping of thermal infrared image sequences, the stitching seam corresponds to building thermal anomalies and is not suitable for excessive smoothing by traditional methods. Therefore, existing texture mapping methods are difficult to be applied to the three-dimensional reconstruction of a building temperature field based on thermal infrared image sequences. Summary of the Invention

[0005] Aiming at the above problems, the purpose of the present invention is to provide a method and a system for constructing a three-dimensional model of a building temperature field by using a handheld dual camera, which can realize the detection of building thermal anomalies without auxiliary positioning data.

[0006] To achieve the above object, the present invention proposes the following technical solutions: A method for reconstructing a three-dimensional model of a building temperature field based on a handheld dual-camera, comprising: acquiring a visible light image sequence and a thermal infrared image sequence; forming a visible light image point cloud model according to the visible light image sequence; using a heterologous image matching method to achieve precise registration of visible light-thermal infrared image pairs; using resection to calculate the relative pose of the thermal infrared image sequence relative to the visible light image point cloud model; using a distance buffer technology to remove occluded wrong textures and determine a candidate texture image set for each three-dimensional ground point; and automatically selecting the optimal thermal infrared texture value for each three-dimensional point with the goal of minimizing the temperature difference of overlapping images to construct an accurate three-dimensional temperature field model.

[0007] Further, the method for forming the visible light image point cloud model is: extracting and matching features of the visible light image sequence; obtaining the internal and external orientation elements of the visible light camera through the extracted feature points and optimizing them; and generating a visible light image point cloud through multi-view dense matching.

[0008] Further, the visible light-thermal infrared image pair matching method is: using the histogram of oriented phase congruency (HOPC) to extract the structural similarity features of the visible light image and the thermal infrared image respectively; achieving rough matching of the visible light-thermal infrared image pair through a pyramid-level guided image matching method; and achieving precise matching of the visible light-thermal infrared image pair through an energy function optimization method based on a thin plate spline function.

[0009] Further, the phase consistency direction information is expressed as:

[0010]

[0011] where o no (θ) represents the convolution result of the log Gabor wavelet with an odd-symmetric filter in the direction θ, and the Euclidean distance of the HOPC descriptor is used as the similarity measure HOPC n to perform homologous point matching.

[0012] Furthermore, the image matching method guided by the pyramid level by level is as follows: Build an image pyramid, and set the number of pyramid levels and the scale factor of pyramid downsampling; Divide a uniform grid on the original visible light image and the thermal infrared image, adopt the block-based Harris operator to extract evenly distributed feature points, and after the feature point extraction is completed, map the coordinates of the feature points to each layer of the image pyramid; Perform matching based on HOPC on the top layer of the image pyramid. Taking the coordinates of the extracted Harris feature points as the center, determine the search radius according to the initial accuracy of the image and the magnification of pyramid downsampling, and use the HOPC algorithm for bidirectional matching; Establish an image geometric transformation model according to the coordinates of the homologous points obtained by the matching, adopt the RANSAC algorithm to eliminate the mismatched points, and transfer the remaining homologous points as the initial values to the next layer of the image pyramid; Under the guidance of the matching result of the previous layer of the image pyramid, perform matching based on HOPC on this layer of the pyramid, and perform matching level by level until the rough matching result of the visible light-thermal infrared image pair is obtained.

[0013] Furthermore, the energy function optimization method based on the thin plate spline function is as follows: Assume that there are N pairs of matching homologous points in the visible light image point cloud model and the thermal infrared image sequence, and their coordinates are represented by X = {X i , i = 1, 2, …, N} and Y = {Y i , i = 1, 2, …, N} respectively. Use f(Y i ) to represent the thin plate spline mapping function from Y to X. The energy function E tps (f) of this transformation is as follows:

[0014]

[0015] where λ is a coefficient. The first term in the above energy function is the accuracy constraint, and the second term is the bending degree constraint. By adjusting the parameters, the bending energy of the thin plate is minimized.

[0016] Furthermore, the method for obtaining the relative pose is as follows: Calculate the three-dimensional position coordinates corresponding to the feature points of the visible light image in the visible light image point cloud model through unidirectional positioning; According to the three-dimensional ground point position coordinates and the thermal infrared image feature point coordinates, calculate the exterior orientation elements of the thermal infrared image relative to the visible light image point cloud through resection; Calculate the relative pose of each thermal infrared image relative to the visible light image point cloud model one by one, so as to complete the relative positioning of the thermal infrared image sequence relative to the visible light image point cloud model.

[0017] Further, the method for removing occluded incorrect textures is as follows: all three-dimensional ground points are assigned to different storage boxes according to the main direction of the building facade, and the projection center of each thermal infrared image is assigned to the corresponding storage box according to its main direction coordinates; taking each thermal infrared image as a basic unit, the three-dimensional ground points corresponding to each unit are determined by searching the storage box where the thermal infrared image is located and the storage boxes around it; for each pixel in the thermal infrared image, only the three-dimensional ground point closest to it is defined as visible, and other three-dimensional ground points are removed.

[0018] Further, the method for determining the candidate texture image of each three-dimensional ground point is as follows: taking each three-dimensional ground point as a basic unit, using the temperature difference between overlapping images as the objective function, calculating the arithmetic mean, geometric mean, and harmonic mean of the temperatures of the three-dimensional ground points, and taking the minimization of the temperature difference between overlapping images as the objective, automatically selecting the method with the smallest temperature difference among the three means as the optimal texture to achieve three-dimensional reconstruction of the temperature field.

[0019] The present invention also discloses a three-dimensional model reconstruction system for the building temperature field based on a handheld dual camera, including: an image acquisition module for acquiring a visible light image sequence and a thermal infrared image sequence; a visible light image point cloud generation module for forming a visible light image point cloud model according to the visible light image sequence; a thermal infrared-visible light image pair matching module for matching the visible light image point cloud model with the thermal infrared image sequence; a relative pose calculation module for calculating the relative pose of the thermal infrared image sequence with respect to the visible light image point cloud model according to the matching result of the visible light-thermal infrared image pair; a temperature field texture mapping module for removing occluded incorrect textures, determining the candidate texture image of each three-dimensional ground point, automatically selecting the optimal texture, and generating a three-dimensional model of the building temperature field.

[0020] Due to the adoption of the above technical solutions, the present invention has the following advantages:

[0021] 1. The prior art mainly uses laser point clouds as the three-dimensional reference model. Since the cost of laser scanners is relatively high, in order to reduce costs, the present invention uses visible light image point clouds as the three-dimensional reference, and realizes thermal infrared texture mapping and three-dimensional modeling of the temperature field by fixedly connecting a visible light camera and a thermal infrared camera.

[0022] 2. Commonly used registration methods for point clouds and image sequences generally require relatively accurate auxiliary positioning data as prior information of the initial pose. The present invention does not require such prior information. First, heterologous image pairs are determined by fixedly connecting a thermal infrared camera and a visible light camera, and then high-precision registration is achieved through thermal infrared-visible light heterologous image pair matching, one-way positioning of visible light images, and resection of thermal infrared images.

[0023] 3. Traditional image matching methods (such as Scale-Invariant Feature Transform, SIFT) are limited to homologous image matching and cannot handle heterologous image matching. The present invention first uses the histogram of oriented phase congruency (HOPC) matching method guided by the image pyramid level by level to achieve rough registration of visible light-thermal infrared image pairs; then uses the image geometric transformation based on thin plate spline function to achieve fine optimization of visible light-thermal infrared image pairs.

[0024] 4. After the registration of the two-dimensional image sequence to the three-dimensional point cloud is achieved, the temperature values corresponding to the same three-dimensional ground point in different thermal infrared images should be unchanged. However, in reality, due to the influence of factors such as external environment changes and calibration errors, there are differences in the temperature values provided on the overlapping images. The present invention proposes an optimal texture selection method based on minimizing the temperature difference of overlapping images to achieve accurate three-dimensional reconstruction of the building temperature field. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of a method for constructing a three-dimensional model of a building temperature field based on a handheld dual camera in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail through specific embodiments. However, it should be understood that the provision of the specific embodiments is only for better understanding of the present invention, and they should not be construed as limitations to the present invention. In the description of the present invention, it should be understood that the terms used are only for the purpose of description and cannot be construed as indicating or implying relative importance.

[0027] In order to solve the technical problem in the prior art that when improving the visual effect of a 3D model, the stitching seam effect caused by uneven illumination conditions is smoothed by models such as conditional random fields. In the texture mapping of thermal infrared image sequences, the stitching seam corresponds to the thermal anomaly of a building, and it is not suitable to over-smooth it by traditional methods. The present invention provides a method and system for constructing a three-dimensional model of the temperature field of a building with a handheld dual-camera. The present invention can realize thermal infrared texture mapping and three-dimensional modeling of the temperature field through a fixedly connected visible light camera and a thermal infrared camera system, without using a relatively expensive laser scanner; the present invention realizes high-precision registration of visible light images and thermal infrared images through the matching of thermal infrared-visible light heterogeneous image pairs, one-way positioning of visible light images, and resection of thermal infrared images, without the need to provide prior information, with higher registration accuracy, better accuracy, and the result not depending on prior information. The present invention realizes the rough registration of visible light-thermal infrared image pairs through a phase consistency direction map (the histogram of oriented phase congruency, HOPC) matching method guided step by step based on an image pyramid; realizes the fine optimization of visible light-thermal infrared image pairs by using an image geometric transformation based on a thin plate spline function; and realizes the optimal selection of thermal infrared texture and the reconstruction of the three-dimensional temperature field of a building by minimizing the temperature difference of overlapping images.

[0028] Embodiment 1

[0029] Figure 1 It is a method for constructing a three-dimensional temperature field of a building based on a handheld dual-camera in an embodiment of the present invention, as Figure 1 shown, including:

[0030] S1 Obtain a visible light image sequence and a thermal infrared image sequence.

[0031] In this embodiment, the handheld dual-camera system includes a fixedly connected visible light camera and a thermal infrared camera. During the process of photographing the facade of a building, the handheld dual-camera system generates a thermal infrared image sequence and a visible light image sequence with a high degree of overlap through horizontal translation and vertical pitching, and the two cameras image simultaneously at any moment. In this embodiment, the imaging system can also adopt other moving methods to achieve imaging, such as: placing it on a drone or an airplane. The visible light image sequence is used to construct a fine three-dimensional model of the building facade, and the thermal infrared image sequence is used to provide an accurate temperature field for the building facade.

[0032] S2 Form a visible light image point cloud model according to the visible light image sequence.

[0033] Due to the low spatial resolution, poor contrast, and blurred edge features of thermal infrared images, it is difficult to achieve high-quality 3D reconstruction based on thermal infrared image sequences using existing 3D reconstruction techniques. In this embodiment, the Structure from Motion (SfM) technique is used to perform 3D reconstruction on the visible light image sequence to obtain a visible light image point cloud. The specific steps include:

[0034] S2.1 Extract and match features from the visible light image sequence;

[0035] S2.2 Obtain the internal and external orientation elements of the visible light image sequence using the feature matching results;

[0036] S2.3 Generate a visible light image point cloud model using the multi-view dense matching technique.

[0037] S3 Visible light-thermal infrared image pair matching. In this embodiment, the method of using a handheld dual camera to simultaneously obtain heterogeneous image sequences is adopted. Therefore, for each visible light image, there naturally exists a thermal infrared image taken of the same scene. In order to establish the pose relationship between the thermal infrared image sequence and the visible light image sequence, this embodiment uses the Hierarchical Orientation Phase Congruency (HOPC) matching method based on the image pyramid and the image geometric transformation method based on the thin plate spline function to achieve the fine optimization of the visible light-thermal infrared image pair. The specific operation process is as follows:

[0038] S3.1 Use the Hierarchical Orientation Phase Congruency (HOPC) algorithm to extract the corner features and edge features of the thermal infrared image and the visible light image as the candidate feature set. Existing methods based on image gray level or gradient cannot solve the matching problem of thermal infrared-visible light heterogeneous images. The reason why these traditional image matching methods cannot be effectively applied is that they mainly use the gradient information in the image spatial domain to describe and detect feature points. Therefore, they can only process small linear gray level changes in the image. However, there are often large non-linear radiation differences in the thermal infrared-visible light image pair. In this embodiment, the HOPC algorithm is used to detect and describe the feature points of the thermal infrared-visible light image pair, and a robust description considering the non-linear radiation differences of heterogeneous images is achieved through the frequency domain phase map.

[0039] The Hierarchical Orientation Phase Congruency, that is, HOPC, calculates the maximum phase value of the superposition of Fourier harmonic components of significant features such as corners from the perspective of the frequency domain. Since the phase congruency theory represents local energy using a cosine function, its smooth peak will cause difficulties in positioning. Therefore, the log Gabor filter is used to improve the local energy formula in the phase congruency, extending it from one-dimensional space to two-dimensional space, and good results can be obtained. The intensity information PC(x, y) obtained by the phase congruency model can be expressed as:

[0040]

[0041] Among them, W o (x, y) is the weight factor for frequency expansion; A no (x, y) is the amplitude of the image point (x, y) at the log Gabor filter scale n and direction o; ξ is the noise threshold; The symbol means taking itself when the value is positive, otherwise taking 0; ε is a small constant to avoid division by zero; A no (x, y)ΔΦ no (x, y) is calculated from the even-symmetric filter response value and odd-symmetric filter response value of the log Gabor wavelet.

[0042] The phase consistency direction histogram makes use of the template structure of the Histogram of Orientated Gradient (HOG). It not only utilizes the phase consistency intensity information but also combines the direction information of the phase consistency to construct a feature descriptor HOPC that can reflect the internal structure of the image. It is applicable to the matching of visible light images and thermal infrared images with similar shapes and structures but significant non-linear radiation differences. The phase consistency direction information Φ can be expressed as:

[0043] Φ = arctan(∑ θ (o no (θ)sinθ), ∑ θ (o no (θ)cosθ)) (2)

[0044] Among them, o no (θ) represents the convolution result of the odd-symmetric filter of the log Gabor wavelet in the direction θ. The Euclidean distance of the HOPC descriptor is used as the similarity measure (HOPC n ) for homologous point matching, and the calculation formula is expressed as:

[0045]

[0046] In the formula, V A and V B respectively represent the HOPC descriptors of the template images A and B.

[0047] The image matching method based on HOPC can better take into account the non - linear radiation differences between visible light images and thermal infrared images. However, HOPC does not have rotational invariance and scale invariance, and is not applicable to the cases where heterologous images have rotational changes and scale differences. In addition, since HOPC needs to extract the geometric structure and shape contour features of images as candidate matching features, when the structural and shape features of images are not rich enough, its matching performance will also decline.

[0048] S3.2 Coarse matching of visible - thermal infrared image pairs is achieved through the image matching method guided by the pyramid level by level.

[0049] Considering that the image matching method based on HOPC is sensitive to rotational and scale changes, the matching strategy guided by the image pyramid level by level is adopted in this embodiment. The specific steps are as follows:

[0050] S3.2.1 Build an image pyramid, and set the number of pyramid levels and the scale factor of pyramid downsampling.

[0051] Due to the limited spatial resolution of thermal infrared images (640×512 pixels), the number of pyramid levels is set to 3, and the scale factor of pyramid downsampling is set to 2, that is, the size of the upper - layer image in the row direction and the column direction is 1 / 2 of that of the adjacent lower - layer image, and each layer of visible light images is downsampled to the number of pixels similar to that of thermal infrared images.

[0052] S3.2.2 Divide a uniform grid on the original image, and use the block - based Harris operator to extract evenly distributed feature points. The extraction method of feature points is the same as that of the HOPC algorithm. After the extraction of feature points, the coordinates of the feature points are mapped to each layer of the image pyramid.

[0053] S3.2.3 Perform HOPC - based matching on the top layer of the image pyramid. Taking the coordinates of the extracted Harris feature points as the center, determine the search radius according to the initial accuracy of the image and the magnification of pyramid downsampling, and use the HOPC algorithm for bidirectional matching.

[0054] S3.2.4 Establish an image geometric transformation model according to the coordinates of the homologous points obtained by matching, and use the RANSAC (random sample consensus) algorithm to eliminate the mismatched points, and transfer the remaining homologous points as the initial values to the next layer of the image pyramid.

[0055] S3.2.5 Under the guidance of the matching result of the upper - layer image pyramid, perform HOPC - based matching on this layer of the pyramid, and perform matching level by level until the coarse matching result of the visible - thermal infrared image pair is obtained.

[0056] S3.3 Achieve the precise matching of visible-light and thermal-infrared image pairs through the energy function optimization method based on thin plate spline function.

[0057] After obtaining sufficient corresponding points between the visible-light image and the thermal-infrared image, determining the optimal geometric transformation model is crucial for the precise matching between images. The original HOPC matching method uses a piecewise linear model for geometric transformation of images. This model is suitable for the case of a small number of feature points. When the number of feature points is large, there are deficiencies such as discontinuous geometric transformation models and being greatly affected by mis-matched points. The thin plate spline function can decompose the geometric transformation relationship between corresponding points into a rigid transformation and a non-rigid transformation. Under the constraint of ensuring one-to-one correspondence between corresponding points, by minimizing the energy function, jointly solving the matching matrix and mapping parameters between point sets, high-precision registration between images is achieved.

[0058] The specific steps are as follows: Assume that there are N pairs of matching corresponding points in the visible-light image point cloud model and the thermal-infrared image sequence, and their coordinates are represented by X = {X i , i = 1, 2, …, N} and Y = {Y i , i = 1, 2, …, N} respectively. Among them, Use f(Y i ) to represent the thin plate spline mapping function from Y to X, and its energy function E tps (f) is as follows:

[0059]

[0060] Among them, λ is a coefficient. The first term in the above energy function is the accuracy constraint, and its main role is to make the points in the thermal-infrared image be mapped to the corresponding points in the visible-light image as accurately as possible; the second term is the bending degree constraint. By adjusting the coefficient λ, the bending energy of the thin plate is minimized, that is, by minimizing the energy function E tps (f), the optimal solution of the mapping function is obtained.

[0061] S4 Calculate the relative pose of the thermal-infrared image sequence with respect to the visible-light image point cloud model, use the visibility test method based on the distance buffer to remove the occluded wrong textures, determine the candidate texture images for each three-dimensional ground point, and take minimizing the temperature difference provided by the overlapping images as the goal. By comparing three different thermal-infrared texture mapping methods, achieve the optimal texture selection and generate a three-dimensional model of the building temperature field.

[0062] S4.1 Calculate the relative pose of the thermal infrared image with respect to the point cloud of the visible light image through visible light-thermal infrared images. First, use one-way positioning to calculate the three-dimensional position coordinates corresponding to the feature points of the visible light image in the visible light image point cloud model; according to the three-dimensional ground point position coordinates and the thermal infrared image feature point coordinates, calculate the exterior orientation elements of the thermal infrared image with respect to the visible light image point cloud through resection. Calculate the relative pose of each thermal infrared image with respect to the visible light image point cloud one by one, so as to complete the relative positioning of the thermal infrared image sequence with respect to the visible light image point cloud model.

[0063] S4.2 Use the visibility test based on the distance buffer to remove the occluded incorrect textures and determine the candidate visible texture feature set for each three-dimensional ground point.

[0064] Before thermal infrared texture mapping, for each point on the visible light three-dimensional model, a list of visible candidate images must be established. Considering that each thermal infrared image can only cover a small part of the entire building facade, in this embodiment, the distance buffer algorithm based on spatial subdivision is used to detect occlusion. The specific method is as follows:

[0065] S4.2.1 All three-dimensional ground points are assigned to different storage boxes according to the main direction of the building facade, and the projection center of each thermal infrared image is assigned to the corresponding storage box according to its main direction coordinates;

[0066] S4.2.2 Take each thermal infrared image as a basic unit, and the three-dimensional ground points corresponding to each unit are determined by searching the storage box where the image is located and its surrounding storage boxes instead of the entire stereo point cloud, which helps to improve the efficiency of the distance buffer algorithm.

[0067] S4.2.3 For each pixel in the thermal infrared image, only the three-dimensional ground point closest to it can be defined as visible, and other three-dimensional ground points are removed. Through the above method, the candidate visible texture set corresponding to each three-dimensional ground point can be determined.

[0068] S4.3 With the goal of minimizing the temperature difference of overlapping images, complete the optimal texture selection and achieve the accurate three-dimensional reconstruction of the building facade temperature field.

[0069] After achieving the registration of the two-dimensional thermal infrared image sequence to the three-dimensional visible light point cloud, there are still certain degrees of radiance differences and a large amount of information redundancy between the overlapping images. For the same ground object, the temperature values provided by the overlapping images should theoretically be exactly the same. However, due to tiny geometric registration errors and radiation temperature differences, the temperature values provided by the overlapping images are not the same. Therefore, this embodiment proposes an optimal texture selection method based on minimizing the temperature differences between overlapping images: taking each three-dimensional ground point as the basic unit and using the temperature differences between the overlapping images as the objective function, calculating the arithmetic mean, geometric mean, and harmonic mean of the temperatures of the three-dimensional ground points, so as to obtain the average temperature value of the three-dimensional ground points, comparing the average temperature value with the corresponding points in the thermal infrared image sequence, obtaining the temperature differences of the overlapping images, minimizing the temperature differences, and thus obtaining the final texture mapping result to achieve the three-dimensional reconstruction of the temperature field.

[0070] Specifically, for the ground point p, the corresponding set of candidate thermal infrared overlapping images is I p , and the number of overlapping images is n p , and the temperature difference e p between the overlapping images is calculated as shown in the following formula:

[0071]

[0072] where (R i , T i ) represents the exterior orientation elements of the image I i , and G i (p, R i , T i ) represents the temperature value of the ground point p on a certain overlapping image I i (I i ∈ I p ), and the average temperature value can be expressed as the arithmetic mean , the geometric mean , or the harmonic mean

[0073]

[0074]

[0075]

[0076] Since the image acquisition time is short, it can be considered that the ground point p remains unchanged during the image acquisition process, and the temperature values provided by the overlapping images should be exactly the same. However, due to tiny geometric registration errors and radiation temperature differences, the temperature values provided by the overlapping images are not the same. Therefore, for the ground point p, in order to make the temperature difference ep When reaching the minimum value, this embodiment compares the advantages and disadvantages of three different texture selection methods (arithmetic mean, geometric mean, harmonic mean), that is, uses formula (5) to calculate the temperature difference of the overlapping images of the three texture selection methods respectively, and automatically selects the method that can obtain the minimum temperature difference to complete texture mapping, so as to realize the three-dimensional reconstruction of the temperature field.

[0077] Embodiment 2

[0078] Based on the same inventive concept, this embodiment discloses a three-dimensional model reconstruction system for the temperature field of a building based on a handheld dual-camera, including:

[0079] An image acquisition module, configured to acquire a visible light image sequence and a thermal infrared image sequence;

[0080] A visible light image point cloud generation module, configured to form a visible light image point cloud model according to the visible light image sequence;

[0081] A thermal infrared-visible light image pair matching module, configured to match the visible light image point cloud model with the thermal infrared image sequence;

[0082] A relative pose calculation module, configured to calculate the relative pose of the thermal infrared image sequence relative to the visible light image point cloud model according to the matching result of the visible light-thermal infrared image pair;

[0083] A temperature field texture mapping module, configured to remove the occluded incorrect textures, determine the candidate texture images of each three-dimensional ground point, automatically select the optimal texture, and generate a three-dimensional model of the building temperature field.

[0084] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0085] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate for implementation in the process Figure 1One or more processes and / or blocks Figure 1 means for the functions specified in one or more blocks.

[0086] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction means that implements the functions in the process Figure 1 One or more processes and / or blocks Figure 1 specified in one or more blocks.

[0087] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus, such that a series of operational steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in the process Figure 1 One or more processes and / or blocks Figure 1 specified in one or more blocks.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: it is still possible to modify the specific implementation manners of the present invention or make equivalent replacements. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention shall be covered by the protection scope of the claims of the present invention. The above content is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or replacements, which should all be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. A method for constructing a three-dimensional model of the building temperature field based on a handheld dual camera, characterized in that, it includes: Obtaining a visible light image sequence and a thermal infrared image sequence; Forming a visible light image point cloud model according to the visible light image sequence; Based on the visible light-thermal infrared image pair matching, calculating the relative pose of the thermal infrared image sequence relative to the visible light image point cloud model; Removing the occluded wrong textures based on the distance buffer algorithm, determining the candidate texture images of each three-dimensional ground point, and realizing the fine optimization of the visible light-thermal infrared image pair by using the image geometric transformation based on the thin plate spline function; realizing the optimal selection of the thermal infrared texture and constructing the three-dimensional temperature field of the building by minimizing the temperature difference of the overlapping images; The method for removing the occluded wrong textures is: All three-dimensional ground points are assigned to different storage boxes according to the main direction of the building facade, and the projection center of each thermal infrared image is assigned to the corresponding storage box according to its main direction coordinates; Taking each thermal infrared image as a basic unit, the three-dimensional ground points corresponding to each unit are determined by searching the storage box where the thermal infrared image is located and its surrounding storage boxes; For each pixel in the thermal infrared image, only the three-dimensional ground point closest to it is defined as visible, and other three-dimensional ground points are removed.

2. The method for constructing a three-dimensional model of the building temperature field according to claim 1, characterized in that, the method for forming the visible light image point cloud model is: Performing feature extraction and matching on the visible light image sequence; Obtaining the internal and external orientation elements of the visible light image sequence through the feature matching results and optimizing them; Generating a visible light image point cloud model by using the multi-view dense matching technology.

3. The method for constructing a three-dimensional model of the building temperature field according to claim 1, characterized in that, the method for matching the visible light image point cloud model with the thermal infrared image sequence is: Extracting the candidate feature sets of the visible light image and the thermal infrared image through the phase consistency direction histogram; Realizing the rough matching of the visible light-thermal infrared image pair through the image matching method guided by the pyramid level by level; Realizing the fine matching of the visible light-thermal infrared image pair through the energy function optimization method based on the thin plate spline function.

4. The method for constructing a three-dimensional model of the building temperature field according to claim 3, characterized in that, the phase consistency direction information is expressed as: Among them, represents the convolution result of the odd-symmetric filter of the log Gabor wavelet in the direction , and the Euclidean distance of the HOPC descriptor is used as the similarity measure to perform corresponding point matching.

5. The method for constructing a three-dimensional model of the building temperature field according to claim 3, characterized in that, the image matching method guided by the pyramid level by level is: Establishing an image pyramid, setting the number of pyramid levels and the scale factor of pyramid downsampling; Dividing a uniform grid on the original thermal infrared image and the visible light image, adopting the block-based Harris operator to extract evenly distributed feature points, and mapping the coordinates of the feature points to each layer of the image pyramid after the feature point extraction; Performing the matching based on HOPC on the top layer of the image pyramid, taking the coordinates of the extracted Harris feature points as the center, determining the search radius according to the initial accuracy of the image and the magnification of pyramid downsampling, and performing two-way matching by using the HOPC algorithm; An image geometric transformation model is established based on the coordinates of the homologous points obtained by matching. The RANSAC algorithm is used to eliminate the mismatched points, and the remaining homologous points are passed as initial values to the next layer of the image pyramid. Under the guidance of the matching results of the previous layer of the image pyramid, matching based on HOPC is performed on this layer of the pyramid, and the matching is carried out layer by layer until the rough matching results of the visible light-thermal infrared image pair are obtained.

6. The method for constructing a three-dimensional model of the building temperature field according to claim 3, characterized in that, The energy function optimization method based on thin plate spline function is as follows: Assume that there are N pairs of matching homologous points in the visible light image point cloud model and the thermal infrared image sequence, and their coordinates are represented by and respectively. Among them, , , and the thin plate spline mapping function from Y to X is represented by . The energy function of this transformation relationship is as follows: Among them, is a coefficient. The first term in the above energy function is the accuracy constraint, and the second term is the bending degree constraint. By adjusting the parameters, the bending energy of the thin plate is minimized.

7. The method for constructing a three-dimensional model of the building temperature field according to claim 2, characterized in that, The method for obtaining the relative pose is as follows: Calculate the three-dimensional position coordinates corresponding to the feature points of the visible light image in the visible light image point cloud model through one-way positioning; According to the three-dimensional ground point position coordinates and the thermal infrared image feature point coordinates, calculate the exterior orientation elements of the thermal infrared image relative to the visible light image point cloud through resection; Calculate the relative pose of each thermal infrared image relative to the visible light image point cloud model one by one, so as to complete the relative positioning of the thermal infrared image sequence relative to the visible light image point cloud model.

8. The method for constructing a three-dimensional model of the building temperature field according to claim 2, characterized in that, The method for determining the optimal texture of each three-dimensional ground point is as follows: taking each three-dimensional ground point as a basic unit, taking the minimization of the temperature difference between overlapping images as the objective function, calculating the arithmetic mean, geometric mean and harmonic mean of the temperatures of the three-dimensional ground points, comparing the average temperature value with the corresponding points in the thermal infrared image sequence to obtain the temperature difference of the overlapping images, and automatically selecting the method that can obtain the minimum temperature difference to complete texture mapping and realize the three-dimensional construction of the temperature field.

9. A three-dimensional model construction system for building temperature field based on a handheld dual camera, characterized in that, including: An image acquisition module for acquiring a visible light image sequence and a thermal infrared image sequence; A visible light image point cloud generation module for forming a visible light image point cloud model according to the visible light image sequence; A thermal infrared-visible light image pair matching module for matching the visible light image point cloud model with the thermal infrared image sequence; A temperature field texture mapping module for calculating the relative pose of the thermal infrared image sequence relative to the visible light image point cloud model according to the matching results of the visible light-thermal infrared image pair, removing the occluded and incorrect textures, and determining the optimal texture image of each three-dimensional ground point, realizing the fine optimization of the visible light-thermal infrared image pair by using the image geometric transformation based on thin plate spline function; realizing the optimal selection of the thermal infrared texture and the construction of the three-dimensional temperature field of the building by minimizing the temperature difference between overlapping images; The method for removing the occluded and incorrect textures is as follows: All three-dimensional ground points are assigned to different storage boxes according to the main direction of the building facade, and the projection center of each thermal infrared image is assigned to the corresponding storage box according to its main direction coordinates; Taking each thermal infrared image as a basic unit, the three-dimensional ground points corresponding to each unit are determined by searching the storage box where the thermal infrared image is located and the storage boxes around it. For each pixel in the thermal infrared image, only the three-dimensional ground point closest to it is defined as visible, and other three-dimensional ground points are removed.

Images