A three-dimensional reconstruction method based on infrared-visible light fusion image

Abstract

The application provides a three-dimensional reconstruction method based on infrared and visible light fusion images, which comprises the following steps: image fusion is performed on infrared images and visible light images taken under the same visual angle to output infrared and visible light fusion images; then feature point extraction and matching are performed on the infrared and visible light fusion images, and corresponding camera poses are calculated, and the camera poses are optimized by minimizing the re-projection error; then the infrared and visible light fusion images and the corresponding camera poses are taken as inputs, and the depth map and the pixel point plane normal vector map corresponding to each image are calculated through a block matching algorithm; finally, the infrared and visible light fusion images, the depth map and the pixel point plane normal vector map are combined in a depth map fusion module to output a three-dimensional model with infrared information and visible light texture information, and the three-dimensional model with infrared and visible light double-layer information can be reconstructed.

Description

Technical Field

[0001] This invention relates to the field of visual image processing, and more specifically, to a three-dimensional reconstruction method based on infrared-visible light fusion images. Background Technology

[0002] With the development of photoelectric detection technology and computer science, traditional two-dimensional visual images can no longer meet human needs for information acquisition, leading to the development of three-dimensional imaging technology. In recent years, three-dimensional reconstruction technology has been widely used in topographic mapping, facial recognition, VR / AR, medical detection, and autonomous driving. Currently, mainstream three-dimensional reconstruction methods extract and match features from visible light images, calculate camera pose using triangulation and reprojection error optimization, and then reconstruct a dense three-dimensional model using block matching algorithms. However, three-dimensional models based on visible light images contain limited information, especially when it's necessary to obtain the three-dimensional temperature distribution of the reconstructed target; visible light-based three-dimensional reconstruction methods cannot acquire temperature information. An intuitive approach is to use three-dimensional reconstruction methods based on infrared images; however, the sparse feature points in infrared images prevent the calculation of camera pose from yielding ideal results, thus hindering the acquisition of a dense infrared three-dimensional model.

[0003] In view of the above, this application is hereby submitted. Summary of the Invention

[0004] The purpose of this invention is to provide a 3D reconstruction method based on infrared and visible light fused images, which can acquire a target 3D model with both infrared and visible light information. This method fuses infrared and visible light images, ensuring that the input image for 3D reconstruction contains both infrared information and rich visible light texture information, thereby reconstructing a 3D model with both infrared and visible light dual-layer information.

[0005] The embodiments of the present invention are implemented as follows:

[0006] A 3D reconstruction method based on infrared-visible light fusion images includes the following steps:

[0007] S1: Image fusion of infrared and visible light images taken from the same viewpoint, including:

[0008] Detect the contour image of a visible light image and determine the first set of features in the contour image;

[0009] The main direction and feature modifiers of the visible light image are determined based on the first feature set, and the estimated scaling factor is calculated.

[0010] The feature modifiers of multiple sets of visible light images are matched bilaterally with the feature modifiers of a set of infrared images. The projection transformation matrix from infrared images to visible light images is calculated based on the scaling factor, and the infrared-visible light fused image is output.

[0011] S2: Extract and match feature points from the infrared and visible light fused images, sort the extracted feature points according to their correlation to generate a second feature set, and then sort them according to their correlation in the second feature set;

[0012] Based on the sorted sequence of the second feature set, the corresponding camera pose is calculated, and the camera pose is optimized by minimizing the reprojection error.

[0013] S3: Using the infrared-visible fused image and the corresponding camera pose as input, calculate the depth map and pixel plane normal map corresponding to each image through the block matching algorithm;

[0014] S4: Combines infrared and visible light fused images, depth maps, and pixel plane normal maps to output a 3D model with infrared and visible light texture information.

[0015] As an optional approach in this embodiment, in step S1 above, the visible light image and the infrared image are preprocessed separately before infrared and visible light image fusion. The preprocessing includes:

[0016] The acquired images are subjected to texture enhancement processing, the acquired image format is converted to grayscale, and its grayscale level is scaled to the range of [0, 255]. The vertical resolution of the visible light image is adjusted to be the same as that of the infrared image, and the horizontal resolution is adjusted according to the original image ratio of the visible light image.

[0017] As an optional approach in this embodiment, in step S1 above, the outline image of the visible light image is detected by the Canny edge detector. The first feature set is the set of corner points in the outline information of the visible light image. The corner points are detected by the curvature scale space corner detector based on local and global curvature as feature points.

[0018] As an optional approach in this embodiment, the contour direction angle of the corresponding corner point is calculated in the contour information based on the set of detected corner points, and the contour direction angle of each corresponding corner point is designated as the main direction.

[0019] As an optional approach in this embodiment, the scale-invariant feature transform descriptor of each corner point is modified to achieve partial intensity invariance of the multimodal image; and the scale-invariant feature transform descriptor of the original visible light grayscale image sampled at multiple scales and the original infrared grayscale image are extracted to estimate the scaling factor.

[0020] Then, the scale-invariant feature transform descriptors of multiple sets of visible light grayscale images are matched bilaterally with the scale-invariant feature transform descriptors of a set of infrared images, and the scaling factor is determined based on the majority voting algorithm.

[0021] A coarse-to-fine matching method was used for optimization, and the projection transformation matrix from infrared image to visible light image was calculated using the least squares method based on the optimization results.

[0022] As an optional method in this embodiment, the formula for calculating the contour direction angle is:

[0023]

[0024]

[0025]

[0026] Where φ is the profile direction angle. Let f be the feature point in the j-th contour set, and let its coordinates be denoted as . Represents the left distance in the j-th contour set The coordinates of the nearest feature point to the feature point. Represents the right distance in the j-th contour set The coordinates of the nearest feature point to the feature point.

[0027] As an optional approach in this embodiment, in step S2 above, the generated second feature set is the set of real-world coordinates of the infrared-visible fusion image, which is sorted according to the fitting degree of the coordinates, and then the camera pose is calculated and optimized; the fitting degree of the infrared-visible fusion image is the degree of overlap between the remapping points of the infrared-visible fusion image and the real-world coordinates in the original second feature set.

[0028] As an optional approach in this embodiment, in step S2 above, the normal viewpoint image of each camera during movement is tracked and acquired, the pose of each camera is calculated, and a pose set is generated, wherein each camera moves independently from the same coordinate position.

[0029] Each pose in the pose set is optimized to generate an optimized pose set; the pose calculation and optimization methods are as follows:

[0030] Based on the fitting degree of the infrared-visible fusion image, the coordinates of three-dimensional points are calculated using epipolar geometry theory; the camera pose is optimized by minimizing the reprojection error based on the initial three-dimensional point coordinates; the minimized reprojection error is represented by the distance between the position of the remapped point in the infrared-visible fusion image and its original position in the real-world coordinate set.

[0031] As an optional approach in this embodiment, the method for obtaining the depth map and normal vector map in step S4 above is as follows:

[0032] For each infrared-visible fused image, a depth map and a normal vector map are randomly initialized;

[0033] Each pixel is homogenized from the top left to the bottom right of the depth map and normal vector map to the adjacent viewpoint image;

[0034] Calculate the cost of the image patch corresponding to the pixel, iterate repeatedly, and output the depth map and normal vector map.

[0035] As an optional approach in this embodiment, it further includes performing a consistency check on adjacent depth maps. For each pixel in any depth map, the difference between its depth value and the depth value of the corresponding matching pixel in its neighboring image is calculated. Pixels with differences outside a preset value are considered invalid pixels that are occluded. Based on the invalid pixels, the depth values ​​and planes of the nearest valid pixels to its left and right are searched. After homography mapping, the normalized cross-correlation values ​​of the two windows are calculated, and the depth value and plane with the better normalized cross-correlation value are used for filling. Finally, median filtering is performed on each depth map to output the corresponding reliable depth map.

[0036] The beneficial effects of the embodiments of the present invention are:

[0037] The method provided by this invention can acquire a target three-dimensional model with infrared and visible light information. By fusing infrared and visible light images, the input image for three-dimensional reconstruction has both infrared information and rich visible light texture information, thereby reconstructing a three-dimensional model with dual infrared and visible light information. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A flowchart illustrating the steps of a three-dimensional reconstruction method for an infrared-visible light fused image provided in an embodiment of the present invention;

[0040] Figure 2 The infrared-visible light fusion process provided for embodiments of the present invention is an algorithm flowchart;

[0041] Figure 3 This is a flowchart of a camera pose algorithm provided in an embodiment of the present invention;

[0042] Figure 4The flowchart illustrates the algorithm for reconstructing a target 3D model as provided in this embodiment of the invention. Detailed Implementation

[0043] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0044] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0045] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0046] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be executed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0047] Those skilled in the art will understand that all or part of the steps in the above facts and methods can be implemented by a program instructing related hardware. The program or the program described therein can be stored in a computer-readable storage medium. When the program is executed, it includes the following steps: at this time, the corresponding method steps are introduced. The storage medium can be ROM / RAM, magnetic disk, optical disk, etc.

[0048] Example

[0049] Please see Figures 1-4 This embodiment provides a three-dimensional reconstruction method based on infrared-visible light fusion images, including the following steps:

[0050] Multiple sets of images from different perspectives were captured using an infrared thermal imager and an RGB camera;

[0051] Infrared and visible light images taken from the same viewpoint are fused to output a fused infrared and visible light image;

[0052] The infrared and visible light fused images are input into the structure of motion restoration (SFM) module for feature point extraction and matching, and then the corresponding camera pose is calculated. The camera pose is optimized by minimizing the reprojection error.

[0053] The infrared-visible fused image and the corresponding camera pose are input to the multi-view stereo geometry module (MVS). The depth map and pixel plane normal map corresponding to each image are calculated by the block matching algorithm. S5: The infrared-visible fused image, depth map and pixel plane normal map are combined in the depth map fusion module to output a 3D model with infrared information and visible light texture information.

[0054] The 3D reconstruction algorithm provided in this embodiment only requires two sets of images, visible light and infrared, as input. Then, based on the fused image, 3D reconstruction is performed to obtain a target 3D model that simultaneously has visible light texture information and infrared information.

[0055] In the image fusion process of infrared and visible light images in this embodiment, the following are included:

[0056] Image preprocessing: To correct all parameters of the proposed registration method, the input image needs to be preprocessed. First, the input image format is converted to grayscale; second, the grayscale levels are scaled to the range of [0, 255] to enhance image texture information; third, the vertical resolution of the visible image is adjusted to be the same as that of the infrared image, while the horizontal resolution is adjusted according to the original image ratio.

[0057] Feature point detection: The image's edge map is detected by the Canny edge detector. Corner points in the contour are detected by a curvature scale-space (CSS) corner detector based on local and global curvature as feature points. Feature points must be detected in the contour because the contour is a prerequisite for calculating the contour orientation angle (CAO).

[0058] Calculate the contour orientation angle (CAO): Assign the principal direction to the CAO of each feature point to achieve invariance to image rotation and perspective differences. The calculation method is as follows:

[0059]

[0060]

[0061]

[0062] Where φ is the profile direction angle. Let f be the feature point in the j-th contour set, and let its coordinates be denoted as . Represents the left distance in the j-th contour set The coordinates of the nearest feature point to the feature point. Represents the right distance in the j-th contour set The coordinates of the nearest feature point to the feature point.

[0063] Next, the modified Scale Invariant Feature Transform (SIFT) descriptors are extracted: SIFT is a robust descriptor commonly used in point-based methods. However, the SIFT descriptor for each point should be modified algorithmically to achieve partial intensity invariance for the multimodal image. Furthermore, in addition to extracting the SIFT descriptor from the original infrared grayscale image, it is also necessary to extract the SIFT descriptor from the visible light grayscale image sampled at multiple scales to estimate the scaling factor ξ.

[0064] SIFT matching: In this embodiment, three sets of SIFTs from visible light grayscale images sampled at multiple scales and one set of SIFTs from infrared images are selected and matched using bilateral matching. Then, a scaling factor is determined based on the number of matching pairs using a majority voting algorithm.

[0065] Then, coarse-to-fine (C2F) matching is used to remove incorrect matches and optimize the positions of correct matches, providing two sets of accurate point coordinates. Finally, the projection transformation matrices of the infrared and visible light images are calculated by least squares on the C2F matching results. The infrared image transformed by the projection transformation matrix and the visible light image are each assigned certain weights and fused to obtain the final fused image.

[0066] In this embodiment, feature points need to be extracted and matched from multiple fused images from different perspectives. The extracted feature points are sorted according to their relevance to generate a second feature set, and then further sorted within the second feature set. Based on the sorting sequence of the second feature set, the corresponding camera pose is calculated, and the camera pose is optimized by minimizing the reprojection error. The RANSAC algorithm is used to remove abnormal matches. Among the matched image pairs, the image pair with the most matching feature point pairs is selected as the initial incremental reconstruction benchmark. The coordinates of the feature points in the benchmark image pair in the real world are calculated using epipolar geometry theory, and the coordinates are sorted according to the fitting degree. Based on this, the PnP (Perspective-n-Point) algorithm is used to calculate the corresponding camera pose of the image. The pose information includes rotation and translation matrices. Specifically, the normal perspective image of each camera movement is tracked and acquired, the pose of each camera is calculated, a pose set is generated, and each pose in the pose set is optimized to generate an optimized pose set. Minimizing the reprojection error is represented by the distance between the position of the remapped point in the infrared-visible fused image in the image and its original position in the real-world coordinate set. New images are added, and their feature points are calculated to have real-world coordinates. The corresponding poses are then determined using the Projection-Negative Projection (PnP) algorithm. Finally, the camera poses for each image are optimized by minimizing the reprojection error. Specifically, image feature points are mapped to 3D space using pose information and then remapped back to the 2D image. The remapped points in the image do not overlap with the original feature points. The process of adjusting the pose to minimize the reprojection error is the pose optimization process.

[0067] In this embodiment, please refer again. Figure 3 The specific steps of step S3, which outputs the depth map and the pixel plane normal map, are as follows:

[0068] For each fused image, a depth map is randomly initialized, where the depth value of each pixel is completely random. Furthermore, each pixel in the depth map is assigned a random plane, which is defined by its normal vector. The process involves an iterative process. Each pixel in the fused image undergoes homography with adjacent fused images using image pose information, its corresponding initialized depth value, and a random plane. A square window of appropriate size is taken centered on the pixel, and the normalized cross-correlation value of the window image patch in both images is calculated. Furthermore, homography is performed using the depth values ​​and random planes of adjacent pixels, and the normalized cross-correlation value is calculated. If the normalized cross-correlation value is better, the depth values ​​and planes of the current pixel are replaced with those of the adjacent pixels. Simultaneously, the better depth values ​​and planes are assigned to the pixels homography of this pixel in the adjacent images. This propagation process of depth values ​​and planes is performed for each pixel in each fused image from top left to bottom right. Additionally, during the propagation process, another depth value and plane are randomly selected for each pixel. After homography, the normalized cross-correlation value is calculated. If the normalized cross-correlation value is better, the current depth values ​​and planes are replaced with the newer, better one. This completes one iteration of spatial propagation, view propagation, and planar refinement. Three to five iterations yield a relatively stable depth map. Next, a consistency check is performed on the depth maps of adjacent images. For each pixel, the difference between its depth value and the depth value of its corresponding matching pixel in its neighboring image is calculated. If the difference is within an acceptable range, the current depth value is retained; otherwise, it is considered an occluded invalid pixel. For these occluded points, the depth values ​​and planes of the nearest valid pixels to their left and right are searched. After homography mapping, the normalized cross-correlation value of the two windows is calculated, and the depth value and plane with the better normalized cross-correlation value are used for filling. Finally, median filtering is applied to each depth map to obtain the corresponding reliable depth map.

[0069] In this embodiment, the specific steps of step S5, which outputs the 3D model, are as follows:

[0070] For each image, each pixel is mapped to 3D space based on its depth map and normal vector map. For the same point in 3D space corresponding to multiple images, it is only retained if the difference between that point and each depth map and normal vector map is less than a certain value. The final retained depth and normal vector values ​​are the average values ​​of that point across all depth maps and normal vector maps. Mapping processing is performed on each pixel of the infrared-visible fused images from each viewpoint to obtain a 3D model of the target with infrared and visible light textures.

[0071] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of protection of the invention. It should be noted that the structures or components illustrated in the accompanying drawings are not necessarily drawn to scale, and descriptions of well-known components, processing techniques, and processes have been omitted to avoid unnecessarily limiting the invention.

Claims

1. A three-dimensional reconstruction method based on infrared-visible light fusion images, characterized in that, Including the following steps: S1: Image fusion of infrared and visible light images taken from the same viewpoint, including: Detect the contour image of a visible light image and determine the first set of features in the contour image; The main direction and feature modifiers of the visible light image are determined based on the first feature set, and the estimated scaling factor is calculated. The feature modifiers of multiple sets of visible light images are matched bilaterally with the feature modifiers of a set of infrared images. The projection transformation matrix from infrared images to visible light images is calculated based on the scaling factor, and the infrared-visible light fused image is output. S2: Extract and match feature points from the infrared-visible fusion image, sort the extracted feature points according to their correlation to generate a second feature set, and sort them according to their correlation in the second feature set; Based on the sorted sequence of the second feature set, the corresponding camera pose is calculated, and the camera pose is optimized by minimizing the reprojection error. S3: Using the infrared-visible fused image and the corresponding camera pose as input, calculate the depth map and pixel plane normal map corresponding to each image through the block matching algorithm; S4: Combines infrared and visible light fused images, depth maps, and pixel plane normal maps to output a 3D model with infrared and visible light texture information.

2. The three-dimensional reconstruction method based on infrared-visible light fusion image according to claim 1, characterized in that, In step S1 above, before the infrared and visible light images are fused, the visible light image and the infrared image are preprocessed separately. The preprocessing includes: The acquired images are texture-enhanced, converted to grayscale, and their grayscale levels are scaled to the range of [0, 255]. The vertical resolution of the visible light image is adjusted to be the same as that of the infrared image, and the horizontal resolution is adjusted according to the original image ratio of the visible light image.

3. The three-dimensional reconstruction method based on infrared-visible light fusion image according to claim 2, characterized in that, In step S1 above, the outline image of the visible light image is detected by the Canny edge detector. The first feature set is the set of corner points in the outline information of the visible light image. The corner points are detected by the curvature scale space corner detector based on local and global curvature as feature points.

4. The three-dimensional reconstruction method based on infrared-visible light fusion image according to claim 3, characterized in that, Based on the set of detected corner points, the contour direction angle of the corresponding corner point is calculated in the contour information, and the contour direction angle of each corresponding corner point is designated as the main direction.

5. A three-dimensional reconstruction method based on infrared-visible light fusion images according to claim 4, characterized in that, To achieve partial intensity invariance of the multimodal image, the scale-invariant feature transform descriptor of the original visible light grayscale image and the original infrared grayscale image are extracted from the multiscale sampled image to estimate the scaling factor. Then, the scale-invariant feature transform descriptors of multiple sets of visible light grayscale images are matched bilaterally with the scale-invariant feature transform descriptors of a set of infrared images, and the scaling factor is determined based on the majority voting algorithm. A coarse-to-fine matching method was used for optimization, and the projection transformation matrix from infrared image to visible light image was calculated using the least squares method based on the optimization results.

6. The three-dimensional reconstruction method based on infrared-visible light fusion image according to claim 4, characterized in that, The formula for calculating the contour direction angle is: ； in, The profile direction angle, Let f be the feature point in the j-th contour set, and let its coordinates be ( ), ( ) represents the left distance in the j-th contour set. The coordinates of the nearest feature point to the feature point, ( ) represents the distance to the right in the j-th contour set. The coordinates of the nearest feature point to the feature point.

7. A three-dimensional reconstruction method based on infrared-visible light fusion images according to claim 2, characterized in that, In step S2 above, the generated second feature set is the set of real-world coordinates of the infrared-visible fusion image. The coordinates are sorted according to the fitting degree, and then the camera pose is calculated and optimized. The fitting degree is the degree of overlap between the remapped points of the infrared-visible fusion image and the real-world coordinates in the original second feature set.

8. A three-dimensional reconstruction method based on infrared-visible light fusion images according to claim 5, characterized in that, In step S2 above, the normal viewpoint image of each camera during movement is tracked and acquired, the pose of each camera is calculated, and a pose set is generated, wherein each camera moves independently from the same coordinate position. Each pose in the pose set is optimized to generate an optimized pose set; the pose calculation and optimization methods are as follows: Based on the fitting degree of the infrared-visible fusion image, the coordinates of three-dimensional spatial points are calculated using epipolar geometry theory; the camera pose is optimized by minimizing the reprojection error according to the initial three-dimensional spatial point coordinates; the minimized reprojection error is a representation of the distance between the position of the remapped point in the infrared-visible fusion image in the image and the original position in the real-world coordinate set.

9. A three-dimensional reconstruction method based on infrared-visible light fusion images according to claim 1, characterized in that, In step S4 above, the depth map and normal vector map are obtained as follows: For each infrared-visible fused image, a depth map and a normal vector map are randomly initialized; Each pixel is homogenized from the top left to the bottom right of the depth map and normal vector map to the adjacent viewpoint image; Calculate the cost of the image patch corresponding to the pixel, iterate repeatedly, and output the depth map and normal vector map.

10. A three-dimensional reconstruction method based on infrared-visible light fusion images according to claim 9, characterized in that, The method also includes performing a consistency check on adjacent depth maps. For each pixel in any depth map, the difference between its depth value and the depth value of the corresponding matching pixel in its neighboring image is calculated. Pixels with differences outside a preset value are considered invalid pixels that are occluded. Based on the invalid pixels, the depth values ​​and planes of the nearest valid pixels to its left and right are searched. After homography mapping, the normalized cross-correlation value of the two windows is calculated, and the depth value and plane with the better normalized cross-correlation value are used for filling. Finally, median filtering is performed on each depth map to output the corresponding reliable depth map.

Images