Target three-dimensional temperature field establishment method and device based on infrared and visible light images

Abstract

The invention provides a target three-dimensional temperature field establishment method and device based on infrared and visible light images. The method comprises the following steps: calibrating a dual-light acquisition platform to obtain a corrected visible light image and a corrected infrared image; registering and converting the corrected visible light images in the same double-light group in the double-light acquisition platform to obtain a visible light type thermal image; performing different-source registration on the visible light type thermal image and the corrected infrared image in the same dual-light pair in the dual-light acquisition platform to obtain a registered infrared image and a registered visible light type thermal image; performing stereo matching and data reconstruction according to the registered infrared image and the registered visible light thermal image to obtain matching points; and performing parallax calculation according to the matching points, and constructing a target single-view-field three-dimensional temperature field according to a parallax calculation result and calibration parameters of the dual-light acquisition platform.

Description

Technical Field

[0001] The present application relates to the field of vision and image processing technology, and in particular to a method and device for establishing a target three-dimensional temperature field based on infrared and visible light images. Background Art

[0002] The 3D reconstruction technology that constructs the 3D geometric information of spatial targets based on 2D image information is widely used in the fields of autonomous driving, medical imaging, design and manufacturing, audio-visual entertainment, and digital cultural heritage protection. At present, 3D reconstruction technologies mainly include visible light stereo vision, structured light scanning, and lidar scanning. However, the above 3D reconstruction technologies only construct the 3D geometric information of the target, and lack other important information sources of the target, such as temperature information. The realization of 3D temperature field reconstruction based on infrared equipment is of great significance for target recognition and monitoring.

[0003] Due to the limitations of the imaging mechanism, infrared equipment images have few texture features, and weak texture features make it difficult to construct dense point clouds, resulting in problems such as difficult reconstruction and low accuracy. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a method and device for establishing a target three-dimensional temperature field based on infrared and visible light images, aiming to solve at least one of the above-mentioned problems.

[0005] To achieve the above objectives, the present application provides a method for establishing a target three-dimensional temperature field based on infrared and visible light images, comprising:

[0006] Calibrate the dual-light acquisition platform to obtain a corrected visible light image and a corrected infrared image;

[0007] Registering and converting the corrected visible light images in the same dual-light group in the dual-light acquisition platform to obtain a visible light thermal image;

[0008] Performing heterogeneous registration on the visible light thermal image and the corrected infrared image in the same dual light pair in the dual light acquisition platform to obtain a registered infrared image and a registered visible light thermal image;

[0009] Perform stereo matching and data reconstruction according to the registered infrared image and the registered visible light thermal image to obtain matching points;

[0010] Parallax calculation is performed according to the matching points, and a target single-view three-dimensional temperature field is constructed according to the parallax calculation result and the calibration parameters of the dual-light acquisition platform.

[0011] In some embodiments, the method further comprises:

[0012] Build a dual-light collection platform; among them,

[0013] The dual-light collection platform includes a plurality of dual-light groups, which are distributed around a target and whose total field of view covers the target. There are overlapping field of view areas between the dual-light groups.

[0014] In some embodiments, the calibrating the dual-light acquisition platform to obtain a corrected visible light image and a corrected infrared image includes:

[0015] Directing the infrared device and the visible light device of each dual light pair of the dual light collection platform toward the composite target;

[0016] Collecting original captured images of the infrared device and the visible light device of each dual light pair based on the composite target;

[0017] The original captured image is calibrated and distortion corrected to obtain a corrected visible light image and a corrected infrared image of each of the dual light pairs.

[0018] In some embodiments, the registering and converting the corrected visible light images in the same dual-light group in the dual-light acquisition platform to obtain a visible light thermal image includes:

[0019] Registering the corrected visible light images in the same dual-light group in the dual-light acquisition platform to obtain a registered visible light image;

[0020] The registered visible light image is modally converted based on a preset image modality conversion algorithm to obtain a visible light thermal image.

[0021] In some embodiments, performing stereo matching and data reconstruction according to the registered infrared image and the registered visible light thermal image to obtain matching points includes:

[0022] Reconstructing the registered infrared image and the registered visible light thermal image to obtain a four-channel data image;

[0023] Based on the epipolar constraint, epipolar correction is performed on the four-channel data image of the visible light device in the dual-light group to obtain an epipolar constrained image;

[0024] Segmenting the registered infrared image to obtain an infrared image segmentation result;

[0025] Stereo matching is performed on the epipolar constraint image according to the infrared image segmentation result to obtain matching points.

[0026] In some embodiments, reconstructing the registered infrared image and the registered visible light thermal image to obtain a four-channel data image includes:

[0027] acquiring temperature data according to the registered infrared image;

[0028] Acquiring RGB data according to the registered visible light thermal image;

[0029] Data reconstruction is performed according to the temperature data and the RGB data to obtain a four-channel data image of RGB-T data.

[0030] In some embodiments, performing stereo matching on the epipolar constraint image according to the infrared image segmentation result to obtain matching points includes:

[0031] Construct an initial matching area centered on the matching pixel;

[0032] Screening the pixel points in the initial matching area based on the infrared image segmentation result to generate an adaptive dynamic matching area;

[0033] The temperature-constrained regional stereo matching algorithm performs row-by-row stereo matching on the epipolar-constrained image according to the adaptive dynamic matching region to obtain matching points.

[0034] In some embodiments, performing parallax calculation according to the matching points, and constructing a target single-view three-dimensional temperature field according to the parallax calculation result and the calibration parameters of the dual-light acquisition platform, includes:

[0035] Calculate the disparity based on the matching points according to the RGB-T data of the epipolar constrained image to obtain an initial disparity image;

[0036] Performing interpolation optimization and outlier correction on the initial disparity image to obtain a target disparity image;

[0037] A target single-view three-dimensional temperature field is constructed according to the target parallax image and the calibration parameters of the dual-light acquisition platform.

[0038] In some embodiments, the method further comprises:

[0039] Registering the corrected visible light images in adjacent dual-light groups in the dual-light acquisition platform and calculating the posture relationship between the adjacent dual-light groups to obtain rotation information and translation information;

[0040] Obtain the three-dimensional temperature information corresponding to the target single-view three-dimensional temperature field of each dual-light group;

[0041] The three-dimensional temperature information of the plurality of dual-light groups is spliced ​​based on the rotation information and the translation information to construct a target multi-view three-dimensional temperature field.

[0042] In addition, to achieve the above purpose, the embodiment of the present application also proposes a target three-dimensional temperature field establishment device based on infrared and visible light images, including:

[0043] A calibration and correction module, configured to calibrate the dual-light acquisition platform to obtain a corrected visible light image and a corrected infrared image;

[0044] A registration and conversion module, configured to register and convert the corrected visible light images in the same dual-light group in the dual-light acquisition platform to obtain a visible light-like thermal image;

[0045] A heterogeneous registration module is configured to perform heterogeneous registration on the visible light thermal image and the corrected infrared image in the same dual light pair in the dual light acquisition platform to obtain a registered infrared image and a registered visible light thermal image;

[0046] A matching and reconstruction module, configured to perform stereo matching and data reconstruction according to the registered infrared image and the registered visible light thermal image to obtain matching points;

[0047] The calculation and construction module is configured to perform parallax calculation according to the matching points, and to construct a target single-view three-dimensional temperature field according to the parallax calculation result and the calibration parameters of the dual-light acquisition platform.

[0048] The embodiment of the present application provides a method for establishing a target three-dimensional temperature field based on infrared and visible light images, including: calibrating a dual-light acquisition platform to obtain a corrected visible light image and a corrected infrared image; aligning and converting the corrected visible light image in the same dual-light group in the dual-light acquisition platform to obtain a visible light thermal image; performing heterogeneous alignment on the visible light thermal image and the corrected infrared image in the same dual-light pair in the dual-light acquisition platform to obtain a registered infrared image and a registered visible light thermal image; performing stereo matching and data reconstruction based on the registered infrared image and the registered visible light thermal image to obtain matching points; performing parallax calculation based on the matching points, and constructing a target single-field three-dimensional temperature field based on the parallax calculation result and the calibration parameters of the dual-light acquisition platform. In the embodiment of the present application, the images of the infrared and visible light bands of the dual-light acquisition platform are fused to achieve synchronous correction and calibration of the infrared and visible light images, reduce the modal conflict of heterogeneous images through modal conversion and registration, improve the registration accuracy of heterogeneous images, construct the registered data for stereo matching and parallax calculation, and realize the reconstruction of the target single-field three-dimensional temperature field, thus overcoming the shortcomings of the weak texture features of a single infrared device, thereby improving the reconstruction accuracy of the target three-dimensional temperature field. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of an embodiment of a method for establishing a target three-dimensional temperature field based on infrared and visible light images of the present application;

[0050] Figure 2 It is a schematic diagram of a specific process of establishing a three-dimensional temperature field in a single field of view involved in the embodiment of the present application;

[0051] Figure 3 It is a schematic diagram of a dual-light collection platform involved in the embodiment of the present application;

[0052] Figure 4 It is a schematic diagram of a composite target involved in the embodiment of the present application;

[0053] Figure 5 It is a schematic diagram of a temperature constraint area stereo matching algorithm involved in an embodiment of the present application;

[0054] Figure 6 This is a structural block diagram of an embodiment of a device for establishing a target three-dimensional temperature field based on infrared and visible light images of the present application;

[0055] Figure 7 It is a structural diagram of an electronic device in a hardware operating environment involved in an embodiment of the present application.

[0056] The realization of the purpose, functional features and advantages of the embodiments of the present application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of them. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] It should be noted that all directional indications in the embodiments of the present application (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0059] In addition, the descriptions of "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by this application. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] The present application proposes a method and device for establishing a target three-dimensional temperature field based on infrared and visible light images.

[0061] The present application embodiment provides a method for establishing a target three-dimensional temperature field based on infrared and visible light images. Figure 1 , Figure 1 This is a flow chart of an embodiment of a method for establishing a target three-dimensional temperature field based on infrared and visible light images of the present application.

[0062] like Figure 1 As shown, the method comprises:

[0063] Step S100: calibrating the dual-light acquisition platform to obtain a corrected visible light image and a corrected infrared image;

[0064] Step S200: registering and converting the corrected visible light images in the same dual-light group in the dual-light acquisition platform to obtain a visible light thermal image;

[0065] Step S300: performing heterogeneous registration on the visible light thermal image and the corrected infrared image in the same dual light pair in the dual light acquisition platform to obtain a registered infrared image and a registered visible light thermal image;

[0066] Step S400: performing stereo matching and data reconstruction according to the registered infrared image and the registered visible light thermal image to obtain matching points;

[0067] Step S500: performing parallax calculation according to the matching points, and constructing a target single-view three-dimensional temperature field according to the parallax calculation result and the calibration parameters of the dual-light acquisition platform.

[0068] It should be noted that the execution subject in this embodiment may be an electronic device, which may be a computer device with data processing functions, or other devices that can achieve the same or similar functions. This embodiment does not limit this. In this embodiment, a computer device is used as an example for explanation.

[0069] It is understandable that, combined with Figure 1 and Figure 2 The specific process of the method for establishing a target three-dimensional temperature field based on infrared and visible light images in an embodiment of the present application is described.

[0070] In one embodiment, the method further includes: building a dual-light collection platform; wherein the dual-light collection platform includes a plurality of dual-light groups, the dual-light groups are distributed around the target and the total field of view of the dual-light groups covers the target, and there are overlapping field of view areas between the dual-light groups.

[0071] It should be noted that the embodiments of the present application are based on Figure 3The dual-light collection platform shown is used as an example for explanation.

[0072] Specifically, the embodiment of the present application is constructed as follows Figure 3 The dual-light collection platform shown in the figure includes a plurality of dual-light groups (eg Figure 3 As shown in the figure, the dual light groups 1, 2, 3, ..., n are arranged around the target, and the total field of view (e.g. Figure 3 The total field of view shown) covers the complete structure of the target.

[0073] In one embodiment, if Figure 3 As shown, each of the dual-light groups includes two dual-light pairs, and each of the dual-light pairs includes an infrared device, a visible light device, and a dichroic mirror; wherein the visible light device in each of the dual-light groups is horizontally assembled with the main optical axis; the infrared device and the visible light device of each of the dual-light pairs are assembled at a vertical angle; the dichroic mirror is used to transmit the light of the visible light band of the target and reflect the light of the infrared band of the target.

[0074] In one embodiment, the baseline distance of each of the dual-light groups in the dual-light collection platform is consistent, wherein a line connecting the optical centers of the visible light devices in each of the dual-light groups is the baseline distance.

[0075] Specifically, each dual-light group includes two dual-light pairs (eg Figure 3 As shown, the dual light pair 1 and dual light pair 2 are composed, each dual light pair includes an infrared camera (such as an infrared thermal imager) and a visible light device (such as a visible light camera), the infrared thermal imager and the visible light device are assembled at a vertical angle and are equipped with a dichroic mirror.

[0076] For example, Figure 3 The infrared camera 1 and the visible light camera 1 in the dual light pair 1 of the dual light group 1 are assembled at a vertical angle and equipped with a dichroic mirror, so that the field of view of the infrared camera and the visible light camera are close to the same, thereby reducing the difference in the field of view of the infrared and visible light devices and reducing the difficulty of subsequent heterogeneous image registration. The dichroic mirror is designed to transmit the light of the target visible light band and reflect the light of the target infrared band, so that the field of view of the visible light device and the infrared device are close to the same, that is, the target photographed by the two is the same.

[0077] Specifically, in the embodiment of the present application, the visible light device (eg Figure 3 As shown, the main optical axes of visible light camera 1 and visible light camera 2 in dual light group 1 are assembled horizontally, and the appropriate baseline distance is set according to the field of view requirements and the consistency of the baseline distance of each dual light group in the dual light acquisition platform is ensured. The above operations reduce the difficulty of subsequent alignment.

[0078] In one embodiment, a dual-light acquisition platform is calibrated to obtain a corrected visible light image and a corrected infrared image, including: pointing the infrared device and the visible light device of each dual-light pair of the dual-light acquisition platform toward a composite target; acquiring original captured images of the infrared device and the visible light device of each dual-light pair based on the composite target; and calibrating and correcting the distortion of the original captured images to obtain a corrected visible light image and a corrected infrared image of each dual-light pair.

[0079] Specifically, taking a dual-light pair as a unit, the binocular cameras (infrared camera and visible light camera) in the dual-light acquisition platform are subjected to distortion correction through a composite target.

[0080] In one embodiment, if Figure 4 As shown, the composite target includes:

[0081] Temperature controlled heating plate;

[0082] A target panel covers the upper surface of the temperature-controlled heating plate; wherein the upper surface of the target panel is divided into M*N calibration areas arranged in an array of M rows and N columns; the M*N calibration areas include first areas and second areas alternately arranged in both the row direction and the column direction, and the first areas and the second areas have different emissivity;

[0083] One of M and N is an odd number, and the other is an even number.

[0084] In one embodiment, a difference between the emissivity of the first region and the emissivity of the second region is greater than 0.6.

[0085] Specifically, the embodiment of the present application is designed as follows Figure 4 The composite target shown includes a target panel, a temperature-controlled heating plate, and a universal adjustment bracket. The material of the target panel includes but is not limited to aviation aluminum sheet. Photolithography (such as laser engraving) technology is used to engrave a chessboard on the surface of the target panel.

[0086] Since binocular camera calibration is required, one of the two numbers of rows and columns of the chessboard designed in the embodiment of the present application is an odd number and the other is an even number. The size is adjusted according to the parameters of the camera to be calibrated and the measurement distance to ensure that the chessboard occupies more than 1 / 4 of the frame in the field of view to be calibrated. A temperature-controlled heating plate is attached to the back of the target panel. The heating temperature can be adjusted between 0°C and 150°C, and the uniformity of the heating temperature is guaranteed. The temperature difference between the highest and lowest temperatures of the target panel can be less than 2°C, and the temperature difference in a single square area (for example, a black square engraved area or a white square unengraved area) can be less than 1°C. The embodiment of the present application uses the above composite target design to ensure that the composite target can be clearly imaged in both the infrared band and the visible light band.

[0087] Specifically, Figure 2 As shown, Step 1: distortion correction. Dual-light device (infrared device and visible light device) distortion correction, use the dual-light device to be corrected to shoot the composite target in the embodiment of the present application, and adjust the position and angle of the composite target. Exemplarily, the dual-light device to be corrected shoots N groups (for example, 30 groups) of image pairs, and uses a preset calibration method (for example, Zhang Zhengyou calibration method) for calibration, obtains the intrinsic parameter matrix, extrinsic parameter matrix and distortion parameters of the infrared camera and visible light camera in the dual-light device, and obtains the corrected visible light image and corrected infrared image. Among them, the distortion includes radial distortion and tangential distortion, and the corrected coordinates can be calculated by the radial distortion parameters.

[0088]

[0089] The corrected coordinates can be calculated from the tangential distortion parameters

[0090]

[0091] Exemplarily, according to Zhang Zhengyou's calibration method, the dual-light device to be calibrated synchronously shoots the composite target in the embodiment of the present application at different angles and then collects 30 sets of image pairs. The 30 sets of image pairs collected by the visible light camera and the infrared camera are calibrated, and the internal and external parameter matrices and distortion coefficients of the visible light camera and the infrared camera are obtained respectively. The original image is dedistorted according to the internal and external parameter matrices and distortion coefficients of the visible light camera and the infrared camera to obtain the dedistorted image, i.e., the corrected visible light image and the corrected infrared image.

[0092] In one embodiment, the corrected visible light images in the same dual-light group in the dual-light acquisition platform are registered and converted to obtain a visible light thermal image; the visible light thermal image and the corrected infrared image in the same dual-light pair in the dual-light acquisition platform are heterogeneously registered to obtain a registered infrared image and a registered visible light thermal image. Wherein, the corrected visible light images in the same dual-light group in the dual-light acquisition platform are registered and converted to obtain a visible light thermal image, including: registering the corrected visible light images in the same dual-light group in the dual-light acquisition platform to obtain a registered visible light image; performing modal conversion on the registered visible light image based on a preset image modal conversion algorithm to obtain a visible light thermal image.

[0093] Specifically, Figure 2As shown, Step 2: Visible light image registration, register the visible light image corrected by the dual light group obtained in Step 1 to obtain a registered visible light image. The registration methods used include but are not limited to matching algorithms based on feature points, matching algorithms based on regional information, and registration algorithms based on deep learning. The affine transformation matrix obtained by the registration is used to construct the scaling, rotation and translation transformation relationship between the two visible light devices:

[0094] [x vis ',y vis ']=[ab;cd]*[x vis ;y vis ]+[e;f]

[0095] Among them, [x vis ,y vis ] is the pixel position of the input visible light image; [x vis ',y vis '] is the pixel position of the output visible light image after the registration of the visible light image; [ab; cd] is the linear transformation matrix; [e; f] is the translation vector.

[0096] Specifically, Figure 2 As shown in Figure 3, Step 3: Modal conversion, visible light image modal conversion. The grayscale value of the infrared image is a mapping of the target's relative temperature, and the grayscale value of the visible light image is a mapping of the target's relative brightness. For example, the sky in the infrared image appears darker due to the lower temperature, while the sky in the visible light image appears brighter due to the higher brightness. In addition to the conflict of grayscale inversion, there is also a conflict in texture information. The conflict between the two modal images, infrared image and visible light image, increases the difficulty of registration.

[0097] In the embodiment of the present application, the registered visible light image can be modally converted based on a preset image modality conversion algorithm to convert the registered visible light image into a visible light-like thermal image (visible light-like thermal image). The preset image modality conversion algorithm includes but is not limited to Rechild image migration algorithm, Pix2Pix (Gan-based image style migration model), Pix2PixHD (Cycle Gan-based image generation model), generative adversarial network Gan, StyleGan and other deep learning algorithms. In the embodiment of the present application, the conflict between the infrared image and the visible light image modality is reduced by modality conversion, and the grayscale mean and variance of the visible light image are converted to be approximately consistent with the infrared image, further reducing the difficulty of registration.

[0098] Specifically, Figure 2As shown, Step 4: heterogeneous registration, registration of visible light and infrared images. The visible light image and infrared image in the dual light alignment are registered, where the infrared image is the corrected infrared image after the distortion correction processing in Step 1, and the visible light image is the visible light-like thermal image (visible light-like thermal image) after processing in Step 1, Step 2 and Step 3. The registration algorithms used in heterogeneous registration include but are not limited to matching algorithms based on feature points, matching algorithms based on regional information, and matching algorithms based on deep learning. Among them, the affine transformation matrix obtained by the registration can be used to construct the scaling, rotation and translation transformation relationship between the infrared device and the visible light device:

[0099] [x vis ”,y vis ”]=[AB;CD]*[x vis ';y vis ']+[E;F]

[0100] Among them, [x vis ',y vis '] is the pixel position of the visible light image after registration of the input visible light image before heterogeneous registration; [x vis ”,y vis ”] is the pixel position of the visible light image output after heterogeneous registration; [AB; CD] is the linear transformation matrix, and [E; F] is the translation vector.

[0101] In one embodiment, stereo matching and data reconstruction are performed on the registered infrared image and the registered visible light thermal image to obtain matching points, including: reconstructing data on the registered infrared image and the registered visible light thermal image to obtain a four-channel data image; performing epipolar correction on the four-channel data image of the visible light device in the dual-light group based on epipolar constraints to obtain an epipolar constrained image; segmenting the registered infrared image to obtain an infrared image segmentation result; and stereo matching the epipolar constrained image based on the infrared image segmentation result to obtain matching points.

[0102] In one embodiment, data is reconstructed on the registered infrared image and the registered visible light thermal image to obtain a four-channel data image, including: acquiring temperature data based on the registered infrared image; acquiring RGB data based on the registered visible light thermal image; and reconstructing data based on the temperature data and the RGB data to obtain a four-channel data image of RGB-T data.

[0103] Specifically, Figure 2As shown, Step 5: Data reconstruction. Data reconstruction is performed based on the infrared image (registered infrared image) and the visible light-like thermal image (registered visible light-like thermal image) registered in Step 4. The registered infrared image and the registered visible light-like thermal image have the same resolution and correspond to each other pixel by pixel. The temperature information corresponding to the pixel points in the infrared image is stored as the fourth channel data to form RGB-T data. It should be noted that the temperature data T is combined with the RGB data to obtain the RGB-T data.

[0104] Specifically, Figure 2 As shown, Step 6: epipolar correction. Affected by installation and assembly, the two visible light cameras in the dual-light group cannot ensure that the optical axes are completely on the same horizontal line. In order to reduce the complexity of registration and increase the accuracy of registration, in the embodiment of the present application, epipolar correction is performed on the visible light devices in the dual-light group based on epipolar constraints, and the global retrieval of feature points is converted into peer retrieval. The image data used is RGB-T data after Step 1 distortion correction, Step 2 registration, Step 3 modal conversion, Step 4 heterogeneous registration and Step 5 data reconstruction.

[0105] It should be noted that in the registration process, it is necessary to find the same features in the two images and perform a global search in the entire frame. In the embodiment of the present application, through epipolar correction, the horizontal registration of the two images only requires a search within each row, which greatly reduces the difficulty and complexity of the registration.

[0106] In one embodiment, stereo matching is performed on the epipolar constrained image according to the infrared image segmentation result to obtain matching points, including: constructing an initial matching area centered on matching pixels; screening pixel points within the initial matching area based on the infrared image segmentation result to generate an adaptive dynamic matching area; and performing row-by-row stereo matching on the epipolar constrained image according to the adaptive dynamic matching area using a regional stereo matching algorithm based on temperature constraints to obtain matching points.

[0107] In one embodiment, the pixel points in the initial matching area are screened based on the infrared image segmentation result to generate an adaptive dynamic matching area, including: determining the background area and the target area according to the infrared image segmentation result; setting the initial matching area to an n*n rectangular area; screening and judging the area to which each pixel point in the initial matching area belongs, retaining the pixel points in the same area as the matching pixel and removing the pixel points that are not in the same area as the matching pixel, to generate an adaptive dynamic matching area; wherein the adaptive dynamic matching area uses the object contour of the infrared image segmentation result as the edge.

[0108] Specifically, Figure 2As shown, Step 7: Stereo matching. In order to better construct the target single-view three-dimensional temperature field, the embodiment of the present application proposes a regional stereo matching algorithm using temperature constraints. The infrared image data after Step 4 registration is segmented, and the segmentation methods used include but are not limited to segmentation methods based on deep learning such as threshold, region and edge detection. Figure 5 As shown, the registered infrared image is segmented into background area and target area (such as Figure 5 The circular area of ​​target 1 and the polygonal area of ​​target 2 are shown in the figure). Perform stereo matching line by line for the RGB-T data generated in Step 6, and build a matching area with the matching pixel as the center. For example, set the initial matching area to an n*n rectangular area, determine the segmentation area where the matching pixel is located, determine the area to which the pixels in the matching area belong, retain the pixels in the same area as the matching pixel, and remove the pixels in the different area from the matching pixel, and generate an adaptive dynamic matching area with the contour of the infrared image segmentation object as the edge.

[0109] In one embodiment, a regional stereo matching algorithm based on temperature constraints performs row-by-row stereo matching on the epipolar constraint image according to the adaptive dynamic matching area to obtain matching points, including: calculating the similarity of pixel distribution in the adaptive dynamic matching area based on the regional stereo matching algorithm based on temperature constraints to obtain grayscale similarity, temperature gradient amplitude similarity and temperature gradient direction similarity; taking the maximum value of the sum of the grayscale similarity, the temperature gradient amplitude similarity and the temperature gradient direction similarity as the pixel value; determining the matching pixel position according to the pixel value, and taking the pixel point corresponding to the matching pixel position as the matching point.

[0110] In one embodiment, a regional stereo matching algorithm based on temperature constraints calculates the similarity of pixel distribution in the adaptive dynamic matching area to obtain grayscale similarity, temperature gradient amplitude similarity and temperature gradient direction similarity, including: obtaining RGB-T data of pixels in the adaptive dynamic matching area; extracting RGB channel data and temperature channel data of the RGB-T data respectively; the regional stereo matching algorithm based on temperature constraints performs correlation calculation based on the RGB channel data to obtain grayscale similarity; the regional stereo matching algorithm based on temperature constraints performs correlation calculation based on the temperature channel data to obtain temperature gradient amplitude similarity and temperature gradient direction similarity.

[0111] Specifically, Figure 2As shown, Step 7: stereo matching. The correlation of pixel distribution in the matching area of ​​the matching pixels is calculated, and the T channel data (temperature data) of the RGB-T data is extracted at the same time, the amplitude and direction of the temperature gradient are calculated, and the similarities are compared to obtain the grayscale similarity, the temperature gradient amplitude similarity and the temperature gradient direction similarity. In the embodiment of the present application, the grayscale similarity (grayscale similarity) of the RGB channels and the similarity of the gradient amplitude and direction of the T channel (temperature gradient amplitude similarity and temperature gradient direction similarity) are comprehensively considered. After the row-by-row traversal is completed, the matching pixel position in the most relevant matching area is obtained through the pixel value MD to form a matching point. The calculation formula of the pixel value MD is:

[0112] MD=max(Gray RGB +Gradient IR_a +Gradient IR_d )

[0113] Among them, Gray RGB is the grayscale similarity of RGB channels; Gradient IR_a is the similarity of the gradient amplitude of the T channel; IR_d is the similarity of the gradient direction of the T channel.

[0114] In one embodiment, disparity calculation is performed based on the matching points, and a target single-field three-dimensional temperature field is constructed based on the disparity calculation results and the calibration parameters of the dual-light acquisition platform, including: calculating the disparity based on the RGB-T data of the epipolar constraint image based on the matching points to obtain an initial disparity image; performing interpolation optimization and outlier correction on the initial disparity image to obtain a target disparity image; and constructing a target single-field three-dimensional temperature field based on the target disparity image and the calibration parameters of the dual-light acquisition platform.

[0115] In one embodiment, a target single-field three-dimensional temperature field is constructed according to the target disparity image and the calibration parameters of the dual-light acquisition platform, including: determining an intrinsic parameter matrix and an extrinsic parameter matrix based on the calibration parameters of the dual-light acquisition platform; calculating the spatial three-dimensional coordinates of each pixel point according to the target disparity image, the intrinsic parameter matrix and the extrinsic parameter matrix; generating a three-dimensional feature point cloud based on the spatial three-dimensional coordinates; and performing surface reconstruction on the three-dimensional feature point cloud based on triangulation to construct the target's three-dimensional temperature field.

[0116] Specifically, Figure 2As shown, Step 8: disparity calculation and optimization. Based on the two sets of RGB-T data after stereo matching in Step 7, disparity calculation is performed to obtain an initial disparity image, and the initial disparity image is interpolated. At the same time, outlier correction, small connected domain removal and filtering are performed to generate a target disparity image. It should be noted that the segmentation result of the infrared image is used to find the most relevant matching point between the two images, and the disparity is calculated based on the coordinate offset between the two matching points. The point cloud data generated by the initial disparity image is relatively sparse, and the target disparity image obtained by interpolation processing, outlier correction, small connected domain removal and filtering can generate dense point cloud data. This embodiment does not limit the processing method of the disparity image.

[0117] It is understandable that the depth information can be calculated based on the disparity image, and the depth information of each pixel point can be converted into coordinates in three-dimensional space to construct point cloud data, such as constructing dense point cloud and dense temperature data through the target disparity image, the intrinsic parameter matrix and the extrinsic parameter matrix. Among them, the intrinsic parameter matrix and the extrinsic parameter matrix include focal length, principal point coordinates, and distortion coefficients, etc. The intrinsic parameter matrix is ​​used to describe the characteristics of the camera device itself, and the extrinsic parameter matrix involves the relative position and posture between the visible light device and the infrared device, so as to generate point cloud according to the obtained target disparity map and camera device parameters (such as the intrinsic parameter matrix and the extrinsic parameter matrix).

[0118] Specifically, the reconstruction process may include reconstructing sparse point clouds, reconstructing dense point clouds, reconstructing curved surfaces, texture mapping, etc. Figure 2 As shown, Step 9: 3D reconstruction. Based on the disparity map of Step 8 and the camera calibration parameters of Step 1, the spatial 3D coordinates of each pixel are calculated, and the point cloud data is reconstructed based on triangulation. The T channel data (temperature data) of the RGB-T image is used for surface mapping, and finally the reconstruction of the target single-view 3D temperature field is achieved. Among them, the camera calibration parameters include but are not limited to the intrinsic parameter matrix, the extrinsic parameter matrix and the distortion parameters.

[0119] In some embodiments, the method also includes: aligning the corrected visible light images in adjacent dual-light groups in the dual-light acquisition platform and calculating the posture relationship between the adjacent dual-light groups to obtain rotation information and translation information; obtaining the three-dimensional temperature information corresponding to the target single-field three-dimensional temperature field of each dual-light group; and splicing the three-dimensional temperature information of multiple dual-light groups based on the rotation information and translation information to construct a target multi-field three-dimensional temperature field.

[0120] It should be noted that Figure 2 The reconstruction process of the 3D temperature field in the field of view of a dual-light group is as follows: Figure 2The process operation shown can obtain the three-dimensional temperature information of each dual-light group. In the embodiment of the present application, the three-dimensional temperature information of multiple dual-light groups can also be spliced ​​to obtain the target multi-field three-dimensional temperature field. Specifically, after the visible light devices in the same dual-light group are registered, the adjacent visible light devices in multiple dual-light groups in the dual-light acquisition platform (for example, Figure 3 The visible light device of the dual light group 2 and the visible light device of the dual light group 3 are aligned, and the rotation information and the translation information can be obtained by calculating the posture relationship between the dual light groups. After the three-dimensional temperature information of each dual light group is obtained through the above steps Step 1 to Step 9, the three-dimensional temperature information of each dual light group is spliced ​​according to the above rotation information and translation information to form the final target multi-view three-dimensional temperature field.

[0121] In the embodiment of the present application, the advantages of dual-light equipment are used to fuse the images of multiple band devices (such as infrared equipment and visible light equipment), which makes up for the shortcomings of weak texture features of single infrared equipment and realizes the construction of single-field three-dimensional temperature field information of the target. A collection platform including multiple dual-light equipment is built, and a composite calibration plate, i.e., a composite target, is designed to realize synchronous correction and calibration of multi-band equipment, perform modal conversion on visible light images, reduce modal conflicts of heterogeneous images, improve the registration accuracy of heterogeneous images, construct registered RGB-T data, and design stereo matching, field of view calculation and optimization algorithms with temperature distribution constraints based on RGB-T data, and finally realize the reconstruction of the target three-dimensional temperature field.

[0122] The embodiment of the present application provides a method for establishing a target three-dimensional temperature field based on infrared and visible light images, including: calibrating a dual-light acquisition platform to obtain a corrected visible light image and a corrected infrared image; converting and registering the corrected visible light image and the corrected infrared image to obtain a registered infrared image and a registered visible light thermal image; performing stereo matching and data reconstruction according to the registered infrared image and the registered visible light thermal image to obtain matching points; performing parallax calculation according to the matching points, and constructing a target single-view three-dimensional temperature field according to the parallax calculation result and the calibration parameters of the dual-light acquisition platform. In the embodiment of the present application, the infrared and visible light band images of the dual-light acquisition platform are fused to realize synchronous correction and calibration of infrared and visible light images, reduce the modal conflict of heterogeneous images through modal conversion and registration, improve the registration accuracy of heterogeneous images, construct the registered data for stereo matching and parallax calculation, realize the reconstruction of the target three-dimensional temperature field, overcome the shortcomings of weak texture features of a single infrared device, and thus improve the reconstruction accuracy of the target three-dimensional temperature field.

[0123] In addition, an embodiment of the present application also proposes a storage medium, on which is stored a target three-dimensional temperature field establishment program based on infrared and visible light images. When the target three-dimensional temperature field establishment program based on infrared and visible light images is executed by a processor, the steps of the target three-dimensional temperature field establishment method based on infrared and visible light images as described above are implemented.

[0124] Reference Figure 6 , Figure 6 This is a structural block diagram of an embodiment of a device for establishing a target three-dimensional temperature field based on infrared and visible light images of the present application.

[0125] like Figure 6 As shown, the target three-dimensional temperature field establishment device based on infrared and visible light images includes:

[0126] The calibration and correction module 10 is configured to calibrate the dual-light acquisition platform to obtain a corrected visible light image and a corrected infrared image;

[0127] A registration and conversion module 20 is configured to register and convert the corrected visible light images in the same dual-light group in the dual-light acquisition platform to obtain a visible light thermal image;

[0128] A heterogeneous registration module 30 is configured to perform heterogeneous registration on the visible light thermal image and the corrected infrared image in the same dual light pair in the dual light acquisition platform to obtain a registered infrared image and a registered visible light thermal image;

[0129] A matching and reconstruction module 40 is configured to perform stereo matching and data reconstruction according to the registered infrared image and the registered visible light thermal image to obtain matching points;

[0130] The calculation and construction module 50 is configured to perform parallax calculation according to the matching points, and to construct a target single-view three-dimensional temperature field according to the parallax calculation result and the calibration parameters of the dual-light acquisition platform.

[0131] In the embodiment of the present application, the advantages of dual-light equipment are used to fuse the images of multiple band devices (such as infrared equipment and visible light equipment), which makes up for the shortcomings of weak texture features of single infrared equipment and realizes the construction of single-field three-dimensional temperature field information of the target. A collection platform including multiple dual-light equipment is built, and a composite calibration plate, i.e., a composite target, is designed to realize synchronous correction and calibration of multi-band equipment, perform modal conversion on visible light images, reduce modal conflicts of heterogeneous images, improve the registration accuracy of heterogeneous images, construct registered RGB-T data, and design stereo matching, field of view calculation and optimization algorithms with temperature distribution constraints based on RGB-T data, and finally realize the reconstruction of the target three-dimensional temperature field.

[0132] It should be noted that the technical details that are not described in detail in the embodiment of the target three-dimensional temperature field establishment device based on infrared and visible light images can be referred to the target three-dimensional temperature field establishment method based on infrared and visible light images as described above provided in any embodiment of the present application, and will not be repeated here.

[0133] Reference Figure 7 , Figure 7 A schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiment of the present application.

[0134] like Figure 7 As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM memory) or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0135] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0136] like Figure 7 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a target three-dimensional temperature field establishment program based on infrared and visible light images.

[0137] exist Figure 7In the electronic device shown, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the electronic device of the embodiment of the present application can be set in the electronic device, and the electronic device calls the target three-dimensional temperature field establishment program based on infrared and visible light images stored in the memory 1005 through the processor 1001, and executes the target three-dimensional temperature field establishment method based on infrared and visible light images provided in the embodiment of the present application.

[0138] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present application. In specific applications, technicians in this field can make settings as needed, and the present application does not impose any limitation on this.

[0139] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present application. In practical applications, technicians in this field can select part or all of it according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0140] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0141] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0142] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0143] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for establishing a target three-dimensional temperature field based on infrared and visible light images, comprising: Calibrate the dual-light acquisition platform to obtain a corrected visible light image and a corrected infrared image; Registering and converting the corrected visible light images in the same dual-light group in the dual-light acquisition platform to obtain a visible light thermal image; Performing heterogeneous registration on the visible light thermal image and the corrected infrared image in the same dual light pair in the dual light acquisition platform to obtain a registered infrared image and a registered visible light thermal image; Perform stereo matching and data reconstruction according to the registered infrared image and the registered visible light thermal image to obtain matching points; Parallax calculation is performed according to the matching points, and a target single-view three-dimensional temperature field is constructed according to the parallax calculation result and the calibration parameters of the dual-light acquisition platform.

2. The method of claim 1, wherein: The method further comprises: Build a dual-light collection platform; among them, The dual-light collection platform includes a plurality of dual-light groups, which are distributed around a target and whose total field of view covers the target. There are overlapping field of view areas between the dual-light groups.

3. The method of claim 1, wherein: The dual-light acquisition platform is calibrated to obtain a corrected visible light image and a corrected infrared image, including: Directing the infrared device and the visible light device of each dual light pair of the dual light collection platform toward the composite target; Collecting original captured images of the infrared device and the visible light device of each dual light pair based on the composite target; The original captured image is calibrated and distortion corrected to obtain a corrected visible light image and a corrected infrared image of each of the dual light pairs.

4. The method of claim 1, wherein: The registering and converting the corrected visible light images in the same dual-light group in the dual-light acquisition platform to obtain a visible light thermal image includes: Registering the corrected visible light images in the same dual-light group in the dual-light acquisition platform to obtain a registered visible light image; The registered visible light image is modally converted based on a preset image modality conversion algorithm to obtain a visible light thermal image.

5. The method of claim 1, wherein: The performing stereo matching and data reconstruction according to the registered infrared image and the registered visible light thermal image to obtain matching points includes: Reconstructing the registered infrared image and the registered visible light thermal image to obtain a four-channel data image; Based on the epipolar constraint, epipolar correction is performed on the four-channel data image of the visible light device in the dual-light group to obtain an epipolar constrained image; Segmenting the registered infrared image to obtain an infrared image segmentation result; Stereo matching is performed on the epipolar constraint image according to the infrared image segmentation result to obtain matching points.

6. The method of claim 5, wherein: The step of reconstructing the registered infrared image and the registered visible light thermal image to obtain a four-channel data image includes: acquiring temperature data according to the registered infrared image; Acquiring RGB data according to the registered visible light thermal image; Data reconstruction is performed according to the temperature data and the RGB data to obtain a four-channel data image of RGB-T data.

7. The method of claim 5, wherein: The step of performing stereo matching on the epipolar constraint image according to the infrared image segmentation result to obtain matching points includes: Construct an initial matching area centered on the matching pixel; Screening the pixel points in the initial matching area based on the infrared image segmentation result to generate an adaptive dynamic matching area; The temperature-constrained regional stereo matching algorithm performs row-by-row stereo matching on the epipolar-constrained image according to the adaptive dynamic matching region to obtain matching points.

8. The method of claim 1, wherein: The disparity calculation is performed according to the matching points, and the target single-view three-dimensional temperature field is constructed according to the disparity calculation result and the calibration parameters of the dual-light acquisition platform, including: Calculate the disparity based on the matching points according to the RGB-T data of the epipolar constrained image to obtain an initial disparity image; Performing interpolation optimization and outlier correction on the initial disparity image to obtain a target disparity image; A target single-view three-dimensional temperature field is constructed according to the target parallax image and the calibration parameters of the dual-light acquisition platform.

9. The method according to any one of claims 1 to 8, wherein: The method further comprises: Registering the corrected visible light images in adjacent dual-light groups in the dual-light acquisition platform and calculating the posture relationship between the adjacent dual-light groups to obtain rotation information and translation information; Obtain the three-dimensional temperature information corresponding to the target single-view three-dimensional temperature field of each dual-light group; The three-dimensional temperature information of the plurality of dual-light groups is spliced ​​based on the rotation information and the translation information to construct a target multi-view three-dimensional temperature field.

10. A device for establishing a target three-dimensional temperature field based on infrared and visible light images, comprising: A calibration and correction module, configured to calibrate the dual-light acquisition platform to obtain a corrected visible light image and a corrected infrared image; A registration and conversion module, configured to register and convert the corrected visible light images in the same dual-light group in the dual-light acquisition platform to obtain a visible light-like thermal image; A heterogeneous registration module is configured to perform heterogeneous registration on the visible light thermal image and the corrected infrared image in the same dual light pair in the dual light acquisition platform to obtain a registered infrared image and a registered visible light thermal image; A matching and reconstruction module, configured to perform stereo matching and data reconstruction according to the registered infrared image and the registered visible light thermal image to obtain matching points; The calculation and construction module is configured to perform parallax calculation according to the matching points, and to construct a target single-view three-dimensional temperature field according to the parallax calculation result and the calibration parameters of the dual-light acquisition platform.

Images