A method, apparatus, and electronic device for generating four-dimensional thermal images

Abstract

This application provides a method, apparatus, and electronic device for generating a four-dimensional thermal image model, belonging to the field of image processing technology. The method includes: acquiring a first infrared image captured by an infrared thermal imaging module against a preset calibration component and a first visible light image captured by a binocular camera module; calibrating the first infrared image to obtain first intrinsic parameters and first extrinsic parameters; calibrating the first visible light image to obtain second intrinsic parameters and second extrinsic parameters; calibrating a first conversion parameter between a left camera and a right camera based on the second extrinsic parameters; performing temperature mapping based on the first and second extrinsic parameters to obtain a second conversion parameter between the infrared thermal imaging module and a target camera; performing three-dimensional reconstruction based on the acquired second visible light image to obtain a three-dimensional point cloud model; acquiring a second infrared image, and obtaining a four-dimensional thermal image model based on the second infrared image, the conversion parameters, and the three-dimensional point cloud model. This application can generate four-dimensional thermal image models with higher accuracy and stronger robustness.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and apparatus for generating a four-dimensional thermal image model, and an electronic device. Background Technology

[0002] Currently, infrared thermal imagers can convert the invisible infrared energy received from objects into recognizable grayscale images, resulting in infrared thermal images that display the temperature distribution on the object's surface. However, because infrared thermal images cannot characterize the geometric information of an object's surface, two-dimensional infrared thermal images cannot accurately determine whether abnormalities have occurred during the heating process for components with complex structures or significant depth variations. In related technologies, the fusion of infrared thermal images and 3D reconstruction techniques generally takes two forms: one is image domain fusion based on image feature extraction, and the other is target calibration-based methods. For the first method, image fusion relies on obvious feature points, which is ineffective for infrared images with low resolution and weak texture features. For the second method, related technologies use a pinhole model of the camera to map a 3D point cloud onto the image coordinate system of the infrared image through matrix transformation to obtain a 3D temperature distribution model. However, the resulting point cloud model lacks more detailed texture information, thus affecting the accuracy of the constructed 4D thermal image model. Therefore, the 4D thermal image model obtained by fusing infrared thermal images and 3D reconstruction techniques suffers from insufficient accuracy and is unsuitable for scenes with low resolution and weak texture features. Summary of the Invention

[0003] The main objective of this application is to propose a method, apparatus, and electronic device for generating four-dimensional thermal images, which can generate four-dimensional thermal images with higher accuracy, stronger robustness, and better applicability in scenarios with lower resolution and weaker texture features.

[0004] To achieve the above objectives, a first aspect of this application proposes a method for generating a four-dimensional thermal image model, applied to a controller, the controller being used for communicative connection with a binocular camera module and an infrared thermal imaging module, the method comprising:

[0005] Acquire the first infrared image of the preset calibration component captured by the infrared thermal imaging module, and acquire the first visible light image of the preset calibration component captured by the binocular camera module;

[0006] The first infrared image is calibrated according to the preset calibration parameters of the preset calibration component to obtain a first internal parameter and a first external parameter. The first internal parameter is used to represent the camera parameters of the infrared thermal imaging module, and the first external parameter is used to represent the relative position parameters of the infrared thermal imaging module in the world coordinate system.

[0007] The first visible light image is calibrated according to the preset calibration parameters to obtain the second intrinsic parameter and the second extrinsic parameter of the binocular camera module. The second intrinsic parameter is used to represent the camera parameters of the binocular camera module, and the second extrinsic parameter is used to represent the relative position parameters of the binocular camera module in the world coordinate system.

[0008] The relative position of the binocular camera module is calibrated according to the second extrinsic parameter to obtain the first conversion parameter, which is used to represent the extrinsic parameter conversion relationship between the left camera and the right camera in the binocular camera module;

[0009] Temperature mapping calibration is performed based on the first external parameter and the second external parameter to obtain the second conversion parameter. The second conversion parameter is used to represent the external parameter conversion relationship between the infrared thermal imaging module and the target camera of the binocular camera module. The target camera is the left camera or the right camera.

[0010] The binocular camera module acquires second visible light images of the object under test under projection gratings at different fringe frequencies, and performs three-dimensional reconstruction based on the second intrinsic parameters, the first conversion parameters, and the second visible light images to obtain a three-dimensional point cloud model of the object under test.

[0011] A second infrared image of the object under test is acquired, and temperature mapping processing is performed based on the first intrinsic parameter, the second conversion parameter, the second infrared image, and the three-dimensional point cloud model to obtain a four-dimensional thermal image model of the object under test.

[0012] In some embodiments, the preset calibration parameters include a preset number of marked circles, a preset roundness threshold, and a preset area ratio range;

[0013] The step of calibrating the first visible light image according to the preset calibration parameters to obtain the second intrinsic parameters and second extrinsic parameters of the binocular camera module includes:

[0014] Corner contours are extracted from the first visible light image to obtain an initial circle and initial contour data of the initial circle. The initial contour data includes the initial contour surface area and the initial contour length.

[0015] The initial roundness value is obtained by calculating the initial contour surface area and initial contour length.

[0016] When the initial roundness value is less than or equal to the preset roundness threshold, the initial contour data is used as candidate contour data, and the initial circle of the initial contour data is used as a candidate circle.

[0017] The average area value of the contour is obtained by calculating the mean area based on the candidate contour data.

[0018] The candidate circles in the candidate contour data are filtered based on the contour surface area, the average area value of the contour, and the preset area ratio range, and the target circle of the first visible light image is determined based on the result of the circle filtering.

[0019] When the number of target circles is equal to the number of preset label circles, the first visible light image is calibrated based on the target circles to obtain the second intrinsic parameter and the second extrinsic parameter.

[0020] In some embodiments, the step of calibrating the relative position of the binocular camera module according to the second extrinsic parameter to obtain the first conversion parameter includes:

[0021] The origin of the first world coordinate system is set according to the first visible light image, and the first world coordinates are obtained according to the origin of the first world coordinate system; wherein, the first world coordinates are used to represent the coordinates of the center of the target circle in the first world coordinate system;

[0022] The first pixel coordinates are obtained based on the first visible light image. The first pixel coordinates are used to represent the coordinates of the center of the target circle in the first pixel coordinate system of the left camera.

[0023] The second pixel coordinates are obtained based on the first visible light image. The second pixel coordinates are used to represent the coordinates of the center of the target circle in the second pixel coordinate system of the right camera.

[0024] The first transformation parameter is calculated based on the first world coordinates, the first pixel coordinates, and the second pixel coordinates.

[0025] In some embodiments, the preset area ratio range includes a preset area ratio upper limit and a preset area ratio lower limit;

[0026] The step of filtering candidate circles in the candidate contour data based on the contour surface area, the average contour area value, and the preset area ratio range, and determining the target circle based on the results of the circle filtering, includes:

[0027] The candidate ratio value is obtained by calculating the area ratio between the surface area of ​​the candidate contour data and the average area value of the contour.

[0028] When the candidate ratio value is less than the upper limit of the preset area ratio value and the candidate ratio value is greater than the lower limit of the preset area ratio value, the candidate circle corresponding to the candidate ratio value is taken as the target circle.

[0029] In some embodiments, the step of performing temperature mapping processing based on the first intrinsic parameter, the second transformation parameter, the second infrared image, and the three-dimensional point cloud model to obtain a four-dimensional thermal image model of the object under test includes:

[0030] Obtain the initial camera coordinates of the 3D point cloud in the 3D point cloud model, wherein the initial camera coordinates are used to represent the coordinates of the 3D point cloud in the camera coordinate system of the target camera;

[0031] The initial camera coordinates are transformed according to the second transformation parameters to obtain the first infrared coordinates. The first infrared coordinates are used to represent the coordinates of the three-dimensional point cloud in the infrared thermal imager coordinate system of the second infrared image.

[0032] A second coordinate transformation is performed based on the first intrinsic parameter and the first infrared coordinate to obtain the second infrared coordinate, which is used to represent the coordinates of the three-dimensional point cloud in the image coordinate system of the second infrared image.

[0033] A third coordinate transformation is performed based on the first intrinsic parameter and the second infrared coordinate to obtain the third infrared coordinate, which is used to represent the coordinates of the three-dimensional point cloud in the pixel coordinate system of the second infrared image;

[0034] Temperature mapping is performed based on the third infrared coordinates and the three-dimensional point cloud model to obtain the four-dimensional thermal image model of the object under test.

[0035] In some embodiments, the step of performing three-dimensional reconstruction based on the second intrinsic parameter, the first transformation parameter, and the second visible light image to obtain a three-dimensional point cloud model of the object under test includes:

[0036] Acquire the average illumination intensity data and intensity modulation data of the second visible light image;

[0037] The first phase principal value is obtained by calculating the phase principal value based on the average illumination intensity data, the intensity modulation data, and the preset first phase difference;

[0038] The second phase principal value is obtained by calculating the phase principal value based on the average illumination intensity data, the intensity modulation data, and the preset second phase difference.

[0039] The target phase is obtained by performing phase calculation based on the preset frequency ratio, the first phase principal value, and the second phase principal value.

[0040] The three-dimensional point cloud model is obtained by performing three-dimensional reconstruction based on the target phase, the second intrinsic parameter, the first conversion parameter, and the second visible light image.

[0041] In some embodiments, the first conversion parameter includes a first conversion sub-parameter and a second conversion sub-parameter, wherein the first conversion sub-parameter is used to represent the conversion parameters for the conversion from the left camera to the right camera, and the second conversion sub-parameter is used to represent the conversion parameters for the conversion from the right camera to the left camera;

[0042] The step of performing three-dimensional reconstruction based on the target phase, the second intrinsic parameter, the first conversion parameter, and the second visible light image to obtain the three-dimensional point cloud model includes:

[0043] When the target camera is the left camera, a left projection image is obtained based on the projection of the second visible light image onto the image plane of the left camera, and a right projection image is obtained based on the projection of the second visible light image onto the image plane of the right camera. The left projection image includes pixels to be matched.

[0044] The coordinates of the left outer pole are calculated based on the second intrinsic parameter and the first transformation sub-parameter. The coordinates of the left outer pole are used to represent the coordinates of the left outer pole center of the left camera in the left camera coordinate system.

[0045] The coordinates of the right outer pole are calculated based on the second intrinsic parameter and the second transformation sub-parameter. The coordinates of the right outer pole are used to represent the coordinates of the right outer pole center of the right camera in the right camera coordinate system.

[0046] The slope of the left outer epipolar line is obtained by calculating the slope based on the coordinates of the pixel to be matched and the coordinates of the left outer epipolar point; the coordinates of the pixel to be matched are used to represent the coordinates of the pixel to be matched in the left camera coordinate system.

[0047] The polar equation is constructed based on the preset slope conversion formula, the slope of the left outer polar line, and the coordinates of the right outer pole, to obtain the right outer polar line equation.

[0048] The initial matching point is determined by selecting points based on the target phase, the right epipolar line equation, and the left projection image.

[0049] The three-dimensional point cloud model is obtained by performing three-dimensional reconstruction based on the slope of the left outer epipolar line and the initial matching point.

[0050] In some embodiments, the step of performing three-dimensional reconstruction based on the slope of the left epipolar line and the initial matching points to obtain the three-dimensional point cloud model includes:

[0051] Based on the initial matching point and the preset neighborhood threshold, matching points are selected to obtain candidate matching points and the coordinates of the right outer pole of the candidate matching points;

[0052] The slope of the right outer epipolar line is obtained by calculating the coordinates of the right outer epipolar point and the coordinates of the candidate matching point; the coordinates of the candidate matching point are used to represent the coordinates of the candidate matching point in the right camera coordinate system.

[0053] The slope difference is calculated by using the slope conversion formula, the slope of the left outer polar line, and the slope of the right outer polar line.

[0054] When the slope difference is less than a preset difference threshold, the candidate matching point is taken as the intermediate matching point;

[0055] The target matching point is determined based on the target phase of the intermediate matching point;

[0056] The three-dimensional point cloud model is obtained by performing three-dimensional reconstruction based on the target matching points.

[0057] A second aspect of this application provides a four-dimensional thermal imaging model generation apparatus, the apparatus comprising:

[0058] A controller, the controller being configured to execute a four-dimensional thermal imaging model generation method as described in the first aspect of the embodiments of this application;

[0059] A binocular camera module, which is communicatively connected to the controller, is used to acquire visible light images;

[0060] An infrared thermal imaging module, which is communicatively connected to the controller, is used to acquire infrared images;

[0061] A preset calibration component is used to calibrate the binocular camera module and the infrared thermal imaging module.

[0062] A third aspect of this application provides an electronic device, comprising:

[0063] At least one memory;

[0064] At least one processor;

[0065] At least one computer program;

[0066] The at least one computer program is stored in the at least one memory, and the at least one processor executes the at least one computer program to perform:

[0067] The method described in the first aspect of the embodiments of this application.

[0068] This application proposes a method, apparatus, and electronic device for generating a four-dimensional thermal image model. First, an infrared image is acquired by an infrared thermal imaging module using a preset calibration component, and a first visible light image is acquired by a binocular camera module using the same preset calibration component. The first infrared image is calibrated according to preset calibration parameters of the preset calibration component to obtain first intrinsic parameters and first extrinsic parameters. The first intrinsic parameters represent the camera parameters of the infrared thermal imaging module, and the first extrinsic parameters represent the relative position parameters between the infrared thermal imaging module and the preset calibration component. Then, the first visible light image is calibrated according to the preset calibration parameters to obtain second intrinsic parameters and second extrinsic parameters of the binocular camera module. The second intrinsic parameters represent the camera parameters of the binocular camera module, and the second extrinsic parameters represent the relative position parameters between the binocular camera module and the preset calibration component. The binocular camera module is then calibrated relative to its position using the second extrinsic parameters to obtain a first transformation parameter. This first transformation parameter represents the extrinsic parameter transformation relationship between the left and right cameras in the binocular camera module. The process involves temperature mapping calibration based on the first and second extrinsic parameters to obtain a second conversion parameter. This second conversion parameter represents the extrinsic parameter conversion relationship between the infrared thermal imaging module and the target camera of the binocular camera module, where the target camera is either the left or right camera. Next, the binocular camera module acquires second visible light images of the object under different fringe frequencies using projection gratings. Based on the second intrinsic parameter, the first conversion parameter, and the second visible light image, a three-dimensional reconstruction is performed to obtain a three-dimensional point cloud model of the object. Finally, the second infrared image of the object is acquired, and temperature mapping processing is performed based on the first intrinsic parameter, the second conversion parameter, the second infrared image, and the three-dimensional point cloud model to obtain a four-dimensional thermal imaging model of the object. This embodiment of the application can generate a more accurate four-dimensional thermal imaging model with stronger robustness and is better applicable in scenes with lower resolution and weaker texture features. Attached Figure Description

[0069] Figure 1 This is a flowchart of one method of the four-dimensional thermal image model generation method provided in the embodiments of this application;

[0070] Figure 2 This is a schematic diagram of the structure of an infrared thermal imaging module and a binocular camera module provided in the embodiments of this application;

[0071] Figure 3 yes Figure 1 The detailed flowchart of step S130;

[0072] Figure 4A This is a schematic diagram of a pre-calibrated board constructed according to an embodiment of this application;

[0073] Figure 4B This is an image of a pre-calibrated plate after heating, captured by an infrared thermal imager according to an embodiment of this application;

[0074] Figure 4C This is an image of the heated preset calibration plate captured by the left camera in the binocular camera module of this application embodiment;

[0075] Figure 4D Yes Figure 4B A schematic diagram of each target circle in the image after sorting;

[0076] Figure 5 yes Figure 3 The detailed flowchart of step S350;

[0077] Figure 6 This is a flowchart of the calibration algorithm for the binocular camera module and the infrared thermal imaging module provided in the embodiments of this application;

[0078] Figure 7 yes Figure 1 The detailed flowchart of step S140;

[0079] Figure 8 yes Figure 1 The detailed flowchart of step S160;

[0080] Figure 9A This is a schematic diagram of the phase information acquisition process during three-dimensional reconstruction provided in an embodiment of this application;

[0081] Figure 9B This is a schematic diagram of a second visible light image of the object under test, captured by the left camera of this application, under the illumination of projection grating patterns at different fringe frequencies.

[0082] Figure 9C This is a schematic diagram of a second visible light image of the object under test, captured by the right camera of this application, under the illumination of projection grating patterns at different fringe frequencies.

[0083] Figure 10 This is a schematic diagram of the image used in this application to solve the target phase using the multi-frequency heterodyne method.

[0084] Figure 11 yes Figure 8 The detailed flowchart of step S850;

[0085] Figure 12 This is a schematic diagram of the epipolar slope-absolute phase matching process provided in an embodiment of this application;

[0086] Figure 13 yes Figure 11 The detailed flowchart of step S1170;

[0087] Figure 14 This is a flowchart of the absolute phase matching algorithm provided in the embodiments of this application;

[0088] Figure 15 yes Figure 1 The detailed flowchart of step S170;

[0089] Figure 16 This is a three-dimensional schematic diagram of the temperature mapping process provided in the embodiments of this application;

[0090] Figure 17A This is a schematic diagram of the prototype structure of the selected test object provided in the embodiments of this application;

[0091] Figure 17B Yes Figure 17A A schematic diagram of the structure of the 3D point cloud model of the object under test after passing through the 3D reconstruction module;

[0092] Figure 17C Yes Figure 17A A schematic diagram of the structure of a four-dimensional thermal imaging model of the object under test after temperature mapping processing.

[0093] Figure 18A This is a schematic diagram of the structure of the three cuboid heating stages provided in the embodiments of this application;

[0094] Figure 18B Yes Figure 18A The depth measurement results of the thermal image model obtained by generating a four-dimensional thermal image model of the object in the image.

[0095] Figure 19 This is another structural schematic diagram of the infrared thermal imaging module and binocular camera module provided in the embodiments of this application;

[0096] Figure 20 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0097] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0098] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0099] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0100] Currently, infrared thermal imagers can convert the invisible infrared energy received from an object into a recognizable grayscale image. The resulting infrared thermal image is an image that can display the temperature distribution on the surface of an object. Infrared thermal images are widely used in many fields such as medicine, agriculture, and military. Compared with visible light images, they are not sensitive to light and can be used for nighttime shooting. However, infrared thermal images taken by infrared thermal imagers cannot well express the spatial temperature distribution of the measured object, mainly because: (1) there is a lack of three-dimensional spatial information of the measured object, which makes it impossible to capture abnormal temperature areas in time; (2) it is difficult to obtain temperature information of the measured object from a non-single perspective. Therefore, for industrial parts with simple structures, analyzing infrared thermal images can accurately determine whether there is an abnormality during the heating process. However, for components with more complex structures or greater depth changes, the limitations become more and more obvious. Two-dimensional infrared thermal images cannot accurately determine whether there is an abnormality during the heating process of the component. Only by analyzing three-dimensional temperature spectrum images can an accurate diagnosis be made. Based on this, combining two-dimensional thermal images with three-dimensional reconstruction technology to construct a four-dimensional thermal image model with temperature information, in order to obtain a visualized three-dimensional component of temperature distribution, has extremely important and far-reaching significance.

[0101] Studies have shown that the fusion of infrared thermal images and 3D reconstruction techniques generally follows two approaches: image-domain fusion based on image feature extraction and target-based calibration. The first approach relies on obvious feature points, resulting in poor performance for infrared images with low resolution and weak texture features. The second approach, using target calibration, generally suffers from low accuracy and poor fusion quality in the resulting thermal models, primarily due to the low quality of the 3D point cloud. Improvements to the quality of 3D point clouds can be achieved through reconstruction using stereo matching based on grayscale regions in binocular vision, or through feature-based stereo matching. While gray-scale region-based stereo matching algorithms can obtain dense disparity maps, they are significantly affected by affine and radiometric distortions in the image. Gray-scale region-based stereo matching methods have obvious drawbacks, being easily affected by environmental interference and exhibiting slightly lower reconstruction accuracy at close range, thus impacting the reconstruction results. Feature-based stereo matching algorithms, while obtaining sparse disparity maps and then estimating dense disparity maps through difference estimation, can extract local features such as points, lines, and surfaces, as well as global features such as polygons and image structures. However, feature extraction is significantly affected by occlusion, lighting, and repetitive textures, thus affecting the reconstruction results. Therefore, the four-dimensional thermal imaging model obtained by fusing infrared thermal images and 3D reconstruction techniques lacks sufficient accuracy and is unsuitable for scenes with low resolution and weak texture features.

[0102] Based on this, the main objective of this application is to propose a method, apparatus, and electronic device for generating four-dimensional thermal images, which can generate four-dimensional thermal images with higher accuracy, stronger robustness, and better applicability in scenarios with lower resolution and weaker texture features.

[0103] The four-dimensional thermal imaging model generation method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the above method, but is not limited to the above forms.

[0104] The embodiments of this application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0105] Please refer to Figure 1 , Figure 1 This is a schematic flowchart of a four-dimensional thermal image model generation method provided in an embodiment of this application. In some embodiments of this application, the four-dimensional thermal image model generation method is applied to a controller, and the controller is used for communication connection with a binocular camera module and an infrared thermal imaging module. The four-dimensional thermal image model generation method provided in this application may specifically include, but is not limited to, steps S110 to S170.

[0106] Step S110: Obtain the first infrared image of the preset calibration component captured by the infrared thermal imaging module, and obtain the first visible light image of the preset calibration component captured by the binocular camera module.

[0107] Step S120: The first infrared image is calibrated according to the preset calibration parameters of the preset calibration component to obtain the first internal parameter and the first external parameter. The first internal parameter is used to represent the internal parameters of the infrared thermal imaging module, and the first external parameter is used to represent the relative position parameters of the infrared thermal imaging module in the world coordinate system.

[0108] Step S130: The first visible light image is calibrated according to the preset calibration parameters to obtain the second intrinsic parameters and the second extrinsic parameters of the binocular camera module. The second intrinsic parameters are used to represent the camera parameters of the binocular camera module, and the second extrinsic parameters are used to represent the relative position parameters of the binocular camera module in the world coordinate system.

[0109] Step S140: The relative position of the binocular camera module is calibrated according to the second extrinsic parameter to obtain the first conversion parameter. The first conversion parameter is used to represent the extrinsic parameter conversion relationship between the left camera and the right camera in the binocular camera module.

[0110] Step S150: Temperature mapping calibration is performed based on the first external parameter and the second external parameter to obtain the second conversion parameter. The second conversion parameter is used to represent the external parameter conversion relationship between the infrared thermal imaging module and the target camera of the binocular camera module. The target camera is either the left camera or the right camera.

[0111] Step S160: The binocular camera module acquires a second visible light image of the object under the projection grating at different fringe frequencies, and performs three-dimensional reconstruction based on the second intrinsic parameter, the first conversion parameter, and the second visible light image to obtain a three-dimensional point cloud model of the object under the test.

[0112] Step S170: Acquire the second infrared image of the object under test, and perform temperature mapping processing based on the first intrinsic parameter, the second transformation parameter, the second infrared image and the three-dimensional point cloud model to obtain the four-dimensional thermal image model of the object under test.

[0113] In step S110 of some embodiments, camera parameter calibration is a crucial step in both image measurement and machine vision applications. The accuracy of the calibration results and the stability of the algorithm directly affect the accuracy of the results produced by the camera. Therefore, in this application, proper camera calibration before generating the four-dimensional thermal image model is a prerequisite for improving the accuracy of the generated model. Specifically, firstly, images are acquired from the preset calibration component using the infrared thermal imaging module to obtain a first infrared image, which can then be used to calibrate the parameters of the infrared thermal imaging module. Furthermore, images are acquired from the preset calibration component using the binocular camera module to obtain a first visible light image, which can then be used to calibrate the parameters of the binocular camera module.

[0114] It should be noted that the infrared thermal imaging module provided in this application embodiment includes a first bracket and an infrared thermal imager. The first bracket is used to fix the infrared thermal imager, enabling it to acquire infrared images of a preset calibration component and the object being measured. Furthermore, the specific model of the infrared thermal imager can be flexibly selected according to actual needs, and no specific limitation is made here.

[0115] It should be noted that the binocular camera module includes a second support, a split-type binocular camera, and a Digital Light Processing (DLP) grating projection optical engine. The second support is used to fix the split-type binocular camera and the DLP grating projection optical engine. The optical engine lens of the DLP grating projection optical engine is used to form a deformed grating fringe pattern in the area where the preset calibration piece and the object under test are placed, and the preset calibration piece and the object under test are located at the center of the matrix of the projected deformed grating fringe pattern, so that the split-type binocular camera can capture the deformed grating fringe pattern projected onto the preset calibration piece and the object under test. The split-type binocular camera and the DLP grating projection optical engine constitute a reconstruction module for reconstructing a 3D point cloud, and this reconstruction module is placed head-to-head with the infrared thermal imager. The split-type binocular camera includes a left camera and a right camera, which are fixed on both sides of the DLP grating projection optical engine, intersecting the optical axes of the DLP grating projection optical engine and the binocular camera to ensure that the common field of view of the left and right cameras is sufficiently large.

[0116] For example, such as Figure 2 As shown, Figure 2 This is a schematic diagram of an infrared thermal imaging module and a binocular camera module provided in an embodiment of this application. The optical axis of the infrared thermal imager 210 of the infrared thermal imaging module is vertically aligned with the reference plane 230 where the heated preset calibration component 220 is located. The left camera 240 and right camera 250 of the split-type binocular camera are respectively placed on both sides of the DLP grating projection optical engine 260, and overlap with the field of view of the infrared thermal imager 210.

[0117] In step S120 of some embodiments, the single-plane checkerboard calibration method used in related technologies, although it can overcome the shortcomings of traditional calibration methods that require high-precision calibration objects, suffers from significant grayscale conversion during calibration due to the checkerboard target's feature points being black and white alternating corner points. This can affect the subsequent calculation of coordinates of points in three-dimensional space. Therefore, this application, based on the OpenCV calibration algorithm, selects a preset calibration component with a circular center point as its feature point, which has better noise resistance, for calibrating the binocular camera and infrared thermal imager. Specifically, this application optimizes the feature point extraction method in the preset calibration component and sorts the extracted feature points to obtain the correspondence between the image points corresponding to the feature points of the preset calibration component and the points in the world coordinate system. Then, it calls the planar calibration method used in related technologies to solve for the first intrinsic parameter and the first extrinsic parameter of the infrared thermal imager. The first intrinsic parameter represents the internal parameters of the infrared thermal imager, and the first extrinsic parameter represents the relative position parameters of the infrared thermal imager in the world coordinate system.

[0118] In step S130 of some embodiments, the correspondence between the object's three-dimensional coordinate reference point and its image pixels captured by the camera usually needs to be determined by the geometric model parameters obtained through camera calibration. The calibration of a stereo camera is to obtain the intrinsic parameters of the two cameras and the extrinsic parameters between the two camera coordinate systems. Specifically, the calibration of a single camera (left and right) is first completed to obtain the second intrinsic parameters, which include the intrinsic parameters of the left camera and the right camera. Then, parameter transformation is performed based on the intrinsic parameters of the left and right cameras to obtain the relative position parameters of the left camera relative to the right camera.

[0119] Please refer to Figure 3 , Figure 3 This is a schematic flowchart of step S130 provided in an embodiment of this application. In some embodiments of this application, the preset calibration parameters include a preset number of marked circles, a preset roundness threshold, and a preset area ratio range. Therefore, step S130 may specifically include, but is not limited to, steps S310 to S360.

[0120] Step S310: Extract corner contours from the first visible light image to obtain an initial circle and initial contour data of the initial circle. The initial contour data includes the initial contour surface area and the initial contour length.

[0121] Step S320: Calculate the roundness based on the initial contour surface area and initial contour length to obtain the initial roundness value;

[0122] Step S330: When the initial roundness value is less than or equal to the preset roundness threshold, the initial contour data is used as candidate contour data, and the initial circle of the initial contour data is used as a candidate circle.

[0123] Step S340: Calculate the average area value of the contour based on the candidate contour data;

[0124] Step S350: Based on the contour surface area, average contour area value and preset area ratio range of the candidate contour data, the candidate circles of the candidate contour data are filtered, and the target circle of the first visible light image is determined based on the result of the circle filtering.

[0125] Step S360: When the number of target circles is equal to the number of preset annotation circles, the first visible light image is calibrated according to the target circles to obtain the second intrinsic parameters and the second extrinsic parameters.

[0126] It should be noted that, in order to complete the calibration process for the binocular camera module and the infrared thermal imaging module, the intrinsic parameter matrices of the binocular camera module and the extrinsic parameter matrices between them are obtained separately. Please refer to... Figure 4A , Figure 4AThis is a schematic diagram of the preset calibration plate constructed according to an embodiment of this application. Specifically, this embodiment uses a 9×11 circular array planar calibration plate with black background and white circles as the preset calibration plate 410. The distance between the center points of each circle in the preset calibration plate 410 is precisely known during its fabrication, allowing a spatial coordinate system to be established based on these known distances. Among the 99 marker circles on the preset calibration plate 410, there are 5 larger marker circles 411 with a diameter of 7 mm and 94 smaller marker circles 412 with a diameter of 3.5 mm, and the center distance between each marker circle is 15 mm. The spatial coordinate system of the preset calibration plate 410 is determined by the position of the larger marker circles 411, based on the order of the calibration points. Furthermore, this embodiment collects a first visible light image captured by a binocular camera and a first infrared image captured by an infrared thermal imager by changing the angle between the plane of the preset calibration plate 410 and the image plane.

[0127] It should be noted that the number of preset marker circles can also be set to an 8×11 circular array planar calibration plate with black background and white circles. It can be flexibly adjusted according to actual needs, and no specific limitation is made here.

[0128] It should be noted that when determining the spatial coordinate system of the calibration plate based on the position of the large marker circle 411, this embodiment of the application can set a rule for marking the center of the large marker circle. Specifically, the center of the uppermost large marker circle 411 can be denoted as N, and the center of the large marker circle 411 appearing directly below the large marker circle 411 corresponding to N can be denoted as S. Then, the center of the large marker circle 411 closest to the large marker circle 411 corresponding to S can be denoted as Q, and the center of the large marker circle 411 in the left region of the large marker circle 411 corresponding to N can be denoted as W, and the center of the large marker circle 411 in the right region of the large marker circle 411 corresponding to N can be denoted as E. Therefore, this application can determine the spatial coordinate system of the preset calibration plate 410 based on the center of the large marker circle according to the order of the calibration points.

[0129] It should be noted that, since different materials have different emissivity and solar radiation absorptivity, this application can use black anodized aluminum plates and nickel-plated aluminum plates to fabricate the preset calibration plate in order to make the infrared imaging boundary clearer. Therefore, unlike the calibration plates made of glass used in related technologies, the preset calibration plate used in this application can be made of black anodized nickel-plated aluminum alloy, and the circular array portion of the preset calibration plate is hollowed out to improve the accuracy of calibration.

[0130] In step S310 of some embodiments, firstly, image preprocessing is performed on the first visible light image, and corner contour extraction is performed on the preprocessed image. Specifically, the OpenCV function cv::findContours() can be used, that is, by setting the contour retrieval mode to RETR_LIST, an array containing four values ​​will be returned for each detected contour data: [Next, Previous, First Child, Parent], that is, the next contour, the previous contour, the first child contour, and the corresponding parent contour of the detected contour. Finally, the cv::drawContours() function is used to draw the extracted contours, that is, to determine multiple initial circles and the initial contour data of each initial circle, and the initial contour data includes the initial contour surface area and the initial contour length.

[0131] It should be noted that in practical applications, after corner point contour extraction, each initial circle will be marked with a red line on the first visible light image, which will not be elaborated here.

[0132] It should be noted that image preprocessing includes grayscale transformation, geometric correction, image enhancement, image filtering, etc., which are not specifically limited here.

[0133] It should be noted that, in order to improve the accuracy of calibration, the embodiments of this application may acquire a preset number of first visible light images, and the preset number of images may be 7, 10, etc., without being specifically limited here.

[0134] In step S320 of some embodiments, since the extracted initial circles are not necessarily all the preset mark circles on the preset calibration plate, it is necessary to perform a first screening on the extracted initial circles. Specifically, since the first screening is based on the roundness value of the extracted initial circles, the roundness value is calculated by combining the roundness value calculation formula of the following formula (1), and the roundness can be calculated based on the initial contour surface area and the initial contour length to obtain the initial roundness value.

[0135]

[0136] Where A is the extracted initial contour surface area, l is the extracted initial contour length, and r is the initial roundness value. Furthermore, for an ideal circle, r = π.

[0137] In step S330 of some embodiments, since the preset calibration plate may be placed at different angles and positions during the shooting process, resulting in the actual captured circle on the preset calibration plate having an elliptical outline, a preset roundness threshold ε is set based on experience. r The preset roundness threshold ε rThis is used to filter initial circles on the first visible light image. Specifically, by iterating through the initial roundness values ​​of each initial circle, when r > ε r When r ≤ ε, the initial circle is discarded; when r ≤ ε r When the initial contour data is used as candidate contour data, the initial circle of the initial contour data is used as the candidate circle.

[0138] In step S340 of some embodiments, the total area is first calculated based on the candidate contour data of all candidate circles after the first screening, to obtain ∑A. i Then, the average area is calculated based on the number of candidate circles n after screening, to obtain the average area value of the contour. For example, the average area value of the contour is calculated according to the following formula (2), and the average area value of the contour is denoted as avgS.

[0139]

[0140] Among them, A i Let i be the surface area of ​​the i-th candidate circle, and let i be the range of [1, n].

[0141] In step S350 of some embodiments, in order to more accurately simulate the feature points of the marker circle in the preset calibration plate, the candidate circles of the candidate contour data are screened a second time according to the contour surface area, the average contour area value and the preset area ratio range of the candidate contour data to determine the target circle.

[0142] In step S360 of some embodiments, if the number of extracted target circles is exactly equal to the number of preset labeled circles, the coordinates of the calibration point of the center of the target circle in the established coordinate system are obtained, and an ellipse equation is constructed according to the calibration point coordinates corresponding to the center of each target circle. The center of the ellipse corresponding to the target circle is obtained by fitting the constructed ellipse equation, and the center of the ellipse is the feature point in the extracted first visible light image.

[0143] It should be noted that, after selecting the required number of target circles and obtaining the corresponding calibration point coordinates, in order to sort the target circles in the first visible light image, this embodiment of the application determines the five target circles with the largest diameters on the preset calibration plate based on the length of their major axes. Then, based on these five target circles with the largest diameters and the center marking rules of the large marker circles above them, the prescribed vertical and horizontal directions of the preset calibration plate in the first visible light image are determined. Therefore, the obtained target circles can be sorted according to the determined vertical and horizontal directions of the preset calibration plate.

[0144] It should be noted that the preset number of marker circles indicates the number of marker circles set on the preset calibration plate. When the preset calibration plate is set to a 9×11 circular array plane, the preset number of marker circles is 99; when the preset calibration plate is set to an 8×11 circular array plane, the preset number of marker circles is 88. No specific limitation is made here.

[0145] For example, when the preset calibration board is set to a 9×11 circular array plane, please refer to... Figure 4B and Figure 4C , Figure 4B It is an image of a heated preset calibration plate taken by an infrared thermal imager, and Figure 4B To quickly move the preset calibration plate to the infrared thermal imager while heating it to 50°C, and to ensure that the preset calibration plate is perpendicular to the optical axis of the infrared thermal imager when taking the image, the following steps are taken: Figure 4B As shown, after the preset calibration plate cools down, the above operation is repeated, and multiple sets of images are taken by placing the heated preset calibration plate in different tilt angles and positions within the field of view of the infrared thermal imager. Figure 4C This is an image of the heated preset calibration board captured by the left camera in the binocular camera module. Therefore, after determining the vertical and horizontal orientations of the preset calibration board, please refer to... Figure 4D , Figure 4D Yes Figure 4B The image shows a schematic diagram of each target circle after sorting, with the target circles sorted and marked from 0 to 98.

[0146] It should be noted that the sorted calibration points have already established a correspondence with the coordinate points in the world coordinate system. Therefore, when calibrating a stereo camera, the external parameters of the left and right cameras relative to the world coordinate system can be determined separately; when calibrating an infrared thermal imager, the external parameters of the infrared thermal imager relative to the world coordinate system can be determined separately.

[0147] It should be noted that, after extracting the feature points from the first visible light image, in order to improve calibration accuracy, this embodiment uses the cvFindCornerSubPix() function to perform sub-pixel processing on the calibration point coordinates of the extracted feature points to obtain more accurate sub-pixel coordinates. The obtained sub-pixel coordinates are then converted from CvPoint2D32f to a CvMat structure to improve the calculation accuracy of the intrinsic and extrinsic parameters of the infrared thermal imaging module and the binocular camera module.

[0148] It should be noted that after obtaining the mapping relationship between the 99 feature points on the calibration board and the corresponding points in the world coordinate system, this embodiment of the application performs calibration processing on the first visible light image according to the cv::CalibrateCamera() function and the 99 feature points to obtain the second intrinsic parameters of the left camera and the right camera respectively, and the second intrinsic parameters are in matrix form.

[0149] Please refer to Figure 5 , Figure 5 This is a schematic flowchart of step S350 provided in an embodiment of this application. In some embodiments of this application, the preset area ratio range includes a preset area ratio upper limit and a preset area ratio lower limit, so step S350 may specifically include, but is not limited to, steps S510 to S520.

[0150] Step S510: Calculate the area ratio based on the contour surface area and average contour area of ​​the candidate contour data to obtain the candidate ratio value;

[0151] Step S520: When the candidate ratio value is less than the upper limit of the preset area ratio value and the candidate ratio value is greater than the lower limit of the preset area ratio value, the candidate circle corresponding to the candidate ratio value is taken as the target circle.

[0152] In steps S510 and S520 of some embodiments, as shown in formula (3) below, the area ratio of the contour surface area and the average area value of the contour data of the candidate contour data is calculated to obtain the candidate ratio value. Among them, the upper limit of the preset area ratio can be any value such as 2 or 1.5, and the lower limit of the preset area ratio can be any value such as 0.5 or 0.8. The upper limit of the preset area ratio is greater than the lower limit of the preset area ratio, thereby avoiding the influence of contours that are not feature circles on this solution.

[0153]

[0154] Among them, A j This is used to represent the contour area value of the j-th target circle, where the value of j is in the range of [1, preset number of marked circles]. j Used to represent the candidate scale value of the j-th target circle.

[0155] It should be noted that, in order to ensure that the parameter acquisition process is consistent between the infrared thermal imager and the binocular camera during calibration, the same calibration rules and algorithms are still used to calibrate the infrared thermal imager in this embodiment. For the calibration of the infrared thermal imager in step S120, the specific settings in steps S310 to S360 of this application can be referred to, and will not be repeated here. That is, the calibration processing algorithm proposed in this embodiment is not only applicable to visible light binocular cameras, but also to infrared thermal imagers, and the calibration program is implemented in Visual C++ 6.0 and the computer vision library OpenCV 2.0.

[0156] It should be noted that after determining the number of feature points for the preset number of marked circles, at least 8 extracted feature points are randomly selected, and the intrinsic and extrinsic parameters are calculated according to the following formula (4). Here, the randomly selected feature point is denoted as P, where P is a point within the field of view of the infrared thermal imager and the binocular camera. Specifically, the coordinates of this feature point in the world coordinate system are denoted as P. W =(X W ,Y C Z W The coordinates of this feature point in the corresponding camera coordinate system are denoted as P. C =(X C ,Y C Z C The relationship between the coordinates (x, y) of the feature point in the image physical coordinate system and its coordinates in the image physical coordinate system is shown in the following formula (4). Among them, the selected camera coordinate system, image physical coordinate system and image pixel coordinate system are flexibly adjusted according to whether the calibrated process is an infrared thermal imager, a left camera or a right camera.

[0157]

[0158] Where dx and dy are the actual dimensions of a single pixel on the coordinate axes when the image physical coordinate system is transformed to the image pixel coordinate system; f is the camera focal length (which can be adjusted according to the selected infrared thermal imager, left camera, or right camera); (u0, v0) is the imaging origin of the image pixel coordinate system; f x For f / dx, f y Let f / dy be the intrinsic parameter matrix. Let the extrinsic parameter matrix M2 be R represents the rotation matrix, and T represents the translation vector.

[0159] It should be noted that the rotation matrix R has a dimension of 3×3, and the translation vector T has a dimension of 3×1.

[0160] It should be noted that the second intrinsic parameter includes the intrinsic parameter matrix M of the left camera. 1L The intrinsic parameter matrix M of the right camera 1W The first intrinsic parameter includes the intrinsic parameter matrix M of the infrared thermal imager. 1I First extrinsic parameter M 2I This is the extrinsic parameter matrix of the infrared thermal imager and the world coordinate system of the infrared thermal imaging module. The second extrinsic parameter includes the extrinsic parameter matrix M of the left camera. 2L And the extrinsic parameter matrix M of the right camera 2R Then the extrinsic parameter matrix M of the left camera 2L Let M represent the extrinsic transformation matrix of the left camera pixel coordinate system relative to the feature points in the world coordinate system, and let M be the extrinsic transformation matrix. 2L Including the left camera rotation matrix R land the left camera translation vector T l Similarly, the extrinsic parameter matrix M of the right camera 2R Let M represent the extrinsic transformation matrix of the right camera pixel coordinate system relative to the feature points in the world coordinate system, and let M be the extrinsic transformation matrix. 2R Including the right camera rotation matrix R r And the right camera translation vector T r .

[0161] For example, please refer to Figure 6 , Figure 6 This is a flowchart illustrating the calibration algorithms for a binocular camera module and an infrared thermal imaging module provided in this application embodiment. The calibration of the binocular camera module mainly includes the calibration of the left and right cameras, while the calibration of the infrared thermal imaging module mainly includes the calibration of the infrared thermal imager. The calibration algorithm flowchart may specifically include, but is not limited to, steps S610 to S6100.

[0162] Step S610, begin calibration;

[0163] Step S620: Read the first infrared image, the first visible light image from the left camera, and the first visible light image from the right camera, respectively;

[0164] Step S630: Extract corner points and contours from the first infrared image and the first visible light image respectively to obtain the initial circle and initial contour data;

[0165] Step S640: Determine the current number of target circles and the preset number of annotation circles. If the number of target circles is less than the preset number of annotation circles, proceed to step S651; if the number of target circles is equal to the preset number of annotation circles, proceed to step S660.

[0166] Step S651: Calculate the initial roundness value based on the initial contour surface area and the initial contour length;

[0167] Step S652: Select initial circles whose initial roundness value is less than or equal to a preset roundness threshold as candidate circles;

[0168] Step S653: Calculate the candidate ratio values ​​of the candidate circles, and filter out the target circles whose candidate ratio values ​​are less than the upper limit of the preset area ratio and greater than the lower limit of the preset area ratio.

[0169] Step S654: Again determine the number of target circles and the number of preset annotation circles. If the number of target circles is less than the number of preset annotation circles, proceed to step S620; if the number of target circles is equal to the number of preset annotation circles, proceed to step S660.

[0170] Step S660: Sort the calibration points corresponding to the target circle;

[0171] Step S670: Obtain the sub-pixel coordinates of each calibration point after sorting, and transform the structure of the sub-pixel coordinates;

[0172] Step S680: Calculate the intrinsic parameter matrix M of the infrared thermal imager. 1I The intrinsic parameter matrix M of the left camera 1L The intrinsic parameter matrix of the right camera is M. 1R The extrinsic parameter matrix M of the left camera 2L And the extrinsic parameter matrix M of the right camera 2R ;

[0173] Step S690, based on the intrinsic parameter matrix M of the infrared thermal imager 1I The second transformation parameters of the infrared thermal imager are determined by the extrinsic parameter matrix of the target camera and the extrinsic parameter matrix M of the left camera. 2L And the extrinsic parameter matrix M of the right camera 2R Determine the first conversion parameters for the binocular camera module;

[0174] Step S6100: Save the results and end the calibration process.

[0175] It should be noted that the embodiments of this application use a specially designed preset calibration board to calibrate the infrared thermal imager. The proposed calibration algorithm is applicable not only to the calibration of visible light binocular cameras but also to infrared thermal imagers, meaning the calibration algorithm has strong universality. Furthermore, the embodiments of this application introduce contour roundness and contour area ratio coefficients (i.e., candidate ratio values) into the calibration algorithm for the first time to screen feature points. Simultaneously, the use of a preset calibration board composed of a circular array further improves the calibration accuracy of the infrared thermal imager. Compared to related technologies for calibrating infrared thermal imagers, better calibration results can be obtained.

[0176] In step S140 of some embodiments, after solving the intrinsic and extrinsic parameters of the left and right binocular cameras respectively, it is necessary to calibrate the extrinsic parameter transformation relationship between the two cameras, that is, to determine the first transformation parameter between the two cameras. The first transformation parameter includes the rotation matrix R0 and translation matrix T0 of the right camera relative to the left camera.

[0177] Please refer to Figure 7 , Figure 7 This is a schematic flowchart of step S140 provided in an embodiment of this application. In some embodiments, step S140 may specifically include, but is not limited to, steps S710 to S740.

[0178] Step S710: Set the origin of the first world coordinate system based on the first visible light image, and obtain the first world coordinates based on the origin of the first world coordinate system; wherein, the first world coordinates are used to represent the coordinates of the center of the target circle in the first world coordinate system;

[0179] Step S720: Obtain the first pixel coordinates based on the first visible light image. The first pixel coordinates are used to represent the coordinates of the center of the target circle in the first pixel coordinate system of the left camera.

[0180] Step S730: Obtain the second pixel coordinates based on the first visible light image. The second pixel coordinates are used to represent the coordinates of the center of the target circle in the second pixel coordinate system of the right camera.

[0181] Step S740: Calculate the first transformation parameters based on the first world coordinates, the first pixel coordinates, and the second pixel coordinates.

[0182] In steps S710 to S740 of some embodiments, the origin of the first world coordinate system is set according to the first visible light image, and the first world coordinates are obtained according to the origin of the first world coordinate system, that is, the coordinates P of the center of the target circle in the first world coordinate system are randomly selected. According to the following formula (5), the first pixel coordinates are obtained from the first visible light image, and the first pixel coordinates are used to represent the coordinates of the center of the target circle in the first pixel coordinate system of the left camera, denoted as P. l According to the following formula (5), the second pixel coordinates are obtained from the first visible light image, and the second pixel coordinates are used to represent the coordinates of the center of the target circle in the second pixel coordinate system of the right camera, denoted as P. r .

[0183]

[0184] Therefore, the label R l and T l Let R be the rotation matrix and translation vector of the left camera relative to point P in the world coordinate system. r and T r Let R0 be the rotation matrix of the right camera relative to point P in the world coordinate system and T0 be the translation vector of the right camera. Then, according to the following formula (6), the transformation relationship between the pixel coordinate systems of the left and right cameras is obtained, that is, the first transformation parameter of the right camera relative to the left camera is obtained, and the first transformation parameter includes the rotation matrix R0 and the translation matrix T0 of the right camera to the left camera.

[0185]

[0186] It should be noted that, in this embodiment of the application, the rotation matrix and translation matrix of the left camera relative to the right camera can also be determined according to formula (6), which will not be elaborated here.

[0187] In step S150 of some embodiments, the four-dimensional thermal imaging model constructed in this application fuses the three-dimensional point cloud and the infrared image captured by the infrared thermal imager through calibration. Based on the relationship between the world coordinate system, camera coordinate system, image coordinate system, and pixel plane coordinate system in a visible light camera (i.e., a binocular camera), this application can analogously establish the relationship between the binocular camera coordinate system, the infrared thermal imager coordinate system, the infrared image coordinate system, and the infrared pixel plane coordinate system. This completes the transformation from the three-dimensional point cloud in the binocular coordinate system to the pixel plane coordinate system of the infrared thermal imager, establishing a one-to-one mapping relationship between the spatial points in the three-dimensional point cloud and the RGB information and temperature information of the pixels in the two-dimensional thermal image, thereby constructing the four-dimensional thermal imaging model. Therefore, in this application embodiment, a mapping relationship is first established between the left camera in the binocular camera module and the infrared thermal imager, that is, the left camera is used as the target camera of the binocular camera module.

[0188] Specifically, firstly, based on formula (5), the transformation relationship between the coordinate system of the left camera in the binocular camera module and the world coordinate system can be obtained, and then the coordinate system O of the infrared thermal imager can be obtained. I X I Y I Z I With world coordinate system O W X W Y W Z W The conversion relationship between them is shown in the following formula (7).

[0189]

[0190] Among them, (X) W ,Y W Z W ) is the world coordinate system O W X W Y W Z W Any point in (X) I ,Y I Z I (O) represents the coordinate system of the infrared thermal imager. I X I Y I Z I In and (X) W ,Y W Z W For the corresponding point, R1 is the rotation matrix from the world coordinate system to the infrared thermal imager coordinate system obtained according to formula (4), and T1 is the translation vector from the world coordinate system to the infrared thermal imager coordinate system obtained according to formula (4). Similarly, the left camera coordinate system O can be obtained. L X L Y L ZL With world coordinate system O W X W Y W Z W The conversion relationship between them is shown in the following formula (8).

[0191]

[0192] Among them, (X) L ,Y L Z L O is the coordinate system of the left camera. L X L Y L Z L For any point in the coordinate system, R2 is the rotation matrix from the world coordinate system to the left camera coordinate system obtained according to formula (4), and T2 is the translation vector from the world coordinate system to the left camera coordinate system obtained according to formula (4). Furthermore, the transformation of formula (8) is as follows: formula (9).

[0193]

[0194] Then, based on formulas (7) and (9), the transformation relationship from the left camera coordinate system to the infrared thermal imager coordinate system can be derived as shown in formula (10).

[0195]

[0196] Therefore, according to formula (10), the second transformation parameter, i.e., the external parameter transformation relationship from the left camera coordinate system to the infrared thermal imager coordinate system, can be obtained, which includes the rotation matrix R from the left camera coordinate system to the infrared thermal imager coordinate system. li Translation vector T li And R li and T li The specific representation is shown in the following formula (11).

[0197]

[0198] Therefore, by simultaneously capturing images of the preset calibration plate within the field of view of both the infrared thermal imager and the left camera of the binocular camera, and extracting the captured images while ensuring that the preset calibration plate is perpendicular to the optical axis of the infrared thermal imager, R can be calculated according to formula (11). li and T li .

[0199] It should be noted that a mapping relationship can also be established between the right camera in the binocular camera module and the infrared thermal imager to construct a four-dimensional thermal image model, but this will not be specifically limited or elaborated here.

[0200] It should be noted that, based on numerous experiments, an example of the calibration result obtained by the calibration processing method of this application for calibrating the left and right cameras of a stereo camera is shown in Table 1 below.

[0201]

[0202] Table 1

[0203] Table 1 includes the second intrinsic parameters (intrinsic parameter matrix M1) and second extrinsic parameters (distortion coefficients [k1,k2,p1,p2]) of the left and right cameras, the rotation matrix R0 and translation matrix T0 of the right camera relative to the left camera, and the reprojection errors of the left and right cameras, respectively. Here, k1 and k2 refer to radial distortion parameters, which occur during the transformation from the camera coordinate system to the physical coordinate system. p1 and p2 refer to tangential distortion parameters, which are caused by the lens not being perfectly parallel to the image. The reprojection error refers to the difference between the projection of a real 3D point onto the image plane (i.e., the pixel in the image) and the reprojection (the calculated virtual pixel). Reducing the reprojection error can effectively improve calibration accuracy. As shown in Table 1, when the left and right cameras each acquire 7 visible light images for calibration, the average reprojection error is less than 1 pixel, reaching the sub-pixel level, thus practically proving that this application can effectively improve the accuracy of the calibration results.

[0204] It should be noted that, based on numerous experiments, an example of the calibration result obtained by calibrating an infrared thermal imager according to the calibration processing method of the embodiments of this application is shown in Table 2 below.

[0205]

[0206] Table 2

[0207] Table 2 records the intrinsic parameter matrix M of the infrared thermal imager. 1I The rotation matrix R from the left camera coordinate system to the infrared thermal imager coordinate system li Translation vector T li The reprojection error of the infrared thermal imager is shown in Table 2. As can be seen from Table 2, when the infrared thermal imager acquires seven infrared images for calibration, the average reprojection error of the calibration results is also less than one pixel, reaching sub-pixel level. Therefore, although the resolution and texture features of infrared images from infrared thermal imagers are inferior to those of visible light images, the calibration algorithm provided in this application can still achieve sub-pixel calibration levels.

[0208] In step S160 of some embodiments, after the calibration of the binocular camera and infrared thermal imager is completed, the coded sinusoidal grating fringe pattern is projected onto the object under test, and the binocular camera is simultaneously triggered to capture a second visible light image. Therefore, in order to obtain the depth and contour information of the object under test for three-dimensional reconstruction, this application uses Phase Shing Profilometry (PMP), which is suitable for static reconstruction and has high reconstruction accuracy, to acquire three-dimensional data.

[0209] Please refer to Figure 8 , Figure 8 This is a schematic flowchart of step S160 provided in an embodiment of this application. In some embodiments of this application, step S160 may specifically include, but is not limited to, steps S810 to S740.

[0210] Step S810: Obtain the average illumination intensity data and intensity modulation data of the second visible light image;

[0211] Step S820: Calculate the principal phase value based on the average illumination intensity data, intensity modulation data, and the preset first phase difference to obtain the first principal phase value;

[0212] Step S830: Calculate the principal phase value based on the average illumination intensity data, intensity modulation data, and the preset second phase difference to obtain the second principal phase value;

[0213] Step S840: Calculate the phase based on the preset frequency ratio, the first phase principal value, and the second phase principal value to obtain the target phase;

[0214] Step S850: Perform three-dimensional reconstruction based on the target phase, the second intrinsic parameter, the first transformation parameter, and the second visible light image to obtain a three-dimensional point cloud model.

[0215] It should be noted that when acquiring three-dimensional point cloud data, this application first uses the phase shift method to calculate the phase principal value of the three-dimensional information of the surface of the object being measured, and then performs phase expansion on the obtained phase principal value to obtain a continuous absolute phase, so as to determine the target phase of each pixel in the image, and determine the three-dimensional point cloud data based on the target phase.

[0216] In step S810 of some embodiments, please refer to Figure 9A , Figure 9AThis is a schematic diagram of the phase information acquisition process during 3D reconstruction. First, a set of grating images 912 with sinusoidal intensity distribution is projected onto the object under test 911 using a DLP grating projection optical engine 910. Then, the left and right cameras 913 of the binocular camera module are controlled to capture the grating fringes on the surface of the object under test 911 that have undergone deformation. A second visible light image is obtained based on the grating images acquired by the left and right cameras 913, and the second visible light image is sent to the controller 914 for grating fringe analysis. Specifically, multiple phase-shifted sinusoidal fringe patterns with spatial intensity variations are projected onto the surface of the object under test. The distorted fringe distribution captured by the left and right cameras can be expressed as shown in the following formula (12).

[0217] I m (x,y)=B(x,y)+C(x,y)cos[φ(x,y)+δ m (12)

[0218] Among them, I n Let I represent the phase shift of pixel (x, y) in the nth sinusoidal fringe pattern, and I n The value range of is [0, 2π), n = 1, 2, ..., M-1, where M is the phase shift step number and M ≥ 3; B(x,y) represents the average illumination intensity data related to pattern brightness and background lighting; C(x,y) represents the intensity modulation data related to pattern contrast and surface reflectivity, and B(x,y) and C(x,y) are preset known data; δ m This represents the phase shift difference, i.e., δ m =2πi / M.

[0219] In step S820 of some embodiments, this application uses the phase-shifting method to calculate the principal phase value wrapped by the grating image of the surface of the object under test at the same frequency. Since the wrapped phase (i.e., the principal phase value) is unique only within one phase period and lacks spatial continuity, this application simultaneously uses the multi-frequency heterodyne method to expand the principal phase value to obtain the absolute phase (i.e., the target phase) with a unique value globally. Please refer to... Figure 9B and Figure 9C , Figure 9B This is a schematic diagram of a second visible light image of the object under test, captured by the left camera and illuminated by projection grating patterns at different fringe frequencies. Figure 9C This is a schematic diagram of a second visible light image of the object under test, captured by the right camera, illuminated by projection grating patterns at different fringe frequencies. Specifically, at the instant the DLP grating projector projects the grating fringe pattern onto the object, the binocular cameras are triggered to simultaneously capture images of the object. Figure 9B and Figure 9CThe first sixteen images in the image correspond to four-step phase shift patterns with frequencies of 1, 3, 9, and 28, respectively, and the last eight images correspond to an eight-step phase shift pattern with a frequency of 85. Specifically, when the M-step multi-frequency heterodyne method is used for calculation, the principal phase value (i.e., the wrapping phase) is determined according to the following formula (13).

[0220]

[0221] Where φ(x,y) represents the phase principal value to be determined, which is either the first phase principal value or the second phase principal value.

[0222] In step S830 of some embodiments, the sinusoidal grating fringes generated by the standard M-step phase-shifting method may not achieve the ideal output effect due to the Gamma nonlinearity of the DLP projector itself, causing the fringe pattern captured by the camera to lose its sinusoidal characteristics, thereby affecting the reconstruction accuracy. Therefore, this embodiment uses a four-step phase-shifting method to obtain the first phase principal value corresponding to the low-frequency fringes. The first phase difference corresponding to the four-step phase-shifting method is π / 2, meaning that at the same frequency, four patterns with different fringe distributions will be generated. Simultaneously, this embodiment uses an eight-step phase-shifting method to obtain the second phase principal value corresponding to the high-frequency fringes. The second phase difference corresponding to the eight-step phase-shifting method is π / 4, meaning that at the same frequency, eight patterns with different fringe distributions will be generated. Thus, this application, by using low-frequency fringes to assist high-frequency fringes in determining the fringe order and then unwrapping the phase of the high-frequency fringes, can effectively reduce the phase error distribution of the fringe pattern caused by the nonlinear response of the digital projector, thereby improving the reconstruction accuracy of the measured object.

[0223] In step S840 of some embodiments, the principle of multi-frequency time phase unwrapping is shown in the following formula (14). Wherein, f h / f l The frequency f of adjacent high-frequency stripes h and the frequency f of the low-frequency stripes l The ratio, i.e., the preset frequency ratio; Φ h (x,y) represents the absolute phase distribution of the high-frequency fringes, Φ l (x,y) represents the absolute phase distribution of the low-frequency fringes.

[0224] Φ h (x,y)=(f l / f l )Φ l (x,y) (14)

[0225] Subsequently, in order to obtain a target phase with a single frequency and continuous distribution, this application performs phase expansion on the principal value of the phase that is not unique within the phase period, that is, by the relationship between the absolute phase obtained by unwrapping and the wrapped phase, the absolute phase of the high-frequency stripe is further obtained according to the following formula (15), that is, the target phase.

[0226]

[0227] Where Round[] represents finding the closest integer value, and represents the fringe order of the fringe. According to formula (15), formula (13) can be phase expanded to solve the high-frequency phase φ. h (x,y),Φ h (x, y) represents the absolute phase of the high-frequency phase, and the fringe frequencies decrease from high to low. Therefore, when calculating the principal phase value of the fringe frequency using the multi-frequency fringe method, the target phase of the lowest frequency fringe frequency is directly equal to its principal phase value, while the target phase of the high-frequency fringe is calculated according to formula (15). The multi-frequency heterodyne method used in this application embodiment can independently calculate the phase information of each pixel, which can better improve the measurement accuracy and anti-interference ability.

[0228] In step S850 of some embodiments, the unfolded phases calculated by the left and right binocular cameras are subjected to phase feature matching, and the three-dimensional spatial coordinates of the measured object and the corresponding three-dimensional point cloud data are calculated using the triangulation principle. Please refer to Figure 10 , Figure 10 This is a schematic diagram of the image used in this application to solve the target phase using the multi-frequency heterodyne method. Figure 10 Subfigures (1)-(4) are four-step phase-shifting fringe projections with an initial phase of 0 and fringe frequencies of 1, 3, 9, and 28, respectively. Subfigure (5) is an eight-step phase-shifting fringe pattern with a fringe frequency of 85. Subfigures (6)-(10) are the principal phase values ​​of the corresponding subfigures (1)-(5). Subfigures (11)-(14) are the target phase diagrams with frequencies of 1, 3, 9, and 28, respectively. Subfigure (15) is a three-dimensional point cloud model after phase matching and three-dimensional coordinate calculation.

[0229] It should be noted that, since the multi-frequency heterodyne method can improve the accuracy and reliability of absolute phase unwrapping, this application uses five different frequency stripe coding patterns of 1, 3, 9, 28, and 85 for projection by the digital projector, based on the horizontal resolution of the DLP projector and the number of pixels occupied by each sinusoidal stripe in the stripe pattern. This enables phase calculation of the grating stripe pattern covering the surface of the object being measured, which is acquired by the binocular camera, to obtain a unique and continuous target phase value across the entire field of view.

[0230] Please refer to Figure 11 , Figure 11This is a schematic flowchart of step S850 provided in an embodiment of this application. In some embodiments of this application, the first conversion parameter includes a first conversion sub-parameter and a second conversion sub-parameter. The first conversion sub-parameter is used to represent the conversion parameter from the left camera to the right camera, and the second conversion sub-parameter is used to represent the conversion parameter from the right camera to the left camera. Therefore, step S850 may specifically include, but is not limited to, steps S1110 to S1170.

[0231] Step S1110: When the target camera is the left camera, a left projection image is obtained based on the projection of the second visible light image onto the image plane of the left camera, and a right projection image is obtained based on the projection of the second visible light image onto the image plane of the right camera. The left projection image includes the pixels to be matched.

[0232] Step S1120: The coordinates of the left outer pole are calculated based on the second intrinsic parameter and the first transformation sub-parameter. The coordinates of the left outer pole are used to represent the coordinates of the left outer pole center of the left camera in the left camera coordinate system.

[0233] Step S1130: Calculate the coordinates of the right outer pole based on the second intrinsic parameter and the second transformation sub-parameter. The coordinates of the right outer pole are used to represent the coordinates of the right outer pole center of the right camera in the right camera coordinate system.

[0234] Step S1140: Calculate the slope based on the coordinates of the pixel to be matched and the coordinates of the left outer epipolar point to obtain the slope of the left outer epipolar line; the coordinates of the pixel to be matched are used to represent the coordinates of the pixel to be matched in the left camera coordinate system.

[0235] Step S1150: Construct the polar equation based on the preset slope conversion formula, the slope of the left outer polar line, and the coordinates of the right outer polar point to obtain the right outer polar line equation;

[0236] Step S1160: Select points based on the target phase, the right outer epipolar equation, and the left projection image to determine the initial matching point;

[0237] Step S1170: Perform three-dimensional reconstruction based on the slope of the left outer epipolar line and the initial matching points to obtain a three-dimensional point cloud model.

[0238] In steps S1110 to S1170 of some embodiments, the present application embodiments match the image pixels acquired by the left and right cameras based on the target phase value and the epipolar constraint method. Successfully matched pixel pairs are marked for the next step of 3D point cloud calculation. Therefore, please refer to... Figure 12 , Figure 12This is a schematic diagram of the epipolar slope-absolute phase matching process provided in an embodiment of this application. This application utilizes the triangulation principle to calculate the three-dimensional coordinates of all points on the surface of the object under test within the field of view in a spatial coordinate system, in order to obtain the three-dimensional shape and position information of the object under test in space. Specifically, it is assumed that the pixel point P1(X1,Y1,Z1) to be matched on the object under test is located on the image plane I of the binocular camera. l and I r The projection points on are p l (u l ,v l ) and p r (u r ,v r ), and the projection point p l (u l ,v l ) and p r (u r ,v r ( ) represents a pair of precisely matched points. The transmission centers of the left and right cameras are denoted as O1 and O2 respectively. r and O r Then the line O l p l and O r p r The intersection of the two points is the spatial location of point P1. Assuming that the left camera coordinate system is located at the origin of the world coordinate system and has no rotation, the three-dimensional coordinates of the spatial point are derived from the spatial transformation matrices R0 and T0 obtained from the aforementioned dual-camera calibration, as shown in the following formula (16).

[0239]

[0240] Among them, f l The focal length of the left camera, f r The focal length of the right camera, and the spatial transformation matrix from the right camera to the left camera. Translation vector from right camera to left camera P l (X l ,Y l ) and P r (X r ,Y r The coordinates are the physical coordinates of the pixels under the stereo camera. In this embodiment, the subscript l represents the coordinates under the left camera, and the subscript r represents the coordinates under the right camera. Therefore, this application uses epiploic geometry constraints to find matching points on the images of the left and right cameras.

[0241] Specifically, suppose (x l ,y l ,z l ) and (xr ,y r ,z r Let A, B, and C be points on the coordinate systems of the left and right cameras, respectively. Then, based on the calibration of the stereo cameras, the first transformation sub-parameters are obtained, and the first transformation sub-parameters include the rotation matrix R for transforming the left camera to the right camera. lr Translation vector T lr And the rotation matrix R for switching from the right camera to the left camera. rl Translation vector T rl And the relational equation shown in formula (17) is obtained.

[0242]

[0243] Among them, the rotation matrix for transforming the left camera to the right camera. Translation vector Rotation matrix for switching from right camera to left camera Translation vector After that, O l and O r The left outer pole center e is obtained by the intersection of the line connecting the left and right image planes. l The coordinates of the left outer pole and the right outer pole center e r The coordinates of the right outer pole are calculated as shown in formula (18).

[0244]

[0245] Therefore, the slope of the epipolar line for each pixel to be matched in the left and right images can be calculated from the pole of its corresponding epipolar plane. Assume the left outer epipolar line l on the left image plane... l The slope is K l And the slope of the transformation from the left image plane to the right image plane is K. lr Right outer polar line l on the right image plane r The slope is K r And the slope of the transformation from the right image plane to the left image plane is K. rl When based on the pixel P to be matched in the left image l With the left outer pole center e l The coordinates are used to calculate the corresponding slope K of the left outer polar line. l The points on the left image are transformed to the right image according to the following slope transformation formula (19), and the transformation slope K is obtained. lr Similarly, points on the right image can be transformed to the left image using the slope transformation formula (19), thus obtaining the transformation slope K. rl .

[0246] (19)

[0247] Then, at any point P on the left image...l (X l ,Y l After that, the slope K of the left outer polar line can be obtained. l and right outer pole center e r The equation for the right external polar line is obtained.

[0248] Please refer to Figure 13 , Figure 13 This is a schematic flowchart of step S1170 provided in an embodiment of this application. In some embodiments of this application, step S1170 may specifically include, but is not limited to, steps S1310 to S1160.

[0249] Step S1310: Select matching points based on the initial matching points and the preset neighborhood threshold to obtain candidate matching points and the coordinates of the right outer pole of the candidate matching points;

[0250] Step S1320: Calculate the slope of the right outer epipolar coordinates and the coordinates of the candidate matching points to obtain the slope of the right outer epipolar line; the coordinates of the candidate matching points are used to represent the coordinates of the candidate matching points in the right camera coordinate system.

[0251] Step S1330: Calculate the difference between the slopes based on the slope conversion formula, the slopes of the left outer polar line and the right outer polar line to obtain the slope difference.

[0252] Step S1340: When the slope difference is less than the preset difference threshold, the candidate matching point is used as the intermediate matching point.

[0253] Step S1350: Determine the target matching point based on the target phase of the intermediate matching point;

[0254] Step S1360: Perform 3D reconstruction based on the target matching points to obtain a 3D point cloud model.

[0255] In steps S1310 to S1360 of some embodiments, the target phase value and P are retrieved on the transformed right outer polar equation. l The point with the smallest difference is denoted as the initial matching point P. r0 Pixels are acquired within a preset neighborhood threshold range of the initial matching point; the data for the preset neighborhood threshold is not specifically limited. For example, setting P... r0 If the preset neighborhood threshold is a circular region with a diameter of 10 pixels, then multiple candidate matching points are obtained. Within this preset neighborhood threshold range, the actual right-side slope K of each pixel is... r The slope can be calculated by using the coordinates of the right outer pole obtained from formula (18). According to formula (19), the slope K of the left outer pole is calculated. l The transformation slope K obtained by performing slope transformation lrThe slope difference between the right outer epipolar slope of the candidate matching point and the slope difference is calculated to obtain the slope difference value, and the slope difference value is the absolute value. Then, when the slope difference value is less than the preset difference threshold, the candidate matching point is used as the intermediate matching point. For example, if the preset difference threshold is 0.5, the slope difference value less than 0.5 is retained as the intermediate matching point, and the pixel points greater than or equal to 0.5 are deleted. Among the intermediate matching points left after filtering, the pixel point with the smallest difference from the target phase value of the pixel point to be matched in the left image is recorded as the target matching point, which is the accurate matching point. Finally, based on the stereo matching process of the absolute phase (i.e. the target phase) described above, and according to the formula (16) to determine the three-dimensional coordinates of the spatial point, the three-dimensional point cloud model of the measured object is obtained.

[0256] Please refer to Figure 14 , Figure 14 This is a flowchart of the absolute phase matching algorithm provided in an embodiment of this application. The specific process includes steps S1410 to S14100.

[0257] Step S1410, matching begins;

[0258] Step S1420: Determine the extrinsic parameter R for switching from the left camera to the right camera. lr and T lr And the external parameter R for switching from the right camera to the left camera rl and T rl ;

[0259] Step S1430: When the target camera is the left camera, calculate the distance between each pixel in the left image and the left outer epipolar center e. l The slope of the left outer polar line;

[0260] Step S1440: According to the slope conversion formula, the slope of the left epipolar line where the pixel to be matched in the left image is located is converted to the right image to obtain the conversion slope K. lr ;

[0261] Step S1450, based on the transformation slope K lr and right outer pole center e r Calculate the equation of the transformed right outer polar line;

[0262] Step S1460: Select the initial matching point with the smallest difference between the target phase value and the pixel to be matched in the left image on the converted right outer epipolar line.

[0263] Step S1470: Based on the right outer epipolar slope and transformation slope K of each pixel within the preset neighborhood threshold... lr The slope difference is obtained by performing a difference calculation.

[0264] Step S1480: Determine the preset difference threshold and the slope difference. If the slope difference is less than the preset difference threshold, proceed to step S1491; if the slope difference is greater than or equal to the preset difference threshold, proceed to step S1492.

[0265] Step S1491: Select the pixel with the smallest difference in target phase value from the remaining intermediate matching points after filtering and record it as the target matching point, and then execute step S14100.

[0266] Step S1492: Delete the pixel and proceed to step S14100;

[0267] Step S14100: Matching ends.

[0268] Please refer to Figure 15 , Figure 15 This is a schematic flowchart of step S170 provided in an embodiment of this application. In some embodiments of this application, step S170 may specifically include, but is not limited to, steps S1510 to S1550.

[0269] Step S1510: Obtain the initial camera coordinates of the 3D point cloud in the 3D point cloud model. The initial camera coordinates are used to represent the coordinates of the 3D point cloud in the camera coordinate system of the target camera.

[0270] Step S1520: Perform a first coordinate transformation on the initial camera coordinates according to the second transformation parameters to obtain the first infrared coordinates. The first infrared coordinates are used to represent the coordinates of the three-dimensional point cloud in the infrared thermal imager coordinate system of the second infrared image.

[0271] Step S1530: Perform a second coordinate transformation based on the first intrinsic parameter and the first infrared coordinate to obtain the second infrared coordinate. The second infrared coordinate is used to represent the coordinates of the three-dimensional point cloud in the image coordinate system of the second infrared image.

[0272] Step S1540: Perform a third coordinate transformation based on the first intrinsic parameter and the second infrared coordinates to obtain the third infrared coordinates. The third infrared coordinates are used to represent the coordinates of the three-dimensional point cloud in the pixel coordinate system of the second infrared image.

[0273] Step S1550: Perform temperature mapping processing based on the third infrared coordinates and the three-dimensional point cloud model to obtain a four-dimensional thermal image model of the object under test.

[0274] In steps S1510 to S1550 of some embodiments, in order to achieve the fusion of the three-dimensional point cloud with the two-dimensional infrared image, please refer to... Figure 16 , Figure 16This is a three-dimensional schematic diagram of the temperature mapping process provided in the embodiments of this application. Specifically, since the calibration of the binocular vision system and the infrared thermal imager has already been completed, the internal parameters of the infrared thermal imager... If it is known, then by analogy with formula (4), the three-dimensional point cloud model with the three-dimensional shape information of the measured object stored under the left camera can be converted into the two-dimensional second infrared image under the infrared thermal imager according to the following formula (20). The three-dimensional point cloud model includes multiple point cloud data.

[0275]

[0276] Among them, O L X L Y L Z L Let O be the coordinate system of the left camera. I X I Y I Z I For the infrared thermal imager coordinate system, o I x I y I Let X be the image coordinate system of the infrared thermal imager, and let X be the point on it. I ,y I ), u I v I Let U be the pixel coordinate system of the infrared thermal imager, and let the points on it be represented as (u I ,v I p is the corresponding imaging point on the second infrared image found through coordinate transformation, i.e., its coordinates are the first infrared coordinates. Based on the first intrinsic parameter and the first infrared coordinates, a second coordinate transformation is performed, and the coordinates of point p in the image coordinate system of the infrared thermal imager are (x...). Ip ,y Ip This yields the second infrared coordinates. Then, a third coordinate transformation is performed based on the first intrinsic parameter and the second infrared coordinates to obtain the coordinates (u) in the pixel coordinate system of the infrared thermal imager. Ip ,v Ip ), that is, obtaining the third infrared coordinates. Furthermore, f I (O I o I The distance between the lines is the focal length of the infrared thermal imager.

[0277] The specific derivation process is as follows: First, according to the following formula (21), the three-dimensional point cloud stored in the left camera is transformed into the infrared thermal imager coordinate system O. I X I Y I Z I Down.

[0278]

[0279] Among them, P LP (X LP ,Y LP Z LP (X) represents a point stored in the 3D point cloud of the left camera coordinate system, which is the initial camera coordinate. IP ,Y IP Z IP ) is P LP After the first coordinate system transformation, at O I X I Y I Z I The corresponding point below. Then, (X) IP ,Y IP Z IP Transform to image coordinate system o I x I y I Below. (e.g.) Figure 16 As shown, the image plane coordinate system of the second infrared image is a two-dimensional coordinate system established on the imaging plane of the infrared thermal imager, with the origin o. I x is the intersection of the imaging plane of the infrared thermal imager and its optical axis. I y I Axis and X I Y I Parallel. Since this transformation process satisfies the similarity theorem of triangles, the transformation is carried out according to the following formula (22).

[0280]

[0281] Among them, (X) I ,Y I Z I (x) is a three-dimensional point in the infrared thermal imager coordinate system. I ,y I () is the corresponding two-dimensional point in the plane coordinate system of the infrared thermal imager. According to the following formula (23), (X) LP ,Y LP Z LP ) Convert to o I x I y I In a coordinate system.

[0282]

[0283] Where, p Ip (x Ip ,y Ip ) is P LP After a second coordinate system transformation, in the image coordinate system o I x Iy I The corresponding points are shown below, and the unit in this coordinate system is millimeters (mm). Furthermore, since infrared thermal images are formed using pixels as the basic unit, it is necessary to convert the infrared image coordinate system (o) which uses length as the basic unit. I x I y I Transformation to the pixel coordinate system u of the infrared image I v I Below. The origin of the pixel coordinate system of the infrared image is the upper left corner of the imaging plane. Let the physical size of the pixels on the imaging plane of the infrared thermal imager be x. I y I dx in the axial direction I dy I Origin of infrared image plane coordinates I The translation amounts to the pixel coordinate origin are u 0I v 0I (Unit is pixels), and the specific conversion formula is shown in formula (24) below.

[0284]

[0285] Where s is the tilt factor of the infrared thermal imager, which is usually set to 0. Therefore, according to the following formula (25), p can be... Ip (x Ip ,y Ip Transform to coordinate system u I v I The following is obtained, and a specific matrix form is obtained.

[0286]

[0287] Where, p Ip (u Ip ,v Ip ) is (x Ip ,y Ip The corresponding points in pixels

[0288] Therefore, by combining equations (21), (23), and (25), we can obtain the following equation (26), which is the 3D point cloud P in the camera coordinate system. LP (X LP ,Y LP Z LP After three coordinate system transformations, the infrared pixel p in the image pixel coordinate system of the infrared thermal imager is obtained. Ip (u Ip ,v Ip Establish a one-to-one mapping relationship.

[0289]

[0290] It should be noted that, for example, when selecting a metal tank, a metal block, and an LED array with applied voltage as the object under test, please refer to [reference needed]. Figures 17A to 17C , Figure 17A This is a schematic diagram of the prototype structure of the selected object to be tested. Figure 17B To Figure 17A A schematic diagram of the structure of the 3D point cloud model of the object under test after passing through the 3D reconstruction module. Figure 17C To Figure 17A A schematic diagram of the structure of a four-dimensional thermal imaging model of the object under test after temperature mapping processing. Figure 17C The mapping relationship between the three-dimensional spatial coordinates of the selected spatial point and the two-dimensional pixel coordinates of the infrared image is shown in Table 3 below.

[0291]

[0292] Table 3

[0293] Therefore, by reading the temperature information contained in each pixel of a two-dimensional infrared image, the temperature data of the three-dimensional point cloud spatial points can be obtained. After three-dimensional reconstruction, a three-dimensional point cloud file is output in .txt format, which can be displayed in Geomagic Studio 2013. Subsequently, temperature mapping is performed on the .txt 3D point cloud data, outputting a .txt format four-dimensional thermal imaging model, which can be displayed in Geomagic Studio 2013 as a point cloud with temperature information. Furthermore, the depth measured by the four-dimensional thermal imaging model constructed in this application has an error of 3.027692% compared to the depth obtained by physical ranging, and the error is -0.107048%, which fully verifies the accuracy of the four-dimensional thermal imaging model measurement method of this application.

[0294] It should be noted that, in order to verify the reconstruction accuracy of the four-dimensional thermal imaging model generation method proposed in this application, the surface depth and local surface area of ​​the object under test can be used as evaluation indicators. For example, a customized heating stage is used, which is made of Al-Mg-Si alloy through hard anodizing, and has three uniformly arranged heating tubes made of nickel-chromium and iron-chromium-aluminum heating wires inside to control the heating temperature of the stage. Specifically, such as... Figure 18A As shown, the dimensions are 250×200 (mm). 2 Three rectangular heating platforms of varying heights are placed inside a casing made of asbestos insulation board. Vertical cylindrical recesses, each 30mm in diameter, are drilled into the surface of the heating platform from top to bottom. These recesses are spaced 3mm deep, with depths increasing in increments of 3mm, resulting in over 30 cylindrical recesses ranging from 3mm to 90mm in depth. Then, as... Figure 18BAs shown, by using the four-dimensional thermal image model generation method proposed in this application, a point cloud containing three-dimensional coordinates and RGB information of spatial points can be obtained. The point cloud can be opened in the point cloud post-processing software Geomagic Studio 2013 to obtain the depth measurement results of the thermal image model.

[0295] Furthermore, the depth measurement results of the cylindrical recess after converting the thermal imaging model from point cloud to polygon and performing hole filling operations are shown in Table 4 below. Therefore, it can be found that the measured depth of the constructed four-dimensional thermal imaging model is within 0.9 mm of the actual depth. Compared with the ideal depth of the cylindrical recess cavity of the tested heating stage, the average absolute percentage error (MAPE) of the measured depth is less than 2.06%. The error bars in the table represent the maximum and minimum errors between 10 measurement results and their arithmetic mean; an error bar (-) indicates a negative value, and an error bar (+) indicates a positive value.

[0296]

[0297] Table 4

[0298] It should be noted that in practical applications, after converting the four-dimensional thermal image point cloud into polygon data, the surface area is calculated by meshing. The surface area measurement results for the cylinder are shown in Table 5 below. As can be seen from Table 5, compared to the ideal surface area of ​​the concave cavity of a cylinder, when the length-to-diameter ratio L / d (ratio of cylinder height to bottom surface dimension) of the concave cavity is less than 13, the difference between the measured surface area of ​​the reconstructed model and the theoretical surface area of ​​the concave region is within 125 mm. 2 Within this range. Furthermore, when the aspect ratio L / d of the concave cavity is greater than or equal to 13, the difference between the measured surface area of ​​the reconstructed model and the theoretical surface area of ​​the concave region increases significantly. This is because the geometry of the cylindrical concave cavity prevents the binocular camera from capturing the obscured area. Overall, when the aspect ratio L / d of the cylindrical concave cavity is relatively small, the reconstruction accuracy of the four-dimensional thermal imaging model generation method proposed in this application can better meet the application requirements and provide more accurate three-dimensional contour information.

[0299] Aspect Ratio L / d <![CDATA[Actual surface area of the cylinder indentation (mm 2 )]]> <![CDATA[Surface area of cylinder depression measurement (mm 2 )]]> 0.1 989.60169 983.72839 0.2 1272.34503 1257.71755 0.3 1555.08836 1540.36676 0.4 1837.8317 1823.92622 0.5 2120.57504 2106.40308 0.6 2403.31838 2393.91846 0.7 2686.06172 2653.32125 0.8 2968.80506 2923.62268 0.9 3251.5484 3147.49285 1.0 3534.29173 3475.68506 1.1 3817.03507 3781.59141 1.2 4099.77841 3977.67403 1.3 4382.52175 4164.18393 1.4 4665.26509 4490.12482 1.5 4948.00843 4661.36538 1.6 5230.75177 4881.61955 1.7 5513.49511 5044.29715 1.8 5796.23845 5230.39951 1.9 6078.98178 5217.59566 2.0 6361.72512 5524.94581 2.1 6644.46846 5588.21353 2.2 6927.2118 5661.69523 2.3 7209.95514 6032.85439 2.4 7492.69848 6147.72215 2.5 7775.44182 6323.76142 2.6 8058.18516 6481.06115 2.7 8340.9285 6525.74484 2.8 8623.67183 6704.9459 2.9 8906.41517 6836.2183 3.0 9189.15851 6937.58921

[0300] Table 5

[0301] Furthermore, experimental data demonstrates that the reconstruction speed of the four-dimensional thermal imaging model constructed in this application can generally be completed within 5 seconds, reaching the application standard for real-time reconstruction.

[0302] In some embodiments, this application also provides a four-dimensional thermal image model generation apparatus for performing the four-dimensional thermal image model generation method of any of the above embodiments. The apparatus includes a controller, a binocular camera module, a binocular camera module, and a preset calibration component.

[0303] In some embodiments, this application also provides a four-dimensional thermal image model generation apparatus, the apparatus comprising:

[0304] A controller for executing a four-dimensional thermal imaging model generation method as provided in the embodiments of this application;

[0305] A binocular camera module, which is connected to the controller, is used to acquire visible light images;

[0306] The infrared thermal imaging module is connected to the controller and is used to acquire infrared images.

[0307] Preset calibration components are used to calibrate binocular camera modules and infrared thermal imaging modules.

[0308] It should be noted that, please refer to Figure 19 After calibrating the infrared thermal imager 1910 of the infrared thermal imaging module and the binocular camera 1920 of the binocular camera module according to the preset calibration components, the DLP projector of the binocular camera module allows the binocular camera 1920 and the infrared thermal imager 1910 to capture the deformed grating fringe pattern projected onto the surface of the object under test 1930. The relative positions between the infrared thermal imager 1910 and the binocular camera 1920 remain unchanged. Then, the images acquired by the binocular camera 1920 and the infrared thermal imager 1910 are uploaded to the controller 1940. The controller 1940 performs 3D reconstruction on the images acquired by the binocular camera 1920 according to the 3D reconstruction module 1941 and the calibrated parameters, obtaining a 3D point cloud model 1942. Subsequently, the controller 1940 performs temperature mapping on the three-dimensional point cloud model 1942 and the images acquired by the infrared thermal imager 1910 according to the temperature mapping module 1943 and the calibrated parameters to obtain a four-dimensional thermal image model 1950 of the object under test 1930.

[0309] A four-dimensional thermal image model generation device according to an embodiment of this application is used to execute a four-dimensional thermal image model generation method in the above embodiment. Its specific processing procedure is the same as that of the four-dimensional thermal image model generation method in the above embodiment, and will not be described in detail here.

[0310] Please refer to Figure 20 This application also provides an electronic device, which includes:

[0311] At least one memory;

[0312] At least one processor;

[0313] At least one computer program;

[0314] At least one computer program is stored in at least one memory, and at least one processor executes at least one computer program to implement: the four-dimensional thermal image model generation method as described in the above embodiments of this application.

[0315] The processor 2010 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0316] The memory can be implemented in the form of read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 2020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 2020 and is called and executed by the processor 2010 to execute the four-dimensional thermal image model generation method of the embodiments of this application.

[0317] Input / output interface 2030 is used to implement information input and output;

[0318] The communication interface 2040 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0319] Bus 2050 transmits information between various components of the device (e.g., processor 2010, memory 2020, input / output interface 2030, and communication interface 2040);

[0320] The processor 2010, memory 2020, input / output interface 2030 and communication interface 2040 are connected to each other within the device via bus 2050.

[0321] This application also provides a computer-readable storage medium storing a computer program for causing a computer to execute the four-dimensional thermal image model generation method described above.

[0322] This application provides a method, apparatus, and electronic device for generating a four-dimensional thermal image model. Compared with related technologies that fuse three-dimensional point clouds with two-dimensional thermal images, this application uses a pre-set calibration method for fusion, which can generate a more accurate four-dimensional thermal image model with better robustness and is more applicable to scenes with low resolution and weak texture features. The method proposed in this application uses structured light technology to reconstruct a high-precision point cloud model and fuse it with a thermal image. Simultaneously, it increases the frequency and number of steps of the projected stripes to obtain better target phase values ​​for subsequent image matching, enabling the calculation of more accurate three-dimensional point cloud data. The overall reconstruction and image fusion time is less than 5 seconds, resulting in higher efficiency. Furthermore, each spatial point in the four-dimensional thermal image model constructed in this application has corresponding temperature information, enabling the visualization of a three-dimensional temperature distribution model in non-contact scenarios. In addition, compared with related infrared camera calibration techniques, the calibration plate manufacturing process required for infrared thermal imager calibration in this application is simple, and the designed and developed calibration algorithm is applicable not only to visible light cameras but also to infrared cameras, demonstrating good versatility. Meanwhile, the calibration algorithm used in this application still yields excellent calibration results for infrared thermal imagers, with reprojection errors reaching the sub-pixel level. Finally, in the 3D reconstruction section, this application employs a stereo matching method based on target phase to find matching corresponding points in the left and right camera images. Compared to stereo matching methods based on grayscale regions and features (such as the SAD algorithm and SGBM algorithm), this method is equally applicable to weakly textured regions and can achieve sub-pixel level matching accuracy.

[0323] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0324] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0325] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0326] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0327] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0328] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0329] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0330] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0331] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for generating a four-dimensional thermal image model, characterized in that, Applied to a controller for communicating with a binocular camera module and an infrared thermal imaging module, the method includes: Acquire the first infrared image of the preset calibration component captured by the infrared thermal imaging module, and acquire the first visible light image of the preset calibration component captured by the binocular camera module; The first infrared image is calibrated according to the preset calibration parameters of the preset calibration component to obtain a first internal parameter and a first external parameter. The first internal parameter is used to represent the camera parameters of the infrared thermal imaging module, and the first external parameter is used to represent the relative position parameters of the infrared thermal imaging module in the world coordinate system. The first visible light image is calibrated according to the preset calibration parameters to obtain the second intrinsic parameter and the second extrinsic parameter of the binocular camera module. The second intrinsic parameter is used to represent the camera parameters of the binocular camera module, and the second extrinsic parameter is used to represent the relative position parameters of the binocular camera module in the world coordinate system. The relative position of the binocular camera module is calibrated according to the second extrinsic parameter to obtain the first conversion parameter. The first conversion parameter is used to represent the extrinsic parameter conversion relationship between the left camera and the right camera in the binocular camera module. The first conversion parameter includes a first conversion sub-parameter and a second conversion sub-parameter. The first conversion sub-parameter is used to represent the conversion parameter from the left camera to the right camera, and the second conversion sub-parameter is used to represent the conversion parameter from the right camera to the left camera. Temperature mapping calibration is performed based on the first external parameter and the second external parameter to obtain the second conversion parameter. The second conversion parameter is used to represent the external parameter conversion relationship between the infrared thermal imaging module and the target camera of the binocular camera module. The target camera is the left camera or the right camera. The binocular camera module acquires second visible light images of the object under different fringe frequencies using projection gratings. It then obtains average illumination intensity data and intensity modulation data from these second visible light images. A first phase principal value is calculated based on the average illumination intensity data, the intensity modulation data, and a preset first phase difference. A second phase principal value is also calculated based on the average illumination intensity data, the intensity modulation data, and a preset second phase difference. Finally, a target phase is calculated based on a preset frequency ratio, the first phase principal value, and the second phase principal value. When the target camera is the left camera... A left projection image is obtained by projecting the second visible light image onto the image plane of the left camera. The left projection image includes pixels to be matched. The slope of the left epipolar line is determined based on the coordinates of the pixels to be matched, the second intrinsic parameter, the first transformation sub-parameter, and the second transformation sub-parameter. An epipolar equation is constructed based on a preset slope transformation formula and the slope of the left epipolar line to obtain the equation of the right epipolar line. Point selection is performed based on the target phase, the equation of the right epipolar line, and the left projection image to determine the initial matching point. Three-dimensional reconstruction is performed based on the slope of the left epipolar line and the initial matching point to obtain a three-dimensional point cloud model of the object under test. A second infrared image of the object under test is acquired, and temperature mapping processing is performed based on the first intrinsic parameter, the second conversion parameter, the second infrared image, and the three-dimensional point cloud model to obtain a four-dimensional thermal image model of the object under test.

2. The method according to claim 1, characterized in that, The preset calibration parameters include the preset number of marked circles, the preset roundness threshold, and the preset area ratio range; The step of calibrating the first visible light image according to the preset calibration parameters to obtain the second intrinsic parameters and second extrinsic parameters of the binocular camera module includes: Corner contours are extracted from the first visible light image to obtain an initial circle and initial contour data of the initial circle. The initial contour data includes the initial contour surface area and the initial contour length. The initial roundness value is obtained by calculating the initial contour surface area and initial contour length. When the initial roundness value is less than or equal to the preset roundness threshold, the initial contour data is used as candidate contour data, and the initial circle of the initial contour data is used as a candidate circle. The average area value of the contour is obtained by calculating the mean area based on the candidate contour data. The candidate circles in the candidate contour data are filtered based on the contour surface area, the average area value of the contour, and the preset area ratio range, and the target circle of the first visible light image is determined based on the result of the circle filtering. When the number of target circles is equal to the number of preset label circles, the first visible light image is calibrated based on the target circles to obtain the second intrinsic parameter and the second extrinsic parameter.

3. The method according to claim 2, characterized in that, The step of calibrating the relative position of the binocular camera module based on the second external parameter to obtain the first conversion parameter includes: The origin of the first world coordinate system is set according to the first visible light image, and the first world coordinates are obtained according to the origin of the first world coordinate system; wherein, the first world coordinates are used to represent the coordinates of the center of the target circle in the first world coordinate system; The first pixel coordinates are obtained based on the first visible light image. The first pixel coordinates are used to represent the coordinates of the center of the target circle in the first pixel coordinate system of the left camera. The second pixel coordinates are obtained based on the first visible light image. The second pixel coordinates are used to represent the coordinates of the center of the target circle in the second pixel coordinate system of the right camera. The first transformation parameter is calculated based on the first world coordinates, the first pixel coordinates, and the second pixel coordinates.

4. The method according to claim 2, characterized in that, The preset area ratio range includes an upper limit value and a lower limit value for the preset area ratio; The step of filtering candidate circles in the candidate contour data based on the contour surface area, the average contour area value, and the preset area ratio range, and determining the target circle based on the results of the circle filtering, includes: The candidate ratio value is obtained by calculating the area ratio between the surface area of ​​the candidate contour data and the average area value of the contour. When the candidate ratio value is less than the upper limit of the preset area ratio value and the candidate ratio value is greater than the lower limit of the preset area ratio value, the candidate circle corresponding to the candidate ratio value is taken as the target circle.

5. The method according to claim 1, characterized in that, The step of performing temperature mapping processing based on the first intrinsic parameter, the second transformation parameter, the second infrared image, and the three-dimensional point cloud model to obtain a four-dimensional thermal image model of the object under test includes: Obtain the initial camera coordinates of the 3D point cloud in the 3D point cloud model, wherein the initial camera coordinates are used to represent the coordinates of the 3D point cloud in the camera coordinate system of the target camera; The initial camera coordinates are transformed according to the second transformation parameters to obtain the first infrared coordinates. The first infrared coordinates are used to represent the coordinates of the three-dimensional point cloud in the infrared thermal imager coordinate system of the second infrared image. A second coordinate transformation is performed based on the first intrinsic parameter and the first infrared coordinate to obtain the second infrared coordinate, which is used to represent the coordinates of the three-dimensional point cloud in the image coordinate system of the second infrared image. A third coordinate transformation is performed based on the first intrinsic parameter and the second infrared coordinate to obtain the third infrared coordinate, which is used to represent the coordinates of the three-dimensional point cloud in the pixel coordinate system of the second infrared image; Temperature mapping is performed based on the third infrared coordinates and the three-dimensional point cloud model to obtain the four-dimensional thermal image model of the object under test.

6. The method according to claim 1, characterized in that, The process of determining the left outer epipolar slope based on the coordinates of the pixel to be matched, the second intrinsic parameter, the first transformation sub-parameter, and the second transformation sub-parameter, and constructing the epipolar equation based on the preset slope transformation formula and the left outer epipolar slope to obtain the right outer epipolar equation includes: The coordinates of the left outer pole are calculated based on the second intrinsic parameter and the first transformation sub-parameter. The coordinates of the left outer pole are used to represent the coordinates of the left outer pole center of the left camera in the left camera coordinate system. The coordinates of the right outer pole are calculated based on the second intrinsic parameter and the second transformation sub-parameter. The coordinates of the right outer pole are used to represent the coordinates of the right outer pole center of the right camera in the right camera coordinate system. The slope of the left outer epipolar line is obtained by calculating the slope based on the coordinates of the pixel to be matched and the coordinates of the left outer epipolar point; the coordinates of the pixel to be matched are used to represent the coordinates of the pixel to be matched in the left camera coordinate system. The polar equation is constructed based on the preset slope conversion formula, the slope of the left outer polar line, and the coordinates of the right outer pole, thus obtaining the right outer polar line equation.

7. The method according to claim 6, characterized in that, The step of performing three-dimensional reconstruction based on the slope of the left outer epipolar line and the initial matching point to obtain a three-dimensional point cloud model of the measured object includes: Based on the initial matching point and the preset neighborhood threshold, matching points are selected to obtain candidate matching points and the coordinates of the right outer pole of the candidate matching points; The slope of the right outer epipolar line is obtained by calculating the coordinates of the right outer epipolar point and the coordinates of the candidate matching point; the coordinates of the candidate matching point are used to represent the coordinates of the candidate matching point in the right camera coordinate system. The slope difference is calculated by using the slope conversion formula, the slope of the left outer polar line, and the slope of the right outer polar line. When the slope difference is less than a preset difference threshold, the candidate matching point is taken as the intermediate matching point; The target matching point is determined based on the target phase of the intermediate matching point; The three-dimensional point cloud model is obtained by performing three-dimensional reconstruction based on the target matching points.

8. A four-dimensional thermal image model generation device, characterized in that, The device includes: Controller, the controller being configured to execute a four-dimensional thermal imaging model generation method as described in any one of claims 1 to 7; A binocular camera module, which is communicatively connected to the controller, is used to acquire visible light images; An infrared thermal imaging module, which is communicatively connected to the controller, is used to acquire infrared images; A preset calibration component is used to calibrate the binocular camera module and the infrared thermal imaging module.

9. An electronic device, characterized in that, include: At least one memory; At least one processor; At least one computer program; The at least one computer program is stored in the at least one memory, and the at least one processor executes the at least one computer program to perform: The method as described in any one of claims 1 to 7.

Images