A dual light fusion calculation method and system based on binocular depth information

Abstract

The application provides a binocular depth information-based dual-light fusion calculation method and system, which comprises the following steps: obtaining and processing engineering actual precision requirement data to calibrate a second visible light camera B and an infrared temperature measurement camera C in a differential depth distance; finding an imaging picture rectangular area M in a specific depth physical space to calibrate a measured target, wherein the imaging picture rectangular area M completely covers the infrared temperature measurement camera C, the area of the imaging picture rectangular area M on the imaging picture of the second visible light camera B is measured to obtain a rectangular pixel point set N; calibrating the vertex coordinates; matching and processing an observed point K in the second visible light camera B; and matching and calculating the area data in the second visible light camera B. The application solves the technical problems of complex design, high cost and unguaranteed fusion matching precision.

Description

Technical Field

[0001] This invention relates to the field of machine vision, specifically to a dual-light fusion computing method and system based on binocular depth information. Background Technology

[0002] Temperature measurement cameras are now widely used in scenarios such as human body temperature measurement, industrial site monitoring, and fire monitoring. The demand for these applications has also evolved from broad field-of-view temperature measurement to precise field-of-view temperature measurement. Especially with the continuous development of AI technology, there is an increasing need for accurate identification of objects and their positions within the visible light field of view, and for precise measurement of the temperature of objects in visible light images (e.g., multi-face area temperature measurement, temperature measurement of specially moving vehicles in industrial sites). A key technical challenge for these applications is solving the problem of accurate image fusion between the visible light and infrared cameras of the same camera device—that is, accurately mapping the position of an object in the visible light image to its position in the infrared image.

[0003] Temperature measurement cameras typically consist of two independent cameras: a visible light camera and an infrared temperature measurement camera. The visible light camera generally has high resolution and a wide field of view, while the infrared camera has low resolution and a narrow field of view. For objects at different depths from the camera, the position coordinates on the visible light camera's screen and their position coordinates on the infrared camera's screen are not in a linear, one-to-one correspondence. Therefore, temperature measurement cameras generally cannot solve the problem of accurate image fusion.

[0004] Existing technologies use a coaxial solution (where visible light and infrared light both enter through the same set of lenses and are then projected onto their respective sensors via a beam splitter) to solve the problem of precise fusion of two beams. However, this solution involves complex optical design, expensive lens sets, and difficult engineering implementation. It is generally found in special industry scenarios where cost is not a major concern, and is very rare in consumer and light industrial applications.

[0005] The existing invention patent application document CN110428008A, entitled "A Target Detection and Recognition Device and Method Based on Multi-Fusion Sensors," includes a multi-fusion sensor module, a system data processing module, a 3D point cloud reconstruction module, a power supply module, a command and control module, and a display module. The multi-fusion sensor system uses a lidar sensor to emit a supercontinuous lidar to detect target objects, acquiring the target's pose and position in real time. Infrared and visible light sensors perform spectral detection on the target, acquiring infrared and visible light video of the target. With the cooperation of each unit in the system, the fused spectral information and spatial pose and position information of the target are obtained, achieving accurate 3D positioning and real-time 3D point cloud imaging. This existing technology uses a deep CNN as a feature extractor to select image features from the video, performs feature overlap and calculates the overlap loss, and selects the video with the lowest loss for tracking. This is executed in the video selection unit, with the tracker performing the tracking and superimposing the tracking result onto the tracked video. Because the scheme disclosed in this existing document requires the use of a deep neural network to process depth dimension information and requires feature overlap to complete multi-sensor data fusion, its processing is complex, and the algorithm complexity is high, resulting in low system applicability.

[0006] The existing invention patent application document CN112258441A, entitled "A Method for Generating Images Based on Visible Light and Infrared Fusion," includes the following steps: S1: Image key point extraction; using machine learning technology, key points of visible light and infrared images are extracted as reference calibration points for image fusion; S2: Image desaturation processing; the acquired visible light image is desaturated to discard chromaticity information while preserving high-frequency contrast information; S3: Temperature information extraction; temperature data is extracted using infrared thermal imaging technology, achieved through the SDK built into the selected camera. As can be seen from the technical details disclosed in the document, the existing technology needs to traverse each depth image point and perform edge detection by finding the location with depth changes in the nearest neighbor region. Based on the surface changes in the nearest neighbor region, a coefficient for measuring surface changes and the main direction of change are determined. It can be seen that although the existing technology considers depth dimension data, it only uses a traversal method for processing, which cannot guarantee the fusion accuracy. At the same time, the neighbor region search process before the traversal process increases the computational load of the system. In addition, the camera used by this technology needs to be implemented using the SDK that comes with the existing device, which increases the cost of using the system.

[0007] In summary, existing technologies suffer from technical problems such as complex design, high cost, and inability to guarantee fusion matching accuracy. Summary of the Invention

[0008] The technical problem to be solved by this invention is how to solve the problems of complex design, high cost and inability to guarantee the accuracy of fusion matching in the prior art.

[0009] This invention solves the above-mentioned technical problems by employing the following technical solution: A dual-light fusion calculation method based on binocular depth information includes:

[0010] S1. Acquire and process the actual accuracy requirements of the project to perform fusion calibration of the second visible light camera B and the infrared temperature measurement camera C at different depth distances. The cameras include: the first visible light camera A, the second visible light camera B, and the infrared temperature measurement camera C.

[0011] S2. Locate the rectangular region M of the imaging screen in the physical space at a specific depth to calibrate the target being measured. The rectangular region M of the imaging screen completely covers the infrared temperature measurement camera C. Measure the area of ​​the rectangular region M of the imaging screen on the imaging screen of the second visible light camera B to obtain the rectangular pixel set N.

[0012] S3, Calibrate vertex coordinates;

[0013] S4. Matching and processing the observed point K in the second visible light camera B. Step S4 includes:

[0014] S41. Obtain the coordinates (xBK, yBK) of the observed point K in the image of the second visible light camera B. Then, based on the principle of binocular parallax, calculate the depth of the image of the first visible light camera A and the image of the second visible light camera B. Calculate the coordinates of the observed point using binocular depth calculation to obtain the depth information deep(xBK, yBK) of the observed point K.

[0015] S42. Process the depth information deep(xBK,yBK) value to obtain the overlapping area of ​​the infrared and visible light camera images;

[0016] S43. Determine whether the observed point K is within the infrared field of view coverage area of ​​the corresponding calibration data;

[0017] S44. If not, it means that the object being measured has not entered the field of view of the infrared temperature measuring camera C, and it is determined that the temperature cannot be measured.

[0018] S45. If so, calculate the coordinates of the observed point K in the infrared camera C, and extract the temperature value matching the observed point, so as to obtain the temperature value of the observed point K from the imaging image of the second visible camera B.

[0019] S5. Match and calculate the area data in the second visible light camera B.

[0020] This invention achieves precise dual-light fusion in a coaxial scheme by processing depth dimension information, the position of the measured point, and the proportion of the measured point in the infrared region. Compared with existing fusion schemes, this invention eliminates the need for complex optical designs and costly lens assemblies, reduces the engineering difficulty of dual-light fusion between visible light cameras and infrared temperature measurement cameras, lowers the technical cost in specific application scenarios, and is suitable for consumer and light industrial applications, demonstrating high system applicability.

[0021] In a more specific technical solution, in step S1, the rectangular pixel set N is described by four vertex coordinates: {Pleft-top(xn,yn), Pright-top(xn,yn), Pleft-bottom(xn,yn), Pright-bottom(xn,yn)}, where the field of view of the infrared temperature measuring camera C is smaller than the field of view of the second visible light camera B; the starting working distance value of the camera is defined as the position at the minimum distance from the camera when the field of view of the infrared temperature measuring camera C is a subset of the field of view of the second visible light camera B.

[0022] In a more specific technical solution, step S3 includes:

[0023] S31. In a preset background environment, place the light-emitting device on an angle position adjustment platform that can adjust the vertical angle and the left, right and up and down movement.

[0024] S32. Adjust the physical spatial position of the light-emitting point device so that the light-emitting point is imaged at a vertex position of the image of the infrared camera C;

[0025] S33. Acquire the image from the second visible light camera B, locate the luminous point, and measure and obtain the coordinates of the luminous point in the image of the second visible light camera B;

[0026] S34. Based on the vertex coordinates measured in the image of the second visible light camera B, repeat steps S32 and S33 until the vertex coordinates of the top left vertex {Pleft-top(xn,yn), top right vertex Pright-top(xn,yn), bottom left vertex Pleft-bottom(xn,yn) and bottom right vertex Pright-bottom(xn,yn)} in the image of the second visible light camera B are obtained, and the vertex coordinates of the top left vertex {Pleft-top(xn,yn), top right vertex Pright-top(xn,yn), bottom left vertex Pleft-bottom(xn,yn)} and bottom right vertex Pright-bottom(xn,yn)} in the image of the second visible light camera B are obtained, so as to describe the overlapping area N (deep) of the infrared and visible light camera images in the image of the second visible light camera B.

[0027] S35. Repeat the above calibration steps S31 to S34 to obtain fusion calibration data at different depths: N(deep1), N(deep2), N(deep3)...N(deeppi).

[0028] In a more specific technical solution, in step S33, color difference data is collected and processed to determine the coordinates of the light-emitting point device in the imaging image of the second visible light camera B.

[0029] In a more specific technical solution, in step S33, temperature difference data is collected and processed to determine the coordinates of the light-emitting device in the image of the infrared temperature measuring camera C.

[0030] In a more specific technical solution, step S42 processes the depth information deep(xBK,yBK) value to find the overlapping area N(deep) of the infrared and visible light camera images that is closest to the current observation point.

[0031] This invention utilizes the characteristic that points at different depths with the same x-coordinate within the field of view of a visible light camera have different x-coordinates within the field of view of an infrared temperature measurement camera. Based on the depth information deep(xBK, yBK), it calculates the x-coordinate of a point in the infrared temperature measurement camera's image using the x-coordinate of the visible light camera's image, and then accurately calculates its y-coordinate using the y-coordinate of the visible light camera's image. Under the aforementioned parameter conditions, this invention incorporates the dimension of depth information to precisely match the visible light camera's image onto the infrared temperature measurement camera's image.

[0032] In a more specific technical solution, step S45 includes:

[0033] S451. Let the width of the image frame of the second visible light camera B be w, the height be h, and the coordinates of a specific point in the image frame of the second visible light camera B be (x, y). Calculate the proportional coordinates of the specific point as x' = x / w, y' = y / h.

[0034] S452. Based on the coordinates of the four vertices {Pleft-top(xn,yn), Pright-top(xn,yn), Pleft-bottom(xn,yn), Pright-bottom(xn,yn)} of the overlapping region N (deep) of the infrared and visible light camera images and the coordinates (xBK, yBK) of the observed point K in the image of the second visible light camera B, the width, height, and coordinates of the observed point K of the overlapping region N (deep) of the infrared and visible light camera images are obtained by the following logical processing:

[0035] wN=Pright-top(xn)-Pleft-top(xn);

[0036] hN=Pleft-bottom(yn)-Pleft-top(yn);

[0037] xNK = xBK - Pleft - top(xn);

[0038] yNK = yBK - Pleft - top(yn);

[0039] S453. Process the width and height of the overlapping area N (deep) of the infrared and visible light camera images and the coordinates of the observed point K using the following logic to obtain the proportional coordinates of the observed point K in the overlapping area N (deep) of the infrared and visible light camera images:

[0040] ratiox = xNK / wN;

[0041] ratioy = yNK / hN;

[0042] S454. The scale coordinates of the observed point K in the image of the infrared camera C are processed using the following logical transformation:

[0043] ratiox = xCK / wC;

[0044] ratioy = yCK / hC;

[0045] xCK = ratiox * wC;

[0046] yCK = ratioy * hC;

[0047] The width wC, height wH, and scale coordinates of the image from infrared camera C are processed using the following logic to obtain the coordinates (xCK, yCK) of the observed point K within the image from infrared camera C, thereby obtaining the temperature value of the observed point K:

[0048] xCK = ratiox * wC

[0049] =xNK / wN*wC

[0050] =(xBK-Pleft-top(xn)) / (Pright-top(xn)-Pleft-top(xn))*wC

[0051] yCK=ratioy*hC

[0052] =yNK / hN*hC

[0053] =(yBK-Pleft-top(yn)) / (Pleft-bottom(yn)-Pleft-top(yn))*hC.

[0054] This invention calculates the relative proportions of the observed point's coordinates within a rectangular region when the observed point appears within the infrared area, thereby obtaining its actual coordinates and temperature value in the infrared image. This improves the matching accuracy of the observed point in both the visible light camera's and infrared temperature measurement camera's images, significantly enhancing the accuracy of dual-light fusion.

[0055] In a more specific technical solution, step S5 includes:

[0056] S51. Traverse the corresponding coordinates of each point in the imaging area of ​​the visible light camera B to obtain the set of temperature values.

[0057] S52. Based on the actual accuracy requirements of the project, the temperature value set is processed to obtain the temperature value of the private area, which is used as the result of dual-light fusion of visible light and temperature measurement camera.

[0058] In a more specific technical solution, the first visible light camera A and the second visible light camera B adopt the same specifications; the optical axes of the first visible light camera A, the second visible light camera B, and the infrared temperature measurement camera C are parallel and their imaging planes are parallel.

[0059] In a more specific technical solution, a dual-light fusion computing system based on binocular depth information includes:

[0060] The engineering requirements acquisition module is used to acquire and process the actual accuracy requirements of the engineering project, so as to perform fusion calibration of the second visible light camera B and the infrared temperature measurement camera C at different depth distances. The cameras include: the first visible light camera A, the second visible light camera B, and the infrared temperature measurement camera C.

[0061] The target calibration module is used to find a rectangular region M of the imaging screen in a specific depth physical space to calibrate the target under test. The rectangular region M of the imaging screen completely covers the infrared temperature measurement camera C. The area of ​​the rectangular region M of the imaging screen on the imaging screen of the second visible light camera B is measured to obtain the rectangular pixel set N. The target calibration module is connected to the engineering requirements acquisition module.

[0062] The vertex processing module is used to calibrate vertex coordinates and is connected to the target calibration module.

[0063] The measured point matching module is used to match and process the observed point K in the second visible light camera B. The measured point matching module is connected to the vertex processing module and the target calibration module. The measured point matching module includes:

[0064] The depth information module is used to obtain the coordinates (xBK, yBK) of the observed point K in the image of the second visible light camera B. Then, based on the principle of binocular parallax, the depth information deep(xBK, yBK) of the image of the first visible light camera A and the second visible light camera B is calculated to obtain the depth information of the observed point K.

[0065] The infrared region acquisition module is used to process the depth information deep(xBK,yBK) value to obtain the overlapping area of ​​the infrared and visible light camera images. The infrared region acquisition module is connected to the depth information module.

[0066] The observed point position determination module is used to determine whether the observed point K is within the infrared field of view coverage area of ​​the corresponding calibration data. The observed point position determination module is connected to the infrared area acquisition module.

[0067] The "Not Entering Field of View" determination module is used to determine that the object being measured has not entered the field of view of the infrared temperature measurement camera C when the observed point K is not in the infrared area of ​​the corresponding calibration data, and to determine that temperature cannot be measured. The "Not Entering Field of View" determination module is connected to the observed point position determination module.

[0068] Enter the field of view determination module to determine the coordinates of the observed point K in the infrared camera C and extract the temperature value matching the observed point. Based on this, the temperature value of the observed point K is obtained from the imaging image of the second visible camera B. The field of view determination module is then connected to the observed point position determination module.

[0069] The region matching module is used to match and calculate the region data in the second visible light camera B. The region matching module is connected to the vertex processing module and the target calibration module.

[0070] Compared with existing technologies, this invention has the following advantages: By processing depth dimension information, the position of the measured point, and the proportion of the measured point in the infrared region, this invention achieves precise dual-light fusion in a coaxial scheme. Compared with existing fusion schemes, this invention eliminates the need for complex optical designs and high-cost lens assemblies, reducing the engineering difficulty of dual-light fusion between visible light cameras and infrared temperature measurement cameras, lowering the technical cost in specific application scenarios, and making it suitable for consumer and light industrial applications with high system applicability.

[0071] This invention utilizes the characteristic that points at different depths with the same x-coordinate within the field of view of a visible light camera have different x-coordinates within the field of view of an infrared temperature measurement camera. Based on the depth information deep(xBK, yBK), it calculates the x-coordinate of a point in the infrared temperature measurement camera's image using the x-coordinate of the visible light camera's image, and then accurately calculates its y-coordinate using the y-coordinate of the visible light camera's image. Under the aforementioned parameter conditions, this invention incorporates the dimension of depth information to precisely match the visible light camera's image onto the infrared temperature measurement camera's image.

[0072] This invention calculates the relative proportions of the observed point's coordinates within a rectangular region when the observed point appears within the infrared area, thereby obtaining its actual coordinates and temperature value in the infrared image. This improves the matching accuracy of the observed point in both visible light camera and infrared temperature measurement camera images, significantly enhancing the accuracy of dual-light fusion. This invention solves the technical problems of complex design, high cost, and inability to guarantee fusion matching accuracy in existing technologies. Attached Figure Description

[0073] Figure 1 This is a schematic diagram of the camera's field of view in Embodiment 1 of the present invention;

[0074] Figure 2 This is a stereo depth diagram of Embodiment 1 of the present invention;

[0075] Figure 3 This is a schematic diagram of the fusion calculation in Embodiment 1 of the present invention;

[0076] Figure 4 This is a schematic diagram of the basic steps of a dual-light fusion calculation method based on binocular depth information according to Embodiment 2 of the present invention. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0078] Example 1

[0079] like Figure 1As shown, this invention uses three cameras: two visible light cameras A and B, and one infrared temperature measurement camera C. Cameras A and B are identical, with the same resolution and field of view; the optical axes and imaging planes of cameras A, B, and C are parallel; the visible light field of view of cameras A and B is larger than the infrared field of view of camera C.

[0080] like Figure 2 As shown, A and B use the binocular parallax depth method to calculate the depth value of each pixel in the visible light imaging image. Binocular depth calculation methods are mature in the industry, such as using the OpenCV library. Here's a brief description of the principle of binocular depth calculation. In this embodiment, the focal length of cameras A and B is f, and the optical axes of A and B are parallel; the baseline length (the distance between the optical centers of A and B) is b; point P is imaged as point Pa on A, with Pa's distance from the optical axis of A being x1, and as point Pb on B, with Pb's distance from the optical axis of B being x2; assuming the distance from point P to the optical axis of camera B is m, then the distance from point P to the optical axis of camera A is (m+b); the distance from point P to the baselines of A and B is z, where z is the object depth information we want to calculate.

[0081] Based on the principle of similar triangles, we can obtain two equations:

[0082] (2.6.a): f / z=x2 / m

[0083] (2.6.b): f / z = x1 / (m+b)

[0084] According to Equation 2.6.a, we get:

[0085] m=(x2*z) / f

[0086] Substituting into equation 2.6.b, we get:

[0087] f / z = x1 / ((x2*z) / f+b)

[0088] Further calculations yielded the following:

[0089] z = f*b / (x1-x2)

[0090] Therefore, the depth (distance) information of an object can be calculated using the focal length f, baseline b, and parallax (x1-x2). The depth information of each pixel in the images captured by cameras A and B can be calculated using binocular parallax. Assuming that the coordinates of any pixel in image B are xB on the imaging width and yB on the imaging height, the depth information of that point is recorded as deep(xB, yB).

[0091] Methods for two-light fusion calculation:

[0092] Assume that the width of the B and C imaging plane coordinate system is the x-axis and the height is the y-axis. Let the width of image B be wB and the height be hB; let the width of image C be wC and the height be hC.

[0093] like Figure 3 As shown, the plane is any cross-section within the fields of view of B and C that is parallel to the x-axis and perpendicular to the imaging plane. BB1 and BB2 are schematic diagrams of the fields of view of the visible light camera B; CC1 and CC2 are schematic diagrams of the fields of view of the infrared camera C; lines BB1 and BB2 are the edge lines of the field of view of B; lines CC1 and CC2 are the edge lines of the field of view of C.

[0094] In this embodiment, four auxiliary lines are drawn: Line0, Line1, Line2, and Line3. Line0 is any auxiliary line passing through B and within the field of view of B; Line1 and Line3 are any two auxiliary lines parallel to the imaging plane of B (and C); Line2 is an auxiliary line passing through the intersection of Line0 and CC2 and parallel to the x-axis. The depths of Line1, Line2, and Line3 from the imaging plane are d1, d2, and d3, respectively. Details of the intersection points of each line can be found in [link to documentation]. Figure 3 :

[0095] Line1 intersects BB1 at point P00, intersects CC1 at point P01, intersects Line0 at point P02, intersects CC2 at point P03, and intersects BB2 at point P04.

[0096] Line2 intersects BB1 at point P10 and also intersects CC1;

[0097] It intersects with CC2 / Line0 at point P11, and with BB2 at point P13;

[0098] Line3 intersects BB1 at point P20, CC1 at point P21, CC2 at point P22, Line0 at point P23, and BB2 at point P24.

[0099] In this embodiment, we first examine the x-coordinates of each point on Line0 on the B-image.

[0100] The x-coordinate of point P02 in B is P02(xB) = (P02P04 / P00P04) * wB

[0101] The x-coordinate of point P12 in B is P12(xB) = (P12P13 / P10P13)*wB

[0102] The x-coordinate of point P23 in B is P22(xB) = (P23P24 / P20P24) * wB

[0103] According to the principle of similar triangles,

[0104] P02P04 / P12P13=d1 / d2

[0105] P00P04 / P10P13=d1 / d2

[0106] so:

[0107] P02P04 / P12P13=P00P04 / P10P13

[0108] and then:

[0109] P02P04 / P00P04=P12P13 / P10P13

[0110] Similarly, it can be deduced that:

[0111] P02P04 / P00P04=P12P13 / P10P13=P23P24 / P20P24

[0112] so:

[0113] P02(xB)=P12(xB)=P22(xB)

[0114] This means that the x-coordinates of all points on Line0 in the image plane B are the same. Similarly, the x-coordinates of all points on CC2 in the image plane C are also the same, that is:

[0115] P03(xC)=P12(xC)=P22(xC)

[0116] from Figure 3 It is obvious

[0117] P02(xC)! = P12(xC)! = P23(xC), and P23 is not even within the field of view of C.

[0118] From the above, we can conclude that points at different depths with the same x-coordinate within the field of view B will have different x-coordinates within the field of view C. Without depth information, the x-coordinate of a point in the image of view C cannot be accurately calculated from the x-coordinate of the image of view B; similarly, the y-coordinate of a point in the image of view C cannot be accurately calculated from the y-coordinate of the image of view B. Under the constraints of the hardware parameters of this invention, theoretically, without incorporating the dimension of depth information, the image of visible light B cannot be accurately matched to the image of infrared light C.

[0119] Example 2

[0120] like Figure 4 As shown, this invention provides a dual-light fusion calculation method based on binocular depth information by incorporating a depth information dimension. The specific steps include:

[0121] S1. According to the actual accuracy requirements of the project, the visible light camera B and the infrared temperature measurement camera C are fused and calibrated at different depths and distances. In this embodiment, for example, the working distance of the camera is required to be 5 meters and the precise fusion accuracy is 0.1 meters. Then, the fusion calibration data at the current depth is obtained every 0.1 meters within the working distance of 5 meters. In this embodiment, the field of view of C is smaller than the field of view of B. Except for a small space near the camera, the field of view of C is completely within the field of view of B. The starting working distance value of the camera is defined as the position of the minimum distance from the camera when the field of view of C is a subset of the field of view of B.

[0122] S2. Locate a rectangular region M in the physical space at a specific depth to calibrate the target being measured. In this embodiment, it is necessary to find a rectangular region M that completely covers the image of C in the physical space at a specific depth, and then measure the area of ​​M on the image of B, thus obtaining a set of pixels N in a rectangular region. In this embodiment, M can be understood as the complete image of C, and N means that the set of pixels N in the image of B at this specific depth is all within the field of view of C, that is, the set of pixels N in B at this depth can measure temperature, while pixels at other positions cannot measure temperature. N is described by four vertex coordinates, such as {Pleft-top(xn,yn), Pright-top(xn,yn), Pleft-bottom(xn,yn), Pright-bottom(xn,yn)}. In this embodiment, because the imaging sensors for visible light and infrared light are rectangular, the image is rectangular.

[0123] S3. Calibrate vertex coordinates; In this embodiment, specific engineering design and implementation are carried out according to the actual hardware environment.

[0124] For example, a calibration method like this can be used: In a simple background environment, a light-emitting device with a color and temperature different from the background can be placed on a platform where its vertical angle and horizontal / vertical movement can be adjusted. The coordinates of the light-emitting device in frame B can be determined by the color difference, and the coordinates of the light-emitting device in frame C can be determined by the temperature difference. By adjusting the position of the light-emitting device to match the top-left vertex of frame C, the Pleft-top(xn,yn) coordinates can then be measured in frame B. Using the same method, the coordinates of the other three points are found. In this embodiment, according to engineering accuracy requirements, the fusion calibration of B and C is performed at different depth positions to obtain the coordinates of four vertices in the B imaging image: {Pleft-top(xn,yn), Pright-top(xn,yn), Pleft-bottom(xn,yn), Pright-bottom(xn,yn)}. These four points describe a rectangular area: the area N(deep) of B covered by the infrared field of view of C at the deep depth position; thus, there are multiple sets of fusion calibration data at different depths: N(deep1), N(deep2), N(deep3)...;

[0125] S4. Calculation of precise matching of point K in visible light camera B; In this embodiment, the coordinates of point K in B are (xBK, yBK); In this embodiment, depth calculation based on the principle of binocular parallax is performed using the imaging images of the first visible light camera A and the second visible light camera B; In this embodiment, the depth information deep(xBK, yBK) is obtained through binocular depth calculation; The calibration data infrared region IR under this depth information is obtained through the deep(xBK, yBK) value: {Pleft-top(xK, yK), Pright ... The algorithm calculates depth based on the principle of binocular parallax using images from the first visible light camera A and the second visible light camera B. It then searches for the nearest depth-based fusion calibration data N(deep) and determines whether point K is within the IR region, such as the infrared field of view N(deep). If not, it indicates the object is not within the field of view of the infrared thermometer C, and temperature measurement is impossible. If it is within the region, the relative proportions (ratiox and ratioy) of point K within the rectangular region are calculated. Based on these proportions, its actual coordinates in the infrared image C are calculated to obtain its temperature value. In this embodiment, the coordinates of K in the infrared camera C are used to extract the temperature value matching that point, thus completing the acquisition of the temperature value of point K in B. The calculation logic is as follows:

[0126] N(deep) is actually the overlapping area of ​​the image of C and the image of B at the depth of deep. Therefore, the relative position of the image point (xBK, yBK) of K in B in N(deep) is the same as the relative position of the image point (xCK, yCK) of K in C in the image of C. The relative position is described using scale coordinates: Assuming the width of the image is w and the height is h, and the coordinates of a point in the image are (x, y), then calculate its scale coordinates x' = x / w, y' = y / h.

[0127] Given the coordinates of the four vertices of N (deep) {Pleft-top(xn,yn), Pright-top(xn,yn), Pleft-bottom(xn,yn), Pright-bottom(xn,yn)} and the coordinates of K in B (xBK, yBK), then:

[0128] The width of N(deep) is wN = Pright - top(xn) - Pleft - top(xn);

[0129] The height hN of N(deep) = Pleft - bottom(yn) - Pleft - top(yn);

[0130] The x-axis coordinate of K in N(deep) is xNK = xBK - Pleft - top(xn);

[0131] The y-coordinate of K in N(deep) is yNK = yBK - Pleft - top(yn);

[0132] so:

[0133] The x-axis proportional coordinate of K in N (deep) is ratiox = xNK / wN;

[0134] The y-axis proportional coordinate of K in N (deep) is ratioy = yNK / hN;

[0135] Meanwhile, the scale coordinates of K in C are equal to the scale coordinates of K in N (deep), and the width wC and height wH of the image frame C are known, therefore:

[0136] ratiox = xCK / wC;

[0137] ratioy = yCK / hC;

[0138] Let's change it:

[0139] xCK = ratiox * wC;

[0140] yCK = ratioy * hC;

[0141] Substitute the known quantities:

[0142] xCK = ratiox * wC

[0143] =xNK / wN*wC

[0144] =(xBK-Pleft-top(xn)) / (Pright-top(xn)-Pleft-top(xn))*wC

[0145] yCK=ratioy*hC

[0146] =yNK / hN*hC

[0147] =(yBK-Pleft-top(yn)) / (Pleft-bottom(yn)-Pleft-top(yn))*hC

[0148] In summary, the coordinates (xCK, yCK) of K in the C image can be calculated using the known coordinates of the four vertices of N (deep), the coordinates of point (xBK, yBK), and the width and height of the C image. This allows us to obtain the temperature value of point K, thus completing the precise matching calculation between points B and C.

[0149] S5. Precise matching calculation of the region in the visible light camera B; In this embodiment, by repeating the aforementioned step S4, the temperature dataset of a certain region in the second visible light camera B can be calculated to realize different precise temperature measurement services; In this embodiment, the principle is the same as the precise matching of points. The corresponding coordinates of each point in the region B that needs to be calculated are traversed to obtain the set of temperature values, and then combined into the temperature values ​​of the private region according to the actual business requirements.

[0150] In summary, this invention achieves precise dual-light fusion in a coaxial scheme by processing depth dimension information, the position of the measured point, and the proportion of the measured point in the infrared region. Compared to existing fusion schemes, this invention eliminates the need for complex optical designs and costly lens assemblies, reduces the engineering difficulty of dual-light fusion between visible light cameras and infrared temperature measurement cameras, lowers the technical cost in specific application scenarios, and is suitable for both consumer and light industrial applications, demonstrating high system applicability.

[0151] This invention utilizes the characteristic that points at different depths with the same x-coordinate within the field of view of a visible light camera have different x-coordinates within the field of view of an infrared temperature measurement camera. Based on the depth information deep(xBK, yBK), it calculates the x-coordinate of a point in the infrared temperature measurement camera's image using the x-coordinate of the visible light camera's image, and then accurately calculates its y-coordinate using the y-coordinate of the visible light camera's image. Under the aforementioned parameter conditions, this invention incorporates the dimension of depth information to precisely match the visible light camera's image onto the infrared temperature measurement camera's image.

[0152] This invention calculates the relative proportions of the observed point's coordinates within a rectangular region when the observed point appears within the infrared area, thereby obtaining its actual coordinates and temperature value in the infrared image. This improves the matching accuracy of the observed point in both visible light camera and infrared temperature measurement camera images, significantly enhancing the accuracy of dual-light fusion. This invention solves the technical problems of complex design, high cost, and inability to guarantee fusion matching accuracy in existing technologies.

[0153] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dual light fusion calculation method based on binocular depth information, characterized in that, The method comprises: S1, acquiring and processing engineering actual precision requirement data to calibrate the second visible light camera B and the infrared temperature measurement camera C in the depth distance difference, wherein the cameras comprise a first visible light camera A, a second visible light camera B, and an infrared light temperature measurement camera C; S2, finding an imaging picture rectangular area M in a specific depth physical space to calibrate a measured target, wherein the imaging picture rectangular area M completely covers the infrared temperature measurement camera C, the area of the imaging picture rectangular area M on the imaging picture of the second visible light camera B is measured to obtain a rectangular pixel point set N; S3, calibrating vertex coordinates; S4, matching processing an observed point K in the second visible light camera B, and the step S4 comprises: S41, acquiring observed point coordinates (xBK, yBK) of the observed point K in the imaging picture of the second visible light camera B, and then calculating the imaging picture of the first visible light camera A and the imaging picture of the second visible light camera B based on the binocular disparity principle to obtain depth information deep(xBK, yBK) of the observed point K; S42, processing the depth information deep(xBK, yBK) to acquire an infrared and visible light camera picture overlapping area; S43, judging whether the observed point K is in a corresponding calibration data infrared field of view covering area; S44, if not, it is determined that the measured object does not enter the field of view of the infrared temperature measurement camera C, and it is determined that temperature measurement is impossible; S45, if yes, the coordinates of the observed point K in the infrared light camera C are calculated, and the temperature value matching the observed point is extracted to acquire the temperature value of the observed point K from the imaging picture of the second visible light camera B; S5, matching and calculating area data in the second visible light camera B.

2. The binocular depth information-based dual light fusion calculation method according to claim 1, characterized in that, In the step S1, the rectangular pixel point set N is described by four vertex coordinates: {Pleft-top(xn,yn), Pright-top(xn,yn), Pleft-bottom(xn,yn), Pright-bottom(xn,yn)}, wherein the field of view angle of the infrared light temperature measurement camera C is smaller than that of the second visible light camera B; the working start distance value of the camera is set as the position of the minimum distance from the camera when the field of view of the infrared light temperature measurement camera C is a subset of the field of view of the second visible light camera B.

3. The binocular depth information-based dual light fusion calculation method according to claim 1, characterized in that, The step S3 comprises: S31, in a preset background environment, placing a light emitting point device on an angle position adjustment platform which can adjust the vertical angle and adjust the left and right and up and down movement; S32, adjusting the physical space position of the light emitting point device so that the light emitting point is imaged at a vertex position of the picture of the infrared camera C; S33, collecting the picture of the second visible light camera B, finding the light emitting point, and measuring and acquiring the coordinates of the light emitting point in the picture of the second visible light camera B; S34, with the vertex coordinates measured in the imaging picture of the second visible light camera B, the steps S32 and S33 are executed cyclically until the vertex coordinates of the top-left vertex Pleft-top(xn, yn), the top-right vertex Pright-top(xn, yn), the bottom-left vertex Pleft-bottom(xn, yn) and the bottom-right vertex Pright-bottom(xn, yn) in the picture of the second visible light camera B are obtained, so as to describe the infrared and visible light camera picture overlapping area N(deep) in the picture of the second visible light camera B; S35, the above calibration steps S31 to S34 are repeated again to obtain fusion calibration data at different depths: N(deep1), N(deep2), N(deep3)... N(deepi).

4. The binocular depth information-based dual light fusion calculation method according to claim 3, characterized in that, In the step S33, color difference data is collected and processed to determine the coordinates of the light emitting point device in the imaging picture of the second visible light camera B.

5. The binocular depth information based dual light fusion calculation method of claim 3, wherein, In the step S33, temperature difference data is collected and processed to determine the coordinates of the light emitting point device in the picture of the infrared temperature measurement camera C.

6. The binocular depth information based dual light fusion calculation method of claim 1, wherein, In the step S42, the depth information deep(xBK, yBK) value is processed to find the infrared and visible light camera picture overlapping area N(deep) with the closest depth to the current observation point.

7. The binocular depth information based dual light fusion calculation method of claim 1, wherein, The step S45 includes: S451, assuming that the width of the imaging picture of the second visible light camera B is w, the height is h, and the coordinates of a specific point in the imaging picture of the second visible light camera B are (x, y), the proportional coordinates of the specific point are calculated as x' = x / w and y' = y / h; S452, according to the four vertex coordinates {Pleft-top(xn, yn), Pright-top(xn, yn), Pleft-bottom(xn, yn), Pright-bottom(xn, yn)} of the infrared and visible light camera picture overlapping area N(deep) and the coordinates (xBK, yBK) of the observed point K in the imaging picture of the second visible light camera B, the width, height of the infrared and visible light camera picture overlapping area N(deep) and the coordinates of the observed point K are obtained by the following logical processing: wN = Pright-top(xn) - Pleft-top(xn); hN = Pleft-bottom(yn) - Pleft-top(yn); xNK = xBK - Pleft-top(xn); yNK = yBK - Pleft-top(yn); S453, the width, height of the infrared and visible light camera picture overlapping area N(deep) and the coordinates of the observed point K are processed by the following logic to obtain the proportional coordinates of the observed point K in the infrared and visible light camera picture overlapping area N(deep): ratiox = xNK / wN; ratioy = yNK / hN; S454, convert the proportional coordinates of the observed point K in the screen of the infrared camera C with the following logic: ratiox = xCK / wC; ratioy = yCK / hC; xCK = ratiox * wC; yCK = ratioy * hC; The width wC and height wH of the screen of the infrared camera C and the proportional coordinates are processed with the following logic, so as to obtain the coordinates (xCK, yCK) of the observed point K in the screen of the infrared camera C, and so as to obtain the temperature value of the observed point K: xCK = ratiox * wC = xNK / wN * wC = (xBK - Pleft-top(xn)) / (Pright-top(xn) - Pleft-top(xn)) * wC yCK = ratioy * hC = yNK / hN * hC = (yBK - Pleft-top(yn)) / (Pleft-bottom(yn) - Pleft-top(yn)) * hC.

8. The binocular depth information based dual light fusion calculation method of claim 1, wherein, The step S5 comprises: S51, traverse the corresponding coordinates of each point in the imaging area of the visible light camera B to be calculated, so as to obtain a set of temperature values; S52, combine and process the set of temperature values according to the engineering actual precision requirement data to obtain the private area temperature value as the visible light and temperature measurement camera dual light fusion result.

9. The binocular depth information based dual light fusion calculation method of claim 1, wherein, The first visible light camera A and the second visible light camera B are of the same specification; the optical axes of the first visible light camera A, the second visible light camera B and the infrared light temperature measurement camera C are parallel and the imaging planes are parallel.

10. A dual light fusion computing system based on binocular depth information, characterized in that, The system comprises: An engineering requirement acquisition module is configured to acquire and process engineering actual precision requirement data to calibrate the second visible light camera B and the infrared temperature measurement camera C at a difference depth distance, wherein the cameras comprise a first visible light camera A, a second visible light camera B and an infrared light temperature measurement camera C; A target calibration module is configured to find an imaging screen rectangular area M in a specific depth physical space to calibrate a measured target, wherein the imaging screen rectangular area M completely covers the infrared temperature measurement camera C, measures the area of the imaging screen rectangular area M on the imaging screen of the second visible light camera B to obtain a rectangular pixel point set N, and the target calibration module is connected with the engineering requirement acquisition module; A vertex processing module is configured to calibrate vertex coordinates, and the vertex processing module is connected with the target calibration module; A measured point matching module is configured to match and process an observed point K in the second visible light camera B, and the measured point matching module is connected with the vertex processing module and the target calibration module, and the measured point matching module comprises: a depth information module, configured to obtain observed point coordinates (xBK, yBK) of the observed point K in the imaging picture of the second visible light camera B, and calculate depth information deep(xBK, yBK) of the observed point K based on a binocular disparity principle and depth calculation of the imaging picture of the first visible light camera A and the imaging picture of the second visible light camera B; an infrared region acquisition module, configured to process the depth information deep(xBK, yBK) to obtain an infrared and visible light camera picture overlapping region, the infrared region acquisition module being connected with the depth information module; an observed point position judgment module, configured to judge whether the observed point K is in a corresponding calibration data infrared field of view covering region, the observed point position judgment module being connected with the infrared region acquisition module; an out-of-field-of-view judgment module, configured to judge that a measured object does not enter a field of view of the infrared temperature measurement camera C and that temperature measurement is impossible when the observed point K is not in the corresponding calibration data infrared region, the out-of-field-of-view judgment module being connected with the observed point position judgment module; an in-field-of-view judgment module, configured to calculate coordinates of the observed point K in the infrared light camera C and extract a temperature value matching the observed point when the observed point K is in the corresponding calibration data infrared region, to obtain the temperature value of the observed point K from the imaging picture of the second visible light camera B, the in-field-of-view judgment module being connected with the observed point position judgment module; a region matching module, configured to match and calculate region data in the second visible light camera B, the region matching module being connected with the vertex processing module and the target calibration module.

Images