A method and device for calibrating dual-light camera imaging, and dual-light camera

Abstract

The present invention discloses a method and device for bi-light camera imaging calibration and a bi-light camera, which obtains a thermal imaging image including a first calibration frame image and a visible light image including a second calibration frame image, wherein the first calibration frame image and the second calibration frame image are images of the same calibration frame taken by the thermal imaging camera and the visible light camera respectively; obtains first position information of the first calibration frame image in the thermal imaging image and second position information of the second calibration frame in the visible light image; generates calibration parameters according to the first position information and the second position information; and adjusts the position of the thermal imaging image synthesized in the visible light image in the visible light image according to the calibration parameters. It can effectively realize fine calibration work before leaving the factory and solve the problem that the imaging cannot be integrated or seamlessly connected due to physical displacement or deformation after long-term use.

Description

Technical Field

[0001] The present invention relates to the technical field of aerial cameras, and in particular to a method and device for bifocal camera imaging calibration and a bifocal camera. Background Art

[0002] Unmanned aerial vehicles (UAVs) are unmanned aircraft that are controlled by radio remote control equipment and self-provided program control terminals. They involve sensor technology, communication technology, information processing technology, intelligent control technology, and power propulsion technology, and are high-tech products of the information age.

[0003] Industrial drones are often used in industrial sites. In order to meet the needs of working in the fields of power monitoring, petrochemicals, crop fields, forest fields, etc., industrial drones often carry dual-light cameras. Common dual-light cameras are cameras that combine thermal imaging and visible light. Such cameras can record and transmit thermal imaging and visible light images at the same time. However, dual-light cameras use two sensors or cameras, and there is a relative physical displacement between the two sensors or cameras in different directions or positions. If two separate images or video streams captured by dual-light cameras are to be combined, fused, or seamlessly displayed, the alignment of the resulting images is particularly important. Traditionally, such registration or alignment of separately captured images or video streams can be achieved by mechanically aligning the position and angle between sensors or cameras, but this makes the cost of manufacturing optical components and imaging sensors extremely high and the requirements are extremely strict, and there is no guarantee that such alignment will not change strictly during use.

[0004] Therefore, a simple and easy-to-operate calibration method is needed to enable dual-light camera or multi-sensor imaging to be free from the limitations of physical displacement and angle, so that images taken by two or more cameras or sensors can be combined, fused or seamlessly displayed. Summary of the invention

[0005] The embodiments of the present invention provide a method and device for bi-focal camera imaging calibration and a bi-focal camera, which can not only effectively implement fine calibration work before leaving the factory, but also solve the problem that the bi-focal shooting structure or camera lens is physically displaced or deformed due to long-term use, resulting in the inability to fuse or seamlessly connect the images.

[0006] According to one aspect of an embodiment of the present invention, a method for calibrating a dual-light camera imaging is provided, which is used for a dual-light camera, wherein the dual-light camera includes a thermal imaging camera and a visible light camera, and the method includes: obtaining a thermal imaging image including a first calibration frame image and a visible light image including a second calibration frame image, wherein the first calibration frame image and the second calibration frame image are images of the same calibration frame taken by the thermal imaging camera and the visible light camera, respectively; obtaining first position information of the first calibration frame image in the thermal imaging image and second position information of the second calibration frame in the visible light image, wherein the first position information includes a width, a height and a coordinate of the first calibration frame image in the thermal imaging image, and the second position information includes a width, a height and a coordinate of the second calibration frame in the visible light image; generating calibration parameters according to the first position information and the second position information, wherein the calibration parameters include a width, a height and a coordinate of the thermal imaging image synthesized in the visible light image; and adjusting a position of the thermal imaging image synthesized in the visible light image in the visible light image according to the calibration parameters.

[0007] Optionally, the calibration frame is a rectangular heating frame.

[0008] Optionally, obtaining first position information of the first calibration frame image in the thermal imaging image and second position information of the second calibration frame in the visible light image includes: performing image processing on the thermal imaging image and the visible light image respectively; and extracting edges of the thermal imaging image and the visible light image after the image processing using an edge detection algorithm, thereby obtaining the first position information and the second position information.

[0009] Optionally, the image processing includes at least one of image denoising and contrast enhancement.

[0010] Optionally, generating the calibration parameter according to the first position information and the second position information includes:

[0011] The calibration parameters are generated using the following formula:

[0012]

[0013] Wherein, w is the width of the thermal imaging image synthesized in the visible light image, w r is the width of the thermal imaging image, w 0 is the width of the first calibration frame image in the thermal imaging image, w 1 is the width of the second calibration frame image in the visible light image, h is the height of the thermal imaging image synthesized in the visible light image, and h ris the height of the thermal imaging image, h 1 is the height of the second calibration frame image in the visible light image, h 0 is the height of the first calibration frame image in the thermal imaging image, x is the abscissa of the top left corner of the thermal imaging image synthesized in the visible light image, and x 1 is the horizontal coordinate of the upper left corner vertex of the second calibration frame image in the visible light image, x 0 is the horizontal coordinate of the top left corner of the first calibration frame image in the thermal imaging image, y is the vertical coordinate of the top left corner of the thermal imaging image synthesized in the visible light image, and y 1 is the ordinate of the upper left corner vertex of the second calibration frame image in the visible light image, y 0 is the ordinate of the upper left corner vertex of the first calibration frame image in the thermal imaging image.

[0014] Another aspect of an embodiment of the present invention provides a dual-light camera, including:

[0015] A thermal imaging camera, used to obtain a thermal imaging image including an image of a first calibration frame;

[0016] a visible light camera, configured to acquire a visible light image including a second calibration frame image, wherein the first calibration frame image and the second calibration frame image are images of the same calibration frame taken by the thermal imaging camera and the visible light camera respectively; and

[0017] A processor, the processor being communicatively connected to the thermal imaging camera and the visible light camera, wherein the processor is configured to:

[0018] Acquire first position information of the first calibration frame image in the thermal imaging image and second position information of the second calibration frame in the visible light image, wherein the first position information includes the width, height and coordinates of the first calibration frame image in the thermal imaging image, and the second position information includes the width, height and coordinates of the second calibration frame in the visible light image;

[0019] generating calibration parameters according to the first position information and the second position information, wherein the calibration parameters include a width, a height, and a coordinate of a thermal imaging image synthesized in the visible light image; and

[0020] The position of the thermal imaging image synthesized in the visible light image is adjusted according to the calibration parameter.

[0021] In one embodiment of the present invention, the processor is further configured to:

[0022] performing image processing on the thermal imaging image and the visible light image respectively;

[0023] The edges of the thermal imaging image and the visible light image after the image processing are extracted by using an edge detection algorithm, so as to obtain the first position information and the second position information.

[0024] In an embodiment of the present invention, the image processing includes at least one of image denoising and contrast enhancement.

[0025] In one embodiment of the present invention, the processor is used to:

[0026] The calibration parameters are generated using the following formula:

[0027]

[0028] Wherein, w is the width of the thermal imaging image synthesized in the visible light image, w r is the width of the thermal imaging image, w 0 is the width of the first calibration frame image in the thermal imaging image, w 1 is the width of the second calibration frame image in the visible light image, h is the height of the thermal imaging image synthesized in the visible light image, and h r is the height of the thermal imaging image, h 1 is the height of the second calibration frame image in the visible light image, h 0 is the height of the first calibration frame image in the thermal imaging image, x is the abscissa of the top left corner of the thermal imaging image synthesized in the visible light image, and x 1 is the horizontal coordinate of the upper left corner vertex of the second calibration frame image in the visible light image, x 0 is the horizontal coordinate of the top left corner of the first calibration frame image in the thermal imaging image, y is the vertical coordinate of the top left corner of the thermal imaging image synthesized in the visible light image, and y 1 is the ordinate of the upper left corner vertex of the second calibration frame image in the visible light image, y 0 is the ordinate of the upper left corner vertex of the first calibration frame image in the thermal imaging image.

[0029] Another aspect of an embodiment of the present invention is a computer-readable storage medium, characterized in that one or more programs are stored on the computer-readable storage medium, and the one or more programs can be executed by one or more processors to implement the method for bi-optical camera imaging calibration as described above.

[0030] Another aspect of an embodiment of the present invention provides a dual-light camera imaging calibration device, including:

[0031] An image acquisition module, used to acquire a thermal imaging image including a first calibration frame image and a visible light image including a second calibration frame image, wherein the first calibration frame image and the second calibration frame image are images of the same calibration frame acquired by the image acquisition module;

[0032] a position acquisition module, configured to acquire first position information of the first calibration frame image in the thermal imaging image and second position information of the second calibration frame in the visible light image, wherein the first position information includes a width, a height, and a coordinate of the first calibration frame image in the thermal imaging image, and the second position information includes a width, a height, and a coordinate of the second calibration frame in the visible light image;

[0033] a calibration module, configured to generate calibration parameters according to the first position information and the second position information, wherein the calibration parameters include a width, a height, and a coordinate of a thermal imaging image synthesized in the visible light image; and

[0034] A position adjustment module is used to adjust the position of the thermal imaging image synthesized in the visible light image in the visible light image according to the calibration parameters.

[0035] Optionally, the calibration frame is a rectangular heating frame.

[0036] Optionally, the position acquisition module is also used to: perform image processing on the thermal imaging image and the visible light image respectively; and use an edge detection algorithm to extract the edges of the thermal imaging image and the visible light image after the image processing, so as to obtain the first position information and the second position information.

[0037] Optionally, the image processing includes at least one of image denoising and contrast enhancement.

[0038] Optionally, the calibration module is further used for:

[0039] Generating the calibration parameter according to the first position information and the second position information includes: generating the calibration parameter using the following formula:

[0040]

[0041] Wherein, w is the width of the thermal imaging image synthesized in the visible light image, w r is the width of the thermal imaging image, w 0 is the width of the first calibration frame image in the thermal imaging image, w 1is the width of the second calibration frame image in the visible light image, h is the height of the thermal imaging image synthesized in the visible light image, and h r is the height of the thermal imaging image, h 1 is the height of the second calibration frame image in the visible light image, h 0 is the height of the first calibration frame image in the thermal imaging image, x is the abscissa of the top left corner of the thermal imaging image synthesized in the visible light image, and x 1 is the horizontal coordinate of the upper left corner vertex of the second calibration frame image in the visible light image, x 0 is the horizontal coordinate of the top left corner of the first calibration frame image in the thermal imaging image, y is the vertical coordinate of the top left corner of the thermal imaging image synthesized in the visible light image, and y 1 is the ordinate of the upper left corner vertex of the second calibration frame image in the visible light image, y 0 is the ordinate of the upper left corner vertex of the first calibration frame image in the thermal imaging image.

[0042] The embodiments of the present invention provide a method, an apparatus, a computer-readable storage medium and a bi-light camera for bi-light camera imaging calibration. The method comprises acquiring a thermal imaging image including a first calibration frame image and a visible light image including a second calibration frame image, wherein the first calibration frame image and the second calibration frame image are images of the same calibration frame taken by the thermal imaging camera and the visible light camera respectively; acquiring first position information of the first calibration frame image in the thermal imaging image and second position information of the second calibration frame in the visible light image, wherein the first position information comprises the width, height and coordinates of the first calibration frame image in the thermal imaging image, and the second position information comprises the width, height and coordinates of the second calibration frame in the visible light image; generating calibration parameters according to the first position information and the second position information, wherein the calibration parameters comprise the width, height and coordinates of the thermal imaging image synthesized in the visible light image; and adjusting the position of the thermal imaging image synthesized in the visible light image according to the calibration parameters; solving the problem that different sensors of the bi-light camera are not accurately fused due to relative physical distance, the calibration method is simple and convenient, and the bi-light imaging display effect is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the embodiments of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0044] Figure 1A schematic structural diagram of a dual-light camera according to an embodiment of the present invention;

[0045] Figure 2 is a schematic diagram of coordinates of a thermal imaging image including a first calibration frame image acquired by a thermal imaging camera in a dual-light camera according to an embodiment of the present invention;

[0046] Figure 3 is a schematic diagram of coordinates of a visible light image including a second calibration frame image acquired by a visible light camera in a dual-light camera according to an embodiment of the present invention;

[0047] Figure 4 A schematic flow chart of a method for calibrating dual-light camera imaging according to an embodiment of the present invention;

[0048] Figure 5 The figure is a structural diagram of a dual-light camera imaging calibration device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention clearer, the embodiments of the present invention will be described in detail in conjunction with the accompanying drawings. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other arbitrarily without conflict. In addition, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order from the module division in the device schematic diagram or the order in the flow chart.

[0050] See also Figure 1 , Figure 1 is a schematic diagram of the structure of a dual-light camera 101. In one embodiment of the present invention, the dual-light camera 101 includes a thermal imaging camera 200, a visible light camera 300, and a processor 400. The thermal imaging camera 200 and the visible light camera 300 are both connected to the processor 400 for communication.

[0051] In one embodiment of the present invention, the thermal imaging camera 200 of the dual-light camera 101 is used to capture a thermal radiation image of a target object, and the visible light camera 300 is used to capture a visible light wavelength range image of the target object.

[0052] The thermal imaging image captured by the thermal imaging camera 200 and the visible light image captured by the visible light camera 300 are fused in a picture-in-picture manner to form a dual light image. After the thermal imaging image and the visible light image are merged in a certain ratio, calibration work needs to be performed. Usually, the calibration work can be performed through a calibration frame.

[0053] The calibration frame is manually selected, and the object selected as the calibration frame needs to have a certain shape, and the uniform heating area and the visible light area are overlapping. This is to ensure that the heating part and the visible light part are consistent, so that the calibration frame images captured by the thermal imaging camera and the visible light camera are the same except that the pixel size, scale size or color may be different. If the heating is uneven or only a certain area is heated, the calibration frame images in the image obtained by the thermal imaging camera and the image obtained by the visible light camera will be completely different. In this way, the variables cannot be controlled and cannot be used as a calibration frame to achieve a reference effect. Therefore, the calibration frame must have a certain shape, and the heating area and the visible light area are overlapping. Therefore, in one embodiment of the present invention, the calibration frame is a rectangular heating frame. In addition, each side of the calibration frame is parallel to the corresponding edge of the thermal imaging image and the corresponding edge of the visible light image. Moreover, the larger the area ratio of the calibration frame to the thermal imaging image, the more accurate the calibration result.

[0054] In actual situations, for example, in factory automatic calibration, a factory-made frame with a higher temperature and an overlapping heating area with a visible light area is set as the calibration frame. The factory has a dedicated calibration wall, which is perpendicular to the optical axis of the thermal imaging camera and the visible light camera, and the wall has a single brightness color and a consistent temperature. The temperature of the calibration frame is higher than that of the calibration wall (for example, the calibration wall does not generate heat). The calibration frame is placed in front of the calibration wall to take an image.

[0055] In user-friendly automatic calibration, a uniform heat source, such as a computer monitor, can be used as a calibration frame. The calibration frame is placed in front of a wall of a single color and uniform temperature (for example, a white wall in a room), and the thermal imaging camera and the visible light camera are used to capture the image.

[0056] In one embodiment of the present invention, after the calibration frame is selected and set, the thermal imaging camera 200 acquires a thermal imaging image including a first calibration frame image, and the visible light camera 300 acquires a visible light image including a second calibration frame image. The first calibration frame image and the second calibration frame image are images of the same calibration frame taken by the thermal imaging camera and the visible light camera, respectively.

[0057] In one embodiment of the present invention, the processor 400 of the dual-light camera 101 is further configured to obtain first position information of the first calibration frame image in the thermal imaging image and second position information of the second calibration frame in the visible light image.

[0058] The thermal imaging camera 200 sends the thermal imaging image including the first calibration frame image to the processor 400, and the processor 400 obtains the first position information of the first calibration frame image in the thermal imaging image, wherein the first position information includes the width, height and coordinates of the first calibration frame image in the thermal imaging image.

[0059] Similarly, the visible light camera 300 sends the visible light image including the second calibration frame image to the processor 400, and the processor 400 obtains the second position information of the second calibration frame image in the visible light image, wherein the second position information includes the width, height and coordinates of the second calibration frame in the visible light image.

[0060] The thermal imaging image and the visible light image are processed by image processing, such as denoising, contrast enhancement, etc., and then edge extraction is performed. In the process of edge extraction, the first position information and the second position information can be obtained. In one embodiment of the present invention, a mature edge detection extraction operator (Canny operator) can be used to implement edge extraction.

[0061] In one embodiment of the present invention, Figure 2 As shown, taking the rectangular calibration frame as an example, the upper left corner vertex of the thermal imaging image is set as the origin (0, 0), then the coordinates of the other three vertices of the entire image are (w r ,0)、(w r ,h r ) and (0, h r ). r -0=h r is the height of the thermal imaging image, w r -0=w r is the width of the thermal image. Image recognition is performed, the thermal image is denoised, contrast enhanced, and then edge extracted, and edge points are connected to form an edge map, thereby obtaining the coordinates of each vertex of the calibration frame, and then the width and height of the calibration frame can be obtained from the distance between the coordinates. Thus, the processor 400 obtains the coordinates of the upper left corner vertex of the calibration frame as (x 0 ,y 0 ), the upper right corner coordinate (x 0r ,y 0 ), and the lower left corner coordinate (x 0 ,y 0b ), and at the same time, the width w of the first calibration frame image in the thermal imaging image is obtained by calculation 0 =x 0r -x 0 , and the height h of the first calibration frame image in the thermal image 0 =y 0b -y 0 .

[0062] In one embodiment of the present invention, Figure 3 As shown, by way of example and not limitation, the top left vertex of the visible light image is set as the origin (0, 0). Image recognition is performed, for example, the visible light image is subjected to denoising, contrast enhancement, edge extraction, and edge points are connected to form an edge map (the image recognition process in the prior art is not specifically described in the present invention), and finally the coordinates of each vertex of the calibration frame are obtained, and then the width and height can be obtained by subtracting the coordinates. Thus, for example, the coordinates of the top left vertex of the calibration frame in the visible light image are obtained as (x 1 ,y 1 ), the upper right corner coordinate (x 1r ,y 1 ), and the lower left corner coordinate (x 1 ,y 1b ), the width w of the second calibration frame image in the visible light image is obtained by calculation 1 =x 1r -x 1 , and the height h of the second calibration frame image in the visible light image 1 =y 1b -y 1 .

[0063] In one embodiment of the present invention, after acquiring the first position information and the second position information, the processor 400 generates calibration parameters according to the first position information and the second position information, wherein the calibration parameters include the width, height and coordinates of the thermal imaging image synthesized in the visible light image.

[0064] Among them, Figure 2 and Figure 3 Take the coordinates in as an example, the calibration parameters include the coordinates of the upper left corner vertex of the thermal imaging image synthesized in the visible light image, as well as the width and height of the thermal imaging image. Assume that the coordinates of the upper left corner vertex of the thermal imaging image synthesized in the visible light image are (x, y), the width is w, and the height is h. Since the ratio between the calibration frame and the entire thermal imaging image is unchanged, we have:

[0065]

[0066]

[0067]

[0068]

[0069] Thus, the calibration parameters can be obtained: the width w, height h, and coordinates (x, y) of the upper left corner of the thermal imaging image synthesized in the visible light image. In addition, the coordinates of other vertices can also be calculated, which are (x+w, y), (x+w, y+h), and (x, y+h) in clockwise order.

[0070] After acquiring the calibration parameters, the processor 400 may adjust the position of the thermal imaging image synthesized in the visible light image in the visible light image according to the calibration parameters.

[0071] Figure 4 A flowchart of a method for calibrating a dual-light camera imaging is provided in accordance with an embodiment of the present invention. The method is performed by a dual-light camera, wherein the dual-light camera includes a thermal imaging camera and a visible light camera, and the method includes:

[0072] Step S401: acquiring a thermal imaging image including a first calibration frame image and a visible light image including a second calibration frame image, wherein the first calibration frame image and the second calibration frame image are images of the same calibration frame taken by the thermal imaging camera and the visible light camera respectively;

[0073] The thermal imaging image captured by the thermal imaging camera 200 and the visible light image captured by the visible light camera 300 are fused in a picture-in-picture manner to form a dual light image. After the thermal imaging image and the visible light image are merged in a certain ratio, calibration work needs to be performed. Usually, the calibration work can be performed through a calibration frame.

[0074] Wherein, the calibration frame is selected artificially,

[0075] Among them, the object selected as the calibration frame needs to have a certain shape, and the heating area and the visible light area are overlapping. This is to ensure that the heating part and the visible light part are consistent, so that the first calibration frame image and the second calibration frame image captured by the thermal imaging camera and the visible light camera are the same except that the pixel size, scale size or color may be different. If the heating is uneven or only a certain area is heated, the calibration frame image in the image obtained by the thermal imaging camera and the image obtained by the visible light camera will be completely different. If the variables cannot be controlled, it cannot be used as a calibration frame to achieve a reference effect. Therefore, the calibration frame must have a certain shape, and the heating area and the visible light area are overlapping. In one embodiment of the present invention, the calibration frame is a rectangular heating frame. Among them, each side of the calibration frame is parallel to the corresponding edges of the thermal imaging image and the visible light image. Moreover, the larger the area ratio of the thermal imaging image occupied by the calibration frame, the more accurate the calibration result.

[0076] In actual situations, for example, in factory automatic calibration, a factory-made frame with a higher temperature and an overlapping heating area with a visible light area is set as a calibration frame. The factory is equipped with a dedicated calibration wall, which is perpendicular to the two cameras and has a single wall brightness and color and a consistent temperature. The temperature of the calibration frame is higher than that of the calibration wall. The calibration frame is set in front of the calibration wall, and a thermal imaging image containing a first calibration frame image and a visible light image containing a second calibration frame image are obtained respectively.

[0077] In the user's simple automatic calibration, a uniform heat source that exists in the user's life and is easily available, such as a computer display screen, is set as a calibration frame. After aligning the dual-light camera with the calibration frame, the user presses the automatic calibration button of the control terminal or issues an instruction to trigger the automatic calibration function of the dual-light camera in other ways (for example, options in a mobile phone APP, etc.), and the processor in the dual-light camera automatically starts calibration. In one embodiment of the present invention, the user's simple automatic calibration also includes a calibration wall, the temperature of the calibration frame is higher than that of the calibration wall, the calibration wall is perpendicular to the dual-light camera (for example, the dual-light lens), and the wall brightness color is single and the temperature is consistent. The calibration wall is selected by the user independently, and the calibration wall can be a pure color wall or background in the user's life, and the temperature of the calibration wall is lower than that of the calibration frame. The calibration frame is set in front of the calibration wall, and a thermal imaging image containing a first calibration frame image and a visible light image containing a second calibration frame image are respectively obtained.

[0078] Step S402: Acquire first position information of the first calibration frame image in the thermal imaging image and second position information of the second calibration frame in the visible light image, wherein the first position information includes the width, height and coordinates of the first calibration frame image in the thermal imaging image, and the second position information includes the width, height and coordinates of the second calibration frame in the visible light image;

[0079] In one embodiment of the present invention, the first position information and the second position information are obtained by: performing image processing on the thermal imaging image; using an edge detection algorithm to extract the edges of the thermal imaging image and the visible light image after the image processing, thereby obtaining the first position information and the second position information.

[0080] The image processing includes at least one of image denoising and contrast enhancement.

[0081] In one embodiment of the present invention, Figure 2 As shown, taking the rectangular calibration frame as an example, for illustrative purposes only and not limiting, the top left vertex of the thermal imaging image is set as the origin (0, 0), then the coordinates of the other three vertices of the entire image are (w r ,0)、(w r,h r ) and (0, h r ). r is the height of the thermal imaging image, w r is the width of the thermal image. Image recognition is performed, the thermal image is subjected to denoising, contrast enhancement, and edge extraction (for example, the existing mature edge detection extraction operator, Canny operator, can be used to implement edge extraction), and the edge points are connected to form an edge map (the image recognition process in the prior art is not specifically described in the present invention), thereby obtaining the coordinates of each vertex of the calibration frame, and then the width and height of the calibration frame can be obtained from the distance between the coordinates. Therefore, the processor 400 obtains the coordinates of the upper left vertex of the calibration frame as (x 0 ,y 0 ), the upper right corner coordinate (x 0r ,y 0 ), and the lower left corner coordinate (x 0 ,y 0b ), and at the same time, the width w of the first calibration frame image in the thermal imaging image is obtained by calculation 0 =x 0r -x 0 , and the height h of the first calibration frame image in the thermal image 0 =y 0b -y 0 .

[0082] In one embodiment of the present invention, Figure 3 As shown, by way of example and not limitation, the top left vertex of the visible light image is set as the origin (0, 0). Image recognition is performed, for example, the visible light image is subjected to denoising, contrast enhancement, edge extraction, and edge points are connected to form an edge map (the image recognition process in the prior art is not specifically described in the present invention), and finally the coordinates of each vertex of the calibration frame are obtained, and then the width and height can be obtained by subtracting the coordinates. Thus, for example, the coordinates of the top left vertex of the calibration frame in the visible light image are obtained as (x 1 ,y 1 ), the upper right corner coordinate (x 1r ,y 1 ), and the lower left corner coordinate (x 1 ,y 1b ), the width w of the second calibration frame image in the visible light image is obtained by calculation 1 =x 1r -x 1 , and the height h of the second calibration frame image in the visible light image 1 =y 1b -y 1 .

[0083] Step S403: Generate calibration parameters according to the first position information and the second position information, wherein the calibration parameters include the width, height and coordinates of the thermal imaging image synthesized in the visible light image:

[0084] Among them, Figure 2 and Figure 3 Taking the coordinates in as an example, the calibration parameters include the coordinates of the upper left corner vertex of the thermal imaging image synthesized in the visible light image, and the width and height of the scaled thermal imaging image synthesized in the visible light image. Assume that the coordinates of the upper left corner vertex of the thermal imaging image synthesized in the visible light image are (x, y), the width is w, and the height is h. Since the ratio between the calibration frame and the entire thermal imaging image is unchanged, we have:

[0085]

[0086]

[0087]

[0088]

[0089] Thus, the calibration parameters can be obtained: the width w, height h, and coordinates (x, y) of the upper left corner of the thermal imaging image synthesized in the visible light image. In addition, the coordinates of other vertices can also be calculated, which are (x+w, y), (x+w, y+h), and (x, y+h) in clockwise order.

[0090] Step S404: adjusting the position of the thermal imaging image synthesized in the visible light image in the visible light image according to the calibration parameters.

[0091] In the embodiments of the present invention, the principle that the proportion of the calibration frame image in the thermal imaging image remains unchanged in the dual-light imaging fusion is utilized to solve the problem that physical deviations in dual-light camera or multi-sensor imaging make the imaging unable to be fused or have errors.

[0092] Therefore, the embodiments of the present invention provide a factory automatic calibration method and a user simple automatic calibration method, which can easily and reliably solve the problem that the jointly generated and displayed images cannot be completely integrated or seamlessly spliced ​​due to the different physical positions and angles of multiple sensors. At the same time, during the user's use, once the problem of image non-fusion or error occurs, the user automatic calibration instruction can be triggered to solve the problem that the jointly generated and displayed images cannot be completely integrated or seamlessly spliced ​​due to the changes in the relative physical positions and different angles of multiple sensors during use. It is simple and easy to operate, greatly improving the user experience.

[0093] The embodiments of the present invention also provide a computer-readable storage medium. A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program. The program can be stored in the computer-readable storage medium. When one or more programs are executed by one or more processors, the processes of the above-mentioned embodiments can be included. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM). The storage medium and the one or more processors are included in the mobile control terminal.

[0094] like Figure 5 As shown, the embodiment of the present invention further provides a dual-light camera imaging calibration device 500, the device 500 includes: an image acquisition module 501, a position acquisition module 502, a calibration module 503, and a position adjustment module 504, and the modules are communicatively connected.

[0095] The image acquisition module 501 is used to acquire a thermal imaging image including a first calibration frame image and a visible light image including a second calibration frame image, wherein the first calibration frame image and the second calibration frame image are images of the same calibration frame acquired by the image acquisition module.

[0096] The acquired thermal imaging image and visible light image are fused in a picture-in-picture manner to form a dual light image. After the thermal imaging image and the visible light image are merged in a certain ratio, calibration is required. Usually, the calibration can be performed through a calibration frame.

[0097] Among them, the calibration frame is artificially selected, and the object selected as the calibration frame needs to have a certain shape, and the heating area and the visible light area are overlapping. This is to ensure that the heating part and the visible light part are consistent, so that the first calibration frame image and the second calibration frame image captured by the thermal imaging camera and the visible light camera are the same except that the pixel size, scale size or color may be different. If the heating is uneven or only a certain area is heated, the calibration frame image in the image obtained by the thermal imaging camera and the image obtained by the visible light camera will be completely different, and the variables cannot be controlled and cannot be used as a calibration frame to achieve a reference effect. Therefore, the calibration frame must have a certain shape, and the heating area and the visible light area are overlapping. In one embodiment of the present invention, the calibration frame is a rectangular heating frame. In addition, each side of the calibration frame is parallel to the edge of the corresponding thermal imaging image and the edge of the visible light image. Moreover, the larger the area ratio of the calibration frame to the thermal imaging image, the more accurate the calibration result.

[0098] The position acquisition module 502 is used to obtain the first position information of the first calibration frame image in the thermal imaging image and the second position information of the second calibration frame in the visible light image, wherein the first position information includes the width, height and coordinates of the first calibration frame image in the thermal imaging image, and the second position information includes the width, height and coordinates of the second calibration frame in the visible light image.

[0099] In one embodiment of the present invention, the position acquisition module 502 is further used to perform image processing on the thermal imaging image; using an edge detection algorithm, extract the edges of the thermal imaging image and the visible light image after the image processing, so as to obtain the first position information and the second position information. The image processing includes at least one of image denoising and contrast enhancement.

[0100] In one embodiment of the present invention, Figure 2 As shown, taking the rectangular calibration frame as an example, for illustrative purposes only and not limiting, the top left vertex of the thermal imaging image is set as the origin (0, 0), then the coordinates of the other three vertices of the entire image are (w r ,0)、(w r ,h r ) and (0, h r ). r is the height of the thermal imaging image, w r is the width of the thermal image. Image recognition is performed, the thermal image is subjected to denoising, contrast enhancement, and edge extraction (for example, the existing mature edge detection extraction operator, Canny operator, can be used to implement edge extraction), and the edge points are connected to form an edge map (the image recognition process in the prior art is not specifically described in the present invention), thereby obtaining the coordinates of each vertex of the calibration frame, and then the width and height of the calibration frame can be obtained from the distance between the coordinates. Therefore, the processor 400 obtains the coordinates of the upper left vertex of the calibration frame as (x 0 ,y 0 ), the upper right corner coordinate (x 0r ,y 0 ), and the lower left corner coordinate (x 0 ,y 0b ), and at the same time, the width w of the first calibration frame image in the thermal imaging image is obtained by calculation 0 =x 0r -x 0 , and the height h of the first calibration frame image in the thermal image 0 =y 0b -y 0 .

[0101] In one embodiment of the present invention, Figure 3As shown, by way of example and not limitation, the top left vertex of the visible light image is set as the origin (0, 0). Image recognition is performed, for example, the visible light image is subjected to denoising, contrast enhancement, edge extraction, and edge points are connected to form an edge map (the image recognition process in the prior art is not specifically described in the present invention), and finally the coordinates of each vertex of the calibration frame are obtained, and then the width and height can be obtained by subtracting the coordinates. Thus, for example, the coordinates of the top left vertex of the calibration frame in the visible light image are obtained as (x 1 ,y 1 ), the upper right corner coordinate (x 1r ,y 1 ), and the lower left corner coordinate (x 1 ,y 1b ), the width w of the second calibration frame image in the visible light image is obtained by calculation 1 =x 1r -x 1 , and the height h of the second calibration frame image in the visible light image 1 =y 1b -y 1 .

[0102] The calibration module 503 is configured to generate calibration parameters according to the first position information and the second position information, wherein the calibration parameters include a width, a height, and coordinates of the thermal imaging image synthesized in the visible light image.

[0103] In one embodiment of the present invention, Figure 2 and Figure 3 The coordinates in are examples.

[0104] The calibration parameters include the coordinates of the upper left corner vertex of the thermal imaging image synthesized in the visible light image, as well as the width and height of the thermal imaging image. Assume that the coordinates of the upper left corner vertex of the thermal imaging image synthesized in the visible light image are (x, y), the width is w, and the height is h. Since the ratio between the calibration frame and the entire thermal imaging image is unchanged, we have:

[0105]

[0106]

[0107]

[0108]

[0109] Thus, the calibration parameters can be obtained: the width w, height h, and coordinates (x, y) of the upper left corner of the thermal imaging image synthesized in the visible light image. In addition, the coordinates of other vertices can also be calculated, which are (x+w, y), (x+w, y+h), and (x, y+h) in clockwise order.

[0110] The position adjustment module 504 is used to adjust the position of the thermal imaging image synthesized in the visible light image in the visible light image according to the calibration parameters.

[0111] In the embodiments of the present invention, the principle that the proportion of the calibration frame in the thermal imaging image remains unchanged in dual-light imaging fusion is utilized to solve the problem that physical deviations in dual-light camera or multi-sensor imaging make the imaging unable to be fused or have errors.

[0112] The present invention proposes a method for bi-optical camera imaging calibration. By setting a calibration frame, obtaining the image position of the calibration frame in thermal imaging and visible light respectively, and then using the proportional relationship to calibrate the relative position of the thermal imaging image and the visible light image, the bi-optical camera or multi-sensor imaging can be free from the restrictions of relative physical displacement and angle change, so that the images of two or more cameras or sensors can be combined, fused or seamlessly connected for display.

[0113] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits, and accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software function modules. The present invention is not limited to any specific form of combination of hardware and software.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for bi-optical camera imaging calibration, characterized in that: include: Acquire a thermal imaging image including a first calibration frame image and a visible light image including a second calibration frame image; The first calibration frame image and the second calibration frame image are images of the same calibration frame taken by a thermal imaging camera and a visible light camera respectively; the calibration frame is a rectangular thermal frame, and each side of the calibration frame is parallel to the corresponding edge of the thermal imaging image and the corresponding edge of the visible light image; Acquire first position information of the first calibration frame image and second position information of the second calibration frame; Generate calibration parameters according to the ratio between the first calibration frame image and the thermal imaging image being equal to the ratio between the second calibration frame image and the thermal imaging image in the visible light image, and the first position information and the second position information, wherein the calibration parameters include the width, height, and coordinates of the thermal imaging image synthesized in the visible light image; and According to the calibration parameters, adjusting the width, height and coordinates of the thermal imaging image synthesized in the visible light image; The generating the calibration parameter according to the first position information and the second position information includes: The calibration parameters are generated using the following formula: Wherein, w is the width of the thermal imaging image synthesized in the visible light image, w r is the width of the thermal imaging image, w0 is the width of the first calibration frame image in the thermal imaging image, w1 is the width of the second calibration frame image in the visible light image, h is the height of the thermal imaging image synthesized in the visible light image, and h r is the height of the thermal imaging image, h1 is the height of the second calibration frame image in the visible light image, h0 is the height of the first calibration frame image in the thermal imaging image, x is the horizontal coordinate of the upper left corner vertex of the thermal imaging image synthesized in the visible light image, x1 is the horizontal coordinate of the upper left corner vertex of the second calibration frame image in the visible light image, x0 is the horizontal coordinate of the upper left corner vertex of the first calibration frame image in the thermal imaging image, y is the vertical coordinate of the upper left corner vertex of the thermal imaging image synthesized in the visible light image, y1 is the vertical coordinate of the upper left corner vertex of the second calibration frame image in the visible light image, and y0 is the vertical coordinate of the upper left corner vertex of the first calibration frame image in the thermal imaging image.

2. The method according to claim 1, characterized in that The first position information includes the width, height and coordinates of the first calibration frame image in the thermal imaging image, and the second position information includes the width, height and coordinates of the second calibration frame in the visible light image.

3. The method according to claim 1, characterized in that Acquiring first position information of the first calibration frame image in the thermal imaging image and second position information of the second calibration frame in the visible light image, wherein the first position information includes the width, height and coordinates of the first calibration frame image in the thermal imaging image, and the second position information includes the width, height and coordinates of the second calibration frame image in the visible light image, including: performing image processing on the thermal imaging image and the visible light image respectively; The edges of the thermal imaging image and the visible light image after the image processing are extracted by using an edge detection algorithm, so as to obtain the first position information and the second position information.

4. The method according to claim 3, characterized in that The image processing includes at least one of image denoising and contrast enhancement.

5. A dual light camera, characterized in that: include: A thermal imaging camera, used to obtain a thermal imaging image including an image of a first calibration frame; a visible light camera, configured to acquire a visible light image including a second calibration frame image, wherein the first calibration frame image and the second calibration frame image are images of the same calibration frame taken by the thermal imaging camera and the visible light camera respectively; the calibration frame is a rectangular thermal frame, and each side of the calibration frame is parallel to a corresponding edge of the thermal imaging image and a corresponding edge of the visible light image; and A processor, the processor being communicatively connected to the thermal imaging camera and the visible light camera, wherein the processor is configured to: Acquire first position information of the first calibration frame image in the thermal imaging image and second position information of the second calibration frame in the visible light image; generating calibration parameters according to the ratio between the first calibration frame image and the thermal imaging image being equal to the ratio between the second calibration frame image and the thermal imaging image in the visible light image, the first position information and the second position information, wherein the calibration parameters include the width, height and coordinates of the thermal imaging image synthesized in the visible light image; and According to the calibration parameters, adjusting the width, height and coordinates of the thermal imaging image synthesized in the visible light image; The processor is further configured to: The calibration parameters are generated using the following formula: Wherein, w is the width of the thermal imaging image synthesized in the visible light image, w r is the width of the thermal imaging image, w0 is the width of the first calibration frame image in the thermal imaging image, w1 is the width of the second calibration frame image in the visible light image, h is the height of the thermal imaging image synthesized in the visible light image, and h r is the height of the thermal imaging image, h1 is the height of the second calibration frame image in the visible light image, h0 is the height of the first calibration frame image in the thermal imaging image, x is the horizontal coordinate of the upper left corner vertex of the thermal imaging image synthesized in the visible light image, x1 is the horizontal coordinate of the upper left corner vertex of the second calibration frame image in the visible light image, x0 is the horizontal coordinate of the upper left corner vertex of the first calibration frame image in the thermal imaging image, y is the vertical coordinate of the upper left corner vertex of the thermal imaging image synthesized in the visible light image, y1 is the vertical coordinate of the upper left corner vertex of the second calibration frame image in the visible light image, and y0 is the vertical coordinate of the upper left corner vertex of the first calibration frame image in the thermal imaging image.

6. The dual light camera according to claim 5, characterized in that: The processor is further configured to: performing image processing on the thermal imaging image and the visible light image respectively; Using an edge detection algorithm, edges of the thermal imaging image and the visible light image after the image processing are extracted to obtain the first position information and the second position information, wherein the first position information includes the width, height and coordinates of the first calibration frame image in the thermal imaging image, and the second position information includes the width, height and coordinates of the second calibration frame in the visible light image.

7. The dual light camera according to claim 6, characterized in that: The image processing includes at least one of image denoising and contrast enhancement.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method according to any one of claims 1 to 4.

9. A dual-light camera imaging calibration device, characterized in that: include: An image acquisition module, used to acquire a thermal imaging image including a first calibration frame image and a visible light image including a second calibration frame image, wherein the first calibration frame image and the second calibration frame image are images of the same calibration frame acquired by the image acquisition module respectively; the calibration frame is a rectangular thermal frame, and each side of the calibration frame is parallel to a corresponding edge of the thermal imaging image and a corresponding edge of the visible light image; a position acquisition module, configured to acquire first position information of the first calibration frame image in the thermal imaging image and second position information of the second calibration frame in the visible light image, wherein the first position information includes a width, a height, and a coordinate of the first calibration frame image in the thermal imaging image, and the second position information includes a width, a height, and a coordinate of the second calibration frame in the visible light image; a calibration module, configured to generate calibration parameters according to the first position information and the second position information, wherein the calibration parameters include a width, a height, and a coordinate of a thermal imaging image synthesized in the visible light image; and A position adjustment module, used for adjusting the position of the thermal imaging image synthesized in the visible light image in the visible light image according to the calibration parameters; The calibration module is also used for: The calibration parameters are generated using the following formula: Wherein, w is the width of the thermal imaging image synthesized in the visible light image, w r is the width of the thermal imaging image, w0 is the width of the first calibration frame image in the thermal imaging image, w1 is the width of the second calibration frame image in the visible light image, h is the height of the thermal imaging image synthesized in the visible light image, and h r is the height of the thermal imaging image, h1 is the height of the second calibration frame image in the visible light image, h0 is the height of the first calibration frame image in the thermal imaging image, x is the horizontal coordinate of the upper left corner vertex of the thermal imaging image synthesized in the visible light image, x1 is the horizontal coordinate of the upper left corner vertex of the second calibration frame image in the visible light image, x0 is the horizontal coordinate of the upper left corner vertex of the first calibration frame image in the thermal imaging image, y is the vertical coordinate of the upper left corner vertex of the thermal imaging image synthesized in the visible light image, y1 is the vertical coordinate of the upper left corner vertex of the second calibration frame image in the visible light image, and y0 is the vertical coordinate of the upper left corner vertex of the first calibration frame image in the thermal imaging image.

10. The device according to claim 9, characterized in that The location acquisition module is also used for: performing image processing on the thermal imaging image and the visible light image respectively; The edges of the thermal imaging image and the visible light image after the image processing are extracted by using an edge detection algorithm, so as to obtain the first position information and the second position information.

11. The device according to claim 10, characterized in that The image processing includes at least one of image denoising and contrast enhancement.

Images