Camera double light fusion calibration method and device, computer equipment and medium

Abstract

The application discloses a kind of camera dual light fusion's calibration method, device, computer equipment and medium, method includes: obtaining visible light sensor and thermal infrared sensor respectively to whiteboard grid pattern shooting obtained visible light image and thermal infrared image;Visible light image and thermal infrared image are handled to obtain the grid intersection point coordinate set of visible light image and thermal infrared image;According to the grid intersection point coordinate set of visible light image and thermal infrared image, the affine change matrix parameter between visible light image and thermal infrared image is calculated.The present application can guarantee that higher precision calibration point can be obtained, and also can guarantee that calibration point can have higher density, in addition, by calculating affine matrix by region, avoid the lens distortion of monitoring camera, improve calibration precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cameras, in particular to a camera dual-light fusion calibration method and device, computer equipment and a medium. BACKGROUND

[0002] Thermal imaging surveillance cameras generally include two imaging sensors, one for capturing visible light and one for capturing object infrared radiation. Through the cooperative work of the two sensors, security and building, forest, and other scene fire monitoring can be achieved. To facilitate use, the visible light picture and the thermal imaging picture support fusion together. In the fusion process, the calibration of the two sensors is involved. The current calibration accuracy is not high, which can cause the final image to not be well fused together. SUMMARY

[0003] The present application aims to overcome the shortcomings of the prior art and provide a camera dual-light fusion calibration method and device, computer equipment and a medium to improve calibration accuracy.

[0004] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0005] In a first aspect, the present application provides a camera dual-light fusion calibration method, the camera is configured with a visible light sensor and a thermal infrared sensor, and the method comprises:

[0006] Obtaining a visible light image and a thermal infrared image obtained by respectively shooting a whiteboard grid pattern by the visible light sensor and the thermal infrared sensor;

[0007] Processing the visible light image and the thermal infrared image to obtain a grid intersection point coordinate set of the visible light image and the thermal infrared image;

[0008] Calculating an affine transformation matrix parameter between the visible light image and the thermal infrared image according to the grid intersection point coordinate set of the visible light image and the thermal infrared image.

[0009] Based on the first aspect, further technical solutions are provided as follows: the processing of the visible light image and the thermal infrared image to obtain the grid intersection point coordinate set of the visible light image and the thermal infrared image comprises:

[0010] Performing gray scale processing on the visible light image and the thermal infrared image to obtain a gray scale image;

[0011] Performing binaryzation processing on the gray scale image to obtain a binaryzation image, and the white points in the binaryzation image are the grid line positions;

[0012] Performing discrete abnormal point processing on the binaryzation image to remove the discrete abnormal points in the binaryzation image;

[0013] Based on the preset algorithm parameters, all white point coordinates in the binary image are obtained by using a random sample consensus algorithm to obtain the straight line equation of each grid line.

[0014] The straight line equation of each grid line is verified for correctness.

[0015] According to the positional relationship of each grid line, the horizontal and vertical lines are sorted.

[0016] According to the sorted lines, the coordinates of each grid intersection point are calculated for the first time.

[0017] According to the position coordinates of the white pixel points and the straight line equation, the starting point position of each line is calculated.

[0018] According to the line starting position and the straight line equation, the center point of the line is extracted to obtain the center point position of the line.

[0019] Using a straight line fitting algorithm, all center point positions of each line are fitted to obtain a high-precision straight line coordinate equation.

[0020] According to the high-precision straight line coordinate equation, the coordinates of each grid intersection point are calculated again.

[0021] Based on the first aspect, further technical solutions are provided: the affine transformation matrix parameters between the visible light image and the thermal infrared image are calculated based on the grid intersection point coordinate set of the visible light image and the thermal infrared image, including:

[0022] The visible light image and the thermal infrared image are divided into multiple regions respectively.

[0023] The grid intersection point coordinates of each region are calculated using an affine transformation equation to obtain the affine transformation matrix parameters of the corresponding region.

[0024] Based on the first aspect, further technical solutions are provided: the affine transformation matrix parameters of each region are calculated using an affine transformation equation, and the affine transformation equation is:

[0025]

[0026] Where X', Y' represents the grid intersection point coordinates of a certain region of the visible light image, X, Y represents the grid intersection point coordinates of a certain region of the thermal infrared image, a, b, c, d, T x , T y are constants of the affine transformation equation.

[0027] Based on the first aspect, further technical solutions are provided: the whiteboard grid pattern is generated by a laser grid projector on a white reflective board.

[0028] In a second aspect, the present application provides a camera dual-optical fusion calibration device, the camera is configured with a visible light sensor and a thermal infrared sensor, comprising an acquisition unit, a processing unit and a calculation unit;

[0029] The acquisition unit is configured to acquire a visible light image and a thermal infrared image obtained by the visible light sensor and the thermal infrared sensor respectively when shooting a whiteboard grid pattern;

[0030] The processing unit is configured to process the visible light image and the thermal infrared image to obtain a grid intersection point coordinate set of the visible light image and the thermal infrared image;

[0031] The calculation unit is configured to calculate an affine transformation matrix parameter between the visible light image and the thermal infrared image according to the grid intersection point coordinate set of the visible light image and the thermal infrared image.

[0032] Based on the second aspect, further technical solutions are provided as follows: the processing unit comprises a gray processing module, a binary processing module, a discrete abnormal point processing module, a straight line equation acquisition module, a correctness verification module, a straight line sorting module, a first intersection point coordinate calculation module, a straight line starting point position calculation module, a center point extraction module, a fitting module and a second intersection point coordinate calculation module;

[0033] The gray processing module is configured to perform gray processing on the visible light image and the thermal infrared image to obtain a gray image;

[0034] The binary processing module is configured to perform binary processing on the gray image to obtain a binary image, and white points in the binary image are grid line positions;

[0035] The discrete abnormal point processing module is configured to perform discrete abnormal point processing on the binary image to remove discrete abnormal points in the binary image;

[0036] The straight line equation acquisition module is configured to use a random sample consensus algorithm to acquire straight line equations of all grid lines based on preset algorithm parameters and all white point coordinates in the binary image;

[0037] The correctness verification module is configured to verify the correctness of the straight line equations of the grid lines;

[0038] The straight line sorting module is configured to sort horizontal and vertical lines according to the position relationship of the grid lines;

[0039] The first intersection point coordinate calculation module is configured to first calculate the positions of the grid intersection points according to the sorted lines;

[0040] The straight line starting point position calculation module is configured to calculate the starting point position of each line according to the position coordinates of the white pixel points and the straight line equation.

[0041] The center point extraction module is configured to extract the center point of the line according to the starting position of the line and the straight line equation, and obtain the center point position of the line.

[0042] The fitting module is configured to use a straight line fitting algorithm to fit all the center point positions of each line, and obtain a high-precision straight line coordinate equation.

[0043] The intersection point coordinate recalculation module is configured to recalculate the grid intersection point coordinate positions according to the high-precision straight line coordinate equation.

[0044] According to the second aspect, further technical solutions are provided as follows: the calculation unit comprises a division module and an affine transformation matrix parameter calculation module.

[0045] The division module is configured to divide the visible light image and the thermal infrared image into multiple regions, respectively.

[0046] The affine transformation matrix parameter calculation module is configured to calculate the affine transformation matrix parameters of each region by using an affine transformation equation according to the grid intersection point coordinates of each region.

[0047] In a third aspect, the present application provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the camera dual-optical fusion calibration method as described above when executing the computer program.

[0048] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program comprises program instructions, and the program instructions are executed by a processor to make the processor execute the camera dual-optical fusion calibration method as described above.

[0049] Compared with the prior art, the present application has the beneficial effects that: the present application comprises obtaining a visible light image and a thermal infrared image obtained by respectively shooting a whiteboard grid pattern by a visible light sensor and a thermal infrared sensor; processing the visible light image and the thermal infrared image to obtain a grid intersection point coordinate set of the visible light image and the thermal infrared image; and calculating an affine transformation matrix parameter between the visible light image and the thermal infrared image according to the grid intersection point coordinate set of the visible light image and the thermal infrared image. By introducing a laser grid projector to project a whiteboard grid pattern, a random sampling consensus method is used to roughly extract a straight line position, the robustness of the calculation result is enhanced, a center line is extracted to realize high-precision straight line position calculation, and then a high-precision straight line equation is used to calculate intersection points to obtain high-precision calibration points, which not only ensures that high-precision calibration points can be obtained, but also ensures that the calibration points can have high density. In addition, by calculating the affine matrix in different regions, the lens distortion of the monitoring camera is avoided, the influence of the lens distortion on a large range of data is avoided, and the calibration precision is improved.

[0050] The above description is only a summary of the technical scheme of the present application, in order to more clearly understand the technical means of the present application, the content of the specification can be implemented, and in order to make the above and other purpose characteristics and advantages of the present application more obvious and easy to understand, the following preferred embodiments are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical scheme of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creating any creative labor.

[0052] Figure 1 The flow of the camera dual light fusion calibration method provided for the specific embodiments of the present application Figure 1 ;

[0053] Figure 2 The flow of the camera dual light fusion calibration method provided for the specific embodiments of the present application Figure 2 ;

[0054] Figure 3 The flow of the camera dual light fusion calibration method provided for the specific embodiments of the present application Figure 3 ;

[0055] Figure 4 The schematic block diagram of the camera dual light fusion calibration device provided for the specific embodiments of the present application Figure 1 ;

[0056] Figure 2The schematic block diagram of the camera dual-optical fusion calibration device provided by the embodiment of the present application Figure 6 ;

[0057] Figure 3 The schematic block diagram of the camera dual-optical fusion calibration device provided by the embodiment of the present application Figure 7 ;

[0058] Figure 1 The schematic block diagram of the computer device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0060] It should be understood that, when used in the specification and the appended claims, the terms "comprise" and "include" indicate the presence of described features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0061] It should also be understood that the terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and the appended claims of the present application, the singular forms "a", "an" and "the" are intended to include the plural forms, unless the context clearly indicates otherwise.

[0062] It should be further understood that the term "and / or" used in the specification and the appended claims of the present application means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0063] The embodiment of the present application provides a camera dual-optical fusion calibration method, the camera is configured with a visible light sensor and a thermal infrared sensor, the visible light sensor is used for capturing visible light, the thermal infrared sensor is used for capturing object radiation infrared, and the two sensors work cooperatively.

[0064] As shown in Figure 2 , the camera dual-optical fusion calibration method comprises the following steps: S10-S30.

[0065] S10, obtaining the visible light image and the thermal infrared image obtained by respectively shooting the whiteboard grid pattern by the visible light sensor and the thermal infrared sensor.

[0066] Before filming, the hardware needs to be prepared and set up. Specifically, the required hardware includes a binocular thermal imaging camera, a computer, a laser grid projector, and a reflector. The computer can be a common PC or laptop. The hardware setup process is as follows: connect the binocular thermal imaging camera to the computer, align the laser grid projector with the reflector so that the laser projector projects a regular, dense grid onto the white reflector (using the principle of diffuse reflection), and adjust the binocular thermal imaging camera to a suitable position so that the projected grid can basically fill the camera's image field of view.

[0067] During filming, the computer acquires visible light and thermal infrared images from the binocular thermal imaging camera and the thermal infrared sensors, respectively.

[0068] S20. Process the visible light image and the thermal infrared image to obtain the coordinate set of the grid intersection points of the visible light image and the thermal infrared image.

[0069] Computer equipment uses different parameters to process visible light images and thermal infrared images respectively. The purpose of the processing is to find which position in the visible light image corresponds to which position in the thermal infrared image.

[0070] It's important to note that the different processing parameters are primarily due to the differences in brightness and color between the visible light image and the thermal infrared image. Therefore, to extract the laser line location, the image processing parameters differ. For example, points with a brightness higher than 200 in the visible light image are treated as laser lines, while points with a brightness higher than 150 in the thermal imaging image are treated as laser lines. However, aside from using different parameters, the specific processing procedure for both visible light and thermal infrared images is the same.

[0071] like Figure 3 As shown, in one embodiment, step S20 specifically includes the following steps: S201-S211.

[0072] S201. Perform grayscale processing on the visible light image and the thermal infrared image to obtain a grayscale image.

[0073] S202. Perform binarization on the grayscale image to obtain a binarized image. The white dots in the binarized image are the grid line positions.

[0074] In this embodiment, the Otsu method is used to binarize visible light images and thermal infrared images to obtain a binarized image. The Otsu method of binarization is prior art and will not be described in detail here.

[0075] S203. Perform discrete outlier processing on the binarized image to remove discrete outliers from the binarized image.

[0076] In the embodiment, morphological operation is performed on the binary image, first erosion and then dilation (opening operation), so as to remove the discrete abnormal points of the binary image.

[0077] S204, based on the preset algorithm parameters, the random sample consensus algorithm is used to obtain the straight line equation of each grid line for all white point coordinates in the binary image.

[0078] In the embodiment, the preset algorithm parameters include the slope of the grid line and the number of pixels of each grid line. For all white point coordinates, the random sample consensus algorithm (RANSAC) is used to obtain the straight line equation of each grid line. The random sample consensus algorithm enhances the robustness of the calculation result.

[0079] S205, the correctness of the straight line equation of each grid line is verified.

[0080] In the embodiment, the correctness of the straight line equation extracted in step S203 is verified based on the preset grid number M*N and the grid line spacing.

[0081] The grid spacing M*N means that the laser grid projector projects a rectangular grid on the reflector, so there are M horizontal lines and N vertical lines.

[0082] The specific verification method is that there must be M horizontal lines which are basically parallel and have the same spacing, and there must be N vertical lines which are basically parallel and have the same spacing, and the horizontal and vertical lines need to be close to vertical. If the above requirements are met, the verification is passed, otherwise the verification is failed.

[0083] S206, according to the positional relationship of each grid line, the horizontal and vertical lines are sorted.

[0084] Since the grid lines in the horizontal and vertical directions are arranged at equal distances, the horizontal lines are sorted according to the Y coordinate from small to large, and the vertical lines are sorted according to the X coordinate.

[0085] S207, according to the sorted lines, the coordinates of the grid intersection points are calculated for the first time.

[0086] According to the sorting information of the straight lines, the starting point and the spacing of the grid are determined. It is assumed that the starting point is the origin (0, 0), the grid spacing in the horizontal direction is dx, and the grid spacing in the vertical direction is dy. The coordinates of the starting point are taken as the coordinates of the first grid intersection point. From the second intersection point, the x coordinate of the new intersection point is the x coordinate of the previous intersection point plus dx, and the y coordinate is the y coordinate of the previous intersection point plus the slope of the straight line multiplied by dx. In this way, the coordinates of all the required grid intersection points are calculated.

[0087] S208. Calculate the starting point position of each line based on the position coordinates of the white pixels and the equation of the line.

[0088] In this embodiment, given the equation of the line Y = ax + b, by projecting the white points near the equation onto the line, we can find the two points on the left and right that are furthest from the center, which are the endpoints (starting points) of the line segment.

[0089] S209. Based on the starting position of the line and the equation of the line, extract the center point of the line to obtain the position of the center point of the line.

[0090] In this embodiment, based on the starting position of the line and the equation of the line, the Steger algorithm is used to extract the center point of the laser line image within the line range (since the approximate position of the intersection point is known, the extraction algorithm skips the pixels within the range of the intersection point) to obtain the precise center point position C of the laser line direction.

[0091] S210. Using a straight line fitting algorithm, fit the positions of all center points of each line to obtain a high-precision straight line coordinate equation.

[0092] In this embodiment, a straight line fitting algorithm is used to fit the position C of all centers of each straight line to obtain a high-precision straight line coordinate equation.

[0093] S211. Based on the high-precision linear coordinate equation, recalculate the coordinate positions of each grid intersection point.

[0094] This step of the calculation yields the set of grid intersection coordinates for the visible light image and the thermal infrared image, which is used to calculate the parameters of the affine transformation matrix.

[0095] S30. Calculate the affine transformation matrix parameters between the visible light image and the thermal infrared image based on the coordinate set of the grid intersection points of the visible light image and the thermal infrared image.

[0096] like Figure 4 As shown, in one embodiment, step S30 specifically includes the following steps S301-S302.

[0097] S301. Divide the visible light image and the thermal infrared image into multiple regions respectively.

[0098] S302. Calculate the affine transformation matrix parameters of the corresponding region by using the affine transformation equation to obtain the coordinates of the grid intersection points of each region.

[0099] For steps S301-S302, in the embodiment, the visible light image and the thermal infrared image are divided into J*K regions respectively (J<=M, K<=N), the cross point coordinates of each region are brought into an affine transformation equation, and J*K affine transformation matrices are calculated.

[0100]

[0101] Wherein, X', Y' represent the grid cross point coordinates of a region of the visible light image, X, Y represent the grid cross point coordinates of a region of the thermal infrared image, a, b, c, d, T x , T y are constants of the affine transformation equation.

[0102] By dividing the image into multiple small range areas, since the distortion of the small range is basically the same, then the small range is used for mapping, and the large range image is spliced, so that the whole mapping is more accurate, the lens distortion of the monitoring camera is avoided, the influence on the large range data is avoided, and the calibration precision is improved.

[0103] When the affine transformation matrix parameters are calculated, the computer device saves the affine matrix parameters to the camera, and after the camera starts the image fusion function, the affine matrix is used to map the visible light image to the thermal imaging channel, and image fusion display is realized.

[0104] The present application introduces a laser grid projector to project a whiteboard grid pattern, simultaneously uses a random sampling consistency method to roughly extract the straight line position, enhances the robustness of the calculation result, then realizes high-precision straight line position calculation through center line extraction, and then calculates the intersection point through the high-precision straight line equation to obtain high-precision calibration points, which can not only obtain high-precision calibration points, but also can guarantee that the calibration points have high density. In addition, by calculating the affine matrix in the region, the lens distortion of the monitoring camera is avoided, the influence on the large range data is avoided, and the calibration precision is improved.

[0105] Corresponding to the above-mentioned camera dual light fusion calibration method, the embodiment of the present application also provides a camera dual light fusion calibration device.

[0106] As Figure 5As shown, the camera dual-optical fusion calibration device 100 is configured with a visible light sensor and a thermal infrared sensor, and the device includes an acquisition unit 110, a processing unit 120, and a calculation unit 130. The acquisition unit 110 is configured to acquire a visible light image and a thermal infrared image obtained by the visible light sensor and the thermal infrared sensor respectively by shooting a whiteboard grid pattern. The processing unit 120 is configured to process the visible light image and the thermal infrared image to obtain a grid intersection point coordinate set of the visible light image and the thermal infrared image. The calculation unit 130 is configured to calculate an affine transformation matrix parameter between the visible light image and the thermal infrared image according to the grid intersection point coordinate set of the visible light image and the thermal infrared image.

[0107] As shown in the embodiment, Figure 6 the processing unit 120 includes a grayscale processing module 121, a binarization processing module 122, a discrete abnormal point processing module 123, a straight line equation acquisition module 124, a correctness verification module 125, a straight line sorting module 126, a first intersection point coordinate calculation module 127, a straight line starting point position calculation module 128, a center point extraction module 129, a fitting module 1291, and a second intersection point coordinate calculation module 1291. The grayscale processing module 121 performs grayscale processing on the visible light image and the thermal infrared image to obtain a grayscale image. The binarization processing module 122 performs binarization processing on the grayscale image to obtain a binarized image, and the white points in the binarized image are the grid line positions. The discrete abnormal point processing module 123 is configured to perform discrete abnormal point processing on the binarized image to remove discrete abnormal points in the binarized image. The straight line equation acquisition module 124 is configured to use a random sample consensus algorithm to acquire a straight line equation of each grid line based on preset algorithm parameters and all white point coordinates in the binarized image. The correctness verification module 125 is configured to verify the correctness of the straight line equation of each grid line. The straight line sorting module 126 is configured to sort the horizontal and vertical lines according to the positional relationship of the grid lines. The first intersection point coordinate calculation module 127 is configured to first calculate the position of each grid intersection point coordinate based on the sorted lines. The straight line starting point position calculation module 128 is configured to calculate the starting point position of each line based on the position coordinates of the white pixel points and the straight line equation. The center point extraction module 129 is configured to extract the center point of each line based on the starting position of the line and the straight line equation to obtain the center point position of the line. The fitting module 1291 is configured to use a straight line fitting algorithm to fit all the center point positions of each line to obtain a high-precision straight line coordinate equation. The second intersection point coordinate calculation module 1291 is configured to recalculate the position of each grid intersection point coordinate based on the high-precision straight line coordinate equation.

[0108] As shown in the embodiment, Figure 7As shown, in an embodiment, the computing unit 130 comprises a division module 131 and an affine transformation matrix parameter calculation module 132. The division module 131 is configured to divide the visible light image and the thermal infrared image into a plurality of regions respectively. The affine transformation matrix parameter calculation module 132 is configured to calculate the affine transformation matrix parameters of each region by using the affine transformation equation based on the grid intersection point coordinates of each region.

[0109] The camera dual-optical fusion calibration method described above can be implemented in the form of a computer program, which can run on a computer device as shown. Figure 7

[0110] Please refer to Figure 7 , Figure 7 is a schematic block diagram of a computer device provided by an embodiment of the present application. The computer device 700 can be a server, wherein the server can be a stand-alone server or a server cluster composed of multiple servers.

[0111] As shown Figure 7 , the computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the camera dual-optical fusion calibration method described above when executing the computer program.

[0112] The computer device 700 can be a terminal or a server. The computer device 700 comprises a processor 720, a memory, and a network interface 750 connected through a system bus 710, wherein the memory can comprise a non-volatile storage medium 730 and an internal memory 740.

[0113] The non-volatile storage medium 730 can store an operating system 731 and a computer program 732. The computer program 732, when executed, can cause the processor 720 to execute any camera dual-optical fusion calibration method.

[0114] The processor 720 is configured to provide computing and control capabilities to support the operation of the entire computer device 700.

[0115] The internal memory 740 provides an environment for the execution of the computer program 732 in the non-volatile storage medium 730, and the computer program 732, when executed by the processor 720, can cause the processor 720 to execute any camera dual-optical fusion calibration method.

[0116] The network interface 750 is configured to perform network communication, such as sending assigned tasks, etc. Those skilled in the art can understand Figure 7 ​The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device 700 to which the scheme of the present application is applied. Specifically, the computer device 700 can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. The processor 720 is configured to run program code stored in the memory to implement the following steps:

[0117] A camera dual-optical fusion calibration method, the camera is configured with a visible light sensor and a thermal infrared sensor, the method comprises:

[0118] Obtaining a visible light image and a thermal infrared image respectively photographed by the visible light sensor and the thermal infrared sensor on a whiteboard grid pattern;

[0119] Processing the visible light image and the thermal infrared image to obtain a grid intersection point coordinate set of the visible light image and the thermal infrared image;

[0120] Calculating an affine transformation matrix parameter between the visible light image and the thermal infrared image according to the grid intersection point coordinate set of the visible light image and the thermal infrared image.

[0121] In an embodiment: the processing of the visible light image and the thermal infrared image to obtain the grid intersection point coordinate set of the visible light image and the thermal infrared image comprises:

[0122] Performing gray scale processing on the visible light image and the thermal infrared image to obtain a gray scale image;

[0123] Performing binaryzation processing on the gray scale image to obtain a binaryzation image, and the white points in the binaryzation image are the grid line positions;

[0124] Performing discrete abnormal point processing on the binaryzation image to remove discrete abnormal points in the binaryzation image;

[0125] Based on preset algorithm parameters, using a random sample consensus algorithm on all white point coordinates in the binaryzation image to obtain a straight line equation of each grid line;

[0126] Verifying the correctness of the straight line equation of each grid line;

[0127] According to the positional relationship of each grid line, sorting the horizontal and vertical lines;

[0128] According to the sorted lines, first calculating the coordinate positions of each grid intersection point;

[0129] According to the coordinate positions of the white pixel points and the straight line equation, calculating the starting point positions of each line;

[0130] According to the line starting position and the straight line equation, the center point of the line is extracted to obtain the center point position of the line;

[0131] Using a straight line fitting algorithm, all center point positions of each line are fitted to obtain a high-precision straight line coordinate equation;

[0132] According to the high-precision straight line coordinate equation, the coordinates of each grid intersection point are calculated again.

[0133] In an embodiment: the affine transformation matrix parameters between the visible light image and the thermal infrared image are calculated according to the grid intersection point coordinate set of the visible light image and the thermal infrared image, comprising:

[0134] The visible light image and the thermal infrared image are respectively divided into multiple regions;

[0135] The grid intersection point coordinates of each region are calculated using the affine transformation equation to obtain the affine transformation matrix parameters of the corresponding region.

[0136] In an embodiment: the affine transformation matrix parameters of each region are calculated using the affine transformation equation of the grid intersection point coordinates of each region, and the affine transformation equation is:

[0137]

[0138] Wherein, X', Y' represents the grid intersection point coordinates of a region of the visible light image, X, Y represents the grid intersection point coordinates of a region of the thermal infrared image, a, b, c, d, T x , T y are constants of the affine transformation equation.

[0139] In an embodiment: the whiteboard grid pattern is generated by projecting a laser grid projector on a white light-reflecting board.

[0140] It should be understood that in the embodiments of the present application, the processor 720 can be a central processing unit (CPU), and the processor 720 can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0141] Those skilled in the art can understand that ​ The computer device 700 structure shown in the middle does not constitute a limitation on the computer device 700, and can include more or fewer components than shown, or combine certain components, or arrange different components.

[0142] In another embodiment of the present application, a computer readable storage medium is provided. The computer readable storage medium can be a non-volatile computer readable storage medium. The computer readable storage medium stores a computer program, wherein the computer program is executed by a processor to implement the camera double light fusion calibration method disclosed in the embodiment of the present application.

[0143] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-mentioned devices, apparatuses and units can refer to the corresponding process in the foregoing method embodiments, which will not be described here. Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in a general manner in the foregoing description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0144] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the apparatus embodiments described above are only schematic. For example, the division of the units is only a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can also be electrical, mechanical or other forms of connection.

[0145] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0146] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0147] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0148] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for calibrating a dual optical fusion camera, the camera being equipped with a visible light sensor and a thermal infrared sensor, characterized in that, The method comprises: acquiring a visible light image and a thermal infrared image obtained by respectively photographing a whiteboard grid pattern by a visible light sensor and a thermal infrared sensor; processing the visible light image and the thermal infrared image to obtain a grid intersection point coordinate set of the visible light image and the thermal infrared image; calculating an affine transformation matrix parameter between the visible light image and the thermal infrared image according to the grid intersection point coordinate set of the visible light image and the thermal infrared image; the processing of the visible light image and the thermal infrared image to obtain the grid intersection point coordinate set of the visible light image and the thermal infrared image comprises: performing grayscale processing on the visible light image and the thermal infrared image to obtain a grayscale image; performing binaryzation processing on the grayscale image to obtain a binary image, and white points in the binary image are positions of grid lines; performing discrete abnormal point processing on the binary image to remove discrete abnormal points in the binary image; based on preset algorithm parameters, obtaining a straight line equation of each grid line by using a random sample consensus algorithm on all white point coordinates in the binary image; performing correctness verification on the straight line equation of each grid line; sorting horizontal and vertical lines according to a positional relationship of the grid lines; first calculating a position of each grid intersection point according to the sorted lines; calculating a starting point position of each line according to a position coordinate of a white pixel and the straight line equation; extracting a center point of each line according to the starting point position and the straight line equation to obtain a center point position of the line; using a straight line fitting algorithm to fit all center point positions of each line to obtain a high-precision straight line coordinate equation; second calculating a position of each grid intersection point according to the high-precision straight line coordinate equation.

2. The method of claim 1, wherein, the calculating of the affine transformation matrix parameter between the visible light image and the thermal infrared image according to the grid intersection point coordinate set of the visible light image and the thermal infrared image comprises: dividing the visible light image and the thermal infrared image into multiple regions respectively; calculating an affine transformation matrix parameter of a corresponding region by using an affine transformation equation on grid intersection point coordinates of each region.

3. The method of claim 2, wherein, the affine transformation equation is: wherein X', Y' represents the grid intersection point coordinates of a certain area of the visible light image, X, Y represents the grid intersection point coordinates of a certain area of the thermal infrared image, a, b, c, d, T x , T y are constants of the affine transformation equation.

4. The method of claim 1, wherein, the whiteboard grid pattern is generated by projecting a laser grid projector on a white light-reflecting board.

5. A device for calibrating binocular fusion of a camera, which operates to perform the method for calibrating binocular fusion of a camera according to any one of claims 1 to 4, the camera being equipped with a visible light sensor and a thermal infrared sensor, characterized in that, comprises an acquisition unit, a processing unit and a calculation unit; the acquisition unit is configured to acquire a visible light image and a thermal infrared image obtained by respectively photographing a whiteboard grid pattern by a visible light sensor and a thermal infrared sensor; the processing unit is configured to process the visible light image and the thermal infrared image to obtain a grid intersection point coordinate set of the visible light image and the thermal infrared image; the calculation unit is configured to calculate an affine transformation matrix parameter between the visible light image and the thermal infrared image according to the grid intersection point coordinate set of the visible light image and the thermal infrared image. The processing unit comprises a grayscale processing module, a binarization processing module, a discrete abnormal point processing module, a straight line equation obtaining module, a correctness verification module, a straight line sorting module, a first intersection point coordinate calculating module, a straight line starting point position calculating module, a center point extracting module, a fitting module, and a second intersection point coordinate calculating module; The grayscale processing module performs grayscale processing on the visible light image and the thermal infrared image to obtain a grayscale image; The binarization processing module performs binarization processing on the grayscale image to obtain a binarization image, and white points in the binarization image are positions of grid lines; The discrete abnormal point processing module is configured to perform discrete abnormal point processing on the binarization image to remove discrete abnormal points in the binarization image; The straight line equation obtaining module is configured to obtain straight line equations of the grid lines by using a random sample consensus algorithm based on preset algorithm parameters and all white point coordinates in the binarization image; The correctness verification module is configured to verify correctness of the straight line equations of the grid lines; The straight line sorting module is configured to sort the horizontal and vertical lines according to position relationships of the grid lines; The first intersection point coordinate calculating module is configured to first calculate positions of intersection points of the grids according to the sorted lines; The straight line starting point position calculating module is configured to calculate starting point positions of each line according to position coordinates of the white pixels and the straight line equations; The center point extracting module is configured to extract center points of the lines according to the starting positions of the lines and the straight line equations to obtain positions of the center points of the lines; The fitting module is configured to fit all the center point positions of each line by using a straight line fitting algorithm to obtain a high-precision straight line coordinate equation; The second intersection point coordinate calculating module is configured to second calculate positions of the intersection points of the grids according to the high-precision straight line coordinate equation.

6. The apparatus for camera dual light fusion calibration of claim 5, wherein, The calculation unit comprises a division module and an affine transformation matrix parameter calculating module; The division module is configured to divide the visible light image and the thermal infrared image into multiple regions respectively; The affine transformation matrix parameter calculating module is configured to calculate affine transformation matrix parameters of a corresponding region by using an affine transformation equation and grid intersection point coordinates of each region.

7. A computer device, comprising: The storage medium stores a computer program, and the computer program includes program instructions. When the program instructions are executed by a processor, the processor performs the camera dual-optical fusion calibration method according to any one of claims 1-4.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program includes program instructions. When the program instructions are executed by a processor, the processor performs the camera dual-optical fusion calibration method according to any one of claims 1-4.

Images