Camera calibration method, device, storage medium and terminal device

Abstract

This application proposes a camera calibration method, apparatus, storage medium, and terminal device. The method performs gradient matching and recognition on a thermal imaging image based on a template image of a heating element to obtain a first coordinate point set. The first coordinate point set is a set of coordinate points of a first target object in the thermal imaging image coordinate system. The first target object is an object whose matching degree with the template image exceeds a preset matching threshold. The coordinate points in the first coordinate point set are sorted so that the sorting of the coordinate points of the first target object in the first coordinate point set matches the sorting of the coordinate points of the heating element in the first calibration coordinate point set. Based on the sorted first coordinate point set and the first calibration coordinate point set, the first rotation matrix and first translation matrix of the thermal imaging camera relative to the calibration coordinate system are determined. Single-target calibration of the thermal imaging camera is completed while automatically completing target object matching and saving manpower and time.

Description

Technical Field

[0001] The present application relates to the field of image acquisition, and more specifically, to a camera calibration method, apparatus, storage medium, and terminal device. Background Art

[0002] In applications such as multispectral photogrammetry, remote sensing, and target surveillance, multispectral combined measurement methods are often used to comprehensively obtain the spectral radiation information of an object. Visible light cameras can obtain rich texture information, while infrared thermal imagers can obtain temperature information. Therefore, combined measurement using visible light and infrared cameras is currently widely used. However, when infrared and visible light cameras do not share the same lens or optical path, their intrinsic camera parameters and their relative position and posture must be rigorously calibrated. The geometric model parameters that determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image are called camera parameters. The process of determining these parameters using calibration equipment and methods is called camera calibration.

[0003] Currently, visible light dual-camera geometric calibration technology is relatively mature and widely used in scenarios such as smartphone dual-camera depth of field photography and zoom photography. However, calibrating the geometric relationship between visible light cameras and infrared cameras has been a difficult problem for those skilled in the art. Summary of the Invention

[0004] The purpose of this application is to provide a camera calibration method, apparatus, storage medium and terminal device to at least partially improve the above-mentioned problems.

[0005] In order to achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows:

[0006] In a first aspect, embodiments of the present application provide a camera calibration method, applied to a terminal device, wherein the terminal device includes a thermal imaging camera, the thermal imaging camera being configured to capture an image of a heated calibration device to obtain a thermal image, the calibration device including a first preset number of heating elements, the method comprising:

[0007] Performing gradient matching recognition on the thermal imaging image based on the template image of the heating element to obtain a first set of coordinate points, wherein the first set of coordinate points is a set of coordinate points of a first target object in the thermal imaging image coordinate system, and the first target object is an object whose matching degree with the template image exceeds a preset matching threshold;

[0008] Sorting the coordinate points in the first coordinate point set so that the order of the coordinate points of the first target object in the first coordinate point set matches the order of the coordinate points of the heating element in the first calibration coordinate point set, wherein the first calibration coordinate point set is a set of coordinate points of the heating element in a calibration coordinate system;

[0009] Based on the sorted first coordinate point set and the first calibration coordinate point set, a first rotation matrix and a first translation matrix of the thermal imaging camera relative to the calibration coordinate system are determined.

[0010] In a second aspect, an embodiment of the present application provides a camera calibration method, which is applied to a terminal device, wherein the terminal device includes a thermal imaging camera and a visible light camera, wherein the relative position of the visible light camera and the thermal imaging camera is fixed, and the visible light camera is used to capture an image of the calibration device to obtain a visible light image, and the thermal imaging camera is used to capture an image of the heated calibration device to obtain a thermal imaging image, wherein the calibration device includes a first preset number of heating elements and a calibration plate, wherein a first preset number of first holes are provided on the calibration plate, wherein all the first holes have the same shape and size, and each of the first holes has a heating element provided therein, and the method includes:

[0011] Acquire a second set of coordinate points based on the visible light image, wherein the second set of coordinate points is a set of coordinate points of a second target object in the visible light image coordinate system, and the second target object is an object in the visible light image corresponding to the first hole;

[0012] sorting the coordinate points in the second coordinate point set so that the order of the coordinate points of the second target object in the second coordinate point set matches the order of the coordinate points of the first hole in the second calibration coordinate point set, wherein the second calibration coordinate point set is a set of coordinate points of the first hole in the calibration coordinate system;

[0013] Determining a second rotation matrix and a second translation matrix of the visible light camera relative to the calibration coordinate system based on the sorted second coordinate point set and the second calibration coordinate point set;

[0014] Based on the second rotation matrix, the second translation matrix, and the first rotation matrix and the first translation matrix obtained in any one of the first aspects, dual-object positioning of the thermal imaging camera and the visible light camera is completed.

[0015] In a third aspect, an embodiment of the present application provides a camera calibration device, applied to a terminal device, wherein the terminal device includes a thermal imaging camera, the thermal imaging camera is configured to capture an image of a heated calibration device to obtain a thermal imaging image, the calibration device including a first preset number of heating elements, and the device comprising:

[0016] a processing unit, configured to perform gradient matching recognition on the thermal imaging image based on the template image of the heating element to obtain a first set of coordinate points, wherein the first set of coordinate points is a set of coordinate points of a first target object in a thermal imaging image coordinate system, and the first target object is an object whose degree of matching with the template image exceeds a preset matching threshold;

[0017] The processing unit is further configured to sort the coordinate points in the first coordinate point set so that the sorting of the coordinate points of the first target object in the first coordinate point set matches the sorting of the coordinate points of the heating element in the first calibration coordinate point set, wherein the first calibration coordinate point set is a set of coordinate points of the heating element in the calibration coordinate system;

[0018] A calibration unit is configured to determine a first rotation matrix and a first translation matrix of the thermal imaging camera relative to the calibration coordinate system based on the sorted first coordinate point set and the first calibration coordinate point set.

[0019] In a fourth aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, and the computer program implements the above method when executed by a processor.

[0020] In a fifth aspect, an embodiment of the present application provides a terminal device, comprising: a processor and a memory, wherein the memory is used to store one or more programs; when the one or more programs are executed by the processor, the above method is implemented.

[0021] Compared to the prior art, the camera calibration method, apparatus, storage medium, and terminal device provided in the embodiments of the present application perform gradient matching and recognition on a thermal imaging image based on a template image of a heating element to obtain a first coordinate point set, wherein the first coordinate point set is a set of coordinate points of a first target object in a thermal imaging image coordinate system, and the first target object is an object whose matching degree with the template image exceeds a preset matching threshold; the coordinate points in the first coordinate point set are sorted so that the sorting of the coordinate points of the first target object in the first coordinate point set matches the sorting of the coordinate points of the heating element in the first calibration coordinate point set, wherein the first calibration coordinate point set is a set of coordinate points of the heating element in a calibration coordinate system; based on the sorted first coordinate point set and the first calibration coordinate point set, the first rotation matrix and the first translation matrix of the thermal imaging camera relative to the calibration coordinate system are determined. Single-target calibration of the thermal imaging camera is completed while automatically completing target object matching and saving manpower and time.

[0022] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 A schematic diagram of the structure of the calibration device provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram comparing a visible light image and a thermal imaging image provided in an embodiment of the present application;

[0027] Figure 4 A flowchart of a camera calibration method provided in an embodiment of the present application;

[0028] Figure 5 Reference schematic diagram of template image and thermal imaging image provided in the embodiment of the present application;

[0029] Figure 6 Schematic diagram of the sub-steps of S102 provided in the embodiment of the present application;

[0030] Figure 7A schematic diagram showing the distribution of four vertices in the first coordinate point set and four vertices in the pre-created normalized coordinate point set provided in an embodiment of the present application;

[0031] Figure 8 Schematic diagram of sub-steps of S102-1 provided in the embodiment of the present application;

[0032] Figure 9 Schematic diagram of sub-steps of S102-3 provided in the embodiment of the present application;

[0033] Figure 10 One of the flowcharts of the camera calibration method provided in an embodiment of the present application;

[0034] Figure 11 Schematic diagram of the units of the camera calibration device provided in an embodiment of the present application.

[0035] In the figure: 10 - processor; 11 - memory; 12 - bus; 13 - communication interface; 201 - processing unit; 202 - calibration unit. DETAILED DESCRIPTION

[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0037] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the present application without creative work are within the scope of protection of the present application.

[0038] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0039] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0040] In the description of this application, it should be noted that the terms "upper", "lower", "inside", "outside", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, or are the orientations or positional relationships in which the product of the application is usually placed when in use. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they should not be understood as limitations on this application.

[0041] It should also be noted that, in the description of this application, unless otherwise expressly specified or limited, the terms "disposed" and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; mechanical connections, or electrical connections; direct connections, indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in this application based on the specific circumstances.

[0042] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.

[0043] The present application embodiment provides a calibration device, such as Figure 1 As shown, the calibration device includes a base plate and a calibration plate, wherein the calibration plate is provided with M first holes, all of which are of the same shape and size, and each of which contains a heating element. Optionally, the calibration device further includes a heat insulating layer and a heating wire, wherein the heating wire is arranged according to the arrangement rule of the M first holes, so that each first hole has an intersection of two heating wires, and the intersection is a heating element.

[0044] The thermal insulation layer is disposed between the heating wire and the calibration plate. The thermal insulation layer has a second hole at a position opposite each first hole. Each first hole and the second hole opposite the first hole form a channel. The heating element is disposed at the bottom of each channel, so that a heating element is located within each first hole. Optionally, the diameter of the second hole is smaller than the diameter of the first hole.

[0045] Alternatively, the insulation layer can be white paper to remove color and facilitate clear image capture. The ends of the heating wire can be secured with screws. The calibration plate can be a film plate. The M first holes can be arranged in a rectangular or square pattern. The base plate can be an acrylic plate. Grooves are etched in the acrylic plate to accommodate the heating wire mesh and ensure the flatness of the checkerboard calibration plate.

[0046] The visible light image captured by the visible light camera is a grid, and the infrared image captured by the infrared thermal imager is an array of N white bright spots on a black calibration plate.

[0047] The embodiment of the present application provides a terminal device, which can be an image acquisition device, a computer device or a server device. Figure 2 , a schematic diagram of the structure of a terminal device. The terminal device includes a processor 10, a memory 11, and a bus 12. The processor 10 and the memory 11 are connected via the bus 12. The processor 10 is used to execute executable modules stored in the memory 11, such as computer programs.

[0048] The processor 10 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the camera calibration method can be completed by the hardware integrated logic circuit in the processor 10 or by instructions in the form of software. The above-mentioned processor 10 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components.

[0049] The memory 11 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.

[0050] The bus 12 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. Figure 2 Only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus 12 or one type of bus 12.

[0051] The memory 11 is used to store programs, such as a program corresponding to the camera calibration device. The camera calibration device includes at least one software functional module that can be stored in the memory 11 in the form of software or firmware, or embedded in the operating system (OS) of the terminal device. After receiving an execution instruction, the processor 10 executes the program to implement the camera calibration method.

[0052] Possibly, the terminal device provided in the embodiment of the present application further includes a communication interface 13. The communication interface 13 is connected to the processor 10 via a bus.

[0053] Optionally, when the terminal device is a computer device or a server device other than the image acquisition device, the terminal device can interact with the image acquisition device to be calibrated through the communication interface 13 to obtain the visible light image and thermal imaging image captured by the camera.

[0054] In an embodiment of the present application, the image capture device includes a thermal imaging camera (e.g., an infrared camera) configured to capture images of the heated calibration device to obtain a thermal image. The calibration device includes a first preset number of heating elements. Optionally, a first preset number of first holes are provided on the calibration plate, all of which have a consistent shape and size, and each of which contains a heating element.

[0055] Optionally, use a 12V / 1A power supply to power the heating wire, stabilize it for a period of time (e.g. 10 minutes), and use a visible light thermal imaging binocular camera to simultaneously capture a set of image sequences S1 and S2. Figure 3 , Figure 3 A schematic diagram comparing a visible light image and a thermal imaging image provided in an embodiment of the present application, wherein the left side is a thermal imaging image and the right side is a visible light image.

[0056] It should be understood that Figure 2 The structure shown is only a schematic diagram of the structure of a portion of the terminal device. The terminal device may also include Figure 2 More or fewer components than shown, or with Figure 2 Different configurations shown. Figure 2 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0057] The camera calibration method provided in the embodiment of the present application can be applied to, but not limited to, Figure 2 For the terminal equipment shown, please refer to Figure 4 The camera calibration method includes S101, S102 and S103, which are described in detail as follows.

[0058] S101 , performing gradient matching recognition on a thermal imaging image based on a template image of a heating body to obtain a first coordinate point set.

[0059] The first coordinate point set is a set of coordinate points of the first target object in the thermal imaging image coordinate system, and the first target object is an object whose matching degree with the template image exceeds a preset matching threshold.

[0060] Optionally, the template image may be an artificially captured image, and the coordinate point of the first target object refers to a coordinate value in the thermal imaging image coordinate system.

[0061] Optionally, refer to Figure 5 , Figure 5 Reference schematic diagrams of the template image and thermal imaging image provided in the embodiments of the present application. The left side is a reference schematic diagram of the template image, and the right side is a reference schematic diagram of the thermal imaging image.

[0062] It should be understood that gradient matching recognition can be performed based on a gradient domain template matching method, thereby accurately capturing the first target object in the thermal image and further determining the set of coordinate points of the first target object in the thermal image coordinate system. Compared to manually marking the first target object in the thermal image and manually adding coordinate information, the camera calibration method provided in this embodiment achieves rapid and efficient capture of the first target object in the thermal image, saving manpower and time.

[0063] S102 , sorting the coordinate points in the first coordinate point set so that the sorting of the coordinate points of the first target object in the first coordinate point set matches the sorting of the coordinate points of the heating element in the first calibration coordinate point set.

[0064] The first calibration coordinate point set is a set of coordinate points of the heating element in the calibration coordinate system.

[0065] Optionally, K i represents the coordinate point of the i-th heating element in the calibration coordinate system, Q i represents the coordinate point of the first target object corresponding to the i-th heating element in the thermal imaging image coordinate system. After sorting the coordinate points in the first coordinate point set, Ki The position (serial number) of the calibration coordinate point set and Q i The positions (serial numbers) in the first coordinate point set are the same. Optionally, a calibration coordinate system is pre-constructed based on the distribution of the heating elements on the calibration plate, and the coordinate points of the heating elements in the calibration coordinate system are combined into a calibration coordinate point set.

[0066] Optionally, the calibration coordinate system may be a three-dimensional coordinate system, such as a world coordinate system.

[0067] S103 : Determine a first rotation matrix and a first translation matrix of the thermal imaging camera relative to the calibration coordinate system based on the sorted first coordinate point set and the first calibration coordinate point set.

[0068] Optionally, a sorted set of first coordinate points corresponding to a second preset number of thermal imaging images can be obtained, where any two thermal imaging images are taken at different angles. Based on the second preset number of sorted first coordinate point sets and the first calibration coordinate point set, a first rotation matrix and a first translation matrix of the thermal imaging camera relative to the calibration coordinate system are determined, thereby completing single-target calibration of the thermal imaging camera.

[0069] In summary, an embodiment of the present application provides a camera calibration method, which performs gradient matching recognition on a thermal imaging image based on a template image of a heating element to obtain a first coordinate point set, wherein the first coordinate point set is a set of coordinate points of a first target object in the thermal imaging image coordinate system, and the first target object is an object whose matching degree with the template image exceeds a preset matching threshold; the coordinate points in the first coordinate point set are sorted so that the sorting of the coordinate points of the first target object in the first coordinate point set matches the sorting of the coordinate points of the heating element in the first calibration coordinate point set, wherein the first calibration coordinate point set is a set of coordinate points of the heating element in the calibration coordinate system; based on the sorted first coordinate point set and the first calibration coordinate point set, the first rotation matrix and the first translation matrix of the thermal imaging camera relative to the calibration coordinate system are determined. Single-target calibration of the thermal imaging camera is completed under the premise of automatically completing target object matching and saving manpower and time.

[0070] exist Figure 4 Based on the content in S102, the present application embodiment also provides a possible implementation method, please refer to Figure 6 , S102 includes S102-1, S102-2, S102-3 and S102-4, which are described in detail as follows.

[0071] S102-1, determine four vertices in the first coordinate point set.

[0072] It should be understood that the four vertices in the first coordinate point set are the coordinate points of the first target object corresponding to the heating elements at the four corners of the calibration plate in the thermal imaging image coordinate system.

[0073] Optionally, regarding how to determine the four vertices in the first coordinate point set, please refer to sub-step S102-1 below.

[0074] S102-2: Construct a homogeneous mapping matrix based on the coordinates of the four vertices in the first coordinate point set and the four vertices in the pre-created normalized coordinate point set.

[0075] The normalized coordinate point set is a set of coordinate points of the first calibration coordinate point set in a corresponding two-dimensional normalized coordinate system; the homogeneous mapping matrix represents a mapping relationship between the normalized coordinate point set and the first coordinate point set.

[0076] Optionally, a normalized coordinate point set is pre-constructed so that the order of the coordinate points in the normalized coordinate point set matches the order of the coordinate points of the heating element in the first calibration coordinate point set.

[0077] Optionally, K i represents the coordinate point of the i-th heating element in the calibration coordinate system, P i Represents the coordinate point of the i-th heating element in the two-dimensional normalized coordinate system. The created normalized coordinate point set has the characteristics of sorting and matching with the first calibration coordinate point set, K i The position (serial number) of the first calibration coordinate point set is the same as P i The positions (serial numbers) in the normalized coordinate point set are the same.

[0078] It should be understood that the coordinates of the four vertices in the normalized coordinate point set are the same as the coordinates of the four vertices in the first calibration coordinate point set.

[0079] Optionally, refer to Figure 7 , Figure 7 A schematic diagram of the distribution of four vertices in the first coordinate point set and four vertices in the pre-created normalized coordinate point set provided in an embodiment of the present application, wherein the left side represents the four vertices in the first coordinate point set (U1, U2, U3 and U4), and the right side represents the four vertices in the normalized coordinate point set (W1, W2, W3 and W4).

[0080] S102-3: Acquire matching information between the normalized coordinate point set and the first coordinate point set based on the mapping relationship.

[0081] The matching information includes a matching relationship between each coordinate point in the normalized coordinate point set and each coordinate point in the first coordinate point set.

[0082] S102-4, sorting the coordinate points in the first coordinate point set according to the matching information and the sorting of the coordinate points of the heating element in the normalized coordinate point set.

[0083] It should be understood that the order of the coordinate points in the normalized coordinate point set matches the order of the coordinate points of the heating element in the first calibration coordinate point set, so the order of the coordinate points in the first coordinate point set matches the order of the coordinate points of the heating element in the first calibration coordinate point set.

[0084] exist Figure 6 On the basis of the content in S102-1, how to accurately obtain the four vertices in the first coordinate point set, the embodiment of the present application also provides a possible implementation method, please refer to Figure 8 , S102-1 includes S102-1A, S102-1B and S102-1C, which are described in detail as follows.

[0085] S102-1A, obtaining the centroid coordinates of the first coordinate point set.

[0086] S102-1B, dividing the first coordinate point set into four quadrants with the barycentric coordinates as the center of the circle.

[0087] S102-1C: Determine the coordinate point in each quadrant that is farthest from the center of gravity coordinate as the vertex of the first coordinate point set.

[0088] Optionally, the distances between all points in each quadrant and the barycentric coordinates are obtained respectively. It should be understood that the coordinate point with the farthest distance from the barycentric coordinates in a quadrant is the vertex in the quadrant.

[0089] exist Figure 6 On the basis of the content in S102-3, how to accurately obtain the matching information of the normalized coordinate point set and the first coordinate point set, the embodiment of the present application also provides a possible implementation method, please refer to Figure 9 , S102-3 includes S102-3A and S102-3B, which are described in detail as follows.

[0090] S102-3A, transforming the calibration coordinate point based on the mapping relationship to obtain a transformed coordinate point.

[0091] The calibration coordinate point is any coordinate point in the normalized coordinate point set, and the conversion coordinate point is a coordinate point after the calibration coordinate point is converted to the thermal imaging image coordinate system.

[0092] S102-3B: Determine the coordinate point in the first coordinate point set that has the smallest distance from the converted coordinate point as the coordinate point that matches the calibration coordinate point.

[0093] It should be understood that due to differences in thermal imaging and target recognition, some of the coordinate points in the first coordinate point set may be offset, resulting in the possibility that the converted coordinate points do not overlap. In this case, an accurate matching relationship is obtained by determining the coordinate point in the first coordinate point set with the smallest distance from the converted coordinate point as the coordinate point that matches the calibration coordinate point. All matching relationships are summarized as matching information, which includes the matching relationship between each coordinate point in the normalized coordinate point set and each coordinate point in the first coordinate point set.

[0094] In one possible implementation, the image acquisition device further includes a visible light camera. The relative positions of the visible light camera and the thermal imaging camera are fixed. The visible light camera is used to acquire images of the calibration device to obtain visible light images. The calibration device further includes a calibration plate. The calibration plate is provided with a first preset number of first holes. The shapes and sizes of all the first holes are consistent. A heating element is provided in each first hole. Figure 4 On the basis of how to complete the calibration of the visible light camera, the embodiment of the present application also provides a possible implementation method, please refer to Figure 10 The camera calibration method also includes S104, S105 and S106, which are described in detail as follows.

[0095] S104: Acquire a second coordinate point set based on the visible light image.

[0096] The second coordinate point set is a set of coordinate points of a second target object in the visible light image coordinate system, and the second target object is an object corresponding to the first hole in the visible light image.

[0097] S105 , sorting the coordinate points in the second coordinate point set so that the sorting of the coordinate points of the second target object in the second coordinate point set matches the sorting of the coordinate points of the first hole in the second calibration coordinate point set.

[0098] The second calibration coordinate point set is a set of coordinate points of the first hole in the calibration coordinate system.

[0099] S106 : Determine a second rotation matrix and a second translation matrix of the visible light camera relative to the calibration coordinate system based on the sorted second coordinate point set and the second calibration coordinate point set.

[0100] Please continue to refer to Figure 10 Regarding how to complete the dual-target positioning of the thermal imaging camera and the visible light camera, the embodiment of the present application also provides a possible implementation method, such as Figure 10 As shown, the camera calibration method further includes S107, which is described in detail as follows.

[0101] S107 , completing dual-target positioning of the thermal imaging camera and the visible light camera according to the first rotation matrix, the first translation matrix, the second rotation matrix, and the second translation matrix.

[0102] Optionally, a target rotation matrix and a target translation matrix between the thermal imaging camera and the visible light camera are determined based on the first rotation matrix, the first translation matrix, the second rotation matrix, and the second translation matrix, thereby completing dual-target positioning of the thermal imaging camera and the visible light camera.

[0103] The embodiment of the present application also provides a camera calibration method, which is applied to a terminal device. The terminal device includes a thermal imaging camera and a visible light camera. The relative posture of the visible light camera and the thermal imaging camera is fixed. The visible light camera is used to capture images of the calibration device to obtain visible light images. The thermal imaging camera is used to capture images of the calibration device that has been heated to obtain thermal imaging images. The calibration device includes a first preset number of heating elements and a calibration plate. The calibration plate is provided with a first preset number of first holes. The shapes and sizes of all the first holes are consistent. A heating element is provided in each first hole. Figure 10 As shown, the camera calibration method includes:

[0104] S104, acquiring a second set of coordinate points based on the visible light image, wherein the second set of coordinate points is a set of coordinate points of a second target object in the visible light image coordinate system, and the second target object is an object corresponding to the first hole in the visible light image;

[0105] S105, sorting the coordinate points in the second coordinate point set so that the order of the coordinate points of the second target object in the second coordinate point set matches the order of the coordinate points of the first hole in the second calibration coordinate point set, wherein the second calibration coordinate point set is a set of coordinate points of the first hole in the calibration coordinate system;

[0106] S106 , determining a second rotation matrix and a second translation matrix of the visible light camera relative to the calibration coordinate system based on the sorted second coordinate point set and the second calibration coordinate point set;

[0107] S107 , completing dual-target positioning of the thermal imaging camera and the visible light camera according to the first rotation matrix, the first translation matrix, the second rotation matrix, and the second translation matrix.

[0108] The first rotation matrix and the first translation matrix may be obtained through the above embodiments.

[0109] See also Figure 11 , Figure 11 A camera calibration device is provided in an embodiment of the present application. Optionally, the camera calibration device is applied to the terminal device described above.

[0110] The camera calibration device includes: a processing unit 201 and a calibration unit 202 .

[0111] Processing unit 201 is configured to perform gradient matching recognition on the thermal imaging image based on the template image of the heating element to obtain a first set of coordinate points, wherein the first set of coordinate points is a set of coordinate points of a first target object in the thermal imaging image coordinate system, and the first target object is an object whose matching degree with the template image exceeds a preset matching threshold;

[0112] The processing unit 201 is further configured to sort the coordinate points in the first coordinate point set so that the sorting of the coordinate points of the first target object in the first coordinate point set matches the sorting of the coordinate points of the heating element in the first calibration coordinate point set, wherein the first calibration coordinate point set is a set of coordinate points of the heating element in the calibration coordinate system;

[0113] The calibration unit 202 is configured to determine a first rotation matrix and a first translation matrix of the thermal imaging camera relative to the calibration coordinate system based on the sorted first coordinate point set and the first calibration coordinate point set.

[0114] Optionally, the processing unit 201 may execute the above-mentioned S101 , S102 , S104 and S105 , and the calibration unit 202 may execute the above-mentioned S103 , S106 and S107 .

[0115] It should be noted that the camera calibration device provided in this embodiment can execute the method flow shown in the above method flow embodiment to achieve the corresponding technical effects. For the sake of brevity, any part not mentioned in this embodiment can be referred to the corresponding content in the above embodiment.

[0116] The present application also provides a storage medium storing computer instructions and programs that, when read and executed, execute the camera calibration method of the above embodiment. The storage medium may include memory, flash memory, registers, or a combination thereof.

[0117] The following provides a terminal device, which can be an image acquisition device, a computer device or a server device. Figure 2 As shown, the above-mentioned camera calibration method can be implemented. Specifically, the terminal device includes: a processor 10, a memory 11, and a bus 12. The processor 10 may be a CPU. The memory 11 is used to store one or more programs. When the one or more programs are executed by the processor 10, the camera calibration method of the above-mentioned embodiment is performed.

[0118] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0119] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0120] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0121] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

[0122] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above and that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the present application is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A camera calibration method, characterized in that: Applied to a terminal device, the terminal device includes a thermal imaging camera, the thermal imaging camera is used to capture an image of a heated calibration device to obtain a thermal imaging image, the calibration device includes a first preset number of heating elements, and the method includes: Performing gradient matching recognition on the thermal imaging image based on the template image of the heating element to obtain a first set of coordinate points, wherein the first set of coordinate points is a set of coordinate points of a first target object in the thermal imaging image coordinate system, and the first target object is an object whose matching degree with the template image exceeds a preset matching threshold; Sorting the coordinate points in the first coordinate point set so that the order of the coordinate points of the first target object in the first coordinate point set matches the order of the coordinate points of the heating element in the first calibration coordinate point set, wherein the first calibration coordinate point set is a set of coordinate points of the heating element in a calibration coordinate system; Determining a first rotation matrix and a first translation matrix of the thermal imaging camera relative to the calibration coordinate system based on the sorted first coordinate point set and the first calibration coordinate point set; The step of sorting the coordinate points in the first coordinate point set includes: Determine four vertices in the first coordinate point set; Constructing a homogeneous mapping matrix based on the coordinates of four vertices in the first coordinate point set and four vertices in a pre-created normalized coordinate point set, wherein the normalized coordinate point set is a set of coordinate points of the first calibration coordinate point set in a corresponding two-dimensional normalized coordinate system; the homogeneous mapping matrix represents a mapping relationship between the normalized coordinate point set and the first coordinate point set; Acquire matching information between the normalized coordinate point set and the first coordinate point set based on the mapping relationship, wherein the matching information includes a matching relationship between each coordinate point in the normalized coordinate point set and each coordinate point in the first coordinate point set; The coordinate points in the first coordinate point set are sorted according to the matching information and the sorting of the coordinate points of the heating element in the normalized coordinate point set.

2. The camera calibration method according to claim 1, wherein: The step of determining four vertices in the first coordinate point set includes: Obtaining the centroid coordinates of the first coordinate point set; Dividing the first coordinate point set into four quadrants with the center of gravity coordinate as the center of the circle; The coordinate point in each quadrant that is farthest from the center of gravity coordinate is determined as the vertex of the first coordinate point set.

3. The camera calibration method according to claim 1, wherein: The step of obtaining matching information between the normalized coordinate point set and the first coordinate point set based on the mapping relationship includes: Converting the calibration coordinate point based on the mapping relationship to obtain a converted coordinate point, wherein the calibration coordinate point is any coordinate point in the normalized coordinate point set, and the converted coordinate point is a coordinate point after the calibration coordinate point is converted to the thermal imaging image coordinate system; The coordinate point in the first coordinate point set that has the shortest distance from the converted coordinate point is determined as the coordinate point that matches the calibration coordinate point.

4. The camera calibration method according to claim 1, wherein: The terminal device further includes a visible light camera, wherein the relative position of the visible light camera and the thermal imaging camera is fixed, and the visible light camera is used to capture images of a calibration device to obtain a visible light image. The calibration device further includes a calibration plate, wherein a first preset number of first holes are provided on the calibration plate, wherein all the first holes have the same shape and size, and a heating element is provided in each of the first holes. The method further includes: Acquire a second set of coordinate points based on the visible light image, wherein the second set of coordinate points is a set of coordinate points of a second target object in the visible light image coordinate system, and the second target object is an object in the visible light image corresponding to the first hole; sorting the coordinate points in the second coordinate point set so that the order of the coordinate points of the second target object in the second coordinate point set matches the order of the coordinate points of the first hole in the second calibration coordinate point set, wherein the second calibration coordinate point set is a set of coordinate points of the first hole in the calibration coordinate system; A second rotation matrix and a second translation matrix of the visible light camera relative to the calibration coordinate system are determined based on the sorted second coordinate point set and the second calibration coordinate point set.

5. The camera calibration method according to claim 4, wherein: After obtaining the first rotation matrix, the first translation matrix, the second rotation matrix, and the second translation matrix, the method further includes: Dual-object positioning of the thermal imaging camera and the visible light camera is completed according to the first rotation matrix, the first translation matrix, the second rotation matrix, and the second translation matrix.

6. A camera calibration method, characterized in that: Applied to a terminal device, the terminal device includes a thermal imaging camera and a visible light camera, the relative position of the visible light camera and the thermal imaging camera is fixed, the visible light camera is used to capture images of a calibration device to obtain a visible light image, and the thermal imaging camera is used to capture images of a heated calibration device to obtain a thermal imaging image, the calibration device includes a first preset number of heating elements and a calibration plate, the calibration plate is provided with a first preset number of first holes, all of the first holes are consistent in shape and size, and each of the first holes is provided with a heating element, the method comprising: Acquire a second set of coordinate points based on the visible light image, wherein the second set of coordinate points is a set of coordinate points of a second target object in the visible light image coordinate system, and the second target object is an object in the visible light image corresponding to the first hole; sorting the coordinate points in the second coordinate point set so that the order of the coordinate points of the second target object in the second coordinate point set matches the order of the coordinate points of the first hole in the second calibration coordinate point set, wherein the second calibration coordinate point set is a set of coordinate points of the first hole in the calibration coordinate system; Determining a second rotation matrix and a second translation matrix of the visible light camera relative to the calibration coordinate system based on the sorted second coordinate point set and the second calibration coordinate point set; Based on the second rotation matrix, the second translation matrix, and the first rotation matrix and the first translation matrix obtained according to any one of claims 1 to 3, dual-object positioning of the thermal imaging camera and the visible light camera is completed.

7. A camera calibration device, characterized in that: Applied to a terminal device, the terminal device includes a thermal imaging camera, the thermal imaging camera is used to capture an image of a heated calibration device to obtain a thermal imaging image, the calibration device includes a first preset number of heating elements, and the device includes: a processing unit, configured to perform gradient matching recognition on the thermal imaging image based on the template image of the heating element to obtain a first set of coordinate points, wherein the first set of coordinate points is a set of coordinate points of a first target object in a thermal imaging image coordinate system, and the first target object is an object whose degree of matching with the template image exceeds a preset matching threshold; The processing unit is further configured to sort the coordinate points in the first coordinate point set so that the sorting of the coordinate points of the first target object in the first coordinate point set matches the sorting of the coordinate points of the heating element in the first calibration coordinate point set, wherein the first calibration coordinate point set is a set of coordinate points of the heating element in the calibration coordinate system; a calibration unit, configured to determine a first rotation matrix and a first translation matrix of the thermal imaging camera relative to the calibration coordinate system based on the sorted first coordinate point set and the first calibration coordinate point set; The sorting of the coordinate points in the first coordinate point set includes: determining four vertices in the first coordinate point set; constructing a homogeneous mapping matrix based on the coordinates of the four vertices in the first coordinate point set and the four vertices in the pre-created normalized coordinate point set, wherein the normalized coordinate point set is a set of coordinate points of the first calibration coordinate point set in the corresponding two-dimensional normalized coordinate system; the homogeneous mapping matrix represents the mapping relationship between the normalized coordinate point set and the first coordinate point set; obtaining matching information between the normalized coordinate point set and the first coordinate point set based on the mapping relationship, wherein the matching information includes the matching relationship between each coordinate point in the normalized coordinate point set and each coordinate point in the first coordinate point set; and sorting the coordinate points in the first coordinate point set based on the matching information and the sorting of the coordinate points of the heating element in the normalized coordinate point set.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. A terminal device, characterized in that: include: a processor and a memory, the memory being configured to store one or more programs; When the one or more programs are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

Images