Spatial registration method and system for visible light image and infrared image

Abstract

The invention discloses a spatial registration method and system for a visible light image and an infrared image, and relates to the technical field of spatial registration. According to the invention, the position coordinate of the position coordinate obtained by the visible light camera in the coordinate system used by the second aerial view is registered with the position coordinate of the position coordinate obtained by the infrared camera in the coordinate system used by the second aerial view, so that a more accurate conversion relation can be obtained; and compared with the prior art, the conversion relation obtaining process is simpler, a large number of training images are not needed, the cost is low, and the practicability is high.

Description

Background Art

[0002] The dual-spectrum integrated machine includes a visible light camera and an infrared camera (specifically, a far-infrared camera), which complement each other during the day and night and can also provide temperature information of the target. It is an important component of vehicle-road collaboration and digital twin systems. Therefore, it is necessary to spatially align the images taken by the visible light camera and the infrared camera of the dual-spectrum integrated machine. Currently, the following existing technologies exist:

[0003] 1) Invention patent publication with publication number “CN103337077A” and subject name “A method for visible light and infrared image registration based on multi-scale segmentation and SIFT”: SIFT feature matching has achieved great success in the visible light field, but there are more complex parameters and engineering adjustment problems between feature extraction and matching of far-infrared images.

[0004] 2) The invention patent with the publication number "CN113628261A" and the subject name "A method for matching infrared and visible light images in power inspection scenarios" discloses: using the SuperPoint feature extraction network to detect the feature points of the two edge images respectively and calculate the descriptors; matching the feature points through the SuperGlue feature matching network according to the feature points of the two edge images, screening out the correct feature point matching pairs, and removing the unmatched feature points; calculating the affine transformation model parameters according to the matching feature point pairs, and performing spatial coordinate transformation on the image to be matched through bilinear interpolation to achieve image matching. The present invention realizes the accurate registration of infrared and visible light images of power equipment, and obtains the temperature information of power equipment in the background of visible light images, but requires more images for training.

[0005] 3) The invention patent with publication number “CN109448035A” and subject name “Infrared image and visible light image registration method based on deep learning” discloses: The affine or homography matrix of the two is directly given by the deep learning model, which requires a large number of samples to train the model, but in actual aftermarket applications, these data are often not guaranteed. Summary of the invention

[0006] The technical problem to be solved by the present invention is to address the deficiencies of the prior art and specifically provide a method and system for spatial registration of visible light images and infrared images, as follows:

[0007] 1) In the first aspect, the present invention provides a method for spatial registration of a visible light image and an infrared image, and the specific technical solution is as follows:

[0008] After the visible light camera and the infrared camera are time-aligned, the first visible light image and the first infrared image are respectively collected at the same time using the visible light camera and the infrared camera;

[0009] Performing target recognition on the first visible light image to obtain first position coordinates of each first preset target in the coordinate system used by the visible light camera; performing target recognition on the first infrared image to obtain second position coordinates of each first preset target in the coordinate system used by the infrared camera;

[0010] Based on the first transformation matrix, perform an inverse perspective projection transformation on the first visible light image to obtain a first bird's-eye view, and transform each first position coordinate using the first transformation matrix to obtain a third position coordinate of each first position coordinate in a coordinate system used by the first bird's-eye view;

[0011] Based on the second transformation matrix, the first infrared image is subjected to inverse perspective projection transformation to obtain a second bird's-eye view, and each second position coordinate is transformed by using the second transformation matrix to obtain fourth position coordinates of each second position coordinate in a coordinate system used by the second bird's-eye view;

[0012] Calculating a rigid transformation matrix between a coordinate system used for the first bird's-eye view and a coordinate system used for the second bird's-eye view;

[0013] Using the rigid transformation matrix, obtaining a fifth position coordinate of each third position coordinate in the coordinate system used in the second bird's-eye view;

[0014] Calculate the error distance between all fourth position coordinates and all fifth position coordinates; determine whether the error distance is less than a preset error distance threshold, and obtain a determination result;

[0015] When the judgment result is yes, determining the conversion relationship between the images respectively captured by the visible light camera and the infrared camera at the same time according to all the fourth position coordinates and all the fifth position coordinates;

[0016] The images taken by the visible light camera and the infrared camera at the same time are spatially registered using the transformation relationship.

[0017] The beneficial effects of the spatial registration method of a visible light image and an infrared image provided by the present invention are as follows:

[0018] The position coordinates of the first preset target obtained by the visible light camera are the coordinates in the coordinate system used by the visible light camera, and the first position coordinates of the first preset target obtained by the infrared camera are the coordinates in the coordinate system used by the infrared camera. In practical applications, it is necessary to convert the second position coordinates of the target obtained by the visible light camera and the infrared camera. Therefore, the accuracy of the conversion relationship between the images taken by the visible light camera and the infrared camera at the same time determines whether the conversion between the first position coordinates and the second position coordinates is accurate. The present invention aligns the position coordinates of the position coordinates obtained by the visible light camera in the coordinate system used by the second bird's-eye view with the position coordinates of the position coordinates obtained by the infrared camera in the coordinate system used by the second bird's-eye view, so as to obtain a more accurate conversion relationship. In addition, the process of obtaining the conversion relationship is simpler than the prior art, does not require a large number of training images, has low cost, and is highly practical.

[0019] Based on the above solution, the spatial registration method of a visible light image and an infrared image of the present invention can be further improved as follows.

[0020] Furthermore, the process of obtaining the first transformation matrix includes:

[0021] The time-aligned visible light camera is used to track the target, the trajectory of each second preset target is obtained, and straight line fitting is performed to fit the first straight line corresponding to each second preset target, and the first transformation matrix is ​​determined based on the two outermost first straight lines.

[0022] Furthermore, the process of obtaining the second transformation matrix includes:

[0023] The time-aligned infrared cameras are used to track the target, the trajectory of each third preset target is obtained, and straight line fitting is performed to fit the second straight line corresponding to each third preset target, and the second transformation matrix is ​​determined based on the two outermost second straight lines.

[0024] Further, calculating a rigid transformation matrix between a coordinate system used by the first bird's-eye view and a coordinate system used by the second bird's-eye view includes:

[0025] The ICP algorithm or the CPD algorithm is used to calculate and obtain a rigid transformation matrix between the coordinate system used by the first bird's-eye view and the coordinate system used by the second bird's-eye view.

[0026] 2) In a second aspect, the present invention also provides a spatial registration system for visible light images and infrared images, and the specific technical solution is as follows:

[0027] It includes an image acquisition module, a target recognition module, a bird's-eye view acquisition module, a position coordinate conversion module, a rigid transformation matrix calculation module, an error distance calculation and judgment module, a conversion relationship determination module and a space registration module;

[0028] The image acquisition module is used to: after time alignment of the visible light camera and the infrared camera, use the visible light camera and the infrared camera to respectively acquire a first visible light image and a first infrared image at the same time;

[0029] The target recognition module is used to: perform target recognition on the first visible light image to obtain the first position coordinates of each first preset target in the coordinate system used by the visible light camera; recognize the first infrared image to obtain the second position coordinates of each first preset target in the coordinate system used by the infrared camera;

[0030] The bird's-eye view acquisition module is used to: perform inverse perspective projection transformation on the first visible light image based on the first transformation matrix to obtain a first bird's-eye view;

[0031] The position coordinate conversion module is used to: transform each first position coordinate using a first transformation matrix to obtain a third position coordinate of each first position coordinate in a coordinate system used in the first bird's-eye view;

[0032] The bird's-eye view acquisition module is further used to: perform inverse perspective projection transformation on the first infrared image based on a second transformation matrix to obtain a second bird's-eye view;

[0033] The position coordinate conversion module is further used to: transform each second position coordinate using a second transformation matrix to obtain a fourth position coordinate of each second position coordinate in a coordinate system used in the second bird's-eye view;

[0034] The rigid transformation matrix calculation module is used to: calculate the rigid transformation matrix between the coordinate system used by the first bird's-eye view and the coordinate system used by the second bird's-eye view;

[0035] The position coordinate conversion module is further used to: obtain a fifth position coordinate of each third position coordinate in the coordinate system used in the second bird's-eye view by using a rigid transformation matrix;

[0036] The error distance calculation and judgment module is used to: calculate the error distance between all the fourth position coordinates and all the fifth position coordinates; judge whether the error distance is less than a preset error distance threshold, and obtain a judgment result;

[0037] The conversion relationship determination module is used to: when the judgment result is yes, determine the conversion relationship between the images respectively taken by the visible light camera and the infrared camera at the same time according to all the fourth position coordinates and all the fifth position coordinates;

[0038] The spatial registration module is used to register the images taken by the visible light camera and the infrared camera at the same time using the conversion relationship.

[0039] Based on the above solution, the spatial registration system of a visible light image and an infrared image of the present invention can also be improved as follows.

[0040] Furthermore, it also includes a first transformation matrix acquisition module, which is used to:

[0041] The time-aligned visible light camera is used to track the target, the trajectory of each second preset target is obtained, and straight line fitting is performed to fit the first straight line corresponding to each second preset target, and the first transformation matrix is ​​determined based on the two outermost first straight lines.

[0042] Furthermore, a second transformation matrix acquisition module is included, and the second transformation matrix acquisition module is used to:

[0043] The time-aligned infrared cameras are used to track the target, the trajectory of each third preset target is obtained, and straight line fitting is performed to fit the second straight line corresponding to each third preset target, and the second transformation matrix is ​​determined based on the two outermost second straight lines.

[0044] Furthermore, the rigid transformation matrix calculation module is specifically used to: use the ICP algorithm or the CPD algorithm to calculate the rigid transformation matrix between the coordinate system used by the first bird's-eye view and the coordinate system used by the second bird's-eye view.

[0045] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor, the processor being coupled to a memory, the memory storing at least one computer program, and the at least one computer program being loaded and executed by the processor so that the electronic device implements any of the above-mentioned methods for spatial registration of visible light images and infrared images.

[0046] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, it implements any of the above-mentioned methods for spatial registration of visible light images and infrared images.

[0047] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments of the present invention:

[0049] Figure 1 : A schematic flow chart of a method for spatial registration of a visible light image and an infrared image according to an embodiment of the present invention;

[0050] Figure 2 is: a geometric diagram used when determining the first transformation matrix;

[0051] Figure 3 The schematic diagram is as follows: a schematic diagram of a rectangle preset in the coordinate system used in the first bird's-eye view;

[0052] Figure 4 is: a geometric diagram used when determining the second transformation matrix;

[0053] Figure 5 The schematic diagram is as follows: a schematic diagram of a rectangle preset in the coordinate system used in the second bird's-eye view;

[0054] Figure 6 : A schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The principles and features of the present invention are described below. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0056] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0057] like Figure 1 As shown, a method for spatial registration of a visible light image and an infrared image according to an embodiment of the present invention comprises the following steps:

[0058] S1, after time alignment of the visible light camera and the infrared camera, using the visible light camera and the infrared camera to respectively collect a first visible light image and a first infrared image at the same time;

[0059] The visible light camera and the infrared camera are used to photograph the same preset area to obtain a first visible light image and a first infrared image collected at the same time. The preset area can be a designated road area, or the preset area can be designated according to actual conditions.

[0060] S2, performing target recognition on the first visible light image to obtain the first position coordinates of each first preset target in the coordinate system used by the visible light camera; and recognizing the first infrared image to obtain the second position coordinates of each first preset target in the coordinate system used by the infrared camera, specifically:

[0061] The first visible light image can be used for target recognition by using algorithms such as the YOLO algorithm or the ViD algorithm, and N1 targets can be detected. The first infrared image can be used for target recognition by using algorithms such as the YOLO algorithm or the ViD algorithm, and N2 targets can be detected. The same multiple targets can be selected from the N1 targets and the N2 targets as the first preset targets, and the number of the first preset targets is recorded as n. When N1=N2, all targets can be used as the first preset targets, and at this time N1=N2=n. It should be noted that the first visible light image can also be used for target recognition by other single-stage detection algorithms.

[0062] Since the detection frame of the first preset target is output by algorithms such as the YOLO algorithm or the ViD algorithm, the coordinates of the bottom edge center point of the detection frame of each first preset target in the coordinate system used by the visible light camera are used as the corresponding first position coordinates. Moreover, since the detection frame of the target is output by algorithms such as the YOLO algorithm or the ViD algorithm, the coordinates of the bottom edge center point of the detection frame of each target in the coordinate system used by the infrared camera are used as the corresponding second position coordinates.

[0063] Among them, the coordinate system used by the visible light camera is: the specified position on the visible light image taken by the visible light camera is taken as the origin, and the rectangular coordinate system is established. The specified position can be the center point of the visible light image or any position on the bottom edge of the visible light image. The straight line where the bottom edge of the visible light image is located can be the x-axis of the rectangular coordinate system. It should be noted that the origin of the coordinate system used by the visible light camera can be set according to actual conditions, and the x-axis and y-axis of the rectangular coordinate system can also be set according to actual conditions.

[0064] Among them, the coordinate system used by the infrared camera is: the specified position on the infrared image taken by the infrared camera is the origin, and the established rectangular coordinate system, the specified position can be the center point of the infrared image, or it can be any position on the bottom edge of the infrared image, and the straight line where the bottom edge of the infrared image is located can be the x-axis of the rectangular coordinate system. It should be noted that the origin of the coordinate system used by the infrared camera can be set according to actual conditions, and the x-axis and y-axis of the rectangular coordinate system can also be set according to actual conditions.

[0065] Among them, the same multiple targets can be selected from N1 targets and N2 targets as the first preset targets, and the number of first preset targets is recorded as n. When N1=N2, all targets can be used as the first preset targets, and at this time N1=N2=n.

[0066] S3. Based on the first transformation matrix, perform an inverse perspective projection transformation on the first visible light image to obtain a first bird's-eye view, and use the first transformation matrix to transform each first position coordinate to obtain a third position coordinate of each first position coordinate in a coordinate system used by the first bird's-eye view;

[0067] The process of obtaining the first transformation matrix includes:

[0068] The time-aligned visible light camera is used to track the target, the trajectory of each second preset target is obtained, and straight line fitting is performed to fit the first straight line corresponding to each second preset target, and the first transformation matrix is ​​determined based on the two outermost first straight lines.

[0069] Among them, the second visible light image taken by the visible light camera can be recognized by algorithms such as the YOLO algorithm or the ViD algorithm, and N3 targets are detected. According to actual conditions, a part of the N3 targets can be selected as the second preset targets, or all the N3 targets can be used as the second preset targets. Since the detection frame of the target will be output by algorithms such as the YOLO algorithm or the ViD algorithm, the coordinates of the center point of the bottom edge of the detection frame of each target are used as the corresponding sixth position coordinates. The sixth position coordinates are the coordinates in the coordinate system used by the visible light camera. It should be noted that other single-stage detection algorithms can also be used to identify the target in the first visible light image.

[0070] And use a visible light camera to track each second preset target to obtain the trajectory of each second preset target, perform straight line fitting on the trajectory of each second preset target to obtain multiple first straight lines, obtain the intersection of all the first straight lines, determine the first vanishing point, select the two outermost first straight lines, and record them as L1 and L2 respectively. When the second preset target is a vehicle, the first vanishing point is: the vanishing point of the road. When the target is different, the specific meaning of the first vanishing point will also be different.

[0071] like Figure 2 As shown, the position coordinates of the first intersection A1 and the second intersection B1 are calculated. The first intersection A1 is the intersection of the first straight line L1 and the x-axis in the coordinate system where the visible light camera is located. The second intersection B1 is the intersection of the second straight line L2 and the x-axis in the coordinate system where the visible light camera is located. The position coordinates of the first intersection A1 and the second intersection B1 are both the position coordinates in the coordinate system where the visible light camera is located.

[0072] Calculate the position coordinates of the first vanishing point VP1 in the coordinate system of the visible light camera;

[0073] The midpoint of the line segment between the first vanishing point VP1 and the first intersection A1 is recorded as the first midpoint A2, and the midpoint of the line segment between the first vanishing point VP1 and the second intersection B1 is recorded as the second midpoint B2, and the position coordinates of the first midpoint A2 and the position coordinates of the second midpoint B2 are obtained. At this time, the first intersection A1, the second intersection B1, the first midpoint A2 and the second midpoint B2 form a trapezoid.

[0074] like Figure 3As shown, in the coordinate system used in the first bird's-eye view, a rectangle is preset, and the four corner points of the rectangle are respectively recorded as: the first corner point a1, the second corner point b1, the third corner point a2 and the fourth corner point b2. The trapezoid composed of the first intersection point A1, the second intersection point B1, the first midpoint A2 and the second midpoint B2 is corresponded to the rectangle composed of the first corner point a1, the second corner point b1, the third corner point a2 and the fourth corner point b2. Specifically, the first intersection point A1 corresponds to the first corner point a1, the second intersection point B1 corresponds to the second corner point b1, the first midpoint A2 corresponds to the third corner point a2, and the second midpoint B2 corresponds to the fourth corner point b2. According to the correspondence between these four pairs of points, the change matrix between the coordinate system used for visible light and the coordinate system used for the first bird's-eye view can be obtained, that is, the first transformation matrix, which is recorded as H vis_bev,1 .

[0075] Among them, the coordinate system used for the first bird's-eye view is: the specified position on the first bird's-eye view is the origin, and the rectangular coordinate system is established. The specified position can be the center point of the first bird's-eye view, or any position on the bottom edge of the first bird's-eye view, and the straight line where the bottom edge of the first bird's-eye view is located can be the x-axis of the rectangular coordinate system. It should be noted that the origin of the first bird's-eye view can be set according to actual conditions, and the x-axis and y-axis of the rectangular coordinate system can also be set according to actual conditions.

[0076] The first transformation matrix is ​​used to transform each first position coordinate to obtain a third position coordinate of each first position coordinate in the coordinate system used in the first bird's-eye view, including:

[0077] The third position coordinates of each first position coordinate in the coordinate system used in the first bird's-eye view are obtained by using the first transformation formula. The first transformation formula is: 2i =H vis_bev,1 P 1i , where P 1i represents: the i-th first position coordinate in P1, the i-th first position coordinate in P1 is the first position coordinate of the i-th first preset target, P1 represents the set of all first position coordinates, P 2i Indicates: P 1i The third position coordinates in the coordinate system used in the first bird's-eye view, 1≤i≤n, i is a positive integer, and the set of the third position coordinates is recorded as P2.

[0078] S4. Based on the second transformation matrix, perform an inverse perspective projection transformation on the first infrared image to obtain a second bird's-eye view, and use the second transformation matrix to transform each second position coordinate to obtain a fourth position coordinate of each second position coordinate in the coordinate system used by the second bird's-eye view;

[0079] The process of obtaining the second transformation matrix includes:

[0080] The time-aligned infrared cameras are used to track the target, the trajectory of each third preset target is obtained, and straight line fitting is performed to fit the second straight line corresponding to each third preset target, and the second transformation matrix is ​​determined based on the two outermost second straight lines.

[0081] Among them, the second infrared image taken by the infrared camera can be identified by algorithms such as the YOLO algorithm or the ViD algorithm, and N4 targets can be detected. According to actual conditions, a part of the N4 targets can be selected as the third preset targets, or all the N4 targets can be used as the third preset targets. Since the detection frame of the target will be output by algorithms such as the YOLO algorithm or the ViD algorithm, the coordinates of the center point of the bottom edge of the detection frame of each target are used as the corresponding sixth position coordinates. The sixth position coordinates are the coordinates in the coordinate system used by the infrared camera. It should be noted that the first visible light image can also be used for target recognition through other single-stage detection algorithms.

[0082] And use an infrared camera to track each third preset target to obtain the trajectory of each third preset target, perform straight line fitting on the trajectory of each third preset target to obtain multiple second straight lines, obtain the intersection of all the second straight lines, calculate the second vanishing point, select the two outermost second straight lines, and record them as L3 and L4 respectively. When the second preset target is a vehicle, the second vanishing point is: the vanishing point of the road. When the target is different, the specific meaning of the vanishing point will also be different.

[0083] like Figure 4 As shown, the position coordinates of the third intersection A3 and the fourth intersection B3 are calculated. The third intersection A3 is the intersection of the third straight line L3 and the x-axis in the coordinate system where the infrared camera is located. The fourth intersection B3 is the intersection of the fourth straight line L4 and the x-axis in the coordinate system where the infrared camera is located. The position coordinates of the third intersection A3 and the fourth intersection B3 are both the position coordinates in the coordinate system where the infrared camera is located.

[0084] Calculate the position coordinates of the second vanishing point VP2 in the coordinate system of the infrared camera;

[0085] The midpoint of the line segment between the second vanishing point VP2 and the third intersection A3 is recorded as the third midpoint A4, and the midpoint of the line segment between the second vanishing point VP2 and the fourth intersection B3 is recorded as the fourth midpoint B4, and the position coordinates of the third midpoint A4 and the position coordinates of the fourth midpoint B4 are obtained. At this time, the third intersection A3, the fourth intersection B3, the third midpoint A4 and the fourth midpoint B4 form a trapezoid.

[0086] like Figure 5As shown, in the coordinate system used in the second bird's-eye view, a rectangle is preset, and the four corner points of the rectangle are respectively recorded as: the fifth corner point a3, the sixth corner point b3, the seventh corner point a4 and the eighth corner point b4. The trapezoid formed by the third intersection point A3, the fourth intersection point B3, the third midpoint A4 and the fourth midpoint B4 is corresponded to the rectangle formed by the fifth corner point a3, the sixth corner point b3, the seventh corner point a4 and the eighth corner point b4. Specifically, the third intersection point A3 corresponds to the fifth corner point a3, the third midpoint A4 corresponds to the seventh corner point a4, the fourth intersection point B4 corresponds to the sixth corner point b3, and the fourth midpoint B4 corresponds to the eighth corner point b4. According to the correspondence between these four pairs of points, the change matrix between the coordinate system used in the infrared and the coordinate system used in the second bird's-eye view can be obtained, that is, the second transformation matrix, which is recorded as H vis_bev,2 .

[0087] The coordinate system used for the second bird's-eye view is: the specified position on the second bird's-eye view is the origin, and the established rectangular coordinate system, the specified position can be the center point of the second bird's-eye view, or any position on the bottom edge of the second bird's-eye view, the straight line where the bottom edge of the second bird's-eye view is located can be the x-axis of the rectangular coordinate system. It should be noted that the origin of the second bird's-eye view can be set according to actual conditions, and the x-axis and y-axis of the rectangular coordinate system can also be set according to actual conditions.

[0088] The second transformation matrix is ​​used to transform each second position coordinate to obtain a fourth position coordinate of each second position coordinate in the coordinate system used in the second bird's-eye view, including:

[0089] The fourth position coordinate of each second position coordinate in the coordinate system used in the second bird's-eye view is obtained by using the second transformation formula. The first transformation formula is: 4i =H vis_bev,2 P 3i , where P 3i represents: the i-th second position coordinate in P2, the i-th second position coordinate in P2 is: the second position coordinate of the i-th first preset target, P3 represents: the set of all second position coordinates, P 4i Indicates: P 3i The fourth position coordinates in the coordinate system used by the second bird's-eye view, 1≤i≤n, i is a positive integer, and the set of the fourth position coordinates is recorded as P4.

[0090] S5, calculating a rigid transformation matrix between a coordinate system used by the first bird's-eye view and a coordinate system used by the second bird's-eye view;

[0091] Specifically, an ICP algorithm (Iterative closest point) or a CPD algorithm (Coherent point drift) may be used to calculate a rigid transformation matrix between a coordinate system used in the first bird's-eye view and a coordinate system used in the second bird's-eye view.

[0092] S6. Using the rigid transformation matrix, obtain the fifth position coordinate of each third position coordinate in the coordinate system used in the second bird's-eye view, which is specifically achieved by the third transformation formula. The third transformation formula is: 2i =H2P 2i , where H2 represents the rigid transformation matrix, R 2i Indicates: P 2i The fifth position coordinates in the coordinate system used by the second bird's-eye view, and the set of all the fifth position coordinates are denoted as R2.

[0093] S7, calculating the error distance between all the fourth position coordinates and all the fifth position coordinates; determining whether the error distance is less than a preset error distance threshold, and obtaining a determination result;

[0094] The error distance may be a Euclidean distance, and the error distance (Euclidean distance) dist is calculated by the following formula:

[0095]

[0096] Among them, the preset error distance threshold can be set according to actual conditions.

[0097] S8. When the judgment result is yes, determining the conversion relationship between the images respectively captured by the visible light camera and the infrared camera at the same time according to all the fourth position coordinates and all the fifth position coordinates;

[0098] S9. Using the conversion relationship, spatially align the images taken by the visible light camera and the infrared camera at the same time.

[0099] The set of all fourth position coordinates is recorded as P4, and the set of all fourth position coordinates is recorded as R2. Through the one-to-one correspondence between the position coordinates of P4 and R2, the one-to-one correspondence between the position coordinates of P1 and P3 can be determined, and then the homography transformation matrix H used to characterize the transformation relationship can be calculated:

[0100]

[0101] Among them, h 00 、h 01 、h 02 、h 10 、h 11 、h 12 、h20 、h 21 and h 22 is the element in the homography transformation matrix H.

[0102] At this time, the time-aligned visible light camera and infrared camera are used to track the target. The position coordinates of the same target monitored at any same time in the corresponding coordinate system satisfy:

[0103]

[0104] Among them, the position coordinates in the coordinate system used by the visible light camera are (u, v), and the position coordinates in the coordinate system used by the infrared camera are (x, y), and (u, v) and (x, y) are: the position coordinates of the same target monitored by the time-aligned visible light camera and infrared camera at any same time, and s represents the coefficient of the homogeneous coordinates.

[0105] Through the homography transformation matrix H, the position coordinates in the coordinate system used by the visible light camera can be accurately converted to the coordinate system used by the infrared camera, and the position coordinates in the coordinate system used by the infrared camera can also be accurately converted to the coordinate system used by the visible light.

[0106] Optionally, the above technical solution also includes: when the judgment result is no, the one-to-one correspondence between the position coordinates of P1 and P3 can be manually aligned, and then the homography transformation matrix used to characterize the transformation relationship can be calculated, or the homography transformation matrix used to characterize the transformation relationship can be obtained through other calculation methods.

[0107] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0108] A spatial registration system for visible light images and infrared images according to an embodiment of the present invention comprises an image acquisition module, a target recognition module, a bird's-eye view acquisition module, a position coordinate conversion module, a rigid transformation matrix calculation module, an error distance calculation and judgment module, a conversion relationship determination module and a spatial registration module;

[0109] The image acquisition module is used to: after time alignment of the visible light camera and the infrared camera, use the visible light camera and the infrared camera to respectively acquire a first visible light image and a first infrared image at the same time;

[0110] The target recognition module is used to: perform target recognition on the first visible light image to obtain the first position coordinates of each first preset target in the coordinate system used by the visible light camera; recognize the first infrared image to obtain the second position coordinates of each first preset target in the coordinate system used by the infrared camera;

[0111] The bird's-eye view acquisition module is used to: perform inverse perspective projection transformation on the first visible light image based on the first transformation matrix to obtain a first bird's-eye view;

[0112] The position coordinate conversion module is used to: transform each first position coordinate using a first transformation matrix to obtain a third position coordinate of each first position coordinate in a coordinate system used in the first bird's-eye view;

[0113] The bird's-eye view acquisition module is further used to: perform inverse perspective projection transformation on the first infrared image based on a second transformation matrix to obtain a second bird's-eye view;

[0114] The position coordinate conversion module is further used to: transform each second position coordinate using a second transformation matrix to obtain a fourth position coordinate of each second position coordinate in a coordinate system used in the second bird's-eye view;

[0115] The rigid transformation matrix calculation module is used to: calculate the rigid transformation matrix between the coordinate system used by the first bird's-eye view and the coordinate system used by the second bird's-eye view;

[0116] The position coordinate conversion module is further used to: obtain a fifth position coordinate of each third position coordinate in the coordinate system used in the second bird's-eye view by using a rigid transformation matrix;

[0117] The error distance calculation and judgment module is used to: calculate the error distance between all the fourth position coordinates and all the fifth position coordinates; judge whether the error distance is less than a preset error distance threshold, and obtain a judgment result;

[0118] The conversion relationship determination module is used to: when the judgment result is yes, determine the conversion relationship between the images respectively taken by the visible light camera and the infrared camera at the same time according to all the fourth position coordinates and all the fifth position coordinates;

[0119] The spatial registration module is used to register the images taken by the visible light camera and the infrared camera at the same time using the conversion relationship.

[0120] Optionally, in the above technical solution, a first transformation matrix acquisition module is further included, and the first transformation matrix acquisition module is used to:

[0121] The time-aligned visible light camera is used to track the target, the trajectory of each second preset target is obtained, and straight line fitting is performed to fit the first straight line corresponding to each second preset target, and the first transformation matrix is ​​determined based on the two outermost first straight lines.

[0122] Optionally, in the above technical solution, a second transformation matrix acquisition module is further included, and the second transformation matrix acquisition module is used to:

[0123] The time-aligned infrared cameras are used to track the target, the trajectory of each third preset target is obtained, and straight line fitting is performed to fit the second straight line corresponding to each third preset target, and the second transformation matrix is ​​determined based on the two outermost second straight lines.

[0124] Optionally, in the above technical solution, the rigid transformation matrix calculation module is specifically used to: use an ICP algorithm or a CPD algorithm to calculate a rigid transformation matrix between a coordinate system used for the first bird's-eye view and a coordinate system used for the second bird's-eye view.

[0125] It should be noted that the beneficial effects of the spatial registration system of a visible light image and an infrared image provided by the above embodiment are the same as the beneficial effects of the spatial registration method of a visible light image and an infrared image, which will not be described in detail here. In addition, when the system provided by the above embodiment realizes its functions, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided by the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be described in detail here.

[0126] Among them, the spatial registration system of a visible light image and an infrared image of the present invention can be a computer program (including program code) running in a computer device. For example, the spatial registration system of a visible light image and an infrared image of the present invention is an application software that can be used to execute the corresponding steps in the spatial registration method of a visible light image and an infrared image of the present invention.

[0127] In some embodiments, a spatial registration system for a visible light image and an infrared image of the present invention can be implemented by a combination of software and hardware. As an example, a spatial registration system for a visible light image and an infrared image of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute a spatial registration method for a visible light image and an infrared image of the present invention. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic components.

[0128] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The name of a module does not limit the module itself in some cases.

[0129] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any one of the above-mentioned methods for spatial registration of visible light images and infrared images is implemented. That is to say, an electronic device according to an embodiment of the present invention may include but is not limited to: a processor and a memory; a memory for storing a computer program; a processor for executing a method for spatial registration of visible light images and infrared images shown in any one of the embodiments of the present invention by calling the computer program.

[0130] In an alternative embodiment, an electronic device is provided, such as Figure 6 As shown, Figure 6 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0131] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0132] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 In the figure, only one thick line is used to represent the bus 4002, but this does not mean that there is only one bus or one type of bus.

[0133] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0134] The memory 4003 is used to store the application code (computer program) for executing the solution of the present invention, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the application code stored in the memory 4003 to implement the content shown in the above method embodiment.

[0135] Among them, the electronic device can also be a terminal device, and the terminal device can be any terminal device that can install applications, including at least one of a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart TV, and a smart car device.

[0136] It should be noted that Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0137] A computer-readable storage medium according to an embodiment of the present invention stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned methods for spatial registration of visible light images and infrared images is implemented.

[0138] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0139] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above-mentioned methods for spatial registration of visible light images and infrared images.

[0140] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0141] It should be understood that the flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] The computer-readable storage medium provided by the embodiment of the present invention may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or component.

[0143] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0144] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) to form a technical solution.

[0145] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0146] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present invention can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0147] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for spatial registration of a visible light image and an infrared image, characterized in that: include: After time alignment of the visible light camera and the infrared camera, using the visible light camera and the infrared camera to respectively collect a first visible light image and a first infrared image at the same time; Performing target recognition on the first visible light image to obtain first position coordinates of each first preset target in the coordinate system used by the visible light camera; Recognize the first infrared image to obtain the second position coordinates of each of the first preset targets in the coordinate system used by the infrared camera; Based on the first transformation matrix, perform an inverse perspective projection transformation on the first visible light image to obtain a first bird's-eye view, and use the first transformation matrix to transform each first position coordinate to obtain a third position coordinate of each first position coordinate in a coordinate system used by the first bird's-eye view; Based on the second transformation matrix, the first infrared image is subjected to inverse perspective projection transformation to obtain a second bird's-eye view, and each second position coordinate is transformed by using the second transformation matrix to obtain fourth position coordinates of each second position coordinate in a coordinate system used by the second bird's-eye view; Calculating a rigid transformation matrix between a coordinate system used by the first bird's-eye view and a coordinate system used by the second bird's-eye view; Using the rigid transformation matrix, obtaining a fifth position coordinate of each third position coordinate in the coordinate system used by the second bird's-eye view; Calculate the error distance between all fourth position coordinates and all fifth position coordinates; determine whether the error distance is less than a preset error distance threshold, and obtain a determination result; When the judgment result is yes, determining a conversion relationship between the images respectively captured by the visible light camera and the infrared camera at the same time according to all the fourth position coordinates and all the fifth position coordinates; The conversion relationship is used to perform spatial registration on the images respectively captured by the visible light camera and the infrared camera at the same time.

2. The method for spatial registration of a visible light image and an infrared image according to claim 1, characterized in that: The process of obtaining the first transformation matrix includes: The time-aligned visible light camera is used to track the target, the trajectory of each second preset target is obtained, and straight line fitting is performed to fit the first straight line corresponding to each second preset target, and the first transformation matrix is ​​determined based on the two outermost first straight lines.

3. The method for spatial registration of a visible light image and an infrared image according to claim 1, characterized in that: The process of obtaining the second transformation matrix includes: The target is tracked using the time-aligned infrared camera to obtain the trajectory of each third preset target and perform straight line fitting to fit the second straight line corresponding to each third preset target, and the second transformation matrix is ​​determined based on the two outermost second straight lines.

4. A method for spatial registration of a visible light image and an infrared image according to any one of claims 1 to 3, characterized in that: Calculating a rigid transformation matrix between a coordinate system used by the first bird's-eye view and a coordinate system used by the second bird's-eye view includes: The ICP algorithm or the CPD algorithm is used to calculate and obtain a rigid transformation matrix between the coordinate system used by the first bird's-eye view and the coordinate system used by the second bird's-eye view.

5. A spatial registration system for visible light images and infrared images, characterized in that: It includes an image acquisition module, a target recognition module, a bird's-eye view acquisition module, a position coordinate conversion module, a rigid transformation matrix calculation module, an error distance calculation and judgment module, a conversion relationship determination module and a space registration module; The image acquisition module is used to: after time alignment of the visible light camera and the infrared camera, use the visible light camera and the infrared camera to respectively acquire a first visible light image and a first infrared image at the same time; The target recognition module is used to: perform target recognition on the first visible light image to obtain the first position coordinates of each first preset target in the coordinate system used by the visible light camera; recognize the first infrared image to obtain the second position coordinates of each first preset target in the coordinate system used by the infrared camera; The bird's-eye view acquisition module is used to: perform inverse perspective projection transformation on the first visible light image based on the first transformation matrix to obtain a first bird's-eye view; The position coordinate conversion module is used to: transform each first position coordinate using the first transformation matrix to obtain a third position coordinate of each first position coordinate in the coordinate system used in the first bird's-eye view; The bird's-eye view acquisition module is further used to: perform inverse perspective projection transformation on the first infrared image based on the second transformation matrix to obtain a second bird's-eye view; The position coordinate conversion module is further used to: transform each second position coordinate using the second transformation matrix to obtain a fourth position coordinate of each second position coordinate in the coordinate system used by the second bird's-eye view; The rigid transformation matrix calculation module is used to: calculate the rigid transformation matrix between the coordinate system used by the first bird's-eye view and the coordinate system used by the second bird's-eye view; The position coordinate conversion module is further used to: obtain a fifth position coordinate of each third position coordinate in the coordinate system used by the second bird's-eye view using the rigid transformation matrix; The error distance calculation and judgment module is used to: calculate the error distance between all the fourth position coordinates and all the fifth position coordinates; determine whether the error distance is less than a preset error distance threshold, and obtain a judgment result; The conversion relationship determination module is used to: when the judgment result is yes, determine the conversion relationship between the images respectively taken by the visible light camera and the infrared camera at the same time according to all the fourth position coordinates and all the fifth position coordinates; The spatial registration module is used to register the images taken by the visible light camera and the infrared camera at the same time using the conversion relationship.

6. The spatial registration system of a visible light image and an infrared image according to claim 5, characterized in that: The invention also includes a first transformation matrix acquisition module, wherein the first transformation matrix acquisition module is used to: The time-aligned visible light camera is used to track the target, the trajectory of each second preset target is obtained, and straight line fitting is performed to fit the first straight line corresponding to each second preset target, and the first transformation matrix is ​​determined based on the two outermost first straight lines.

7. The spatial registration system of a visible light image and an infrared image according to claim 5, characterized in that: The invention also includes a second transformation matrix acquisition module, wherein the second transformation matrix acquisition module is used to: The target is tracked using the time-aligned infrared camera to obtain the trajectory of each third preset target and perform straight line fitting to fit the second straight line corresponding to each third preset target, and the second transformation matrix is ​​determined based on the two outermost second straight lines.

8. A spatial registration system for visible light images and infrared images according to any one of claims 5 to 7, characterized in that: The rigid transformation matrix calculation module is specifically used to: use the ICP algorithm or the CPD algorithm to calculate the rigid transformation matrix between the coordinate system used by the first bird's-eye view and the coordinate system used by the second bird's-eye view.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a method for spatial registration of a visible light image and an infrared image as claimed in any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for spatial registration of a visible light image and an infrared image according to any one of claims 1 to 4 is implemented.

Images