Image registration method and device, electronic equipment and readable storage medium

Abstract

The application discloses a kind of image registration method, device, electronic equipment and readable storage medium, belong to image processing technical field.The method includes: respectively obtaining the first infrared image of infrared lens acquisition and the first visible light image of visible light lens acquisition;Determine the first coordinate mapping relationship and the second coordinate mapping relationship corresponding to the first object distance based on the first object distance, the first coordinate mapping relationship is used to describe the transformation relationship between real visible light pixel coordinate system and ideal pixel coordinate system, the second coordinate mapping relationship is used to describe the transformation relationship between real infrared pixel coordinate system and ideal pixel coordinate system;According to the first coordinate mapping relationship, the first visible light image is carried out coordinate transformation, obtains the target visible light image after registration, and according to the second coordinate mapping relationship, the first infrared image is carried out coordinate transformation, obtains the target infrared image after registration.The application converts two images into the same coordinate system, improves the accuracy of image registration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a registration method and device of images, an electronic device and a readable storage medium. BACKGROUND

[0002] With the development of society, the cooperation of an infrared lens and a visible light lens has a wide application in the field of image processing. For example, the infrared lens and the visible light lens can be used in a temperature measurement scene. In the cooperation of the infrared lens and the visible light lens, an infrared image and a visible light image acquired are usually fused so that a user can obtain more information from the fused image. However, due to the different resolutions of the infrared lens and the visible light lens, the difference between the spatial positions of the two lenses, the inconsistency of lens distortion parameters and other reasons, the image information between the infrared image and the visible light image is deviated. Therefore, in order to better fuse the two images and enable the user to obtain accurate information from the fused image, the infrared image and the visible light image usually need to be registered.

[0003] At present, in the case of needing to register the infrared image and the visible light image, feature points of the infrared image and the visible light image can be extracted, and then feature point matching is performed. The transformation parameters of the infrared image to the visible light image are obtained according to the matched feature points, and the infrared image is transformed according to the transformation parameters, so as to realize the registration of the images.

[0004] However, in the process of extracting the feature points and performing the matching according to the feature points, the feature point extraction may be inaccurate due to noise or similar structures in the image, which may cause incorrect feature point matching and inaccurate registration results. SUMMARY

[0005] Embodiments of the present application provide a registration method and device of images, an electronic device and a readable storage medium, which can solve the problem of inaccurate image registration in the related art. The technical solution is as follows:

[0006] In one aspect, a registration method of images is provided, applied to an electronic device, and the method comprises:

[0007] A first infrared image collected by an infrared lens and a first visible light image collected by a visible light lens are acquired respectively, the infrared lens and the visible light lens are used cooperatively, and the first infrared image and the first visible light image are images containing a target object;

[0008] determine, based on the first object distance, a first coordinate mapping relationship and a second coordinate mapping relationship corresponding to the first object distance, the first object distance being a distance between the target object and a lens connecting line, the lens connecting line being a connecting line between an optical center of the visible light lens and an optical center of the infrared lens, the first coordinate mapping relationship being used to describe a transformation relationship between a real visible light pixel coordinate system and an ideal pixel coordinate system, the second coordinate mapping relationship being used to describe a transformation relationship between a real infrared pixel coordinate system and the ideal pixel coordinate system, the ideal pixel coordinate system being an ideal infrared pixel coordinate system or an ideal visible light pixel coordinate system, the ideal infrared pixel coordinate system being a pixel coordinate system in which an infrared image without distortion is located, and the ideal visible light pixel coordinate system being a pixel coordinate system in which a visible light image without distortion is located;

[0009] perform coordinate transformation on the first visible light image according to the first coordinate mapping relationship to obtain a target visible light image after registration, and perform coordinate transformation on the first infrared image according to the second coordinate mapping relationship to obtain a target infrared image after registration.

[0010] In another aspect, an image registration apparatus is provided, which is applied to an electronic device and includes:

[0011] an acquisition module configured to acquire a first infrared image captured by an infrared lens and a first visible light image captured by a visible light lens, the infrared lens and the visible light lens being used in cooperation, and the first infrared image and the first visible light image both being images containing a target object;

[0012] a determination module configured to determine, based on a first object distance, a first coordinate mapping relationship and a second coordinate mapping relationship corresponding to the first object distance, the first object distance being a distance between the target object and a lens connecting line, the lens connecting line being a connecting line between an optical center of the visible light lens and an optical center of the infrared lens, the first coordinate mapping relationship being used to describe a transformation relationship between a real visible light pixel coordinate system and an ideal pixel coordinate system, the second coordinate mapping relationship being used to describe a transformation relationship between a real infrared pixel coordinate system and the ideal pixel coordinate system, the ideal pixel coordinate system being an ideal infrared pixel coordinate system or an ideal visible light pixel coordinate system, the ideal infrared pixel coordinate system being a pixel coordinate system in which an infrared image without distortion is located, and the ideal visible light pixel coordinate system being a pixel coordinate system in which a visible light image without distortion is located;

[0013] a coordinate transformation module configured to perform coordinate transformation on the first visible light image according to the first coordinate mapping relationship to obtain a target visible light image after registration, and perform coordinate transformation on the first infrared image according to the second coordinate mapping relationship to obtain a target infrared image after registration.

[0014] In a third aspect, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the image registration method of the first aspect when executing the computer program.

[0015] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores instructions, and the instructions are executed by a processor to implement the image registration method of the first aspect.

[0016] In a fifth aspect, a computer program product is provided, and the computer program product includes instructions, and the instructions make the computer execute the image registration method of the first aspect when the computer program product is executed on the computer.

[0017] It can be understood that the beneficial effects of the second aspect to the fifth aspect can be referred to the related description in the first aspect, and will not be repeated here.

[0018] The technical scheme provided by the embodiments of the present application has the beneficial effects that:

[0019] In the embodiments of the present application, after the first infrared image and the first visible light image are acquired, the corresponding first coordinate mapping relationship and the second coordinate mapping relationship can be determined according to the first object distance, and the first infrared image and the first visible light image are respectively converted into the ideal pixel coordinate system according to the first coordinate mapping relationship and the second coordinate mapping relationship. Since the first infrared image and the first visible light image are converted into the same ideal pixel coordinate system, the image registration is realized, and since the corresponding first coordinate mapping relationship and the second coordinate mapping relationship can be determined according to the first object distance in the registration process, the time required for registration is reduced. In addition, since the image registration is the coordinate conversion between the real pixel coordinate system and the ideal pixel coordinate system, that is, the image registration is realized based on the lens imaging principle, the operations of feature point extraction and feature point matching are not required, thereby improving the accuracy and stability of the image registration. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] Figure 1 is a flow chart of an image registration method according to an exemplary embodiment;

[0022] Figure 2is a schematic diagram of a geometric relationship between a target object, an infrared lens and a visible light lens according to an example embodiment;

[0023] Figure 3 is a flow chart of a method for determining a first coordinate mapping relationship and a second coordinate mapping relationship according to an example embodiment;

[0024] Figure 4 is a schematic diagram of a coordinate relationship conversion according to an example embodiment;

[0025] Figure 5 is another schematic diagram of a coordinate relationship conversion according to another example embodiment;

[0026] Figure 6 is a structural schematic diagram of an image registration device according to an example embodiment;

[0027] Figure 7 is a structural schematic diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0028] In order to make the objects, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0029] It should be understood that the "multiple" mentioned in the present application refers to two or more than two. In the description of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B; "and / or" in the present application only describes the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent three cases: A exists alone, A and B exist together, and B exists alone. In addition, in order to clearly describe the technical solutions of the present application, the same items or similar items with basically the same functions and effects are distinguished by using "first", "second" and the like. Those skilled in the art can understand that "first", "second" and the like do not limit the quantity and execution order, and "first", "second" and the like do not necessarily mean different.

[0030] Reference within the specification of this application to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" or "in some embodiments" in various places within specified

[0031] Before the image registration method provided by the embodiments of the present application is explained in detail, an application scenario provided by the embodiments of the present application is described.

[0032] In some scenarios, because the infrared image and the visible light image can respectively convey different image information, in order to enable a user to obtain more image information, the infrared lens and the visible light lens can be used in cooperation. For example, the infrared lens and the visible light lens can be used in cooperation in a temperature measurement scenario. Because the images captured by the infrared lens and the visible light lens both have certain distortion, and because of the reasons such as different resolutions of the infrared lens and the visible light lens, differences in spatial positions of the two lenses, and inconsistent lens distortion parameters, there are some structural differences between the infrared image and the visible light image, and thus there is a deviation in the image information between the infrared image and the visible light image, and the user can be difficult to match the infrared image and the visible light image, and thus it is inconvenient to comprehensively utilize the image information of the infrared image and the visible light image. Therefore, in order to improve the deviation problem of the image information between the infrared image and the visible light image, the infrared image and the visible light image can be registered by using a feature point matching manner. However, because the feature point matching manner needs to extract feature points, and in the process of extracting the feature points, the feature point extraction can be inaccurate due to reasons such as noise or similar structures in the two images, and thus incorrect feature point matching can be caused, resulting in inaccurate registration result.

[0033] Based on such an application scenario, the embodiments of the present application provide an image registration method capable of improving the accuracy of image registration.

[0034] Figure 1 is a flowchart of an image registration method according to an exemplary embodiment. The method is applied in an electronic device, and the method can include the following steps:

[0035] Step 101: respectively acquire a first infrared image captured by an infrared lens and a first visible light image captured by a visible light lens.

[0036] It should be noted that the infrared lens and the visible light lens are used in cooperation, and the first infrared image and the first visible light image are images containing the target object.

[0037] As an example, the infrared lens and the visible light lens can be both installed in an electronic device, for example, the electronic device is a temperature measurement device, and the infrared lens and the visible light lens are included in the temperature measurement device. Alternatively, the infrared lens and the electronic device are two devices independently installed with each other, and the visible light lens is installed in the electronic device, and the infrared lens and the electronic device are communicatively connected. Alternatively, the visible light lens and the electronic device are two devices independently installed with each other, and the infrared lens is installed in the electronic device, and the visible light lens and the electronic device are communicatively connected. Alternatively, the infrared lens, the visible light lens and the electronic device are three devices independently installed with each other, and the infrared lens and the visible light lens are communicatively connected with the electronic device. The embodiments of the present application do not make specific limitation on this.

[0038] In some embodiments, the electronic device can directly obtain the first infrared image from the infrared lens and directly obtain the first visible light image from the visible light lens. Alternatively, the electronic device can also obtain the first infrared image collected by the infrared lens from other devices and obtain the first visible light image collected by the visible light lens from other devices.

[0039] Step 102: determining a first coordinate mapping relationship and a second coordinate mapping relationship corresponding to the first object distance based on the first object distance.

[0040] It should be noted that the first object distance is the distance between the target object and the lens connection line, and the lens connection line is the connection line between the optical center of the visible light lens and the optical center of the infrared lens. For example, referring to Figure 2 , the first object distance between the target object A and the visible light lens is d obj , the first object distance between the target object A and the infrared lens is d obj That is, the first object distance can be the distance from the target object A to the lens connection line.

[0041] It should also be noted that the first coordinate mapping relationship is used to describe the transformation relationship between the real visible light pixel coordinate system and the ideal pixel coordinate system, and the second coordinate mapping relationship is used to describe the transformation relationship between the real infrared pixel coordinate system and the ideal pixel coordinate system. The ideal pixel coordinate system is an ideal infrared pixel coordinate system or an ideal visible light pixel coordinate system. The ideal infrared pixel coordinate system is a pixel coordinate system in which the infrared image does not occur distortion, and the ideal visible light pixel coordinate system is a pixel coordinate system in which the visible light image does not occur distortion.

[0042] Since there can be a deviation in image information between the first infrared image and the first visible light image, in order to improve the deviation in image information between the first infrared image and the first visible light image, it is usually necessary to convert (or map) the first infrared image and the first visible light image into the same coordinate system. Different object distances can correspond to different coordinate mapping relationships, and therefore the electronic device needs to determine the first coordinate mapping relationship and the second coordinate mapping relationship corresponding to the first object distance based on the first object distance.

[0043] In some embodiments, referring to Figure 3 , the operation of determining, by the electronic device and based on the first object distance, the first coordinate mapping relationship and the second coordinate mapping relationship corresponding to the first object distance includes: step A, obtaining the first object distance. Step B, determining whether a preset first mapping relationship set and a preset second mapping relationship set are obtained, and if so, performing the operation of step C below, and if not, performing the operation of step E below. Step C, determining whether the first object distance is located in a target object distance range in the at least one object distance range, and if so, performing the operation of step D, and if not, performing the operation of step E. Step D, obtaining the first coordinate mapping relationship corresponding to the target object distance range in the first mapping relationship set, and obtaining the second coordinate mapping relationship corresponding to the target object distance range in the second mapping relationship set. Step E, determining the first coordinate mapping relationship and the second coordinate mapping relationship based on the geometric relationship between the target object, the infrared lens and the visible light lens, and the lens parameters of the infrared lens and the lens parameters of the visible light lens, respectively.

[0044] As an example, the first object distance can be obtained by a distance measuring instrument installed in the visible light lens or the infrared lens after measuring the distance to the target object, or can be manually input by the user into the electronic device. The embodiments of the present application do not make specific limitations on this.

[0045] In order to facilitate the understanding of the operations of steps B-E above, the operations of the above steps are expressed in another way, that is: in the case that the preset first mapping relationship set and the preset second mapping relationship set are obtained, and the first object distance is located in a target object distance range in the at least one object distance range, the first coordinate mapping relationship corresponding to the target object distance range in the first mapping relationship set is obtained, and the second coordinate mapping relationship corresponding to the target object distance range in the second mapping relationship set is obtained, and the first coordinate mapping relationship and the second coordinate mapping relationship corresponding to the target object distance range are the first coordinate mapping relationship and the second coordinate mapping relationship corresponding to the first object distance; in the case that the first mapping relationship set and the second mapping relationship set are not obtained, or the first object distance is located outside each object distance range in the at least one object distance range, the first coordinate mapping relationship and the second coordinate mapping relationship are determined based on the geometric relationship between the target object, the infrared lens and the visible light lens, and the lens parameters of the infrared lens and the lens parameters of the visible light lens, respectively.

[0046] It should be noted that the first mapping relationship set includes at least one preset first coordinate mapping relationship, the second mapping relationship set includes at least one preset second coordinate mapping relationship, and the target object distance range is any one of at least one object distance range, and the at least one object distance range, the at least one first coordinate mapping relationship in the first mapping relationship set and the at least one second coordinate mapping relationship in the second mapping relationship set are one-to-one corresponding. That is, each first coordinate mapping relationship included in the first mapping relationship set respectively corresponds to a second coordinate mapping relationship in the second mapping relationship set, and each first coordinate mapping relationship and the corresponding second coordinate mapping relationship have the same object distance range.

[0047] Exemplarily, the object distance range a is [1.6 meters-2.4 meters], which corresponds to the first coordinate mapping relationship a in the first mapping relationship set and the second coordinate mapping relationship a in the second mapping relationship set; the object distance range b is [3 meters-3.6 meters], which corresponds to the first coordinate mapping relationship b in the first mapping relationship set and the second coordinate mapping relationship b in the second mapping relationship set.

[0048] As an example, each first coordinate mapping relationship in the first mapping relationship set and each second coordinate mapping relationship in the second mapping relationship set can be determined in advance based on the corresponding object distance. The determination process can refer to the operation of determining the first coordinate mapping relationship and the second coordinate mapping relationship based on the geometric relationship between the target object, the infrared lens and the visible light lens, and the lens parameters of the infrared lens and the lens parameters of the visible light lens, respectively, of the electronic device described below, and the embodiments of the present application will not be described one by one.

[0049] Since the distance of the object collected by the infrared lens and the visible light lens can change, different object distances correspond to different first coordinate mapping relationships and different second coordinate mapping relationships, and in the case that the object distance changes little, the image coordinates change little. In this case, the first coordinate mapping relationship and the second coordinate mapping relationship corresponding to the object distance can be directly used, therefore, the corresponding object distance range can be set for the first coordinate mapping relationship and the second coordinate mapping relationship corresponding to the object distance, so as to expand the use range of the first coordinate mapping relationship and the corresponding second coordinate mapping relationship, and reduce the time required for image registration.

[0050] In some embodiments, in the case where the first mapping relationship set and the second mapping relationship set exist, the electronic device can further determine whether the first coordinate mapping relationship corresponding to the first object distance and the second coordinate mapping relationship exist in other manners. Illustratively, at least one first coordinate mapping relationship in the first mapping relationship set corresponds to at least one object distance in a one-to-one manner, and at least one second coordinate mapping relationship in the second mapping relationship set also corresponds to the at least one object distance in a one-to-one manner. In the case where the first mapping relationship set and the second mapping relationship set exist, the electronic device can further determine the difference between the first object distance and the object distance corresponding to each first coordinate mapping relationship in the first mapping relationship set, to obtain at least one object distance difference. In the case where there is a target object distance difference less than or equal to the distance threshold in the at least one object distance difference, the first coordinate mapping relationship and the second coordinate mapping relationship corresponding to the second object distance are obtained, and it is determined that the first coordinate mapping relationship and the second coordinate mapping relationship corresponding to the second object distance are the coordinate mapping relationship and the second coordinate mapping relationship corresponding to the first object distance, and the difference between the first object distance and the second object distance is the target object distance difference.

[0051] It should be noted that the distance threshold can be set in advance, for example, the distance threshold can be 0.2 meters, 0.3 meters or 0.4 meters, etc.

[0052] Since in the case where the preset first mapping relationship set and the preset second mapping relationship set are obtained, and the first object distance is located in the target object distance range in the at least one object distance range, it is indicated that the first mapping relationship set corresponding to the first object distance exists in the first mapping relationship set, therefore, the electronic device can obtain the first coordinate mapping relationship corresponding to the target object distance range in the first mapping relationship set, and obtain the second coordinate mapping relationship corresponding to the target object distance range in the second mapping relationship set.

[0053] As an example, the lens parameters of the infrared lens include a focal length, intrinsic parameters, a physical size represented by a single infrared pixel point, and distortion parameters of the infrared lens, the lens parameters of the visible light lens include a focal length, intrinsic parameters, a physical size represented by a single visible light pixel point, and distortion parameters of the visible light lens, the focal length of the infrared lens is the same as the focal length of the visible light lens, and the physical size represented by a single infrared pixel point is the same as the physical size represented by a single visible light pixel point; if the ideal pixel coordinate system is an ideal infrared pixel coordinate system, in a case where the first mapping relationship set and the second mapping relationship set are not acquired, or in a case where the first object distance is located outside each object distance range, the operation of determining the first coordinate mapping relationship and the second coordinate mapping relationship based on the geometric relationship among the target object, the infrared lens, and the visible light lens, and the lens parameters of the infrared lens and the lens parameters of the visible light lens includes: determining a horizontal pixel difference in a horizontal direction and a vertical pixel difference in a vertical direction between the infrared image collected by the infrared lens and the visible light image collected by the visible light lens according to the geometric relationship among the target object, the infrared lens, and the visible light lens, and the focal length and the physical size; determining the first coordinate mapping relationship based on the horizontal pixel difference, the vertical pixel difference, intrinsic parameters of the infrared lens, distortion parameters of the visible light lens, and intrinsic parameters of the visible light lens; and determining the second coordinate mapping relationship based on the intrinsic parameters of the infrared lens and the distortion parameters of the infrared lens.

[0054] To realize field of view alignment in the image registration process, the electronic device can determine a horizontal pixel difference in a horizontal direction and a vertical pixel difference in a vertical direction between the infrared image collected by the infrared lens and the visible light image collected by the visible light lens.

[0055] In some embodiments, the geometric relationship among the target object, the infrared lens, and the visible light lens includes the first object distance, an optical center distance between the infrared lens and the visible light lens, a first included angle between an optical axis of the visible light lens and a lens connection line, and a second included angle between an optical axis of the infrared lens and the lens connection line; in a case where the first mapping relationship set and the second mapping relationship set are not acquired, or in a case where the first object distance is located outside each object distance range, the electronic device determines the horizontal pixel difference by a first formula and determines the vertical pixel difference by a second formula according to the first object distance, the optical center distance, the first included angle, the second included angle, the focal length, and the physical size.

[0056] The first formula is:

[0057] The second formula is:

[0058] It should be noted that, in the above first formula (1) and second formula (2), Δx is the horizontal pixel difference, Δy is the vertical pixel difference, d obj is the first object distance, F is the focal length of the infrared lens or the visible light lens, dx is a physical size of a single infrared pixel point or a single visible light pixel point in a horizontal direction, d y is a physical size of a single infrared pixel point or a single visible light pixel point in a vertical direction, d υs is an angle of inclination of the target object relative to the optical center of the visible light lens, β ir is an angle of inclination of the target object relative to the optical center of the infrared lens, α υs is a first included angle, α ir is a second included angle.

[0059] In order to facilitate understanding of the above first formula and second formula, the embodiments of the present application combine Figure 2 to explain the above first formula and second formula. Referring to Figure 2 , since the infrared lens and the visible light lens are usually fixedly installed, the distance between the infrared lens and the visible light lens, i.e. the optical center distance d, is fixed, and since the focal length of the infrared lens is the same as the focal length of the visible light lens, the physical size of a single infrared pixel point in a horizontal direction is the same as the physical size of a single visible light pixel point in a horizontal direction, and the physical size of a single infrared pixel point in a vertical direction is the same as the physical size of a single visible light pixel point in a vertical direction, therefore, one of the intrinsic parameters f x of the infrared lens (wherein f x =F / d x ) is the same as one of the intrinsic parameters f x of the visible light lens, and the other intrinsic parameter f y of the infrared lens (wherein f y =F / d y ) is the same as the other intrinsic parameter f y of the visible light lens. However, when the same target object is imaged on the infrared lens and the visible light lens respectively, the distances of the respective imaging to the respective optical axes are not equal, i.e. Figure 2 d υs ≠d ir , (wherein d υs is the distance of the imaging of the target object in the visible light lens to the optical axis M of the visible light lens, and d ir is the distance of the imaging of the target object in the infrared lens to the optical axis N of the infrared lens). Therefore, referring to Figure 2 , the above first formula (1) can be derived by the following third formula, and the above second formula (2) can be derived by the following fourth formula.

[0060] The third formula is:

[0061] The fourth formula is:

[0062] From the above, f x = F / d x , f y = F / d y Therefore, F / d x in the first formula can also be replaced by f x , and F / d y in the second formula (2) can be replaced by f y . The embodiments of the present application do not make specific limitations on this.

[0063] In some embodiments, the operation of determining the first coordinate mapping relationship based on the horizontal pixel difference, the vertical pixel difference, the intrinsic parameter of the infrared lens, the distortion parameter of the visible light lens, and the intrinsic parameter of the visible light lens can include: determining a third coordinate mapping relationship between an ideal infrared pixel coordinate system and a reference pixel coordinate system according to the horizontal pixel difference and the vertical pixel difference, the reference pixel coordinate system being a pixel coordinate system of an undistorted infrared image collected by the infrared lens when the infrared lens is installed at a position where the visible light lens is located; determining a fourth coordinate mapping relationship between the reference pixel coordinate system and an ideal visible light image coordinate system according to the intrinsic parameter of the infrared lens, the ideal visible light image coordinate system being an image coordinate system of an undistorted visible light image; determining a fifth coordinate mapping relationship between the ideal visible light image coordinate system and a real visible light image coordinate system according to the distortion parameter of the visible light lens; determining a sixth coordinate mapping relationship between the real visible light image coordinate system and a real visible light pixel coordinate system according to the intrinsic parameter of the visible light lens; and determining the first coordinate mapping relationship according to the third coordinate mapping relationship, the fourth coordinate mapping relationship, the fifth coordinate mapping relationship, and the sixth coordinate mapping relationship.

[0064] As can be known from the imaging principle of a lens (including an infrared lens and a visible light lens), the image shooting process of the lens is actually an optical imaging process. The imaging process of the lens involves four coordinate systems, i.e., a world coordinate system, a camera coordinate system, an image coordinate system, a pixel coordinate system and conversion between the four coordinate systems. Among them, the target object is transformed from the world coordinate system to the camera coordinate system through rigid body transformation, and is then converted to the image coordinate system through perspective projection. Finally, in order to convert into a digital image, the image is sampled and converted to the pixel coordinate system. The pixel coordinate system and the image coordinate system are located in the same plane, the origins and the axis directions of the pixel coordinate system and the image coordinate system are different, and after the image is changed from the image coordinate system to the pixel coordinate system, there is also a certain scale zoom due to the difference in lens internal parameters. In addition, the image coordinate system includes an ideal image coordinate system and a real image coordinate system. The process of object projection onto the image plane involves pinhole imaging and imaging distortion. In the process of pinhole imaging, the image does not have distortion, and at this time the image is in the ideal image coordinate system. After imaging distortion occurs, the image is converted to the real image coordinate system. Therefore, the imaging process can be regarded as a series of coordinate conversion processes, i.e., the process of converting the world coordinate system to the camera coordinate system, converting the camera coordinate system to the ideal image coordinate system, converting the ideal image coordinate system to the real image coordinate system, and converting the real image coordinate system to the pixel image coordinate system. For the infrared lens and the visible light lens used in cooperation, the two are in the same world coordinate system. Therefore, the electronic device can determine an ideal visible light image coordinate system, an ideal infrared image coordinate system, a real visible light image coordinate system and a real infrared image coordinate system based on a horizontal pixel difference, a vertical pixel difference, a reference coordinate system, internal parameters of the infrared lens, distortion parameters of the infrared lens, internal parameters of the visible light lens and distortion parameters of the visible light lens, and obtain the corresponding relationship between the ideal infrared pixel coordinate system and the real visible light pixel coordinate system, and the corresponding relationship between the ideal infrared pixel coordinate system and the real infrared pixel coordinate system through conversion between the coordinate systems; or, obtain the corresponding relationship between the ideal visible light pixel coordinate system and the real visible light pixel coordinate system, and the corresponding relationship between the ideal visible light pixel coordinate system and the real infrared pixel coordinate system.

[0065] It should be noted that the distortion parameters of the infrared lens include radial distortion parameters and tangential distortion parameters, and the distortion parameters of the visible light lens also include radial distortion parameters and tangential distortion parameters, and the distortion parameters of the infrared lens are different from the distortion parameters of the visible light lens.

[0066] Exemplarily, the conversion relationship between the third coordinate mapping relationship, the fourth coordinate mapping relationship, the fifth coordinate mapping relationship and the sixth coordinate mapping relationship can be obtained through Figure 4 It is indicated that, in Figure 4In the method, the ideal infrared pixel coordinate system is converted into a reference pixel coordinate system according to horizontal pixel difference and vertical pixel difference (i.e. the field of view of the infrared lens is converted into the field of view of the visible light lens), the reference pixel coordinate system is converted into an ideal visible light image coordinate system according to internal parameters of the infrared lens, the ideal visible light image coordinate system is converted into a real visible light image coordinate system according to distortion parameters of the visible light lens, and the real visible light image coordinate is converted into a real visible light pixel coordinate system according to internal parameters of the visible light lens, so that a corresponding relationship between the ideal infrared pixel coordinate system and the real visible light pixel coordinate system is established through the conversion of the coordinate systems.

[0067] In some embodiments, the third coordinate mapping relationship can be represented by the following fifth formula, the fourth coordinate mapping relationship can be represented by the following sixth formula, the fifth coordinate mapping relationship can be represented by the following seventh formula, and the sixth coordinate mapping relationship can be represented by the following eighth formula.

[0068] The fifth formula is:

[0069] The sixth formula is:

[0070] The seventh formula is:

[0071] The eighth formula is:

[0072] It should be noted that in the above fifth formula (5), (u, v) is the reference pixel coordinate system, (u', v') is the ideal infrared pixel coordinate system, Δx is the horizontal pixel difference, and Δy is the vertical pixel difference. In the above sixth formula (6), (x', y') is the ideal visible light image coordinate system, f x , f y , u 0ir , and u 0ir are internal parameters of the infrared lens. In the above seventh formula (7), (x, y) is the real visible light image coordinate system, k 1υs , k 2υs , and k 3υs are radial distortion parameters of the visible light lens, p 1υs , and p 2υs are tangential distortion parameters of the visible light lens, and r is the distance from a coordinate point in the ideal visible light image coordinate system to the origin of the ideal visible light image coordinate system, where In the above eighth formula (8), (u υs , v υs ) is the real visible light pixel coordinate system, f x , f y , u 0υs , and u 0υs are internal parameters of the visible light lens.0υs and υ 0υs This is an internal reference for visible light lenses.

[0073] In some embodiments, when the ideal pixel coordinate system is an ideal infrared pixel coordinate system, the operation of the electronic device determining the second coordinate mapping relationship based on the intrinsic parameters and distortion parameters of the infrared lens includes: determining a seventh coordinate mapping relationship between the ideal infrared pixel coordinate system and the ideal infrared image coordinate system based on the intrinsic parameters of the infrared lens, wherein the ideal infrared image coordinate system is the image coordinate system in which the undistorted infrared image is located; determining an eighth coordinate mapping relationship between the ideal infrared image coordinate system and the real infrared image coordinate system based on the distortion parameters of the infrared lens; determining a ninth coordinate mapping relationship between the real infrared image coordinate system and the real infrared pixel coordinate system based on the intrinsic parameters of the infrared lens; and determining the second coordinate mapping relationship based on the seventh, eighth, and ninth coordinate mapping relationships.

[0074] For example, the transformation relationship between the seventh coordinate mapping relationship, the eighth coordinate mapping relationship, and the ninth coordinate mapping relationship can be achieved through... Figure 5 It means that, in Figure 5 In this process, the ideal infrared pixel coordinate system can be transformed into the ideal infrared image coordinate system based on the intrinsic parameters of the infrared lens. The ideal infrared image coordinate system can be transformed into the real infrared image coordinate system based on the distortion parameters of the infrared lens. The real infrared image coordinate system can be transformed into the real infrared pixel coordinate system based on the intrinsic parameters of the infrared lens. Thus, through the transformation of each coordinate system, a correspondence exists between the ideal infrared pixel coordinate system and the real infrared pixel coordinate system.

[0075] In some embodiments, the seventh coordinate mapping relationship can be represented by the following ninth formula, the eighth coordinate mapping relationship can be represented by the following tenth formula, and the ninth coordinate mapping relationship can be represented by the following eleventh formula.

[0076] Ninth Formula:

[0077] Formula 10:

[0078] Eleventh Formula:

[0079] It should be noted that in the above ninth formula (9), (x'1, y'1) is the ideal infrared image coordinate system, (u'1, y'1) is the ideal infrared image coordinate system, and (u'1, y'1) is the ideal infrared image coordinate system. ir υ′ ir (f) is an ideal infrared pixel coordinate system. x f y u 0ir and u 0iris an internal parameter of the infrared lens. In the tenth formula (10) above, (x1, y1) is a real infrared image coordinate system, k 1ir , k 2ir , and k 3ir are radial distortion parameters of the infrared lens, p 1ir and p 2ir are tangential distortion parameters of the infrared lens, and r1 is a distance from a coordinate point in an ideal infrared image coordinate system to an origin of the ideal infrared image coordinate system, where In the eleventh formula (11) above, (u 0ir , υ 0ir ) is a real infrared pixel coordinate system.

[0080] In some embodiments, if the ideal pixel coordinate system is an ideal visible light pixel coordinate system, and the first mapping relationship set and the second mapping relationship set are not acquired, or the first object distance is located outside each object distance range, the operation of determining the first coordinate mapping relationship and the second coordinate mapping relationship based on the geometric relationship between the target object, the infrared lens, and the visible light lens, and the lens parameters of the infrared lens and the visible light lens includes: determining, in a case where the first mapping relationship set and the second mapping relationship set are not acquired, or the first object distance is located outside each object distance range, a horizontal pixel difference in a horizontal direction and a vertical pixel difference in a vertical direction between an infrared image collected by the infrared lens and a visible light image collected by the visible light lens according to the geometric relationship between the target object, the infrared lens, and the visible light lens, and a focal length and a physical size; determining the first coordinate mapping relationship based on the horizontal pixel difference, the vertical pixel difference, an internal parameter of the visible light lens, distortion parameters of the infrared lens, and an internal parameter of the infrared lens; and determining the second coordinate mapping relationship based on the internal parameter of the visible light lens and the distortion parameters of the visible light lens.

[0081] It should be noted that, in a case where the ideal pixel coordinate system is an ideal visible light pixel coordinate system, the operation of determining the first coordinate mapping relationship based on the horizontal pixel difference, the vertical pixel difference, the internal parameter of the visible light lens, the distortion parameters of the infrared lens, and the internal parameter of the infrared lens, and the operation of determining the second coordinate mapping relationship based on the internal parameter of the visible light lens and the distortion parameters of the visible light lens can be respectively referred to the operation of determining the first coordinate mapping relationship based on the horizontal pixel difference, the vertical pixel difference, the internal parameter of the infrared lens, the distortion parameters of the visible light lens, and the internal parameter of the visible light lens, and the operation of determining the second coordinate mapping relationship based on the internal parameter of the infrared lens and the distortion parameters of the infrared lens in a case where the ideal pixel coordinate system is an ideal infrared pixel coordinate system, which will not be described herein again.

[0082] In some embodiments, after determining the first coordinate mapping relationship and the second coordinate mapping relationship based on the geometric relationship among the target object, the infrared lens and the visible light lens, and the lens parameters of the infrared lens and the lens parameters of the visible light lens, the electronic device can further update the first mapping relationship set and the second mapping relationship set according to the first coordinate mapping relationship and the second coordinate mapping relationship. For example, the electronic device can store the first coordinate mapping relationship corresponding to the first object distance into the first mapping relationship set, and store the second coordinate mapping relationship corresponding to the first object distance into the second mapping relationship set.

[0083] It should be noted that the object distance or the object distance range, the corresponding first coordinate mapping relationship and the corresponding second coordinate mapping relationship can be stored in the form of a table, or can be stored in other forms, and the embodiments of the present application do not make specific limitations thereto.

[0084] Step 103: performing coordinate transformation on the first visible light image according to the first coordinate mapping relationship to obtain a target visible light image after registration, and performing coordinate transformation on the first infrared image according to the second coordinate mapping relationship to obtain a target infrared image after registration.

[0085] Since the first infrared image is in the real infrared pixel coordinate system after being captured by the infrared lens, and the first visible light image is in the real visible light pixel coordinate system after being captured by the visible light lens. Therefore, in the case that the ideal pixel coordinate system is the ideal infrared pixel coordinate system, the electronic device can convert the first visible light image from the real visible light pixel coordinate system to the ideal infrared pixel coordinate system according to the first coordinate mapping relationship to obtain a target visible light image, and convert the first infrared image from the real infrared pixel coordinate system to the ideal infrared pixel coordinate system according to the second coordinate mapping relationship to obtain a target infrared image; in the case that the ideal pixel coordinate system is the ideal visible light pixel coordinate system, the first visible light image is converted from the real visible light pixel coordinate system to the ideal visible light pixel coordinate system according to the first coordinate mapping relationship to obtain a target visible light image, and the first infrared image is converted from the real infrared pixel coordinate system to the ideal visible light pixel coordinate system according to the second coordinate mapping relationship to obtain a target infrared image.

[0086] It should be noted that in order to enable the target visible light image and the target infrared image to be used together, the coordinates need to be converted to the same coordinate system during the registration of the first visible light image and the first infrared image.

[0087] Since the size of the infrared image is usually smaller than the size of the visible light image, in the case of image registration, the first infrared image and the first visible light image can be both converted to the ideal infrared pixel coordinate system to reduce the computational amount of image registration.

[0088] In the embodiments of the present application, after the first infrared image and the first visible light image are acquired, the corresponding first coordinate mapping relationship and the second coordinate mapping relationship can be determined according to the first object distance, and the first infrared image and the first visible light image are respectively converted into the ideal pixel coordinate system according to the first coordinate mapping relationship and the second coordinate mapping relationship. Since the first infrared image and the first visible light image are converted into the same ideal pixel coordinate system, the image registration is realized, and since the corresponding first coordinate mapping relationship and the second coordinate mapping relationship can be determined according to the first object distance in the registration process, the time required for registration is reduced. In addition, since the image registration is the coordinate conversion between the real pixel coordinate system and the ideal pixel coordinate system, that is, the image registration is realized based on the lens imaging principle, the feature point extraction and feature point matching operations are not required, thereby improving the accuracy and stability of the image registration.

[0089] All the optional technical solutions described above can be combined to form optional embodiments of the present application, and the embodiments of the present application will not be described one by one.

[0090] Figure 6 FIG. 1 is a structural schematic diagram of an image registration device according to an exemplary embodiment, which can be realized by software, hardware or a combination of both. The device is applied to an electronic device and includes:

[0091] The acquisition module 601 is configured to acquire a first infrared image collected by an infrared lens and a first visible light image collected by a visible light lens, the infrared lens and the visible light lens are used in cooperation, and the first infrared image and the first visible light image are images containing a target object;

[0092] The determination module 602 is configured to determine a first coordinate mapping relationship and a second coordinate mapping relationship corresponding to a first object distance based on the first object distance, the first object distance is a distance between the target object and a lens connection line, the lens connection line is a connection line between an optical center of the visible light lens and an optical center of the infrared lens, the first coordinate mapping relationship is used to describe a transformation relationship between a real visible light pixel coordinate system and an ideal pixel coordinate system, the second coordinate mapping relationship is used to describe a transformation relationship between a real infrared pixel coordinate system and the ideal pixel coordinate system, the ideal pixel coordinate system is an ideal infrared pixel coordinate system or an ideal visible light pixel coordinate system, the ideal infrared pixel coordinate system is a pixel coordinate system in which an infrared image without distortion is located, and the ideal visible light pixel coordinate system is a pixel coordinate system in which a visible light image without distortion is located;

[0093] The coordinate transformation module 603 is configured to perform coordinate transformation on the first visible light image according to the first coordinate mapping relationship to obtain a target visible light image after registration, and perform coordinate transformation on the first infrared image according to the second coordinate mapping relationship to obtain a target infrared image after registration.

[0094] As an example of the present application, the determination module 602 is configured to:

[0095] Obtain the first object distance;

[0096] In a case where the first mapping relationship set and the second mapping relationship set are obtained, and the first object distance is located in a target object distance range in the at least one object distance range, obtain a first coordinate mapping relationship corresponding to the target object distance range in the first mapping relationship set, and obtain a second coordinate mapping relationship corresponding to the target object distance range in the second mapping relationship set, the first coordinate mapping relationship and the second coordinate mapping relationship corresponding to the target object distance range being the first coordinate mapping relationship and the second coordinate mapping relationship corresponding to the first object distance, the first mapping relationship set including at least one first coordinate mapping relationship, the second mapping relationship set including at least one second coordinate mapping relationship, and the target object distance range being any one of the at least one object distance range, the at least one object distance range, the at least one first coordinate mapping relationship, and the at least one second coordinate mapping relationship corresponding to each other;

[0097] In a case where the first mapping relationship set and the second mapping relationship set are not obtained, or the first object distance is located outside each object distance range in the at least one object distance range, respectively determine the first coordinate mapping relationship and the second coordinate mapping relationship based on a geometric relationship between the target object, the infrared lens, and the visible light lens, and lens parameters of the infrared lens and lens parameters of the visible light lens.

[0098] As an example of the present application, the lens parameters of the infrared lens include a focal length, intrinsic parameters, a physical size represented by a single infrared pixel point, and distortion parameters of the infrared lens, the lens parameters of the visible light lens include a focal length, intrinsic parameters, a physical size represented by a single visible light pixel point, and distortion parameters of the visible light lens, the focal length of the infrared lens is the same as the focal length of the visible light lens, and the physical size represented by the single infrared pixel point is the same as the physical size represented by the single visible light pixel point; the ideal pixel coordinate system is the ideal infrared pixel coordinate system;

[0099] The determination module 602 is configured to:

[0100] determine, according to the geometric relationship among the target object, the infrared lens and the visible light lens, and the focal length and the physical size, a horizontal pixel difference and a vertical pixel difference between an infrared image captured by the infrared lens and a visible light image captured by the visible light lens in a horizontal direction and in a vertical direction;

[0101] determine the first coordinate mapping relationship based on the horizontal pixel difference, the vertical pixel difference, an intrinsic parameter of the infrared lens, a distortion parameter of the visible light lens and an intrinsic parameter of the visible light lens;

[0102] determine the second coordinate mapping relationship based on the intrinsic parameter of the infrared lens and the distortion parameter of the infrared lens.

[0103] As an example of the present application, the geometric relationship among the target object, the infrared lens and the visible light lens includes the first object distance, an optical center distance between the infrared lens and the visible light lens, a first included angle between an optical axis of the visible light lens and the lens connection line, and a second included angle between an optical axis of the infrared lens and the lens connection line;

[0104] The determination module 602 is configured to:

[0105] determine the horizontal pixel difference by a first formula and determine the vertical pixel difference by a second formula according to the first object distance, the optical center distance, the first included angle, the second included angle, the focal length and the physical size, in a case where the first mapping relationship set and the second mapping relationship set are not acquired, or in a case where the first object distance is located outside each object distance range;

[0106] The first formula is:

[0107] The second formula is:

[0108] wherein the Δx is the horizontal pixel difference, the Δy is the vertical pixel difference, the d obj is the first object distance, the F is the focal length, the d x is a physical size represented by the single infrared pixel point or the single visible light pixel point in the horizontal direction, the d y is a physical size represented by the single infrared pixel point or the single visible light pixel point in the vertical direction, the d is the optical center distance, the β υs is an inclination angle of the target object relative to an optical center of the visible light lens, and the βir is an inclination angle of the target object relative to an optical center of the infrared lens, and the a υs is the first included angle, and the a ir is the second included angle.

[0109] As an example of the present application, the determining module 602 is configured to:

[0110] According to the horizontal pixel difference and the vertical pixel difference, a third coordinate mapping relationship between the ideal infrared pixel coordinate system and a reference pixel coordinate system is determined, the reference pixel coordinate system being a pixel coordinate system of an undistorted infrared image collected by the infrared lens when the infrared lens is installed at a position of the visible light lens;

[0111] According to intrinsic parameters of the infrared lens, a fourth coordinate mapping relationship between the reference pixel coordinate system and an ideal visible light image coordinate system is determined, the ideal visible light image coordinate system being an image coordinate system of an undistorted visible light image;

[0112] According to distortion parameters of the visible light lens, a fifth coordinate mapping relationship between the ideal visible light image coordinate system and a real visible light image coordinate system is determined;

[0113] According to intrinsic parameters of the visible light lens, a sixth coordinate mapping relationship between the real visible light image coordinate system and a real visible light pixel coordinate system is determined;

[0114] According to the third coordinate mapping relationship, the fourth coordinate mapping relationship, the fifth coordinate mapping relationship and the sixth coordinate mapping relationship, the first coordinate mapping relationship is determined.

[0115] As an example of the present application, the determining module 602 is configured to:

[0116] According to intrinsic parameters of the infrared lens, a seventh coordinate mapping relationship between the ideal infrared pixel coordinate system and an ideal infrared image coordinate system is determined, the ideal infrared image coordinate system being an image coordinate system of an undistorted infrared image;

[0117] According to distortion parameters of the infrared lens, an eighth coordinate mapping relationship between the ideal infrared image coordinate system and a real infrared image coordinate system is determined;

[0118] According to intrinsic parameters of the infrared lens, a ninth coordinate mapping relationship between the real infrared image coordinate system and a real infrared pixel coordinate system is determined;

[0119] According to the seventh coordinate mapping relationship, the eighth coordinate mapping relationship and the ninth coordinate mapping relationship, the second coordinate mapping relationship is determined.

[0120] As an example of the present application, the coordinate transformation module 603 is configured to:

[0121] In the case that the ideal pixel coordinate system is the ideal infrared pixel coordinate system, the first visible light image is converted from the real visible light pixel coordinate system to the ideal infrared pixel coordinate system according to the first coordinate mapping relationship to obtain the target visible light image, and the first infrared image is converted from the real infrared pixel coordinate system to the ideal infrared pixel coordinate system according to the second coordinate mapping relationship to obtain the target infrared image.

[0122] In the case that the ideal pixel coordinate system is the ideal visible light pixel coordinate system, the first visible light image is converted from the real visible light pixel coordinate system to the ideal visible light pixel coordinate system according to the first coordinate mapping relationship to obtain the target visible light image, and the first infrared image is converted from the real infrared pixel coordinate system to the ideal visible light pixel coordinate system according to the second coordinate mapping relationship to obtain the target infrared image.

[0123] In the embodiments of the present application, after the first infrared image and the first visible light image are acquired, the corresponding first coordinate mapping relationship and the second coordinate mapping relationship can be determined according to the first object distance, and the first infrared image and the first visible light image are both converted to the ideal pixel coordinate system according to the first coordinate mapping relationship and the second coordinate mapping relationship. Since the first infrared image and the first visible light image are converted to the same ideal pixel coordinate system, image registration is achieved, and since the corresponding first coordinate mapping relationship and the second coordinate mapping relationship can be determined according to the first object distance in the registration process, the time required for registration is reduced. In addition, since the image registration is a coordinate conversion between the real pixel coordinate system and the ideal pixel coordinate system, i.e., the image registration is achieved based on the lens imaging principle, the operations of feature point extraction and feature point matching are not required, thereby improving the accuracy and stability of image registration.

[0124] Figure 7 The structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 7. As shown in FIG. 7, the electronic device 7 of this embodiment includes at least one processor 70 (only one processor is shown in the figure), a memory 71, and a computer program 72 stored in the memory 71 and executable on the at least one processor 70, wherein the processor 70 implements the steps in any of the method embodiments described above when executing the computer program 72. Figure 7 Figure 7

[0125] ​​The electronic device 7 can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The electronic device can include, but is not limited to, a processor 70, a memory 71. Those skilled in the art can understand that Figure 7 The electronic device 7 is only an example and does not limit the electronic device 7, and can include more or fewer components than shown, or combine certain components, or include different components, for example, can also include an input / output device, a network access device, and the like.

[0126] The processor 70 can be a CPU (Central Processing Unit), and can also be other general-purpose processors, DSPs (Digital Signal Processors), ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. The general-purpose processor can be a microprocessor or can also be any conventional processor.

[0127] The memory 71 can be an internal storage unit of the electronic device 7 in some embodiments, for example, a hard disk or a memory of the electronic device 7. The memory 71 can also be an external storage device of the electronic device 7 in other embodiments, for example, a plug-in hard disk, an SMC (Smart Media Card), an SD (Secure Digital) card, a Flash Card, and the like. Further, the memory 71 can include both the internal storage unit and the external storage device of the electronic device 7. The memory 71 is used to store an operating system, application programs, a BootLoader, data, and other programs, for example, program codes of the computer program, and the like. The memory 71 can also be used to temporarily store data that has been output or will be output.

[0128] It should be noted that the information interaction, execution process, and the like between the above-mentioned devices / units, since based on the same concept as the method embodiments of the present application, the specific functions and the technical effects brought by them can be referred to the method embodiments part, and will not be repeated here.

[0129] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for convenient distinction, and do not limit the protection scope of the application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0130] The above-described embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method of registration of images, characterized in that, The method is applied to an electronic device, and the method comprises: respectively acquiring a first infrared image collected by an infrared lens and a first visible light image collected by a visible light lens, the infrared lens and the visible light lens being used in cooperation, and the first infrared image and the first visible light image being images containing a target object; acquiring a first object distance, the first object distance being a distance between the target object and a lens connection line, the lens connection line being a connection line between an optical center of the visible light lens and an optical center of the infrared lens; in a case where a preset first mapping relationship set and a preset second mapping relationship set are acquired, and the first object distance is located within a target object distance range in at least one object distance range, acquiring a first coordinate mapping relationship corresponding to the target object distance range in the first mapping relationship set, and acquiring a second coordinate mapping relationship corresponding to the target object distance range in the second mapping relationship set, the first mapping relationship set comprising at least one first coordinate mapping relationship, the first coordinate mapping relationship and the second coordinate mapping relationship corresponding to the target object distance range being the first coordinate mapping relationship and the second coordinate mapping relationship corresponding to the first object distance, the second mapping relationship set comprising at least one second coordinate mapping relationship, and the target object distance range being any one of the at least one object distance range, the at least one object distance range, the at least one first coordinate mapping relationship, and the at least one second coordinate mapping relationship corresponding to each other, the first coordinate mapping relationship being used to describe a transformation relationship between a real visible light pixel coordinate system and an ideal pixel coordinate system, the second coordinate mapping relationship being used to describe a transformation relationship between a real infrared pixel coordinate system and the ideal pixel coordinate system, the ideal pixel coordinate system being an ideal infrared pixel coordinate system or an ideal visible light pixel coordinate system, the ideal infrared pixel coordinate system being a pixel coordinate system in which an infrared image without distortion is located, and the ideal visible light pixel coordinate system being a pixel coordinate system in which a visible light image without distortion is located; in a case where the first mapping relationship set and the second mapping relationship set are not acquired, or in a case where the first object distance is located outside each object distance range in the at least one object distance range, respectively determining the first coordinate mapping relationship and the second coordinate mapping relationship based on a geometric relationship among the target object, the infrared lens, and the visible light lens, and lens parameters of the infrared lens and lens parameters of the visible light lens; performing coordinate transformation on the first visible light image according to the first coordinate mapping relationship to obtain a target visible light image after registration, and performing coordinate transformation on the first infrared image according to the second coordinate mapping relationship to obtain a target infrared image after registration.

2. The method of claim 1, wherein, The lens parameters of the infrared lens include a focal length, intrinsic parameters, a physical size represented by a single infrared pixel point, and distortion parameters of the infrared lens, and the lens parameters of the visible light lens include a focal length, intrinsic parameters, a physical size represented by a single visible light pixel point, and distortion parameters of the visible light lens, the focal length of the infrared lens is the same as the focal length of the visible light lens, and the physical size represented by the single infrared pixel point is the same as the physical size represented by the single visible light pixel point; the ideal pixel coordinate system is the ideal infrared pixel coordinate system; The first coordinate mapping relationship and the second coordinate mapping relationship are determined based on a geometric relationship among the target object, the infrared lens, and the visible light lens, and lens parameters of the infrared lens and lens parameters of the visible light lens in a case where the first mapping relationship set and the second mapping relationship set are not acquired or in a case where the first object distance is located outside each object distance range in the at least one object distance range, including: In a case where the first mapping relationship set and the second mapping relationship set are not acquired or in a case where the first object distance is located outside each object distance range, horizontal pixel difference and vertical pixel difference between an infrared image collected by the infrared lens and a visible light image collected by the visible light lens are determined according to a geometric relationship among the target object, the infrared lens, and the visible light lens, and the focal length and the physical size in the case; The first coordinate mapping relationship is determined based on the horizontal pixel difference, the vertical pixel difference, intrinsic parameters of the infrared lens, distortion parameters of the visible light lens, and intrinsic parameters of the visible light lens. The second coordinate mapping relationship is determined based on the intrinsic parameters of the infrared lens and the distortion parameters of the infrared lens.

3. The method of claim 2, wherein, The geometric relationship among the target object, the infrared lens, and the visible light lens includes the first object distance, an optical center distance between the infrared lens and the visible light lens, a first included angle between an optical axis of the visible light lens and the lens connection line, and a second included angle between an optical axis of the infrared lens and the lens connection line; In a case where the first mapping relationship set and the second mapping relationship set are not acquired or in a case where the first object distance is located outside each object distance range, horizontal pixel difference and vertical pixel difference between an infrared image collected by the infrared lens and a visible light image collected by the visible light lens are determined according to a geometric relationship among the target object, the infrared lens, and the visible light lens, and the focal length and the physical size in the case, including: In a case where the first mapping relationship set and the second mapping relationship set are not acquired or in a case where the first object distance is located outside each object distance range, the horizontal pixel difference is determined by a first formula and the vertical pixel difference is determined by a second formula according to the first object distance, the optical center distance, the first included angle, the second included angle, the focal length, and the physical size; The first formula is: The second formula is: wherein the is the horizontal pixel difference, the is the vertical pixel difference, the is the first object distance, the is the focal length, the is the physical size of the single infrared pixel point or the single visible light pixel point in the horizontal direction, the is the physical size of the single infrared pixel point or the single visible light pixel point in the vertical direction, the is the optical center distance, the is the inclination angle of the target object relative to the optical center of the visible light lens, the is the inclination angle of the target object relative to the optical center of the infrared lens, the is the first included angle, the is the second included angle.

4. The method of claim 2, wherein, The first coordinate mapping relationship is determined based on the horizontal pixel difference, the vertical pixel difference, the intrinsic parameter of the infrared lens, the distortion parameter of the visible light lens, and the intrinsic parameter of the visible light lens, and the first coordinate mapping relationship includes: A third coordinate mapping relationship between the ideal infrared pixel coordinate system and a reference pixel coordinate system is determined according to the horizontal pixel difference and the vertical pixel difference, the reference pixel coordinate system being a pixel coordinate system of an undistorted infrared image collected by the infrared lens when the infrared lens is installed at a position of the visible light lens; A fourth coordinate mapping relationship between the reference pixel coordinate system and an ideal visible light image coordinate system is determined according to the intrinsic parameter of the infrared lens, the ideal visible light image coordinate system being an image coordinate system of an undistorted visible light image; A fifth coordinate mapping relationship between the ideal visible light image coordinate system and a real visible light image coordinate system is determined according to the distortion parameter of the visible light lens; A sixth coordinate mapping relationship between the real visible light image coordinate system and a real visible light pixel coordinate system is determined according to the intrinsic parameter of the visible light lens; The first coordinate mapping relationship is determined according to the third coordinate mapping relationship, the fourth coordinate mapping relationship, the fifth coordinate mapping relationship, and the sixth coordinate mapping relationship.

5. The method of claim 2, wherein, The second coordinate mapping relationship is determined based on the intrinsic parameter of the infrared lens and the distortion parameter of the infrared lens, and the second coordinate mapping relationship includes: A seventh coordinate mapping relationship between the ideal infrared pixel coordinate system and an ideal infrared image coordinate system is determined according to the intrinsic parameter of the infrared lens, the ideal infrared image coordinate system being an image coordinate system of an undistorted infrared image; An eighth coordinate mapping relationship between the ideal infrared image coordinate system and a real infrared image coordinate system is determined according to the distortion parameter of the infrared lens; A ninth coordinate mapping relationship between the real infrared image coordinate system and a real infrared pixel coordinate system is determined according to the intrinsic parameter of the infrared lens; The second coordinate mapping relationship is determined according to the seventh coordinate mapping relationship, the eighth coordinate mapping relationship, and the ninth coordinate mapping relationship.

6. The method of any one of claims 1-5, wherein, The first visible light image is subjected to coordinate transformation according to the first coordinate mapping relationship to obtain a target visible light image after registration, and the first infrared image is subjected to coordinate transformation according to the second coordinate mapping relationship to obtain a target infrared image after registration, and the first coordinate mapping relationship includes: In a case where the ideal pixel coordinate system is the ideal infrared pixel coordinate system, the first visible light image is converted from the real visible light pixel coordinate system to the ideal infrared pixel coordinate system according to the first coordinate mapping relationship to obtain the target visible light image, and the first infrared image is converted from the real infrared pixel coordinate system to the ideal infrared pixel coordinate system according to the second coordinate mapping relationship to obtain the target infrared image. In a case where the ideal pixel coordinate system is the ideal visible light pixel coordinate system, the first visible light image is converted from the real visible light pixel coordinate system to the ideal visible light pixel coordinate system according to the first coordinate mapping relationship, to obtain the target visible light image, and the first infrared image is converted from the real infrared pixel coordinate system to the ideal visible light pixel coordinate system according to the second coordinate mapping relationship, to obtain the target infrared image.

7. An apparatus for registering images, characterized by The device is applied to an electronic device, and the device comprises: An acquisition module is configured to acquire a first infrared image captured by an infrared lens and a first visible light image captured by a visible light lens, the infrared lens and the visible light lens being used in cooperation, and the first infrared image and the first visible light image both being images containing a target object; A determination module is configured to acquire a first object distance, the first object distance being a distance between the target object and a lens connection line, the lens connection line being a connection line between an optical center of the visible light lens and an optical center of the infrared lens; in a case where a preset first mapping relationship set and a preset second mapping relationship set are acquired, and the first object distance is located within a target object distance range in at least one object distance range, a first coordinate mapping relationship corresponding to the target object distance range in the first mapping relationship set is acquired, and a second coordinate mapping relationship corresponding to the target object distance range in the second mapping relationship set is acquired, the first mapping relationship set comprises at least one first coordinate mapping relationship, the first coordinate mapping relationship and the second coordinate mapping relationship corresponding to the target object distance range are the first coordinate mapping relationship and the second coordinate mapping relationship corresponding to the first object distance, the second mapping relationship set comprises at least one second coordinate mapping relationship, and the target object distance range is any one of the at least one object distance range, the at least one object distance range, the at least one first coordinate mapping relationship, and the at least one second coordinate mapping relationship correspond to each other, the first coordinate mapping relationship is used to describe a transformation relationship between a real visible light pixel coordinate system and an ideal pixel coordinate system, the second coordinate mapping relationship is used to describe a transformation relationship between a real infrared pixel coordinate system and the ideal pixel coordinate system, the ideal pixel coordinate system is an ideal infrared pixel coordinate system or an ideal visible light pixel coordinate system, the ideal infrared pixel coordinate system is a pixel coordinate system in which an undistorted infrared image is located, and the ideal visible light pixel coordinate system is a pixel coordinate system in which an undistorted visible light image is located; In a case where the first mapping relationship set and the second mapping relationship set are not acquired, or in a case where the first object distance is located outside each object distance range in the at least one object distance range, the first coordinate mapping relationship and the second coordinate mapping relationship are respectively determined based on a geometric relationship among the target object, the infrared lens, and the visible light lens, and lens parameters of the infrared lens and lens parameters of the visible light lens. A coordinate transformation module is configured to perform coordinate transformation on the first visible light image according to the first coordinate mapping relationship to obtain a registered target visible light image, and perform coordinate transformation on the first infrared image according to the second coordinate mapping relationship to obtain a registered target infrared image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor implements the method of any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium having stored thereon instructions, the computer-readable storage medium comprising: The instructions, when executed by the processor, implement the steps of the method of any one of claims 1 to 6.

Images