Method and device for registration and fusion of visible light images and infrared images

Abstract

The present invention provides a method and apparatus for registering and fusing visible light images and infrared images, which are applied to a device equipped with a visible light camera and an infrared camera, wherein the optical axis of the infrared camera is perpendicular to the plane of the device's front view. The method comprises: establishing a registration model for the coordinate positions of a visible light image of a target object captured by the visible light camera and an infrared image of the target object captured by the infrared camera based on the spatial relative positions of the visible light camera and the infrared camera, conversion parameters, and the horizontal distance of the target object from the visible light camera; and registering and fusing the visible light image and infrared image of the target object based on the registration model. In the present invention, the established registration model can be used to fuse the visible light image data and the infrared image data of a device equipped with dual cameras into a single image.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and device for registering and fusing visible light images and infrared images. Background Art

[0002] Currently, there are many infrared temperature measurement and facial detection and recognition devices. These devices are all fixed-mount devices, often integrating an infrared camera module with a visible light camera module. The relative positions and angles of the two camera modules are fixed. For example, a dual-light (infrared and visible or white light) module design ensures that the central optical axes of the two camera modules are parallel and fixed. The two camera modules have the same Z-axis position, the same and fixed vertical (Y-axis) height, and a very small relative horizontal (X-axis) distance. Alternatively, the two camera modules have a fixed vertical (Y-axis) height deviation and are identical and fixed horizontal (X-axis) positions. These fixed-mount infrared temperature measurement and facial detection and recognition devices share the following three characteristics: the central optical axes of the two camera modules are parallel, and the Z-axis zero coordinates of the two camera modules are the same, the Y-axis zero coordinates are the same, or the X-axis zero coordinates are the same.

[0003] These characteristics provide favorable conditions for the data fusion of two different camera images, but in product design and various practical solutions, the infrared camera and visible light camera modules are not designed in the same position (the zero-point coordinate positions of the X, Y, and Z axes are also different). For example, in wearable application scenarios, it is often necessary to adjust the angle of the visible light camera or infrared camera when worn by people of different heights and used in different scenarios. However, the two camera modules of infrared cameras and visible light cameras are often not implemented using binocular modules, which results in the central optical axes of the two camera modules being non-parallel (there is an angle), and the zero-point coordinate positions of the X, Y, and Z axes of the two camera modules are also different. The above are significantly different from the three characteristics of these fixed infrared temperature measurement and face detection and recognition devices. These differences bring great challenges to the data fusion of two different camera images of wearable devices.

[0004] Although existing equipment is equipped with infrared cameras and visible light cameras, it does not achieve dual-image registration and fusion. It can only perform data analysis and processing on one camera image. The equipment cannot automatically fuse the data of the visible light image and the infrared image into one image. Summary of the Invention

[0005] The present invention provides a method and device for registering and fusing visible light images and infrared images, so as to at least solve the problem in the related art that a device equipped with dual cameras cannot automatically fuse the data of the visible light image and the data of the infrared image into one image.

[0006] According to one aspect of the present invention, a method for registration and fusion of visible light images and infrared images is provided, which is applied to a device equipped with a visible light camera and an infrared camera, wherein the optical axis of the infrared camera is perpendicular to the plane of the device's front view, and the method comprises: establishing a registration model of the coordinate positions of the visible light image of the target object captured by the visible light camera and the infrared image of the target object captured by the infrared camera based on the spatial relative positions of the visible light camera and the infrared camera, conversion parameters, and the horizontal distance of the target object from the visible light camera; and registering and fusing the visible light image and the infrared image of the target object according to the registration model.

[0007] In an exemplary embodiment, the registration model is:

[0008]

[0009] Among them, A, B, C, and D are the first, second, third, and fourth conversion parameters respectively, and m, n, and d are the relative distances of the X, Y, and Z axes of the visible light camera and the infrared camera respectively. and γ are the horizontal angle and vertical angle of the optical axis of the visible light camera and the infrared camera respectively, L is the horizontal distance between the target object and the visible light camera, (x VR ,y VR ) is the pixel coordinate of the visible light image, (x IR ,y IR ) is the pixel coordinate of the infrared image.

[0010] In an exemplary embodiment, before registering and fusing the visible light image and the infrared image of the target object according to the registration model, the transverse angle and the longitudinal angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model are calibrated.

[0011] In an exemplary embodiment, calibrating the lateral angle and longitudinal angle of intersection between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model includes: selecting a first reference object whose horizontal distance from the visible light camera is a first set distance; simultaneously capturing an image of the first reference object through the visible light camera and the infrared camera, and obtaining the coordinates of the same position of the first reference object in the visible light image and the infrared image; calibrating the lateral angle and longitudinal angle of intersection between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model according to the first set distance and the coordinates.

[0012] In an exemplary embodiment, the lateral angle between the optical axes of the two cameras in the registration model can be calibrated by the following formula: And the longitudinal angle γ:

[0013]

[0014]

[0015] Among them, L C is the first set distance, (x VR-C ,y VR-C ) is the coordinate of the same position of the first reference object in the visible light image, and the coordinate of the same position of the first reference object in the infrared image (x IR-C ,y IR-C ).

[0016] In an exemplary embodiment, calibrating the lateral angle and longitudinal angle of intersection between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model includes: selecting a second reference object whose horizontal distance from the visible light camera is a second set distance; adjusting the optical axis of the visible light camera so that the same position of the second reference object is located at a specific position in the visible light image and the infrared image; and calibrating the lateral angle and longitudinal angle of intersection between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model based on the second set distance and the coordinate value of the specific position.

[0017] In an exemplary embodiment, the lateral angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model is calibrated by the following formula: And the longitudinal angle γ:

[0018]

[0019] Among them, L C is a second set distance, the coordinates of the specific position of the same position of the second reference object in the visible light image are (0, 0), and the coordinates of the specific position of the same position of the second reference object in the visible light image are (0, 0);

[0020] or,

[0021] Among them, L C The second set distance is the same as the coordinates of the specific position of the second reference object in the visible light image (200, 100), and the coordinates of the specific position of the second reference object in the visible light image is (0, 0).

[0022] In an exemplary embodiment, after calibrating the lateral angle and longitudinal angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model, the following steps are also included: substituting the lateral angle and longitudinal angle between the optical axis of the visible light camera and the optical axis of the infrared camera into the registration model to establish a height and width mapping model between the visible light image and the infrared image; based on multiple different horizontal distances between the target object and the visible light camera, establishing a mapping relationship between the height of the specified area of ​​the target object in multiple groups of visible light images and the horizontal distance of the target object from the visible light camera.

[0023] In an exemplary embodiment, registering and fusing the visible light image and the infrared image of the target object according to the registration model includes: obtaining the center position coordinate value of the designated area of ​​the target object in the visible light image, and the height and width of the designated area of ​​the target object in the visible light image, and finding the horizontal distance value corresponding to the target object and the visible light camera in the height and width mapping model according to the height and width of the designated area of ​​the target object in the visible light image; inputting the corresponding horizontal distance value and the center position coordinate value of the designated area of ​​the target object into the registration model, and calculating the center position coordinate value of the designated area of ​​the target object in the infrared image; inputting the height and width of the designated area of ​​the target object in the visible light image into the height and width mapping model, and calculating the height and width of the designated area of ​​the target object in the infrared image; and determining the designated area of ​​the target object in the infrared image according to the center position coordinate value of the designated area of ​​the target object in the infrared image and the height and width of the designated area of ​​the target object in the infrared image.

[0024] In an exemplary embodiment, after determining the designated area of ​​the target object in the infrared image, it also includes: obtaining the highest temperature value in the designated area of ​​the target object in the infrared image; and marking the temperature value at a designated position of the designated area of ​​the target object in the visible light image.

[0025] According to another aspect of the present invention, a device for registration and fusion of visible light images and infrared images is provided, which is located on a device equipped with a visible light camera and an infrared camera, wherein the optical axis of the infrared camera is perpendicular to the plane of the front view of the device. The device includes: a registration model establishment module, which is used to establish a registration model of the coordinate position of the visible light image of the target object captured by the visible light camera and the infrared image of the target object captured by the infrared camera based on the spatial relative position of the visible light camera and the infrared camera, the conversion parameters, and the horizontal distance of the target object from the visible light camera; and an image fusion module, which is used to register and fuse the visible light image and the infrared image of the target object according to the registration model.

[0026] According to another aspect of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0027] According to another aspect of the present invention, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above method embodiment when executing the computer program.

[0028] In the above-mentioned embodiment of the present invention, a registration model of the coordinate positions of the visible light image of the target object captured by the visible light camera and the infrared image of the target object captured by the infrared camera is established according to the spatial relative positions of the visible light camera and the infrared camera, and the horizontal distance of the target object from the visible light camera, and the visible light image and the infrared image of the target object are registered and fused according to the registration model, thereby solving the problem that a device equipped with dual cameras capable of both light and infrared cannot fuse the data of the visible light image and the data of the infrared image into one picture. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0030] Figure 1 is a flow chart of a method for registration and fusion of visible light images and infrared images according to an embodiment of the present invention;

[0031] Figure 2 is a structural block diagram of a device for registration and fusion of visible light images and infrared images according to an embodiment of the present invention;

[0032] Figure 3is a schematic diagram of the horizontal positions of a visible light camera and an infrared camera according to an embodiment of the present invention;

[0033] Figure 4 is a schematic diagram of the horizontal positions of a visible light camera and an infrared camera according to another embodiment of the present invention;

[0034] Figure 5 is a flow chart of a method for registering and fusing visible light images and infrared images according to embodiment 1 of the present invention;

[0035] Figure 6 4 is a flowchart of a method for registering and fusing visible light images and infrared images according to a second embodiment of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0037] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0038] This embodiment provides a method for registering and fusing visible light images and infrared images. The method is applied to a device equipped with a visible light camera and an infrared camera. For example, the device is a helmet equipped with both visible light and infrared cameras. One camera is mounted directly in front of the helmet, and the other is mounted on the side of the helmet via an external mount. When the user wears the helmet, both cameras face forward, capturing images close to the wearer's perspective. However, because the optical axes of the two cameras cannot be guaranteed to be parallel and they are spaced a certain distance apart, it is difficult for the images from the two cameras to "overlap," making it impossible to directly calibrate the temperature at that location on the visible light image. Therefore, a fusion operation is required.

[0039] In a feasible example, the optical axis of the infrared camera is perpendicular to the plane of the front view of the device. Figure 1 FIG. 1 is a flow chart of a method for registering and fusing visible light images and infrared images according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0040] Step S102, establishing a registration model for the coordinate positions of the visible light image of the target object captured by the visible light camera and the infrared image of the target object captured by the infrared camera based on the spatial relative positions of the visible light camera and the infrared camera, and the horizontal distance of the target object from the visible light camera;

[0041] Step S104 : registering and fusing the visible light image and the infrared image of the target object according to the registration model.

[0042] In step S102 of this embodiment, a registration model such as the following may be established:

[0043]

[0044] Among them, A, B, C, and D are the first, second, third, and fourth conversion parameters respectively, and m, n, and d are the relative distances of the X, Y, and Z axes of the visible light camera and the infrared camera respectively. and γ are the horizontal angle and vertical angle of the optical axis of the visible light camera and the infrared camera respectively, L is the horizontal distance between the target object and the visible light camera, (x VR ,y VR ) is the pixel coordinate of the visible light image, (x IR ,y IR ) is the pixel coordinate of the infrared image.

[0045] In this embodiment, this registration model can be used to rapidly register and fuse the coordinate positions of objects in two camera images. This registration model shows that the non-zero entries in the registration model matrix are not fixed values; rather, they are functions of the horizontal distance L of the target object from the visible light camera. This value changes as the horizontal distance L changes. Therefore, it is impossible to determine the non-zero entries in the registration model matrix by selecting images captured at a specific horizontal distance from the target object to the visible light camera.

[0046] Before applying the registration model of this embodiment, it is necessary to calibrate the unknown parameters in the registration model, that is, to calibrate the lateral angle between the two optical axes in the registration model of the visible light image and the infrared image. For example, in this embodiment, the lateral included angle and the longitudinal intersection angle between the optical axis of the visible light camera and the optical axis of the infrared camera can be calibrated in the following two ways:

[0047] The first calibration method includes the following steps:

[0048] 1) selecting a first reference object whose horizontal distance from the visible light camera is a first set distance;

[0049] 2) simultaneously capturing images of the first reference object using a visible light camera and an infrared camera, and obtaining coordinates of the same position of the first reference object in the visible light image and the infrared image respectively;

[0050] 3) Calibrate the lateral angle and longitudinal angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model according to the first set distance and the coordinates of the same position of the first reference object in the visible light image and the infrared image respectively.

[0051] For example, the lateral angle between the optical axes of the two cameras in the registration model can be calibrated by the following formula: And the longitudinal angle γ:

[0052]

[0053]

[0054] Among them, L C is the first set distance, (x VR-C ,y VR-C ) is the coordinate of the same position of the first reference object in the visible light image, and the coordinate of the same position of the first reference object in the infrared image (x IR-C ,y IR-C ).

[0055] The second calibration method includes the following steps:

[0056] 1) selecting a second reference object whose horizontal distance from the visible light camera is a second set distance;

[0057] 2) adjusting the optical axis of the visible light camera so that the same position of the second reference object is located at a specific position in the visible light image and the infrared image;

[0058] 3) Based on the second set distance and the coordinate value of the specific position, calibrate the horizontal angle and the vertical angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model.

[0059] For example, suppose the coordinate value x of a specific location is VR 、y VR , x IR 、y IR are all 0. According to the registration model of visible light image and infrared image, the lateral angle between the two optical axes can be calculated. The longitudinal intersection angles γ of the two optical axes are:

[0060]

[0061] For example: Assume that the coordinate value x of a specific location IR 、y IR are all 0, x VR is 200, y VR = 100, the lateral angle between the two optical axes can be calculated based on the registration model of visible light image and infrared image The longitudinal intersection angles γ of the two optical axes are:

[0062]

[0063] In this embodiment, after calibrating the transverse angle and longitudinal angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model, the following steps may also be included: substituting the transverse angle and longitudinal angle between the optical axis of the visible light camera and the optical axis of the infrared camera into the registration model to establish a height and width mapping model between the visible light image and the infrared image; based on multiple different horizontal distances between the target object and the visible light camera, establishing a mapping relationship between the height of the specified area of ​​the target object in multiple groups of visible light images and the horizontal distance of the target object from the visible light camera.

[0064] For example, if the registration model of this embodiment is applied to a human body temperature measurement scenario, the following height and width mapping model of the visible light image and the infrared image is established:

[0065]

[0066] Among them, A, C, γ, n, and L are all known, and λ is a configuration parameter, with the range of λ being 0.1 < λ ≤ 1. In this embodiment, by adjusting the size of λ, interference from background temperature in areas other than the face can be avoided.

[0067] In this embodiment, step S104 may include: obtaining the center position coordinate value of the designated area of ​​the target object in the visible light image, and the height and width of the designated area of ​​the target object in the visible light image, and finding the horizontal distance value corresponding to the target object and the visible light camera in the height and width mapping model based on the height and width of the designated area of ​​the target object in the visible light image; inputting the corresponding horizontal distance value and the center position coordinate value of the designated area of ​​the target object into the registration model, and calculating the center position coordinate value of the designated area of ​​the target object in the infrared image; inputting the height and width of the designated area of ​​the target object in the visible light image into the height and width mapping model, and calculating the height and width of the designated area of ​​the target object in the infrared image; determining the designated area of ​​the target object in the infrared image based on the center position coordinate value of the designated area of ​​the target object in the infrared image, and the height and width of the designated area of ​​the target object in the infrared image.

[0068] In this embodiment, after determining the designated area of ​​the target object in the infrared image, it can also include: obtaining the highest temperature value in the designated area of ​​the target object in the infrared image; and marking the temperature value at a designated position of the designated area of ​​the target object in the visible light image.

[0069] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0070] This embodiment also provides a device for registering and fusing visible light images and infrared images. This device is used to implement the above-mentioned embodiments and preferred implementations, and details already described will not be repeated. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0071] Figure 2 is a structural block diagram of a device for registering and fusing visible light images and infrared images according to an embodiment of the present invention. The device is located on a device equipped with a visible light camera and an infrared camera, wherein the optical axis of the infrared camera is perpendicular to the plane of the front view of the device, as shown in FIG. Figure 2 As shown, the device includes a registration model building module 10 and an image fusion module 20.

[0072] The registration model establishment module 10 is used to establish a registration model of the coordinate position of the visible light image of the target object captured by the visible light camera and the infrared image of the target object captured by the infrared camera based on the spatial relative position of the visible light camera and the infrared camera, and the horizontal distance of the target object from the visible light camera.

[0073] The image fusion module 20 is configured to register and fuse the visible light image and the infrared image of the target object according to the registration model.

[0074] It should be noted that the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are respectively located in multiple processors.

[0075] In order to facilitate the understanding of the technical solution provided by the present invention, a detailed description will be given below in conjunction with specific scenario embodiments.

[0076] Example 1

[0077] This embodiment provides a method for registering and fusing visible light and infrared images, which is applied to a device equipped with a visible light camera and an infrared camera. Figure 3 and Figure 4 All of them are schematic diagrams of the horizontal positions of the visible light camera and the infrared camera on the device according to the embodiment of the present invention. Figure 3 It shows the device area 1, infrared camera 2, visible light camera 3, horizontal line 4 of the device front view plane, infrared camera optical axis 5, visible light camera optical axis 6, and the horizontal angle 7 between the two optical axes.

[0078] like Figure 3 As shown, the infrared camera optical axis 5 is perpendicular to the plane of the device front view, and the visible light camera 3 is located below the infrared camera 2, and its optical axis 6 is not perpendicular to the plane of the device front view, but intersects with the optical axis 5 of the infrared camera.

[0079] exist Figure 4 The figure shows the device area 1, infrared camera 2, visible light camera 3, the horizontal line 4 of the device front view plane, the infrared camera optical axis 5, the visible light camera optical axis 6, and the horizontal angle 7 between the two optical axes. Figure 4 As shown, the infrared camera optical axis 5 is perpendicular to the plane of the device front view, the visible light camera 3 is located below the infrared camera 2, and the visible light camera optical axis 6 is not perpendicular to the plane of the device front view, but intersects with the infrared camera optical axis 5.

[0080] like Figure 3 and Figure 4 As shown, in this embodiment, the optical axis of the infrared camera is perpendicular to the plane of the device's front view, the optical axis of the visible light camera may not be perpendicular to the plane of the device's front view, and the relative angle of the same object in two different camera images is 0 (that is, the position of the same object in the image has not rotated).

[0081] The following will describe this embodiment in detail in conjunction with the scenario of using a visible light camera and an infrared camera for temperature measurement. Of course, the technical solution provided by this embodiment can also be applied to other scenarios requiring image fusion. Figure 5 As shown, the registration and fusion of visible light and infrared images provided in this embodiment may include the following steps:

[0082] Step S501: establishing a registration model for the visible light image and the infrared image.

[0083] Specifically, in this step, the following registration model for the visible light image and the infrared image can be established based on the parameters of the relative distances m, d, and n of the X, Y, and Z axis spatial positions of the visible light and infrared cameras, the horizontal and vertical field of view angles, the display resolution, the transverse angle between the two optical axes (i.e., the angle between the ZOY planes of the two cameras), the longitudinal angle between the two optical axes (i.e., the angle between the ZOX planes of the two cameras), and the horizontal distance L of the target object from the visible light camera:

[0084]

[0085] Among them, A, B, C, and D are conversion parameters, and m, n, and d are the relative distances between the two cameras in the X, Y, and Z axis space, which are known constants. and γ are the horizontal angle and vertical angle of the optical axes of the two cameras, and the horizontal distance L of the target object from the visible light camera is the change. VR and y VR is the pixel coordinate value of the visible light image, x IR and y IR is the pixel coordinate value of the infrared image.

[0086] In this embodiment, the conversion parameters A, B, C, and D can be calculated from the visible light horizontal and vertical viewing angles, infrared horizontal and vertical viewing angles, and infrared and visible light display resolution parameters. For example,

[0087]

[0088] Among them, w VR is the horizontal display resolution of the visible light camera, h VR is the vertical display resolution of the visible light camera, w IR is the infrared camera horizontal display resolution, h IR is the vertical display resolution of the infrared camera, α is the horizontal field of view of the visible light camera, β is the vertical field of view of the visible light camera, θ is the horizontal field of view of the infrared camera, and φ is the vertical field of view of the infrared camera.

[0089] Step S502: calibrate the horizontal angle and vertical angle between the optical axes of the visible light camera and the infrared camera in the registration model. In this embodiment, there is no need to manually adjust the visible light and infrared cameras. The horizontal angle and vertical angle can be calibrated as follows:

[0090] 1)1) Select a horizontal distance L from the visible light camera C Target object. For example, LC The selection range can be [0.5m~7m];

[0091] 2) The visible light and infrared cameras simultaneously capture the image of the target object (the target object can be a regular cuboid, cube or a part of the human body, such as the head or glasses), and find the coordinates (x VR-C ,y VR-C ) and the coordinates in the infrared image (x IR-C ,y IR-C );

[0092] 3) According to the registration model of visible light image and infrared image, the lateral angle between the two optical axes can be calculated The longitudinal angle γ between the two optical axes is:

[0093]

[0094]

[0095] Step S503: Establish a height and width mapping model between the visible light image and the infrared image as follows:

[0096]

[0097] Among them, A, C, γ, n, and L are all known, and λ is a configuration parameter with a range of 0.1<λ≤1. By adjusting the size of λ, interference from background temperature outside the face can be completely avoided.

[0098] In this embodiment, the height and width mapping model established can further quickly integrate the size ratio relationship of the object. For example, for a given distance L where the center of the face image is located, the face image height H in the visible light image is VR-Face and width W VR-Face , H can be obtained based on the height and width mapping ratio model of the visible light image and the infrared image. ir_face and W ir_face , and further according to x IR 、y IR 、H ir_face 、W ir_face The collected temperature information corresponding to the face area of ​​the infrared image can be obtained.

[0099] Step S504: Create a mapping table of the ranges corresponding to different face heights in the visible light image and the horizontal distance L between the face and the visible light camera. For example, as shown in Table 1, H vr_face The range interval is divided into multiple intervals, among which H k+1 >H k >Hk-1 >......>H6>H5>H4>H3>H2>H1, L1>L2>L3>L4>......>L k-2 >L k-1 >L k .

[0100] Table 1

[0101] Hvr_face L <![CDATA[[H1,H2]]]> <![CDATA[L1]]> <![CDATA[(H2,H3]]]> <![CDATA[L2]]> <![CDATA[(H3,H4]]]> <![CDATA[L3]]> … … … … … … <![CDATA[(H k-2 ,H k-1 ]]]> <![CDATA[L k-2 ]]> <![CDATA[(H k-1 ,H k ]]]> <![CDATA[L k-1 ]]> <![CDATA[(H k ,H k+1 ]]]> <![CDATA[L k ]]>

[0102] Step S505: The face detection module outputs one or more face center position coordinates (x VR ,y VR ), and the height H of the face frame image vr_face , width W vr_face , find the corresponding distance L value for the height or width of each detected face image in the mapping model, and compare the corresponding distance L value and the face center position coordinate value (x VR ,y VR ) is input into the registration model of visible light image and infrared image, and the coordinate value of the center position of the face image corresponding to the infrared image (x IR ,y IR ) until the calculation is completed for the center position coordinates (x VR ,y VR ).

[0103] Step S506: Input the height and width of each detected face image into the height and width mapping model of the visible light image and the infrared image, and calculate the height H of the corresponding face image. ir_face and width W ir_face .

[0104] Step S507: The center position coordinates (x IR ,y IR ), the height H of the face image ir_face and width W ir_face Determine the corresponding infrared image area, take the highest temperature from the corresponding infrared image area and record it as the corresponding person's face temperature, and mark each corresponding person's temperature value around or within the face frame of each corresponding face visible light image.

[0105] Through the above steps of this embodiment, the image registration and fusion problems are solved when the central optical axes of the visible light camera and the infrared camera are not parallel (there is an angle between the two optical axes horizontally and vertically) and the positions of the two cameras in the X-axis, Y-axis, and Z-axis directions are different. It further solves the problem of interference with facial temperature detection caused by abnormal ambient temperature around the face in temperature measurement scenario applications.

[0106] When the method of this embodiment is applied to a temperature measurement scenario, when the location and area range data of multiple faces are detected in the visible light image, the face range corresponding to all detected faces in the infrared image can be quickly and accurately obtained, and the face temperature data of the corresponding face range can be further accurately obtained. This can completely solve the problem of interference with face temperature detection caused by abnormal temperature of the environment around the face, and greatly improve the efficiency and accuracy of face temperature detection.

[0107] Example 2

[0108] This embodiment provides another method for registering and fusing visible light and infrared images. This method can be applied to a device equipped with a visible light camera and an infrared camera. In this embodiment, the spatial position relationship between the visible light camera and the infrared camera can be seen in Figure 3 and Figure 4 .

[0109] like Figure 3 and Figure 4 As shown, in this embodiment, the optical axis of the infrared camera is perpendicular to the plane of the device's front view, the optical axis of the visible light camera may not be perpendicular to the plane of the device's front view, and the relative angle of the same object in two different camera images is 0 (that is, the position of the same object in the image has not rotated).

[0110] The following describes this embodiment in detail in conjunction with a scenario where a visible light camera and an infrared camera are used to measure the temperature of multiple people. Of course, the technical solution provided by this embodiment can also be applied to other scenarios requiring image fusion. Figure 6 As shown, the shooting scene is a helmet equipped with a visible light camera and an infrared camera shooting a face in front of the helmet. The registration and fusion of visible light and infrared images provided in this embodiment may include the following steps:

[0111] Step S601: establishing a registration model for the visible light image and the infrared image.

[0112] Specifically, in this step, the following registration model for the visible light image and the infrared image can be established based on the parameters such as the relative distances of the X, Y, and Z axes of the visible light and infrared cameras in space (m, d, and n), the horizontal and vertical field of view angles, the display resolution, the transverse angle between the two optical axes (i.e., the angle between the ZOY planes of the two cameras), the longitudinal angle between the two optical axes (i.e., the angle between the ZOX planes of the two cameras), and the horizontal distance L between the target object and the visible light camera:

[0113]

[0114] Among them, A, B, C, and D are conversion parameters, and m, n, and d are the relative distances between the two cameras in the X, Y, and Z axis space, which are known constants. and γ are the horizontal angle and vertical angle between the optical axis of the visible light camera and the optical axis of the infrared camera, and the horizontal distance L of the target object from the visible light camera is the change. VR and y VR is the pixel coordinate value of the visible light image, x IR and y IR is the pixel coordinate value of the infrared image.

[0115] In this embodiment, the conversion parameters A, B, C, and D can be calculated from the horizontal and vertical field angles of the visible light camera, the horizontal and vertical field angles of the infrared camera, and the display resolution size parameters of the infrared camera and the visible light camera. For example,

[0116]

[0117] Among them, w VR is the horizontal display resolution of the visible light camera, h VR is the vertical display resolution of the visible light camera, w IR is the infrared camera horizontal display resolution, h IR is the vertical display resolution of the infrared camera, α is the horizontal field of view of the visible light camera, β is the vertical field of view of the visible light camera, θ is the horizontal field of view of the infrared camera, and φ is the vertical field of view of the infrared camera.

[0118] Step S602: Calibrate the transverse angle and longitudinal angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model. In this embodiment, another method for calibrating the transverse angle and longitudinal angle in the registration model is provided, which specifically includes the following steps:

[0119] 1) Do not manually adjust the visible light and infrared cameras. Select a horizontal distance L from the visible light camera. C Object. For example, L C The selection range can be [0.3m~7m];

[0120] 2) Adjust the optical axis of the visible light camera so that the object or human body is located at the same specific position in the visible light image and the infrared image. The specific position is displayed with a marker in both the visible light image and the infrared image (such as a crosshair or other marker image), for example: x IR 、y IR are all 0, x VR is 200, y VR is 100;

[0121] 3) According to the registration model of visible light image and infrared image, the lateral angle between the two optical axes can be calculated The longitudinal intersection angles γ of the two optical axes are:

[0122]

[0123] The specific position of this embodiment can be flexibly selected. For example, in another embodiment, x VR 、y VR , x IR 、y IR are all 0. According to the registration model of visible light image and infrared image, the lateral angle between the two optical axes can be calculated. The longitudinal intersection angles γ of the two optical axes are:

[0124]

[0125] Step S603: Establish a height and width mapping model between the visible light image and the infrared image as follows:

[0126]

[0127] Among them, A, C, γ, n, and L are all known, and λ is a configuration parameter with a range of 0.1<λ≤1. By adjusting the size of λ, interference from background temperature outside the face can be avoided.

[0128] In this embodiment, the height and width mapping model established can further quickly integrate the size ratio relationship of the object. For example, for a given distance L where the center of the face image is located, the face image height H in the visible light image is VR-Face and width W VR-Face , H can be obtained based on the height and width mapping ratio model of the visible light image and the infrared image. ir_face and W ir_face , and further according to x IR 、y IR 、H ir_face and W ir_face The collected temperature information corresponding to the face area of ​​the infrared image can be obtained.

[0129] Step S604: Create a mapping table of the ranges corresponding to different face heights in the visible light image and the horizontal distance L between the face and the visible light camera. For example, as shown in Table 1, H vr_face The range interval is divided into multiple intervals, among which Hk+1>Hk>Hk-1>......>H6>H5>H4>H3>H2>H1, L1>L2>L3>L4>......>Lk-2>Lk-1>Lk.

[0130] Table 1

[0131] Hvr_face L [H1,H2] L1 (H2,H3] L2 (H3,H4] L3 … … … … … … (Hk-2,Hk-1] Lk-2 (Hk-1,Hk] Lk-1 (Hk,Hk+1] Lk

[0132] Step S605: The face detection module outputs one or more face center position coordinates (x VR ,y VR ), and the height H of the face frame image vr_face , width W vr_face , find the corresponding distance L value for the height or width of each detected face image in the mapping model, and compare the corresponding distance L value and the face center position coordinate value (x VR ,y VR ) is input into the registration model of visible light image and infrared image to calculate the coordinate value of the center position of the face image corresponding to the infrared image (x IR ,y IR ) until the calculation is completed for the center position coordinates (x VR ,y VR ).

[0133] Step S606: Input the height and width of each detected face image into the height and width mapping model of the visible light image and the infrared image, and calculate the height H of the corresponding face image. ir_face and width W ir_face .

[0134] Step S607: The center position coordinates (x IR ,y IR ), the height H of the face image ir_face and width W ir_face Determine the corresponding infrared image area, take the highest temperature from the corresponding infrared image area and record it as the corresponding person's face temperature, and mark each corresponding person's temperature value around or within the face frame of each corresponding face visible light image.

[0135] Through the above steps of this embodiment, the image registration and fusion problems are solved when the central optical axes of the visible light camera and the infrared camera are not parallel (there is an angle between the two optical axes horizontally and vertically) and the positions of the two cameras in the X-axis, Y-axis, and Z-axis directions are different. It further solves the problem of interference with facial temperature detection caused by abnormal ambient temperature around the face in temperature measurement scenario applications.

[0136] The technical solution provided by this embodiment can solve the image registration and fusion problem in wearable devices where the central optical axes of the visible light camera and the infrared camera are not parallel, and the positions of the two cameras in the X-axis, Y-axis, and Z-axis directions are different. It can also solve the problem of interference with facial temperature detection caused by abnormal ambient temperature around the face in temperature measurement scenarios. When the technical solution provided by this embodiment is applied to the visible light image when multiple faces are detected and the area range data is displayed, the facial range corresponding to all detected faces in the infrared image can be quickly and accurately obtained, and the facial temperature data of the corresponding facial range can be further accurately obtained. This can completely solve the problem of interference with facial temperature detection caused by abnormal ambient temperature around the face, greatly improving the efficiency and accuracy of facial temperature detection. In addition, the calibration process of the image registration and fusion model provided by this embodiment is simple and fast, avoiding the consumption of a large amount of complex computing resources. Compared with other image fusion algorithms that require more computing resources to support, the image registration and fusion model provided by this embodiment requires fewer computing resources and is more efficient.

[0137] An embodiment of the present invention further provides a storage medium. Optionally, in this embodiment, the storage medium may be configured to store program codes for executing the following steps:

[0138] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0139] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, centralized on a single computing device, or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. In some cases, the steps shown or described can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0140] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for registering and fusing visible light images and infrared images, applied to a device equipped with a visible light camera and an infrared camera, wherein: The optical axis of the infrared camera is perpendicular to the plane of the front view of the device, and is characterized by comprising: Establishing a registration model for the coordinate positions of a visible light image of the target object captured by the visible light camera and an infrared image of the target object captured by the infrared camera according to the spatial relative position of the visible light camera and the infrared camera, the conversion parameter, and the horizontal distance of the target object from the visible light camera; Registering and fusing the visible light image and the infrared image of the target object according to the registration model; Wherein, the registration model is: Among them, A, B, C, and D are the first, second, third, and fourth conversion parameters respectively, and m, n, and d are the relative distances of the X, Y, and Z axes of the visible light camera and the infrared camera respectively. and γ are the horizontal angle and vertical angle of the optical axis of the visible light camera and the infrared camera respectively, L is the horizontal distance between the target object and the visible light camera, (x VR ,y VR ) is the pixel coordinate of the visible light image, (x IR ,y IR ) is the pixel coordinate of the infrared image.

2. The method according to claim 1, characterized in that The transverse angle and longitudinal angle of the optical axes of the visible light camera and the infrared camera in the registration model are obtained by the following steps: Selecting a first reference object whose horizontal distance from the visible light camera is a first set distance; Simultaneously capturing images of the first reference object using a visible light camera and an infrared camera, and obtaining coordinates of the same position of the first reference object in the visible light image and the infrared image respectively; The horizontal angle and the vertical angle between the optical axis of the visible light camera and the optical axis of the infrared camera are calibrated in the registration model according to the first set distance and the coordinates of the same position of the first reference object in the visible light image and the infrared image respectively.

3. The method according to claim 2, characterized in that , the lateral angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model The longitudinal intersection angle γ is: Among them, L C is the first set distance, (x VR-C ,y VR-C ) is the coordinate of the same position of the first reference object in the visible light image, and the coordinate of the same position of the first reference object in the infrared image (x IR-C ,y IR-C ).

4. The method according to claim 1, wherein The transverse angle and longitudinal angle of the optical axes of the visible light camera and the infrared camera in the registration model are obtained by the following steps: Selecting a second reference object whose horizontal distance from the visible light camera is a second set distance; adjusting the optical axis of the visible light camera so that the same position of the second reference object is located at a specific position in the visible light image and the infrared image; According to the second set distance and the coordinate value of the specific position, the transverse angle and the longitudinal angle between the optical axis of the visible light camera and the optical axis of the infrared camera are calibrated in the registration model.

5. The method according to claim 4, characterized in that , the lateral angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model The longitudinal intersection angle γ is: Among them, L C is a second set distance, the coordinates of the specific position of the same position of the second reference object in the visible light image are (0, 0), and the coordinates of the specific position of the same position of the second reference object in the infrared image are (0, 0); or, Among them, L C For the second set distance, the coordinates of the specific position of the same position of the second reference object in the visible light image are (200, 100), and the coordinates of the specific position of the same position of the second reference object in the infrared image are (0, 0).

6. The method according to claim 2 or 4, characterized in that After calibrating the transverse angle and the longitudinal angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model, the method further includes: Substituting the transverse angle and the longitudinal angle between the optical axis of the visible light camera and the optical axis of the infrared camera into the registration model to establish a height and width mapping model between the visible light image and the infrared image; Based on a plurality of different horizontal distances between the target object and the visible light camera, a mapping relationship between the height of a designated area of ​​the target object in a plurality of sets of visible light images and the horizontal distance of the target object from the visible light camera is established.

7. The method according to claim 6, characterized in that Registering and fusing the visible light image and the infrared image of the target object according to the registration model includes: Obtaining the center coordinates of the designated area of ​​the target object in the visible light image, as well as the height and width of the designated area of ​​the target object in the visible light image, and finding the horizontal distance value corresponding to the target object and the visible light camera in the height and width mapping model based on the height and width of the designated area of ​​the target object in the visible light image; Inputting the corresponding horizontal distance value and the center position coordinate value of the designated area of ​​the target object into the registration model, and calculating the center position coordinate value of the designated area of ​​the target object in the infrared image; Inputting the height and width of the designated area of ​​the target object in the visible light image into the height and width mapping model to calculate the height and width of the designated area of ​​the target object in the infrared image; The designated area of ​​the target object in the infrared image is determined according to the center position coordinates of the designated area of ​​the target object in the infrared image, and the height and width of the designated area of ​​the target object in the infrared image.

8. The method according to claim 7, characterized in that After determining the designated area of ​​the target object in the infrared image, the method further includes: Obtaining a maximum temperature value in a specified area of ​​the target object in the infrared image; The temperature value is marked at a specified location of a specified area of ​​the target object in the visible light image.

9. A device for registering and fusing visible light images and infrared images, located on a device equipped with a visible light camera and an infrared camera, wherein: The optical axis of the infrared camera is perpendicular to the plane of the front view of the device, and is characterized by comprising: a registration model establishment module for establishing a registration model of the coordinate positions of the visible light image of the target object captured by the visible light camera and the infrared image of the target object captured by the infrared camera based on the spatial relative position of the visible light camera and the infrared camera, the conversion parameter, and the horizontal distance of the target object from the visible light camera; An image fusion module, configured to perform registration and fusion of the visible light image and the infrared image of the target object according to the registration model; Wherein, the registration model is: Among them, A, B, C, and D are the first, second, third, and fourth conversion parameters respectively, and m, n, and d are the relative distances of the X, Y, and Z axes of the visible light camera and the infrared camera respectively. and γ are the horizontal angle and vertical angle of the optical axis of the visible light camera and the infrared camera respectively, L is the horizontal distance between the target object and the visible light camera, (x VR ,y VR ) is the pixel coordinate of the visible light image, (x IR ,y IR ) is the pixel coordinate of the infrared image.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images