Method and apparatus for registration fusion of visible light image and infrared image

Abstract

The application provides a method and device for registration and fusion of visible light images and infrared images, which are applied to a device provided with a visible light camera and an infrared camera, wherein the optical axis of the visible light camera is perpendicular to the front view plane of the device, the method comprising: establishing a registration model of coordinate positions of a visible light image of a target object collected by the visible light camera and an infrared image of the target object collected by the infrared camera according to the spatial relative positions of the visible light camera and the infrared camera, conversion parameters and a horizontal distance of the target object from the visible light camera; and registration and fusion of the visible light image and the infrared image of the target object according to the registration model. In the application, the data of the visible light image and the data of the infrared image in the device provided with the double cameras can be fused into one image through the established registration model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a method and device for registration fusion of visible light images and infrared images. BACKGROUND

[0002] There are many infrared temperature measurement and face detection and recognition devices at present, and these temperature measurement and face detection and recognition devices are fixed installation devices, wherein the infrared camera module and the visible light camera module are integrated together, and the relative positions and relative angles of the two camera modules are fixed. For example, a dual light (infrared and visible light or white light) module design is adopted, and since the dual light module design is adopted, the central optical axes of the two camera modules are parallel and fixed and do not change, the positions of the Z axes of the centers of the two camera modules are the same, the vertical (Y axis direction) heights of the centers of the two camera modules are the same and fixed, the relative positions in the horizontal (X axis direction) are fixed and the distance is very small, or the relative position deviations of the vertical (Y axis direction) heights of the centers of the two camera modules are fixed, the positions in the horizontal (X axis direction) are the same and fixed. These fixed infrared temperature measurement and face detection and recognition devices have the following three characteristics: the central optical axes of the two camera modules are parallel, the zero point coordinate positions of the Z axes of the two camera modules are the same, and the zero point coordinate positions of the Y axes or the zero point coordinate positions of the X axes are the same.

[0003] These characteristics provide favorable conditions for data fusion of pictures of two different cameras, but in product design and various actual solutions, the infrared camera and the visible light camera are not designed at the same position (the zero point coordinate positions of the X, Y and Z axes are not the same). For example, in a wearable application scenario, the angle of the visible light camera or the infrared camera needs to be adjusted when different height people wear and different scene applications, and the infrared camera and the visible light camera are not often implemented by using a binocular module, so that the central optical axes of the two camera modules are not parallel (an angle exists), and the zero point coordinate positions of the X, Y and Z axes of the space positions of the two camera modules are not the same. The above differences are obviously different from the three characteristics of the fixed infrared temperature measurement and face detection and recognition devices, and these differences bring great challenges to data fusion of pictures of two different cameras of a wearable device.

[0004] Although existing devices are equipped with infrared cameras and visible light cameras, registration fusion of dual images is not achieved, and only data analysis and processing in one camera picture can be achieved, and the device cannot automatically fuse the data of the visible light picture and the data of the infrared picture into one image. SUMMARY

[0005] The application provides a registration fusion method and device for visible light images and infrared images to at least solve the problem that a device equipped with double cameras cannot automatically fuse data of a visible light picture and data of an infrared picture into one picture in the prior art.

[0006] According to an aspect of the application, a registration fusion method for visible light images and infrared images is provided, which is applied to a device configured with a visible light camera and an infrared camera, for example, the device is a helmet equipped with a visible light camera and an infrared camera, one of the cameras is installed at the front of the helmet, and the other camera is installed on one side of the helmet in an external hanging mode, when a user wears the helmet, the two cameras shoot forward to obtain a view angle close to the wearer. However, the optical axes of the two cameras cannot be guaranteed to be parallel, and the two cameras have a certain distance in space, which makes the images of the two cameras difficult to "overlap", and the temperature at the position cannot be directly calibrated on the visible light image. Therefore, the images need to be fused.

[0007] In an implementable example, the optical axis of the visible light camera is perpendicular to the front view plane of the device, and the method comprises: establishing a registration model of coordinate positions of a visible light image of a target object collected by the visible light camera and an infrared image of the target object collected by the infrared camera according to the spatial relative positions of the visible light camera and the infrared camera, conversion parameters, and a horizontal distance of the target object from the visible light camera; and performing registration fusion on the visible light image and the infrared image of the target object according to the registration model.

[0008] In an exemplary embodiment, the registration model is:

[0009]

[0010] wherein A, B, C and D are respectively first, second, third and fourth conversion parameters, m, n and d are respectively spatial position relative distances of X, Y and Z axes of the visible light camera and the infrared camera, and γ are respectively a transverse included angle and a longitudinal intersection angle of the optical axes of the visible light camera and the infrared camera, L is a horizontal distance of a target object from the visible light camera, (x VR , y VR ) are pixel coordinates of the visible light image, and (x IR , y IR ) are pixel coordinates of the infrared image.

[0011] In an exemplary embodiment, before performing registration fusion on the visible light image and the infrared image of the target object according to the registration model, the transverse included angle and the longitudinal intersection angle of the optical axes of the visible light camera and the infrared camera in the registration model are calibrated.

[0012] In an example embodiment, calibrating the transverse included angle and the longitudinal intersection angle of the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model comprises: selecting a first reference object with a horizontal distance of a first set distance from the visible light camera; simultaneously collecting images of the first reference object by the visible light camera and the infrared camera, and measuring the transverse length and the longitudinal length of the same position of the first reference object in the visible light image and the infrared image respectively; and calibrating the transverse included angle and the longitudinal intersection angle of the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model according to the first set distance and the transverse length and the longitudinal length.

[0013] In an example embodiment, the transverse included angle and the longitudinal intersection angle of the optical axes of the two cameras in the registration model can be calibrated by the following formulae:

[0014]

[0015] wherein L C is the first set distance, L VR and W VR are the transverse length and the longitudinal length of the same position of the first reference object in the visible light image respectively, L IR and W IR are the transverse length and the longitudinal length of the same position of the first reference object in the infrared image respectively.

[0016] In an example embodiment, calibrating the transverse included angle and the longitudinal intersection angle of the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model comprises: selecting a second reference object with a horizontal distance of a second set distance from the visible light camera; adjusting the optical axis of the visible light camera so that the same position of the second reference object is located at a specific position in the visible light image and the infrared image; and calibrating the transverse included angle and the longitudinal intersection angle of the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model according to the second set distance and the coordinate values of the specific position.

[0017] In an example embodiment, the transverse included angle and the longitudinal intersection angle of the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model are calibrated by the following formulae:

[0018]

[0019] wherein L C is the second set distance, the coordinate of the specific position of the same position of the second reference object in the visible light image is (0, 0), and the coordinate of the specific position of the same position of the second reference object in the infrared image is (0, 0).​​

[0020] or,

[0021] wherein, L C is a second set distance, the same position of the second reference object is located at a coordinate of (200, 100) of the specific position in the visible light image, and the same position of the second reference object is located at a coordinate of (0, 0) of the specific position in the visible light image.

[0022] In an exemplary embodiment, after calibrating the transverse included angle and the longitudinal intersection angle of the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model, the following steps are further included: substituting the transverse included angle and the longitudinal intersection angle of the optical axis of the visible light camera and the optical axis of the infrared camera into the registration model to establish the height and width mapping model of the visible light image and the infrared image; and based on a plurality of different horizontal distances of the target object from the visible light camera, establishing a mapping relationship between the height of the specified region of the target object in the visible light image and the horizontal distance of the target object from the visible light camera.

[0023] In an exemplary embodiment, the registration and fusion of the visible light image and the infrared image of the target object according to the registration model includes: obtaining the coordinate value of the center position of the specified region of the target object in the visible light image, and the height and width of the specified region of the target object in the visible light image, and finding the corresponding horizontal distance value of the target object from the visible light camera in the height and width mapping model according to the height and width of the specified region of the target object in the visible light image; inputting the corresponding horizontal distance value and the coordinate value of the center position of the specified region of the target object into the registration model to calculate and obtain the coordinate value of the center position of the specified region of the target object in the infrared image; inputting the height and width of the specified region of the target object in the visible light image into the height and width mapping model to calculate and obtain the height and width of the specified region of the target object in the infrared image; and determining the specified region of the target object in the infrared image according to the coordinate value of the center position of the specified region of the target object in the infrared image, and the height and width of the specified region of the target object in the infrared image.

[0024] In an exemplary embodiment, after determining the specified region of the target object in the infrared image, the following steps are further included: obtaining the highest temperature value in the specified region of the target object in the infrared image; and marking the temperature value at a specified position of the specified region of the target object in the visible light image.

[0025] According to another aspect of the present application, there is provided a device for registering and fusing a visible light image and an infrared image, which is located on a device configured with a visible light camera and an infrared camera, wherein an optical axis of the visible light camera is perpendicular to an orthographic plane of the device, the device comprising: a registration model establishing module configured to establish a registration model of coordinate positions of a visible light image of a target object captured by the visible light camera and an infrared image of the target object captured by the infrared camera according to a spatial relative position of the visible light camera and the infrared camera, a conversion parameter, and a horizontal distance of the target object from the visible light camera; and an image fusing module configured to register and fuse the visible light image and the infrared image of the target object according to the registration model.

[0026] According to still another aspect of the present application, there is also provided a computer readable storage medium having a computer program stored therein, wherein the computer program, when executed by a processor, implements the steps of the above method embodiments.

[0027] According to still another aspect of the present application, there is also provided an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method embodiments when executing the computer program.

[0028] In the above embodiments of the present application, the registration model of coordinate positions of a visible light image of a target object captured by a visible light camera and an infrared image of the target object captured by an infrared camera is established according to a spatial relative position of the visible light camera and the infrared camera, and a horizontal distance of the target object from the visible light camera, and the visible light image and the infrared image of the target object are registered and fused according to the registration model, thereby solving the problem that a device configured with a visible light camera and an infrared camera cannot fuse data of a visible light image and data of an infrared image into one picture. BRIEF DESCRIPTION OF DRAWINGS

[0029] The accompanying drawings, which are included to provide a further understanding of the present application and are incorporated in and constitute a part of this application, illustrate embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0030] Figure 1 is a flow chart of a method for registering and fusing a visible light image and an infrared image according to an embodiment of the present application;

[0031] Figure 2 is a structural block diagram of a device for registering and fusing a visible light image and an infrared image according to an embodiment of the present application;

[0032] Figure 3is a visible light camera and infrared camera horizontal position schematic diagram according to an embodiment of the present application;

[0033] Figure 4 is a visible light camera and infrared camera horizontal position schematic diagram according to another embodiment of the present application;

[0034] Figure 5 is a flow chart of a visible light image and infrared image registration fusion method according to an embodiment of the present application;

[0035] Figure 6 is a flow chart of a visible light image and infrared image registration fusion method according to another embodiment of the present application. DETAILED DESCRIPTION

[0036] Hereinafter, the present application will be described in detail with reference to the accompanying drawings and embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0037] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0038] In the present embodiment, a visible light image and infrared image registration fusion method is provided, applied to a device configured with a visible light camera and an infrared camera, wherein the optical axis of the visible light camera is perpendicular to the front view plane of the device. Figure 1 is a flow chart of a visible light image and infrared image registration fusion method according to an embodiment of the present application, as shown in Figure 1 The flow chart includes the following steps:

[0039] Step S102, establishing a registration model of coordinate positions of a visible light image of a target object collected by the visible light camera and an infrared image of the target object collected by the infrared camera according to the spatial relative position of the visible light camera and the infrared camera and the horizontal distance of the target object from the visible light camera;

[0040] Step S104, registration fusion of the visible light image and the infrared image of the target object according to the registration model.

[0041] In step S102 of the present embodiment, for example, a registration model such as the following can be established:

[0042]

[0043] Wherein, A, B, C, D are respectively the first, second, third and fourth conversion parameters, m, n, d are respectively the relative distance of the X, Y, Z axes of the visible light camera and the infrared camera, and γ are respectively the transverse angle and the longitudinal intersection angle of the optical axes of the visible light camera and the infrared camera, L is the horizontal distance of the target object from the visible light camera, (x VR , y VR ) are the pixel coordinates of the visible light image, (x IR , y IR ) are the pixel coordinates of the infrared image.

[0044] In the present embodiment, the registration model can be used to quickly register and fuse the coordinate positions of the objects in the two camera pictures. From the registration model, it can be known that the non-zero items in the registration model matrix are not fixed values, but functions of the horizontal distance L of the target object from the visible light camera, and thus vary with the horizontal distance L of the target object from the visible light camera. Therefore, the non-zero items in the registration model matrix cannot be determined by selecting a picture image collected in a scene in which a certain target object is at a certain horizontal distance from the visible light camera.

[0045] Before the application of the registration model in the present embodiment, the unknown parameters in the registration model need to be calibrated, i.e., the transverse angle and the longitudinal intersection angle γ of the two optical axes in the registration model of the visible light image and the infrared image. For example, in the present embodiment, the transverse angle and the longitudinal intersection angle of the optical axes of the visible light camera and the infrared camera can be calibrated in the following two ways:

[0046] The first calibration method comprises the following steps:

[0047] 1) selecting a first reference object at a first set distance from the visible light camera;

[0048] 2) simultaneously collecting images of the first reference object by the visible light camera and the infrared camera, and measuring the transverse length and the longitudinal length of the same position of the first reference object in the visible light image and the infrared image respectively;

[0049] 3) calibrating the transverse angle and the longitudinal intersection angle of the optical axes of the visible light camera and the infrared camera in the registration model according to the first set distance, and the transverse length and the longitudinal length of the same position of the first reference object in the visible light image and the infrared image.

[0050] For example, the transverse angle and the longitudinal intersection angle γ of the optical axes of the two cameras in the registration model can be calibrated by the following formula:

[0051]

[0052] wherein L C is a first set distance, L VR and W VR are the horizontal length and vertical length of the same position of the first reference object in the visible light image, respectively, L IR and W IR are the horizontal length and vertical length of the same position of the first reference object in the infrared image, respectively.

[0053] The second calibration method comprises the following steps:

[0054] 1) selecting a second reference object with a horizontal distance from the visible light camera of a second set distance;

[0055] 2) adjusting the optical axis of the visible light camera so that the same position of the second reference object is located at a specific position in the visible light image and the infrared image;

[0056] 3) according to the second set distance and the coordinate values of the specific position, calibrating the horizontal included angle and the vertical intersection angle of the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model.

[0057] For example, assuming that the coordinate values of the specific position x VR , y VR , x IR , y IR are all 0, according to the registration model of the visible light image and the infrared image, the horizontal included angle and the vertical intersection angle γ of the two optical axes are respectively:

[0058]

[0059] For example: assuming that the coordinate values of the specific position x IR , y IR are all 0, x VR is 200, and y VR is 100, according to the registration model of the visible light image and the infrared image, the horizontal included angle and the vertical intersection angle γ of the two optical axes are respectively:

[0060]

[0061] In the embodiment, after calibrating the transverse included angle and the longitudinal intersection angle of the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model, the following steps can be further included: substituting the transverse included angle and the longitudinal intersection angle of the optical axis of the visible light camera and the optical axis of the infrared camera into the registration model to establish the height and width mapping model of the visible light image and the infrared image; and establishing a mapping relationship between the height of the specified region of the target object in the visible light image and the horizontal distance of the target object from the visible light camera based on different horizontal distances of the target object from the visible light camera.

[0062] For example, if the registration model of the embodiment is applied to a human body temperature measurement scene, the following height and width mapping model of the visible light image and the infrared image is established:

[0063]

[0064] wherein A, C, γ, n, L are known, and λ is a configuration parameter, and λ ranges from 0.1 < λ < 1. In the embodiment, by adjusting the size of λ, interference from the background temperature in the region other than the face can be avoided.

[0065] In the embodiment, step S104 can include: obtaining the center position coordinate value of the specified region of the target object in the visible light image, and the height and width of the specified region of the target object in the visible light image, and finding the corresponding horizontal distance value of the target object from the visible light camera in the height and width mapping model according to the height and width of the specified region of the target object in the visible light image; inputting the corresponding horizontal distance value and the center position coordinate value of the specified region of the target object into the registration model to calculate and obtain the center position coordinate value of the specified region of the target object in the infrared image; inputting the height and width of the specified region of the target object in the visible light image into the height and width mapping model to calculate and obtain the height and width of the specified region of the target object in the infrared image; and determining the specified region of the target object in the infrared image according to the center position coordinate of the specified region of the target object in the infrared image and the height and width of the specified region of the target object in the infrared image.

[0066] In the embodiment, after determining the specified region of the target object in the infrared image, the following steps can be further included: obtaining the highest temperature value in the specified region of the target object in the infrared image; and marking the temperature value at a specified position of the specified region of the target object in the visible light image.

[0067] Those skilled in the art can clearly understand that the method according to the above-mentioned embodiments can be realized by means of software and necessary general hardware platform, of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server or network device) execute the method described in each embodiment of the present application.

[0068] In this embodiment, a visible light image and infrared image registration fusion device is also provided, which is used to realize the above-mentioned embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that realizes a predetermined function. Although the device described in the following embodiments is preferably realized in software, hardware, or a combination of software and hardware is also possible and is contemplated.

[0069] Figure 2 is a structural block diagram of a visible light image and infrared image registration fusion device according to an embodiment of the present application, which is located on a device configured with a visible light camera and an infrared camera, wherein the optical axis of the visible light camera is perpendicular to the front view plane of the device, as shown in Figure 2 The device includes a registration model establishing module 10 and an image fusion module 20.

[0070] The registration model establishing module 10 is used to establish a registration model of the coordinate positions of the visible light image of the target object collected by the visible light camera and the infrared image of the target object collected by the infrared camera according to the spatial relative position of the visible light camera and the infrared camera and the horizontal distance of the target object from the visible light camera.

[0071] The image fusion module 20 is used to registration fuse the visible light image and the infrared image of the target object according to the registration model.

[0072] It should be noted that each of the above-mentioned modules can be realized by software or hardware, and for the latter, the following implementation methods can be used, but are not limited thereto: all the above-mentioned modules are located in the same processor; or the above-mentioned modules are located in multiple processors respectively.

[0073] In order to facilitate the understanding of the technical solutions provided by the present application, the following will be described in detail in conjunction with specific scene embodiments.

[0074] Embodiment 1

[0075] The embodiment provides a registration fusion method of visible light and infrared images. The method is applied to a device provided with a visible light camera and an infrared camera. Figure 3 and Figure 4 are schematic diagrams of horizontal positions of the visible light camera and the infrared camera on the device according to the embodiment of the present application, wherein in Figure 3 the device area 1, the infrared camera 2, the visible light camera 3, the horizontal line of the front view plane of the device 4, the optical axis of the infrared camera 5, the optical axis of the visible light camera 6 and the transverse included angle of the two optical axes 7 are shown.

[0076] As shown in Figure 3 , the optical axis 6 of the visible light camera is perpendicular to the front view plane of the device, the infrared camera 2 is located above the visible light camera 3, and the optical axis of the infrared camera is not perpendicular to the front view plane of the device but intersects with the optical axis 6 of the visible light camera.

[0077] In Figure 4 , the device area 1, the infrared camera 2, the visible light camera 3, the horizontal line of the front view plane of the device 4, the optical axis of the infrared camera 5, the optical axis of the visible light camera 6 and the transverse included angle of the two optical axes 7 are shown. As shown in Figure 4 , the optical axis 6 of the visible light camera is perpendicular to the front view plane of the device, the infrared camera 2 is located above the visible light camera 3, and the optical axis 5 of the infrared camera is not perpendicular to the front view plane of the device but intersects with the optical axis 6 of the visible light camera.

[0078] As shown in Figure 3 and Figure 4 , in the embodiment, the optical axis of the visible light camera is perpendicular to the front view plane of the device, and the optical axis of the infrared camera can not be perpendicular to the front view plane of the device, and the relative angle of the same object in the two different camera pictures is 0 (that is, the position of the same object in the picture does not rotate).

[0079] The embodiment will be described in detail in combination with a scene of temperature measurement by using the visible light camera and the infrared camera. Of course, the technical solution provided by the embodiment can also be applied to other scenes requiring image fusion. As shown in Figure 5 , the registration fusion of the visible light and the infrared image provided by the embodiment can include the following steps:

[0080] Step S501, a registration model of the visible light image and the infrared image is established.

[0081] Specifically, in this step, the following registration model of the visible light image and the infrared image can be established according to the relative distances m, d, n of the X, Y, Z axes of the two cameras, the horizontal and vertical field angles, the display resolution, the horizontal angle between the two optical axes (i.e., the angle between the ZOY planes of the two cameras), the longitudinal intersection angle between the two optical axes (i.e., the angle between the ZOX planes of the two cameras), and the horizontal distance L of the target object from the visible light camera:

[0082]

[0083] where A, B, C, and D are conversion parameters, m, n, and d are the relative distances of the X, Y, and Z axes of the two cameras, and are known constants, and the horizontal angle between the optical axes of the two cameras and the longitudinal intersection angle are variable. VR and y VR are the pixel coordinate values of the visible light image, x IR and y IR are the pixel coordinate values of the infrared image.

[0084] In this embodiment, the conversion parameters A, B, C, and D can be calculated from the horizontal and vertical field angles of the visible light, the horizontal and vertical field angles of the infrared, and the display resolution parameters of the infrared and visible light. For example,

[0085]

[0086] where w VR is the horizontal display resolution of the visible light camera, h VR is the vertical display resolution of the visible light camera, w IR is the horizontal display resolution of the infrared camera, h IR is the vertical display resolution of the infrared camera, a is the horizontal field angle of the visible light camera, b is the vertical field angle of the visible light camera, q is the horizontal field angle of the infrared camera, and f is the vertical field angle of the infrared camera.

[0087] In step S502, the horizontal angle and the longitudinal intersection angle between the optical axes of the visible light camera and the infrared camera in the registration model are calibrated. In this embodiment, the horizontal angle and the longitudinal intersection angle can be calibrated without manually adjusting the visible light and infrared cameras as follows:

[0088] 1) Select a target object at a horizontal distance L C from the visible light camera. For example, the selection range of L C may be [0.3 m-7 m];

[0089] 2) Visible light and infrared camera simultaneously collect the image of the target object (the target object can be a regular cuboid, a cube or a part of the human body, such as the head of a person), and measure the transverse length and longitudinal length of the same position of the object. The measurement method can be automatically calculated and measured by the built-in software of the device, or manually calculated and measured by the collected image. The collected image can be obtained through the server software connected to the device. The transverse length of the visible light image of the object is L VR , the longitudinal length is W VR , the transverse length of the infrared image of the object is L IR , and the longitudinal length is W IR .

[0090] 3) According to the registration model of visible light image and infrared image, the transverse angle of the two optical axes and the longitudinal intersection angle γ of the two optical axes can be calculated:

[0091]

[0092] Step S503, establish a height and width mapping model of visible light image and infrared image as follows:

[0093]

[0094] Where A, C, γ, n, L are known, and λ is a configuration parameter, λ is in the range of 0.1<λ≤1. In this embodiment, by adjusting the size of λ, the interference of the background temperature of the area other than the face in the temperature measurement application can be avoided.

[0095] In this embodiment, the height and width mapping model can further quickly fuse the size ratio of the object. For example, for the face picture height H VR-Face and width W VR-Face of the face picture center in the visible light image of a given distance L, the height H ir_face and width W ir_face can be obtained according to the height and width mapping ratio model of the visible light image and the infrared image, and further according to x IR , y IR , H ir_face , W ir_face , the corresponding collected temperature information of the face area of the infrared image can be obtained.

[0096] Step S504, establish a mapping table of the range interval corresponding to different face heights in the visible light image and the horizontal distance L between the face and the visible light camera, for example, as shown in Table 1, the range of H vr_face can be divided into multiple intervals, wherein H k+1 >H k >Hk-1 >......>H6>H5>H4>H3>H2>H1, L1>L2>L3>L4>......>L k-2 >L k-1 >L k .

[0097] Table 1

[0098] Hvr_face L [[H1,H2]] ​ (H2, H3) ​ (H3, H4) ​ … … … … … … (H k-2 ,H k-1 ]]]> [[ L k-2 ]]> (H k-1 ,H k ]]]> [[ L k-1 ]]> (H k ,H k+1 ]]]> [[ L k ]]>

[0099] Step S505, the face detection module outputs one or more face center position coordinate values (x VR , y VR ) in the visible light image and the height H vr_face and width W vr_face of the face frame picture. The height or width of each detected face picture is used to find the corresponding distance L value in the mapping model. The corresponding distance L value and the face center position coordinate values (x VR , y VR ) are input into the registration model of the visible light image and the infrared image to calculate the face picture center position coordinate values (x IR , y IR ) of the corresponding infrared image. This process is repeated until the face picture center position coordinate values (x VR , y VR ) of the corresponding infrared image of the multiple detected faces are calculated.

[0100] Step S506, the height and width of each detected face picture are input into the height and width mapping model of the visible light image and the infrared image to calculate the corresponding face picture height H ir_face and width W ir_face .

[0101] Step S507, according to the face picture center position coordinate (x IR , y IR ), the height H ir_face and the width W ir_face of the corresponding infrared image of each detected face, the corresponding infrared image region is determined. The highest temperature record in the corresponding infrared image region is taken as the face temperature of the corresponding person, and each corresponding person temperature value is marked around or in the face frame of the corresponding face visible light picture.

[0102] By the above steps of the embodiment, the image registration fusion problem in the case that the center optical axes of the visible light camera and the infrared camera are not parallel (there is an intersection angle between the two optical axes in the transverse direction, and there is also an intersection angle between the two optical axes in the longitudinal direction), and the positions of the X-axis, Y-axis and Z-axis of the two cameras are not the same, is solved, and the problem that the abnormal ambient temperature around the face causes interference with the face temperature detection in the temperature measurement scene application is further solved.

[0103] When the method of the embodiment is applied to the temperature measurement scene, when the visible light picture detects the position and area range data of multiple faces, the face range corresponding to all the detected faces in the infrared picture can be quickly and accurately obtained, the face temperature data of the corresponding face range is further accurately obtained, the interference problem of face temperature detection caused by the abnormal ambient temperature around the face can be completely solved, and the detection efficiency and accuracy of the face temperature are greatly improved.

[0104] Embodiment 2

[0105] Another visible light and infrared image registration fusion method is provided in the embodiment, which can be applied to a device provided with a visible light camera and an infrared camera. In the embodiment, the spatial position relationship of the visible light camera and the infrared camera can be referred to as Figure 3 and Figure 4 .

[0106] As shown in Figure 3 and Figure 4 , in the embodiment, the optical axis of the visible light camera is perpendicular to the front view plane of the device, the optical axis of the infrared camera can not be perpendicular to the front view plane of the device, and the relative angle of the same object in the two different camera pictures is 0 (that is, the position of the same object in the picture does not rotate).

[0107] The embodiment will be described in detail below in combination with the scene of measuring the temperature of multiple people by using a visible light camera and an infrared camera. Of course, the technical solution provided in the embodiment can also be applied to other scenes requiring image fusion. As shown in Figure 6 , the shooting scene is that a helmet provided with a visible light camera and an infrared camera shoots the face in front of the helmet, and the registration and fusion of the visible light and infrared images provided in the embodiment can include the following steps:

[0108] Step S601, a registration model of the visible light image and the infrared image is established.

[0109] Specifically, in this step, the following registration model of the visible light image and the infrared image can be established according to the relative distances of the X, Y, Z axis spatial positions of the two cameras, the horizontal and vertical field angles, the display resolution, the transverse angle of the two optical axes (i.e., the angle of the ZOY plane of the two cameras), the longitudinal intersection angle of the two optical axes (i.e., the angle of the ZOX plane of the two cameras), and the horizontal distance L of the target object from the visible light camera:

[0110]

[0111] wherein A, B, C, and D are conversion parameters, m, n, and d are the relative distances of the X, Y, Z axis spatial positions of the two cameras, and are known constants, and γ is the transverse angle and the longitudinal intersection angle of the optical axes of the two cameras, and the horizontal distance L of the target object from the visible light camera is a variable. VR and y VR are the pixel coordinate values of the visible light image, x IR and y IR are the pixel coordinate values of the infrared image.

[0112] In this embodiment, the conversion parameters A, B, C, and D can be calculated from the horizontal and vertical field angles of the visible light, the horizontal and vertical field angles of the infrared, and the display resolution parameters of the infrared and the visible light. For example,

[0113]

[0114] wherein w VR is the horizontal display resolution of the visible light camera, h VR is the vertical display resolution of the visible light camera, w IR is the horizontal display resolution of the infrared camera, h IR is the vertical display resolution of the infrared camera, α is the horizontal field angle of the visible light camera, β is the vertical field angle of the visible light camera, θ is the horizontal field angle of the infrared camera, and φ is the vertical field angle of the infrared camera.

[0115] In step S602, the transverse angle and the longitudinal intersection angle between the optical axes of the visible light camera and the infrared camera in the registration model are calibrated. In this embodiment, another calibration method of the transverse angle and the longitudinal intersection angle in the registration model is provided, which specifically can include the following steps:

[0116] 1) Selecting an object at a horizontal distance L C from the visible light camera. For example, the selection range of L C may be [0.3m-7m];

[0117] 2) Adjust the optical axis of the visible light camera so that the same position of the object or human body is located in a specific position of the visible light picture and the infrared picture, and the specific position is displayed by a marker (such as a cross line or other marker image) in both the visible light picture and the infrared picture, for example: x IR , y IR are all 0, and x VR is 200 and y VR is 100.

[0118] 3) According to the registration model of the visible light image and the infrared image, the horizontal included angle θ of the two optical axes and the longitudinal intersection angle γ of the two optical axes are calculated as follows:

[0119]

[0120] The specific position of the embodiment can be flexibly selected. For example, in another embodiment, x VR , y VR , x IR , y IR are all 0, and according to the registration model of the visible light image and the infrared image, the horizontal included angle θ of the two optical axes and the longitudinal intersection angle γ of the two optical axes are calculated as follows:

[0121]

[0122] Step S603: A height and width mapping model of the visible light image and the infrared image is established as follows:

[0123]

[0124] Wherein, A, C, γ, n, L are known, and λ is a configuration parameter, and λ ranges from 0.1 to 1 (0.1<λ≤1). By adjusting the size of λ, interference caused by the background temperature of the area other than the face can be avoided.

[0125] In the embodiment, the size proportion relationship of the object can be further quickly fused by the established height and width mapping model. For example, for the face picture height H VR-Face and width W VR-Face of the face picture center in the visible light image at a given distance L, H ir_face and W ir_face can be obtained according to the height and width mapping proportion model of the visible light image and the infrared image, and the corresponding collected temperature information of the face area of the infrared image can be further obtained according to x IR , y IR , H ir_face and W ir_face .

[0126] ​​Step S604, a mapping table of the range interval corresponding to the different face heights in the visible light image and the horizontal distance L between the face and the visible light camera is established, for example, as shown in Table 1, H vr_face The range interval is divided into multiple intervals, where Hk+1>Hk>Hk-1>......>H6>H5>H4>H3>H2>H1, L1>L2>L3>L4>......>Lk-2>Lk-1>Lk.

[0127] Table 1

[0128] Hvr_face L [H1,H2] L1 (H2,H3] L2 (H3,H4] L3 … … … … … … (Hk-2,Hk-1] Lk-2 (Hk-1,Hk] Lk-1 (Hk,Hk+1] Lk

[0129] Step S605, the face detection module outputs one or more face center position coordinate values (x VR , y VR ) in the visible light image, and the height H vr_face and width W vr_face of the face frame picture, finds the corresponding distance L value of each detected face picture height or width in the mapping model, and inputs the corresponding distance L value and the face center position coordinate values (x VR , y VR ) into the registration model of the visible light image and the infrared image to calculate the face picture center position coordinate values (x IR , y IR ) of the corresponding infrared image, until the face picture center position coordinate values (x VR , y VR ) of the corresponding infrared image of the multiple persons detected are calculated.

[0130] Step S606, the height and width of each detected face picture are input into the height and width mapping model of the visible light image and the infrared image to calculate the corresponding face picture height H ir_face and width W ir_face .

[0131] Step S607, according to the face picture center position coordinate (x IR , y IR ) of each detected face corresponding to the infrared image, the height H ir_face and width W ir_face of the face picture, the corresponding infrared image area is determined, the highest temperature record in the corresponding infrared image area is taken as the face temperature of the corresponding person, and each corresponding person temperature value is marked around or in the face frame of each corresponding face visible light picture.

[0132] The above steps in this embodiment solve the image registration and fusion problem when the central optical axes of the visible light camera and the infrared camera are not parallel (the two optical axes intersect at an angle in the horizontal direction and also in the vertical direction) and the positions of the two cameras in the X-axis, Y-axis and Z-axis directions are not the same. Furthermore, it solves the problem of interference with face temperature detection caused by abnormal ambient temperature around the face in temperature measurement scenarios.

[0133] The technical solution provided in this embodiment can solve the image registration and fusion problem in wearable devices when the central optical axes of visible light cameras and infrared cameras are not parallel, and the positions of the two cameras in the X, Y, and Z axes are not the same. It also addresses the interference of abnormal ambient temperature around the face in temperature measurement applications. When the technical solution provided in this embodiment is applied to the visible light image to detect the location and area of ​​multiple faces, it quickly and accurately obtains the corresponding face area in the infrared image for all detected faces, and further accurately obtains the face temperature data for the corresponding face area. This completely solves the problem of interference caused by abnormal ambient temperature around the face, greatly improving the efficiency and accuracy of face temperature detection. Furthermore, the calibration process of the image registration and fusion model provided in this embodiment is simple and quick, avoiding the consumption of a large amount of complex computational resources. Compared with other image fusion algorithms that require more computational resources, the image registration and fusion model provided in this embodiment requires fewer computational resources and is more efficient.

[0134] Embodiments of the present invention also provide a storage medium. Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:

[0135] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0136] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0137] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application. The present application can have various changes and modifications for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of registering and fusing a visible light image and an infrared image, applied to a device configured with a visible light camera and an infrared camera, wherein, The optical axis of the visible light camera is perpendicular to the front view plane of the device, and the device comprises: According to the spatial relative position of the visible light camera and the infrared camera, the conversion parameter, and the horizontal distance of the target object from the visible light camera, a registration model of the coordinate positions of the visible light image of the target object collected by the visible light camera and the infrared image of the target object collected by the infrared camera is established; According to the registration model, the visible light image and the infrared image of the target object are registered and fused; The registration model is: ， Wherein, A, B, C, D are the first, second, third and fourth conversion parameters, m, n, d are the relative distance of the X, Y, Z axis of the visible light camera and the infrared camera, And The transverse angle and the longitudinal intersection angle of the optical axis of the visible light camera and the infrared camera, respectively, L is the horizontal distance of the target object from the visible light camera, , ) is the pixel coordinate of the visible light image, , ) is the pixel coordinate of the infrared image.

2. The method of claim 1, wherein, In the registration model, the transverse included angle and the longitudinal intersection angle of the optical axes of the visible light camera and the infrared camera are obtained by the following steps: A first reference object with a horizontal distance from the visible light camera of a first set distance is selected; The images of the first reference object are simultaneously collected by the visible light camera and the infrared camera, and the transverse length and the longitudinal length of the same position of the first reference object in the visible light image and the infrared image are measured, respectively; According to the first set distance and the transverse length and the longitudinal length of the same position of the first reference object in the visible light image and the infrared image, the transverse included angle and the longitudinal intersection angle of the optical axes of the visible light camera and the infrared camera in the registration model are calibrated.

3. The method of claim 2, wherein , the transverse included angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model is calibrated by the following formula and the longitudinal included angle : ， ， wherein, is a first set distance, and are a lateral length and a longitudinal length, respectively, of the same position of the first reference object in the visible light image, and are a lateral length and a longitudinal length, respectively, of the same position of the first reference object in the infrared image.

4. The method of claim 2, wherein, The transverse included angle and the longitudinal intersection angle of the optical axes of the visible light camera and the infrared camera in the registration model are obtained by the following steps: A second reference object with a horizontal distance from the visible light camera of a second set distance is selected; The optical axis of the visible light camera is adjusted so that the same position of the second reference object is located at a specific position in the visible light image and the infrared image; According to the second set distance and the coordinate value of the specific position, the transverse included angle and the longitudinal intersection angle of the optical axes of the visible light camera and the infrared camera in the registration model are calibrated.

5. The method of claim 4, wherein , the transverse included angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model is calibrated by the following formula and the longitudinal included angle : ， ， wherein, is a second set distance, the same position of the second reference object is located at a specific position in the visible light image with a coordinate of (0, 0), and the same position of the second reference object is located at a specific position in the infrared light image with a coordinate of (0, 0); or, , , wherein, is a second set distance, the coordinate of the same position of the second reference object in the specific position in the visible light image is (200, 100), and the coordinate of the same position of the second reference object in the specific position in the infrared light image is (0, 0).

6. The method according to claim 2 or 4, characterized in that, After calibrating the transverse included angle and the longitudinal intersection angle of the optical axes of the visible light camera and the infrared camera in the registration model, the following steps are further included: The transverse included angle and the longitudinal intersection angle of the optical axes of the visible light camera and the infrared camera are substituted into the registration model to establish a height and width mapping model of the visible light image and the infrared image; Based on the different horizontal distances of the target object from the visible light camera, a mapping relationship between the height of the specified region of the target object in multiple visible light images and the horizontal distance of the target object from the visible light camera is established.

7. The method of claim 6, wherein, According to the registration model, the visible light image and the infrared image of the target object are registered and fused, including: The center position coordinate value of the specified region of the target object in the visible light image, and the height and width of the specified region of the target object in the visible light image are obtained, and according to the height and width of the specified region of the target object in the visible light image, the corresponding horizontal distance value of the target object from the visible light camera is found in the height and width mapping model; input the corresponding horizontal distance value and the center position coordinate value of the specified region of the target object into the registration model, and obtain the center position coordinate value of the specified region of the target object in the infrared image by calculation; input the height and width of the specified region of the target object in the visible light image into the height and width mapping model, and obtain the height and width of the specified region of the target object in the infrared image by calculation; determine the specified region of the target object in the infrared image according to the center position coordinate of the specified region of the target object in the infrared image and the height and width of the specified region of the target object in the infrared image.

8. The method of claim 7, wherein, After determining the specified region of the target object in the infrared image, the method further comprises: obtaining the highest temperature value in the specified region of the target object in the infrared image; marking the temperature value at the specified position of the specified region of the target object in the visible light image.

9. An apparatus for registered fusion of a visible light image and an infrared image, on a device configured with a visible light camera and an infrared camera, wherein, The optical axis of the visible light camera is perpendicular to the front view plane of the device, and the method comprises: a registration model establishing module, configured to establish a registration model of the coordinate positions of the visible light image of the target object collected by the visible light camera and the infrared image of the target object collected by the infrared camera according to the spatial relative positions of the visible light camera and the infrared camera, the conversion parameters, and the horizontal distance of the target object from the visible light camera; the registration model is: ， Wherein, A, B, C, D are the first, second, third and fourth conversion parameters respectively, m, n, d are the relative distances of the X, Y, Z axes of the visible light camera and the infrared camera respectively, And Respectively, the transverse angle and the longitudinal intersection angle of the optical axis of the visible light camera and the infrared camera, L is the horizontal distance of the target object from the visible light camera, , ) is the pixel coordinate of the visible light image, , ) is the pixel coordinate of the infrared image; an image fusion module, configured to register and fuse the visible light image and the infrared image of the target object according to the registration model.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, wherein the computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 8.

11. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 8.

Images