Image registration method and apparatus

Abstract

The application discloses an image registration method and device, relates to the field of image processing, and realizes pixel-level registration of images with different imaging light rays. The method comprises the following steps: acquiring a first image to be registered, a reference second image with the smallest time interval from the first image, and one or more non-reference second images, the imaging light rays of the first image and the reference second image are different, and the imaging light rays of the reference second image and the non-reference second images are the same; acquiring a first optical flow of each non-reference second image and the reference second image; acquiring an optical flow scaling factor corresponding to each non-reference second image; determining a second optical flow of the first image to be registered and the reference second image according to the first optical flow and the optical flow scaling factor; and registering the first image to be registered and the reference second image according to the second optical flow, so as to obtain a registered first image.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image registration method and apparatus. Background Technology

[0002] With the development of image processing technology, people have increasingly higher requirements for image quality. The application of fusing multiple images to obtain high-quality images is becoming more and more widespread.

[0003] For example, in low-light scenes for traffic enforcement, supplementary lighting equipment is used to obtain high-quality images. Alternatively, a visible light image reflecting the true color information of an object can be acquired without supplementary lighting, while an infrared image can be obtained using infrared supplementary lighting. In this approach, the time-division acquired visible light and infrared images can be fused together to obtain a high-quality image, preserving the true color information of the object as much as possible.

[0004] For example, in surveillance scenarios, a true image of the interior of an object with a reflective surface can be obtained by using enhanced lighting, while a true image of the surface of the object can be obtained by using weakened lighting or no lighting. In this scheme, the time-division acquired enhanced and weakened lighting images can be fused together to obtain a high-quality image, preserving as much of the true image of the object's interior and exterior as possible.

[0005] However, during image acquisition, object movement or camera movement is unavoidable. Pixels indicating the same object in different images will experience pixel displacement. Therefore, image registration is necessary before image fusion. However, due to differences in acquisition parameters (such as different spectra and imaging lighting), significant texture and color differences can occur between images. Consequently, existing registration algorithms based on feature similarity estimation have low image registration accuracy. Summary of the Invention

[0006] This application provides an image registration method and apparatus to improve the registration accuracy of images with different imaging lighting.

[0007] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0008] In a first aspect, an image registration method is provided, which may include: acquiring a first image to be registered, a reference second image with the smallest temporal interval to the first image, and one or more non-reference second images, wherein the imaging rays of the first image and the reference second image are different, and the imaging rays of the reference second image and the non-reference second images are the same; acquiring a first optical flow between each non-reference second image and the reference second image; acquiring an optical flow scaling factor corresponding to each non-reference second image, wherein the optical flow scaling factor is used to indicate the proportional relationship between the first optical flow and the second optical flow; the second optical flow is the optical flow between the first image to be registered and the reference second image; determining the second optical flow between the first image to be registered and the reference second image based on the first optical flow and the optical flow scaling factor; and registering the first image to be registered and the reference second image based on the second optical flow to obtain a registered first image.

[0009] The image registration method provided in this application first obtains the optical flow between images with the same imaging rays, converts it to obtain the optical flow between images with different imaging rays, and then registers the images with different imaging rays according to the obtained optical flow. Since the accuracy of obtaining the optical flow between images with the same imaging rays is high, the accuracy of the converted optical flow between images with different imaging rays can be improved, thereby improving the registration accuracy of images with different imaging rays.

[0010] The same object described in this application can be a fixed point on a person or object, or a pixel of a fixed point on a person or object in an image. Alternatively, the same object described in this application can also be the entire person or object in an image. When the same object includes multiple pixels, the coordinate difference of the pixels of the same object can be the sum of the coordinate differences of all pixels of the object in two frames of images, or a simple average or weighted average of the coordinate differences of all pixels in two frames of images.

[0011] The imaging principle of an imaging device (equipment) is to collect light through a lens, allowing the light to enter the imaging sensor. The imaging sensor then forms an image based on the incoming light. Imaging light refers to the light that enters the imaging sensor during image acquisition. Different imaging lights can be understood as either different types of imaging light or different intensities of imaging light. In practical applications, the light entering the lens can either directly enter the imaging sensor, or it can be processed, with the processed light then entering the imaging sensor.

[0012] In one possible implementation, the first image to be registered and the reference second image have different illumination types, while the reference second image and the non-reference second image have the same illumination type, so as to improve the registration accuracy of images with different illumination types.

[0013] Supplemental lighting refers to adding light to the environment of the imaging device (equipment). Different types of supplemental lighting can refer to adding light of different wavelengths or different frequency bands in the optical spectrum to the environment of the imaging device (equipment), resulting in different types of imaging light entering the imaging sensor.

[0014] In another possible implementation, the types of light entering the imaging sensor are different when acquiring the first image and the reference second image, and the types of light entering the imaging sensor are the same when acquiring the reference second image and the non-reference second image, so as to improve the registration accuracy of images with different types of light entering the imaging sensor.

[0015] This allows for the filtering of light entering the lens, resulting in different types of light entering the imaging sensor. For example, a filter (e.g., a filter plate) can be placed before the imaging sensor to allow only infrared light to pass through, thus capturing an infrared image. Similarly, a filter (e.g., a filter plate) can be placed before the imaging sensor to allow only visible light to pass through, thus capturing a visible light image.

[0016] In another possible implementation, the light intensity entering the imaging sensor is different when acquiring the first image and the reference second image, and the light intensity entering the imaging sensor is the same when acquiring the reference second image and the non-reference second image, so as to improve the registration accuracy of images with different light intensities entering the imaging sensor.

[0017] This technology can attenuate or amplify the light entering the lens, resulting in light of different intensities entering the imaging sensor, and thus capturing images with different imaging light intensities.

[0018] In another possible implementation, the first image to be registered is an infrared image acquired by an infrared light-in-the-image sensor, and the reference second image and the non-reference second image are visible light images acquired by a visible light-in-the-image sensor, so as to improve the registration accuracy of the infrared image to the visible light image.

[0019] In another possible implementation, the first image to be registered is a visible light image acquired by the imaging sensor, and the reference second image and the non-reference second image are infrared images acquired by the imaging sensor, so as to improve the registration accuracy of the visible light image to the infrared image.

[0020] Infrared light entering the imaging sensor can be achieved by supplementing the ambient light of the imaging device (equipment) with infrared light, or by placing a filter device (e.g., a filter) that only allows infrared light to pass through before the imaging sensor, ensuring that the light entering the imaging sensor is infrared light. Similarly, visible light entering the imaging sensor can be achieved by supplementing the ambient light of the imaging device (equipment) with visible light or not supplementing it, or by placing a filter device (e.g., a filter) that only allows visible light to pass through before the imaging sensor, ensuring that the light entering the imaging sensor is visible light.

[0021] In another possible implementation, the supplementary light intensities of the first image to be registered and the reference second image are different, while the supplementary light intensities of the reference second image and the non-reference second image are the same. The supplementary light intensity determines the intensity of the imaging light, thereby improving the registration accuracy of images with different supplementary light intensities.

[0022] Different supplementary light intensities can refer to supplementing the environment of the shooting device (equipment) with light of different intensities, so that the imaging light intensity entering the imaging sensor is different.

[0023] In another possible implementation, obtaining the optical flow scaling factor corresponding to each non-reference second image includes: obtaining a first acquisition time interval between the reference second image and the first image to be registered, and a second acquisition time interval between the reference second image and each non-reference second image; determining the optical flow scaling factor based on the first acquisition time interval, the second acquisition time interval, and the parameters of the acquisition device. Since the ratio of image acquisition time intervals can accurately reflect pixel displacement, the optical flow scaling factor can be determined based on the acquisition time intervals to obtain a highly accurate optical flow scaling factor.

[0024] In another possible implementation, the optical flow scaling factor is determined based on the first acquisition time interval, the second acquisition time interval, and the parameters of the acquisition device. This includes: the optical flow scaling factor sf and the first acquisition time interval Δt, the second acquisition time interval ΔT, and the parameters of the acquisition device satisfying the following relationship 1: D = hsinα + dcosα. Where h is the height of the acquisition device, d is the capture distance of the acquisition device, α is the pitch angle between the optical axis of the acquisition device and the horizontal plane, and v is the moving speed of the object captured by the acquisition device.

[0025] In Equation 1, if the first image to be registered is acquired after the reference second image in terms of timing, the denominator is negative; if the first image to be registered is acquired before the reference second image, the denominator is positive. Similarly, if the non-reference second image is acquired after the reference second image in terms of timing, the numerator is negative; if the non-reference second image is acquired before the reference second image, the numerator is positive.

[0026] In another possible implementation, obtaining the optical flow scaling factor corresponding to each non-reference second image includes: obtaining one or more matching pixel pairs in the reference second image and the first image to be registered, wherein a matching pixel pair includes pixels in the reference second image and the first image to be registered whose feature similarity is greater than or equal to a second threshold; and determining the optical flow scaling factor based on the pixel displacement of the first matching pixel pair and the first optical flow. The first matching pixel pair is one of the one or more matching pixel pairs. Obtaining the optical flow scaling factor by actually acquiring the pixel displacement of pixels with high feature value is highly accurate because the pixel displacement obtained through feature value matching best matches the actual pixel displacement.

[0027] In another possible implementation, the first matching pixel pair is the matching pixel pair with the highest matching degree among the one or more matching pixel pairs mentioned above. Since the pixel pair with the highest matching degree in the two images is used for the same object, selecting the pixel displacement of the pixel pair with the highest matching degree represents the actual pixel displacement of each pixel in the two images, resulting in the highest accuracy and the smallest error.

[0028] In another possible implementation, the determination of the optical flow scaling factor corresponding to the non-reference second image based on the pixel displacement of the first matching pixel pair and the first optical flow can be specifically implemented as follows: based on the first optical flow, determine the corresponding pixel P1 in the non-reference second image for pixel P0 in the reference second image included in the first matching pixel pair; based on the two-dimensional vector of P0 and P1... And the pixel P in the first image to be registered, which is included in the first matching pixel pair P0. t Two-dimensional vector Determine the optical flow scaling factor sf. Wherein, the optical flow scaling factor sf and... Satisfy the following relationship:

[0029] In another possible implementation, the second optical flow is determined based on the first optical flow and the optical flow scaling factor, including: if the first optical flow and the second optical flow have the same direction, the second optical flow is determined to be the product of the optical flow scaling factor and the first optical flow; if the first optical flow and the second optical flow have different directions, the second optical flow is determined to be the negative of the product of the optical flow scaling factor and the first optical flow.

[0030] Among them, the second optical flow f vn With optical flow scaling factor sf and first optical flow f vv It satisfies the following expression:

[0031]

[0032] In another possible implementation, there are multiple non-reference second images. Correspondingly, determining the second optical flow based on the first optical flow and the optical flow scaling factor corresponding to each non-reference second image includes: obtaining multiple second optical flows based on the first optical flow of each non-reference second image and the reference second image, and the optical flow scaling factor corresponding to each non-reference second image. Registering the first image to be registered with the reference second image based on the second optical flows to obtain a registered first image includes: determining a target second optical flow based on the multiple second optical flows, and registering the first image to be registered with the reference second image according to the target second optical flow to obtain a registered first image. By selecting multiple non-reference second images to obtain multiple second optical flows, the accuracy of the second optical flow is improved, thereby improving the registration accuracy between the first image and the reference second image.

[0033] In another possible implementation, the target second optical flow is determined based on multiple second optical flows. Specifically, this can be achieved by taking a simple average, weighted average, maximum value, or minimum value of the multiple second optical flows as the target second optical flow.

[0034] In another possible implementation, each non-reference second image is temporally adjacent to the reference second image. In temporally adjacent images, the pixel displacement of the same object is minimized, which improves the accuracy of the second optical flow and thus enhances the registration accuracy between the first image and the reference second image.

[0035] In another possible implementation, the second optical flow is a two-dimensional vector diagram (u,v). The u component of the second optical flow indicates the displacement of each pixel in the image along the X-axis, and the v component indicates the displacement of each pixel in the image along the Y-axis. Based on the second optical flow, the first image to be registered is registered with the reference second image to obtain the registered first image, including: the registered first image. With the first image I to be registered t The second optical flow (u,v) satisfies the following relationship: Where x, y are the coordinates of a pixel in the image. u(x,y) is the value of the pixel at coordinates (x,y) in the first image in the u component of the second optical flow, and v(x,y) is the value of the pixel at coordinates (x,y) in the first image in the v component of the second optical flow.

[0036] It should be noted that the directional parameters described in this application (such as optical flow, displacement, etc.) all refer to these parameters in the same direction. However, this application does not limit the specific direction, as long as all directional parameters have the same direction.

[0037] In another possible implementation, the coordinate difference between pixels indicating the same object in the registered first image and the reference second image is less than or equal to a first threshold to achieve pixel-level registration.

[0038] Secondly, an image registration apparatus is provided, which may include: a first acquisition unit, a second acquisition unit, a third acquisition unit, a determination unit, and a registration unit. Wherein:

[0039] The first acquisition unit is used to acquire a first image to be registered, a reference second image with the smallest temporal interval to the first image, and one or more non-reference second images. The imaging rays of the first image and the reference second image are different, while the imaging rays of the reference second image and the non-reference second images are the same.

[0040] The second acquisition unit is used to acquire the first optical flow of each non-reference second image and the reference second image.

[0041] The third acquisition unit is used to acquire the optical flow scaling factor corresponding to each non-reference second image. The optical flow scaling factor is used to indicate the ratio between the first optical flow and the second optical flow. The second optical flow is the optical flow between the first image to be registered and the reference second image.

[0042] The determining unit is used to determine the second optical flow based on the first optical flow and the optical flow scaling factor.

[0043] The registration unit is used to register the first image to be registered with the reference second image according to the second optical flow, so as to obtain the registered first image.

[0044] The image registration apparatus provided in this application first acquires the optical flow between images with the same imaging light rays, converts it to obtain the optical flow between images with different imaging light rays, and then registers the images with different imaging light rays according to the acquired optical flow. Since the accuracy of acquiring the optical flow between images with the same imaging light rays is high, the accuracy of the converted optical flow between images with different imaging light rays can be improved, thereby improving the registration accuracy of images with different imaging light rays.

[0045] It should be noted that the specific implementation of each unit in the second aspect is the same as the method description in the first aspect, and will not be repeated here.

[0046] Thirdly, this application provides an image registration device that can implement the functions described in the method example of the first aspect above. These functions can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functions. This image registration device can exist in the form of a chip.

[0047] In one possible implementation, the image registration device includes a processor and a transceiver. The processor is configured to support the image registration device in performing the corresponding functions described in the above methods. The transceiver supports communication between the image registration device and other devices. The image registration device may also include a memory coupled to the processor, which stores necessary program instructions and data for the image registration device.

[0048] Fourthly, a computer-readable storage medium is provided, including instructions that, when executed on a computer, cause the computer to perform the image registration method provided in the first aspect or any possible implementation thereof.

[0049] Fifthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to execute the image registration method provided in the first aspect or any possible implementation thereof.

[0050] Sixthly, this application provides a chip system including a processor and potentially a memory for implementing the corresponding functions in the above-described methods. The chip system can be composed of chips or may include chips and other discrete devices.

[0051] In a seventh aspect, this application provides an image registration system, which includes the image registration device described in the fifth aspect, the image registration device having the functions of the first aspect and any possible implementation thereof.

[0052] It should be noted that any of the possible implementations of any of the above aspects can be combined, provided that the solutions do not contradict each other. Attached Figure Description

[0053] Figure 1 A schematic diagram of a color wheel device;

[0054] Figure 2 This is a schematic diagram of images captured by a traffic capture camera based on rotating color wheel imaging in time-division format;

[0055] Figure 3 This is a schematic diagram of a time-division acquired multispectral image;

[0056] Figure 4 A schematic diagram of the system architecture of an image capturing system;

[0057] Figure 5 This application provides a schematic diagram of the architecture of a photography system.

[0058] Figure 6 A schematic diagram of another photographic system provided in this application;

[0059] Figure 7 A schematic diagram of an image sequence provided in this application;

[0060] Figure 8 A schematic diagram of the structure of an image registration device provided in this application;

[0061] Figure 9 A schematic diagram of the system architecture of an image registration method provided in this application;

[0062] Figure 10 A schematic diagram of an image sequence of time-division alternating visible light and infrared images provided in this application;

[0063] Figure 11 A schematic diagram of an image pair after image sequence registration provided in this application;

[0064] Figure 12 A flowchart illustrating an image registration method provided in this application;

[0065] Figure 13 A schematic diagram illustrating the principle of determining the optical flow scaling factor based on the image acquisition time interval provided in this application;

[0066] Figure 14 A schematic diagram of another image registration device provided in this application;

[0067] Figure 15 A schematic diagram of an image registration scene provided for this application;

[0068] Figure 16 A schematic diagram of an image registration process provided in this application;

[0069] Figure 17 A schematic diagram of another image registration scenario provided for this application;

[0070] Figure 18 A schematic diagram illustrating another image registration process provided for this application;

[0071] Figure 19 A schematic diagram of another image registration scenario provided for this application;

[0072] Figure 20 A schematic diagram illustrating another image registration process provided for this application;

[0073] Figure 21 A schematic diagram showing the comparison of images before and after registration provided for this application;

[0074] Figure 22 A schematic diagram of another image registration device provided in this application;

[0075] Figure 23 A schematic diagram of the structure of a computing device provided in this application. Detailed Implementation

[0076] In the embodiments of this application, in order to clearly describe the technical solutions of the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different. There is no sequential or major order among the technical features described by "first" and "second".

[0077] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner to facilitate understanding.

[0078] In the embodiments of this application, at least one can also be described as one or more, and multiple can be two, three, four or more, and this application does not impose any restrictions.

[0079] Furthermore, the network architecture and scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0080] Before describing the embodiments of this application, the terms used in this application will be explained in a unified manner, and will not be explained in detail hereafter.

[0081] Optical flow: The movement of an image object between two consecutive frames due to the movement of the target object or camera is called optical flow. Optical flow is a two-dimensional vector field that can be used to represent the displacement of a point from the first frame to the second frame. The optical flow between two images is a two-dimensional vector diagram that includes the displacement of each pixel in the image in different dimensions. For example, optical flow is a two-dimensional vector diagram (u,v), where the u component indicates the displacement of each pixel in the image along the X-axis, and the v component indicates the displacement of each pixel in the image along the Y-axis.

[0082] Optical flow scaling factor refers to the proportional relationship between optical flows. In the embodiments of this application, the optical flow scaling factor is used to indicate the proportional relationship between the first optical flow between a reference second image and a non-reference second image with the same supplementary lighting, and the second optical flow between a reference second image and a first image to be registered with different imaging lights.

[0083] The same object in an image can be a fixed point on a person or object in the image, or a pixel of a fixed point on a person or object in the image.

[0084] Supplemental lighting refers to providing additional light to the environment of the shooting device (equipment).

[0085] Supplemental lighting type can refer to the wavelength or frequency band of the light that supplements the environment of the shooting device (equipment).

[0086] Supplemental lighting intensity can refer to the amount of light energy that is supplied to the environment of the shooting device (equipment).

[0087] Imaging light refers to the light that enters the imaging sensor during image acquisition. Different imaging light sources can be of different types or intensities. Different types of supplementary lighting during image acquisition result in unfiltered light entering the sensor and acquiring the image, thus different imaging light types. Even if the supplementary lighting type is the same, or there is no supplementary lighting, different filtering methods are applied before entering the sensor to acquire the image, resulting in different imaging light types. Finally, different supplementary lighting intensities or intensity processing methods are also considered when acquiring the image before entering the sensor and acquiring the image, resulting in different imaging light intensities.

[0088] Currently, the industry has proposed a dual-frame fusion surveillance camera that acquires infrared and visible images in a time-division manner. This scheme uses an image sensor that alternately uses different supplementary lighting methods in time-division to obtain different imaging light rays, thereby acquiring visible light frames and infrared supplementary lighting frames. The visible light images and infrared images acquired in time-division alternating ways are fused to obtain a high-quality image that retains the true color information of the object.

[0089] In traffic monitoring scenarios, the industry has proposed a traffic capture camera based on rotating color wheel imaging, which adds a color wheel device between the lens and the image sensor. This color wheel device is as follows: Figure 1 As shown. Figure 1The rectangle represents the image sensor. The color wheel assembly includes an infrared cut-off filter (preventing infrared light from passing through) and a full-pass filter (allowing infrared light to pass through). By rotating the color wheel assembly, different filters are positioned in front of the image sensor at different times, allowing different types of light to enter the sensor at different times (i.e., different types of imaging light are captured at different times), resulting in images of different imaging light. When the infrared cut-off filter is in front of the image sensor, the imaging light is visible light, and a visible light image is captured; when the full-pass filter is in front of the image sensor, the imaging light is infrared light, and an infrared image is captured. The periodic rotation of the color wheel assembly switches between the infrared cut-off filter and the full-pass filter, allowing a single image sensor to alternately acquire visible light and infrared images, with the time interval between acquiring adjacent infrared and visible light images depending on the switching transition time. Figure 2 The image shown illustrates the time-division captured by a traffic camera based on rotating color wheel imaging. The horizontal lines in the image indicate that the pixels in the time-division captured image have undergone significant displacement. Figure 3 This illustrates a time-division acquired multispectral image, including images of the same vehicle captured under infrared illumination and images captured without illumination. Figure 3 A clear displacement can be seen in the image.

[0090] In systems based on this type of time-division multispectral image acquisition, pixel displacement occurs between adjacent infrared and visible light images when capturing moving objects due to the different acquisition times. Before multispectral image fusion, image registration between adjacent infrared and visible light images is necessary. However, due to differences in exposure parameters, spectra, and illumination between the visible light image sequence and the captured infrared supplementary lighting image, such as… Figure 2 As shown, adjacent infrared and visible light images can exhibit significant differences in texture and color. Therefore, registration algorithms that rely on feature similarity to estimate the correspondence between images struggle to achieve pixel-level registration accuracy.

[0091] To improve the registration accuracy between infrared and visible frames, the industry practice is to use two image sensors with a beam-splitting prism placed between them and the lens. The prism is made of two pieces of high-transmittance glass bonded together, with a film of a certain thickness coated on the bonded surface to allow for the reflection and transmission of different wavelengths of light. The incident light is split into visible and infrared light by the prism, and then the two image sensors image the split visible and infrared light respectively. The system architecture of this image capture system is as follows: Figure 4As shown. Provided the prism and two image sensors are precisely assembled, this scheme can directly obtain registered infrared and visible frame images. However, this scheme requires very high assembly precision for the two image sensors, resulting in high production complexity; it requires a beam-splitting prism and two image sensors, leading to expensive material costs; and the beam-splitting prism, located between the lens and image sensors, is incompatible with existing standard back focal length lenses.

[0092] Based on this, this application provides an image registration method that utilizes the principle that a moving object can be regarded as uniform linear motion in a very short time. It obtains the optical flow between images with the same supplementary light through feature similarity, converts it into the optical flow between images with different imaging rays, and finally registers the images with different imaging rays according to the optical flow between the images with different imaging rays, thereby improving the registration accuracy of images with different imaging rays.

[0093] Compared to the beam splitter prism approach, this application improves registration accuracy, reduces costs, and is compatible with various lens designs.

[0094] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0095] The solution provided in this application embodiment can be executed by an image registration device for registering images of different imaging light rays acquired by the imaging system in time-division mode.

[0096] Figure 5 This is a schematic diagram of the architecture of a photography system provided in an embodiment of this application. Figure 5 The illustrated imaging system is used to acquire visible light and infrared images in time-division format. For example... Figure 5 As shown, the imaging system may include at least: an imaging device 501, an infrared supplementary lighting device 502, and a visible light supplementary lighting device 503.

[0097] The infrared supplementary lighting device 502 is used to emit infrared light to supplement the environmental scene within the field of view of the imaging device 501, so as to improve the illumination of the environmental scene.

[0098] The visible light supplementary lighting device 503 is used to emit visible light to supplement the environmental scene within the field of view of the imaging device 501, so as to improve the illumination of the environmental scene.

[0099] The photographing device 501 is used to take pictures of the environmental scene within its field of view. Figure 5 The illustrated camera system can acquire visible light and infrared images in real time.

[0100] For example, Figure 5The illustrated photo capture system can be a traffic capture system, the photo capture device 501 can be a traffic capture camera, and the photo capture device 201 can be installed on an F pole installed above the road to capture traffic scenes on the road.

[0101] Figure 6 This is a schematic diagram of the architecture of another imaging system provided in an embodiment of this application. Figure 6 As shown, the photography system may include at least: a photography device 601 and a visible light supplementary lighting device 602. The photography device 601 is used to take pictures of the environmental scene within its field of view. Figure 6 The illustrated imaging system can acquire visible light images with different imaging light intensities in real time.

[0102] Figure 6 In the illustrated imaging system, different filters can be changed in front of the imaging sensor of the imaging device 601 to allow different types of light to enter the imaging sensor, thereby capturing images with different types of light.

[0103] Before describing the solution of this application, the principles of this application will be explained.

[0104] For a moving object, its motion can be considered uniform and linear over a very short time interval, meaning the displacement (pixel displacement in an image) is proportional to the time interval. For example... Figure 7 As shown, image I0, image I t Image I1 consists of three adjacent time-domain images acquired in a time-division multiplexing manner. If object A moves linearly, then the coordinates of point P on the object in the three images are: Figure 7 On the straight line shown in the diagram, point P on the object lies in image I. t The middle point is P t Point P on the object is point P in image I1. 1 .

[0105] Point P on image I0 0 With Image I t The optical flow between them is P. 0 Pointing to image I t The corresponding point P t motion vector Point P on image I0 0 The optical flow between the image I1 and the image I1 is P. 0 Point to the corresponding point P on image I1 1 motion vector Point P on image I0 0 Pointing to image I t The corresponding point P t motion vector And point P on image I0 0 Point to the corresponding point P on image I11 motion vector The following relationship must be satisfied:

[0106]

[0107] Where sf is the optical flow scaling factor, used to indicate the proportional relationship of optical flow.

[0108] Based on this principle, images of different modalities (different supplementary lighting, or others) (I0 and I) can be compared. t The optical flow estimation is transformed into optical flow estimation of the same modal images (I0 and I1).

[0109] It should be noted that the directional parameters described in this application (such as optical flow, displacement, etc.) all refer to these parameters in the same direction. However, this application does not limit the specific direction, as long as all directional parameters have the same direction.

[0110] On the one hand, embodiments of this application provide an image registration apparatus 80 for performing the image registration method provided in this application.

[0111] Figure 8 This diagram illustrates the structure of the image registration apparatus 80 provided in an embodiment of this application. Figure 8 As shown, the image registration device 80 may include a processor 801, a memory 802, and a transceiver 803.

[0112] The following is combined Figure 8 The various components of the image registration device 80 will be described in detail below:

[0113] The memory 802 may be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, used to store application code, configuration files, data information, or other content that can implement the methods of this application. In other possible cases, the memory 802 may also be deployed in other devices independent of the image registration device 80.

[0114] Transceiver 803 is used for information exchange between image registration device 80 and other devices.

[0115] The processor 801 can be the control center of the image registration device 80. For example, the processor 801 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0116] Processor 801 performs the following functions by running or executing software programs and / or modules stored in memory 802:

[0117] The process involves acquiring a first image to be registered, a reference second image with the smallest temporal interval to the first image, and one or more non-reference second images. The imaging rays of the first image and the reference second image are different, while the imaging rays of the reference second image and the non-reference second images are the same. The process also involves acquiring the first optical flow between each non-reference second image and the reference second image; acquiring the optical flow scaling factor corresponding to each non-reference second image, which indicates the ratio between the first and second optical flows; the second optical flow being the optical flow between the first image to be registered and the reference second image; determining the second optical flow based on the first optical flow and the optical flow scaling factor; and registering the first image to be registered with the reference second image based on the second optical flow to obtain the registered first image.

[0118] The method provided in this application embodiment can be executed by an image registration device, and the system architecture of the solution provided in this application embodiment can be as follows: Figure 9 As shown. Figure 9 As shown, the system includes an acquisition unit, an image registration unit, and an image fusion unit. The acquisition unit acquires consecutive first and second images in a time-division alternating manner. The first and second images acquired by the acquisition unit at the same time are input to the image registration unit. After the image registration unit performs registration according to the scheme provided in this application, the images are then fused by the image fusion unit to output the final high-quality fused image for that moment.

[0119] The imaging light in the first image is different from that in the second image.

[0120] For example, the first image and the second image may have different types of imaging light or different imaging light intensities.

[0121] For example, Figure 9The image sequence of time-division alternating visible light and infrared images acquired by the acquisition unit shown in the diagram can be as follows: Figure 10 As shown. Figure 10 The image pairs after the illustrated image sequence registration can be as follows: Figure 11 As shown.

[0122] On the other hand, embodiments of this application provide an image registration method for registering a first image to be registered to a reference second image. For example... Figure 12 As shown, the image registration method provided in this application may include:

[0123] S1201, The image registration device acquires a first image to be registered, a reference second image with the smallest temporal interval to the first image, and one or more non-reference second images.

[0124] Among them, the imaging rays of the first image and the reference second image are different, while the imaging rays of the reference second image and the non-reference second image are the same.

[0125] In one possible implementation, the first image to be registered and the reference second image have different illumination types, while the reference second image and the non-reference second image have the same illumination type, in order to improve the registration accuracy of images with different illumination types.

[0126] For example, the first image to be registered can be an infrared image acquired with supplemented infrared light, and the reference second image and the non-reference second image can be visible light images acquired with supplemented visible light.

[0127] For example, the first image to be registered is a visible light image acquired with supplemented visible light, while the reference second image and the non-reference second image are infrared images acquired with supplemented infrared light.

[0128] In another possible implementation, the supplementary light intensities of the first image to be registered and the reference second image are different, while the supplementary light intensities of the reference second image and the non-reference second image are the same, so as to improve the registration accuracy of images with different supplementary light intensities.

[0129] In another possible implementation, the types of light entering the imaging sensor are different when acquiring the first image and the reference second image, while the types of light entering the imaging sensor are the same when acquiring the reference second image and the non-reference second image.

[0130] For example, when acquiring the first image to be registered, infrared light can enter the imaging sensor, and the first image to be registered is an infrared image. When acquiring the reference second image and the non-reference second image, visible light can enter the imaging sensor, and the reference second image and the non-reference second image can be visible light images.

[0131] For example, when acquiring the first image to be registered, visible light can enter the imaging sensor, and the first image to be registered is a visible light image. When acquiring the reference second image and the non-reference second image, infrared light can enter the imaging sensor, and the reference second image and the non-reference second image are infrared images.

[0132] Infrared light entering the imaging sensor can be achieved by supplementing the ambient light of the imaging device (equipment) with infrared light, or by placing a filter device (e.g., a filter) that only allows infrared light to pass through before the imaging sensor, ensuring that the light entering the imaging sensor is infrared light. Similarly, visible light entering the imaging sensor can be achieved by supplementing the ambient light of the imaging device (equipment) with visible light or not supplementing it, or by placing a filter device (e.g., a filter) that only allows visible light to pass through before the imaging sensor, ensuring that the light entering the imaging sensor is visible light.

[0133] Specifically, the reference second image is the target registration object of the first image to be registered in the time-division alternating image sequence where the first image to be registered is located, and it is also the second image with the smallest time interval to the first image to be registered.

[0134] For example, in Figure 1 In the scenario where the color wheel device acquires alternating infrared and visible light images in time division, the reference second image and the first image to be registered are images captured by the color wheel device during one rotation.

[0135] Furthermore, a non-reference second image serves as the second image for auxiliary registration. The non-reference second image is the second image other than the reference second image within the time-division alternating image sequence containing the first image to be registered. There can be one or more non-reference second images.

[0136] In practical applications, non-reference second images can be selected according to actual needs. This application embodiment does not limit the number of non-reference second images or their temporal positional relationship with the reference second image.

[0137] In one possible implementation, each non-reference second image is temporally adjacent to the reference second image. In temporally adjacent images, the pixel displacement of the same object is minimized, which improves the accuracy of the second optical flow and thus enhances the registration accuracy between the first image and the reference second image.

[0138] Optionally, the non-reference second image is the second image preceding the reference second image. Alternatively, the non-reference second image is the second image following the reference second image. The non-reference second image can be both the second image preceding and following the reference second image.

[0139] For example, in Figure 10In the image sequence shown, the first image to be registered is infrared frame 3, the reference second image is visible frame 3, and the non-reference second image can be visible frame 2.

[0140] For example, in Figure 10 In the image sequence shown, the first image to be registered is infrared frame 3, the reference second image is visible frame 3, and the non-reference second image can be visible frame 4.

[0141] For example, in Figure 10 In the image sequence shown, the first image to be registered is infrared frame 3, the reference second image is visible frame 3, and the non-reference second images can be visible frame 2 and visible frame 4.

[0142] S1202, The image registration device acquires the first optical flow of each non-reference second image and the reference second image.

[0143] Specifically, in S1202, the image registration device can estimate and obtain the first optical flow between each non-reference second image and the reference second image through deep learning methods or traditional optical flow estimation methods. The process of obtaining the first optical flow will not be described in detail in this embodiment.

[0144] The first optical flow is used to indicate the displacement relationship between each pixel in the non-reference second image and the reference second image. The relationship between the pixels in the non-reference second image and the reference second image and the first optical flow between them is described as follows:

[0145] Assume that image I0 is the nearest visible light image in the time domain of the infrared image to be registered, and image I1 is the adjacent visible light image of I0. The optical flow f of images I1 and I0 is: a two-dimensional vector diagram (u, v) with image I0 as the reference, used to indicate the pixel displacement of each pixel in image I0 to the corresponding pixel in image I1.

[0146] For any pixel I0(x,y) at any position (x,y) in image I0, there exists a two-dimensional vector (u(x,y), v(x,y)) in the two-dimensional vector diagram (u,v) that points to the corresponding matching pixel I1(x+u(x,y), y+v(x,y)) in image I1, which can be expressed by the following formula:

[0147]

[0148] in, This indicates that two pixels are in a corresponding relationship.

[0149] S1203, The image registration device obtains the optical flow scaling factor corresponding to each non-reference second image.

[0150] One of the optical flow scaling factors is used to indicate the ratio between the first optical flow and the second optical flow.

[0151] Specifically, the number of optical flow scaling factors acquired in S1203 is the same as the number of non-reference second images. It should be understood that each optical flow scaling factor acquired in S1203 is used to indicate the proportional relationship between the first optical flow between each non-reference second image and the reference second image, and the second optical flow between the reference second image and the first image to be registered. The second optical flow is the optical flow between the first image to be registered and the reference second image.

[0152] Optionally, obtaining the optical flow scaling factor corresponding to a non-reference second image in S1203 can be achieved through, but is not limited to, any of the following two schemes:

[0153] Option 1: The physical meaning of the optical flow scaling factor can be the proportional relationship of the imaging displacement of pixels. The imaging displacement is highly related to the image acquisition time. In S1203, the optical flow scaling factor corresponding to the non-reference second image is obtained, which can be specifically implemented as follows: obtaining the first acquisition time interval between the reference second image and the first image to be registered, and the second acquisition time interval between the reference second image and the non-reference second image; determining the optical flow scaling factor based on the first acquisition time interval, the second acquisition time interval, and the parameters of the acquisition device.

[0154] The parameters of the acquisition device refer to the parameters that affect the imaging displacement. This application does not limit the content of the parameters of the acquisition device.

[0155] For example, the acquisition parameters may include: the installation height of the acquisition device, the capture distance of the acquisition device, the pitch angle between the optical axis of the acquisition device and the horizontal plane, and the moving speed of the object captured by the acquisition device.

[0156] For example, based on the first acquisition time interval, the second acquisition time interval, and the parameters of the acquisition device, the optical flow scaling factor is determined. Specifically, the optical flow scaling factor sf and the first acquisition time interval Δt, the second acquisition time interval ΔT, and the parameters of the acquisition device satisfy the following relationship:

[0157] Where D = h sinα + d cosα, h is the height of the acquisition device, d is the capture distance of the acquisition device, α is the pitch angle between the optical axis of the acquisition device and the horizontal plane, and v is the moving speed of the object captured by the acquisition device.

[0158] In Equation 1, if the first image to be registered is acquired after the reference second image in terms of timing, the denominator is negative; if the first image to be registered is acquired before the reference second image, the denominator is positive. Similarly, if the non-reference second image is acquired after the reference second image in terms of timing, the numerator is negative; if the non-reference second image is acquired before the reference second image, the numerator is positive.

[0159] For example, the principle of determining the optical flow scaling factor based on the image acquisition time interval is as follows: Figure 13 As shown. Figure 13 The solid rectangles represent the positions of moving objects in the previous frame, while the dashed rectangles represent the positions of moving objects after time vΔt.

[0160] Option 2: Since optical flow is used to indicate the pixel displacement of a pixel, the optical flow scaling factor can be understood as the proportional relationship of the pixel displacement of the same object in the image. In S1203, the optical flow scaling factor corresponding to the non-reference second image is obtained. Specifically, this can be achieved by: obtaining one or more matching pixel pairs in the reference second image and the first image to be registered, wherein a matching pixel pair includes pixels in the reference second image and the first image to be registered whose feature similarity is greater than or equal to a second threshold; and determining the optical flow scaling factor based on the pixel displacement of the first matching pixel pair and the first optical flow.

[0161] The second threshold is a boundary value for determining whether pixels correspond. The value of the second threshold can be configured according to actual needs. For example, when the similarity is 1, which means that they are completely identical, the second threshold can be set to any value greater than 0.5 and less than or equal to 1, such as 0.8, 0.9, or 0.95. When other indicators are used to measure similarity, the setting method of the second threshold can be similar. This application does not limit this aspect.

[0162] The first matching pixel pair is one of one or more matching pixel pairs.

[0163] In one possible implementation, the first matching pixel pair is the matching pixel pair with the highest matching degree among the above one or more matching pixel pairs.

[0164] For example, in Scheme 2, the optical flow scaling factor corresponding to the non-reference second image is determined based on the pixel displacement of the first matching pixel pair and the first optical flow. Specifically, this can be implemented as follows: based on the first optical flow, the corresponding pixel P1 in the non-reference second image is determined for the pixel P0 in the reference second image included in the first matching pixel pair; based on the two-dimensional vector of P0 and P1... And the pixel P in the first image to be registered, which is included in the first matching pixel pair P0. t Two-dimensional vector Determine the above optical flow scaling factor

[0165] It should be noted that the above two schemes only describe the process of obtaining the optical flow scaling factor corresponding to a non-reference second image. When registering the first image to be registered, if multiple non-reference second images are obtained in step S1201, the optical flow scaling factor corresponding to each non-reference second image can be obtained in step S1203. The acquisition process is the same and will not be described in detail.

[0166] The two schemes described above are merely examples to illustrate possible implementations of S1203, and are not specific limitations on the schemes for obtaining the optical flow scaling factor.

[0167] S1204. The image registration device determines the second optical flow based on the first optical flow and the optical flow scaling factor.

[0168] Specifically, since the optical flow scaling factor is used to indicate the ratio between the first optical flow and the second optical flow, after obtaining the optical flow scaling factor in S1203, the second optical flow of the first image to be registered and the reference second image can be determined based on the first optical flow.

[0169] In one possible implementation, in S1204, the second optical flow is determined based on the first optical flow and the optical flow scaling factor. Specifically, this can be achieved as follows: if the first optical flow and the second optical flow have the same direction, the second optical flow is determined to be the product of the optical flow scaling factor and the first optical flow; if the first optical flow and the second optical flow have different directions, the second optical flow is determined to be the negative of the product of the optical flow scaling factor and the first optical flow.

[0170] The first optical flow and the second optical flow have the same direction, which means that the first image to be registered is located between the reference second image and the non-reference second image in terms of timing. The first optical flow and the second optical flow take the reference second image as the reference. When the direction of the first optical flow and the second optical flow points outward from the reference second image or both point to the reference second image, the directions are the same.

[0171] The first optical flow and the second optical flow have different directions, which means that the first image to be registered is located outside the reference second image and the non-reference second image in terms of timing. The first optical flow and the second optical flow take the reference second image as the reference, and the directions of the first optical flow and the second optical flow are either pointing outward from the reference second image or both pointing to the reference second image, with opposite directions.

[0172] Among them, the second optical flow f vn With optical flow scaling factor sf and first optical flow f vv It can satisfy the following expression:

[0173]

[0174] In one possible implementation, there are multiple non-reference second images. Accordingly, determining the second optical flow based on the first optical flow and the optical flow scaling factor corresponding to each non-reference second image includes: obtaining multiple second optical flows based on the first optical flow of each non-reference second image and the reference second image, and the optical flow scaling factor corresponding to each non-reference second image. That is, in S1203, the optical flow scaling factor corresponding to each non-reference second image is obtained, and in S1204, the second optical flow is determined according to each optical flow scaling factor, resulting in a total number of second optical flows equal to the number of non-reference second images.

[0175] In another possible implementation, when there are multiple non-reference second images, the optical flow scaling factor corresponding to each non-reference second image is obtained in S1203, the calculated value of the multiple optical flow scaling factors is obtained in S1204, and then a second optical flow is determined based on the calculated value.

[0176] The calculated value can be a simple average, a weighted average, a maximum value, or a minimum value, etc.

[0177] S1205. The image registration device registers the first image to be registered with the reference second image according to the second optical flow to obtain the registered first image.

[0178] In one possible implementation, there is one non-reference second image. In S1205, the first image to be registered is registered with the reference second image based on a second optical flow obtained in S1204 to obtain the registered first image.

[0179] In another possible implementation, there are multiple non-reference second images. In S1204, a second optical flow is obtained. In S1205, the first image to be registered is registered with the reference second image based on the second optical flow obtained in S1204 to obtain the registered first image.

[0180] In another possible implementation, there are multiple non-reference second images. In S1204, the same number of second optical flows as the number of non-reference second images are obtained. In S1205, the first image to be registered is registered with the reference second image according to the second optical flow to obtain the registered first image. Specifically, it can be implemented as follows: based on the multiple second optical flows, a target second optical flow is determined, and the first image to be registered is registered with the reference second image according to the target second optical flow to obtain the registered first image.

[0181] Specifically, determining the target second optical flow based on multiple second optical flows can be achieved by taking a simple average, weighted average, maximum, or minimum value of the multiple second optical flows as the target second optical flow.

[0182] Specifically, the second optical flow indicates the relationship between the pixel displacements of the first image to be registered and the reference second image. Therefore, registering the first image to be registered with the reference second image according to the second optical flow can improve the registration accuracy.

[0183] Optionally, the coordinate difference between pixels indicating the same object in the registered first image and the reference second image is less than or equal to a first threshold.

[0184] The first threshold mentioned above is the allowable error for pixel-level alignment. The value of the first threshold can be configured according to actual needs. For example, when complete and accurate registration is required, the first threshold can be set to 0. When relatively accurate registration is required, the first threshold can be set to any positive integer such as 10, 50, or 100. This application embodiment does not limit this.

[0185] For example, the second optical flow is a two-dimensional vector image (u, v). The u component in the second optical flow indicates the displacement of each pixel in the image along the X-axis, and the v component indicates the displacement of each pixel in the image along the Y-axis. In S1205, based on the second optical flow, the first image to be registered is registered with the reference second image to obtain the registered first image. Specifically, this can be implemented as follows: the registered first image... With the first image I to be registered t The second optical flow (u,v) satisfies the following relationship:

[0186] Where x, y are the coordinates of a pixel in the image. u(x,y) is the value of the pixel at coordinates (x,y) in the first image in the u component of the second optical flow, and v(x,y) is the value of the pixel at coordinates (x,y) in the first image in the v component of the second optical flow.

[0187] The image registration method provided in this application first obtains the optical flow between images with the same imaging rays, converts it to obtain the optical flow between images with different imaging rays, and then registers the images according to the obtained optical flow. Since the accuracy of obtaining the optical flow between images with the same imaging rays is high, the accuracy of the converted optical flow between images with different imaging rays can be improved, thereby improving the registration accuracy of images with different imaging rays.

[0188] The solution provided in this application will be described in detail below through specific embodiments. The following embodiments utilize traffic capture cameras based on rotating color wheel imaging to acquire... Figure 10 The registration process of the image sequence shown is illustrated as an example and does not constitute a specific limitation.

[0189] In the following embodiments, by Figure 14The image registration apparatus shown implements the scheme of this application, registering alternating visible light and infrared images acquired in a time-division manner. For example... Figure 14 As shown, the image registration device includes an input module, an optical flow estimation module, an optical flow scaling factor calculation module, an optical flow correction module, and a pixel alignment module.

[0190] The input module is used to execute S1201 to acquire the first image to be registered, the reference second image, and the non-reference second image.

[0191] The optical flow estimation module is used to execute S1202 to obtain the first optical flow of the non-reference second image and the reference second image.

[0192] The optical flow scaling factor calculation module is used to execute S1203 to obtain the optical flow scaling factor corresponding to the non-reference second image.

[0193] The optical flow correction module is used to execute S1204, which determines the second optical flow of the first image to be registered and the reference second image according to the optical flow scaling factor and the first optical flow.

[0194] The pixel alignment module is used to execute S1205, which, based on the second optical flow obtained by correction, registers the first image to be registered to the reference second image to obtain the registered first image.

[0195] In the following embodiments, an increase is made between the lens and the image sensor. Figure 1 The rotating color wheel shown periodically switches between an infrared cutoff filter and a full-pass filter, allowing a single image sensor to alternately acquire visible light and infrared images. The sensor's output frame rate is 50 frames per second, or 25 frames of visible light images and 25 frames of infrared images per second. The acquired image sequence can be... Figure 10 As shown. In the following embodiments, the first image is an infrared image and the second image is a visible light image, which will be used as an example for illustration.

[0196] Example 1

[0197] Input module from Figure 10 From the image sequence shown, select one frame of infrared image I to be registered. t And the two adjacent visible light images I0 and I1 preceding it in time sequence, where I1 is the infrared image I to be registered. t The registration objects are I1, which is the reference second image, I0, which is the reference second image during registration, and I0 is the non-reference second image.

[0198] The image selected by the input module can be as follows Figure 15 As shown. Figure 15 The image shown is via Figure 14 The registration process of the image registration device shown can be as follows: Figure 16 As shown, the details are as follows:

[0199] Visible light images I0 and I1 are input into the optical flow estimation module. The optical flow estimation module estimates the first optical flow f of two adjacent visible light images I0 and I1 using deep learning methods or traditional optical flow estimation methods. vv The first optical flow f vv The nearest visible light image I1 in the time domain of the infrared image can be used as a reference.

[0200] The optical flow scaling factor calculation module can obtain the time interval ΔT = 40ms between images I0 and I1 based on the frame rate. Based on the rotation speed and radius of the color wheel and the size of the image sensor, the transition time between the infrared and visible light images is Δt = 5ms. According to the survey assumptions for the traffic checkpoint camera, the camera mounting height h is 6 meters, the capture distance d is 24 meters, and the pitch angle α between the camera's optical axis and the horizontal plane is 12°. Assuming a road speed limit of v = 80 km / h, the optical flow scaling factor corresponding to image I0 can be obtained through scheme 1 in S1203.

[0201] Since the infrared image to be registered is located outside the two visible light images in time sequence (i.e., the directions of the first and second optical flows are different), the optical flow correction module determines the second optical flow between the infrared image to be registered and the visible light image I1, which is the closest in time, to be f. vn =-sf*f vv =-0.09f vv .

[0202] The pixel alignment module utilizes the obtained second optical flow f vn The infrared image to be registered is mapped and transformed to achieve pixel-level alignment with the nearest visible light image I1 in the time domain. The pixel-level aligned infrared and visible light image pair can then be further processed by image fusion to output a fused image with excellent detail and color.

[0203] Example 2

[0204] Input module from Figure 11 From the image sequence shown, select one frame of infrared image I to be registered. t And the adjacent visible light image I1 that precedes it in time, and the adjacent visible light image I2 that follows it in time. I1 is the infrared image I to be registered. t The registration objects are I1, which is the reference second image, I2, which is the reference second image during registration, and I3, which is the non-reference second image.

[0205] The image selected by the input module can be as follows Figure 17 As shown. Figure 17 The image shown is via Figure 14 The registration process of the image registration device shown can be as follows: Figure 18 As shown, the details are as follows:

[0206] Visible light images I0 and I2 are input into the optical flow estimation module. The optical flow estimation module estimates the first optical flow f of two adjacent visible light images I0 and I2 using deep learning methods or traditional optical flow estimation methods. vv The first optical flow f vv The nearest visible light image I1 in the time domain of the infrared image can be used as a reference.

[0207] The optical flow scaling factor calculation module can obtain the time interval ΔT = 40ms between images I0 and I2 based on the frame rate. Based on the rotation speed and radius of the color wheel and the size of the image sensor, the transition time between the infrared and visible light images is Δt = 5ms. According to the survey assumptions for the traffic checkpoint camera, the camera mounting height h is 6 meters, the capture distance d is 24 meters, and the pitch angle α between the camera's optical axis and the horizontal plane is 12°. Assuming a road speed limit of v = 80 km / h, the optical flow scaling factor corresponding to image I2 can be obtained through scheme 1 in S1203.

[0208] Since the infrared image to be registered is located between two visible light images in time (i.e., the directions of the first and second optical flows are the same), the optical flow correction module determines the second optical flow between the infrared image to be registered and the nearest visible light image I1 in the time domain as f. vn =sf*f vv =0.08f vv .

[0209] The pixel alignment module utilizes the obtained second optical flow as f vn The infrared image to be registered is mapped and transformed to achieve pixel-level alignment with the nearest visible light image I1 in the time domain. The pixel-level aligned infrared and visible light image pair can then be further processed by image fusion to output a fused image with excellent detail and color.

[0210] Example 3

[0211] Input module from Figure 10 From the image sequence shown, select one frame of infrared image I to be registered. t The two adjacent visible light images I0 and I1 preceding it in time sequence, and the adjacent visible light image I2 following it in time sequence. I1 is the infrared image I to be registered. t The registration objects are I1, which is the reference second image, I0 and I2, which are the reference second images during registration, and I0 and I2 are the non-reference second images.

[0212] The image selected by the input module can be as follows Figure 19 As shown. Figure 19 The image shown is via Figure 14 The registration process of the image registration device shown can be as follows: Figure 20 As shown, the details are as follows:

[0213] Visible light images I0 and I1 are input into the optical flow estimation module. The optical flow estimation module estimates the first optical flow of two adjacent visible light images I0 and I1 using deep learning methods or traditional optical flow estimation methods. The first optical flow The nearest visible light image I1 in the time domain of the infrared image can be used as a reference.

[0214] Visible light images I0 and I2 are input into the optical flow estimation module. The optical flow estimation module estimates the first optical flow of two adjacent visible light images I0 and I2 using deep learning methods or traditional optical flow estimation methods. The first optical flow The nearest visible light image I1 in the time domain of the infrared image can be used as a reference.

[0215] The optical flow scaling factor calculation module can obtain the time interval ΔT = 40ms between images I0 and I2 based on the frame rate. Based on the rotation speed and radius of the color wheel and the size of the image sensor, the transition time between the infrared and visible light images is Δt = 5ms. According to the site survey assumptions for the traffic checkpoint camera, the camera mounting height h is 6 meters, the capture distance d is 24 meters, and the pitch angle α between the camera's optical axis and the horizontal plane is 12°. Assuming a road speed limit of v = 80 km / h, the optical flow scaling factor corresponding to I0 can be obtained through scheme 1 in S1203. Obtain the optical flow scaling factor corresponding to I2

[0216] Since the infrared image to be registered is temporally outside of images I0 and I1 (i.e., the directions of the first and second optical flows are different), the optical flow correction module determines that the second optical flow between the infrared image to be registered and the temporally nearest visible light image I1 is... Since the infrared image to be registered is located between images I0 and I2 in time (i.e., the directions of the first and second optical flows are the same), the optical flow correction module determines that the second optical flow between the infrared image to be registered and the nearest visible light image I1 in the time domain is...

[0217] The pixel alignment module performs a second optical flow estimation on two estimated values. and Take the average or the maximum value to obtain the final infrared image I to be registered. t The second optical flow f between it and its temporally nearest visible light image I1 vn Finally, the second optical flow f is used.vn The infrared image to be registered is mapped and transformed to achieve pixel-level alignment with the nearest visible light image I1 in the time domain. The pixel-level aligned infrared and visible light image pair can then be further processed by image fusion to output a fused image with excellent detail and color.

[0218] For example, suppose the infrared image I before registration t And visible light image I1 such Figure 21 As shown in (a) above, the registered infrared image obtained through the schemes of the three embodiments described above And visible light image I1 such Figure 21 As shown in (b) in the figure, pixel-level alignment is achieved.

[0219] The above primarily describes the solution provided in the embodiments of this application from the perspective of the device's working principle. It is understood that the image registration device described above includes corresponding hardware structures and / or software modules for performing each function in order to achieve the aforementioned functions. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0220] This application embodiment can divide the image registration device provided in this application into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. The module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0221] When dividing each function into modules according to its corresponding function. Figure 22 A schematic diagram of a possible structure of an image registration device deployed in the electronic device involved in the above embodiments is shown. This image registration device 220 can be a functional module or a chip. For example... Figure 22 As shown, the image registration device 220 may include: a first acquisition unit 2201, a second acquisition unit 2202, a third acquisition unit 2203, a determination unit 2204, and a registration unit 2205. The first acquisition unit 2201 is used to perform... Figure 12 The process S1204; the second acquisition unit 2202 is used to execute Figure 12The process S1202; the third acquisition unit 2203 is used to execute Figure 12 The process S1203; the determination unit 2204 is used to execute Figure 12 The process S1204; the registration unit 2205 is used to execute Figure 12 The process S1205 is described above. All relevant content regarding each step in the above method embodiment can be found in the functional descriptions of the corresponding functional modules, and will not be repeated here.

[0222] When using integrated units, Figure 23 A possible structural diagram of the computing device involved in the above embodiments is shown. The computing device 230 may include: a processing module 2301 and a communication module 2302. The processing module 2301 is used to control and manage the operation of the computing device, and the communication module 2302 is used to communicate with other devices. For example, the processing module 2301 is used to execute... Figure 12 The computing device 230 may include any one of processes S1201 to S1205. The computing device 230 may also include a storage module 2303 for storing the program code and data of the computing device 230.

[0223] Among them, the processing module 2301 can be Figure 8 The processor 801 in the physical structure of the image registration device 80 shown can be a processor or a controller. For example, it can be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processing module 2601 can also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc. The communication module 2302 can be... Figure 8 The transceiver 803 and communication module 2302 in the physical structure of the image registration device 80 shown can be a communication port, or a transceiver, transceiver circuit, or communication interface, etc. Alternatively, the aforementioned communication interface can achieve communication with other devices through the aforementioned transceiver components. The aforementioned transceiver components can be implemented by antennas and / or radio frequency devices. The storage module 2303 can be... Figure 8 The memory 802 is located in the physical structure of the image registration device 80 shown.

[0224] As mentioned above, the image registration device 220 or computing device 230 provided in the embodiments of this application can be used to implement the corresponding functions in the methods implemented in the various embodiments of this application. For ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the various embodiments of this application.

[0225] As another embodiment of this invention, a computer-readable storage medium is provided, on which instructions are stored, which, when executed, perform the image registration method described in the above method embodiment.

[0226] As another form of this embodiment, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to execute the image registration method in the above method embodiment.

[0227] This application provides another chip system, which includes a processor for implementing the technical methods of the embodiments of the present invention. In one possible design, the chip system further includes a memory for storing program instructions and / or data necessary for the embodiments of the present invention. In another possible design, the chip system further includes a memory for the processor to call application code stored in the memory. This chip system may be composed of one or more chips, or may include chips and other discrete devices; this application does not specifically limit this.

[0228] The steps of the methods or algorithms described in this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in RAM, flash memory, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, read-only optical discs (CD-ROMs), or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a core network interface device. Of course, the processor and storage medium can also exist as discrete components in the core network interface device. Alternatively, the memory can be coupled to the processor; for example, the memory can exist independently and be connected to the processor via a bus. The memory can also be integrated with the processor. The memory can be used to store application code that executes the technical solutions provided in the embodiments of this application, and its execution is controlled by the processor. The processor is used to execute application code stored in memory, thereby implementing the technical solutions provided in the embodiments of this application.

[0229] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0230] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0231] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0232] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0233] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, essentially, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. The above descriptions are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image registration method, characterized in that, The method includes: Acquire a first image to be registered, a reference second image with the smallest temporal interval to the first image, and one or more non-reference second images, wherein the imaging rays of the first image and the reference second image are different, and the imaging rays of the reference second image and the non-reference second image are the same; Obtain the first optical flow of each of the non-reference second images and the reference second image; Obtain the optical flow scaling factor corresponding to each of the non-reference second images, wherein the optical flow scaling factor is used to indicate the proportional relationship between the first optical flow and the second optical flow; the second optical flow is the optical flow between the first image and the reference second image. The second optical flow is determined based on the first optical flow and the optical flow scaling factor; Based on the second optical flow, the first image is registered with the reference second image to obtain the registered first image.

2. The method according to claim 1, characterized in that, The first image and the reference second image have different fill light types, while the reference second image and the non-reference second image have the same fill light type. or, The first image and the reference second image have different fill light intensities, while the reference second image and the non-reference second image have the same fill light intensity.

3. The method according to claim 1, characterized in that, The first image is an infrared image acquired by an infrared light imaging sensor, and the reference second image and the non-reference second image are visible light images acquired by a visible light imaging sensor. or, The first image is a visible light image acquired by the visible light entering the imaging sensor, and the reference second image and the non-reference second image are infrared images acquired by the infrared light entering the imaging sensor.

4. The method according to any one of claims 1-3, characterized in that, The step of obtaining the optical flow scaling factor corresponding to each of the non-reference second images includes: Acquire a first acquisition time interval between the reference second image and the first image, and a second acquisition time interval between the reference second image and each of the non-reference second images; The optical flow scaling factor is determined based on the first acquisition time interval, the second acquisition time interval, and the parameters of the acquisition device.

5. The method according to any one of claims 1-3, characterized in that, The step of obtaining the optical flow scaling factor corresponding to each of the non-reference second images includes: Obtain one or more matching pixel pairs in the reference second image and the first image, wherein a matching pixel pair includes pixels in the reference second image and the first image whose feature similarity is greater than or equal to a second threshold; The optical flow scaling factor is determined based on the pixel displacement of the first matched pixel pair and the first optical flow; wherein the first matched pixel pair is one of the one or more matched pixel pairs.

6. The method according to any one of claims 1-3, characterized in that, Determining the second optical flow based on the first optical flow and the optical flow scaling factor includes: If the first optical flow and the second optical flow have the same direction, the second optical flow is determined to be the product of the optical flow scaling factor and the first optical flow; If the directions of the first optical flow and the second optical flow are different, the second optical flow is determined to be the negative of the product of the optical flow scaling factor and the first optical flow.

7. The method according to any one of claims 1-3, characterized in that, There are multiple non-reference second images; Determining the second optical flow based on the first optical flow and the optical flow scaling factor includes: obtaining a plurality of second optical flows based on the first optical flow of each non-reference second image and the reference second image, and the optical flow scaling factor corresponding to each non-reference second image; According to the second optical flow, registering the first image with the reference second image to obtain a registered first image includes: determining a target second optical flow according to the plurality of second optical flows, and registering the first image with the reference second image according to the target second optical flow to obtain a registered first image.

8. The method according to any one of claims 1-3, characterized in that, Each of the non-reference second images is temporally adjacent to the reference second image.

9. The method according to any one of claims 1-3, characterized in that, The coordinate difference between the pixels indicating the same object in the registered first image and the reference second image is less than or equal to a first threshold.

10. An image registration device, characterized in that, The device includes: The first acquisition unit is used to acquire a first image to be registered, a reference second image with the smallest temporal interval to the first image, and one or more non-reference second images, wherein the imaging rays of the first image and the reference second image are different, and the imaging rays of the reference second image and the non-reference second image are the same. The second acquisition unit is used to acquire the first optical flow of each of the non-reference second images and the reference second image; The third acquisition unit is used to acquire the optical flow scaling factor corresponding to each of the non-reference second images, wherein the optical flow scaling factor is used to indicate the ratio between the first optical flow and the second optical flow; the second optical flow is the optical flow between the first image and the reference second image. A determining unit is configured to determine the second optical flow based on the first optical flow and the optical flow scaling factor; The registration unit is used to register the first image with the reference second image according to the second optical flow to obtain the registered first image.

11. The apparatus according to claim 10, characterized in that, The first image and the reference second image have different fill light types, while the reference second image and the non-reference second image have the same fill light type. or, The first image and the reference second image have different fill light intensities, while the reference second image and the non-reference second image have the same fill light intensity.

12. The apparatus according to claim 10, characterized in that, The first image is an infrared image acquired by an infrared light imaging sensor, and the reference second image and the non-reference second image are visible light images acquired by a visible light imaging sensor. or, The first image is a visible light image acquired by the visible light entering the imaging sensor, and the reference second image and the non-reference second image are infrared images acquired by the infrared light entering the imaging sensor.

13. The apparatus according to any one of claims 10-12, characterized in that, The third acquisition unit is specifically used for: Acquire a first acquisition time interval between the reference second image and the first image, and a second acquisition time interval between the reference second image and each of the non-reference second images; The optical flow scaling factor is determined based on the first acquisition time interval, the second acquisition time interval, and the parameters of the acquisition device.

14. The apparatus according to any one of claims 10-12, characterized in that, The third acquisition unit is specifically used for: Obtain one or more matching pixel pairs in the reference second image and the first image, wherein a matching pixel pair includes pixels in the reference second image and the first image whose feature similarity is greater than or equal to a second threshold; The optical flow scaling factor is determined based on the pixel displacement of the first matched pixel pair and the first optical flow; wherein the first matched pixel pair is one of the one or more matched pixel pairs.

15. The apparatus according to any one of claims 10-12, characterized in that, The determining unit is specifically used for: If the first optical flow and the second optical flow have the same direction, the second optical flow is determined to be the product of the optical flow scaling factor and the first optical flow; If the directions of the first optical flow and the second optical flow are different, the second optical flow is determined to be the negative of the product of the optical flow scaling factor and the first optical flow.

16. The apparatus according to any one of claims 10-12, characterized in that, There are multiple non-reference second images; The determining unit is specifically used to obtain a plurality of second optical flows based on the first optical flow of each non-reference second image and the reference second image, and the optical flow scaling factor corresponding to each non-reference second image; The registration unit is specifically used to determine a target second optical flow based on the plurality of second optical flows, and to register the first image with the reference second image according to the target second optical flow to obtain a registered first image.

17. The apparatus according to any one of claims 10-12, characterized in that, Each of the non-reference second images is temporally adjacent to the reference second image.

18. The apparatus according to any one of claims 10-12, characterized in that, The coordinate difference between the pixels indicating the same object in the registered first image and the reference second image is less than or equal to a first threshold.

19. An image registration device, characterized in that, The image registration device includes: a processor and a memory; The memory is connected to the processor; The memory is used to store computer instructions, and when the processor executes the computer instructions, the image registration device performs the method as described in any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that, Includes instructions that, when run on a computer, cause the computer to perform the method of any one of claims 1 to 9.

21. A computer program product containing instructions, characterized in that, When it is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 9.

Images