Dual-light imaging automatic registration method of near-infrared and visible light shared sensor

Abstract

The invention discloses a dual-light imaging automatic registration method of a near-infrared and visible light shared sensor, relates to the field of dual-light imaging image registration, and aims to solve the problem of difficult dual-light camera image registration in the prior art. An infrared cut-off filter and a single near-infrared pass filter alternately pass through a lens through a filter circulation switching mechanism, an image formed by interweaving a visible light image and a near-infrared image is shot, optical flow calculation is carried out through the image which is well imaged, pixel motion of an image frame with low imaging quality is estimated, and the image quality is improved. And finally, registering and outputting the high-imaging-quality image and the estimated image. According to the method, accurate matching of the visible light image and the near-infrared image in the aspect of spatial parameters is achieved, meanwhile, a mapping transformation matrix with complex calculation is converted into simple optical flow estimation, rapid, efficient and accurate registration of the visible light image and the near-infrared image is achieved, and therefore a registration image with clear pictures and unified spatial parameters is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of dual-light imaging image registration, in particular to a dual-light imaging automatic registration method of a near-infrared and visible light shared sensor. BACKGROUND

[0002] For the requirement of simultaneously acquiring visible light images and near-infrared images, a traditional dual-light camera device usually adopts a design scheme of dual sensors and dual lenses. Under this scheme, each lens corresponds to an independent sensor, which are respectively used for acquiring visible light images and near-infrared images. However, this traditional design has a serious image registration problem. Due to the inevitable differences in optical characteristics, installation positions, and response characteristics of the sensors of the two lenses, it is difficult to accurately match the acquired visible light images and near-infrared images in terms of spatial position, scale, angle, etc., which leads to image misregistration, distortion, etc. when the two kinds of images are fused to obtain more comprehensive information, greatly affecting the quality of image fusion and the final application effect.

[0003] For example, in a security monitoring scene, when it is necessary to perform fusion analysis by using the detail information of the visible light images and the night recognition advantage of the near-infrared images, inaccurate image registration will cause the positions of the target objects in the two kinds of images to be inconsistent, so that the behavior and characteristics of the target cannot be accurately judged. In industrial detection, for some products that need to use visible light appearance detection and near-infrared internal defect detection at the same time, the image registration problem may lead to misjudgment or missed judgment of product defects.

[0004] In terms of image registration methods, the traditional way is usually to achieve registration by calculating the mapping transformation matrix between the visible light images and the near-infrared images. However, due to the different imaging principles of visible light and near-infrared images, their image features differ greatly, making it very difficult to directly calculate the mapping transformation matrix. And the unified mapping transformation matrix has different registration accuracies for objects at different distances, and has poor distance applicability. Although the visible light optical flow calculation based on deep learning can accurately estimate the pixel motion relationship between the visible light images, so as to perform registration according to the motion relationship, the optical flow needs to accurately find the motion trajectory of the corresponding points in the two images, and the differences in gray distribution, texture features, etc. between the visible light and near-infrared images make it difficult to accurately match these corresponding points, thereby leading to large optical flow calculation errors and failing to achieve high-precision image registration. SUMMARY

[0005] The technical problem to be solved by the present application is to overcome the existing defects and provide a dual-light imaging automatic registration method of a near-infrared and visible light shared sensor, which can effectively solve the problems in the background art.

[0006] In order to achieve the above object, the application discloses a kind of near-infrared and visible light common sensor's dual optical imaging automatic registration method, the technical solution is, including the following steps:

[0007] Step 1, start shooting signal, synchronous trigger camera equipment and filter cycle switching mechanism;Camera equipment starts to execute image acquisition job, and filter cycle switching mechanism starts to act;

[0008] Step 2, filter cycle switching mechanism makes infrared cut-off filter and single near-infrared pass filter alternately pass through lens;So that there is a frame of near-infrared image between every adjacent two frames of visible light image, realize that visible light image and near-infrared image are interlaced together;

[0009] Step 3, judge the imaging quality of visible light image and near-infrared image, mark the image with better imaging quality in visible light image and near-infrared image as a class of images, mark the image with poor imaging quality as a class of images;Every two adjacent frames of a class of images are sandwiched with a frame of a class of images;For different illumination degree, the corresponding relationship of visible light image, near-infrared image and a class of images and a class of images is different;

[0010] Step 4, optical flow calculation is carried out to a class of images, and the optical flow of a class of images is estimated through the calculation result, so as to adjust the pixel position of a class of images, and the adjusted image is marked as quasi-a class of images;

[0011] Step 5, a class of images and quasi-a class of images are registered and output.

[0012] As a preferred technical scheme of the application, in step 1, the filter cycle switching mechanism is one of reciprocating mechanism and rotating mechanism. The reciprocating mechanism includes translation mechanism, fan swing mechanism, etc., and the rotating mechanism is a filter holder driven by a motor, a plurality of filters are staggered on the filter holder, and the switching of the filters is realized by rotating the filter holder.

[0013] As a preferred technical scheme of the application, in step 3, when the light condition is good, the imaging quality of visible light image is high, the visible light image is marked as a class of images, and the near-infrared image is marked as a class of images;When the light condition is poor, the imaging quality of near-infrared image is high, the near-infrared image is marked as a class of images, and the visible light image is marked as a class of images.

[0014] As a preferred technical scheme of the application, in step 4, the optical flow of a class of images is calculated by Neuflow_V2 algorithm. The optical flow of pixel point is calculated as follows:

[0015]

[0016] Among them, is the first frame of the image of the first type is the first frame of the image of the second type is the optical flow field from the first frame of the image of the first type to the first frame of the image of the second type, and the calculation result includes a horizontal component and a vertical component , is the pixel value of the pixel point with coordinate in the first frame of the image of the first type is the pixel value of the pixel point with coordinate in the first frame of the image of the first type in the corresponding pixel point in the first frame of the image of the second type is the pixel value of the pixel point with coordinate in the first frame of the image of the first type in the corresponding pixel point in the first frame of the image of the second type is the pixel value of the pixel point with coordinate in the first frame of the image of the first type in the corresponding pixel point in the first frame of the image of the second type

[0017] then, the optical flow field of the image of the second type between the first frame of the image of the first type and the first frame of the image of the second type is:

[0018]

[0019] wherein, is the optical flow field from the first frame of the image of the first type to the first frame of the image of the second type is the optical flow field from the first frame of the image of the first type to the first frame of the image of the second type

[0020] then, the corresponding position of the pixel point with coordinate in the first frame of the image of the first type in the first frame of the image of the second type is:

[0021]

[0022] wherein, is the coordinate of the corresponding position of the pixel point with coordinate in the first frame of the image of the first type in the first frame of the image of the second type is the horizontal coordinate of the corresponding point of the pixel point with coordinate in the first frame of the image of the first type in the first frame of the image of the second type is the vertical coordinate of the corresponding point of the pixel point with coordinate in the first frame of the image of the first type in the first frame of the image of the second type is the vertical coordinate of the corresponding point of the pixel point with coordinate in the first frame of the image of the first type in the first frame of the image of the second type is the vertical coordinate of the corresponding point of the pixel point with coordinate in the first frame of the image of the first type in the first frame of the image of the second type is the vertical coordinate of the corresponding point of the pixel point with coordinate in the first frame of the image of the first type in the first frame of the image of the second type

[0023] The optical flow estimation of the image of the second type is based on that the optical flow path from the image of the first type to the image of the second type is half of the optical flow path between two adjacent images of the first type.

[0024] As a preferred technical solution of the present application, the imaging quality is judged by information entropy and contrast analysis. ​​​​

[0025] Compared with the prior art, the present application has the advantages that: the present application obtains images with visible light image frames and near-infrared image frames alternately interlaced from the same sensor by using the filter cyclic switching mechanism, and then obtains images with higher imaging quality by judging the imaging quality of the two kinds of images, and estimates the pixel motion of the low imaging quality image by using the half optical flow of the optical flow field formed by the high imaging quality image frames of the adjacent two frames, so that the visible light image and the near-infrared image are accurately matched in terms of spatial parameters such as position, scale and angle, the complex mapping transformation matrix is converted into simple optical flow estimation, and the visible light image and the near-infrared image are quickly, efficiently and accurately registered, so that the registered image with clear picture and unified spatial parameters is obtained. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 The installation position diagram of the filter cyclic switching mechanism of the present application is shown in the figure.

[0027] Figure 2 The structure diagram of the filter cyclic switching mechanism of the first embodiment of the present application is shown in the figure.

[0028] Figure 3 The circuit connection principle diagram of the first embodiment of the present application is shown in the figure.

[0029] Figure 4 The flow chart of the automatic registration method of the first embodiment of the present application is shown in the figure.

[0030] Figure 5 The first frame of visible light image taken by the first embodiment of the present application is shown in the figure.

[0031] Figure 6 The first frame of near-infrared image taken by the first embodiment of the present application is shown in the figure.

[0032] Figure 7 The second frame of visible light image taken by the first embodiment of the present application is shown in the figure.

[0033] Figure 8 The synthesized image of the visible light image and the near-infrared image before registration of the first embodiment of the present application is shown in the figure.

[0034] Figure 9 The synthesized image of the visible light image and the near-infrared image after registration of the first embodiment of the present application is shown in the figure.

[0035] Figure 10 The structure diagram of the filter cyclic switching mechanism of the second embodiment of the present application is shown in the figure.

[0036] Figure 11 The circuit connection principle diagram of the second embodiment of the present application is shown in the figure.

[0037] Figure 12 Flow chart of automatic registration method for the second embodiment of the present application;

[0038] Figure 13 Structure diagram of filter cycle switching mechanism for the third embodiment of the present application.

[0039] In the figure: 1, housing; 101, slide; 2, push-pull electromagnet; 3, sliding groove; 4, traction sliding block; 401, first toggle pin; 5, toggle lever; 501, pivot; 502, first toggle slot; 503, second toggle slot; 6, filter loading piece; 601, second toggle pin; 602, intermittent dial; 7, infrared cut-off filter; 8, single near-infrared pass filter; 9, first photoelectric sensor; 10, second photoelectric sensor; 11, lens; 12, filter cycle switching mechanism; 13, control unit; 14, imaging sensor; 15, dial wheel. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0041] Embodiment 1

[0042] As Figures 1 to 3As shown, the embodiment discloses a first embodiment of the application, a filter cycle switching mechanism 12 is installed in front of the lens 11 of the camera equipment, the shell 1 of the filter cycle switching mechanism 12 is connected with the shell of the lens 11, the shell 1 is provided with a light inlet hole corresponding to the position of the lens 11 and matching in size, the shell 1 is provided with a sliding channel 101, the sliding channel 101 is provided with a push-pull electromagnet 2, the telescopic rod end of the push-pull electromagnet 2 is provided with a traction sliding block 4, the traction sliding block 4 is slidingly connected in the sliding channel 101, the traction sliding block 4 is provided with a first shift pin 401, the shell 1 is further provided with a rectangular sliding groove 3, the sliding groove 3 is slidingly connected with a filter loading piece 6, the filter loading piece 6 is a movable sliding block, the filter loading piece 6 is loaded with an infrared cut-off filter 7 and a single near-infrared passing filter 8, in order to drive the filter loading piece 6 to move, the filter loading piece 6 is provided with a second shift pin 601, the sliding groove 3 is further provided with a rotating shaft 501, the rotating shaft 501 is hingedly connected with a shift rod 5, the shift rod 5 is a strip-shaped structure, the shift rod 5 is provided with a first shift groove 502 and a second shift groove 503 at both ends, the first shift pin 401 is slidingly contacted in the first shift groove 502, and the second shift pin 601 is slidingly contacted in the second shift groove 503. The telescopic rod of the push-pull electromagnet 2 can push the traction sliding block 4 to slide, and then the shift rod 5 drives the filter loading piece 6 to move, when the push-pull electromagnet 2 is pushed out by a set distance, the infrared cut-off filter 7 is coaxial with the lens 11, at this time, the right end of the filter loading piece 6 moves to the right side inner wall of the sliding groove 3, when the push-pull electromagnet 2 is pulled back by a set distance, the single near-infrared passing filter 8 is coaxial with the lens 11, at this time, the left end of the filter loading piece 6 moves to the left side inner wall of the sliding groove 3. In order to detect the position of the filter loading piece 6, a first photoelectric sensor 9 and a second photoelectric sensor 10 are respectively installed at the left side inner wall and the right side inner wall of the sliding groove 3. The control unit 13 of the camera equipment is electrically connected with the push-pull electromagnet 2, the first photoelectric sensor 9 and the second photoelectric sensor 10.

[0043] The embodiment discloses a dual-light imaging automatic registration method of a near-infrared and visible light shared sensor, and the technical scheme is as follows: Figure 4 As shown, the method comprises the following steps:

[0044] Step 1, a shooting signal is started, the shooting day is a sunny day, the starting shooting time is 10 o'clock in the morning, the control unit 13 synchronously starts the imaging sensor 14 of the camera equipment and the filter cycle switching mechanism 12; the camera equipment starts to perform image acquisition work, and the filter cycle switching mechanism 12 starts to act; the control unit 13 continuously switches the current direction of the push-pull electromagnet 2, changes the push-pull state of the push-pull electromagnet 2, so that the filter loading piece 6 reciprocally slides in the sliding groove 3, the filter in front of the lens 11 is switched, and the switching frequency of the filter is the same as the video recording frame rate;

[0045] Step 2, the filter cycle switching mechanism 12 makes the infrared cut-off filter 7 and the single near-infrared pass filter 8 alternately pass through the lens 11; so that there is a near-infrared image between every two adjacent visible light images;

[0046] Step 3, the imaging quality of the visible light image and the near-infrared image is judged by information entropy and contrast analysis:

[0047]

[0048] Wherein, represents the final image quality index value, the higher the index value represents the higher the image quality; and are weight coefficients (0 + =1), in this embodiment, it is assumed that =0.4, =0.6;

[0049] H represents information entropy, which is used to measure the uncertainty of the pixel gray scale distribution in the image, the higher the entropy value represents the richer the image contains the information, the richer the image information of the higher quality, and the calculation formula is as follows:

[0050]

[0051] Wherein, L is the image gray scale, is the probability of the pixel of the gray scale i appearing in the image, and the calculation formula is as follows:

[0052]

[0053] Wherein, is the number of pixels with the gray value i, is the total number of pixels in the image;

[0054] In the image quality index value calculation formula, represents the contrast, which reflects the dispersion degree of the pixel gray scale, the larger the variance, the higher the contrast, and the image with high quality should have high contrast:

[0055]

[0056] Wherein g(x,y) is the gray value of the pixel (x,y), is the average gray value of the image.

[0057] In this embodiment, the light condition is good, the imaging quality of the visible light image is higher than that of the near-infrared image, the visible light image is marked as a type of image, and the near-infrared image is marked as a type of image; there is a near-infrared image between every two adjacent visible light images;

[0058] Step 4: Use Neuflow_v2 algorithm to calculate the coordinates in the visible light image. The optical flow field of the pixel point is calculated as follows:

[0059]

[0060] in, For the Frame visible light image to the The optical flow field of the frame visible light image, the calculation results include the horizontal component and vertical component , For the Coordinates in the frame visible light image The pixel value of the pixel, For the Coordinates in the frame visible light image Pixel point The pixel value of the corresponding pixel in the frame visible light image;

[0061] Then, Frame visible light image and The optical flow field of the near-infrared image sandwiched between frames of visible light images is:

[0062]

[0063] in, For the Frame visible light image to the Optical flow field of frame near infrared image;

[0064] Then, Coordinates in the frame visible light image Pixel point The corresponding position in the frame near infrared image is:

[0065]

[0066] in, For the Coordinates in the frame visible light image Pixel point The coordinates of the corresponding position in the frame near-infrared image, For the Coordinates in the frame visible light image Pixel point The horizontal coordinate of the corresponding point in the frame visible light image, For the Coordinates in the frame visible light image Pixel point Vertical coordinate of corresponding point in frame visible light image

[0067] Physical meaning of step 4: the pixel of near-infrared frame is determined by Frame pixel motion Position after time is determined, and half of optical flow needs to be subtracted when reverse mapping;

[0068] The estimated near-infrared image in two adjacent visible light images is obtained by the above optical flow estimation method;

[0069] Step 5, register the visible light image and the estimated near-infrared image and output.

[0070] Image acquisition is performed using the method, and image comparison before and after registration is performed, as shown in Figures 5 to 9 , it can be seen from the comparison of Figure 8 and Figure 9 that the method can effectively improve the output accuracy of the image.

[0071] Example 2

[0072] As shown in Figure 10 , Figure 11 , the filter cycle switching mechanism 12 in the shell 1 is provided with a driving motor, the output shaft of the driving motor is key connected with a filter loading piece 6, the filter loading piece 6 is a disc structure, and is shaft connected on the shell 1, six groups of filters are mounted on the filter loading piece 6, which are three groups of infrared cut-off filters 7 and three groups of single near-infrared passing filters 8, the infrared cut-off filters 7 and the single near-infrared passing filters 8 are arranged in an annular array and alternately arranged, and the driving motor and the control unit 13 are electrically connected.

[0073] As shown in Figure 12 , the difference between the dual-light imaging automatic registration method of the near-infrared and visible light shared sensor of the present embodiment and the dual-light imaging automatic registration method of example 1 is:

[0074] In step 1, the shooting time is 2 a.m., when the filter cycle switching mechanism 12 is in action, the control unit 13 controls the intermittent operation of the driving motor to drive the intermittent rotation of the filter loading piece 6 to switch the filter in front of the lens 11;

[0075] In steps 3 and 4, the first type of image is a near-infrared image, and the second type of image is a visible light image.

[0076] Example 3

[0077] As shown in Figure 12 , the difference between the present embodiment and example 2 is that the filter loading piece 6 is provided with an intermittent dial 602, the intermittent dial 602 has a dial slot, a dial wheel 15 is hinged in the shell 1, the dial wheel 15 is connected with a driving motor, the dial wheel 15 has a dial rod 5, and the dial rod 5 is slidingly connected in the dial slot.

[0078] The automatic registration method for bi-optical imaging using a common sensor for near-infrared and visible light in this embodiment differs from the automatic registration method for bi-optical imaging in Example 2 in that:

[0079] In step 1, when the filter cycle switching mechanism 12 is in action, the control unit 13 controls the driving motor to run continuously, and the dial wheel 15 rotates continuously. When the lever 5 enters the slot of the filter loader 6, the filter loader 6 can be driven to rotate. When the lever 5 leaves the slot, the filter loader 6 stops rotating, thereby achieving an intermittent rotation effect.

[0080] The circuits and mechanical connections involved in the present invention are conventional means used by those skilled in the art, and technical inspiration can be obtained through limited experiments, and they are common knowledge.

[0081] The control unit 13 of the present invention adopts the Rockchip RV1126 quad-core processor. The connection method between the control unit 13 and the control device can be obtained by those skilled in the art by referring to textbooks or technical manuals, and belongs to the prior art; the components not described in detail in this article are prior art.

[0082] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for automatic registration of bi-optical imaging using a common sensor for near-infrared and visible light, characterized in that: The following steps are involved: Step 1: Start the shooting signal, synchronously trigger the camera device and the filter cycle switching mechanism; the camera device starts to perform image acquisition operations, and the filter cycle switching mechanism starts to operate; Step 2: The filter cycle switching mechanism allows the infrared cutoff filter and the single near-infrared pass filter to alternately pass through the lens; thereby, there is a near-infrared image frame between every two adjacent visible light image frames; Step 3: Determine the imaging quality of the visible light image and the near-infrared image, and mark the visible light image and the near-infrared image with better imaging quality as Class 1 images, and mark the images with worse imaging quality as Class 2 images; a Class 2 image is sandwiched between every two adjacent Class 1 images. Step 4: Calculate the optical flow of the first-class image and estimate the optical flow of the second-class image based on the calculation result, thereby adjusting the pixel position of the second-class image and marking the adjusted image as a quasi-first-class image; Step 5: Register and output the first-class image and the quasi-first-class image.

2. The automatic registration method for bi-optical imaging using a common near-infrared and visible light sensor according to claim 1, characterized in that: In step 1, the filter cycle switching mechanism is a reciprocating mechanism or a rotating mechanism.

3. The automatic registration method for bi-optical imaging using a common near-infrared and visible light sensor according to claim 1, characterized in that: In step 3, when the lighting conditions are good, the imaging quality of the visible light image is high, and the visible light image is marked as a first-class image, and the near-infrared image is marked as a second-class image; when the lighting conditions are poor, the imaging quality of the near-infrared image is high, and the near-infrared image is marked as a first-class image, and the visible light image is marked as a second-class image.

4. The automatic registration method for bi-optical imaging using a common near-infrared and visible light sensor according to claim 1 or 3, characterized in that: In step 4, the coordinates in a class of images The optical flow of a pixel is calculated as follows: ； in, For the Frame 1 image to frame The optical flow field of a frame type image, the calculation results include the horizontal component and vertical component , For the Frame-type image coordinates The pixel value of the pixel, For the Frame-type image coordinates Pixel point The pixel value of the corresponding pixel in the frame type image; Then, Frame 1 image and The optical flow field of the second type of image sandwiched between the first type of images is: ； in, For the Frame 1 image to frame Optical flow field of frame two images; Then, Frame-type image coordinates Pixel point The corresponding estimated position in the frame class II image is: ； in, For the Frame-type image coordinates Pixel point The coordinates of the estimated position in the frame class II image, For the Frame-type image coordinates Pixel point The horizontal coordinate of the corresponding point in the frame type image, For the Frame-type image coordinates Pixel point The vertical coordinate of the corresponding point in the frame image.

5. The automatic registration method for bi-optical imaging using a common near-infrared and visible light sensor according to claim 3, characterized in that: The imaging quality is judged through information entropy and contrast analysis.

Images