A method for long-short focal camera image fusion

Abstract

The application is a kind of long-short focus camera image fusion method, specifically, an original long-focus camera image and an original short-focus camera image are obtained, SIFT features are extracted, the long-focus camera image is down-sampled to obtain a reduced long-focus camera image, SIFT feature extraction is also performed, the long-focus camera image feature set and the short-focus camera image feature set are violently matched to obtain the corresponding feature point relationship set, then the reduced long-focus camera image feature set and the short-focus camera image feature set are violently matched to obtain the corresponding feature point relationship set, in the latter feature relationship set, the feature points in the image are restored to the positions in the original long-focus image, and the corresponding feature point relationship set is updated, the two groups of feature point relationships are combined, the RANSAC algorithm is used to remove poor relationship points, the final matching feature data is optimized, the conversion relationship of the long-short focus image is calculated, and image fusion is performed.

Description

Technical Field

[0001] This invention relates to the technical field of camera image processing, and in particular to a method for image fusion of long and short focal length cameras, which is mainly used in fields or scenarios such as autonomous driving, target detection, and video surveillance. Background Technology

[0002] Short-focus cameras, such as those equipped with an 8mm lens, can typically clearly view images of targets at close ranges of 2-30 meters; long-focus cameras, such as those equipped with a 50mm lens, can clearly view images of targets at around 150 meters. In real-world applications, with the development of technologies such as autonomous driving, vehicle-to-everything (V2X) communication, and intelligent monitoring equipment, the requirements for image visibility and viewing distance are becoming increasingly stringent. This highlights the limitations of both long-focus and short-focus cameras in terms of their visibility and range. Summary of the Invention

[0003] The present invention aims to address the shortcomings of the prior art by providing a method for image fusion between long and short focal length cameras.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] A method for image fusion between long-focus and short-focus cameras, specifically:

[0006] S1. Obtain one original telephoto camera image and one original short-focus camera image. Extract SIFT features from the telephoto camera image and the short-focus camera image respectively to obtain the feature set PL of the telephoto camera image and the feature set PS of the short-focus camera image.

[0007] S2. Downsample the telephoto camera image to obtain a scaled-down telephoto camera image, and perform SIFT feature extraction on the image to obtain the scaled-down telephoto camera image feature set PL_small;

[0008] S3. Perform brute-force matching between the long-focus camera image feature set PL obtained in step S1 and the short-focus camera image feature set PS to obtain the feature relationship set D1 of long-focus and short-focus images.

[0009] S4. Perform brute-force matching between the reduced telephoto camera image feature set PL_small obtained in step S2 and the short-focus camera image feature set PS obtained in step S1 to obtain the feature relationship set D2 of the reduced telephoto camera image and the short-focus camera image.

[0010] S5. In the feature point relationship set D2 of step S4, restore the feature points in the image to their positions in the original telephoto image, and update the corresponding feature point relationship set D2'.

[0011] S6. Use the RANSAC algorithm to remove undesirable relationship points from the feature point relationship set D2 in step S3 and the feature point relationship set D2' in step S5, and optimize to obtain the final feature relationship set D3.

[0012] S7. Through the final feature relationship set D3, the conversion relationship between long and short focal length images is calculated, which can then be used for image fusion.

[0013] In step S2, the telephoto camera image is downsampled, and the downsampling ratio is related to the focal length of the telephoto camera and the short-focal-length camera.

[0014] Let the focal length of the telephoto camera be fl, the focal length of the short-focus camera be fs, the size of the light-collecting chip and the image frame size of the telephoto and short-focus cameras be the same, the horizontal pixel length be w, and the vertical pixel length be h.

[0015] The downsampling ratio of the telephoto camera The size of the zoomed-out telephoto camera image is: horizontal pixel length Vertical pixel length

[0016] In step S5, the pixel coordinates of the feature points involved in the scaled-down telephoto camera image in the feature relationship set D2 are restored to the original size telephoto camera image, and the corresponding pixel positions in the telephoto camera are replaced with the pixel coordinates of the scaled-down telephoto camera in the feature relationship set D2, and updated to the feature relationship set D2'.

[0017] In step S5, the pixel coordinates of feature points in the scaled-down telephoto camera image are restored to the original size telephoto camera image. The specific method is as follows:

[0018] The current pixel's x and y coordinates are (w_s, h_s). Multiplying this by the reciprocal of the sampling scaling factor gives the restoration magnification factor. The horizontal coordinate of the restored pixel is The vertical axis is

[0019] In step S6, the RANSAC algorithm is used to filter out unqualified feature relations in feature relation sets D1 and D2' respectively. Then, the relation sets D1 and D2' are merged, overlapping feature relations are removed, and a new feature relation set D3 is formed.

[0020] In step S7, based on the feature relationship set D3 of the fusion, the homography matrix H of the telephoto camera image and the short-focus camera image is obtained, and the relationship between the telephoto camera image and the short-focus camera image is obtained. Then, the telephoto camera image is fused into the short-focus camera image through the homography matrix H.

[0021] In step S1, Gaussian filtering is applied to the acquired original telephoto camera image and the original short-focus camera image, and the quality and sharpness of the telephoto camera image and the short-focus camera image are optimized according to the V component in the HSV format of the image.

[0022] In step S2, after acquiring the scaled-down telephoto camera image, Gaussian filtering and sharpening are applied to the image to highlight its features.

[0023] The beneficial effects of this invention are: this invention performs target recognition on images captured by a telephoto camera and images captured by a short-focus camera respectively, selects the target area, and combines the telephoto camera image with the short-focus camera image. The fused image can show targets identified at close range as well as targets identified at a distance. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the operation of the present invention;

[0025] The following will describe in detail, with reference to the accompanying drawings, embodiments of the present invention. Detailed Implementation

[0026] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are for illustrative purposes only and are not intended to limit the scope of the invention. The invention is described more specifically in the following paragraphs by way of example with reference to the accompanying drawings. The advantages and features of the invention will become clearer from the following description. It should be noted that the drawings are in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the invention.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0029] A method for image fusion between long-focus and short-focus cameras, the specific steps of which are as follows: Figure 1 As shown, it includes:

[0030] S1. Acquire one original telephoto camera image and one original short-focus camera image. Perform Gaussian filtering on the acquired original telephoto camera image and the original short-focus camera image. Optimize the quality and sharpness of the telephoto camera image and the short-focus camera image based on the V component in the HSV format of the image. Extract SIFT features from the optimized telephoto camera image and the short-focus camera image respectively to obtain the feature set PL of the telephoto camera image and the feature set PS of the short-focus camera image.

[0031] The SIFT feature algorithm not only maintains invariance to rotation, scaling, and brightness changes, but also maintains a certain degree of stability to viewpoint changes, affine transformations, and noise, thus avoiding some problems caused by poor image acquisition.

[0032] S2. Downsample the telephoto camera image to obtain a reduced telephoto camera image. Perform Gaussian filtering and sharpening on the image to highlight the features of the reduced image. Perform SIFT feature extraction on the processed image to obtain the feature set PL_small of the reduced telephoto camera image.

[0033] When downsampling images from telephoto cameras, the downsampling ratio is related to the focal length of the telephoto or short-focal-length camera.

[0034] Let the focal length of the telephoto camera be fl, the focal length of the short-focus camera be fs, the size of the light-collecting chip and the image frame size of the telephoto and short-focus cameras be the same, the horizontal pixel length be w, and the vertical pixel length be h.

[0035] The downsampling ratio of the telephoto camera The size of the zoomed-out telephoto camera image is: horizontal pixel length Vertical pixel length

[0036] S3. Perform brute-force matching between the long-focus camera image feature set PL obtained in step S1 and the short-focus camera image feature set PS to obtain the feature relationship set D1 of long-focus and short-focus images.

[0037] S4. Perform brute-force matching between the reduced telephoto camera image feature set PL_small obtained in step S2 and the short-focus camera image feature set PS obtained in step S1 to obtain the feature relationship set D2 of the reduced telephoto camera image and the short-focus camera image.

[0038] S5. In the feature relationship set D2 of step S4, the feature point pixel coordinates involved in the reduced telephoto camera image are restored to the original size telephoto camera image and the corresponding pixel positions in the telephoto camera are replaced with the feature point pixel coordinates of the reduced telephoto camera in the feature relationship set D2, and updated to the feature relationship set D2'.

[0039] The method for restoring the pixel coordinates of feature points in a scaled-down telephoto camera image to the original telephoto camera image is as follows:

[0040] The current pixel's x and y coordinates are (w_s, h_s). Multiplying this by the reciprocal of the sampling scaling factor gives the restoration magnification factor. The horizontal coordinate of the restored pixel is The vertical axis is

[0041] S6. Use the RANSAC algorithm to filter out the unqualified feature relations in feature relation sets D1 and D2' respectively. Then merge the relation sets D1 and D2', remove overlapping feature relations, and merge to form a new feature relation set D3.

[0042] S7. Based on the feature relationship set D3 of the fusion, calculate the homography matrix H of the telephoto camera image and the short-focus camera image to obtain the relationship between the telephoto camera image and the short-focus camera image. Then, fuse the telephoto camera image into the short-focus camera image through the homography matrix H.

[0043] Telephoto and short-focus cameras have parallel optical axes, meaning that objects in a short-focus camera's image completely encompass objects in the telephoto camera's image. Telephoto cameras can see distant targets and objects clearly, but their field of view is smaller. Short-focus cameras have a larger field of view, but can only see nearby targets and objects clearly. For example, in target recognition applications, using images captured only by a short-focus camera can identify targets within a larger field of view, but it usually cannot identify targets farther away. Conversely, using images captured only by a telephoto camera can identify targets farther away, but the recognizable field of view is smaller.

[0044] This invention performs target recognition on images captured by a telephoto camera and images captured by a short-focus camera, respectively. After selecting the target area, the telephoto camera image and the short-focus camera image are combined. The fused image can show targets that are identified at close range as well as targets that are identified at a distance.

[0045] This invention does not require a fixed calibration plate or calibration device. It can obtain the correspondence between telephoto camera images and short-focus camera images by comparing the features of the original image and the features of the downsampled telephoto image, and then fuse the images based on the correspondence.

[0046] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any improvements made using the inventive concept and technical solution of the present invention, or direct application to other occasions without modification, are all within the protection scope of the present invention.

Claims

1. A method of long-short focal camera image fusion, characterized in that, Specifically: S1, obtain an original long-focus camera image and an original short-focus camera image, extract SIFT features from the long-focus camera image and the short-focus camera image respectively to obtain a long-focus camera image feature set PL and a short-focus camera image feature set PS; S2, downsample the long-focus camera image to obtain a reduced long-focus camera image, and perform SIFT feature extraction on the image to obtain a reduced long-focus camera image feature set PL_small; S3, perform brute-force matching between the long-focus camera image feature set PL obtained in step S1 and the short-focus camera image feature set PS to obtain a feature relationship set D1 of the long-focus and short-focus images; S4, perform brute-force matching between the reduced long-focus camera image feature set PL_small obtained in step S2 and the short-focus camera image feature set PS obtained in step S1 to obtain a feature relationship set D2 of the reduced long-focus camera image and the short-focus camera image; S5, in the feature point relationship set D2 of step S4, restore the feature points in the image to their positions in the original long-focus image and update the corresponding feature point relationship set D2'; S6, use the RANSAC algorithm to remove poor relationship points from the feature point relationship set D2 in step S3 and the feature point relationship set D2' in step S5 respectively to obtain a final feature relationship set D3; S7, calculate the conversion relationship of the long-focus and short-focus images based on the final feature relationship set D3, and then use it for image fusion.

2. The method of claim 1, wherein, In step S2, the long-focus camera image is downsampled, and the downsample ratio is related to the focal lengths of the long-focus camera and the short-focus camera; Let the focal length of the long-focus camera be fl, the focal length of the short-focus camera be fs, the light sensing chip size and the image frame size of the long-focus camera and the short-focus camera be consistent, the horizontal pixel length be w, and the vertical pixel length be h; then the long camera down-sampling ratio The long camera image size after down-sampling is: horizontal pixel length vertical pixel length 3. The method of claim 2, wherein, In step S5, the feature point pixel coordinates involved in the reduced long-focus camera image in the feature relationship set D2 are restored to the original size long-focus camera image, and the corresponding pixel positions in the long-focus camera are restored to replace the feature point pixel coordinates of the reduced long-focus camera in the feature relationship set D2, and updated to the feature relationship set D2'.

4. The method of claim 3, wherein, In step S5, the feature point pixel coordinates involved in the reduced long-focus camera image are restored to the original size long-focus camera image, and the specific method is: The horizontal and vertical coordinates of the current pixel point are (w_s, h_s), and the reciprocal of the downsampling ratio coefficient is called the restoration magnification coefficient The horizontal coordinate of the restored pixel point is The vertical coordinate is 5. The method of long-short focal length camera image fusion according to claim 4, wherein, In step S6, use the RANSAC algorithm to filter out unqualified feature relationships in the feature relationship sets D1 and D2', then merge the relationship sets D1 and D2', remove the coincident feature relationships, and fuse to form a new feature relationship set D3.

6. The method of long-short focal length camera image fusion according to claim 5, wherein, In step S7, based on the fused feature relationship set D3, the homography matrix H of the long-focus camera image and the short-focus camera image is calculated to obtain the mutual relationship of the long-focus camera image and the short-focus camera image, and then the long-focus camera image is fused into the short-focus camera image through the homography matrix H.

7. The method of long-short focal length camera image fusion according to claim 6, wherein, In step S1, perform Gaussian filtering on the obtained original long-focus camera image and original short-focus camera image, and optimize the long-focus camera image and short-focus camera image quality and clarity based on the V component in the HSV format.

8. The method of long-short focal length camera image fusion according to claim 7, wherein, In step S2, after the zoomed-out long-focus camera image is acquired, Gaussian filtering and sharpening are respectively performed on the image to highlight the features of the zoomed-out image.

Images