Multi-focus image fusion method, storage medium and computer
Through the multi-focus image fusion method, the depth of field map is generated using the clarity evaluation function and center of gravity sorting, which solves the image blur problem caused by the lens depth of field limitation, and realizes the full focus fusion of multiple images, improving information utilization and image quality.
Patent Information
- Application Number
- CN202210109832.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-29
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-01-29
AI Technical Summary
During the real-life imaging process, the lens is limited by the depth of field and cannot focus all objects in the field of view at the same time, resulting in local blurring of the captured images and it is difficult to obtain the complete information it contains.
Provide a multi-focus image fusion method, which calculates the sharpness of each original image through the definition evaluation function, obtains the sharpness image and significance image, sorts and registers the images based on the center of gravity, generates the depth of field map and reconstructs the clear fusion image.
It has realized the integration of multiple images with different focus areas in the same scene into a completely clear image in each area, which has improved the information utilization rate of the image and is suitable for industrial machine vision, medical imaging diagnosis, and aviation target recognition.
Smart Images

Figure CN114612352B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a multi-focus image fusion method, a storage medium and a computer. Background Art
[0002] In modern society, images are an important source of information for humans, and the demand for image clarity is getting higher and higher. However, in the actual imaging process, the lens is often limited by the depth of field and cannot focus on all targets in the field of view at the same time. Therefore, the captured images always have local blur, making it difficult to use them to obtain the complete information they contain.
[0003] Multi-focus image fusion technology can fuse multiple images with different focus areas in the same scene into an image with completely clear areas, providing a high-quality information source for subsequent image analysis and processing, while greatly improving the information utilization rate of the image. This advantage has played a huge role in industrial machine vision, medical imaging diagnosis, aerial target recognition and other fields.
[0004] In the actual multi-focus image fusion process, it is necessary to preserve all the useful information in the original image as much as possible, consider the interference factors of the shooting environment, enhance the robustness of the method, and consider the speed of practical application. Therefore, the research on multi-focus image fusion method is of great significance. Summary of the invention
[0005] The object of the present invention is to provide a multi-focus image fusion method, a storage medium and a computer, which can fuse multiple original images with the same scene but different focuses to form a clear fused image.
[0006] To achieve the purpose of the invention, according to one aspect of the present invention, a multi-focus image fusion method is provided, which includes: providing multiple original images of the same scene but different focuses; using a clarity evaluation function to calculate the sharpness of each pixel of each original image to obtain a clarity image of each original image, and obtaining a saliency image of each original image based on the clarity image of each original image; calculating the center of gravity of each image based on each clarity image, and sorting the multiple original images and the multiple saliency images according to the center of gravity of each image to obtain an original image set and a saliency image set; performing pairwise registration between adjacent images in the original image set and between adjacent images in the saliency image set according to the image sorting to obtain a registered saliency image set and a registered original image set; generating a depth map according to the registered saliency image set, wherein the depth map records the image index number of each pixel position; and reconstructing a clear fused image according to the image index number of each pixel position in the depth map.
[0007] According to another aspect of the present invention, the present invention provides a storage medium storing program instructions, wherein the multi-focus image fusion method described above is executed in the program execution.
[0008] According to yet another aspect of the present invention, the present invention provides a computer, comprising a processor and a memory, wherein the memory stores program instructions, and the processor runs the program instructions to execute the multi-focus image fusion method described above.
[0009] Compared with the existing technology, the present invention can fuse multiple images with different focus areas in the same scene into an image with completely clear areas, providing a high-quality information source for subsequent image analysis and processing, while greatly improving the information utilization rate of the image. This advantage has played a huge role in industrial machine vision, medical imaging diagnosis, aerial target recognition and other fields. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 is a schematic diagram of a flow chart of a multi-focus image fusion method in one embodiment of the present invention;
[0011] Figure 2 This is an example of the multi-focus image fusion method in the present invention, which shows three original images, three saliency images, a depth map and a fused image. DETAILED DESCRIPTION
[0012] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.
[0013] The present invention provides a multi-focus image fusion method, which can fuse multiple original images with the same scene but different focuses to form a clear fused image.
[0014] Figure 1 FIG. 1 is a flow chart of a multi-focus image fusion method 100 in one embodiment of the present invention. Figure 1 As shown, the multi-focus image fusion method 100 includes the following steps.
[0015] Step 110 , providing a plurality of original images with the same scene but different focuses.
[0016] In one embodiment, multiple original images may be taken by a camera at the same scene. Due to different focuses during shooting, different areas of the multiple original images are relatively cleaner. Figure 2As shown, three original images 210 with the same scene but different focuses are shown. The same screw is shown in the figure. Due to the different focuses, the clear areas in different original images are different. In addition, since the camera may shake or move during shooting, the three original images may not be completely aligned, so image alignment is required in the subsequent image fusion process.
[0017] Step 120 , using a clarity evaluation function to calculate the sharpness of each pixel of each original image to obtain a clarity image of each original image, and based on the clarity image of each original image to obtain a saliency image of each original image.
[0018] In one embodiment, the Tenengrad function is used as a sharpness evaluation function. The Tenengrad function is a gradient-based image sharpness evaluation function, because it is generally believed that the focused area of the image has a sharper edge, that is, has a larger gradient function value. Therefore, it is more appropriate to calculate the sharpness of the pixel based on the Tenengrad function.
[0019] Specifically, the Tenengrad function first uses the Sobel operator to extract the horizontal and vertical gradient values of the original image. x and G y They are the convolution kernels in the horizontal and vertical directions respectively:
[0020]
[0021]
[0022] Then the sharpness of the definition image at the pixel position (x, y) is:
[0023]
[0024] I is the original image, and I(x,y) is the pixel value of the original image at the pixel position (x,y).
[0025] After that, each clarity image is grayscale stretched to obtain image L. Considering the diffusion factor of the unfocused area, image L is Gaussian filtered to obtain the saliency image S of each original image.
[0026] Figure 2 Three saliency images 220 corresponding to the three original images are shown.
[0027] Step 130 , calculating the centroid of each image based on each definition image, and sorting the multiple original images and the multiple saliency images according to the centroid of each image to obtain an original image set and a saliency image set.
[0028] In one embodiment, calculating the center of gravity of each image based on each clarity image includes: performing grayscale stretching on each clarity image to obtain image L; binarizing image L into a binary image T; calculating the spatial moment of the binary image T to obtain the center of gravity of the binary image, wherein the center of gravity of the binary image is the center of gravity of the image.
[0029] Specifically, the grayscale value of the image L is 0-255, and a fixed threshold can be set to binarize the image L, and the fixed threshold can be 100.
[0030] The calculation formula of the spatial moment of the binary graph T is:
[0031]
[0032] The calculation formula of the centroid P(X,Y) of the binary image is:
[0033]
[0034] T(x,y) is the value of the pixel at position (x, y) in the binary image, m 00 is the zero-order spatial moment, m 01 , is the first-order spatial moment, m 10 is the first-order spatial moment.
[0035] According to the center of gravity of each image, multiple original images and multiple saliency images are sorted so that the distance between the center of gravity of each image and the images before and after it is the shortest. This is equivalent to sorting multiple original images and multiple saliency images in the order of focus movement to improve the accuracy of subsequent pairwise registration of images. If the focus distance between two images is too large, the overlapped focus area will be too blurred, making it difficult to determine whether the registration is completed, which will greatly affect the registration accuracy.
[0036] It should be noted that the center of gravity of each original image and its corresponding saliency image is consistent, so the order of the original images will be consistent with the order of the saliency images, and the center of gravity of each image can be either the center of gravity of the original image or the center of gravity of the saliency image.
[0037] Step 140 , performing pairwise registration between adjacent images in the original image set and between adjacent images in the salient image set according to the image order, so as to obtain a registered salient image set and a registered original image set.
[0038] In practical industrial applications, the time complexity of the algorithm needs to be considered. If feature point matching is used to register two images, the speed is slow. Considering that in actual shooting, due to the fixation of the mechanical arm, the shooting field of view will not change in a large range, the translation difference method can be used to perform pairwise registration between adjacent images in the original image set and the salient image set according to the image sorting, wherein the translation difference method is a method of using multiple image translations in each direction to calculate the minimum difference between two adjacent images to obtain the translation transformation matrix. Specifically, one of the adjacent images in the original image set is translated multiple times in each direction, and the directions include four directions of x-, y-, x+, and y+, including simultaneous translation of x and y; after each translation, the absolute value sum of the pixel grayscale difference of the image overlapping area of the adjacent images in the original image set is calculated, and the one with the smallest absolute value sum within the specified range is the best match, so as to obtain the translation transformation matrix, and the adjacent two original images are registered according to the translation transformation matrix, and the adjacent two salient images are registered, so as to obtain the registered salient image set and the registered original image set.
[0039] Step 150: Generate a depth map based on the registered saliency image set, wherein the depth map records the image index number of each pixel position. The image index number is the index number in the saliency image set or the original image set, that is, the number of the original image or saliency image.
[0040] In one embodiment, the sharpness value of each pixel position in the saliency image set on each saliency image is compared, and the image index number of the saliency image corresponding to the maximum value is taken as the value of the pixel position in the depth image. In one embodiment, considering the diffusion factor of the out-of-focus area, the formed depth image is expanded so that the edge of the clear area is easy to form an oscillating area in the out-of-focus image and is not misjudged as a focused area. Preferably, the formed depth image is further converted into a floating point type, and the mean filtering edge feathering is performed, and the floating point part index number is increased to reduce the stiffness of subsequent fusion.
[0041] Figure 2 FIG. 2 shows a depth map 230 generated from three saliency images 220 .
[0042] Step 160: reconstruct a clear fused image according to the image index number of each pixel position in the depth image.
[0043] In one embodiment, when the image index number of the pixel position (x, y) is an integer n, the pixel value of the pixel position (x, y) is the value of the pixel position of the nth original image in the registered original image set; when the image index number of the pixel position (x, y) is a floating point number f, two image index numbers (i+1) and i are obtained by rounding up and down respectively, and the values of the i+1th and ith original images at the pixel position are superimposed according to the weights of (fi) and (1-f+i) to achieve the effect of soft edge transition and fusion, thereby forming a pixel-level fully focused and clear image.
[0044] Figure 2 , a clear fused image 240 obtained based on the depth map 230 and the original image 210 is shown.
[0045] The present invention fuses multiple images with the same field of view but different focuses, and can generate a high-quality fully focused image based on clarity analysis and compatibility with interference factors caused by slight deviation of the field of view for subsequent analysis and observation.
[0046] According to another aspect of the present invention, the present invention provides a storage medium storing program instructions, which are executed to execute the multi-focus image fusion method 100 described above. For simplicity, the specific content of the multi-focus image fusion method 100 is not repeated here.
[0047] According to another aspect of the present invention, the present invention provides a computer, comprising a processor and a memory, wherein the memory stores program instructions, and the processor runs the program instructions to execute the multi-focus image fusion method 100 described above. For simplicity, the specific content of the image target matching and positioning method is not repeated here.
[0048] In this document, the terms "comprises," "comprising," or any other variations thereof, are intended to cover a non-exclusive inclusion of elements other than those listed and may also include additional elements not expressly listed.
[0049] In this document, the directional words such as front, back, top, and bottom are defined by the positions of the components in the drawings and the positions of the components relative to each other, and are only for the sake of clarity and convenience in expressing the technical solution. It should be understood that the use of the directional words should not limit the scope of protection claimed in this application.
[0050] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0051] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0052] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0053] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0054] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
Claims
1. A multi-focus image fusion method, characterized in that: It includes: Provide multiple original images with the same scene but different focuses; The sharpness of each pixel of each original image is calculated by using a clarity evaluation function to obtain a clarity image of each original image, and a saliency image of each original image is obtained based on the clarity image of each original image; The center of gravity of each image is calculated based on each clarity image, and multiple original images and multiple saliency images are sorted according to the center of gravity of each image to obtain an original image set and a saliency image set; Performing pairwise registration between adjacent images in the original image set and between adjacent images in the salient image set according to the image sorting, so as to obtain a registered salient image set and a registered original image set; Generate a depth map according to the registered saliency image set, wherein the depth map records the image index number of each pixel position; and A clear fused image is reconstructed according to the image index number of each pixel position in the depth map.
2. The multi-focus image fusion method according to claim 1, characterized in that: Using Tenengrad function as the clarity evaluation function, The method of calculating the sharpness of each pixel of each original image by using the clarity evaluation function to obtain the clarity image of each original image includes: The Tenengrad function first uses the Sobel operator to extract the horizontal and vertical gradient values of the original image. x and G y They are the convolution kernels in the horizontal and vertical directions respectively: Then the sharpness of the definition image at the pixel position (x, y) is: I is the original image, and I(x,y) is the pixel value of the original image at the pixel position (x,y).
3. The multi-focus image fusion method according to claim 1, characterized in that: The obtaining of the saliency image of each original image based on the clarity image of each original image comprises: The image L is obtained by grayscale stretching each clarity image, and the image L is Gaussian filtered to obtain the saliency image S of each original image.
4. The multi-focus image fusion method according to claim 1, characterized in that: According to the center of gravity of each image, multiple original images and multiple saliency images are sorted so that the distance between the center of gravity of each image and the images before and after it is the shortest. The calculating the center of gravity of each image based on each definition image includes: Perform grayscale stretching on each definition image to obtain image L; Binarize the image L into a binary image T; Calculate the spatial moment of the binary image T and obtain the centroid of the binary image, where the centroid of the binary image is the centroid of the image.
5. The multi-focus image fusion method according to claim 4, characterized in that: The calculation formula of the spatial moment of the binary graph T is: The calculation formula of the centroid P(X,Y) of the binary image is: T(x,y) is the value of the pixel at position (x, y) in the binary image, m 00 is the zero-order spatial moment, m 01 , is the first-order spatial moment, m 10 is the first-order spatial moment.
6. The multi-focus image fusion method according to claim 1, characterized in that: The translation difference method is used to perform pairwise registration between adjacent images in the original image set and the salient image set according to the image sorting. The translation difference method is a method that uses multiple image translations in various directions to calculate the minimum difference between two adjacent images to obtain the translation transformation matrix.
7. The multi-focus image fusion method according to claim 6, characterized in that: The translation difference method is: One of the adjacent images in the original image set is translated multiple times in directions, where the directions include four directions: x-, y-, x+, and y+; After each translation, the absolute value sum of the pixel grayscale differences in the image overlapping areas of adjacent images in the original image set is calculated. The one with the smallest absolute value sum within the specified range is the best match, so as to obtain the translation transformation matrix. According to the translation transformation matrix, the two adjacent original images are aligned, and the two adjacent saliency images are aligned.
8. The multi-focus image fusion method according to claim 1, characterized in that: Generating a depth map according to the registered saliency image set comprises: Comparing the sharpness values of each pixel position in the saliency image set on each saliency image, and taking the image index number of the saliency image corresponding to the maximum value as the value of the pixel position of the depth image; Taking into account the diffusion factor of the out-of-focus area, the formed depth map is expanded; The depth map is converted into floating point type, the mean filter edge feathering is performed, and the index number of the floating point part is increased.
9. The multi-focus image fusion method according to claim 1, characterized in that: The step of reconstructing a clear fused image according to the image index number of each pixel position of the depth map includes: When the image index number of the pixel position (x, y) is an integer n, the pixel value of the pixel position (x, y) is the value of the pixel position of the nth original image in the registered original image set; When the image index number of the pixel position (x, y) is a floating point number f, it is rounded up and down to obtain two image index numbers (i+1) and i respectively, and the values of the i+1th and ith original images at the pixel position are superimposed according to the weights of (fi) and (1-f+i).
10. A storage medium, characterized in that: It stores program instructions, and the program is executed to execute the multi-focus image fusion method as described in any one of claims 1-9.
11. A computer, characterized in that: It includes a processor and a memory, wherein program instructions are stored in the memory, and the processor runs the program instructions to execute the multi-focus image fusion method as described in any one of claims 1-9.
