A multi-process-based focus stack image fusion method and system

Abstract

The application discloses a kind of based on multi-process focus stack image fusion method and system, belong to industrial blemish detection and photography field, including the following steps: quickly acquire the mth, m=1,2,......,N, N is the total number of image, the image of the same object is shot;In the application, traditional image edge detection method is changed into multi-image gradient edge detection method based on multi-process, so as to realize the synthesis of focus stack image, which greatly improves the running time of synthesis, especially when CPU is in multi-core condition, while the method also has the processing process for image, which can effectively improve the quality of acquired image, thereby effectively improving the subsequent focus stacking effect, ensuring the quality of final focus stack image, and in the end, image evaluation process is also provided, which can learn high-quality images according to deep learning algorithm, compare and evaluate, and evaluate the overall focus stack image fusion method, which is powerful and suitable for promotion.

Description

Technical Field

[0001] This invention belongs to the field of industrial defect detection and photography technology, and particularly relates to a multi-process focus stacking image fusion method and system. Background Technology

[0002] With the development of image processing technology, focus stacking technology has gradually become popular. During the photography process, in order to focus and make the target object sharp, some parts of the overall image will inevitably be blurred, making it difficult to obtain an image where the entire field of view is sharp. However, focus stacking technology can overcome this difficulty and achieve an overall sharp image.

[0003] The key feature of focus stacking technology lies in the accuracy of detecting the edges of target objects. To ensure high precision in the synthesized image, this inevitably leads to complex programs, numerous loops, and ultimately, excessively long program execution times. This poses a significant challenge to image synthesis using focus stacking technology. Therefore, a solution based on multi-process data processing becomes a feasible approach.

[0004] Prior art 1 discloses a focal stacking method for retinal imaging (CN111093470A), comprising illuminating the interior of the eye with one or more light beams from a light source, and said illuminating being configured to trigger a change in the pupil of the eye. During a predetermined time frame in which the pupil width changes in response to said illuminating, an image sensor captures a sequence of images of light reflected from the interior of the eye. A processing device combines the images in the image sequence to form a composite image having a depth of field greater than that of each image in the image sequence. The depth of field of each image corresponds to the pupil width at the time each image is captured.

[0005] Prior art 2 discloses an automatic gimbal for multi-focus stacked macro photography (CN112212142A). This invention discloses an automatic gimbal for multi-focus stacked macro photography, including a fixed plate. The top of the fixed plate is provided with a sliding groove. A slide block is slidably installed inside the sliding groove. A clamp is connected to the top of the slide block. A fixed plate is fixedly installed on one side of the top of the clamp. A clamp plate is installed on the side of the clamp away from the fixed plate. The clamp plate and the fixed plate are parallel to each other. A lead screw is fixedly installed inside the sliding groove. The lead screw passes through the left and right sides of the slide block. The fixed plate is provided with a mounting groove at one end near the lead screw. A first motor is fixedly installed in the mounting groove. The first motor is connected to the lead screw for transmission.

[0006] The prior art 3 discloses a feature selection method based on the Laplacian operator (CN104408480A). This invention discloses a feature selection method based on the Laplacian operator, which takes into account both the association between samples and class labels and preserves the interdependencies between samples.

[0007] The prior art 4 discloses a method for extracting illumination-invariant facial features using logarithmic transformation and the Laplacian operator (CN106934399A). The method transforms the facial image to the logarithmic domain and uses the Laplacian operator to sharpen the facial image, extracting the detailed features of the face while eliminating the illumination component. The result is the illumination-invariant facial features that need to be extracted.

[0008] The prior art 5 discloses a method for extracting illumination-invariant facial features based on the Laplacian operator (CN106934343A). The Laplacian operator is used to sharpen the facial image and extract the facial detail features. The corresponding pixels of the original image are divided to eliminate the slowly changing facial illumination part. The result is the illumination-invariant facial features that need to be extracted.

[0009] The prior art 6 discloses a method and system for image enhancement and denoising based on the Laplacian operator (CN113034376A). The method involves reading the original image A to obtain the maximum number of rows m and columns n; performing a 3x3 neighborhood Laplacian operator template operation on each row and column of the middle pixels to obtain an initial output; binarizing the initial output, assigning a value of 1 to points with absolute values ​​higher than a pre-defined parameter a, and assigning a value of 0 to points lower than a, to obtain a binarized initial edge image; searching for the number of connected pixels for points assigned a value of 1 in the previous step, identifying points with values ​​lower than a pre-defined second parameter b as noise points, and using a Gaussian module to denoise the original pixels; identifying points with values ​​higher than b as edge points for enhancement, increasing the original pixel value by a specified number of pixels if the corresponding A(i,j) is greater than 0, and decreasing the original pixel value by a specified number of pixels if the corresponding A(i,j) is less than 0, to obtain the final enhanced image.

[0010] The prior art 7 discloses a method and system for image edge extraction based on the Laplacian operator (CN112464962A). The method involves reading the original image A to obtain the maximum number of rows m and columns n; performing a 3x3 neighborhood Laplacian operator template operation on each row and column of the middle pixels to obtain an initial output; binarizing the initial output, assigning a value of 1 to points with an absolute value higher than a pre-defined parameter a, and assigning a value of 0 to points with an absolute value lower than a, to obtain a binarized initial edge image; searching for the number of connected pixels for points assigned a value of 1 in the previous step, identifying points with a value lower than a pre-defined second parameter b as noise and deleting them from the initial edge image, and identifying points with a value higher than b as true edge points; and finally obtaining the final edge image.

[0011] The first technology 8 discloses an image fusion transformation (CN109993716A), which fuses the image of an object in the original image into the corresponding object image in the template image. During the fusion process, the pixel weight map is used to perform fusion processing on each pixel, so that the fused image can better integrate into the template image while better preserving the features of the object in the original image.

[0012] The prior art 9 discloses an image fusion method (CN109064436A), which acquires a visible light image and an infrared light image of the same scene; decomposes the visible light image to obtain a first high-frequency sub-image and a first low-frequency sub-image, and decomposes the infrared light image to obtain a second high-frequency sub-image and a second low-frequency sub-image; generates a high-frequency fused image of the scene based on the first high-frequency sub-image and the second high-frequency sub-image, and generates a low-frequency fused image of the scene based on the first low-frequency sub-image and the second low-frequency sub-image; and calculates a fused image of the scene based on the low-frequency fused image and the high-frequency fused image of the scene.

[0013] The prior art 10 discloses an image fusion method (CN113362261A). Based on the first fused image obtained by one fusion, a second fusion can be performed by color transfer so that the brightness and color of the first fused image can be transferred to the visible light image, thereby obtaining a second fused image with brightness and color closer to the visible light image.

[0014] In summary, the aforementioned prior art involves the application and device of focus stacking technology; or improves the accuracy of image information extraction by setting a threshold, transforming the coordinate system, and performing multiplication and division operations with the original image based on the Laplacian operator; or fuses the target image with the template image, or fuses the non-visible light image with the visible light image. Summary of the Invention

[0015] The technical problems to be solved by this invention are different from those of the prior art. It provides a multi-process-based focus stacking image fusion method and system, which extracts the maximum gradient edge information of each image and simultaneously adopts a multi-process data processing method to realize a multi-process-based focus stacking image.

[0016] To achieve the above objectives, the present invention adopts the following technical solution: a multi-process-based focus stacking image fusion method, comprising the following steps:

[0017] S1. Quickly acquire the m-th image, where m = 1, 2, ..., N, and N is the total number of images taken of the same object;

[0018] S2. Upload the acquired image information and process the image at the same time;

[0019] S3. Use a suitable gradient edge detection operator to quickly detect the gradient of the m-th image;

[0020] S4. Quickly obtain the maximum gradient information in the m-th image;

[0021] S5. Repeat the above steps until the maximum gradient information of the Nth image has been obtained, and then superimpose the maximum gradient information of the N images.

[0022] S6. Normalize the superimposed image;

[0023] S7. Evaluate the image after focus stacking.

[0024] As a further description of the above technical solution:

[0025] In step S1, m images are quickly acquired, where m = 1, 2, ..., N, and N is the total number of images. The same object is photographed, and the images are taken at different focal lengths.

[0026] As a further description of the above technical solution:

[0027] In step S2, the acquired image information is uploaded, and the image is processed. The specific steps are as follows: first, the brightness of the image is adjusted, and the color space is converted from RGB space to YCbCr space to reduce the influence of light on image processing. Then, the grayscale and contrast of the image are adjusted.

[0028] As a further description of the above technical solution:

[0029] In step S3, a suitable gradient edge detection operator is used to quickly detect the gradient of the m-th image, thereby obtaining all gradient information of the m-th image.

[0030] As a further description of the above technical solution:

[0031] In S1, S2, S3 and S4, multi-process data processing is used, which includes pooling and the addition of a just-in-time compiler.

[0032] As a further description of the above technical solution:

[0033] In step S6, the superimposed images are normalized to obtain a multi-process focus stacked image.

[0034] As a further description of the above technical solution:

[0035] In step S7, a deep learning algorithm is used to evaluate the image after focus stacking. The evaluation factors are pixels, noise reduction effect, and color softness.

[0036] The present invention also discloses a multi-process focus stacking image fusion system, including an image fast acquisition module, wherein the output end of the image fast acquisition module is electrically connected to the input end of an image processing module, and the output end of the image processing module is electrically connected to the input end of an image gradient detection module.

[0037] As a further description of the above technical solution:

[0038] The output of the image gradient detection module is electrically connected to the input of the maximum gradient information acquisition module, and the output of the maximum gradient information acquisition module is electrically connected to the input of the gradient information superposition module.

[0039] As a further description of the above technical solution:

[0040] The output of the gradient information overlay module is electrically connected to the input of the overlay image normalization module, and the output of the overlay image normalization module is electrically connected to the input of the focus stacking image evaluation module.

[0041] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0042] This invention processes a set of images of the same object taken at different focal lengths. By selecting an appropriate gradient edge detection operator, a gradient detection image is obtained. The maximum gradient value of each image is acquired, superimposed, and normalized to obtain a multi-process-based focus stacking image. This invention offers advantages in high-precision and high-speed image processing. The focus stacking image fusion method is universally applicable to stacked image processing, featuring convenient implementation and reasonable program design. Furthermore, it transforms traditional image edge detection methods into a multi-process, multi-image gradient edge detection method, thereby achieving the synthesis of focus stacking images. This significantly improves the synthesis runtime, especially under multi-core CPU conditions. The method also incorporates image processing to effectively improve the quality of acquired images, thus enhancing the subsequent focus stacking effect and ensuring the quality of the final focus stacking image. Finally, an image evaluation process is included, using deep learning algorithms to learn from high-quality images and perform comparative evaluation, thus assessing the overall focus stacking image fusion method. This powerful functionality is suitable for widespread application. Attached Figure Description

[0043] Figure 1 This is a flowchart of a multi-process-based focus stacking image fusion method.

[0044] Figure 2 This is a schematic diagram of the structural components of multi-process data processing in a multi-process-based focus stacking image fusion method.

[0045] Figure 3 This is a schematic diagram of the optical path in a multi-process-based focus stacking image fusion method.

[0046] Figure 4 This is a schematic diagram of the vibration damping platform structure in a multi-process-based focus stacking image fusion method.

[0047] Figure 5 This is a sample image in a multi-process-based focus stacking image fusion method.

[0048] Figure 6 for Figure 5 Synthesized sample images.

[0049] Figure 7 This is a schematic diagram of the module structure of a multi-process focus stacking image fusion system.

[0050] Legend:

[0051] 1. LD light source; 2. Sample to be tested; 3. CMOS camera; 4. Computer; 5. Vibration damping platform; 6. Sliding bracket. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Please see Figure 1-7 This invention provides a technical solution: a multi-process-based focus stacking image fusion method, comprising the following steps:

[0054] S1. Quickly acquire the m-th image, where m = 1, 2, ..., N, and N is the total number of images. Take images of the same object with different focal lengths.

[0055] S2. Upload the acquired image information and process the image. The specific steps are as follows: first, adjust the brightness of the image, convert the color space, convert the acquired different images from RGB space to YCbCr space to reduce the impact of light on image processing, and then adjust the grayscale and contrast of the image.

[0056] S3. Use a suitable gradient edge detection operator to quickly detect the gradient of the m-th image and obtain all gradient information of the m-th image;

[0057] S4. Quickly obtain the maximum gradient information in the m-th image. In S1, S2, S3 and S4, multi-process data processing is used. Multi-process data processing includes pooling and adding a just-in-time compiler.

[0058] S5. Repeat the above steps until the maximum gradient information of the Nth image has been obtained, and then superimpose the maximum gradient information of the N images.

[0059] S6. Normalize the superimposed images to obtain a multi-process focus stacked image;

[0060] S7. Using deep learning algorithms, evaluate the image after focus stacking. The evaluation factors are pixels, noise reduction effect and color softness.

[0061] The present invention also discloses a multi-process-based focus stacking image fusion system, including an image fast acquisition module, the output of which is electrically connected to the input of an image processing module, the output of which is electrically connected to the input of an image gradient detection module, the output of which is electrically connected to the input of a maximum gradient information acquisition module, the output of which is electrically connected to the input of a gradient information overlay module, the output of which is electrically connected to the input of an overlay image normalization module, and the output of which is electrically connected to the input of a focus stacking image evaluation module.

[0062] In this embodiment, the method processes a set of images of the same object taken at different focal lengths. By selecting a suitable gradient edge detection operator, a gradient detection image is obtained. The maximum gradient value of each image is acquired, superimposed, and normalized to finally obtain a multi-process-based focus stacking image. This invention offers advantages in high-precision and high-speed image processing. This focus stacking image fusion method is universally applicable to stacked image processing, featuring convenient implementation and reasonable program design. Furthermore, it transforms traditional image edge detection methods into a multi-process, multi-image gradient edge detection method, thereby achieving the synthesis of focus stacking images. This significantly improves the synthesis runtime, especially under multi-core CPU conditions. The method also includes an image processing step to effectively improve the quality of the acquired images, thereby enhancing the subsequent focus stacking effect and ensuring the quality of the final focus stacking image. Finally, an image evaluation process is included, which uses deep learning algorithms to learn from high-quality images and perform comparative evaluation, thus evaluating the overall focus stacking image fusion method. This powerful functionality is suitable for widespread application.

[0063] In the embodiments, it should be noted that:

[0064] The optical path diagram of the multi-process focus stacking image fusion method and system of this invention is shown below. Figure 3 As shown, its components include an LD light source 1, a sample under test 2, a CMOS camera 3, a computer 4, a vibration damping platform 5, and a sliding support 6. The sample under test 2 is illuminated from the bottom by the LD light source 1, and the light intensity information of the sample under test 2 is received and recorded by the CMOS camera 3. The CMOS camera 3 can be translated. The computer 4 is connected to the output terminal of the CMOS camera 3 and has a processing program for the acquired images using a multi-process focus stacking image fusion method.

[0065] The positional relationships on the shock absorption platform 5 are as follows: Figure 4 As shown, the sample 2 to be tested is fixed on the surface of the vibration damping platform 5. The LD light source 1 is embedded in the vibration damping platform 5 and can illuminate the CMOS camera 3. The CMOS camera 3 is mounted on the sliding bracket 6, and the CMOS camera 3 can move left and right on the sliding bracket 6. The sliding bracket 6 is mounted on the vibration damping platform 5, and the sliding bracket 6 can move up and down and back and forth on the vibration damping platform 5.

[0066] Figure 5 (a) shows the case where the focus is at the leftmost end of the image, (b) shows the case where the focus is in the middle of the image, and (c) shows the case where the focus is at the rightmost end of the image.

[0067] The multi-process-based focus stacking image fusion method of this embodiment includes the following steps:

[0068] H1 and CMOS camera 3 move from left to right on the sliding bracket 6, taking a picture every 0.5 seconds. Figure 3 As shown, focus and capture images from the leftmost edge of sample 2 until the rightmost edge of sample 2 is focused and captured, and finally m images are acquired, m = 1, 2, ..., N, where N is the total number of images;

[0069] H2. Employ multi-process data processing to quickly acquire the m-th image of the same object taken at different focal lengths;

[0070] H3. Multi-process data processing is employed, and a Gaussian filter operator is used to quickly perform a convolution operation on the m-th image to remove noise from the image. The formula is as follows:

[0071] O m (x,y)=I m (x,y)*K m (x,y,σ)

[0072] Among them O m (x, y) represents the output of the m-th image, I m (x, y) represents the m-th input image, K m (x, y, σ) represents the convolution kernel for the m-th image, and * represents the convolution operator. At this point, K... m (x, y, σ) represents the Gaussian convolution kernel for the m-th image, as shown in the following formula:

[0073]

[0074] Where x is the horizontal coordinate of the input image, y is the vertical coordinate of the input image, σ is the standard deviation, W is the width of the input image, and H is the height of the input image;

[0075] H4. Multi-process data processing is adopted, and the Laplacian operator is used to perform gradient edge detection on the m-th image to obtain the gradient information of the m-th image. The formula is as follows:

[0076]

[0077] H5 uses multi-process data processing to quickly obtain the maximum gradient information in the m-th image;

[0078] H6. Repeat steps <2> , <3> , <4> , <5> This process continues until the maximum gradient information of the Nth image has been obtained. Then, multi-process data processing is used to superimpose the maximum gradient information of the N images.

[0079] H7. Normalize the superimposed images to obtain a multi-process focus stacked image.

[0080] The multi-process-based focus stacking image fusion method and system employs multi-process data processing in steps H2, H3, H4, H5, and H6. This multi-process data processing includes pooling and the addition of a just-in-time (JIT) compiler. Pooling means that when the number of processes is less than the maximum value in the pool, user requests can be treated as a new process for parallel processing; however, when the number of processes is greater than or equal to the maximum value, user requests must wait until a process finishes before they can be treated as a new process for parallel processing. The addition of a JIT compiler improves efficiency when a piece of code is frequently executed by translating it into machine code, which is advantageous in situations with nested loops or a large number of loops.

[0081] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multi-process-based focus stacking image fusion method, characterized in that: Includes the following steps: S1. Quickly acquire the m-th image, where m = 1, 2, ..., N, and N is the total number of images taken of the same object; Among them, images were taken at different focal lengths; S2. Upload the acquired image information and process the image at the same time; To process an image, the brightness is first adjusted, the color space is converted from RGB to YCbCr, and then the grayscale and contrast are adjusted. S3. Use the gradient edge detection operator to quickly detect the gradient of the m-th image; S4. Quickly obtain the maximum gradient information in the m-th image; In S1, S2, S3 and S4, multi-process data processing is used, which includes pooling and adding a just-in-time compiler. S5. Repeat the above steps until the maximum gradient information of the Nth image has been obtained, and then superimpose the maximum gradient information of the N images. S6. Normalize the superimposed image; S7. Evaluate the image after focus stacking.

2. The multi-process-based focus stacking image fusion method according to claim 1, characterized in that, In step S3, the gradient edge detection operator is used to quickly detect the gradient of the m-th image, thereby obtaining all gradient information of the m-th image.

3. The multi-process-based focus stacking image fusion method according to claim 1, characterized in that, In step S6, the superimposed images are normalized to obtain a multi-process focus stacked image.

4. The multi-process-based focus stacking image fusion method according to claim 1, characterized in that, In step S7, a deep learning algorithm is used to evaluate the image after focus stacking. The evaluation factors are pixels, noise reduction effect, and color softness.

Images