Multi-focus image fusion method and device

Abstract

The present invention provides a multi-focus image fusion method and device, the method comprising: acquiring a target image, performing Gaussian blur processing and binarization processing on the target image, obtaining a Gaussian map and a true value map corresponding to the target image; generating an inverse true value map corresponding to the true value map, the inverse true value map and the true value map are in a complementary relationship; based on the target image, the Gaussian map, the true value map and the inverse true value map, generating a first focus map and a second focus map corresponding to the target image, the first focus map and the second focus map are in a complementary relationship; inputting the first focus map and the second focus map into a pre-trained image fusion network, obtaining a decision map output by the image fusion network, the decision map is a binary map predicted by the image fusion network; fusing the first focus image, the second focus image and the decision map to obtain a multi-focus fusion image. The method provided by the present invention can simplify the multi-focus image fusion process.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a multi-focus image fusion method and device. Background Art

[0002] Image fusion technology is a visual information fusion technology and an extended branch of information fusion. Image fusion mainly involves various methods of processing the content information of multiple channels obtained by multi-source sensors in the same scene, extracting useful content information from each image, and finally fusing it into a high-quality, information-rich single image. Multi-focus image fusion is one of the important branches of image fusion. It uses a single sensor device to fuse the image content information obtained by multiple focuses in a certain scene.

[0003] The existing multi-focus image fusion method is based on prediction and processing of convolutional neural networks, which also requires binary segmentation, hole removal, watershed and other operations on the image, and some related data need to be manually extracted for features, which is a very cumbersome process. Summary of the invention

[0004] In view of this, the present invention provides a multi-focus image fusion method, by which the multi-focus image fusion process can be simplified.

[0005] The present invention also provides a multi-focus image fusion device to ensure the implementation and application of the above method in practice.

[0006] A multi-focus image fusion method, comprising:

[0007] Acquire a target image, perform Gaussian blur processing and binarization processing on the target image, and obtain a Gaussian map and a true value map corresponding to the target image;

[0008] Generate an inverse truth value graph corresponding to the truth value graph, wherein the inverse truth value graph is in a complementary relationship with the truth value graph;

[0009] Based on the target image, the Gaussian map, the true value map and the inverse true value map, a first focus map and a second focus map corresponding to the target image are generated, wherein the first focus map and the second focus map are in a complementary relationship;

[0010] Inputting the first focus map and the second focus map into a pre-trained image fusion network to obtain a decision map output by the image fusion network, wherein the decision map is a binary map predicted by the image fusion network;

[0011] The first focused image, the second focused image and the decision graph are fused to obtain a multi-focus fused image.

[0012] In the above method, optionally, generating a first focus map and a second focus map corresponding to the target image based on the target image, the Gaussian map, the true value map and the inverse true value map comprises:

[0013] Applying a preset first calculation formula to calculate the target image, the Gaussian image, the true value image and the inverse true value image, to generate a first focusing image corresponding to the target image;

[0014] Applying a preset second calculation formula to calculate the target image, the Gaussian image, the true value image and the inverse true value image, to generate a second focusing image corresponding to the target image;

[0015] The first calculation formula is:

[0016]

[0017] The second calculation formula is:

[0018]

[0019] Among them, y1 is the first focus image, y2 is the second focus image, x is the target image, is the true value graph, is the inverse truth value graph, x blur It is a Gaussian graph.

[0020] In the above method, optionally, the process of training the image fusion network includes:

[0021] Acquire a first training image, a second training image, and a training target, wherein the first training image and the second training image are focus images in a complementary relationship, and the training target is a binary image corresponding to the first training image and the second training image;

[0022] Based on the first training image, the second training image and the training target, executing a training process corresponding to the image fusion network;

[0023] The training process includes: inputting the first training image and the second training image into the first coding network and the second coding network of the image fusion network respectively, obtaining first context information corresponding to the first training image output by the first coding network and second context information corresponding to the second training image output by the second coding network; fusing the first context information with the second context information to obtain fusion information, and inputting the fusion information into the decoder of the image fusion network to obtain a target binary image output by the decoder;

[0024] Comparing the target binary image with the training target to obtain a comparison result;

[0025] When the comparison result does not meet the preset training condition, applying a preset binary cross entropy function to adjust the image fusion network, and re-executing the training process;

[0026] When the comparison result satisfies the training condition, the training of the image fusion network is terminated.

[0027] In the above method, optionally, fusing the first focused image, the second focused image and the decision graph to obtain a multi-focus fused image includes:

[0028] Applying a preset conditional random field method CRF to perform boundary smoothing processing on the decision graph to obtain a processed decision graph;

[0029] The first focused image, the second focused image and the processed decision graph are fused to obtain a multi-focus fused image.

[0030] In the above method, optionally, fusing the first focused image, the second focused image and the decision graph to obtain a fused image includes:

[0031] Applying a preset third calculation formula to calculate the first focused image, the second focused image and the decision graph to generate a multi-focus fusion image;

[0032] The third calculation formula is:

[0033] F fused =F D (x,y)×F A (x,y)+(1-F D (x,y))×F B (x,y)

[0034] Among them, F fused is the fused image, F A is the first focused image, F B is the second focused image, F D is a graph decision graph, (x, y) is a pixel in the image.

[0035] A multi-focus image fusion method, comprising:

[0036] A first processing unit is used to acquire a target image, perform Gaussian blur processing and binarization processing on the target image, and obtain a Gaussian map and a true value map corresponding to the target image;

[0037] A first generating unit, configured to generate an inverse truth value graph corresponding to the truth value graph, wherein the inverse truth value graph and the truth value graph are in a complementary relationship;

[0038] A second generating unit is configured to generate a first focusing map and a second focusing map corresponding to the target image based on the target image, the Gaussian map, the true value map and the inverse true value map, wherein the first focusing map and the second focusing map are in a complementary relationship;

[0039] A second processing unit is used to input the first focus map and the second focus map into a pre-trained image fusion network to obtain a decision map output by the image fusion network, where the decision map is a binary map predicted by the image fusion network;

[0040] The fusion unit is used to fuse the first focused image, the second focused image and the decision graph to obtain a multi-focus fused image.

[0041] In the above device, optionally, the second generating unit includes:

[0042] A first sub-calculation unit, configured to apply a preset first calculation formula to calculate the target image, the Gaussian image, the true value image, and the inverse true value image, to generate a first focus image corresponding to the target image;

[0043] A second calculation subunit is used to apply a preset second calculation formula to calculate the target image, the Gaussian image, the true value image and the inverse true value image to generate a second focus image corresponding to the target image;

[0044] The first calculation formula is:

[0045]

[0046] The second calculation formula is:

[0047]

[0048] Among them, y1 is the first focus image, y2 is the second focus image, x is the target image, is the true value graph, is the inverse truth value graph, x blur It is a Gaussian graph.

[0049] The above device may optionally further include:

[0050] An acquisition unit, configured to acquire a first training image, a second training image, and a training target, wherein the first training image and the second training image are focus images in a complementary relationship, and the training target is a binary image corresponding to the first training image and the second training image;

[0051] A training unit, configured to execute a training process corresponding to the image fusion network based on the first training image, the second training image and the training target;

[0052] The training process includes: inputting the first training image and the second training image into the first coding network and the second coding network of the image fusion network respectively, obtaining first context information corresponding to the first training image output by the first coding network and second context information corresponding to the second training image output by the second coding network; fusing the first context information with the second context information to obtain fusion information, and inputting the fusion information into the decoder of the image fusion network to obtain a target binary image output by the decoder;

[0053] A comparison unit is used to compare the target binary image with the training target to obtain a comparison result; when the comparison result does not meet the preset training conditions, a preset binary cross entropy function is applied to adjust the image fusion network and the training process is re-executed; when the comparison result meets the training conditions, the training of the image fusion network is terminated.

[0054] In the above device, optionally, the fusion unit comprises:

[0055] A first processing subunit is used to apply a preset conditional random field method CRF to perform boundary smoothing processing on the decision graph to obtain a processed decision graph;

[0056] The fusion subunit is used to fuse the first focused image, the second focused image and the processed decision graph to obtain a multi-focus fused image.

[0057] In the above device, optionally, the fusion unit comprises:

[0058] a generating subunit, configured to calculate the first focused image, the second focused image and the decision graph by applying a preset third calculation formula to generate a multi-focus fusion image;

[0059] The third calculation formula is:

[0060] F fused =F D (x,y)×F A (x,y)+(1-F D (x,y))×F B (x,y)

[0061] Among them, F fused is the fused image, F A is the first focused image, F B is the second focused image, F D is a graph decision graph, (x, y) is a pixel in the image.

[0062] A storage medium includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the multi-focus image fusion method.

[0063] An electronic device includes a memory and one or more instructions, wherein the one or more instructions are stored in the memory and are configured to execute the multi-focus image fusion method by one or more processors.

[0064] Compared with the prior art, the present invention has the following advantages:

[0065] The present invention provides a multi-focus image fusion method, including: acquiring a target image, performing Gaussian blur processing and binarization processing on the target image, and obtaining a Gaussian map and a true value map corresponding to the target image; generating an inverse true value map corresponding to the true value map, wherein the inverse true value map and the true value map are in a complementary relationship; based on the target image, the Gaussian map, the true value map and the inverse true value map, generating a first focus map and a second focus map corresponding to the target image, wherein the first focus map and the second focus map are in a complementary relationship; inputting the first focus map and the second focus map into a pre-trained image fusion network, and obtaining a decision map output by the image fusion network, wherein the decision map is a binary map predicted by the image fusion network; fusing the first focus image, the second focus image and the decision map to obtain a multi-focus fusion image. The method provided by the present invention can simplify the multi-focus image fusion process. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0067] Figure 1 A method flow chart of a multi-focus image fusion method provided by an embodiment of the present invention;

[0068] Figure 2 A flowchart of another method of a multi-focus image fusion method provided by an embodiment of the present invention;

[0069] Figure 3 A schematic diagram of a training process of an image fusion network provided by an embodiment of the present invention;

[0070] Figure 4 A device structure diagram of a multi-focus image fusion device provided by an embodiment of the present invention;

[0071] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0074] The present invention can be used in many general or special computing device environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or devices, etc.

[0075] The embodiment of the present invention provides a multi-focus image fusion method, which can be applied to a variety of system platforms, and its execution subject can be a processor of a computer terminal or various mobile devices. The method flow chart of the method is as follows: Figure 1 As shown, specifically including:

[0076] S101: Acquire a target image, perform Gaussian blur processing and binarization processing on the target image, and obtain a Gaussian map and a true value map corresponding to the target image.

[0077] It should be noted that the target image may be a saliency image, which is an image showing the uniqueness of each pixel.

[0078] Among them, performing Gaussian blur processing on the target image can simulate the defocus property of the camera, thereby obtaining a non-focused Gaussian image. Performing binarization processing on the target image is to set the grayscale value of the pixel in the image to 0 or 255, and obtain a true value image with only black and white.

[0079] S102: Generate an inverse truth value graph corresponding to the truth value graph.

[0080] The inverse truth map and the true value map are complementary, that is, the grayscale value of a pixel with a grayscale value of 0 in the true value map is 255 in the inverse truth map, and the grayscale value of a pixel with a grayscale value of 255 in the true value map is 0 in the inverse truth map.

[0081] S103: Based on the target image, the Gaussian map, the true value map and the inverse true value map, generate a first focus map and a second focus map corresponding to the target image.

[0082] The first focusing map and the second focusing map are in a complementary relationship.

[0083] Specifically, the process of obtaining the first focusing map and the second focusing map is as follows:

[0084] Applying a preset first calculation formula to calculate the target image, the Gaussian image, the true value image and the inverse true value image, to generate a first focusing image corresponding to the target image;

[0085] Applying a preset second calculation formula to calculate the target image, the Gaussian image, the true value image and the inverse true value image, to generate a second focusing image corresponding to the target image;

[0086] The first calculation formula is:

[0087]

[0088] The second calculation formula is:

[0089]

[0090] Among them, y1 is the first focus image, y2 is the second focus image, x is the target image, is the true value graph, is the inverse truth value graph, x blur It is a Gaussian graph.

[0091] It should be noted that the focus map includes a focus area and a non-focus area. The focus area refers to the image area with clear focus in the focus map. The non-focus area refers to the image area that is not clear in the focus image. In the embodiment of the present invention, the first focus map and the second focus map are complementary to each other. Therefore, the focus area in the first focus map is a non-focus area in the second focus map, and the non-focus area in the first focus map is a focus area in the second focus map.

[0092] S104: Inputting the first focus map and the second focus map into a pre-trained image fusion network to obtain a decision map output by the image fusion network.

[0093] Wherein, the decision graph is a binary graph predicted by the image fusion network.

[0094] It should be noted that the image fusion network is a dual-branch end-to-end encoding and decoding network, which extracts the first focus map and the second focus map as input related information of the network to obtain a decision map.

[0095] It should also be noted that the decision graph is a binary graph obtained by fusion analysis of the first focus graph and the second focus graph by the image fusion network. The decision graph output by the image fusion network can be compared with the true value. Figure 1 It can also be different from the true value graph.

[0096] S105: Fusing the first focused image, the second focused image and the decision graph to obtain a multi-focus fused image.

[0097] Specifically, the process of fusing the image is as follows:

[0098] Applying a preset third calculation formula to calculate the first focused image, the second focused image and the decision graph to generate a multi-focus fusion image;

[0099] The third calculation formula is:

[0100] F fused =F D (x,y)×F A (x,y)+(1-F D (x,y))×F B (x,y)

[0101] Among them, F fused is the fused image, F A is the first focused image, F B is the second focused image, F D is a graph decision graph, (x, y) is a pixel in the image.

[0102] It should be noted that the third calculation formula is applied to calculate each pixel point, and each pixel point in the first focus map, the second focus map and the decision map is fused to generate a multi-focus fusion image.

[0103] In the method provided by the embodiment of the present invention, a target image is obtained, and a Gaussian blur process is performed on the image to obtain a Gaussian map, and a binarization process is performed to obtain a true value map. The pixel points with a grayscale value of 0 in the true value map are converted into pixel points with a grayscale value of 255, and the pixel points with a grayscale value of 255 are converted into pixel points with a grayscale value of 0, and an anti-true value map that is complementary to the true value map is generated. The target image, Gaussian map, true value map and anti-true value map are calculated according to the first calculation formula to obtain a first focus map, that is, the target image is multiplied by the true value map and the structure of multiplying the Gaussian map and the anti-true value map is added to obtain the first focus map. The target image, Gaussian map, true value map and anti-true value map are calculated according to the second calculation formula to obtain a second focus map, that is, the target image is multiplied by the anti-true value map and the structure of multiplying the Gaussian map and the true value map is added to obtain the first focus map. The first focus map and the second focus map are analyzed by the image fusion network, and a decision map is output, which is also a binary map, and the grayscale value of each pixel in the decision map is 0 or 255. The first focus map, the second focus map and the decision map are fused to obtain a multi-focus fusion image. In the embodiment of the present invention, there is no need to use a large number of cumbersome processing operations such as difference, segmentation and hole removal in the image fusion process, which simplifies the multi-focus image fusion process.

[0104] Furthermore, in the multi-focus image fusion process, the decision map may be further smoothed to ensure the smoothness of the focus area and the non-focus area of ​​the multi-focus image. Therefore, the process of fusing the first focus image, the second focus image and the decision map may specifically include:

[0105] Applying a preset conditional random field method CRF to perform boundary smoothing processing on the decision graph to obtain a processed decision graph;

[0106] The first focused image, the second focused image and the processed decision graph are fused to obtain a multi-focus fused image.

[0107] It should be noted that the main idea of ​​the conditional random field is to use assumptions such as label consistency between pixels to convert the pixel-level label classification problem into a probability solution problem. Its energy function is as follows:

[0108] E(x)=∑ i ψ u (x i )+∑ i≠j ψ p (x i ,x j )

[0109] ψ u (x i ) represents the probability that each pixel belongs to each category, that is, the probability that each pixel belongs to the focused area or the non-focused area. p(x i ,x j ) is a pairwise potential function term, which is used to measure the probability that two pixels belong to the same region and is defined as the weighted Gaussian kernel function. p (x i ,x j ) is calculated as:

[0110]

[0111] Among them, ω1 and ω2 are Gaussian weight terms, I i ,I j ,p i ,p j Represents the pixel value and position of pixel i and j respectively, μ(x i ,x j ) is the compatibility measure between two labels, θ α ,θ β ,θ γ is the Gaussian kernel parameter.

[0112] It should also be noted that after the decision graph is processed, the third calculation formula may be applied to calculate the first focused image, the second focused image and the processed decision graph to generate a multi-focus fusion image.

[0113] In the process of fusing the first focus map, the second focus map and the decision map using the method provided in the embodiment of the present invention, the CRF is applied to process and fuse each image, thereby ensuring the consistency between the focus area and the non-focus area and the smooth transition of the boundary.

[0114] In the present invention, in order to simplify the image fusion process, the image fusion network is applied to process the first focus map and the second focus map to obtain a decision map. Figure 2 , the process of training the image fusion network can specifically include:

[0115] S201: Acquire a first training image, a second training image, and a training target.

[0116] The first training image and the second training image are complementary focus images, and the training target is a binary image corresponding to the first training image and the second training image.

[0117] Optionally, the first training graph and the second training graph may be the first cluster graph and the second cluster graph mentioned in S103 above, and the training target may be the true value graph mentioned in S101 above.

[0118] S202: Input the first training image and the second training image into the first encoding network and the second encoding network of the image fusion network respectively, and obtain first context information corresponding to the first training image output by the first encoding network and second context information corresponding to the second training image output by the second encoding network.

[0119] It should be noted that the first training image is input into the first coding network, and the first coding network outputs the first context information corresponding to the first training image; the second training image is input into the second coding network, and the second coding network outputs the second context information corresponding to the second training image. The image fusion network is a dual-branch end-to-end encoding and decoding network, and the dual branches in the network are the first encoder and the second encoder.

[0120] S203: Fusing the first context information with the second context information to obtain fused information, and inputting the fused information into a decoder of the image fusion network to obtain a target binary image output by the decoder.

[0121] It should be noted that the first context information and the second context information are feature fused, and after being processed by a multi-stage convolution module in the decoder, a target binary image with a grayscale of black and white is output.

[0122] S204: Compare the target binary image with the training target to obtain a comparison result.

[0123] It should be noted that the target binary image is compared with the training target, specifically, the similarity between the target binary image and the training target is calculated. If the similarity is greater than a preset threshold, the preset training condition is met, and S205 is continued to be executed. If the similarity is not greater than the preset threshold, the preset training condition is not met, and S206 is continued to be executed.

[0124] S205: When the comparison result does not meet the preset training condition, a preset binary cross entropy function is applied to adjust the image fusion network, and the process returns to S202.

[0125] It should be noted that the binary cross entropy function is:

[0126] L=-∑G(m,n)log(S(m,n))-∑(1-G(m,n))log(1-S(m,n))

[0127] Among them, G(m,n)∈(0,1) is the value of the true label (training target) G at position (m,n), and S(m,n)∈(0,1) is the value of the target binary image prediction result S at position (m,n).

[0128] S206: When the comparison result satisfies the training condition, the training of the image fusion network is terminated.

[0129] Optionally, the present invention may apply multiple paired complementary focus maps to train the image fusion network, so as to improve the accuracy of the image fusion network after training with multiple pairs of images.

[0130] In the method provided in the embodiment of the present invention, reference is made to Figure 3 , Figure 3 The figure is a diagram of the training process of the image fusion network, which includes two encoder modules and a decoder module. After the two decoders input paired complementary training images, the decoders output corresponding binary images. The encoder contains a residual dense convolution block (RDA) for feature extraction to obtain the context information of the training image.

[0131] The method provided in the embodiment of the present invention designs a dual-branch end-to-end encoding and decoding network to achieve end-to-end prediction of the decision graph; it avoids the use of a large number of cumbersome post-processing operations such as interpolation, binary segmentation, and hole removal in the existing method to optimize the results.

[0132] The specific implementation processes and derivative methods of the above-mentioned embodiments are all within the protection scope of the present invention.

[0133] and Figure 1 Corresponding to the method described above, the embodiment of the present invention further provides a multi-focus image fusion device for Figure 1 In the specific implementation of the method, the multi-focus image fusion device provided in the embodiment of the present invention can be applied to a computer terminal or various mobile devices, and its structural schematic diagram is as follows Figure 4 As shown, specifically including:

[0134] The first processing unit 401 is used to obtain a target image, perform Gaussian blur processing and binarization processing on the target image, and obtain a Gaussian map and a true value map corresponding to the target image;

[0135] A first generating unit 402 is used to generate an inverse truth value graph corresponding to the truth value graph, wherein the inverse truth value graph is in a complementary relationship with the truth value graph;

[0136] A second generating unit 403 is configured to generate a first focusing map and a second focusing map corresponding to the target image based on the target image, the Gaussian map, the true value map and the inverse true value map, wherein the first focusing map and the second focusing map are in a complementary relationship;

[0137] A second processing unit 404 is configured to input the first focus map and the second focus map into a pre-trained image fusion network to obtain a decision map output by the image fusion network, where the decision map is a binary map predicted by the image fusion network;

[0138] The fusion unit 405 is used to fuse the first focused image, the second focused image and the decision graph to obtain a multi-focus fused image.

[0139] In the device provided by the embodiment of the present invention, a target image is obtained, and a Gaussian blur process is performed on the image to obtain a Gaussian map, and a binarization process is performed to obtain a true value map. The pixel points with a grayscale value of 0 in the true value map are converted into pixel points with a grayscale value of 255, and the pixel points with a grayscale value of 255 are converted into pixel points with a grayscale value of 0, and an anti-true value map that is complementary to the true value map is generated. The target image, the Gaussian map, the true value map and the anti-true value map are calculated according to the first calculation formula to obtain a first focus map, that is, the target image is multiplied by the true value map and the structure of multiplying the Gaussian map and the anti-true value map is added to obtain the first focus map. The target image, the Gaussian map, the true value map and the anti-true value map are calculated according to the second calculation formula to obtain a second focus map, that is, the target image is multiplied by the anti-true value map and the structure of multiplying the Gaussian map and the true value map is added to obtain the first focus map. The first focus map and the second focus map are analyzed by the image fusion network, and a decision map is output, which is also a binary map, and the grayscale value of each pixel in the decision map is 0 or 255. The first focus map, the second focus map and the decision map are fused to obtain a multi-focus fusion image. In the embodiment of the present invention, there is no need to use a large number of cumbersome processing operations such as difference, segmentation and hole removal in the image fusion process, which simplifies the multi-focus image fusion process.

[0140] In the device provided in the embodiment of the present invention, the second generating unit 403 includes:

[0141] A first sub-calculation unit, configured to apply a preset first calculation formula to calculate the target image, the Gaussian image, the true value image, and the inverse true value image, to generate a first focus image corresponding to the target image;

[0142] A second calculation subunit is used to apply a preset second calculation formula to calculate the target image, the Gaussian image, the true value image and the inverse true value image to generate a second focus image corresponding to the target image;

[0143] The first calculation formula is:

[0144]

[0145] The second calculation formula is:

[0146]

[0147] Among them, y1 is the first focus image, y2 is the second focus image, x is the target image, is the true value graph, is the inverse truth value graph, x blur It is a Gaussian graph.

[0148] The device provided in the embodiment of the present invention further includes:

[0149] An acquisition unit, configured to acquire a first training image, a second training image, and a training target, wherein the first training image and the second training image are focus images in a complementary relationship, and the training target is a binary image corresponding to the first training image and the second training image;

[0150] A training unit, configured to execute a training process corresponding to the image fusion network based on the first training image, the second training image and the training target;

[0151] The training process includes: inputting the first training image and the second training image into the first coding network and the second coding network of the image fusion network respectively, obtaining first context information corresponding to the first training image output by the first coding network and second context information corresponding to the second training image output by the second coding network; fusing the first context information with the second context information to obtain fusion information, and inputting the fusion information into the decoder of the image fusion network to obtain a target binary image output by the decoder;

[0152] A comparison unit is used to compare the target binary image with the training target to obtain a comparison result; when the comparison result does not meet the preset training conditions, a preset binary cross entropy function is applied to adjust the image fusion network and the training process is re-executed; when the comparison result meets the training conditions, the training of the image fusion network is terminated.

[0153] In the device provided in the embodiment of the present invention, the fusion unit 405 includes:

[0154] A first processing subunit is used to apply a preset conditional random field method CRF to perform boundary smoothing processing on the decision graph to obtain a processed decision graph;

[0155] The fusion subunit is used to fuse the first focused image, the second focused image and the processed decision graph to obtain a multi-focus fused image.

[0156] In the device provided in the embodiment of the present invention, the fusion unit 405 includes:

[0157] a generating subunit, configured to calculate the first focused image, the second focused image and the decision graph by applying a preset third calculation formula to generate a multi-focus fusion image;

[0158] The third calculation formula is:

[0159] F fused =F D (x,y)×F A (x,y)+(1-F D (x,y))×F B (x,y)

[0160] Among them, F fused is the fused image, F A is the first focused image, F B is the second focused image, F D is a graph decision graph, (x, y) is a pixel in the image.

[0161] For the specific working process of each unit and sub-unit in the multi-focus image fusion device disclosed in the above embodiment of the present invention, reference may be made to the corresponding content in the multi-focus image fusion method disclosed in the above embodiment of the present invention, and will not be repeated here.

[0162] An embodiment of the present invention further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned multi-focus image fusion method.

[0163] The embodiment of the present invention further provides an electronic device, the structural diagram of which is shown in FIG. Figure 5 As shown, it specifically includes a memory 501 and one or more instructions 502, wherein the one or more instructions 502 are stored in the memory 501 and are configured to be executed by one or more processors 503 to perform the following operations:

[0164] Acquire a target image, perform Gaussian blur processing and binarization processing on the target image, and obtain a Gaussian map and a true value map corresponding to the target image;

[0165] Generate an inverse truth value graph corresponding to the truth value graph, wherein the inverse truth value graph is in a complementary relationship with the truth value graph;

[0166] Based on the target image, the Gaussian map, the true value map and the inverse true value map, a first focus map and a second focus map corresponding to the target image are generated, wherein the first focus map and the second focus map are in a complementary relationship;

[0167] Inputting the first focus map and the second focus map into a pre-trained image fusion network to obtain a decision map output by the image fusion network, wherein the decision map is a binary map predicted by the image fusion network;

[0168] The first focused image, the second focused image and the decision graph are fused to obtain a multi-focus fused image.

[0169] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0170] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, computer software, or a combination of both.

[0171] In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0172] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-focus image fusion method, characterized in that: include: Acquire a target image, perform Gaussian blur processing and binarization processing on the target image, and obtain a Gaussian map and a true value map corresponding to the target image; Generate an inverse truth map corresponding to the truth map, wherein the inverse truth map and the truth map are in a complementary relationship; the complementary relationship is that a pixel point with a grayscale value of 0 in the truth map has a grayscale value of 255 in the inverse truth map, and a pixel point with a grayscale value of 255 in the truth map has a grayscale value of 0 in the inverse truth map; Based on the target image, the Gaussian map, the true value map and the inverse true value map, a first focus map and a second focus map corresponding to the target image are generated, wherein the first focus map and the second focus map are in a complementary relationship; Inputting the first focus map and the second focus map into a pre-trained image fusion network to obtain a decision map output by the image fusion network, wherein the decision map is a binary map predicted by the image fusion network; fusing the first focused image, the second focused image and the decision graph to obtain a multi-focus fused image; The step of generating a first focus map and a second focus map corresponding to the target image based on the target image, the Gaussian map, the true value map and the inverse true value map comprises: Applying a preset first calculation formula to calculate the target image, the Gaussian image, the true value image and the inverse true value image, to generate a first focusing image corresponding to the target image; Applying a preset second calculation formula to calculate the target image, the Gaussian image, the true value image and the inverse true value image, to generate a second focusing image corresponding to the target image; The first calculation formula is: ； The second calculation formula is: ； Among them, y1 is the first focus image, y2 is the second focus image, x is the target image, is the true value graph, is the inverse truth value graph, x blur is a Gaussian graph; The step of fusing the first focused image, the second focused image and the decision graph to obtain a multi-focus fused image includes: Applying a preset third calculation formula to calculate the first focused image, the second focused image and the decision graph to generate a multi-focus fusion image; The third calculation formula is: ； Among them, F fused is the fused image, F A is the first focused image, F B is the second focused image, F D is a graph decision graph, (x, y) is a pixel in the image.

2. The method according to claim 1, characterized in that The process of training the image fusion network includes: Acquire a first training image, a second training image, and a training target, wherein the first training image and the second training image are focus images in a complementary relationship, and the training target is a binary image corresponding to the first training image and the second training image; Based on the first training image, the second training image and the training target, executing a training process corresponding to the image fusion network; The training process includes: inputting the first training image and the second training image into the first coding network and the second coding network of the image fusion network respectively, obtaining first context information corresponding to the first training image output by the first coding network and second context information corresponding to the second training image output by the second coding network; fusing the first context information with the second context information to obtain fusion information, and inputting the fusion information into the decoder of the image fusion network to obtain a target binary image output by the decoder; Comparing the target binary image with the training target to obtain a comparison result; When the comparison result does not meet the preset training condition, applying a preset binary cross entropy function to adjust the image fusion network, and re-executing the training process; When the comparison result satisfies the training condition, the training of the image fusion network is terminated.

3. The method according to claim 1, characterized in that: The step of fusing the first focused image, the second focused image and the decision graph to obtain a multi-focus fused image includes: Applying a preset conditional random field method CRF to perform boundary smoothing processing on the decision graph to obtain a processed decision graph; The first focused image, the second focused image and the processed decision graph are fused to obtain a multi-focus fused image.

4. A multi-focus image fusion device, characterized in that: include: A first processing unit is used to acquire a target image, perform Gaussian blur processing and binarization processing on the target image, and obtain a Gaussian map and a true value map corresponding to the target image; A first generating unit, configured to generate an inverse truth value graph corresponding to the truth value graph, wherein the inverse truth value graph and the truth value graph are in a complementary relationship; The complementary relationship is that the grayscale value of a pixel in the true value image with a grayscale value of 0 is 255 in the inverse true value image, and the grayscale value of a pixel in the true value image with a grayscale value of 255 is 0 in the inverse true value image; A second generating unit is configured to generate a first focusing map and a second focusing map corresponding to the target image based on the target image, the Gaussian map, the true value map and the inverse true value map, wherein the first focusing map and the second focusing map are in a complementary relationship; A second processing unit is used to input the first focus map and the second focus map into a pre-trained image fusion network to obtain a decision map output by the image fusion network, where the decision map is a binary map predicted by the image fusion network; a fusion unit, configured to fuse the first focused image, the second focused image and the decision graph to obtain a multi-focus fused image; Wherein, the second generating unit includes: A first sub-calculation unit, configured to apply a preset first calculation formula to calculate the target image, the Gaussian image, the true value image, and the inverse true value image, to generate a first focus image corresponding to the target image; A second calculation subunit is used to apply a preset second calculation formula to calculate the target image, the Gaussian image, the true value image and the inverse true value image to generate a second focus image corresponding to the target image; The first calculation formula is: ； The second calculation formula is: ； Among them, y1 is the first focus image, y2 is the second focus image, x is the target image, is the true value graph, is the inverse truth value graph, x blur is a Gaussian graph; The fusion unit comprises: a generating subunit, configured to calculate the first focused image, the second focused image and the decision graph by applying a preset third calculation formula to generate a multi-focus fusion image; The third calculation formula is: ； Among them, F fused is the fused image, F A is the first focused image, F B is the second focused image, F D is a graph decision graph, (x, y) is a pixel in the image.

5. The device according to claim 4, characterized in that Also includes: An acquisition unit, configured to acquire a first training image, a second training image, and a training target, wherein the first training image and the second training image are focus images in a complementary relationship, and the training target is a binary image corresponding to the first training image and the second training image; A training unit, configured to execute a training process corresponding to the image fusion network based on the first training image, the second training image and the training target; The training process includes: inputting the first training image and the second training image into the first coding network and the second coding network of the image fusion network respectively, obtaining first context information corresponding to the first training image output by the first coding network and second context information corresponding to the second training image output by the second coding network; fusing the first context information with the second context information to obtain fusion information, and inputting the fusion information into the decoder of the image fusion network to obtain a target binary image output by the decoder; A comparison unit is used to compare the target binary image with the training target to obtain a comparison result; when the comparison result does not meet the preset training conditions, a preset binary cross entropy function is applied to adjust the image fusion network and the training process is re-executed; when the comparison result meets the training conditions, the training of the image fusion network is terminated.

6. The device according to claim 4, characterized in that The fusion unit comprises: A first processing subunit is used to apply a preset conditional random field method CRF to perform boundary smoothing processing on the decision graph to obtain a processed decision graph; The fusion subunit is used to fuse the first focused image, the second focused image and the processed decision graph to obtain a multi-focus fused image.

Images