A multi-focus image fusion method and device based on deep feature perception, a terminal and a storage medium

Abstract

The disclosure provides a multi-focus image fusion method and device based on deep feature perception, a terminal and a storage medium, including obtaining a pair of partial focus images under the same scene, converting them into grayscale images, and taking them as input data of a fusion model; inputting the grayscale images into one branch of a double-flow network respectively, extracting and integrating the depth feature maps of the source images through the encoder and decoder of the U-module; obtaining the initial decision map through the feature fusion module of the depth feature map of each branch, and further performing decision reinforcement to obtain the final decision map; fusing the final decision map and the source image to obtain the fusion result; introducing a target function based on pixel sparse representation, and iteratively training on a large-scale training data set to obtain an optimal fusion model. The disclosure aims to extract the depth perception features of the source images, thereby obtaining a high-quality fusion result.

Description

Technical Field

[0001] This disclosure relates to the field of image fusion technology, and in particular to a multifocal image fusion method, apparatus, terminal and storage medium based on depth feature perception. Background Technology

[0002] Multifocal image fusion (MFIF) is a key technique for improving image sharpness by combining images captured at different depths of focus. Specifically, MFIF integrates images captured on different focal planes to produce a fully focused composite image. This method effectively addresses the problem of limited sharpness due to the limited depth of field of optical lenses, resulting in blurred objects outside this range. MFIF plays a crucial role in enhancing feature extraction, object recognition, and tracking.

[0003] Multifocal image fusion methods can be categorized into three types: transform domain, spatial domain, and deep learning-based methods. Transform domain methods transform the image to an alternative domain for fusion, then transform it back to the spatial domain for the final output. In contrast, spatial domain methods utilize linear or nonlinear methods to achieve fusion directly at the pixel or region level. Deep learning-based methods are efficient and adaptable, representing a significant improvement over traditional techniques. These methods leverage neural networks, enabling them to autonomously learn and extract complex features from source images, providing more sophisticated processing methods for multifocal image fusion. However, key challenges remain in feature extraction and fusion. The main problem is the loss of important information at these stages, which adversely affects the quality of the fusion result.

[0004] In two-branch deep learning networks, common feature fusion strategies, such as pixel-wise subtraction or concatenation, often fail to preserve complete gradient information in the feature maps. This leads to the loss of texture details and brightness. Furthermore, while subtraction after calculating spatial frequencies preserves gradient information, it neglects information exchange with the source image, resulting in the loss of detail in the fusion result. Convolutional processes in deep learning-based feature extraction often blur the distinction between important and unimportant features, leading to feature homogenization and thus masking crucial details. Moreover, the reliance on post-processing in many methods complicates the process and limits their generalization across various fusion scenarios.

[0005] In view of the above situation, there is an urgent need to study a robust model that overcomes the drawbacks of feature extraction and fusion in order to obtain high-quality multifocal image fusion results. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, this disclosure provides a multifocal image fusion method, apparatus, terminal, and storage medium based on depth feature perception.

[0007] According to a first aspect of this disclosure, a multifocal image fusion method based on depth feature awareness is provided, comprising:

[0008] Obtain partially focused image pairs from the same scene in the dataset to be trained as source images, convert them into grayscale images, and use them as input data for the fusion model to be trained;

[0009] A U-module is set up to extract depth features. The grayscale image is input into one branch of a two-stream network. The encoder and decoder of the U-module extract and integrate the features to obtain the depth feature map of the source image.

[0010] The deep feature map of each branch is processed by the feature fusion module to obtain the initial decision map, and further decision enhancement is performed to obtain the final decision map. The final decision map and the source image are then fused to obtain the fusion result.

[0011] Define the true value of the final decision map and the true value of the fusion result. Then, pass the final decision map, the fusion result, and the two true values ​​together through a target function based on pixel sparse representation. Perform supervised training on a large-scale dataset to iterate the encoding and decoding parameters of the U-module to obtain the fusion model.

[0012] The multi-focus image pairs are input into the fusion model, and the full-focus image is output.

[0013] Preferably, the grayscale images are input into one branch of a two-stream network, including:

[0014] The network architecture of the two-stream network consists of two identical sub-branches, each of which is responsible for processing one input. The two sub-branches share the same weights and structure, share parameters, and have consistent feature extraction.

[0015] Preferably, the depth feature map of the source image is extracted and integrated through the encoder and decoder of the U-module, including:

[0016] The grayscale image is processed by encoding and decoding. Then, a skip connection mechanism is used to fuse the low-level features extracted by the encoder with the high-level features extracted by the decoder in order to preserve the details and structural information of the image.

[0017] Preferably, the encoding process includes: two-dimensional convolution, downsampling, BatchNorm, and ReLU activation function; the decoding process includes: two-dimensional convolution, upsampling, BatchNorm, and ReLU activation function; and the skip connection mechanism includes: two-dimensional convolution, BatchNorm, and ReLU activation function.

[0018] Preferably, the step of fusing the deep features of each branch to obtain an initial decision map and then performing decision enhancement to obtain a final decision map includes:

[0019] The deep feature maps A and B on the two branches are used to obtain the output after feature fusion through formulas (1)-(4):

[0020]

[0021] In the formula, AP is the result of feature fusion, (x,y) are the location coordinates, A and B represent the two feature extraction branches respectively, and ESF is the local energy spatial frequency.

[0022]

[0023] In the formula, X represents the source image, X ′ This represents the depth feature map, where SF and E represent the spatial frequency and local energy operator operations, respectively.

[0024]

[0025] In the formula, CF and RF represent the row vector frequency and column vector frequency, respectively, and r is the kernel radius. In some embodiments, r is set to 5.

[0026] Preferably, the true values ​​of the final decision map and the fusion result are defined. The final decision map, the fusion result, and the two true values ​​are then subjected to supervised training on a large-scale dataset using an objective function based on pixel sparse representation to iterate the encoding and decoding parameters of the U-module, thereby obtaining the fusion model. Specifically, the supervised training on a large-scale dataset using an objective function based on pixel sparse representation includes:

[0027]

[0028] In the formula, and These represent the pixel sparse representation loss and the structural similarity loss, respectively:

[0029]

[0030] Where N is the number of pixels, w i p represents the weight of pixel i; i and q i These are the pixel values ​​in the final decision map and the true values ​​in the final decision map, respectively. Pixel weights allow the model to adjust its attention to different pixels based on the frequency of different pixels in the source image, thereby enhancing the model's performance in multifocal image fusion tasks. Specifically:

[0031]

[0032] In the formula:

[0033]

[0034] In the formula N and N represent the pixel value p, respectively. i The total number of occurrences and the total number of pixels in the two partially focused images. The formula is as follows:

[0035]

[0036] In the formula σ O σ I and σ OI This represents the average intensity and standard deviation of images O and I, as well as their covariance.

[0037] According to a second aspect of the present invention, a multifocal image fusion apparatus based on depth feature perception is provided, characterized in that it comprises:

[0038] The data preprocessing module is used to obtain partially focused image pairs in the same scene in the dataset to be trained as source images, convert them into grayscale images, and use them as input data for the fusion model to be trained.

[0039] The U-feature extraction and integration module is used to input the grayscale image into one branch of a two-stream network, and extract and integrate the feature maps of the source image through the encoder and decoder of the U-module to obtain the depth feature maps of the source image.

[0040] The feature fusion module is used to obtain an initial decision map by passing the deep feature map of each branch through the feature fusion module, and further perform decision enhancement to obtain a final decision map. The final decision map and the source image are then fused to obtain a fusion result.

[0041] The image fusion module is used to define the true value of the final decision map and the true value of the fusion result. The final decision map, the fusion result and the two true values ​​are passed together through a target function based on pixel sparse representation. Supervised training is performed on a large-scale dataset to iterate the encoding and decoding parameters of the U-module to obtain the fusion model.

[0042] The image output module is used to input multi-focus image pairs into the fusion model and output a full-focus image.

[0043] According to a third aspect of this disclosure, a terminal is provided that includes one or more processors;

[0044] Memory for storing the one or more processor-executable instructions;

[0045] Wherein, the one or more processors are configured as follows:

[0046] The method for multifocal image fusion based on depth feature perception is implemented.

[0047] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of the multifocal image fusion method based on depth feature perception.

[0048] The principle of this disclosed technical solution is as follows:

[0049] First, a U-module for extracting deep features was designed. This module transforms the source image according to task requirements and addresses feature homogenization by employing the encoder-decoder block structure and skip connection mechanism within the U-module, thus preserving key details. Skip connections establish direct connections between the encoder and decoder, fusing low-level features extracted by the encoder with high-level features from the decoder, thereby maintaining multi-scale and multi-level information. Second, a feature fusion module based on local energy spatial frequency was designed to fuse the deep feature map output from the U-module with the source image. Next, a pixel-sparse representation-based objective function suitable for multifocal image fusion tasks was designed and iteratively trained on a large-scale training dataset to obtain the optimal fusion model. Finally, the fused image is obtained by inputting a set of partially focused images into this model.

[0050] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0051] This disclosure provides a method, apparatus, terminal, and storage medium for multifocal image fusion based on depth feature perception. By designing a U-module and a feature fusion module based on depth feature perception, the model can effectively preserve multifocal image features, including boundary information and texture details. Furthermore, a supervised learning method is adopted, and a loss function based on pixel sparse representation is designed to encourage the model to focus on sparse pixels containing more boundary and texture features during training, thereby improving fusion performance and obtaining high-quality fusion results. Attached Figure Description

[0052] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the present invention and are not intended to limit the scope of this disclosure. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0053] Figure 1A flowchart of a multifocal image fusion method based on depth feature perception provided in this disclosure;

[0054] Figure 2 This is a schematic block diagram of a multifocal image fusion device based on depth feature perception provided in this disclosure;

[0055] Figure 3 This is a schematic diagram illustrating the implementation framework of a multifocal image fusion method based on depth feature perception provided in this disclosure. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0057] like Figure 1 and Figure 3 As shown, in one embodiment, a multifocal image fusion method based on depth feature awareness is provided, including the following steps:

[0058] Step 101: Obtain partial focused image pairs in the same scene from the dataset to be trained as source images, convert them into grayscale images, and use them as input data for the fusion model to be trained.

[0059] Step 102: Set up a U-module for extracting depth features. Input the grayscale image into one branch of a two-stream network. The network architecture of the two-stream network consists of two complete sub-branches. Each sub-branch is responsible for processing one input. The two sub-branches share the same weights and structure, share parameters, and have consistent feature extraction. The depth feature map of the source image is obtained by extracting and integrating the features through the encoder and decoder of the U-module.

[0060] In some embodiments, the grayscale image is processed by an encoding module and a decoding module, and then the low-level features extracted by the encoder are fused with the high-level features extracted by the decoder using a skip connection mechanism to preserve the details and structural information of the image. The resulting depth feature map is then used as the input for subsequent processing steps.

[0061] In some embodiments, the encoding process includes: two-dimensional convolution, downsampling, BatchNorm, and ReLU activation function; the decoding process includes: two-dimensional convolution, upsampling, BatchNorm, and ReLU activation function; the skip connection mechanism includes: two-dimensional convolution, BatchNorm, and ReLU activation function.

[0062] Step 103: The deep features of each branch are processed by the feature fusion module to obtain the initial decision map, and further decision enhancement is performed to obtain the final decision map;

[0063] In some embodiments, the deep feature maps A and B on the two branches are used to obtain the feature fusion output through formulas (1)-(4):

[0064]

[0065] In the formula, AP is the result of feature fusion, (x,y) are the location coordinates, A and B represent the two feature extraction branches respectively, and ESF is the local energy spatial frequency.

[0066]

[0067] In the formula, X represents the source image, X ′ This represents the depth feature map, where SF and E represent the spatial frequency and local energy operator operations, respectively.

[0068]

[0069] In the formula, CF and RF represent the row vector frequency and column vector frequency, respectively, and r is the kernel radius. In some embodiments, r is set to 5.

[0070] In some embodiments, a guided filter is applied to smooth the boundaries of the initial decision graph and further enhance and refine the features of the initial decision graph to obtain a final decision graph that is more accurate and vivid.

[0071] In some embodiments, the final decision map and the source image are fused to obtain a fusion result;

[0072] Step 104: In the constructed multifocal image fusion neural network structure based on deep feature perception, an objective function based on pixel sparse representation is introduced. Iterative training is performed on a large-scale training dataset to obtain the optimal fusion model. The specific content is as follows:

[0073]

[0074] In the formula, and These represent the pixel sparse representation loss and the structural similarity loss, respectively:

[0075]

[0076] Where N is the number of pixels, w i p represents the weight of pixel i; i and q iThese are the pixel values ​​in the final decision map and the true values ​​in the final decision map, respectively; pixel weights allow the model to adjust its attention to different pixels based on the frequency of different pixels in the source image, thereby enhancing the model's performance in multifocal image fusion tasks, specifically:

[0077]

[0078] In the formula:

[0079]

[0080] In the formula N and N represent the pixel value p, respectively. i The total number of occurrences and the total number of pixels in the two partially focused images. The formula is as follows:

[0081]

[0082] In the formula σ O σ I and σ OI This represents the mean intensity, standard deviation, and covariance of images O and I;

[0083] Step 105: Input the multi-focus image pair into the fusion model and output a full-focus image.

[0084] To objectively evaluate the comparison results, the average objective evaluation index of each fusion method was calculated on both datasets, and the results are shown in Tables 1 and 2. Higher objective evaluation index values ​​indicate better fusion performance. The tables show that the fusion method outperformed the compared methods on most metrics across each dataset. These results demonstrate the benefits of the U-module, the feature fusion module, and the objective function based on pixel sparse representation.

[0085]

[0086] Table 1

[0087]

[0088] Table 2

[0089] like Figure 2 A schematic block diagram of a multifocal image fusion device based on depth feature perception is provided in one embodiment. The device includes:

[0090] The data preprocessing module 210 is used to obtain partially focused image pairs in the same scene in the dataset to be trained as source images, convert them into grayscale images, and use them as input data for the fusion model to be trained.

[0091] The U-feature extraction and integration module 220 is used to input the grayscale image into one branch of a dual-stream network, and extract and integrate the source image's depth feature map through the encoder and decoder of the U-module.

[0092] The feature fusion module 230 is used to obtain an initial decision map by passing the deep feature map of each branch through the feature fusion module, and further perform decision enhancement to obtain a final decision map. The final decision map and the source image are then fused to obtain a fusion result.

[0093] Image fusion module 240 is used to define the true value of the final decision map and the true value of the fusion result, and to perform supervised training on a large-scale dataset by passing the final decision map, the fusion result and the two true values ​​together through a target function based on pixel sparse representation to iterate the encoding and decoding parameters of the U-module to obtain the fusion model;

[0094] The image output module 250 is used to input multi-focus image pairs into the fusion model and output a full-focus image.

[0095] This disclosure designs a U-module and a feature fusion module based on deep feature awareness, enabling the model to effectively preserve multifocal image features, including boundary information and texture details. It also employs a supervised learning method and designs a loss function based on pixel sparse representation to encourage the model to focus on sparse pixels containing more boundary and texture features during training, thereby improving fusion performance and obtaining high-quality fusion results.

[0096] Embodiments of this disclosure also provide a terminal, characterized in that it includes:

[0097] One or more processors;

[0098] Memory for storing the one or more processor-executable instructions;

[0099] Wherein, the one or more processors are configured as follows:

[0100] The multifocal image fusion method based on depth feature perception is executed.

[0101] Embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of the multifocal image fusion method based on depth feature perception.

[0102] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this disclosure can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0103] The embodiments described above are merely illustrative of several implementations of this disclosure, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this disclosure, and these all fall within the protection scope of this application. Therefore, the protection scope of this disclosure should be determined by the appended claims.

Claims

1. A multifocal image fusion method based on depth feature perception, characterized in that, Includes the following steps: Obtain partially focused image pairs from the same scene in the dataset to be trained as source images, convert them into grayscale images, and use them as input data for the fusion model to be trained; A U-module is set up to extract depth features. The grayscale image is input into one branch of a two-stream network. The encoder and decoder of the U-module extract and integrate the features to obtain the depth feature map of the source image. The deep feature map of each branch is processed by the feature fusion module to obtain an initial decision map, and further decision enhancement is performed to obtain a final decision map. The final decision map and the source image are then fused to obtain a fusion result. The process of fusing the deep feature maps of each branch to obtain an initial decision map, and then performing decision enhancement to obtain a final decision map, includes: The deep feature maps on the two branches , The output after feature fusion is obtained through formulas (1)-(4): （1） in The result of feature fusion, These are position coordinates. and These represent two feature extraction branches. For local energy spatial frequency: （2） in Represents the source image. Represents a depth feature map. and These represent the spatial frequency and local energy operator operations, respectively: （3） （4） in and These represent the row vector frequency and the column vector frequency, respectively. The radius of the core; Define the true value of the final decision graph and the true value of the fusion result. Combine the final decision graph, the fusion result, the true value of the final decision graph, and the true value of the fusion result with a pixel-sparse representation-based objective function. Perform supervised training on a large-scale dataset to iterate the encoding and decoding parameters of the U-module to obtain the fusion model. The multi-focus image pairs are input into the fusion model, and the full-focus image is output.

2. The multifocal image fusion method based on depth feature perception according to claim 1, characterized in that, The grayscale images are input into one branch of a two-stream network, including: The network architecture of the two-stream network consists of two identical sub-branches, each responsible for processing one input. The two sub-branches share the same weights and structure, and their feature extraction is consistent and their parameters are shared.

3. The multifocal image fusion method based on depth feature perception according to claim 1, characterized in that, The U-module's encoder and decoder extract and integrate the depth feature map of the source image, including: The grayscale image is processed by encoding and decoding. Then, a skip connection mechanism is used to fuse the low-level features extracted by the encoder with the high-level features extracted by the decoder in order to preserve the details and structural information of the image.

4. The multifocal image fusion method based on depth feature perception according to claim 3, characterized in that, The encoding process includes: two-dimensional convolution, downsampling, BatchNorm, and ReLU activation function; the decoding process includes: two-dimensional convolution, upsampling, BatchNorm, and ReLU activation function; the skip connection mechanism includes: two-dimensional convolution, BatchNorm, and ReLU activation function.

5. The multifocal image fusion method based on depth feature perception according to claim 1, characterized in that, Define the true value of the final decision graph and the true value of the fusion result. Then, train the final decision graph, the fusion result, the true value of the final decision graph, and the true value of the fusion result together on a large-scale dataset through a target function based on pixel sparse representation to iterate the encoding and decoding parameters of the U-module and obtain the fusion model.

6. The multifocal image fusion method based on depth feature perception according to claim 5, characterized in that, The supervised training on a large-scale dataset using an objective function based on pixel sparse representation includes: （5） In the formula, and These represent the pixel sparse representation loss and the structural similarity loss, respectively: （6） in Number of pixels Represents pixels The weights; and These are the pixel values ​​in the final decision map and the true values ​​in the final decision map, respectively. The degree of attention given to different pixels is adjusted based on the frequency of different pixels in the source image: （7） in: （8） in, and Each represents a pixel value. The total number of occurrences and the total number of pixels in the two partially focused images The formula is as follows: （9） In the formula, , , , and Representing an image and The average strength, standard deviation, and covariance.

7. A multifocal image fusion device based on depth feature perception, characterized in that, include: The data preprocessing module is used to obtain partially focused image pairs in the same scene in the dataset to be trained as source images, convert them into grayscale images, and use them as input data for the fusion model to be trained. The U-feature extraction and integration module is used to input the grayscale image into one branch of a two-stream network, and extract and integrate the feature maps of the source image through the encoder and decoder of the U-module to obtain the depth feature map of the source image. The feature fusion module is used to obtain an initial decision map by passing the deep feature map of each branch through the feature fusion module, and further perform decision enhancement to obtain a final decision map. The final decision map and the source image are then fused to obtain a fusion result. The process of fusing the deep feature maps of each branch to obtain an initial decision map, and then performing decision enhancement to obtain a final decision map, includes: The deep feature maps on the two branches , The output after feature fusion is obtained through formulas (1)-(4): （1） in The result of feature fusion, These are position coordinates. and These represent two feature extraction branches. For local energy spatial frequency: （2） in Represents the source image. Represents a depth feature map. and These represent the spatial frequency and local energy operator operations, respectively: （3） （4） in and These represent the row vector frequency and the column vector frequency, respectively. The radius of the core; The image fusion module is used to define the ground truth value of the final decision map and the ground truth value of the fusion result. The final decision map, the fusion result and the two ground truth values ​​are passed together through a target function based on pixel sparse representation. Supervised training is performed on a large-scale dataset to iterate the encoding and decoding parameters of the U-module to obtain the fusion model. The image output module is used to input multi-focus image pairs into the fusion model and output a full-focus image.

8. A terminal, characterized in that, include: One or more processors; Memory for storing the one or more processor-executable instructions; Wherein, the one or more processors are configured as follows: Perform the multifocal image fusion method based on depth feature perception as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multifocal image fusion method based on depth feature perception as described in any one of claims 1 to 6.

Images