Image fusion method, device, processor and electronic device

Abstract

The application discloses an image fusion method and device, a processor and an electronic device. It relates to the field of image processing. The method comprises the following steps: acquiring at least two images to be fused, wherein the at least two images indicate the same scene and have different focuses, and the at least two images comprise a first image and a second image; performing feature extraction processing on the first image to obtain first image features associated with the first image, and performing feature extraction processing on the second image to obtain second image features associated with the second image; performing global recursive processing on the first image features to obtain a first decision map associated with the image, and performing global recursive processing on the second image to obtain a second decision map associated with the image; and obtaining a target fusion image of the first image and the second image based on the first decision map and the second decision map. The application solves the problem of low accuracy of image fusion in the related art.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to an image fusion method, apparatus, processor, and electronic device. Background Technology

[0002] Traditional multi-focus image fusion methods are not comprehensive enough in preserving image details, resulting in insufficient extraction and retention of detailed information in complex scenes, and even defects in fused image distortion, thus leading to low accuracy of image fusion results.

[0003] Multi-focus image fusion technology aims to combine multiple images containing the same scene obtained under different focusing settings into a single, more complete, and sharp image. The resulting sharp image is more suitable for subsequent computer vision tasks.

[0004] In the aforementioned multi-focus image fusion scenarios, traditional multi-focus image fusion methods are not comprehensive enough in terms of image detail preservation, resulting in insufficient extraction and retention of detailed information in complex scenes, and even defects in fused image distortion, thus leading to technical problems of low image fusion accuracy.

[0005] There is currently no effective solution to the problem of low accuracy in image fusion in related technologies. Summary of the Invention

[0006] The main objective of this application is to provide an image fusion method, apparatus, processor, and electronic device to solve the problem of low accuracy in image fusion in related technologies.

[0007] To achieve the above objectives, according to one aspect of this application, an image fusion method is provided. The method includes: acquiring at least two images to be fused, wherein the at least two images indicate the same scene but have different focuses, and the at least two images include a first image and a second image; performing feature extraction processing on the first image to obtain first image features associated with the first image, and performing feature extraction processing on the second image to obtain second image features associated with the second image; performing global recursive processing on the first image features to obtain a first decision map associated with the image, and performing global recursive processing on the second image to obtain a second decision map associated with the second image, wherein the first decision map is used to indicate global decision information of the first image, and the second decision map is used to indicate global decision information of the second image; and obtaining a target fused image of the first image and the second image based on the first decision map and the second decision map.

[0008] As an optional approach, the above-mentioned global recursive processing of the first image features and the second image features to obtain the first decision map associated with the first image and the second decision map associated with the second image includes: recursively processing each pixel in the first image features to obtain the third image features, and recursively processing each pixel in the second image features to obtain the fourth image features; normalizing the third image features to obtain the first decision map, and normalizing the third image features to obtain the second decision map.

[0009] As an optional approach, the recursive processing of each pixel in the first image feature to obtain the third image feature includes: determining the current pixel from the first image feature, and determining other pixels of the current pixel in the target direction, wherein the target direction includes: up, down, left, and right; adjusting the current pixel using the other pixels to obtain the adjusted current pixel, wherein the third image feature includes the adjusted current pixel; and normalizing the third image feature to obtain the first decision map includes: normalizing the third image feature using an objective function to obtain the first decision map.

[0010] As an optional approach, obtaining the target fused image of the first image and the second image based on the first decision map and the second decision map includes: performing correction processing on the first decision map to obtain a third decision map associated with the first image, and performing correction processing on the second decision map to obtain a fourth decision map associated with the second image; performing region smoothing processing on the third decision map to obtain a fifth decision map associated with the first image, and performing region smoothing processing on the fourth decision map to obtain a sixth decision map associated with the second image; and determining the target fused image of the first image and the second image based on the first fusion result of the first image and the fifth decision map, and the second fusion result of the second image and the sixth decision map.

[0011] As an optional approach, the above-mentioned correction processing of the first decision map to obtain the third decision map associated with the first image includes: using a target correction algorithm to correct the first decision map and determining the corrected decision map as the third decision map; performing region smoothing processing on the third decision map to obtain the fifth decision map associated with the first image includes: using a target region smoothing algorithm to divide the third decision map into N windows and determining the fifth decision map based on the divided N windows, where N is a positive integer.

[0012] As an optional approach, after acquiring at least two images to be fused, the method further includes: inputting the first image and the second image into a preset model for feature extraction processing to obtain first image features associated with the first image and second image features associated with the second image; using the preset model, performing global recursive processing, correction processing, and regional smoothing processing on the first image features and the second image features to obtain a fifth decision map associated with the first image and a sixth decision map associated with the second image; and determining and outputting the target fused image of the first image and the second image based on the first image, the second image, the fifth decision map, and the sixth decision map.

[0013] As an optional approach, before inputting the first image and the second image into the preset model for feature extraction processing, the method further includes: determining an initialization model for obtaining the preset model; training the initialization model using a training sample set, wherein the training sample set includes multiple training sample pairs, each training sample pair including a first training image, a second training image, and a ground truth image, wherein the first training image and the second training image are used to input the first model to obtain a fused training image, the fused training image and the ground truth image are used to determine a first loss value at the image block level and a second loss value at the pixel level for the first model, and the joint loss value is determined based on the first loss value and the second loss value; if the joint loss value is less than a preset convergence threshold, the initialization model is determined to be trained successfully, and the trained initialization model is determined as the preset model.

[0014] To achieve the above objectives, according to another aspect of this application, an image fusion apparatus is provided. The apparatus includes: an acquisition unit for acquiring at least two images to be fused, wherein the at least two images indicate the same scene but have different focuses, and the at least two images include a first image and a second image; an extraction unit for performing feature extraction processing on the first image to obtain first image features associated with the first image, and performing feature extraction processing on the second image to obtain second image features associated with the second image; a recursive unit for performing global recursive processing on the first image features to obtain a first decision map associated with the image, and performing global recursive processing on the second image to obtain a second decision map associated with the second image, wherein the first decision map indicates global decision information of the first image, and the second decision map indicates global decision information of the second image; and a determination unit for obtaining a target fused image of the first image and the second image based on the first decision map and the second decision map.

[0015] As an optional scheme, the recursive unit includes: a recursive module, used to recursively process each pixel in the first image feature to obtain the third image feature, and to recursively process each pixel in the second image feature to obtain the fourth image feature; and a normalization module, used to normalize the third image feature to obtain the first decision map, and to normalize the third image feature to obtain the second decision map.

[0016] As an optional approach, the recursive module includes: a determination submodule, used to determine the current pixel from the first image features, and to determine other pixels of the current pixel in the target direction, wherein the target direction includes: up, down, left, and right directions; an adjustment submodule, used to adjust the current pixel using the other pixels to obtain the adjusted current pixel, wherein the third image features include the adjusted current pixel; and a normalization module including: a normalization submodule, used to perform a normalization transformation on the third image features using an objective function to obtain the first decision map.

[0017] As an optional scheme, the determining unit includes: a correction module, used to perform correction processing on the first decision map to obtain a third decision map associated with the first image, and to perform correction processing on the second decision map to obtain a fourth decision map associated with the second image; a smoothing module, used to perform regional smoothing processing on the third decision map to obtain a fifth decision map associated with the first image, and to perform regional smoothing processing on the fourth decision map to obtain a sixth decision map associated with the second image; and a first determining module, used to determine the target fused image of the first image and the second image based on the first fusion result of the first image and the fifth decision map, and the second fusion result of the second image and the sixth decision map.

[0018] As an optional scheme, the correction module includes: a correction submodule, used to correct the first decision graph using a target correction algorithm, and determine the corrected decision graph as the third decision graph; the smoothing module includes: a smoothing submodule, used to divide the third decision graph into N windows using a target region smoothing algorithm, and determine the fifth decision graph based on the divided N windows, where N is a positive integer.

[0019] As an optional solution, the device further includes: an extraction processing module, used to, after acquiring at least two images to be fused, input the first image and the second image into a preset model for feature extraction processing respectively, to obtain first image features associated with the first image and second image features associated with the second image; a recursive processing module, used to, after acquiring at least two images to be fused, use the preset model to perform global recursive processing, correction processing, and region smoothing processing on the first image features and the second image features respectively, to obtain a fifth decision map associated with the first image and a sixth decision map associated with the second image; and a fusion processing module, used to, after acquiring at least two images to be fused, determine and output the target fused image of the first image and the second image based on the first image, the second image, the fifth decision map, and the sixth decision map.

[0020] As an optional solution, the device further includes: a second determining module, used to determine an initialization model for obtaining the preset model before inputting the first image and the second image into the preset model for feature extraction processing respectively; a training module, used to train the initialization model using a training sample set before inputting the first image and the second image into the preset model for feature extraction processing respectively, wherein the training sample set includes multiple training sample pairs, each training sample pair including a first training image, a second training image and a ground truth image, wherein the first training image and the second training image are used to input the first model to obtain a fused training image, the fused training image and the ground truth image are used to determine a first loss value at the image block level and a second loss value at the pixel level for the first model, and the joint loss value is determined based on the first loss value and the second loss value; a third determining module, used to determine that the initialization model training is complete before inputting the first image and the second image into the preset model for feature extraction processing respectively, if the joint loss value is less than a preset convergence threshold, and determine the initialization model after training as the preset model.

[0021] This application employs the following steps: acquiring at least two images to be fused, wherein the at least two images indicate the same scene but have different focuses, and the at least two images include a first image and a second image; performing feature extraction processing on the first image to obtain first image features associated with the first image, and performing feature extraction processing on the second image to obtain second image features associated with the second image; performing global recursive processing on the first image features to obtain a first decision map associated with the image, and performing global recursive processing on the second image to obtain a second decision map associated with the second image, wherein the first decision map is used to indicate the global decision information of the first image, and the second decision map is used to indicate the global decision information of the second image; and obtaining a target fused image of the first image and the second image based on the first decision map and the second decision map. After acquiring the images to be fused and extracting features, the extracted features are subjected to global recursive processing, resulting in a decision map that can indicate the global decision information of the image. Image fusion is then performed based on the decision map containing global information, ensuring that the information processing during the image fusion process is sufficiently comprehensive. This effectively improves the accuracy of image fusion and solves the technical problem of low accuracy in image fusion in related technologies. Attached Figure Description

[0022] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0023] Figure 1 This is a flowchart of the image fusion method provided according to the embodiments of this application;

[0024] Figure 2 This is a schematic diagram of an image fusion method provided according to an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of an image fusion method provided according to an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of an image fusion apparatus provided according to an embodiment of this application;

[0027] Figure 5 This is a schematic diagram of an image fusion electronic device provided according to an embodiment of this application. Detailed Implementation

[0028] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are information and data authorized by the user or fully authorized by all parties. For example, this system has an interface with relevant users or organizations. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving consent information from the aforementioned user or organization.

[0032] The present invention will now be described in conjunction with preferred implementation steps. Figure 1 This is a flowchart of the image fusion method provided according to the embodiments of this application, such as... Figure 1 As shown, the method includes the following steps:

[0033] Step S101: Obtain at least two images to be fused, wherein the at least two images indicate the same scene but have different focuses, and the at least two images include a first image and a second image;

[0034] Step S102: Perform feature extraction processing on the first image to obtain the first image features associated with the first image, and perform feature extraction processing on the second image to obtain the second image features associated with the second image;

[0035] Step S103: Perform global recursive processing on the features of the first image to obtain a first decision map associated with the image, and perform global recursive processing on the second image to obtain a second decision map associated with the second image. The first decision map is used to indicate the global decision information of the first image, and the second decision map is used to indicate the global decision information of the second image.

[0036] Step S104: Based on the first decision map and the second decision map, obtain the target fusion image of the first image and the second image.

[0037] Optionally, in this embodiment, the image fusion method described above can be used, but is not limited to, multi-focus image fusion scenarios. The multi-focus image fusion technique in these scenarios aims to fuse multiple images containing the same scene obtained under different focusing settings into a single, fully clear image with more complete information. The resulting fully clear image is more suitable for subsequent computer vision tasks, such as recognition and supervision.

[0038] Optionally, in multi-focus image fusion scenarios, traditional multi-focus image fusion methods can be used, but are not limited to, such as spatial domain-based fusion methods and transform domain-based fusion methods. However, these methods require manual design of activity level detection and image fusion rules, which greatly increases the difficulty of algorithm design and the complexity of the algorithm. Furthermore, it is difficult to achieve satisfactory fusion results at the boundary between focused and non-focused regions.

[0039] Optionally, in multi-focus image fusion scenarios, deep learning-based multi-focus image fusion methods can also be used, such as image transformation-based fusion methods and decision map-based fusion methods. However, image transformation-based deep learning fusion methods rely on the accuracy of the training dataset, and the preservation of image details needs improvement. Decision map-based deep learning fusion methods, because they obtain the image's decision map (0 or 1) through the algorithm model, result in sharpness at non-focused edges, which does not conform to human visual perception, leading to poor performance.

[0040] Optionally, in multi-focus image fusion scenarios, the above-mentioned image fusion method can also be used, but not limited to, after acquiring the image to be fused and extracting features, the extracted features are subjected to global recursive processing, so that a decision map that can indicate global decision information of the image is obtained after recursive processing. Then, image fusion is performed based on the decision map of global information, so that the information processing in the image fusion process is comprehensive enough, thereby achieving the technical effect of effectively improving the accuracy of image fusion and solving the problems of inaccurate or poor image fusion.

[0041] Optionally, in this embodiment, the at least two images to be fused may be, but are not limited to, images that indicate the same scene and have different focus settings. Indicating the same scene may be, but is not limited to, indicating that the captured images of the at least two images are the same, and different focus settings may be, but are not limited to, indicating that the images of the at least two images have different display sharpness.

[0042] To further illustrate, such as Figure 2 As shown, Figure 2 (a) in the text indicates the two actual entities in the current scene. Figure 2 (b) and Figure 2 (c) represents images of the same current scene. The two images are identical except for focus settings (a solid circle indicates the object is in sharp focus, while a dashed circle indicates the object is in blurry focus). Figure 2 The scenario in (a) of the text.

[0043] Optionally, in this embodiment, when the first image and the second image to be fused are obtained, feature extraction processing is performed on the first image and the second image respectively to obtain the first image features associated with the first image and the second image features associated with the second image.

[0044] It should be noted that this embodiment does not actually limit the specific implementation of the above feature extraction process. Appropriate feature extraction methods can be selected based on specific application scenarios and requirements. For example, Scale-Invariant Feature Transform (SIFT): by detecting and describing key points in an image, image features with scale invariance are obtained; Principal Component Analysis (PCA): by performing dimensionality reduction on the image, representative principal component features are obtained; Deep learning features: by using deep learning models, such as Convolutional Neural Networks (CNNs), high-level semantic features of the image are extracted.

[0045] Optionally, in this embodiment, global recursive processing can be used, but is not limited to, to instruct the application of a target recursive algorithm to each pixel indicated by the image features and other pixels in the target direction of each pixel to obtain global decision information, so as to obtain a decision map corresponding to the image features.

[0046] Optionally, in this embodiment, taking the global recursive processing of the first image feature as an example, the current pixel can be adjusted by using other pixels in the target direction to obtain the adjusted current pixel. After all pixels of the first image feature have been adjusted, the first decision map associated with the first image is obtained based on the adjusted pixels.

[0047] Optionally, in this embodiment, the target direction may include, but is not limited to, the up direction, the down direction, the left direction, and the right direction.

[0048] It should be noted that other pixels in different directions and at different distances from the current pixel may, but are not limited to, have different adjustment weights. For example, the closer other pixels are to the current pixel, the greater their adjustment weight and the higher their influence.

[0049] It should be noted that, but not limited to, setting a preset number of other pixels is possible. For example, the number of other pixels associated with each current pixel may have a maximum preset threshold. If the number of other pixels exceeds the maximum preset threshold, the selection is based on the distance from the current pixel in the target direction, and the other pixel closest to the maximum preset threshold is selected.

[0050] Optionally, in this embodiment, when a first decision map and a second decision map are obtained, the first decision map and the second decision map can be corrected and region smoothed respectively, and the first image can be adjusted based on the first decision map after the above processing, and the second image can be adjusted based on the second decision map after the above processing, to obtain the adjusted first image and the adjusted second image, and then the target fused image can be obtained by fusing the adjusted first image and the adjusted second image.

[0051] The embodiments provided in this application acquire at least two images to be fused, wherein the at least two images indicate the same scene but have different focuses, and the at least two images include a first image and a second image. Feature extraction processing is performed on the first image to obtain first image features associated with the first image, and feature extraction processing is performed on the second image to obtain second image features associated with the second image. Global recursive processing is performed on the first image features to obtain a first decision map associated with the image, and global recursive processing is performed on the second image to obtain a second decision map associated with the second image. The first decision map is used to indicate the global decision information of the first image, and the second decision map is used to indicate the global decision information of the second image. Based on the first and second decision maps, a target fused image of the first and second images is obtained. After acquiring the images to be fused and extracting features, global recursive processing is performed on the extracted features, resulting in a decision map that indicates the global decision information of the image. Image fusion is then performed based on this global decision map, ensuring that the information processing during the image fusion process is sufficiently comprehensive, thereby effectively improving the accuracy of image fusion.

[0052] As an optional approach, global recursive processing is performed on the first image features and the second image features to obtain a first decision map associated with the first image and a second decision map associated with the second image, including:

[0053] S1, recursively process each pixel in the first image feature to obtain the third image feature, and recursively process each pixel in the second image feature to obtain the fourth image feature.

[0054] S2, normalize the third image features to obtain the first decision map, and normalize the third image features to obtain the second decision map.

[0055] Optionally, in this embodiment, taking the global recursive processing of the first image feature as an example, the current pixel can be adjusted by using other pixels in the target direction to obtain the adjusted current pixel. After all pixels of the first image feature have been adjusted, the third image feature after the recursive processing of the first image feature is obtained.

[0056] Optionally, in this embodiment, the normalization process may, but is not limited to, instructing the image features to be converted into a decision map, for example, using an objective function to normalize the third image features to obtain a first decision map.

[0057] Through the embodiments provided in this application, each pixel in the first image feature is recursively processed to obtain the third image feature, and each pixel in the second image feature is recursively processed to obtain the fourth image feature; the third image feature is normalized to obtain the first decision map, and the third image feature is normalized to obtain the second decision map.

[0058] As an optional approach, recursively processing each pixel in the first image feature yields the third image feature, which includes:

[0059] S1, determine the current pixel from the first image features, and determine other pixels of the current pixel in the target direction, wherein the target direction includes: up direction, down direction, left direction, and right direction;

[0060] S2, using other pixels, adjust the current pixel to obtain the adjusted current pixel, wherein the third image feature includes the adjusted current pixel;

[0061] The third image features are normalized to obtain the first decision map, which includes:

[0062] S3. Using the objective function, the features of the third image are normalized and transformed to obtain the first decision map.

[0063] Optionally, in this embodiment, each pixel is adjusted using the target pixel (i.e., the other pixels mentioned above) associated with each pixel to obtain the adjusted pixels, and the third image feature after the first image feature is adjusted is obtained after all pixels are adjusted.

[0064] It should be noted that the above pixel adjustment may include, but is not limited to: when N target pixels associated with the current first pixel are obtained, according to the weight of each pixel in the N target pixels, the image information indicated by each pixel in the N target pixels is fused with the image information indicated by the first pixel, so as to adjust and optimize the image information indicated by the first pixel, so that the adjusted and optimized first pixel indicates as much global information as possible.

[0065] Optionally, in this embodiment, the normalization process may, but is not limited to, instructing the image features to be converted into a decision map, for example, using an objective function to normalize the third image features to obtain a first decision map.

[0066] Optionally, in this embodiment, the objective function may be, but is not limited to, obtaining the association scores of each pixel indicated by the image features, classifying or otherwise processing each pixel based on the association scores, and generating a corresponding decision map after classification or other processing. For example, the objective function may be, but is not limited to, the softmax function.

[0067] The embodiments provided in this application determine the current pixel from the first image features and other pixels in the target direction, where the target direction includes: up, down, left, and right. The current pixel is adjusted using these other pixels to obtain an adjusted current pixel, where the third image feature includes the adjusted current pixel. Normalizing the third image feature to obtain a first decision map involves using a target function to perform a normalization transformation on the third image feature. By recursively combining information from pixels in multiple directions, the recursively processed image features indicate more comprehensive global information. Image fusion is then performed based on the decision map containing global information, ensuring sufficient information processing during image fusion and effectively improving the accuracy of image fusion.

[0068] As an optional approach, based on the first decision map and the second decision map, the target fused image obtained from the first image and the second image includes:

[0069] S1, perform correction processing on the first decision graph to obtain the third decision graph associated with the first image, and perform correction processing on the second decision graph to obtain the fourth decision graph associated with the second image;

[0070] S2, perform region smoothing on the third decision graph to obtain the fifth decision graph associated with the first image, and perform region smoothing on the fourth decision graph to obtain the sixth decision graph associated with the second image;

[0071] S3, based on the first fusion result of the first image and the fifth decision map, and the second fusion result of the second image and the sixth decision map, determine the target fused image of the first image and the second image.

[0072] Optionally, in this embodiment, after obtaining the first decision map and the second decision map, the first decision map and the second decision map are respectively corrected, and then the corrected third decision map and the fourth decision map are respectively subjected to region smoothing to obtain the corrected and smoothed fifth decision map and the sixth decision map.

[0073] Optionally, in this embodiment, the correction process may be used, but is not limited to, to instruct the first decision graph to be corrected using a target correction algorithm, and to determine the corrected decision graph as the third decision graph.

[0074] Optionally, in this embodiment, the region smoothing process may be used, but is not limited to, to instruct the third decision graph to be divided into N windows using a target region smoothing algorithm, and to determine the fifth decision graph based on the N windows after division.

[0075] Optionally, in this embodiment, the first image is fused and adjusted using the fifth decision graph to obtain the first fusion result, and the second image is fused and adjusted using the sixth decision graph to obtain the second fusion result. The first fusion result and the second fusion result are then fused to obtain the target fused image.

[0076] The embodiments provided in this application involve correcting a first decision map to obtain a third decision map associated with the first image, and correcting a second decision map to obtain a fourth decision map associated with the second image; performing region smoothing on the third decision map to obtain a fifth decision map associated with the first image, and performing region smoothing on the fourth decision map to obtain a sixth decision map associated with the second image; based on the first fusion result of the first image and the fifth decision map, and the second fusion result of the second image and the sixth decision map, a target fused image of the first image and the second image is determined. By performing correction processing to achieve "self-correction" in the low dimension of the decision map, and by performing region smoothing processing to achieve "other-party correction" in the high dimension of the decision map, the technical effect of comprehensively improving the information accuracy of the decision map is achieved.

[0077] As an optional approach, the third decision map associated with the first image is obtained by correcting the first decision map, including:

[0078] S1, use the target correction algorithm to correct the first decision graph, and determine the corrected decision graph as the third decision graph;

[0079] S2, performing region smoothing on the third decision map to obtain the fifth decision map associated with the first image, includes:

[0080] S3. Using the target region smoothing algorithm, the third decision graph is divided into N windows, and the fifth decision graph is determined based on the N windows after division, where N is a positive integer.

[0081] Optionally, in this embodiment, the target correction algorithm may be used, but is not limited to, to instruct the first decision map / second decision map to undergo convolutional processing. The correction processing of the first decision map is described below as an example:

[0082] D2 = soft max(conv(f) add (D1), w), μ)

[0083] Where D2 is the third decision map obtained after correction, D1 is the first decision map to be corrected, conv(, w) indicates the convolution operation, w indicates the convolution parameter set, and f add() indicates an addition operation, where μ is the parameter of the soft max function.

[0084] Optionally, in this embodiment, the target region smoothing algorithm can be used, but is not limited to, to instruct the third decision graph / fourth decision graph to perform region smoothing processing to obtain several windows after division, and to adjust / correct the current window using the decision information of the previous and next windows. The following description uses the region smoothing processing of the third decision graph as an example:

[0085] D3 = f slide (D2, a)

[0086] Where D3 is the fifth decision graph obtained after region smoothing, D2 is the third decision graph to be smoothed, and f slide (,a) is used to indicate the target region smoothing algorithm, and a is used to indicate the region size. For example, when a=3, it means that the size of the sliding window is 3x3.

[0087] To further illustrate, the above f slide (a) The algorithm can be computed, but is not limited to, the following algorithms:

[0088]

[0089] Among them, D n 3 The fifth decision graph (n=1) / sixth decision graph (n=2) obtained after region smoothing, D n 2 For the third decision graph (n=1) / fourth decision graph (n=2) before region smoothing, avg() represents the averaging operation, and (m,i) represents the pixel in the m-th row and i-th column.

[0090] The embodiments provided in this application utilize a target correction algorithm to correct the first decision graph, and determine the corrected decision graph as the third decision graph. A target region smoothing algorithm is then used to divide the third decision graph into N windows, and a fifth decision graph is determined based on these N windows, where N is a positive integer. By employing the target correction algorithm and the target region smoothing algorithm, "self-correction" in the low-dimensional aspect and "other-correction" in the high-dimensional aspect of the decision graph are achieved, respectively, thereby comprehensively improving the information accuracy of the decision graph.

[0091] As an alternative approach, after obtaining at least two images to be fused, the method further includes:

[0092] S1, Input the first image and the second image into the preset model and perform feature extraction processing respectively to obtain the first image features associated with the first image and the second image features associated with the second image;

[0093] S2, using a preset model, perform global recursive processing, correction processing, and region smoothing processing on the first image features and the second image features respectively to obtain the fifth decision map associated with the first image and the sixth decision map associated with the second image.

[0094] S3, based on the first image, the second image, the fifth decision map, and the sixth decision map, determines and outputs the target fused image of the first image and the second image.

[0095] Optionally, in this embodiment, the preset model may be, but is not limited to, a model obtained by training and optimizing the initial model multiple times.

[0096] Optionally, in this embodiment, when at least two images to be fused are obtained, the first image and the second image can be input into a preset model for feature extraction, global recursive processing, correction processing, and region smoothing processing to obtain a fifth decision map associated with the first image and a sixth decision map associated with the second image. Based on the first fusion result of the first image and the fifth decision map, and the second fusion result of the second image and the sixth decision map, the target fused image of the first image and the second image is obtained.

[0097] The embodiments provided in this application involve inputting a first image and a second image into a preset model for feature extraction, resulting in first image features associated with the first image and second image features associated with the second image. Using the preset model, global recursive processing, correction processing, and region smoothing processing are performed on the first and second image features respectively, resulting in a fifth decision map associated with the first image and a sixth decision map associated with the second image. Based on the first image, the second image, the fifth decision map, and the sixth decision map, the target fused image of the first and second images is determined and output. Using a trained model for image fusion effectively improves the processing efficiency of image fusion, and allows for further selective adjustment and optimization of the trained model based on the image fusion results, thereby improving the accuracy of subsequent image fusion.

[0098] As an optional approach, before inputting the first image and the second image into a preset model for feature extraction processing, the method further includes:

[0099] S1, determine the initialization model used to obtain the preset model;

[0100] S2, the initialization model is trained using a training sample set, wherein the training sample set includes multiple training sample pairs, each training sample pair includes a first training image, a second training image and a ground truth image, wherein the first training image and the second training image are used as input to the first model to obtain a fused training image, the fused training image and the ground truth image are used to determine the first loss value of the first model at the image block level and the second loss value at the pixel level, and the joint loss value is determined based on the first loss value and the second loss value;

[0101] S3. If the joint loss value is less than the preset convergence threshold, determine that the initialization model training is complete, and determine the initialization model after training as the preset model.

[0102] Optionally, in this embodiment, a general training dataset of multi-focus images is used to train the network. Since the images in the dataset have arbitrary sizes, during training, the images can be uniformly transformed to a size of 180×180, but are not limited to this. On the training dataset, the obtained fused images are trained using a deep supervision mechanism to train the algorithm network, optimize the joint loss function, complete the network training, and obtain the network model parameters.

[0103] Optionally, in this embodiment, the joint loss value may be, but is not limited to, the sum of the first loss value and then the second loss value, wherein the first loss value may be, but is not limited to, the image patch-level loss value, corresponding to the structural similarity function L. SSIM The second loss value can be, but is not limited to, a pixel-level loss value, corresponding to the mean squared error loss function L. MSR .

[0104] Structural similarity loss function L SSIM The definition is as follows:

[0105]

[0106] L SSIM =1-SSIM(G,P)

[0107] Where, μ G μ P C1 and C2 represent the mean values ​​of the ground truth image and the fused image, respectively; C1 and C2 are two constants used to prevent division by zero errors, set to 0.01 and 0.03, respectively; σ G , σ P σ represents the variance between the ground truth image and the fused image; GP This represents the covariance between the ground truth image and the fused image.

[0108] Mean squared error loss function L MSR The definition is as follows:

[0109]

[0110] Where H and W represent the dimensions of the image; G and P represent the ground truth image and the fused image, respectively; and (i,j) represents the spatial location in the image.

[0111] The embodiments provided in this application determine an initialization model for obtaining a preset model. The initialization model is trained using a training sample set, which includes multiple training sample pairs. Each training sample pair includes a first training image, a second training image, and a ground truth image. The first and second training images are input into the first model to obtain a fused training image. The fused training image and the ground truth image are used to determine a first loss value at the image block level and a second loss value at the pixel level for the first model. The joint loss value is determined based on the first and second loss values. If the joint loss value is less than a preset convergence threshold, the initialization model is determined to be trained successfully, and the trained initialization model is identified as the preset model. By jointly calculating the loss values ​​at two levels, a more accurate model loss value can be obtained, making the image fusion results obtained by the trained preset model more accurate in subsequent image fusion applications, thereby achieving the technical effect of improving the accuracy of image fusion.

[0112] As an alternative approach, the above image fusion method is applied to a multi-focus image fusion scenario based on a recursive global information self-correcting decision map. The specific steps include:

[0113] Step 1, Image Feature Preprocessing:

[0114] Dual-channel image processing is employed to extract decision maps from two pre-registered multi-focus images. First, image preprocessing yields easily processed 64-channel depth image features Fn. d d = 1, n = 1 or 2 (Fn) d This represents the d-th feature of the nth input image, specifically when d=0 it represents the input image.

[0115] Step 2: Construct a recursive global decision information acquisition model

[0116] The preprocessed image features Fn obtained in step 1 d Iterate through Fn one by one d Given d = 2, n = 1 or 2 pixels, and using a recursive algorithm to obtain global information for the pixels in the four directions (top, bottom, left, and right), then Fn 2 It can be calculated using the following formula:

[0117] Fn 2 =f(Fn) 1 a)

[0118] Where f(, a) represents a recursive algorithm, The parameters represent the recursive algorithm.

[0119] Step 3, construct the calibration model:

[0120] The image features Fn obtained in step 2 2 Use softmax() to obtain the decision graph Dn 1 :

[0121] Dn 1 =softmax(Fn 2 ,μ), where μ represents the parameter of the softmax() function.

[0122] Step 4, Decision Map Correction:

[0123] The decision graph Dn obtained in step 3 1 The decision map Dn is obtained using a calibration algorithm. 2 :

[0124] Dn 2 =soft max(conv(fadd(Dn) 1 ), w), μ)

[0125] Where conv(, w) represents the convolution operation, w represents the convolution parameter set, and fadd represents the addition operation.

[0126] For the obtained decision graph Dn 2 A sliding window smoothing operation is used to obtain the final decision graph Dn. 3 :

[0127] Dn 3 =fslide(Dn) 2 a)

[0128] Where fslide(, a) represents the region smoothing algorithm, and a represents the region size. In this algorithm, the value is 3, which means that the sliding window size is 3x3.

[0129] The region smoothing algorithm fslide(,a) can be calculated as follows:

[0130]

[0131] Here, avg() represents the averaging operation, and (m,i) represents the pixel in the m-th row and i-th column.

[0132] Step 5, construct the joint loss function

[0133] To optimize the quality of the fused images, a joint loss function LMS = LSSIM + LMSE is used, which includes a block-level loss (structural similarity loss function LSSIM) and a pixel-level loss (mean squared error loss function LMSE).

[0134] The mean squared error loss function (LMSE) is defined as follows:

[0135]

[0136] Where H and W represent the dimensions of the image; G and P represent the ground truth image and the fused image, respectively; and (i,j) represents the spatial location in the image.

[0137] The structural similarity loss function LSSIM is defined as follows:

[0138]

[0139] L SSIM =1-SSIM(G,P)

[0140] Where, μ G μ P C1 and C2 represent the mean values ​​of the ground truth image and the fused image, respectively; C1 and C2 are two constants used to prevent division by zero errors, set to 0.01 and 0.03, respectively; σ G , σ P σ represents the variance between the ground truth image and the fused image; GP This represents the covariance between the ground truth image and the fused image.

[0141] Step 6, Algorithm Model Training:

[0142] The network was trained using a general training dataset of multi-focus images. Since the images in the dataset have arbitrary sizes, they were uniformly transformed to 180×180 during training. On the training dataset, the fused images obtained in step (3) were used to train the algorithm network using a deep supervision mechanism, optimize the joint loss function, complete the network training, and obtain the network model parameters.

[0143] Step 7, Construct the fused image

[0144] Step 7.1: The fused image F′ can be calculated using the following formula:

[0145]

[0146] To further illustrate, such as Figure 3As shown in (a), after the algorithm model has been trained, the trained algorithm model is used to fuse the input multifocus image 1 and multifocus image 2 to output the target fused image. The training process of the algorithm model is as follows: Figure 3 As shown in (b), taking the multi-focus training image 1 and multi-focus training image 2 included in the training dataset as an example, after image preprocessing to obtain the corresponding image features of multi-focus training images 1 and 2, global decision information is further obtained based on the recursive algorithm to generate the corresponding decision map, and the decision map is corrected and related processing is performed to obtain the final decision information corresponding to multi-focus training images 1 and 2. Based on the final decision information of multi-focus training image 1 and multi-focus training image 2, and the final decision information of multi-focus training image 2, a fused image is obtained. Based on the fused image and the ground image included in the training dataset, the loss function is minimized and compared to train the algorithm model until the joint loss value of the algorithm model meets the preset convergence condition, such as being less than the preset convergence threshold.

[0147] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0148] This application also provides an image fusion apparatus. It should be noted that the image fusion apparatus of this application can be used to execute the image fusion method provided in this application. The image fusion apparatus provided in this application is described below.

[0149] Figure 4 This is a schematic diagram of an image fusion apparatus according to an embodiment of this application. Figure 4 As shown, the device includes:

[0150] The acquisition unit 401 is used to acquire at least two images to be fused, wherein the at least two images indicate the same scene but have different focuses, and the at least two images include a first image and a second image;

[0151] The extraction unit 402 is used to perform feature extraction processing on the first image to obtain the first image features associated with the first image, and to perform feature extraction processing on the second image to obtain the second image features associated with the second image;

[0152] The recursive unit 403 is used to perform global recursive processing on the features of the first image to obtain a first decision graph associated with the image, and to perform global recursive processing on the second image to obtain a second decision graph associated with the second image. The first decision graph is used to indicate the global decision information of the first image, and the second decision graph is used to indicate the global decision information of the second image.

[0153] The determining unit 404 is used to obtain a target fusion image of the first image and the second image based on the first decision map and the second decision map.

[0154] Optionally, in the image fusion apparatus provided in this application embodiment, the recursive unit 404 includes:

[0155] The recursive module is used to recursively process each pixel in the first image feature to obtain the third image feature, and to recursively process each pixel in the second image feature to obtain the fourth image feature.

[0156] The normalization module is used to normalize the features of the third image to obtain the first decision map, and to normalize the features of the third image to obtain the second decision map.

[0157] Optionally, in the image fusion apparatus provided in this application embodiment, the recursive module includes:

[0158] The determination submodule is used to determine the current pixel from the first image features, and to determine other pixels of the current pixel in the target direction, wherein the target direction includes: up direction, down direction, left direction, and right direction;

[0159] The adjustment submodule is used to adjust the current pixel using other pixels to obtain the adjusted current pixel, wherein the third image feature includes the adjusted current pixel;

[0160] The normalization module includes:

[0161] The normalization submodule is used to perform a normalization transformation on the third image features using the objective function to obtain the first decision map.

[0162] Optionally, in the image fusion apparatus provided in this application embodiment, the determining unit 404 includes:

[0163] The correction module is used to correct the first decision graph to obtain the third decision graph associated with the first image, and to correct the second decision graph to obtain the fourth decision graph associated with the second image.

[0164] The smoothing module is used to perform region smoothing on the third decision graph to obtain the fifth decision graph associated with the first image, and to perform region smoothing on the fourth decision graph to obtain the sixth decision graph associated with the second image.

[0165] The first determining module is used to determine the target fused image of the first image and the second image based on the first fusion result of the first image and the fifth decision map, and the second fusion result of the second image and the sixth decision map.

[0166] Optionally, in the image fusion apparatus provided in this application embodiment, the correction module includes:

[0167] The correction submodule is used to correct the first decision graph using the target correction algorithm and determine the corrected decision graph as the third decision graph.

[0168] The smoothing module includes:

[0169] The smoothing submodule is used to divide the third decision graph into N windows using a target region smoothing algorithm, and to determine the fifth decision graph based on the N windows after division, where N is a positive integer.

[0170] Optionally, in the image fusion apparatus provided in the embodiments of this application, the apparatus further includes:

[0171] The extraction and processing module is used to, after acquiring at least two images to be fused, input the first image and the second image into a preset model for feature extraction processing respectively, to obtain the first image features associated with the first image and the second image features associated with the second image;

[0172] The recursive processing module is used to perform global recursive processing, correction processing, and region smoothing processing on the features of the first image and the features of the second image respectively, after acquiring at least two images to be fused, using a preset model, to obtain the fifth decision map associated with the first image and the sixth decision map associated with the second image.

[0173] The fusion processing module is used to determine and output the target fused image of the first image and the second image based on the first image, the second image, the fifth decision map and the sixth decision map after acquiring at least two images to be fused.

[0174] Optionally, in the image fusion apparatus provided in the embodiments of this application, the apparatus further includes:

[0175] The second determining module is used to determine the initialization model for obtaining the preset model before inputting the first image and the second image into the preset model for feature extraction processing respectively.

[0176] The training module is used to train the initialization model using a training sample set before inputting the first image and the second image into the preset model for feature extraction processing. The training sample set includes multiple training sample pairs, each of which includes a first training image, a second training image, and a ground truth image. The first training image and the second training image are used to input the first model to obtain a fused training image. The fused training image and the ground truth image are used to determine the first loss value at the image block level and the second loss value at the pixel level of the first model. The joint loss function includes the first loss value and the second loss value.

[0177] The third determining module is used to determine that the initialization model training is complete before inputting the first image and the second image into the preset model for feature extraction processing, provided that the joint loss value is less than the preset convergence threshold, and to determine the initialization model after training as the preset model.

[0178] The image fusion apparatus provided in this application, after acquiring the image to be fused and extracting features, performs global recursive processing on the extracted features, so that a decision map that can indicate global decision information of the image is obtained after recursive processing. Then, image fusion is performed based on the decision map of global information, so that the information processing in the image fusion process is sufficiently comprehensive, thereby achieving the technical effect of effectively improving the accuracy of image fusion and solving the technical problem of low accuracy of image fusion in related technologies.

[0179] The image fusion device includes a processor and a memory. The aforementioned acquisition unit, extraction unit, recursive unit, determination unit, etc., are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0180] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can ensure the uniformity of centroid selection, thereby improving clustering accuracy.

[0181] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0182] This invention provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the image fusion method.

[0183] This invention provides a processor for running a program, wherein the program executes the image fusion method during runtime.

[0184] like Figure 5 As shown, an embodiment of the present invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps:

[0185] Acquire at least two images to be fused, wherein the at least two images indicate the same scene but have different focuses, and the at least two images include a first image and a second image;

[0186] The first image is processed by feature extraction to obtain the first image features associated with the first image, and the second image is processed by feature extraction to obtain the second image features associated with the second image;

[0187] Global recursive processing is performed on the features of the first image to obtain a first decision map associated with the image, and global recursive processing is performed on the second image to obtain a second decision map associated with the second image. The first decision map is used to indicate the global decision information of the first image, and the second decision map is used to indicate the global decision information of the second image.

[0188] Based on the first decision map and the second decision map, a target fusion image of the first image and the second image is obtained.

[0189] As an optional approach, global recursive processing is performed on the first image features and the second image features to obtain a first decision map associated with the first image and a second decision map associated with the second image, including:

[0190] The third image feature is obtained by recursively processing each pixel in the first image feature, and the fourth image feature is obtained by recursively processing each pixel in the second image feature.

[0191] The third image features are normalized to obtain the first decision map, and the third image features are normalized to obtain the second decision map.

[0192] As an optional approach, recursively processing each pixel in the first image feature yields the third image feature, which includes:

[0193] The current pixel is determined from the first image features, and other pixels in the target direction are determined, wherein the target direction includes: up direction, down direction, left direction, and right direction;

[0194] The current pixel is adjusted using other pixels to obtain the adjusted current pixel, wherein the third image feature includes the adjusted current pixel;

[0195] The third image features are normalized to obtain the first decision map, which includes:

[0196] The first decision map is obtained by normalizing the features of the third image using the objective function.

[0197] As an optional approach, based on the first decision map and the second decision map, the target fused image obtained from the first image and the second image includes:

[0198] The first decision graph is corrected to obtain the third decision graph associated with the first image, and the second decision graph is corrected to obtain the fourth decision graph associated with the second image.

[0199] The third decision graph is smoothed to obtain the fifth decision graph associated with the first image, and the fourth decision graph is smoothed to obtain the sixth decision graph associated with the second image.

[0200] Based on the first fusion result of the first image and the fifth decision map, and the second fusion result of the second image and the sixth decision map, the target fused image of the first image and the second image is determined.

[0201] As an optional approach, the third decision map associated with the first image is obtained by correcting the first decision map, including:

[0202] The first decision graph is corrected using a target correction algorithm, and the corrected decision graph is then used as the third decision graph.

[0203] The fifth decision map, obtained by performing region smoothing on the third decision map and associating it with the first image, includes:

[0204] Using a target region smoothing algorithm, the third decision graph is divided into N windows, and the fifth decision graph is determined based on the N windows after division, where N is a positive integer.

[0205] As an alternative approach, after obtaining at least two images to be fused, the method further includes:

[0206] The first image and the second image are input into a preset model for feature extraction processing to obtain the first image features associated with the first image and the second image features associated with the second image.

[0207] Using a pre-defined model, global recursive processing, correction processing, and region smoothing processing are performed on the first image features and the second image features respectively to obtain the fifth decision map associated with the first image and the sixth decision map associated with the second image.

[0208] Based on the first image, the second image, the fifth decision map, and the sixth decision map, the target fused image of the first image and the second image is determined and output.

[0209] As an optional approach, before inputting the first image and the second image into a preset model for feature extraction processing, the method further includes:

[0210] Determine the initialization model used to obtain the preset model;

[0211] The initial model is trained using a training sample set, which includes multiple training sample pairs. Each training sample pair includes a first training image, a second training image, and a ground truth image. The first and second training images are used as input to the first model to obtain a fused training image. The fused training image and the ground truth image are used to determine the first loss value at the image block level and the second loss value at the pixel level for the first model. The joint loss function includes the first loss value and the second loss value.

[0212] If the joint loss value is less than the preset convergence threshold, the initialization model is determined to be completed, and the initialization model after training is determined as the preset model.

[0213] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0214] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that initializes the following method steps:

[0215] Acquire at least two images to be fused, wherein the at least two images indicate the same scene but have different focuses, and the at least two images include a first image and a second image;

[0216] The first image is processed by feature extraction to obtain the first image features associated with the first image, and the second image is processed by feature extraction to obtain the second image features associated with the second image;

[0217] Global recursive processing is performed on the features of the first image to obtain a first decision map associated with the image, and global recursive processing is performed on the second image to obtain a second decision map associated with the second image. The first decision map is used to indicate the global decision information of the first image, and the second decision map is used to indicate the global decision information of the second image.

[0218] Based on the first decision map and the second decision map, a target fusion image of the first image and the second image is obtained.

[0219] As an optional approach, global recursive processing is performed on the first image features and the second image features to obtain a first decision map associated with the first image and a second decision map associated with the second image, including:

[0220] The third image feature is obtained by recursively processing each pixel in the first image feature, and the fourth image feature is obtained by recursively processing each pixel in the second image feature.

[0221] The third image features are normalized to obtain the first decision map, and the third image features are normalized to obtain the second decision map.

[0222] As an optional approach, recursively processing each pixel in the first image feature yields the third image feature, which includes:

[0223] The current pixel is determined from the first image features, and other pixels in the target direction are determined, wherein the target direction includes: up direction, down direction, left direction, and right direction;

[0224] The current pixel is adjusted using other pixels to obtain the adjusted current pixel, wherein the third image feature includes the adjusted current pixel;

[0225] The third image features are normalized to obtain the first decision map, which includes:

[0226] The first decision map is obtained by normalizing the features of the third image using the objective function.

[0227] As an optional approach, based on the first decision map and the second decision map, the target fused image obtained from the first image and the second image includes:

[0228] The first decision graph is corrected to obtain the third decision graph associated with the first image, and the second decision graph is corrected to obtain the fourth decision graph associated with the second image.

[0229] The third decision graph is smoothed to obtain the fifth decision graph associated with the first image, and the fourth decision graph is smoothed to obtain the sixth decision graph associated with the second image.

[0230] Based on the first fusion result of the first image and the fifth decision map, and the second fusion result of the second image and the sixth decision map, the target fused image of the first image and the second image is determined.

[0231] As an optional approach, the third decision map associated with the first image is obtained by correcting the first decision map, including:

[0232] The first decision graph is corrected using a target correction algorithm, and the corrected decision graph is then used as the third decision graph.

[0233] The fifth decision map, obtained by performing region smoothing on the third decision map and associating it with the first image, includes:

[0234] Using a target region smoothing algorithm, the third decision graph is divided into N windows, and the fifth decision graph is determined based on the N windows after division, where N is a positive integer.

[0235] As an alternative approach, after obtaining at least two images to be fused, the method further includes:

[0236] The first image and the second image are input into a preset model for feature extraction processing to obtain the first image features associated with the first image and the second image features associated with the second image.

[0237] Using a pre-defined model, global recursive processing, correction processing, and region smoothing processing are performed on the first image features and the second image features respectively to obtain the fifth decision map associated with the first image and the sixth decision map associated with the second image.

[0238] Based on the first image, the second image, the fifth decision map, and the sixth decision map, the target fused image of the first image and the second image is determined and output.

[0239] As an optional approach, before inputting the first image and the second image into a preset model for feature extraction processing, the method further includes:

[0240] Determine the initialization model used to obtain the preset model;

[0241] The initial model is trained using a training sample set, which includes multiple training sample pairs. Each training sample pair includes a first training image, a second training image, and a ground truth image. The first and second training images are used as input to the first model to obtain a fused training image. The fused training image and the ground truth image are used to determine the first loss value at the image block level and the second loss value at the pixel level for the first model. The joint loss function includes the first loss value and the second loss value.

[0242] If the joint loss value is less than the preset convergence threshold, the initialization model is determined to be completed, and the initialization model after training is determined as the preset model.

[0243] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0244] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0245] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0246] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0247] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0248] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0249] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0250] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0251] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0252] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An image fusion method, characterized in that, include: The initialization model is trained using a training sample set, wherein the training sample set includes multiple training sample pairs, each training sample pair including a first training image, a second training image and a ground truth image, wherein the first training image and the second training image are used as input to the initialization model to obtain a fused training image. Based on the structural similarity loss function, a first loss is calculated on the fused training image and the ground truth image to determine the first loss value of the initialization model at the image block level; Based on the mean squared error loss function, a second loss is calculated on the fused training image and the ground truth image to determine the second loss value of the initialized model at the pixel level; If the joint loss value is less than a preset convergence threshold, the initialization model is determined to be trained and the trained initialization model is determined as the preset model, wherein the joint loss value is the sum of the first loss value and the second loss value; Acquire at least two images to be fused, wherein the at least two images indicate the same scene but have different focuses, and the at least two images include a first image and a second image; The first image and the second image are input into the preset model. Feature extraction processing is performed on the first image to obtain the first image feature associated with the first image, and feature extraction processing is performed on the second image to obtain the second image feature associated with the second image. The first image feature is recursively processed to obtain the third image feature, and the second image feature is recursively processed to obtain the fourth image feature. The third image features are normalized to obtain a first decision map associated with the first image, and the fourth image features are normalized to obtain a second decision map associated with the second image. The first decision map is used to indicate the global decision information of the first image, and the second decision map is used to indicate the global decision information of the second image. The first decision graph is corrected to obtain the third decision graph associated with the first image, and the second decision graph is corrected to obtain the fourth decision graph associated with the second image. The third decision map is smoothed to obtain the fifth decision map associated with the first image, and the fourth decision map is smoothed to obtain the sixth decision map associated with the second image. The fifth decision map and the first image are fused to obtain a first fused image, and the sixth decision map and the second image are fused to obtain a second fused image. The first fused image and the second fused image are fused together to obtain a target fused image of the first image and the second image.

2. The method according to claim 1, characterized in that, The recursive processing of each pixel in the first image feature to obtain the third image feature includes: The current pixel is determined from the first image features, and other pixels of the current pixel in the target direction are determined, wherein the target direction includes: up direction, down direction, left direction, and right direction; The current pixel is adjusted using the other pixels to obtain the adjusted current pixel, wherein the third image feature includes the adjusted current pixel; The normalization process for the third image features to obtain the first decision map includes: The third image features are normalized using an objective function to obtain the first decision map.

3. The method according to claim 1, characterized in that, The step of correcting the first decision map to obtain the third decision map associated with the first image includes: The first decision graph is corrected using a target correction algorithm, and the corrected decision graph is determined as the third decision graph. The process of performing region smoothing on the third decision map to obtain the fifth decision map associated with the first image includes: Using a target region smoothing algorithm, the third decision graph is divided into N windows, and the fifth decision graph is determined based on the N windows, where N is a positive integer.

4. The method according to claim 1, characterized in that, After acquiring at least two images to be fused, the method further includes: The first image and the second image are input into the preset model for feature extraction processing to obtain the first image features associated with the first image and the second image features associated with the second image. Using the preset model, global recursive processing, correction processing, and region smoothing processing are performed on the first image features and the second image features respectively to obtain the fifth decision map associated with the first image and the sixth decision map associated with the second image; Based on the first image, the second image, the fifth decision map, and the sixth decision map, the target fused image of the first image and the second image is determined and output.

5. An image fusion apparatus, characterized in that, include: The apparatus is used to train an initialization model using a training sample set, wherein the training sample set includes multiple training sample pairs, each training sample pair including a first training image, a second training image, and a ground truth image, wherein the first training image and the second training image are used as input to the initialization model to obtain a fused training image; based on a structural similarity loss function, a first loss calculation is performed on the fused training image and the ground truth image to determine a first loss value of the initialization model at the image block level; based on a mean squared error loss function, a second loss calculation is performed on the fused training image and the ground truth image to determine a second loss value of the initialization model at the pixel level; if the joint loss value is less than a preset convergence threshold, the initialization model is determined to be trained successfully, and the trained initialization model is determined as the preset model, wherein the joint loss value is the sum of the first loss value and the second loss value; An acquisition unit is used to acquire at least two images to be fused, wherein the at least two images indicate the same scene but have different focuses, and the at least two images include a first image and a second image; The extraction unit is used to input the first image and the second image into the preset model, perform feature extraction processing on the first image to obtain the first image feature associated with the first image, and perform feature extraction processing on the second image to obtain the second image feature associated with the second image; The apparatus is further configured to: recursively process each pixel in the first image feature to obtain a third image feature; recursively process each pixel in the second image feature to obtain a fourth image feature; normalize the third image feature to obtain a first decision map associated with the first image; normalize the fourth image feature to obtain a second decision map associated with the second image, wherein the first decision map indicates global decision information of the first image, and the second decision map indicates global decision information of the second image; correct the first decision map to obtain a third decision map associated with the first image; correct the second decision map to obtain a fourth decision map associated with the second image; perform region smoothing on the third decision map to obtain a fifth decision map associated with the first image; perform region smoothing on the fourth decision map to obtain a sixth decision map associated with the second image; perform image fusion on the fifth decision map and the first image to obtain a first fused image; perform image fusion on the sixth decision map and the second image to obtain a second fused image; and perform image fusion on the first fused image and the second fused image to obtain a target fused image of the first image and the second image.

6. A processor, characterized in that, The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 4 when it runs.

7. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of any one of claims 1 to 4.

Images