Image fusion processing method, device, equipment, storage medium and chip

Abstract

The present disclosure relates to an image fusion processing method, device, equipment, storage medium and chip. The method comprises: acquiring a first image and a second image; performing affine transformation on the second image to the coordinate system of the first image to obtain a third image; determining a target fusion mask image based on the first image and the third image, the target fusion mask image comprising a first indication area and a second indication area, the boundary of the first indication area and the second indication area being a target fusion boundary, the target fusion boundary being arranged at a position where the image difference meets a set condition, the image difference comprising the difference between the first image and the third image; and fusing the third image and the first image based on the target fusion mask image to obtain a target fusion image. The present disclosure can avoid the problem of ghosting at the fusion boundary, thereby improving the quality of the output image of the electronic device.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image fusion processing method, apparatus, device, storage medium, and chip. Background Technology

[0002] With the development of terminal technology, the functions of smartphones and other terminal devices are becoming increasingly rich, and image processing is one of the important functions of these devices. Due to the spatial limitations of terminal devices, their camera modules cannot achieve the shooting quality of professional cameras. Therefore, related technologies typically improve image quality by fusing multiple frames captured by multiple cameras in the camera module to obtain a fused image. However, these image fusion methods suffer from image ghosting issues, which affect the quality of the output image. Summary of the Invention

[0003] To overcome the problems existing in the related technologies, the present disclosure provides an image fusion processing method, apparatus, device, storage medium, and chip to solve the defects in the related technologies.

[0004] According to a first aspect of the present disclosure, an image fusion processing method is provided, the method comprising:

[0005] Acquire a first image and a second image, wherein the first image is captured by a first camera of an electronic device and the second image is captured by a second camera of the electronic device, and the field of view of the first camera is greater than the field of view of the second camera;

[0006] The second image is affinely transformed to the coordinate system of the first image to obtain the third image;

[0007] A target fusion mask image is determined based on the first image and the third image. The target fusion mask image includes a first indicator area and a second indicator area. The first indicator area is used to indicate the fusion area, and the second indicator area is used to indicate the non-fusion area. The boundary between the first indicator area and the second indicator area is the target fusion boundary. The target fusion boundary is set at a position where the image difference meets a set condition. The image difference includes the difference between the first image and the third image.

[0008] The third image and the first image are fused based on the target fusion mask image to obtain the target fusion image.

[0009] In some embodiments, the step of affinely transforming the second image to the coordinate system of the first image to obtain the third image includes:

[0010] Image registration is performed based on the first image and the second image to obtain the image affine transformation relationship;

[0011] Based on the affine transformation relationship of the images, the second image is affinely transformed to the coordinate system of the first image to obtain the third image.

[0012] In some embodiments, the method further includes:

[0013] The first image and the second image are preprocessed based on a preset processing method to obtain the preprocessed first image and the second image. The preset processing method includes at least one of brightness correction and color correction.

[0014] Based on the preprocessed first image and second image, the operation of image registration based on the first image and second image to obtain the image affine transformation relationship is performed.

[0015] In some embodiments, determining the target fusion mask image based on the first image and the third image includes:

[0016] Determine the pixel differences and optical flow information between the first image and the third image;

[0017] A first fusion mask image is determined based on the pixel differences and the optical flow information. The first fusion mask image includes a third indicator region and a fourth indicator region. The third indicator region is used to indicate the fusion region, and the fourth indicator region is used to indicate the non-fusion region. The boundary between the third indicator region and the fourth indicator region is the first fusion boundary.

[0018] The first fused mask image is downsampled to obtain a second fused mask image. The second fused mask image includes a fifth indicator region and a sixth indicator region. The fifth indicator region is used to indicate the fused region, and the sixth indicator region is used to indicate the non-fused region. The boundary between the fifth indicator region and the sixth indicator region is the second fused boundary.

[0019] The second fusion boundary in the second fusion mask image is adjusted to a position where the image difference meets the set conditions to obtain the adjusted second fusion mask image, wherein the image difference includes the difference between the first image and the third image;

[0020] The adjusted second fusion mask image is upsampled to obtain the target fusion mask image, and the target fusion boundary is obtained based on the upsampling of the adjusted second fusion boundary.

[0021] In some embodiments, determining the first fused mask image based on the pixel differences and the optical flow information includes:

[0022] Based on the pixel differences and the optical flow information, a first fused mask image is determined using a preset algorithm, wherein the preset algorithm includes at least one of a motion detection algorithm and an occlusion detection algorithm.

[0023] In some embodiments, adjusting the second fusion boundary in the second fusion mask image to a position where the image differences satisfy a set condition, to obtain the adjusted second fusion mask image, includes:

[0024] In the second fused mask image, the sixth indicated region is subjected to distance transformation to obtain a first distance-transformed image;

[0025] The adjustment range of the second fusion boundary in the second fusion mask image is determined based on the first distance transform image;

[0026] Determine the pixel differences between the first image and the third image;

[0027] Based on the pixel differences, the Gaussian weighted average value of the pixel differences within the set window corresponding to each pixel within the adjustment range is determined, and a Gaussian weighted average value image is obtained;

[0028] Based on the Gaussian weighted average image, an objective function is constructed using a graph cut algorithm. The objective function includes a constraint term and a smoothing term. The constraint term is used to limit the positions of the fifth and sixth indicator regions in the second fused mask image. The smoothing term is used to adjust the second fused boundary within the adjustment range based on the Gaussian weighted average image to a position where the image difference meets the set conditions. The image difference includes the difference between the first image and the third image.

[0029] The objective function is solved based on a preset function solving algorithm to obtain the adjusted second fused mask image.

[0030] In some embodiments, fusing the third image and the first image based on the target fusion mask image to obtain the target fusion image includes:

[0031] Within the fusion transition region of the target fused image, the third image and the first image are fused based on a predetermined fusion transition weight, and the fusion transition region corresponds to a preset fusion transition region in the target fused mask image;

[0032] Within the fusion region of the target fused image, the pixel values ​​of the third image are retained, and the fusion region corresponds to the region in the first indication region excluding the preset fusion transition region;

[0033] Within the non-fusion region of the target fused image, the pixel values ​​of the first image are retained, and the non-fusion region corresponds to the second indicated region.

[0034] In some embodiments, the method further includes pre-determining the fusion transition weights based on the following:

[0035] A distance transformation is performed on the second indicated region in the target fusion mask image to obtain a second distance-transformed image;

[0036] The fusion transition weights corresponding to the preset fusion transition region are determined based on the second distance transform image.

[0037] In some embodiments, the method further includes:

[0038] The third image is style-transferred to obtain the fourth image;

[0039] The operation of fusing the third image with the first image based on the target fusion mask image to obtain the target fusion image is performed after replacing the third image with the fourth image.

[0040] In some embodiments, performing style transfer on the third image to obtain a fourth image includes:

[0041] The third image is input into the generator network in a pre-trained generative adversarial network to obtain the style transfer residual image from the third image to the first image;

[0042] Based on a predetermined style transition weight, the style transfer residual image is superimposed onto the style transition region of the third image to obtain a fourth image, wherein the style transition region of the third image corresponds to the preset style transition region of the target fusion mask image.

[0043] In some embodiments, the method further includes pre-determining the style transition weights based on the following:

[0044] A distance transformation is performed on the second indicated region in the target fusion mask image to obtain a third distance-transformed image;

[0045] The style transition weights corresponding to the preset style transition regions of the target fusion mask image are determined based on the third distance transform image.

[0046] In some embodiments, the method further includes pre-training the generative adversarial network based on the following:

[0047] Multiple first sample images and multiple third sample images are acquired. The third sample images include images obtained by affine transformation of the second sample images to the coordinate system of the first sample images. The first sample images are captured by a first sample camera, and the second sample images are captured by a second sample camera. The field of view of the first sample camera is greater than that of the second sample camera.

[0048] The multiple first sample images and the multiple third sample images are used as the decision set to train the decision network in the generative adversarial network, resulting in a trained decision network. The decision network is used for classification based on the comparison results of image styles.

[0049] The sliding window is used to slide on the first sample image and the third sample image respectively.

[0050] In response to detecting that the structural similarity (SSIM) between two images within the sliding window is greater than or equal to a set threshold, the two images are used as a set of training images.

[0051] Repeat the process of acquiring a set of training images, and use the resulting multiple sets of training images as the generator training set;

[0052] The generator network in the generative adversarial network is trained based on the generator training set to obtain the trained generator network.

[0053] In some embodiments, training the generator network in the generative adversarial network based on the generator training set includes:

[0054] A generator loss function is constructed, which includes at least one of a constraint term, a feature similarity term, and a generation term. The constraint term is determined based on the difference in image content between the style-transformed third sample image and its own image content. The feature similarity term is determined based on the difference in image content between the style-transformed third sample image and the first sample image. The generation term is determined based on the output probability of the decision network.

[0055] The loss function is solved based on a preset optimization algorithm to obtain the trained generator network.

[0056] According to a second aspect of the present disclosure, an image fusion processing apparatus is provided, the apparatus comprising:

[0057] An image acquisition module is used to acquire a first image and a second image. The first image is captured by a first camera of an electronic device, and the second image is captured by a second camera of the electronic device. The field of view of the first camera is greater than the field of view of the second camera.

[0058] The image transformation module is used to affinely transform the second image to the coordinate system of the first image to obtain the third image;

[0059] A mask determination module is used to determine a target fusion mask image based on the first image and the third image. The target fusion mask image includes a first indicator area and a second indicator area. The first indicator area is used to indicate the fusion area, and the second indicator area is used to indicate the non-fusion area. The boundary between the first indicator area and the second indicator area is the target fusion boundary. The target fusion boundary is set at a position where the image difference meets a set condition. The image difference includes the difference between the first image and the third image.

[0060] An image fusion module is used to fuse the third image and the first image based on the target fusion mask image to obtain a target fusion image.

[0061] In some embodiments, the image transformation module includes:

[0062] The relationship acquisition unit is used to perform image registration based on the first image and the second image to obtain the image affine transformation relationship;

[0063] The image transformation unit is used to affinely transform the second image to the coordinate system of the first image based on the affine transformation relationship of the image, so as to obtain the third image.

[0064] In some embodiments, the image transformation module includes:

[0065] A preprocessing unit is configured to preprocess the first image and the second image based on a preset processing method to obtain the preprocessed first image and the second image, wherein the preset processing method includes at least one of brightness correction and color correction.

[0066] The relationship acquisition unit is also used to perform the operation of image registration based on the first image and the second image to obtain the image affine transformation relationship based on the preprocessed first image and the second image.

[0067] In some embodiments, the mask determination module includes:

[0068] A difference information determination unit is used to determine the pixel differences and optical flow information between the first image and the third image;

[0069] The first mask determination unit is used to determine a first fusion mask image based on the pixel difference and the optical flow information. The first fusion mask image includes a third indicator region and a fourth indicator region. The third indicator region is used to indicate the fusion region, and the fourth indicator region is used to indicate the non-fusion region. The boundary between the third indicator region and the fourth indicator region is the first fusion boundary.

[0070] The second mask acquisition unit is used to downsample the first fused mask image to obtain a second fused mask image. The second fused mask image includes a fifth indicator region and a sixth indicator region. The fifth indicator region is used to indicate the fused region, and the sixth indicator region is used to indicate the non-fused region. The boundary between the fifth indicator region and the sixth indicator region is the second fused boundary.

[0071] A fusion boundary adjustment unit is used to adjust the second fusion boundary in the second fusion mask image to a position where the image difference meets a set condition, thereby obtaining an adjusted second fusion mask image, wherein the image difference includes the difference between the first image and the third image;

[0072] The target mask acquisition unit is used to upsample the adjusted second fusion mask image to obtain the target fusion mask image, wherein the target fusion boundary is obtained based on the upsampling of the adjusted second fusion boundary.

[0073] In some embodiments, the first mask determination unit is further configured to determine a first fused mask image based on the pixel differences and the optical flow information using a preset algorithm, wherein the preset algorithm includes at least one of a motion detection algorithm and an occlusion detection algorithm.

[0074] In some embodiments, the fusion boundary adjustment unit is further configured to:

[0075] In the second fused mask image, the sixth indicated region is subjected to distance transformation to obtain a first distance-transformed image;

[0076] The adjustment range of the second fusion boundary in the second fusion mask image is determined based on the first distance transform image;

[0077] Determine the pixel differences between the first image and the third image;

[0078] Based on the pixel differences, the Gaussian weighted average value of the pixel differences within the set window corresponding to each pixel within the adjustment range is determined, and a Gaussian weighted average value image is obtained;

[0079] Based on the Gaussian weighted average image, an objective function is constructed using a graph cut algorithm. The objective function includes a constraint term and a smoothing term. The constraint term is used to limit the positions of the fifth and sixth indicator regions in the second fused mask image. The smoothing term is used to adjust the second fused boundary within the adjustment range based on the Gaussian weighted average image to a position where the image difference meets the set conditions. The image difference includes the difference between the first image and the third image.

[0080] The objective function is solved based on a preset function solving algorithm to obtain the adjusted second fused mask image.

[0081] In some embodiments, the image fusion module includes:

[0082] The first fusion unit is configured to fuse the third image and the first image within the fusion transition region of the target fusion image based on a predetermined fusion transition weight, wherein the fusion transition region corresponds to a preset fusion transition region in the target fusion mask image.

[0083] The second fusion unit is used to retain the pixel values ​​of the third image within the fusion region of the target fused image, wherein the fusion region corresponds to the region in the first indication region other than the preset fusion transition region;

[0084] The third fusion unit is used to retain the pixel values ​​of the first image within the non-fusion region of the target fused image, wherein the non-fusion region corresponds to the second indicated region.

[0085] In some embodiments, the image fusion module further includes a fusion transition weight determination unit;

[0086] The fusion transition weight determination unit is used for:

[0087] A distance transformation is performed on the second indicated region in the target fusion mask image to obtain a second distance-transformed image;

[0088] The fusion transition weights corresponding to the preset fusion transition region are determined based on the second distance transform image.

[0089] In some embodiments, the apparatus further includes:

[0090] A style transfer module is used to perform style transfer on the third image to obtain a fourth image;

[0091] The image fusion module is further configured to replace the third image with the fourth image and perform the operation of fusing the third image with the first image based on the target fusion mask image to obtain a target fusion image.

[0092] In some embodiments, the style transfer module includes:

[0093] The residual image acquisition unit is used to input the third image into the generator network in the pre-trained generative adversarial network to obtain the style transfer residual image from the third image to the first image;

[0094] The fourth image acquisition unit is used to superimpose the style transfer residual image onto the style transition region of the third image based on a predetermined style transition weight to obtain a fourth image, wherein the style transition region of the third image corresponds to the preset style transition region of the target fusion mask image.

[0095] In some embodiments, the style transfer module further includes a style transition weight determination unit;

[0096] The style transition weight determination unit is used for:

[0097] A distance transformation is performed on the second indicated region in the target fusion mask image to obtain a third distance-transformed image;

[0098] The style transition weights corresponding to the preset style transition regions of the target fusion mask image are determined based on the third distance transform image.

[0099] In some embodiments, the apparatus further includes a generative adversarial network training module;

[0100] The generative adversarial network training module includes:

[0101] The sample acquisition unit is used to acquire multiple first sample images and multiple third sample images. The third sample images include images obtained by affine transformation of the second sample images to the coordinate system of the first sample images. The first sample images are acquired by a first sample camera, and the second sample images are acquired by a second sample camera. The field of view of the first sample camera is greater than the field of view of the second sample camera.

[0102] The decision training unit is used to train the decision network in the generative adversarial network using the multiple first sample images and the multiple third sample images as the decision training set, so as to obtain the trained decision network. The decision network is used to classify based on the comparison results of image styles.

[0103] A window sliding unit is used to slide on the first sample image and the third sample image respectively based on a preset sliding window;

[0104] An image determination unit is configured to, in response to detecting that the structural similarity (SSIM) between two images within the sliding window is greater than or equal to a set threshold, use the two images as a set of training images.

[0105] The training set determination unit is used to repeatedly acquire a set of training images and use the resulting multiple sets of training images as the generator training set.

[0106] The generator training unit is used to train the generator network in the generative adversarial network based on the generator training set, so as to obtain the trained generator network.

[0107] In some embodiments, the generator training unit is further configured to:

[0108] A generator loss function is constructed, which includes at least one of a constraint term, a feature similarity term, and a generation term. The constraint term is determined based on the difference in image content between the style-transformed third sample image and its own image content. The feature similarity term is determined based on the difference in image content between the style-transformed third sample image and the first sample image. The generation term is determined based on the output probability of the decision network.

[0109] The loss function is solved based on a preset optimization algorithm to obtain the trained generator network.

[0110] According to a third aspect of the present disclosure, an electronic device is provided, the device comprising:

[0111] The first camera, the second camera, the processor, and the memory for storing computer programs;

[0112] The processor is configured to, when executing the computer program, implement:

[0113] Acquire a first image and a second image, wherein the first image is captured by the first camera and the second image is captured by the second camera, and the field of view of the first camera is greater than the field of view of the second camera;

[0114] The second image is affinely transformed to the coordinate system of the first image to obtain the third image;

[0115] A target fusion mask image is determined based on the first image and the third image. The target fusion mask image includes a first indicator area and a second indicator area. The first indicator area is used to indicate the fusion area, and the second indicator area is used to indicate the non-fusion area. The boundary between the first indicator area and the second indicator area is the target fusion boundary. The target fusion boundary is set at a position where the image difference meets a set condition. The image difference includes the difference between the first image and the third image.

[0116] The third image and the first image are fused based on the target fusion mask image to obtain the target fusion image.

[0117] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, the program being implemented when executed by a processor:

[0118] Acquire a first image and a second image, wherein the first image is captured by a first camera of an electronic device and the second image is captured by a second camera of the electronic device, and the field of view of the first camera is greater than the field of view of the second camera;

[0119] The second image is affinely transformed to the coordinate system of the first image to obtain the third image;

[0120] A target fusion mask image is determined based on the first image and the third image. The target fusion mask image includes a first indicator area and a second indicator area. The first indicator area is used to indicate the fusion area, and the second indicator area is used to indicate the non-fusion area. The boundary between the first indicator area and the second indicator area is the target fusion boundary. The target fusion boundary is set at a position where the image difference meets a set condition. The image difference includes the difference between the first image and the third image.

[0121] The third image and the first image are fused based on the target fusion mask image to obtain the target fusion image.

[0122] According to a fifth aspect of the present disclosure, a chip is provided, comprising:

[0123] Processors and interfaces;

[0124] The processor is used to read instructions through the interface to execute any of the image fusion processing methods described above.

[0125] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0126] This disclosure obtains a first image and a second image, and performs an affine transformation of the second image to the coordinate system of the first image to obtain a third image. Then, a target fusion mask image is determined based on the first image and the third image. Subsequently, the third image and the first image can be fused based on the target fusion mask image to obtain a target fused image. Since the target fusion boundary in the target fusion mask image is set at the position where the image difference meets the set conditions, the target fusion boundary can be restricted to the position with the minimum image difference by setting the conditions. This can avoid the problem of ghosting at the fusion boundary in the target fused image during image fusion. Furthermore, it can appropriately reduce the width of the fusion transition area while ensuring the fusion effect. Compared with the fusion transition method in related technologies, it can achieve a narrow fusion transition effect, which can further avoid the problem of ghosting at the fusion boundary, thereby improving the quality of the image output by the electronic device.

[0127] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0128] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0129] Figure 1A This is a flowchart illustrating an image fusion processing method according to an exemplary embodiment of the present disclosure;

[0130] Figure 1B This is a schematic diagram of a first image shown according to an exemplary embodiment of the present disclosure;

[0131] Figure 1C This is a schematic diagram of a third image shown according to an exemplary embodiment of the present disclosure;

[0132] Figure 1D This is a schematic diagram illustrating a target fusion mask image according to an exemplary embodiment of the present disclosure;

[0133] Figure 1E This is a schematic diagram of a target fused image according to an exemplary embodiment of the present disclosure;

[0134] Figure 2 This is a flowchart illustrating how to obtain a third image according to an exemplary embodiment of this disclosure;

[0135] Figure 3 This is a flowchart illustrating how to determine a target fusion mask image based on the first image and the third image, according to an exemplary embodiment of this disclosure;

[0136] Figure 4A This is a flowchart illustrating, according to an exemplary embodiment of the present disclosure, how to adjust the second fusion boundary in the second fusion mask image to a position where the image differences satisfy a set condition;

[0137] Figure 4B This is a schematic diagram of a second fused mask image according to an exemplary embodiment of the present disclosure;

[0138] Figure 4C This is a schematic diagram of a first distance-transformed image according to an exemplary embodiment of the present disclosure;

[0139] Figure 4D This is a schematic diagram illustrating the adjustment range of the second fusion boundary according to an exemplary embodiment of the present disclosure;

[0140] Figure 5 This is a flowchart illustrating how to determine the fusion transition weights according to yet another exemplary embodiment of this disclosure;

[0141] Figure 6A This is a flowchart illustrating how to perform style transfer on the third image according to an exemplary embodiment of this disclosure;

[0142] Figure 6B This is a schematic diagram of a third image shown according to an exemplary embodiment of the present disclosure;

[0143] Figure 7 This is a flowchart illustrating how to determine the style transition weights according to an exemplary embodiment of this disclosure;

[0144] Figure 8 This is a flowchart illustrating how to train the generative adversarial network according to an exemplary embodiment of this disclosure;

[0145] Figure 9 This is a flowchart illustrating, according to an exemplary embodiment of the present disclosure, how to train the generator network in the generative adversarial network based on the generator training set;

[0146] Figure 10 This is a block diagram illustrating an image fusion processing apparatus according to an exemplary embodiment of the present disclosure;

[0147] Figure 11 This is a block diagram of another image fusion processing apparatus according to an exemplary embodiment of the present disclosure;

[0148] Figure 12 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0149] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0150] Figure 1A This is a flowchart illustrating an image fusion processing method according to an exemplary embodiment; the method of this embodiment can be applied to an electronic device having a first camera (e.g., a wide-angle camera) and a second camera (e.g., a telephoto camera).

[0151] like Figure 1A As shown, the method includes the following steps S101-S102:

[0152] In step S101, the first image and the second image are acquired.

[0153] In this embodiment, the first image can be captured by the first camera of the electronic device, and the second image can be captured by the second camera of the electronic device.

[0154] For example, the first image mentioned above could be an image with lower resolution but a wider field of view taken with a wide-angle lens (such as...). Figure 1B As shown in the figure), the second image mentioned above can be an image with high resolution but a small field of view taken by a telephoto lens (not shown in the figure).

[0155] In step S102, the second image is affinely transformed to the coordinate system of the first image to obtain the third image.

[0156] In this embodiment, after acquiring the first image and the second image, the second image can be affinely transformed (warped) to the coordinate system of the first image to obtain the third image (e.g., ...). Figure 1C (As shown).

[0157] It is worth noting that the above-described method of performing affine transformation on the second image can be found in the explanations and descriptions in related technologies, and this embodiment does not limit it.

[0158] In other embodiments, the method of affinely transforming the second image to the coordinate system of the first image described above can also be found in the following: Figure 2 The embodiments shown will not be described in detail here.

[0159] In step S103, a target fusion mask image is determined based on the first image and the third image.

[0160] For example, Figure 1D This is a schematic diagram illustrating a target fusion mask image according to an exemplary embodiment of the present disclosure. Figure 1D As shown, the target fusion mask image includes a first indication region (i.e., Figure 1D The white area in the middle) and the second indicator area (i.e., Figure 1D The first indicator area is used to indicate the fused area, and the second indicator area is used to indicate the non-fused area. The boundary between the first indicator area and the second indicator area is the target fused boundary.

[0161] In this embodiment, the target fusion boundary is set at the position where the image difference meets the set condition (e.g., the image difference is the minimum), wherein the image difference includes the difference between the first image and the third image.

[0162] In some embodiments, after the second image is affinely transformed to the coordinate system of the first image to obtain the third image, an initial fusion mask image can be generated based on the mask image generation method in related technologies. Then, the fusion boundary in the initial fusion mask image is adjusted so that the fusion boundary is located at the position with the least image difference.

[0163] In other embodiments, the method of determining the target fusion mask image based on the first image and the third image described above can also be found in the following: Figure 3 The embodiments shown will not be described in detail here.

[0164] In step S104, the third image and the first image are fused based on the target fusion mask image to obtain the target fusion image.

[0165] In this embodiment, after determining the target fusion mask image based on the first image and the third image, the third image and the first image can be fused based on the target fusion mask image to obtain the target fusion image.

[0166] For example, Figure 1E This is a schematic diagram of a target fusion image according to an exemplary embodiment of the present disclosure. In this embodiment, when performing image fusion, in order to make the image fusion transition more natural, a fusion transition region (hereinafter referred to as the preset fusion transition region for easy distinction) can be set in the target fusion mask image in advance. For example, the region with a width (e.g., 5) set outside the target fusion boundary in the target fusion mask image can be used as the preset fusion transition region.

[0167] like Figure 1EAs shown, within the fusion transition region (i.e., Sb, which corresponds to the preset fusion transition region in the target fusion mask image) of the target fusion image, the third image and the first image can be fused based on a predetermined fusion transition weight.

[0168] In one embodiment, the method for determining the aforementioned fusion transition weights can be found below. Figure 5 The embodiments shown will not be described in detail here.

[0169] On the other hand, within the fusion region of the target fused image, the pixel values ​​of the aforementioned third image can be retained, wherein the fusion region corresponds to the region in the first indication region of the target fused mask image excluding the preset fusion transition region (i.e., the portion of the fusion region excluding the fusion transition region).

[0170] In the non-fusion region of the target fused image, the pixel values ​​of the first image can be retained, wherein the non-fusion region corresponds to the second indication region in the target fused mask image.

[0171] In some embodiments, in order to ensure that the overall image style of the target fused image is consistent, the third image can be style-transferred to obtain a fourth image before fusing the third image with the first image based on the target fused mask image. Then, the third image is replaced by the fourth image and fused with the first image to obtain the target fused image.

[0172] It is worth noting that the above-described method for style transfer of the third image can refer to image style transfer schemes in related technologies, and this embodiment does not limit it.

[0173] In other embodiments, the style transfer method for the third image described above can also be found in the following... Figure 6A The embodiments shown will not be described in detail here.

[0174] As described above, this embodiment obtains a first image and a second image, and performs an affine transformation of the second image to the coordinate system of the first image to obtain a third image. Then, a target fusion mask image is determined based on the first image and the third image. Subsequently, the third image and the first image can be fused based on the target fusion mask image to obtain a target fusion image. Since the target fusion boundary in the target fusion mask image is set at the position where the image difference meets the set conditions, the target fusion boundary can be restricted to the position where the image difference is minimal by setting the conditions. This can avoid the problem of ghosting at the fusion boundary in the target fusion image during image fusion. Furthermore, it can appropriately reduce the width of the fusion transition area while ensuring the fusion effect. Compared with the fusion transition method in related technologies, it can achieve a narrow fusion transition effect, which can further avoid the problem of ghosting at the fusion boundary, thereby improving the quality of the image output by the electronic device.

[0175] Figure 2 This is a flowchart illustrating how to obtain a third image according to an exemplary embodiment of this disclosure; this embodiment is an illustrative example of how to obtain a third image, based on the above embodiment. Figure 2 As shown, the step S102 above, which involves affinely transforming the second image to the coordinate system of the first image to obtain the third image, may include the following steps S201-S202:

[0176] In step S201, image registration is performed based on the first image and the second image to obtain the image affine transformation relationship.

[0177] In this embodiment, the registration of the first image and the second image can be completed using algorithms such as feature point matching and optical flow. For example, feature point matching can be performed using the ORB (Oriented Fast and Rotated Brief) method, and / or the optical flow information of the first image and the second image can be determined based on the PWC-Net (Optical Flow Learning Network) algorithm.

[0178] In some embodiments, to ensure that the color and / or brightness of the target fused image obtained by subsequent image fusion are consistent, the first image and the second image can be preprocessed according to a preset processing method before image registration is performed based on the first image and the second image to obtain the preprocessed first image and the preprocessed second image. Then, image registration is performed based on the preprocessed first image and the preprocessed second image to obtain the image affine transformation relationship.

[0179] The aforementioned preset processing method may include at least one of brightness correction and color correction.

[0180] In step S202, based on the affine transformation relationship of the image, the second image is affinely transformed to the coordinate system of the first image to obtain the third image.

[0181] In this embodiment, after image registration is performed based on the first image and the second image to obtain the image affine transformation relationship, the second image can be affinely transformed to the coordinate system of the first image based on the image affine transformation relationship to obtain the third image.

[0182] For example, after obtaining the above affine transformation relationship of the image, each pixel in the second image can be affinely transformed to the coordinate system of the first image based on the transformation relationship of each pixel recorded in the affine transformation relationship, thereby obtaining the third image.

[0183] As described above, this embodiment obtains an affine transformation relationship by performing image registration based on the first image and the second image, and then affinely transforms the second image to the coordinate system of the first image based on the affine transformation relationship to obtain the third image. This can accurately transform the second image to the coordinate system of the first image, thereby enabling subsequent fusion of the obtained third image with the first image. This avoids the problem of ghosting at the fusion boundary, thus improving the quality of the image output by the electronic device.

[0184] Figure 3 This is a flowchart illustrating how to determine a target fusion mask image based on the first image and the third image according to an exemplary embodiment of the present disclosure; this embodiment is an exemplary description based on the above embodiment, taking how to determine the target fusion mask image based on the first image and the third image as an example.

[0185] like Figure 3 As shown, the determination of the target fusion mask image based on the first image and the third image in step S103 above may include the following steps S301-S302:

[0186] In step S301, the pixel differences and optical flow information between the first image and the third image are determined.

[0187] In this embodiment, when it is necessary to determine the target fusion mask image based on the first image and the third image, the pixel difference and optical flow information between the first image and the third image can be determined first.

[0188] The pixel difference between the first image and the third image can be calculated based on the pixel difference calculation method in related technologies, and this embodiment does not limit this.

[0189] In some embodiments, the optical flow information of the first and second images can be determined based on the PWC-Net (Optical Flow Learning Network) algorithm.

[0190] In step S302, a first fusion mask image is determined based on the pixel differences and the optical flow information.

[0191] In this embodiment, after determining the pixel difference and optical flow information between the first image and the third image, a first fusion mask image can be determined based on the pixel difference and the optical flow information. The first fusion mask image includes a third indicator region and a fourth indicator region. The third indicator region is used to indicate the fusion region, and the fourth indicator region is used to indicate the non-fusion region. The boundary between the third indicator region and the fourth indicator region is the first fusion boundary.

[0192] In some embodiments, a first fused mask image can be determined using a preset algorithm based on the aforementioned pixel differences and optical flow information. For example, the preset algorithm may include at least one of motion detection algorithms and occlusion detection algorithms from related technologies.

[0193] In step S303, the first fused mask image is downsampled to obtain the second fused mask image.

[0194] In this embodiment, after determining the first fusion mask image based on the pixel differences and the optical flow information, in order to improve the speed of subsequent fusion boundary adjustment, the first fusion mask image is downsampled to obtain the second fusion mask image.

[0195] It is worth noting that the frequency of downsampling can be flexibly set based on business experience or the needs of the actual scenario, such as setting it to 4:1 (i.e., retaining 1 point out of every 4 points), etc. This embodiment does not limit this.

[0196] The second fusion mask image includes a fifth indicator region and a sixth indicator region. The fifth indicator region is used to indicate the fusion region, and the sixth indicator region is used to indicate the non-fusion region. The boundary between the fifth indicator region and the sixth indicator region is the second fusion boundary (i.e., the first fusion boundary after downsampling).

[0197] In step S304, the second fusion boundary in the second fusion mask image is adjusted to a position where the image difference meets the set conditions, thereby obtaining the adjusted second fusion mask image.

[0198] In this embodiment, after downsampling the first fused mask image to obtain the second fused mask image, the second fusion boundary in the second fused mask image can be adjusted to a position where the image difference meets the set conditions, thus obtaining the adjusted second fused mask image. The aforementioned image difference includes the difference between the first image and the third image.

[0199] For example, the locations where the image differences meet the set conditions include the locations where the image differences are the smallest.

[0200] In some embodiments, the second fusion boundary in the second fusion mask image can be adjusted to a position where the image differences meet a set condition, based on the differences between the first image and the third image.

[0201] In other embodiments, the method of adjusting the second fusion boundary in the second fusion mask image to a position where the image differences meet the set conditions can also be referred to below. Figure 4A The embodiments shown will not be described in detail here.

[0202] In step S305, the adjusted second fusion mask image is upsampled to obtain the target fusion mask image.

[0203] In this embodiment, after adjusting the second fusion boundary in the second fusion mask image to a position where the image difference meets the set conditions, and obtaining the adjusted second fusion mask image, the adjusted second fusion mask image can be upsampled to obtain the target fusion mask image. Specifically, the adjusted second fusion mask image is resized to the size of the first fusion mask image to obtain the target fusion mask image. The target fusion boundary in the target fusion mask image is the aforementioned "adjusted second fusion boundary" after upsampling.

[0204] It is understood that if the downsampling frequency is 4:1 as described above, the upsampling frequency in this step can be 1:4 (i.e., inserting 3 new points). This embodiment does not limit this.

[0205] As described above, this embodiment determines the pixel differences and optical flow information between the first image and the third image, determines a first fusion mask image based on the pixel differences and optical flow information, then downsamples the first fusion mask image to obtain a second fusion mask image, and adjusts the second fusion boundary in the second fusion mask image to a position where the image differences meet the set conditions to obtain an adjusted second fusion mask image. Then, upsampling is performed on the adjusted second fusion mask image to obtain the target fusion mask image. This allows for accurate determination of the target fusion mask image, laying the foundation for subsequent fusion of the third image and the first image based on the target fusion mask image. Furthermore, since the first fusion mask image is downsampled first, and then the fusion boundary is adjusted on the obtained second fusion mask image, small-scale fusion boundary adjustment can be achieved. Compared to directly adjusting the fusion boundary at the original scale, this increases the speed of boundary adjustment and thus improves the efficiency of subsequent output images.

[0206] Figure 4A This is a flowchart illustrating, according to an exemplary embodiment of the present disclosure, how to adjust the second fusion boundary in the second fusion mask image to a position where the image differences satisfy a set condition; Figure 4B This is a schematic diagram of a second fused mask image according to an exemplary embodiment of the present disclosure; Figure 4C This is a schematic diagram of a first distance-transformed image according to an exemplary embodiment of the present disclosure; Figure 4D This is a schematic diagram illustrating the adjustment range of the second fusion boundary according to an exemplary embodiment of the present disclosure.

[0207] This embodiment, based on the above embodiment, takes as an example how to adjust the second fusion boundary in the second fusion mask image to a position where the image difference meets the set conditions.

[0208] like Figure 3 As shown, the step S304 above, which involves adjusting the second fusion boundary in the second fusion mask image to a position where the image differences meet the set conditions, to obtain the adjusted second fusion mask image, may include the following steps S401-S406:

[0209] In step S401, a distance transformation is performed on the sixth indication region in the second fused mask image to obtain a first distance transformed image.

[0210] In this embodiment, when it is necessary to adjust the second fusion mask image (such as...) Figure 4B The second fusion boundary (i.e., in the image shown) Figure 4BWhen the boundary between the black and white areas is shown, a distance transformation can first be performed on the sixth indicated area in the second fusion mask image to obtain a first distance transformation image (such as...). Figure 4C (The image shown).

[0211] like Figure 4C As shown, the first distance-transformed image can be used to characterize the distance between each pixel in the second fused mask image and the second fused boundary. That is, different colored regions in the image can be used to indicate different distances from the second fused boundary.

[0212] It is worth noting that the above-described method of distance transformation can be found in the explanations and descriptions in related technologies, and this embodiment does not limit it.

[0213] In step S402, the adjustment range of the second fusion boundary in the second fusion mask image is determined based on the first distance transformation image.

[0214] In this embodiment, after performing a distance transformation on the sixth indication region in the second fusion mask image to obtain a first distance transformation image, the adjustment range of the second fusion boundary in the second fusion mask image can be determined based on the first distance transformation image (e.g., Figure 4D The adjustment range S shown.

[0215] like Figure 4D As shown, the range [a, b] can be defined as the adjustment range S of the second fusion boundary. For example, a ≤ 0 (i.e., a can be negative), b = 20. That is to say, the second fusion boundary is expanded outward by a width of 20, which is used as the adjustment range S of the second fusion boundary.

[0216] In step S403, the pixel differences between the first image and the third image are determined.

[0217] In this embodiment, after determining the adjustment range of the second fusion boundary in the second fusion mask image based on the first distance transformation image, the pixel difference between the first image and the third image can be determined.

[0218] For example, suppose the first image is I1(p) i The third image is I3(p) i ), p i Let i be the i-th point in the corresponding image. Then the pixel difference between the first image and the third image can be:

[0219] |I1(p i )-I3(p i )|;(4-1)

[0220] In step S404, the Gaussian weighted average value of the pixel differences within the set window corresponding to each pixel in the adjustment range is determined based on the pixel differences, and a Gaussian weighted average value image is obtained.

[0221] In this embodiment, after determining the pixel difference between the first image and the third image, the Gaussian weighted average value of the pixel difference within the set window corresponding to each pixel in the adjustment range can be determined based on the pixel difference to obtain a Gaussian weighted average value image, which can be used to describe the pixel difference between the first image and the third image.

[0222] For example, the Gaussian weighted average image D(p) can be determined based on the following equation (4-2):

[0223] D(p)=∑ i∈N w i |I1(p i )-I3(p i (4-2)

[0224] In the above formula, N is p i A point corresponds to the set of pixels within the window at the original image's location, w i To set the Gaussian weights within the window, I1(p) i ) and I3(p i () represent the first image and the third image, respectively. For example, the window size can be set to 7x7.

[0225] In step S405, an objective function is constructed based on the Gaussian weighted average image using a graph cut algorithm.

[0226] In this embodiment, the objective function includes a constraint term and a smoothing term. The constraint term is used to limit the positions of the fifth and sixth indication regions in the second fused mask image. The smoothing term is used to adjust the second fused boundary within the adjustment range based on the Gaussian weighted average image to a position where the image difference meets the set conditions. The image difference includes the difference between the first image and the third image.

[0227] For example, the objective function E can be constructed as shown in equation (4-3):

[0228] E = ∑ p∈S E d (p)+∑ p,q∈N E s (p,q); (4-3)

[0229] In the above formula, S is the set of points within the adjustment range, N is the set of pixels within the setting window corresponding to the original image position of point P, and q is the point within the setting window located around point P; E d (p) is a constraint term used to limit the positions of the fifth indicator region (i.e., the region indicating the fusion region) and the sixth indicator region (i.e., the region indicating the non-fusion region) in the second fusion mask image; E s (p,q) is a smoothing term used to adjust the second fusion boundary within the adjustment range based on the Gaussian weighted average image to a position where the image difference meets a set condition (e.g., the image difference is minimal).

[0230] In some embodiments, the aforementioned constraint E d (p) can take the form shown in equation (4-5):

[0231]

[0232] In the above formula, This is the adjusted second fusion mask.

[0233] From equation (4-5) above, we can see that:

[0234] (1) When S(p) ≠ {a,b}, that is, when point p in S is not within the adjustment range {a,b}, let E d (p) = 0, indicating that the cost of the expression is small;

[0235] As can be seen from (1), this embodiment does not focus on points located outside the adjustment range when adjusting the second fusion boundary.

[0236] (2) When That is, when point p in S is on boundary line a, and this point is located in the sixth indicator region (the region indicating the non-merged region), let constraint term E... d (p) = 0, indicating that the cost of the expression is small;

[0237] (3) When That is, when point p in S is on boundary line a, and this point is located in the fifth indicator region (the region indicating the fusion region), let constraint term E... d (p) = ∞, indicating that the cost of this expression is very high;

[0238] As can be seen from (2) and (3), in this embodiment, when adjusting the second fusion boundary, the boundary line a is defined as belonging to the sixth indication area.

[0239] (4) When That is, when point p in S is on boundary line b and is located in the fifth indication area, let constraint term E d (p) = 0, indicating that the cost of the expression is small;

[0240] (5) When That is, when point p in S is on boundary line b and is located in the sixth indication area, let constraint term E d (p) = ∞, indicating that the cost of this expression is very high;

[0241] As can be seen from (4) and (5), in this embodiment, when adjusting the second fusion boundary, the boundary line b is defined as belonging to the fifth indication area.

[0242] In other words, in this embodiment, when adjusting the second fusion boundary in the second fusion mask image, the adjustment range is first divided into {a,b} (i.e., the area between boundary line a and boundary line b). The area inside boundary line a (including boundary line a) is set as the sixth indicator area (i.e., the area indicating the non-fusion area), and the area outside boundary line b (including boundary line b) is set as the fifth indicator area (i.e., the area indicating the fusion area). In subsequent steps, the second fusion boundary is adjusted within the adjustment range {a,b}.

[0243] In some embodiments, the above smoothing term E s The form of (p,q) can be shown in equation (4-6):

[0244]

[0245] Where D(p) and D(q) are the Gaussian weighted average images of points p and q, respectively. and These are the fused mask images for points p and q, respectively.

[0246] From equation (4-6) above, it can be seen that when At that time, Then E s (p,q) = max(D(p),D(q)), indicating that the cost of this expression is very high; while when At that time, Then E s (p,q)=0 indicates that the cost of the expression is relatively small.

[0247] In other words, in this embodiment, when adjusting the second fusion boundary in the second fusion mask image, the second fusion boundary can be forced towards... Furthermore, the direction in which "max(D(p),D(q))" takes a smaller value is adjusted, because it can adjust the second fusion boundary in the second fusion mask image to the position with the smallest image difference.

[0248] In step S406, the objective function is solved based on a preset function solving algorithm to obtain the adjusted second fusion mask image.

[0249] In this embodiment, after constructing the objective function based on the Gaussian weighted average image using the graph cut algorithm, the objective function can be solved based on the preset function solving algorithm to obtain the adjusted second fused mask image.

[0250] It is worth noting that the above-mentioned preset function solving algorithm can be set according to actual business needs, such as setting it to the Boykov-Kolmogorov algorithm, etc. This embodiment does not limit it in this way.

[0251] As described above, this embodiment obtains a first distance-transformed image by performing a distance transformation on the sixth indication region in the second fused mask image. Based on the first distance-transformed image, the adjustment range of the second fusion boundary in the second fused mask image is determined, and the pixel difference between the first image and the third image is determined. Based on the pixel difference, the Gaussian weighted average value of the pixel difference within a set window corresponding to each pixel in the adjustment range is determined, resulting in a Gaussian weighted average value image. Then, based on the Gaussian weighted average value image, a target function is constructed using a graph cut algorithm, and the target function is solved using a preset function solving algorithm to obtain the adjusted second fused mask image. By constructing and solving the target function, the second fusion boundary in the second fused mask image can be adjusted to a position where the image difference meets the set conditions. This lays the foundation for subsequent upsampling processing of the adjusted second fused mask image to obtain the target fused mask image.

[0252] Figure 5 This is a flowchart illustrating how to determine the fusion transition weight according to another exemplary embodiment of the present disclosure; this embodiment is an exemplary description based on the above embodiment, taking the determination of the fusion transition weight as an example.

[0253] like Figure 5 As shown, the image fusion processing method of this embodiment, based on the above embodiment, may further include pre-determining the fusion transition weights based on the following steps S501-S502:

[0254] In step S501, the second indicated region is subjected to distance transformation in the target fusion mask image to obtain a second distance transformed image.

[0255] In this embodiment, after obtaining the target fusion mask image, a distance transformation can be performed on the second indicated region in the target fusion mask image to obtain a second distance transformed image.

[0256] The aforementioned second distance transformation image can be used to characterize the distance between each pixel in the target fusion mask image and the target fusion boundary.

[0257] It is worth noting that the above-described method of distance transformation can be found in the explanations and descriptions in related technologies, and this embodiment does not limit it.

[0258] In step S502, the fusion transition weights corresponding to the preset fusion transition region are determined based on the second distance transform image.

[0259] In this embodiment, after performing distance transformation on the second indicated region in the target fusion mask image to obtain a second distance transformed image, the fusion transition weight corresponding to the preset fusion transition region can be determined based on the second distance transformed image.

[0260] In some embodiments, after obtaining the second distance transformation image, the fusion transition weight corresponding to the point closer to the boundary of the preset fusion transition region can be set to a larger value based on the second distance transformation image, while the fusion transition weight corresponding to the point farther from the boundary of the preset fusion transition region can be set to a smaller value, wherein the above-mentioned points are points in the preset fusion transition region.

[0261] For example, the fusion transition weight W can be determined based on the following equation (5-1). b :

[0262]

[0263] Among them, s b The boundary distance of the preset fusion transition region is defined as [0, s]. b Therefore, d b The points mentioned above can be represented as points within a preset fusion transition region.

[0264] As described above, this embodiment obtains a second distance-transformed image by performing a distance transformation on the second indicated region in the target fusion mask image, and determines the fusion transition weight corresponding to the preset fusion transition region based on the second distance-transformed image. This allows for accurate determination of the fusion transition weight, and subsequently, the third image and the first image can be fused based on the fusion transition weight within the fusion transition region of the target fusion image. This better avoids the problem of ghosting at the fusion boundary, thereby improving the quality of the image output by the electronic device.

[0265] Figure 6A This is a flowchart illustrating how to perform style transfer on the third image according to an exemplary embodiment of the present disclosure; this embodiment is based on the above embodiment and takes how to perform style transfer on the third image as an example for illustrative explanation.

[0266] like Figure 6A ​As shown, the image fusion processing method of this embodiment, based on the above embodiment, may further include performing style transfer on the third image in advance based on the following steps S601-S602:

[0267] In step S601, the third image is input into the generator network in the pre-trained generative adversarial network to obtain the style transfer residual image from the third image to the first image.

[0268] In this embodiment, when style transfer is required on the third image, the third image can be input into the generator network in a pre-trained generative adversarial network to generate a style transfer residual image from the third image to the first image through the generator network.

[0269] For example, a third sample image can be obtained by using a first sample image captured by a first sample camera and a second sample image captured by a second sample camera, respectively. The second sample image can then be affinely transformed to the coordinate system of the first sample image to obtain a third sample image. The aforementioned Generative Adversarial Network (GAN) can then be trained based on the first and second sample images. The generator network in this trained GAN can be used to generate a style transfer residual image from the third image to the first image based on the input third image.

[0270] The first sample camera can be a wide-angle camera of the same model as the first camera, and the second sample camera can be a telephoto camera of the same model as the second camera.

[0271] In step S602, based on a predetermined style transition weight, the style transfer residual image is superimposed onto the style transition region of the third image to obtain a fourth image.

[0272] For example, Figure 6B This is a schematic diagram of a third image shown according to an exemplary embodiment of the present disclosure;

[0273] In this embodiment, when performing image fusion, in order to make the image style fusion more natural, a style transition region (hereinafter referred to as the preset style transition region) can be set in the target fusion mask image in advance. For example, the region with a set width (e.g., 20) outside the target fusion boundary in the target fusion mask image can be used as the preset style transition region.

[0274] like Figure 6BAs shown, after obtaining the style transfer residual image, based on the predetermined style transition weight, the style transfer residual image is superimposed on the style transition region of the third image (i.e., Sa, which corresponds to the preset fusion transition region in the target fusion mask image), thereby obtaining the fourth image.

[0275] In one embodiment, the method for determining the aforementioned style transition weights can be found below. Figure 7 The embodiments shown will not be described in detail here.

[0276] As described above, this embodiment inputs the third image into the generator network of a pre-trained generative adversarial network to obtain a style transfer residual image from the third image to the first image. Based on a predetermined style transition weight, the style transfer residual image is superimposed onto the style transition region of the third image to obtain a fourth image. This allows for style transfer of the third image, which in turn enables subsequent image fusion with the first image based on the fourth image, ensuring a consistent overall image style and avoiding stitching marks at the fusion boundary. This further improves the quality of the output image from the electronic device.

[0277] Figure 7 This is a flowchart illustrating how to determine the style transition weight according to an exemplary embodiment of the present disclosure; this embodiment is an illustrative example of how to determine the style transition weight based on the above embodiment.

[0278] like Figure 7 As shown, the image fusion processing method of this embodiment, based on the above embodiment, may further include pre-determining style transition weights based on the following steps S701-S702:

[0279] In step S701, a distance transformation is performed on the second indicated region in the target fusion mask image to obtain a third distance transformed image;

[0280] In this embodiment, after obtaining the target fusion mask image, a distance transformation can be performed on the second indicated region in the target fusion mask image to obtain a third distance transformed image.

[0281] The aforementioned third distance transformation image can be used to characterize the distance between each pixel in the target fusion mask image and the target fusion boundary.

[0282] It is worth noting that the above-described method of distance transformation can be found in the explanations and descriptions in related technologies, and this embodiment does not limit it.

[0283] In step S702, the style transition weights corresponding to the preset style transition regions of the target fusion mask image are determined based on the third distance transform image.

[0284] In this embodiment, after performing distance transformation on the second indicated region in the target fusion mask image to obtain a third distance transformed image, the style transition weight corresponding to the preset style transition region of the target fusion mask image can be determined based on the third distance transformed image.

[0285] In some embodiments, after obtaining the third distance transformation image, the fusion transition weight corresponding to the point closer to the boundary of the preset style transition region can be set to a larger value based on the third distance transformation image, while the fusion transition weight corresponding to the point farther from the boundary of the preset style transition region can be set to a smaller value, wherein the above-mentioned points are points in the preset fusion transition region.

[0286] For example, the style transition weight W can be determined based on the following equation (7-1). a :

[0287]

[0288] Among them, s a The boundary distance of the preset style transition region, that is, the range of the preset style transition region is [0, s]. a Therefore, d a The points mentioned above can be represented as points within the preset style transition area.

[0289] As described above, this embodiment obtains a third distance-transformed image by performing a distance transformation on the second indicated region in the target fusion mask image, and determines the style transition weight corresponding to the preset style transition region of the target fusion mask image based on the third distance-transformed image. This allows for accurate determination of the style transition weight, and subsequently, based on the style transition weight, the style transfer residual image can be superimposed onto the style transition region of the third image to obtain a fourth image. This ensures style consistency in the fused images, thereby avoiding stitching marks at the fusion boundary and further improving the quality of the output image of the electronic device.

[0290] Figure 8 This is a flowchart illustrating how to train the generative adversarial network according to an exemplary embodiment of the present disclosure; this embodiment is based on the above embodiment and takes the training of the generative adversarial network as an example for illustrative explanation.

[0291] like Figure 8 ​As shown, the image fusion processing method of this embodiment may further include training the generative adversarial network based on the following steps S801-S806:

[0292] In step S801, multiple first sample images and multiple third sample images are acquired.

[0293] In this embodiment, the third sample image may include an image obtained by affine transformation of the second sample image to the coordinate system of the first sample image. The first sample image may be captured by a first sample camera, and the second sample image may be captured by a second sample camera. The first sample camera may be a wide-angle camera of the same model as the first camera, and the second sample camera may be a telephoto camera of the same model as the second camera. Therefore, the field of view of the first sample camera is larger than that of the second sample camera.

[0294] In step S802, the multiple first sample images and the multiple third sample images are used as the decision set to train the decision network in the generative adversarial network, thereby obtaining the trained decision network.

[0295] The aforementioned decision network can be used to classify images based on the comparison results of image styles (i.e., the result of comparing the style of the third image with that of the style transfer residual image to obtain the fourth image). For example, when the similarity is higher than or equal to a set similarity threshold, the output is 1; when the similarity is lower than the set similarity threshold, the output is 0.

[0296] For example, a large number of first sample images can be labeled as 1, and a large number of third sample images can be labeled as 0. These sample images can be used as the training set for the decision network to perform classification training on a pre-built decision network, resulting in the trained decision network D. The type of decision network can be set according to actual needs, such as setting it to a VGG16 model, which can have 13 convolutional layers and 3 fully connected layers.

[0297] In step S803, a preset sliding window is used to slide over the first sample image and the third sample image respectively.

[0298] In this embodiment, a preset sliding window can be used to slide on the first sample image and the third sample image, and the structural similarity (SSIM) between the two images within the sliding window can be detected.

[0299] It is worth noting that the size of the sliding window can be set according to actual needs, such as 512x512, etc. This embodiment does not limit this.

[0300] In step S804, in response to detecting that the structural similarity SSIM between two images within the sliding window is greater than or equal to a set threshold, the two images are used as a set of training images.

[0301] In this embodiment, when the structural similarity SSIM between two images within the sliding window is detected to be greater than or equal to a set threshold T, the two images can be used as a set of training images.

[0302] It is worth noting that the value of the threshold T can be set according to actual needs, such as 0.95, etc. This embodiment does not limit this.

[0303] In step S805, the process of acquiring a set of training images is repeated, and the resulting multiple sets of training images are used as the generator training set.

[0304] In this embodiment, by repeating the process described in step S804 above in multiple pairs of images, multiple sets of training images can be obtained, which can then constitute a generator training set.

[0305] In step S806, the generator network in the generative adversarial network is trained based on the generator training set to obtain the trained generator network.

[0306] In this embodiment, after obtaining the generator training set, the generator network in the generative adversarial network is trained based on the generator training set to obtain the trained generator network G.

[0307] Once training is complete, the style transfer residual image from the third image to the first image can be generated based on the generator network described above.

[0308] As described above, this embodiment can train an adversarial network based on sample images, and then input the third image into the generator network of the trained generative adversarial network to obtain the style transfer residual image from the third image to the first image, thereby realizing style transfer on the third image. This can ensure the overall image style of the target fused image is consistent, thereby avoiding the problem of stitching marks at the fusion boundary and further improving the quality of the output image of the electronic device.

[0309] Figure 9 This is a flowchart illustrating how to train the generator network in the generative adversarial network based on the generator training set, according to an exemplary embodiment of the present disclosure. This embodiment is based on the above embodiment and provides an exemplary description of how to train the generator network in the generative adversarial network based on the generator training set.

[0310] like Figure 9As shown, training the generator network in the generative adversarial network based on the generator training set in step S806 above may include the following steps S901-S902:

[0311] In step S901, the generator loss function is constructed.

[0312] The loss function includes at least one of a constraint term, a feature similarity term, and a generation term. The constraint term is determined based on the difference in image content between the style-transformed third sample image and its own image content. The feature similarity term is determined based on the difference in image content between the style-transformed third sample image and the first sample image. The generation term is determined based on the output probability of the decision network.

[0313] For example, the generator loss function Loss can be constructed based on the following equation (9-1):

[0314] Loss=α·l mse +β·l vgg / n +γ·l gen (9-1)

[0315] In the above formula, l mse As a constraint term, it represents the MSE (Mean Squared Error) of the third image after style transformation and the third image itself, so that the image content of the third image remains stable after style transformation;

[0316] l vgg / n The feature similarity term represents the MSE of the nth layer VGG network feature map of the style-transformed third image and the first image; for example, n=7.

[0317] l gen The generated term represents the logarithm of the output probability of the decision network, which can force an increase in the probability that the style-transformed third image is judged as the first image (i.e., the probability that the similarity between the style-transformed third image and the first image is higher than a set similarity threshold).

[0318] α, β, and γ are the weight adjustment factors for each item.

[0319] For example, constraint item l mse The form can be shown in the following equation (9-2):

[0320]

[0321] In this formula, W and H are the width and height of the third image, respectively. G(I) represents the point with coordinates (x, y) in the third image. (3) ) x,yThis represents the point with coordinates (x, y) in the style transfer residual image from the third image to the first image.

[0322] For example, feature similarity item l vgg / n The form can be shown in the following equation (9-3):

[0323]

[0324] In this formula, W n and H n These are the width and height of the feature map of the nth layer of the VGG network, respectively. Let (x, y) be the point with coordinates (x, y) in the nth layer VGG network feature map of the first image.

[0325] For example, generating item l gen The form can be shown in the following equation (9-4):

[0326] l gen =-log{D(G(I (3) )+I (3) (9-4)

[0327] In this formula, D(G(I) (3) )+I (3) ) represents the output probability of the decision network D.

[0328] In step S902, the loss function is solved based on a preset optimization algorithm to obtain the trained generator network.

[0329] In this embodiment, after the generator loss function is constructed, the loss function can be solved based on a preset optimization algorithm to obtain the trained generator network. That is, based on the preset optimization algorithm, the generator network parameters that minimize the loss function are determined and used as the parameters of the trained generator network.

[0330] It is worth noting that the above-mentioned preset optimization algorithm can be set according to actual business needs, such as setting it to the Boykov-Kolmogorov algorithm, etc. This embodiment does not limit this.

[0331] As described above, this embodiment constructs a generator loss function and solves the loss function based on a preset optimization algorithm to obtain a trained generator network. This allows for accurate training of the generator network, enabling the subsequent input of the third image into the generator network within a pre-trained generative adversarial network to obtain a style transfer residual image from the third image to the first image. This achieves style transfer of the third image, ensuring the overall image style consistency of the target fused image and avoiding stitching marks at the fusion boundary. This further improves the quality of the output image from the electronic device.

[0332] Figure 10 This is a block diagram illustrating an image fusion processing apparatus according to an exemplary embodiment of the present disclosure; the apparatus of this embodiment can be applied to an electronic device having a first camera (e.g., a wide-angle camera, etc.) and a second camera (e.g., a telephoto camera, etc.).

[0333] like Figure 10 As shown, the device includes: an image acquisition module 110, an image transformation module 120, a mask determination module 130, and an image fusion module 140, wherein:

[0334] Image acquisition module 110 is used to acquire a first image and a second image. The first image is captured by a first camera of the electronic device, and the second image is captured by a second camera of the electronic device. The field of view of the first camera is greater than the field of view of the second camera.

[0335] The image transformation module 120 is used to affinely transform the second image to the coordinate system of the first image to obtain the third image;

[0336] The mask determination module 130 is used to determine a target fusion mask image based on the first image and the third image. The target fusion mask image includes a first indicator area and a second indicator area. The first indicator area is used to indicate the fusion area, and the second indicator area is used to indicate the non-fusion area. The boundary between the first indicator area and the second indicator area is the target fusion boundary. The target fusion boundary is set at a position where the image difference meets a set condition. The image difference includes the difference between the first image and the third image.

[0337] The image fusion module 140 is used to fuse the third image and the first image based on the target fusion mask image to obtain a target fusion image.

[0338] As described above, the device in this embodiment acquires a first image and a second image, and performs an affine transformation of the second image to the coordinate system of the first image to obtain a third image. Then, a target fusion mask image is determined based on the first image and the third image. Subsequently, the third image and the first image can be fused based on the target fusion mask image to obtain a target fusion image. Since the target fusion boundary in the target fusion mask image is set at the position where the image difference meets the set conditions, the target fusion boundary can be restricted to the position where the image difference is minimal by setting the conditions. This avoids the problem of ghosting at the fusion boundary in the target fusion image during image fusion. Furthermore, it can appropriately reduce the width of the fusion transition area while ensuring the fusion effect. Compared with the fusion transition method in related technologies, it can achieve a narrow fusion transition effect, further avoiding the problem of ghosting at the fusion boundary, thereby improving the quality of the image output by the electronic device.

[0339] Figure 11 This is a block diagram illustrating another image fusion processing apparatus according to an exemplary embodiment of the present disclosure; the apparatus of this embodiment can be applied to an electronic device having a first camera (e.g., a wide-angle camera, etc.) and a second camera (e.g., a telephoto camera, etc.).

[0340] Among them, the image acquisition module 210, image transformation module 220, mask determination module 230 and image fusion module 240 are the same as those mentioned above. Figure 10 The image acquisition module 110, image transformation module 120, mask determination module 130 and image fusion module 140 in the illustrated embodiment have the same functions, which will not be described in detail here.

[0341] like Figure 11 As shown, the image transformation module 220 may include:

[0342] The relation acquisition unit 221 is used to perform image registration based on the first image and the second image to obtain the image affine transformation relation;

[0343] Image transformation unit 222 is used to affinely transform the second image to the coordinate system of the first image based on the affine transformation relationship of the image to obtain the third image.

[0344] In some embodiments, the image transformation module 220 may further include:

[0345] Preprocessing unit 223 is used to preprocess the first image and the second image based on a preset processing method to obtain the preprocessed first image and the second image. The preset processing method includes at least one of brightness correction and color correction.

[0346] Furthermore, the relationship acquisition unit 221 can also be used to perform the operation of image registration based on the first image and the second image to obtain the image affine transformation relationship based on the preprocessed first image and the second image.

[0347] In some embodiments, the mask determination module 230 may include:

[0348] The difference information determination unit 231 is used to determine the pixel difference and optical flow information between the first image and the third image;

[0349] The first mask determination unit 232 is used to determine a first fusion mask image based on the pixel difference and the optical flow information. The first fusion mask image includes a third indicator region and a fourth indicator region. The third indicator region is used to indicate the fusion region, and the fourth indicator region is used to indicate the non-fusion region. The boundary between the third indicator region and the fourth indicator region is the first fusion boundary.

[0350] The second mask acquisition unit 233 is used to downsample the first fused mask image to obtain a second fused mask image. The second fused mask image includes a fifth indicator region and a sixth indicator region. The fifth indicator region is used to indicate the fused region, and the sixth indicator region is used to indicate the non-fused region. The boundary between the fifth indicator region and the sixth indicator region is the second fused boundary.

[0351] The fusion boundary adjustment unit 234 is used to adjust the second fusion boundary in the second fusion mask image to a position where the image difference meets the set conditions, so as to obtain the adjusted second fusion mask image, wherein the image difference includes the difference between the first image and the third image;

[0352] The target mask acquisition unit 235 is used to perform upsampling processing on the adjusted second fusion mask image to obtain the target fusion mask image, wherein the target fusion boundary is obtained based on upsampling of the adjusted second fusion boundary.

[0353] In some embodiments, the first mask determination unit 232 may also be used to determine a first fused mask image based on the pixel differences and the optical flow information using a preset algorithm, wherein the preset algorithm includes at least one of a motion detection algorithm and an occlusion detection algorithm.

[0354] In some embodiments, the fusion boundary adjustment unit 234 can also be used for:

[0355] In the second fused mask image, the sixth indicated region is subjected to distance transformation to obtain a first distance-transformed image;

[0356] The adjustment range of the second fusion boundary in the second fusion mask image is determined based on the first distance transform image;

[0357] Determine the pixel differences between the first image and the third image;

[0358] Based on the pixel differences, the Gaussian weighted average value of the pixel differences within the set window corresponding to each pixel within the adjustment range is determined, and a Gaussian weighted average value image is obtained;

[0359] Based on the Gaussian weighted average image, an objective function is constructed using a graph cut algorithm. The objective function includes a constraint term and a smoothing term. The constraint term is used to limit the positions of the fifth and sixth indicator regions in the second fused mask image. The smoothing term is used to adjust the second fused boundary within the adjustment range based on the Gaussian weighted average image to a position where the image difference meets the set conditions. The image difference includes the difference between the first image and the third image.

[0360] The objective function is solved based on a preset function solving algorithm to obtain the adjusted second fused mask image.

[0361] In some embodiments, the image fusion module 240 may include:

[0362] The first fusion unit 241 is used to fuse the third image and the first image in the fusion transition region of the target fusion image based on a predetermined fusion transition weight, wherein the fusion transition region corresponds to a preset fusion transition region in the target fusion mask image.

[0363] The second fusion unit 242 is used to retain the pixel values ​​of the third image within the fusion area of ​​the target fused image, wherein the fusion area corresponds to the area in the first indication area excluding the preset fusion transition area;

[0364] The third fusion unit 243 is used to retain the pixel values ​​of the first image in the non-fusion area of ​​the target fused image, wherein the non-fusion area corresponds to the second indicated area.

[0365] In some embodiments, the image fusion module 240 may further include a fusion transition weight determination unit 244;

[0366] The fusion transition weight determination unit 244 can be used for:

[0367] A distance transformation is performed on the second indicated region in the target fusion mask image to obtain a second distance-transformed image;

[0368] The fusion transition weights corresponding to the preset fusion transition region are determined based on the second distance transform image.

[0369] In some embodiments, the above-described apparatus may further include:

[0370] Style transfer module 250 is used to perform style transfer on the third image to obtain a fourth image;

[0371] Furthermore, the image fusion module 240 can also be used to replace the third image with the fourth image and perform the operation of fusing the third image with the first image based on the target fusion mask image to obtain a target fusion image.

[0372] In some embodiments, the style transfer module 250 described above may include:

[0373] The residual image acquisition unit 251 is used to input the third image into the generator network in the pre-trained generative adversarial network to obtain the style transfer residual image from the third image to the first image;

[0374] The fourth image acquisition unit 252 is used to superimpose the style transfer residual image onto the style transition region of the third image based on a predetermined style transition weight to obtain a fourth image, wherein the style transition region of the third image corresponds to the preset style transition region of the target fusion mask image.

[0375] In some embodiments, the style transfer module 250 may further include a style transition weight determination unit 253;

[0376] Style transition weight determination unit 253 can be used for:

[0377] A distance transformation is performed on the second indicated region in the target fusion mask image to obtain a third distance-transformed image;

[0378] The style transition weights corresponding to the preset style transition regions of the target fusion mask image are determined based on the third distance transform image.

[0379] In some embodiments, the above-described apparatus generates an adversarial network training module 260;

[0380] Generative Adversarial Network Training Module 260 may include:

[0381] The sample acquisition unit 261 is used to acquire multiple first sample images and multiple third sample images. The third sample images include images obtained by affine transformation of the second sample images to the coordinate system of the first sample images. The first sample images are acquired by a first sample camera, and the second sample images are acquired by a second sample camera. The field of view of the first sample camera is greater than the field of view of the second sample camera.

[0382] The decision training unit 262 is used to train the decision network in the generative adversarial network by using the multiple first sample images and the multiple third sample images as the decision training set, so as to obtain the trained decision network. The decision network is used to classify based on the comparison results of image styles.

[0383] The window sliding unit 263 is used to slide on the first sample image and the third sample image respectively based on a preset sliding window;

[0384] Image determination unit 264 is configured to, in response to detecting that the structural similarity SSIM between two images within the sliding window is greater than or equal to a set threshold, use the two images as a set of training images.

[0385] The training set determination unit 265 is used to repeatedly acquire a set of training images and use the resulting multiple sets of training images as the generator training set.

[0386] The generator training unit 266 is used to train the generator network in the generative adversarial network based on the generator training set, so as to obtain the trained generator network.

[0387] In some embodiments, the generator training unit 266 is further configured to:

[0388] A generator loss function is constructed, which includes at least one of a constraint term, a feature similarity term, and a generation term. The constraint term is determined based on the difference in image content between the style-transformed third sample image and its own image content. The feature similarity term is determined based on the difference in image content between the style-transformed third sample image and the first sample image. The generation term is determined based on the output probability of the decision network.

[0389] The loss function is solved based on a preset optimization algorithm to obtain the trained generator network.

[0390] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0391] Figure 12This is a block diagram illustrating an electronic device according to an exemplary embodiment. For example, device 900 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0392] Reference Figure 12 The device 900 may include one or more of the following components: a processing component 902, a memory 904, a power supply component 906, a multimedia component 908, an audio component 910, an input / output (I / O) interface 912, a sensor component 914, and a communication component 916.

[0393] Processing component 902 typically controls the overall operation of device 900, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 902 may include one or more processors 920 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

[0394] Memory 904 is configured to store various types of data to support the operation of device 900. Examples of this data include instructions for any application or method operating on device 900, contact data, phonebook data, messages, pictures, videos, etc. Memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0395] Power supply component 906 provides power to various components of device 900. Power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 900.

[0396] Multimedia component 908 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0397] Audio component 910 is configured to output and / or input audio signals. For example, audio component 910 includes a microphone (MIC) configured to receive external audio signals when device 900 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 904 or transmitted via communication component 916. In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

[0398] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0399] Sensor assembly 914 includes one or more sensors for providing status assessments of various aspects of device 900. For example, sensor assembly 914 may detect the on / off state of device 900, the relative positioning of components such as the display and keypad of device 900, changes in the position of device 900 or a component of device 900, the presence or absence of user contact with device 900, the orientation or acceleration / deceleration of device 900, and temperature changes of device 900. Sensor assembly 914 may also include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 914 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 914 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0400] Communication component 916 is configured to facilitate wired or wireless communication between device 900 and other devices. Device 900 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G or 5G, or combinations thereof. In one exemplary embodiment, communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 916 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0401] In an exemplary embodiment, device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0402] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, which can be executed by a processor 920 of device 900 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0403] In an exemplary embodiment, a chip is also provided, including a processor and an interface;

[0404] The processor is used to read instructions through the interface to execute the above method.

[0405] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0406] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image fusion processing method, characterized in that, The method includes: Acquire a first image and a second image, wherein the first image is captured by a first camera of an electronic device and the second image is captured by a second camera of the electronic device, and the field of view of the first camera is greater than the field of view of the second camera; The second image is affinely transformed to the coordinate system of the first image to obtain the third image; A target fusion mask image is determined based on the first image and the third image. The target fusion mask image includes a first indicator area and a second indicator area. The first indicator area is used to indicate the fusion area, and the second indicator area is used to indicate the non-fusion area. The boundary between the first indicator area and the second indicator area is the target fusion boundary. The target fusion boundary is set at a position where the image difference meets a set condition. The image difference includes the difference between the first image and the third image. The third image and the first image are fused based on the target fusion mask image to obtain the target fusion image; The step of determining the target fusion mask image based on the first image and the third image includes: Determine the pixel differences and optical flow information between the first image and the third image; A first fusion mask image is determined based on the pixel differences and the optical flow information; The first fused mask image is downsampled to obtain the second fused mask image; The second fusion boundary in the second fusion mask image is adjusted to a position where the image differences meet the set conditions, thus obtaining the adjusted second fusion mask image; The adjusted second fused mask image is upsampled to obtain the target fused mask image.

2. The method according to claim 1, characterized in that, The step of affinely transforming the second image to the coordinate system of the first image to obtain the third image includes: Image registration is performed based on the first image and the second image to obtain the image affine transformation relationship; Based on the affine transformation relationship of the images, the second image is affinely transformed to the coordinate system of the first image to obtain the third image.

3. The method according to claim 2, characterized in that, The method further includes: The first image and the second image are preprocessed based on a preset processing method to obtain the preprocessed first image and the second image. The preset processing method includes at least one of brightness correction and color correction. Based on the preprocessed first image and second image, the operation of image registration based on the first image and second image to obtain the image affine transformation relationship is performed.

4. The method according to claim 1, characterized in that, The first fused mask image includes a third indicator region and a fourth indicator region. The third indicator region is used to indicate the fused region, and the fourth indicator region is used to indicate the non-fused region. The boundary between the third indicator region and the fourth indicator region is the first fused boundary. The second fused mask image includes a fifth indicator region and a sixth indicator region. The fifth indicator region is used to indicate the fused region, and the sixth indicator region is used to indicate the non-fused region. The boundary between the fifth indicator region and the sixth indicator region is the second fused boundary. The image differences include the differences between the first image and the third image; The target fusion boundary is obtained by upsampling the adjusted second fusion boundary.

5. The method according to claim 1, characterized in that, Determining the first fused mask image based on the pixel differences and the optical flow information includes: Based on the pixel differences and the optical flow information, a first fused mask image is determined using a preset algorithm, wherein the preset algorithm includes at least one of a motion detection algorithm and an occlusion detection algorithm.

6. The method according to claim 4, characterized in that, The step of adjusting the second fusion boundary in the second fusion mask image to a position where the image difference meets a set condition, to obtain the adjusted second fusion mask image, includes: In the second fused mask image, the sixth indicated region is subjected to distance transformation to obtain a first distance-transformed image; The adjustment range of the second fusion boundary in the second fusion mask image is determined based on the first distance transform image; Determine the pixel differences between the first image and the third image; Based on the pixel differences, the Gaussian weighted average value of the pixel differences within the set window corresponding to each pixel within the adjustment range is determined, and a Gaussian weighted average value image is obtained; Based on the Gaussian weighted average image, an objective function is constructed using a graph cut algorithm. The objective function includes a constraint term and a smoothing term. The constraint term is used to limit the positions of the fifth and sixth indicator regions in the second fused mask image. The smoothing term is used to adjust the second fused boundary within the adjustment range based on the Gaussian weighted average image to a position where the image difference meets the set conditions. The image difference includes the difference between the first image and the third image. The objective function is solved based on a preset function solving algorithm to obtain the adjusted second fused mask image.

7. The method according to claim 1, characterized in that, The step of fusing the third image and the first image based on the target fusion mask image to obtain the target fusion image includes: Within the fusion transition region of the target fused image, the third image and the first image are fused based on a predetermined fusion transition weight, and the fusion transition region corresponds to a preset fusion transition region in the target fused mask image; Within the fusion region of the target fused image, the pixel values ​​of the third image are retained, and the fusion region corresponds to the region in the first indication region excluding the preset fusion transition region; Within the non-fusion region of the target fused image, the pixel values ​​of the first image are retained, and the non-fusion region corresponds to the second indicated region.

8. The method according to claim 7, characterized in that, The method further includes pre-determining the fusion transition weights based on the following: A distance transformation is performed on the second indicated region in the target fusion mask image to obtain a second distance-transformed image; The fusion transition weights corresponding to the preset fusion transition region are determined based on the second distance transform image.

9. The method according to claim 1 or 7, characterized in that, The method further includes: The third image is style-transferred to obtain the fourth image; The operation of fusing the third image with the first image based on the target fusion mask image to obtain the target fusion image is performed after replacing the third image with the fourth image.

10. The method according to claim 9, characterized in that, The process of performing style transfer on the third image to obtain the fourth image includes: The third image is input into the generator network in a pre-trained generative adversarial network to obtain the style transfer residual image from the third image to the first image; Based on a predetermined style transition weight, the style transfer residual image is superimposed onto the style transition region of the third image to obtain a fourth image, wherein the style transition region of the third image corresponds to the preset style transition region of the target fusion mask image.

11. The method according to claim 10, characterized in that, The method also includes pre-determining the style transition weights based on the following: A distance transformation is performed on the second indicated region in the target fusion mask image to obtain a third distance-transformed image; The style transition weights corresponding to the preset style transition regions of the target fusion mask image are determined based on the third distance transform image.

12. The method according to claim 10, characterized in that, The method further includes pre-training the generative adversarial network based on the following: Multiple first sample images and multiple third sample images are acquired. The third sample images include images obtained by affine transformation of the second sample images to the coordinate system of the first sample images. The first sample images are captured by a first sample camera, and the second sample images are captured by a second sample camera. The field of view of the first sample camera is greater than that of the second sample camera. The multiple first sample images and the multiple third sample images are used as the decision set to train the decision network in the generative adversarial network, resulting in a trained decision network. The decision network is used for classification based on the comparison results of image styles. The sliding window is used to slide on the first sample image and the third sample image respectively. In response to detecting that the structural similarity (SSIM) between two images within the sliding window is greater than or equal to a set threshold, the two images are used as a set of training images. Repeat the process of acquiring a set of training images, and use the resulting multiple sets of training images as the generator training set; The generator network in the generative adversarial network is trained based on the generator training set to obtain the trained generator network.

13. The method according to claim 12, characterized in that, Training the generator network in the generative adversarial network based on the generator training set includes: A generator loss function is constructed, which includes at least one of a constraint term, a feature similarity term, and a generation term. The constraint term is determined based on the difference in image content between the style-transformed third sample image and its own image content. The feature similarity term is determined based on the difference in image content between the style-transformed third sample image and the first sample image. The generation term is determined based on the output probability of the decision network. The loss function is solved based on a preset optimization algorithm to obtain the trained generator network.

14. An image fusion processing apparatus, characterized in that, The device includes: An image acquisition module is used to acquire a first image and a second image. The first image is captured by a first camera of an electronic device, and the second image is captured by a second camera of the electronic device. The field of view of the first camera is greater than the field of view of the second camera. The image transformation module is used to affinely transform the second image to the coordinate system of the first image to obtain the third image; A mask determination module is used to determine a target fusion mask image based on the first image and the third image. The target fusion mask image includes a first indicator area and a second indicator area. The first indicator area is used to indicate the fusion area, and the second indicator area is used to indicate the non-fusion area. The boundary between the first indicator area and the second indicator area is the target fusion boundary. The target fusion boundary is set at a position where the image difference meets a set condition. The image difference includes the difference between the first image and the third image. The image fusion module is used to fuse the third image and the first image based on the target fusion mask image to obtain the target fusion image; The mask determination module includes: A difference information determination unit is used to determine the pixel differences and optical flow information between the first image and the third image; The first mask determination unit is used to determine a first fused mask image based on the pixel differences and the optical flow information; The second mask acquisition unit is used to downsample the first fused mask image to obtain the second fused mask image; The fusion boundary adjustment unit is used to adjust the second fusion boundary in the second fusion mask image to a position where the image difference meets the set conditions, so as to obtain the adjusted second fusion mask image; The target mask acquisition unit is used to upsample the adjusted second fused mask image to obtain the target fused mask image.

15. An electronic device, characterized in that, The device includes: The first camera, the second camera, the processor, and the memory for storing computer programs; The processor is configured to, when executing the computer program, implement: Acquire a first image and a second image, wherein the first image is captured by the first camera and the second image is captured by the second camera, and the field of view of the first camera is greater than the field of view of the second camera; The second image is affinely transformed to the coordinate system of the first image to obtain the third image; A target fusion mask image is determined based on the first image and the third image. The target fusion mask image includes a first indicator area and a second indicator area. The first indicator area is used to indicate the fusion area, and the second indicator area is used to indicate the non-fusion area. The boundary between the first indicator area and the second indicator area is the target fusion boundary. The target fusion boundary is set at a position where the image difference meets a set condition. The image difference includes the difference between the first image and the third image. The third image and the first image are fused based on the target fusion mask image to obtain the target fusion image; The step of determining the target fusion mask image based on the first image and the third image includes: Determine the pixel differences and optical flow information between the first image and the third image; A first fusion mask image is determined based on the pixel differences and the optical flow information; The first fused mask image is downsampled to obtain the second fused mask image; The second fusion boundary in the second fusion mask image is adjusted to a position where the image differences meet the set conditions, thus obtaining the adjusted second fusion mask image; The adjusted second fused mask image is upsampled to obtain the target fused mask image.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, the following is achieved: Acquire a first image and a second image, wherein the first image is captured by a first camera of an electronic device and the second image is captured by a second camera of the electronic device, and the field of view of the first camera is greater than the field of view of the second camera; The second image is affinely transformed to the coordinate system of the first image to obtain the third image; A target fusion mask image is determined based on the first image and the third image. The target fusion mask image includes a first indicator area and a second indicator area. The first indicator area is used to indicate the fusion area, and the second indicator area is used to indicate the non-fusion area. The boundary between the first indicator area and the second indicator area is the target fusion boundary. The target fusion boundary is set at a position where the image difference meets a set condition. The image difference includes the difference between the first image and the third image. The third image and the first image are fused based on the target fusion mask image to obtain the target fusion image; The step of determining the target fusion mask image based on the first image and the third image includes: Determine the pixel differences and optical flow information between the first image and the third image; A first fusion mask image is determined based on the pixel differences and the optical flow information; The first fused mask image is downsampled to obtain the second fused mask image; The second fusion boundary in the second fusion mask image is adjusted to a position where the image differences meet the set conditions, thus obtaining the adjusted second fusion mask image; The adjusted second fused mask image is upsampled to obtain the target fused mask image.

17. A chip, characterized in that, include: Processors and interfaces; The processor is used to read instructions through the interface to execute the image fusion processing method according to any one of claims 1 to 13.

Images