Two-stage sky replacement method, device, and storage medium based on neural network

Abstract

The application provides a two-stage sky replacement method, device and storage medium based on a neural network, which comprises the following steps: a sky dataset construction step; a plurality of groups of samples are extracted from the sky dataset, a plurality of sky materials meeting preset requirements are selected to construct a sky material library, sky replacement is performed on the samples and the sky materials in the sky material library, and the color, saturation and brightness of the original sky materials and the samples are adjusted in terms of fusion coordination to obtain a sky fusion dataset; a sky matting model and a sky fusion model are constructed based on the sky dataset and the sky fusion dataset; and a sky replacement step: a sky replacement image is output. The application makes the boundary between the sky and non-sky areas in the image more natural and smooth, and improves the fusion effect of the sky in the image.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a two-stage sky replacement method, device, and storage medium based on neural networks. Background Technology

[0002] With the rapid development of digital SLR cameras, smartphones, and other devices, photography has gradually become an integral part of people's lives, used to record and share them. Outdoor photography is a crucial aspect of this, as clear blue skies or stunning sunsets are not always readily available. Therefore, sky replacement has become an essential skill for photo retouchers. Thanks to advancements in computer vision and deep learning technologies, an increasing number of methods have been proposed to solve the problem of automatic sky replacement. These methods only require the input of a photo and the sky image to be replaced, and the sky replacement can be completed automatically. Yi-Hsuan Tsai et al. proposed a coarse-to-fine method in Sky is Not the Limit: Semantic-Aware Sky Replacement (ACM Transactions on Graphics, 2016) to obtain accurate sky segmentation for sky replacement. Sky segmentation is essentially a "hard segmentation" with values ​​of either 0 or 1, which can easily lead to a "harsh" appearance at the boundary between the replaced sky and the original sky. More recent work Castle in the Sky: Dynamic Sky Replacement and Harmonization in Videos (2020) changed the sky segmentation map used for sky replacement to a softer matting method. The matting is essentially a probability map with values ​​ranging from [0,1], similar to the alpha channel of a PNG. The closer a pixel value is to 1, the more likely that the pixel belongs to the sky region. Based on the sky matting, an alpha fusion operation is then performed to obtain the sky replacement result. During the fusion process, this method designed recolor and relight operations to solve the color and light coordination problem between the sky and the original image by making the sky and the original image closer in terms of image mean.

[0003] In summary, to obtain high-quality sky replacement results, two issues need to be addressed: 1) a smooth transition between the sky and non-sky boundaries; and 2) the overall color harmony of the image after sky replacement. It's worth noting that previous methods primarily focused on issue 1), aiming to achieve smoother blending with a more refined and accurate sky mask. The blending techniques were generally based on simple alpha blending. However, neither sky segmentation nor sky matting yields satisfactory results. Separating the blending and matting issues means that while the sky matting result may be satisfactory from a purely matting perspective, it may still be insufficient for sky blending. Furthermore, current methods are somewhat coarse in their handling of color harmony; simply adjusting the image mean is clearly insufficient to achieve a realistic and natural blending effect. Summary of the Invention

[0004] To address the aforementioned issues, embodiments of this application provide a two-stage sky replacement method, device, and storage medium based on neural networks, which makes the boundary between sky and non-sky regions in an image transition naturally and smoothly, improving the fusion effect of the sky in the image.

[0005] Therefore, one aspect of this application provides a two-stage sky replacement method based on a neural network, the method comprising the following steps:

[0006] Steps to build a sky dataset: Select several images containing sky regions and build a sky dataset;

[0007] Steps for constructing a sky fusion dataset: Extract several sets of samples from the sky dataset, select several sky materials that meet the preset requirements to construct a sky material library, perform sky replacement on the samples and the sky materials in the sky material library, and adjust the color, saturation, and brightness between the original sky materials and the samples in terms of fusion coordination to obtain the sky fusion dataset;

[0008] Model building steps: Construct a sky matting model and a sky fusion model based on the sky dataset and the sky fusion dataset;

[0009] Sky replacement steps: Input the original image and sky material, obtain the sky matting and preliminary sky replacement results of the original image through the sky matting model, and fine-tune the sky fusion model according to the sky matting and preliminary sky replacement results of the original image to output the sky-replaced image.

[0010] Alternatively, in combination with any of the above aspects, in another implementation of this aspect, the input of the sky matting model is the original image and the sky dataset, and the output is the sky matting of the original image and the preliminary sky replacement result.

[0011] The input to the sky fusion model is the original image, the sky cutout of the original image, the sky material, and the random cropped pieces corresponding to the preliminary sky replacement result, and the output is the fused sky replacement image.

[0012] Alternatively, in conjunction with any of the above aspects, in another implementation of this aspect, the step of constructing the sky dataset specifically includes:

[0013] Select N images containing sky regions, and determine the x-axis for each image. i Corresponding sky cutout m i Sky cutout m i The original value range is between [0, 255];

[0014] Cut out the sky m i Dividing by 255 ensures the value range is between [0,1], resulting in the sky dataset D. matte ={(x i ,m i )|x i ∈X N ,m i ∈M N Let X = {i = 1, 2, ..., N}, where X = ... N M represents the original set of images. N This represents the set of sky cutouts corresponding to the original image, and N represents the size of the dataset.

[0015] Alternatively, in combination with any of the above aspects, in another implementation of this aspect, the step of constructing the sky fusion dataset specifically involves:

[0016] From the sky dataset D matte K sets of samples are extracted from the data, and then L sky images that meet the preset requirements are selected as the sky image library S. L ={(s1,s2,…,s K}; For the aforementioned sky dataset D matte Each group (x) i ,m i ) and Sky Material Library S L Each of the s in j After pre-processing, the result image y with sky replacement is obtained. ij That is, to obtain the sky fusion dataset D fusion ={(x i ,m i ,s j ,y ij )|x i ∈X K ,m i ∈M K ,s j∈S L ,y ij ∈Y KL ,i=1,2,…K;j=1,2,…L}, where X K M K They are X N and M N A subset of Y KL The set of resulting images for sky replacement, K=500, L=50.

[0017] Alternatively, in combination with any of the above aspects, in another implementation of this aspect, constructing the sky matting model includes the following steps:

[0018] Image x i The image is compressed to 1024*1024 pixels. The model input is a compressed image with dimensions [B, 3, 1024, 1024], and the model output is a sky cutout image with dimensions [B, 1, 1024, 1024]. i ′ The pixel value ranges from [0,1], where B is the size of the batch, which is usually 64.

[0019] The loss function uses the mean squared error (MSE) loss, and the calculation formula is as follows:

[0020]

[0021] During training, the epoch was set to 50, the Adam optimizer was used, and the learning rate lr was set to 1e-4.

[0022] Optionally, in combination with any of the above aspects, in another implementation of this aspect, the sky fusion model includes a multi-source information fusion network and a backbone detail correction network. The multi-source information fusion network integrates the heterogeneous input information, obtains the compensation amount required for the fusion result after preliminary alpha fusion in terms of fusion edge transition and color coordination, and outputs high-dimensional fusion compensation features and high-dimensional compensation features. The backbone detail correction network receives the high-dimensional features of the preliminary sky replacement result after alpha fusion after convolutional dimensionality increase and the detail correction features output by the multi-source information fusion network.

[0023] Alternatively, in conjunction with any of the above aspects, in another implementation of this aspect, the input size of the multi-source information fusion network is an image of size [B, 7, 512, 512]. Sky cutout sky material The input is concatenated, and the output is a high-dimensional fusion compensation feature T of size [B, 64, 512, 512]. add ; to the image Sky cutout sky material The initial sky replacement result was obtained after alpha fusion. The alpha fusion result is shown in the following formula.

[0024]

[0025] Will After two convolutional layers and a ReLU layer, a high-dimensional feature T of [B, 64, 512, 512] is obtained. init High-dimensional fusion compensation feature T add and high-dimensional features T init After concatenation, the result is input into the backbone detail correction network, and the output is a fused sky replacement image with a resolution of [B,3,512,512].

[0026] The loss function uses VGG-perceptual loss, and the calculation formula is as follows:

[0027]

[0028] in

[0029] B represents the batch size, w l This represents the weights corresponding to the outputs of each layer in VGG; during training, the epoch is set to 600, the Adam optimizer is used, and the learning rate is lr = 1e-4.

[0030] Alternatively, in combination with any of the above aspects, in another implementation of this aspect, the pre-defined processing in the step of constructing the sky fusion dataset includes sky replacement, color adjustment for the fused dataset, and handling of detail issues.

[0031] Alternatively, in combination with any of the above aspects, in another implementation of this aspect,

[0032] In another aspect of this application, an electronic device is provided, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements a two-stage sky replacement method based on a neural network as described above.

[0033] In another aspect of this application, a storage medium is provided having a computer program stored thereon, which, when executed, implements a two-stage sky replacement method based on a neural network as described above.

[0034] As described above, this application provides a two-stage sky replacement method, device, and storage medium based on neural networks. The method constructs a two-stage sky replacement approach. The first stage builds a relatively simple sky matting model to obtain sky matting of the original image, yielding a preliminary sky replacement result. The second stage builds a sky fusion model to fine-tune the sky replacement result from the first stage, ultimately obtaining a natural and harmonious sky replacement result. The boundary transition between the sky and non-sky regions in the image obtained by this method is natural and smooth, and the overall image appears harmonious and natural.

[0035] The above summary provides a simplified overview of some concepts, which will be further described in detail in the following specific embodiments. The above summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter. The claimed subject matter is not limited to embodiments that address any or all the shortcomings pointed out in the background art. Attached Figure Description

[0036] The accompanying drawings, incorporated in and forming part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without any creative effort. These drawings and textual descriptions are not intended to limit the scope of the concept of this application in any way, but rather to illustrate the concepts of this application to those skilled in the art by referring to specific embodiments.

[0037] Figure 1 A schematic diagram of a two-stage sky replacement method based on a neural network provided in this application;

[0038] Figure 2 A schematic diagram of the first and second stages of a two-stage sky replacement method based on a neural network provided in this application;

[0039] Figure 3 A schematic diagram of the sky fusion model in a two-stage sky replacement method based on a neural network provided in this application;

[0040] Figure 4 A schematic diagram of the original image used in a two-stage sky replacement method based on a neural network provided in this application;

[0041] Figure 5 This is a schematic diagram of the sky material to be replaced in the flowchart of a two-stage sky replacement method based on a neural network provided in this application.

[0042] Figure 6 A schematic diagram of the sky matting output by the sky matting model in a two-stage sky replacement method based on neural networks provided in this application;

[0043] Figure 7 A schematic diagram of the preliminary sky replacement result obtained after the original image and sky material are processed by a sky matting model and alpha fusion in a two-stage sky replacement method based on neural networks provided in this application;

[0044] Figure 8 The sky replacement result is output in the two-stage sky replacement method based on neural networks provided in this application. Detailed Implementation

[0045] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0046] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0047] It should be understood that although the terms first, second, third, etc., may be used herein to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this document, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if," as used herein, may be interpreted as "when," "when," or "in response to determination." Furthermore, as used herein, the singular forms "a," "an," and "the" are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms "comprising," "including," indicate the presence of the stated feature, step, operation, element, component, item, kind, and / or group, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, kinds, and / or groups. The terms "or," "and / or," "including at least one of the following," etc., as used in this application, may be interpreted as inclusive, or mean any one or any combination thereof. An exception to this definition will only occur if the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0048] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0049] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”

[0050] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0051] Please see Figure 1This method provides a two-stage sky replacement method based on neural networks. The boundary between the sky and non-sky regions in the image obtained by this method is natural and smooth, which improves the fusion effect of the sky in the image.

[0052] Specifically, this method includes the following steps:

[0053] Step S1: Constructing the Sky Dataset. Select several images containing sky regions to construct a sky dataset. Select N images containing sky regions and determine the x-axis for each image. i All were obtained by using image processing tools such as Photoshop to cut out the sky. i m i The original value range is between [0, 255]. Sky cutout m i Dividing by 255 ensures the value range is between [0,1]. The closer a pixel value is to 1, the more likely it is to belong to the sky region, resulting in the sky dataset D. matte ={(x i ,m i )|x i ∈X N ,m i ∈M N Let X = {i = 1, 2, ..., N}, where X = ... N M represents the original set of images. N This represents the set of sky cutouts corresponding to the original image, and N represents the size of the dataset, N = 5000.

[0054] Step S2, Building a Sky Fusion Dataset: Extract several sets of samples from the sky dataset, select several sky materials that meet the preset requirements to build a sky material library, replace the sky in the samples and the sky materials in the sky material library, and adjust the color, saturation, and brightness between the original sky materials and the samples in terms of fusion coordination to obtain the sky fusion dataset;

[0055] The sky dataset D obtained from step S1 matte K groups of samples were extracted from the data, and then L sky images with certain differences in color temperature, exposure, hue, etc., were selected as the sky image library S. L ={(s1,s2,…,s K}, for D matte Each group (x) i ,m i ) and S L Each of the s in j All images were pre-processed to obtain the result image y after sky replacement. ij The preset processing in this implementation includes: the retoucher using image processing tools such as Photoshop to process D... matteEach group (x) i ,m i ) and S L Each of the s in j The digital retouching process includes at least sky replacement, color correction for post-blending consistency, and handling of some minor details, resulting in a sky-blended dataset D. fusion ={(x i ,m i ,s j ,y ij )|x i ∈X K ,m i ∈M K ,s j ∈S L ,y ij ∈Y KL ,i=1,2,…K;j=1,2,…L}, where X K M K They are X N and M N A subset of Y KL The set of resulting images is used to replace the sky, where K = 500 and L = 50.

[0056] Step S3: Model building step: Construct a sky matting model and a sky fusion model based on the sky dataset and the sky fusion dataset.

[0057] The first stage involves constructing a sky matting model based on the Unet network. A relatively simple sky matting model is used to obtain the sky matting of the original image, which is then alpha-blended to obtain a preliminary sky replacement result. The specific steps for constructing the sky matting model are as follows:

[0058] Image x i The dimensions are compressed to 1024*1024, and the model input is a compressed image with dimensions [B, 3, 1024, 1024].

[0059] ′

[0060] The image outputs a sky cutout m of [B, 1, 1024, 1024]. i The pixel value ranges from [0,1], where B refers to the batch size, which is usually 64.

[0061] The loss function uses the mean squared error (MSE) loss, and the calculation formula is as follows:

[0062]

[0063] During training, the epoch was set to 50, the Adam optimizer was used, and the learning rate lr was set to 1e-4.

[0064] Secondly, the second stage involves constructing a sky fusion model, which fine-tunes the edges and colors of the sky replacement results from the first stage to ultimately obtain a natural and harmonious sky replacement result. The sky fusion model is as follows: Figure 2 As shown, the model includes a multi-source information fusion network and a backbone detail correction network. The model input includes... These correspond to the original image x in the sky that needs to be replaced. i Sky cutout from the original image m i and sky materials j The corresponding randomly cropped slices have slice sizes [H,W] = [512,512]. The cropping operation is performed using the torchvision library, and the model output is the merged sky replacement result.

[0065] The multi-source information fusion network integrates heterogeneous input information to obtain the compensation amounts required for complex details such as fusion edge transitions and color coordination in the preliminary alpha fusion result, and outputs high-dimensional fusion compensation features and high-dimensional compensation features. The backbone detail correction network receives the high-dimensional features of the preliminary sky replacement result after alpha fusion through convolution and the detail correction features output by the multi-source information fusion network.

[0066] The input size of the multi-source information fusion network model is an image of size [B, 7, 512, 512]. Sky cutout sky material The input is concatenated, and the output is a high-dimensional fusion compensation feature T of size [B, 64, 512, 512]. add ; to the image Sky cutout sky material The initial sky replacement result was obtained after alpha fusion. The alpha fusion result is shown in the following formula:

[0067]

[0068] Will After two convolutional layers and a ReLU layer, a high-dimensional feature T of [B, 64, 512, 512] is obtained. init High-dimensional fusion compensation feature T add and high-dimensional features T init After being stitched together, the result is input into the backbone detail correction network, which is built based on the Unet network. The output is a fused sky replacement image with a resolution of [B, 3, 512, 512].

[0069] The loss function uses VGG-perceptual loss, and the calculation formula is as follows:

[0070]

[0071] in B represents the batch size, w l This represents the weights corresponding to the outputs of each layer in VGG; during training, the epoch is set to 600, the Adam optimizer is used, and the learning rate is lr = 1e-4.

[0072] The sky fusion model obtained through this step can solve the following two problems:

[0073] 1. After the initial alpha blending, the blending boundary is further refined at the pixel level to achieve a more natural and smooth boundary transition;

[0074] 2. Solve the problem of blending and coordination, that is, how to harmoniously blend a photo where the sky image is taken on a sunny day and the sample photo is taken on a cloudy day, even though the two photos have different brightness levels. Figure 3 As shown, the color of the sky after fusion is significantly different from that of the original sky material. This solves the coordination issues of color, saturation, and brightness between the original sky material and the replaced sky sample, making the processed photo look more realistic and natural overall.

[0075] Step S4, Sky Replacement Step: Input the original image and sky material, obtain the sky matting and preliminary sky replacement results of the original image through the sky matting model, and fine-tune the sky fusion model according to the sky matting and preliminary sky replacement results of the original image to output the sky-replaced image.

[0076] like Figure 4 , 5 This is a set of original photos and sky materials that need to be replaced with sky data. Figure 6 This is the sky matting obtained using the sky matting model described above. Figure 7 The sky replacement result obtained from the classic alpha fusion method easily reveals two obvious problems:

[0077] 1. Obvious fusion artifacts exist at the boundary between the sky and non-sky areas;

[0078] and Figure 8 This is the sky replacement image obtained through this method, and... Figure 6 , 7 In comparison, the method obtained Figure 8 The transition between the sky and the non-sky is smooth and natural, and the overall tone of the merged photo is harmonious, resulting in a realistic and natural blending effect.

[0079] This application provides a two-stage sky replacement method, device, and storage medium based on neural networks. The method constructs a two-stage sky replacement approach. The first stage builds a relatively simple sky matting model to obtain sky matting from the original image, yielding a preliminary sky replacement result. The second stage builds a sky fusion model to fine-tune the sky replacement result from the first stage, ultimately obtaining a natural and harmonious sky replacement result. The boundary between the sky and non-sky regions in the image obtained by this method is natural and smooth, and the overall image appears harmonious and natural.

[0080] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0081] In this application, the same or similar terms, concepts, technical solutions and / or application scenario descriptions are generally described in detail only when they appear for the first time. When they appear again, they are generally not repeated for the sake of brevity. When understanding the technical solutions and other contents of this application, the same or similar terms, concepts, technical solutions and / or application scenario descriptions that are not described in detail later can be referred to their previous relevant detailed descriptions.

[0082] In this application, the descriptions of the various embodiments have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0083] The technical features of the present application can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present application.

[0084] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be an electrical device or a network device, etc.) to execute the methods of each embodiment of this application.

[0085] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A two-stage sky replacement method based on neural networks, characterized in that, This method includes the following steps: Steps to build a sky dataset: Select several images containing sky regions and build a sky dataset; Steps for constructing a sky fusion dataset: Extract several sets of samples from the sky dataset, select several sky materials that meet the preset requirements to construct a sky material library, perform sky replacement on the samples and the sky materials in the sky material library, and adjust the color, saturation, and brightness between the original sky materials and the samples in terms of fusion coordination to obtain the sky fusion dataset; Model building steps: Construct a sky matting model and a sky fusion model based on the sky dataset and the sky fusion dataset; Sky replacement steps: Input the original image and sky material, obtain the sky matting and preliminary sky replacement results of the original image through the sky matting model, and fine-tune the sky fusion model according to the sky matting and preliminary sky replacement results of the original image to output the sky-replaced image; The first stage of the model building process is to build a sky matting model based on the Unet network. The sky matting of the original image is obtained through a relatively simple sky matting model, and then the preliminary sky replacement result is obtained through alpha fusion. The second stage involves constructing a sky fusion model, which fine-tunes the edges and colors of the sky replacement results from the first stage to obtain a natural and harmonious sky replacement result. The sky fusion model includes a multi-source information fusion network and a backbone detail correction network, with inputs including... 、 , ， These correspond to the original images of the sky that need to be replaced. Sky cutout from the original image and sky materials The corresponding randomly cropped slices; the output is the sky replacement result after merging. ; The multi-source information fusion network integrates heterogeneous input information, obtains the compensation amount required for the fusion result after the initial alpha fusion in terms of fusion edge transition and color coordination, and outputs high-dimensional fusion compensation features and high-dimensional compensation features; the backbone detail correction network receives the high-dimensional features of the initial sky replacement result after alpha fusion after convolution and the detail correction features output by the multi-source information fusion network. The input size to the multi-source information fusion network model is the size. pictures 、 Sky cutout 、 sky material Input is concatenated, output is size. High-dimensional fusion compensation features ; to the image 、 Sky cutout 、 sky material The initial sky replacement result was obtained after alpha fusion. alpha The fusion result is shown in the following formula: Will After two convolutional layers and a ReLU layer, we obtain... High-dimensional features ， High-dimensional fusion compensation features and High-dimensional features After being concatenated, the data is input into the backbone detail correction network, which is built based on the Unet network, and outputs... The result of the sky replacement after fusion .

2. The two-stage sky replacement method based on neural network of claim 1, wherein: The input to the sky matting model is the original image and the sky dataset, and the output is the sky matting of the original image and the preliminary sky replacement result. The input to the sky fusion model is the original image, the sky cutout of the original image, the sky material, and the random cropped pieces corresponding to the preliminary sky replacement result, and the output is the fused sky replacement image.

3. The two-stage sky replacement method based on neural networks as described in claim 1, characterized in that: The steps for constructing the sky dataset are as follows: choose N Images of the sky area were collected to identify each image. Corresponding sky cutout Sky cutout The original value range is within between; Cut out the sky Dividing by 255 makes the value range fall within the range of 255. In between, we obtained the sky dataset. ， in Represents the original set of images. This represents the set of sky cutouts corresponding to the original image. Indicates the size of the dataset.

4. The two-stage sky replacement method based on a neural network as described in claim 3, characterized in that: The specific steps for constructing a sky fusion dataset are as follows: From the sky dataset Extracted from Group samples, then select Zhang uses sky materials that meet the preset difference requirements as the sky material library. For the aforementioned sky dataset Each group in And Sky Material Library Each of the images All images were pre-processed to obtain the result image with the sky replaced. That is, to obtain the sky fusion dataset ,in , They are and a subset of Replace the sky with the resulting image set. , .

5. The two-stage sky replacement method based on neural network of claim 4, wherein: Constructing a sky matting model involves the following steps: Image and Size compressed to 1024 1024, then combine the two to form The dimension is used as the model input, and the model output is... Sky cutout The pixel value range is within Between, among Set the batch size to 64.

6. The two-stage sky replacement method based on a neural network as described in claim 4, characterized in that: In the process of building the sky fusion dataset, the preset processing includes sky replacement, color adjustment for the fused data, and handling of details.

7. An electronic device, comprising: It includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the two-stage sky replacement method based on a neural network as described in any one of claims 1 to 6.

8. A storage medium, characterized by It stores a computer program that, when executed, implements the two-stage sky replacement method based on a neural network as described in any one of claims 1 to 6.

Images