A multi-style fusion image generation method based on deep learning

Abstract

This invention relates to the field of image fusion technology, specifically to a deep learning-based method for generating multi-style fusion images. The invention aligns an input binary semantic mask into a two-dimensional grid space composed of the width and height of an encoded feature tensor to obtain an aligned binary semantic mask. Based on the distribution of mask values ​​at different positions of the aligned binary semantic mask, spatial core positions for two semantic styles are selected respectively. Based on the feature vectors corresponding to different spatial core positions for each semantic style, and the feature vectors corresponding to each spatial position, the geometric residual components and inter-style center position coefficients for each spatial position are obtained, and output feature vectors for different spatial positions are obtained. These are input into a pre-trained image decoder to output a style-fused image. This invention enhances the quality of the fused image by accurately obtaining the optimized output feature vector through feature recombination based on residual preservation.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, and more specifically to a method for generating multi-style fusion images based on deep learning. Background Technology

[0002] Deep learning image generation models typically employ an encoder-decoder structure to handle multi-style fusion or style transfer tasks. The encoder is responsible for mapping the input image to a deep feature space, while the decoder restores the features to pixel images. In this process, the encoder generally contains convolutional and pooling layers, which causes the spatial resolution of the feature map to gradually decrease. However, at the boundaries of different semantic regions of the input image, a single feature vector on the deep feature map actually corresponds to the region that crosses the boundary in the original image, representing a linear mixture of two different style feature distributions.

[0003] In existing technologies, simple spatial interpolation or hard partitioning based on masks is usually relied upon to determine the region assignment, but the mixed components in the feature vector are not removed. This results in color leakage, edge blurring or unnatural artifacts at the boundaries of the generated image when the decoder parses the mixed features in the intermediate transition state, leading to poor image fusion quality. Summary of the Invention

[0004] To address the technical problem of poor image fusion quality due to the failure to numerically remove mixed components from feature vectors, this invention aims to provide a deep learning-based multi-style fusion image generation method. The specific technical solution adopted is as follows: This invention proposes a multi-style fusion image generation method based on deep learning, the method comprising: Obtain the input binary semantic mask and the encoded feature tensor of the image. The encoded feature tensor contains the feature vectors of the encoded feature map along the channel dimension, as well as the corresponding width and height. Align the input binary semantic mask to a two-dimensional grid space consisting of the width and height of the encoded feature tensor to obtain an aligned binary semantic mask; based on the distribution of mask values ​​at different positions of the aligned binary semantic mask, select the spatial core positions of the two semantic styles respectively; based on the feature vectors corresponding to the different spatial core positions of the different semantic styles, obtain the average feature vector of each semantic style; Based on the average feature vector distribution of the two semantic styles and the feature vector distribution corresponding to each spatial location, the geometric residual component and the inter-style center position coefficient of each spatial location are obtained, and the weighted repair coefficient of each spatial location is obtained; based on the weighted repair coefficient distribution of different spatial locations, the mask value distribution in the aligned two-dimensional semantic mask, and the average feature vector, geometric residual component and corresponding feature vector of the corresponding semantic style, the output feature vector of different spatial locations is obtained. The output feature vectors from different spatial locations are input into a pre-trained image decoder, which outputs a style fusion image.

[0005] Furthermore, the method for obtaining the spatial core location includes: In an aligned binary semantic mask, mask values ​​1 and 0 each correspond to a semantic style; For any semantic style, if the mask values ​​of all positions within the neighborhood of a spatial location are the corresponding semantic style, then the mask value of the corresponding spatial location is taken as the spatial core position of the corresponding semantic style. If the mask values ​​of all locations within the neighborhood of a spatial location are of the corresponding semantic style, then the spatial location whose mask value in the aligned binary semantic mask is of the corresponding semantic style is taken as the spatial core location of the corresponding semantic style.

[0006] Furthermore, the method for obtaining the average feature vector includes: The mean of the feature vectors corresponding to all core spatial locations for each semantic style is obtained, and this mean is used as the average feature vector for each semantic style.

[0007] Furthermore, the method for obtaining the geometric residual components includes: Take the average feature vector of any semantic style as the starting feature vector, calculate the difference between the average feature vector of another semantic style and the starting feature vector, and use it as the principal displacement axis vector; The displacement vector is normalized based on the magnitude of the principal axis displacement to obtain the unit vector of the principal axis displacement. Based on the principal displacement axis vector, the unit vector of the principal style displacement axis, and the eigenvector corresponding to each spatial location, the geometric residual components of each spatial location are obtained.

[0008] Furthermore, the method for obtaining the geometric residual components includes: The difference between the feature vector corresponding to each spatial location and the feature vector of the starting point is obtained and used as the offset vector relative to the starting point. Obtain the product between the relative starting point offset vector and the unit vector of the displacement principal axis for each spatial location, and use it as the projection component; Obtain the product between the projection component and the unit vector of the principal displacement axis, calculate the difference between the relative starting point offset vector of each spatial location and the product result, and use it as the geometric residual component of each spatial location.

[0009] Furthermore, the method for obtaining the inter-style center position coefficient includes: The projection components are normalized based on the magnitude of the principal axis of displacement to obtain the position ratio; the position ratio is then linearly mapped to the center peak to obtain the center position coefficient between styles.

[0010] Furthermore, the method for obtaining the weighted repair coefficients includes: Negative correlation mapping is performed on the magnitude of the principal axial displacement, which serves as a structural sensitivity factor. The product of the modulus of the geometric residual component and the structural sensitivity factor at each spatial location is obtained as the geometric structural strength. The sum of the inter-style center position coefficient, the positive integer 1, and the geometric structure strength is obtained and used as the weighted repair coefficient.

[0011] Furthermore, the method for obtaining the output feature vector includes: Based on the weighted repair coefficient distribution of different spatial locations, the spatial locations to be repaired are selected; For the spatial location to be repaired, the output feature vector of the spatial location to be repaired is obtained based on the average feature vector of the corresponding semantic style in the aligned two-dimensional semantic mask and the geometric residual component. For spatial locations other than the location to be repaired, the feature vector of the corresponding spatial location is used as the output feature vector.

[0012] Furthermore, the method for obtaining the location of the space to be repaired includes: The mean of the weighted repair coefficients for all spatial locations is obtained as the average weighted repair level; the standard deviation of the weighted repair coefficients for all spatial locations is obtained as the repair fluctuation characteristic. The sum of the repair fluctuation characteristics obtained from the average weighted repair level and the preset multiple is used as the repair confidence threshold; the maximum value between the repair confidence threshold and the preset minimum repair threshold is selected as the location repair benchmark threshold. If the weighted repair coefficient for each spatial location is greater than the location repair baseline threshold, the corresponding spatial location will be designated as the spatial location to be repaired.

[0013] Furthermore, the method for obtaining the output feature vector of the spatial location to be repaired includes: The average feature vector of the semantic style corresponding to the mask value of each spatial location to be repaired in the aligned two-dimensional semantic mask is used as the target vector of each spatial location to be repaired. The sum of the target vector and the geometric residual vector for each spatial location to be repaired is obtained and used as the output feature vector for each spatial location to be repaired.

[0014] The present invention has the following beneficial effects: This invention aligns the input binary semantic mask into a two-dimensional grid space composed of the width and height of the encoded feature tensor to obtain an aligned binary semantic mask, reflecting the semantic category to which the feature vector of the encoded feature tensor belongs at each position. Based on the distribution of mask values ​​at different positions of the aligned binary semantic mask, the spatial core positions of the two semantic styles are selected respectively, far away from the boundary region where convolutional mixing occurs. Based on the feature vectors corresponding to the different spatial core positions of different semantic styles, the average feature vector of each semantic style is obtained, reflecting the most essential distribution state of the semantic style in the feature space. Based on the distribution of the average feature vectors of the two semantic styles and the distribution of the feature vectors corresponding to each spatial position, the geometric residual component and the inter-style center position coefficient of each spatial position are obtained, representing the image structure information and whether it is in the intermediate transition zone between the two style regions. The weighted restoration coefficient of each spatial position is also obtained, reflecting the degree of mixing artifacts. Combining the mask value distribution in the aligned two-dimensional semantic mask and the average feature vector, geometric residual component and corresponding feature vector of the corresponding semantic style, the output feature vector of different spatial positions is obtained, which is input into the pre-trained image decoder to output the style fusion image. This invention enhances the quality of fused images by performing feature recombination based on residual preservation, accurately obtaining the optimized output feature vector of feature recombination. Attached Figure Description

[0015] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating a deep learning-based multi-style fusion image generation method provided in one embodiment of the present invention; Figure 2 This is a flowchart illustrating a method for obtaining an output feature vector according to an embodiment of the present invention. Detailed Implementation

[0017] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a deep learning-based multi-style fusion image generation method proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0019] The following description, in conjunction with the accompanying drawings, details a specific scheme for a deep learning-based multi-style fusion image generation method provided by the present invention.

[0020] Please see Figure 1 The diagram illustrates a flowchart of a deep learning-based multi-style fusion image generation method according to an embodiment of the present invention. The specific method includes: Step S1: Obtain the input binary semantic mask and the encoded feature tensor of the image. The encoded feature tensor contains the feature vectors along the channel dimension of the encoded feature map, as well as the corresponding width and height.

[0021] In embodiments of the present invention, the encoder generally contains convolutional and pooling layers, which leads to a gradual decrease in the spatial resolution of the feature map. This results in a single feature vector on the deep feature map actually corresponding to a region in the original image that crosses the boundary at the intersection of different semantic regions of the input image. Therefore, it is necessary to perform alignment analysis on the input semantic mask and the encoded feature tensor. First, the system receives the input image and the corresponding input binary semantic code. The input binary semantic mask is a binary matrix whose dimensions consist of the width and height of the input image. The input image is processed by a pre-trained image encoder to generate an encoded feature tensor with dimensions of the number of channels, height, and width. The height can be the pixel dimension of the encoded feature tensor in the vertical direction, and the width can be the pixel dimension of the encoded feature tensor in the horizontal direction. A two-dimensional grid is constructed based on the image height and width. The encoded feature tensor can be regarded as a channel with a height × width of the number of channels. Each spatial position in the two-dimensional network corresponds to a feature vector with a dimension of the number of channels. For example, if the number of channels is n, at any height × width position in the two-dimensional grid space, the feature vector [C1, C2, ..., Cn] is obtained, where C1 represents the feature value of the first channel, C2 represents the feature value of the second channel, and Cn represents the feature value of the nth channel.

[0022] Step S2: Align the input binary semantic mask to a two-dimensional grid space consisting of the width and height of the encoded feature tensor to obtain an aligned binary semantic mask; based on the distribution of mask values ​​at different positions of the aligned binary semantic mask, select the spatial core positions of the two semantic styles respectively; based on the feature vectors corresponding to the different spatial core positions of each semantic style, obtain the average feature vector of each semantic style.

[0023] Since the encoder contains pooling layers or strided convolutional layers, the spatial resolution reflected by the encoded feature tensor is lower than the original resolution of the input semantic mask. Furthermore, features at the boundaries are statistically contaminated due to overlapping receptive fields. To facilitate subsequent fusion processing, they need to be aligned to the same space for analysis. The input binary semantic mask is aligned to a two-dimensional grid space composed of the width and height of the encoded feature tensor to obtain an aligned binary semantic mask.

[0024] It should be noted that the height can be the pixel dimension of the encoded feature tensor in the vertical direction; the width can be the pixel dimension of the encoded feature tensor in the horizontal direction, and a two-dimensional grid space is formed based on the height and width; the nearest neighbor interpolation algorithm is used to downsample the input binary semantic mask to the two-dimensional grid space formed by the width and height of the encoded feature tensor, and the output is an aligned binary semantic mask.

[0025] The mask value at each position in the aligned binary semantic mask reflects the semantic category to which the feature vector of the encoded feature tensor belongs at each position. Based on the distribution of the mask values ​​at different positions in the aligned binary semantic mask, the spatial core positions of the two semantic styles are selected respectively.

[0026] Preferably, in one embodiment of the present invention, the method for obtaining the spatial core location includes: In an aligned binary semantic mask, mask values ​​1 and 0 each correspond to a semantic style; For any semantic style, if the mask values ​​of all positions within the neighborhood of a spatial location are the corresponding semantic style, then the mask value of the corresponding spatial location is taken as the spatial core position of the corresponding semantic style. If the mask values ​​of all locations within the neighborhood of a spatial location are of the corresponding semantic style, then the spatial location of the corresponding semantic style will be used as the spatial core location of the corresponding semantic style in the aligned binary semantic mask.

[0027] It should be noted that in order to obtain a pure style statistical benchmark, it is necessary to remove the mixed feature positions located near the semantic boundary. The selected spatial core position is far away from the boundary region where convolution mixing occurs, so the feature vector corresponding to the spatial core position is not contaminated by the mixing effect. In the embodiments of the present invention, the neighborhood range is obtained by taking any spatial position as the center and forming a range of 8 neighboring spatial positions.

[0028] The spatial core location can accurately reflect the original statistical distribution of semantic styles, and the feature vector reflects the most essential distribution state of semantic styles in the feature space. Based on the feature vectors corresponding to different spatial core locations of each semantic style, the average feature vector of each semantic style is obtained.

[0029] Preferably, in one embodiment of the present invention, the method for obtaining the average feature vector includes: The mean of the feature vectors corresponding to all core spatial locations for each semantic style is obtained, and this mean is used as the average feature vector for each semantic style.

[0030] It should be noted that the overall characteristics of each semantic style are quantified by calculating the mean, reflecting the baseline characteristics of each semantic style.

[0031] Step S3: Based on the average feature vector distribution of the two semantic styles and the feature vector distribution corresponding to each spatial location, obtain the geometric residual component and the inter-style center position coefficient for each spatial location, and obtain the weighted repair coefficient for each spatial location; based on the weighted repair coefficient distribution of different spatial locations, the mask value distribution in the aligned two-dimensional semantic mask, and the average feature vector, geometric residual component, and corresponding feature vector of the corresponding semantic style, obtain the output feature vector for different spatial locations.

[0032] The average feature vector reflects the cluster center of semantic styles in the feature space. The feature vector of each spatial location reflects the semantic style and image structure corresponding to the location. By analyzing the geometric relationship between the feature vector and the average feature vector of the two semantic styles, it is helpful to understand the information that each spatial location cannot be explained by the two semantic styles and to determine whether it is in the intermediate transition zone between the two style regions. Based on the distribution of the average feature vector of the two semantic styles and the distribution of the feature vector corresponding to each spatial location, the geometric residual components and the inter-style center position coefficients of each spatial location are obtained.

[0033] Preferably, in one embodiment of the present invention, the method for obtaining the geometric residual components includes: Take the average feature vector of any semantic style as the starting feature vector, calculate the difference between the average feature vector of another semantic style and the starting feature vector, and use it as the principal displacement axis vector; The displacement vector is normalized based on the magnitude of the principal axis displacement to obtain the unit vector of the principal axis displacement. It should be noted that the method for normalizing the displacement vector based on the magnitude of the principal displacement axis is as follows: calculate the ratio between the displacement vector and the magnitude of the principal displacement axis, and use it as the unit vector of the principal displacement axis; where, considering that the magnitude of the principal displacement axis may be 0, in order to avoid the formula being meaningless, a non-zero minimum positive number, such as 0.1, is added to the denominator during the calculation. Its value can be set according to the specific range of the denominator.

[0034] Based on the principal displacement axis vector, the unit vector of the principal style displacement axis, and the eigenvector corresponding to each spatial location, the geometric residual components of each spatial location are obtained.

[0035] Preferably, in one embodiment of the present invention, the method for obtaining the geometric residual components includes: The difference between the feature vector corresponding to each spatial location and the feature vector of the starting point is obtained and used as the offset vector relative to the starting point. Obtain the product between the relative starting point offset vector and the unit vector of the displacement principal axis for each spatial location, and use it as the projection component; Obtain the product between the projection component and the unit vector of the principal displacement axis, calculate the difference between the relative starting point offset vector of each spatial location and the product result, and use it as the geometric residual component of each spatial location.

[0036] Based on this, the product between the relative starting point offset vector and the unit vector of the displacement principal axis at each spatial location can reflect the projection length of the relative starting point offset vector on the unit vector of the displacement principal axis. The product between the projection component and the unit vector of the displacement principal axis is obtained as a vector along the direction of the unit vector of the displacement principal axis, and then the geometric residual component perpendicular to the principal axis is obtained.

[0037] Preferably, in one embodiment of the present invention, the method for obtaining the inter-style center position coefficient includes: The projection components are normalized based on the magnitude of the principal axis of displacement to obtain the position ratio; the position ratio is then linearly mapped to the center peak to obtain the center position coefficient between styles.

[0038] It should be noted that the method for obtaining the normalization of the projection component based on the magnitude of the principal displacement axis is as follows: calculate the ratio between the projection component and the magnitude of the principal displacement axis as the position ratio; where, considering that the magnitude of the principal displacement axis may be 0, in order to avoid the formula being meaningless, a non-zero minimum positive number, such as 0.1, is added to the denominator during the calculation. Its value can be set according to the specific range of the denominator.

[0039] It should be noted that the projection component reflects the relative position of each spatial location on the semantic style change principal axis. The larger the projection component, the longer the component on the principal axis, and the larger the position ratio. A central peak linear mapping is performed on the position ratio. When the position ratio is larger or smaller, each spatial location is closer to a certain semantic style location, and the inter-style central position coefficient is smaller. When the position ratio is closer to the middle, each spatial location is closer to the middle position of the two semantic style displacement unit principal axes, and the inter-style central position coefficient is larger. Therefore, the method for obtaining the central peak linear mapping of the position ratio is as follows: the position ratio ranges from [0,1]. Calculate the absolute value of the difference between the position ratio and half of the range's extreme value. Calculate the product of the absolute value of the difference and the multiple 2. Map the result to the range of 0-1. Calculate the difference between the positive integer 1 and the product result as the inter-style central position coefficient. The formula is expressed as: ;in, Indicates spatial location Style center position coefficient; Indicates spatial location The projected components; The modulus representing the principal axial displacement; Represents a non-zero, extremely small positive number; Indicates the computational spatial location The ratio of the projected component to the magnitude of the principal axial displacement, i.e., the position ratio.

[0040] While true semantic edges may be in an intermediate state at the projection location, they are usually accompanied by strong geometric structural changes. Artifacts caused by convolutional mixing usually appear as smooth mean transitions and lack independent structures. These may all be true edges that are misjudged as artifacts. When they are in an intermediate position and lack structure, the edges become blurred and need to be repaired. Therefore, the weighted repair coefficient is quantified by combining geometric residual components and inter-style center position coefficients.

[0041] Preferably, in one embodiment of the present invention, the method for obtaining the weighted repair coefficient includes: Negative correlation mapping is performed on the magnitude of the principal axial displacement, which serves as a structural sensitivity factor. It should be noted that, in one embodiment of the present invention, negative correlation mapping can be performed by taking the reciprocal of the magnitude of the principal axial displacement. Considering that the magnitude of the principal axial displacement may be 0, to avoid the formula becoming meaningless, a manually set non-zero minimum positive number, such as 0.1, is added to the denominator during calculation. Its value can be specifically set according to the range of values ​​in the denominator. In other embodiments of the present invention, an exponential function with the natural constant as its base is used. Negative correlation mapping is performed using techniques well-known to those skilled in the art, which will not be elaborated upon here.

[0042] The product of the modulus of the geometric residual component and the structural sensitivity factor at each spatial location is obtained as the geometric structural strength. The sum of the inter-style center position coefficient, the positive integer 1, and the geometric structure strength is obtained and used as the weighted repair coefficient.

[0043] Based on this, the larger the center position coefficient between styles, the more susceptible it is to the influence of intermediate mixing features, and the smaller the geometric structure strength, the less realistic the structure, and the greater the degree of repair required. The formula for the weighted repair coefficient is expressed as: ;in, Indicates spatial location The weighted repair coefficient; Indicates spatial location Style center position coefficient; Indicates spatial location The modulus of the geometric residual components; Indicates spatial location Structural sensitive factors; Indicates spatial location Geometric strength.

[0044] The weighted restoration coefficient distribution quantifies the overall mixing degree of the image as a benchmark; the mask value reflects the semantic style of the location, the average feature vector reflects the benchmark characteristics of the semantic style, and the geometric residual vector eliminates the transition axis projection components that cause visual blurring, retaining only the structural information perpendicular to the direction of style change, which helps to accurately reflect the feature information of the spatial location; based on the weighted restoration coefficient distribution of different spatial locations, the mask value distribution in the aligned two-dimensional semantic mask, and the average feature vector, geometric residual components, and corresponding feature vectors of the corresponding semantic styles, the output feature vectors of different spatial locations are obtained.

[0045] Preferably, in one embodiment of the present invention, the method for obtaining the output feature vector is described in [reference needed]. Figure 2 It illustrates a flowchart of a method for obtaining output feature vectors, including: Step S201: Select the spatial locations to be repaired based on the weighted repair coefficient distribution of different spatial locations.

[0046] Preferably, in one embodiment of the present invention, the method for obtaining the location of the space to be repaired includes: The mean of the weighted repair coefficients for all spatial locations is obtained as the average weighted repair level; the standard deviation of the weighted repair coefficients for all spatial locations is obtained as the repair fluctuation characteristic. The sum of the repair fluctuation characteristics obtained from the average weighted repair level and the preset multiple is used as the repair confidence threshold; the maximum value between the repair confidence threshold and the preset minimum repair threshold is selected as the repair confidence benchmark threshold. If the weighted repair coefficient for each spatial location is greater than the repair confidence benchmark threshold, the corresponding spatial location will be designated as the spatial location to be repaired.

[0047] Based on this, the mean reflects the baseline mixing level of the image. The larger the weighted restoration coefficient, the greater the possibility of it being in the middle mixing position. The standard deviation reflects the dispersion of the weighted restoration level. The larger the standard deviation, the greater the fluctuation of the weighted restoration level, the more discrete the distribution, and the larger the dynamic adjustment threshold. Combining the mean and standard deviation of the weighted restoration coefficient reflects the baseline of the feature mixing degree distribution of the image. If it exceeds the baseline threshold of restoration confidence, restoration is more necessary.

[0048] It should be noted that in a normal distribution, the mean ± 1 standard deviation covers approximately 68% of the data, and the mean ± 2 standard deviation covers approximately 95% of the data. To avoid filtering out too many small fluctuations and to ensure that the analysis requires repair of positions with higher reliability, the preset multiple is set relatively large. In one embodiment of the present invention, the preset multiple is set to 2 based on relevant historical experience. In other embodiments of the present invention, the preset multiple can be set according to specific circumstances, and will not be limited or elaborated here.

[0049] It should be noted that, considering that when the image is generally clean, the restoration level and restoration fluctuation characteristics are closer to 0, which makes normal small calculation fluctuations incorrectly identified as the presence of mixing artifacts, in one embodiment of the present invention, the minimum restoration threshold is preset to 0.1 based on relevant historical experience. In other embodiments of the present invention, the size of the preset minimum restoration threshold can be set according to specific circumstances, and will not be limited or elaborated here.

[0050] Step S202: For the spatial location to be repaired, obtain the output feature vector of the spatial location to be repaired based on the average feature vector of the corresponding semantic style in the aligned two-dimensional semantic mask and the geometric residual component.

[0051] Preferably, the average feature vector of semantic style reflects the base tone of the location, and combined with the residual components of geometric information, accurately reflects the texture features of the location; in one embodiment of the present invention, the method for obtaining the output feature vector of the spatial location to be repaired includes: The average feature vector of the semantic style corresponding to the mask value of each spatial location to be repaired in the aligned two-dimensional semantic mask is used as the target vector of each spatial location to be repaired. The sum of the target vector and the geometric residual vector for each spatial location to be repaired is obtained and used as the output feature vector for each spatial location to be repaired.

[0052] Step S203: For spatial locations other than the location to be repaired, use the feature vector of the corresponding spatial location as the output feature vector.

[0053] For spatial locations that do not require repair, such as those with a clean style and realistic structural edges, feature adjustments are less necessary.

[0054] Step S4: Input the output feature vectors from different spatial locations into the pre-trained image decoder to output a style fusion image.

[0055] The output feature vector eliminates the linear blending components at the boundary. The decoder maps the feature vector back to the pixel space. After receiving the feature vector, the decoder can automatically infer and recover the texture details that conform to the target style distribution based on the skeleton information. This eliminates color bleeding while avoiding the texture loss problem that may be caused by simple mean replacement. The generated image will show clear color boundaries and continuous structural textures at the boundaries of different styles.

[0056] In summary, this invention aligns the input binary semantic mask into a two-dimensional grid space composed of the width and height of the encoded feature tensor to obtain an aligned binary semantic mask. Based on the distribution of mask values ​​at different positions of the aligned binary semantic mask, spatial core positions for two semantic styles are selected respectively. Based on the feature vectors corresponding to different spatial core positions for each semantic style, and the feature vectors corresponding to each spatial position, the geometric residual components and inter-style center position coefficients for each spatial position are obtained, and output feature vectors for different spatial positions are obtained. These are input into a pre-trained image decoder to output a style-fused image. This invention enhances the quality of the fused image by accurately obtaining the optimized output feature vector through feature reorganization based on residual preservation.

[0057] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0058] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A multi-style fusion image generation method based on deep learning, characterized in that, The method includes: Obtain the input binary semantic mask and the encoded feature tensor of the image. The encoded feature tensor includes the feature vectors of the encoded feature map along the channel dimension, as well as the corresponding width and height. Align the input binary semantic mask to a two-dimensional grid space consisting of the width and height of the encoded feature tensor to obtain an aligned binary semantic mask; based on the distribution of mask values ​​at different positions of the aligned binary semantic mask, select the spatial core positions of the two semantic styles respectively; based on the feature vectors corresponding to the different spatial core positions of the different semantic styles, obtain the average feature vector of each semantic style; Based on the average feature vector distribution of the two semantic styles and the feature vector distribution corresponding to each spatial location, the geometric residual component and the inter-style center position coefficient of each spatial location are obtained, and the weighted repair coefficient of each spatial location is obtained; based on the weighted repair coefficient distribution of different spatial locations, the mask value distribution in the aligned two-dimensional semantic mask, and the average feature vector, geometric residual component and corresponding feature vector of the corresponding semantic style, the output feature vector of different spatial locations is obtained. The output feature vectors from different spatial locations are input into a pre-trained image decoder, which outputs a style fusion image.

2. The method for generating multi-style fusion images based on deep learning according to claim 1, characterized in that, The method for obtaining the location of the spatial core includes: In an aligned binary semantic mask, mask values ​​1 and 0 each correspond to a semantic style; For any semantic style, if the mask values ​​of all positions within the neighborhood of a spatial location are the corresponding semantic style, then the mask value of the corresponding spatial location is taken as the spatial core position of the corresponding semantic style. If the mask values ​​of all locations within the neighborhood of a spatial location are of the corresponding semantic style, then the spatial location whose mask value in the aligned binary semantic mask is of the corresponding semantic style is taken as the spatial core location of the corresponding semantic style.

3. The method for generating multi-style fusion images based on deep learning according to claim 1, characterized in that, The method for obtaining the average feature vector includes: The mean of the feature vectors corresponding to all core spatial locations for each semantic style is obtained, and this mean is used as the average feature vector for each semantic style.

4. The method for generating multi-style fusion images based on deep learning according to claim 1, characterized in that, The method for obtaining the geometric residual components includes: Take the average feature vector of any semantic style as the starting feature vector, calculate the difference between the average feature vector of another semantic style and the starting feature vector, and use it as the principal displacement axis vector; The displacement vector is normalized based on the magnitude of the principal axis displacement to obtain the unit vector of the principal axis displacement. Based on the principal displacement axis vector, the unit vector of the principal style displacement axis, and the eigenvector corresponding to each spatial location, the geometric residual components of each spatial location are obtained.

5. The method for generating multi-style fusion images based on deep learning according to claim 4, characterized in that, The method for obtaining the geometric residual components includes: The difference between the feature vector corresponding to each spatial location and the feature vector of the starting point is obtained and used as the offset vector relative to the starting point. Obtain the product between the relative starting point offset vector and the unit vector of the displacement principal axis for each spatial location, and use it as the projection component; Obtain the product between the projection component and the unit vector of the principal displacement axis, calculate the difference between the relative starting point offset vector of each spatial location and the product result, and use it as the geometric residual component of each spatial location.

6. The method for generating multi-style fusion images based on deep learning according to claim 5, characterized in that, The method for obtaining the inter-style center position coefficient includes: The projection components are normalized based on the magnitude of the principal axis of displacement to obtain the position ratio; the position ratio is then linearly mapped to the center peak to obtain the center position coefficient between styles.

7. The method for generating multi-style fusion images based on deep learning according to claim 6, characterized in that, The method for obtaining the weighted repair coefficient includes: Negative correlation mapping is performed on the magnitude of the principal axial displacement, which serves as a structural sensitivity factor. The product of the modulus of the geometric residual component and the structural sensitivity factor at each spatial location is obtained as the geometric structural strength. The sum of the inter-style center position coefficient, the positive integer 1, and the geometric structure strength is obtained and used as the weighted repair coefficient.

8. The multi-style fusion image generation method based on deep learning according to claim 1, characterized in that, The method for obtaining the output feature vector includes: Based on the weighted repair coefficient distribution of different spatial locations, the spatial locations to be repaired are selected; For the spatial location to be repaired, the output feature vector of the spatial location to be repaired is obtained based on the average feature vector of the corresponding semantic style in the aligned two-dimensional semantic mask and the geometric residual component. For spatial locations other than the location to be repaired, the feature vector of the corresponding spatial location is used as the output feature vector.

9. The method for generating multi-style fusion images based on deep learning according to claim 8, characterized in that, The method for obtaining the location of the space to be repaired includes: The mean of the weighted repair coefficients for all spatial locations is obtained as the average weighted repair level; the standard deviation of the weighted repair coefficients for all spatial locations is obtained as the repair fluctuation characteristic. The sum of the repair fluctuation characteristics obtained from the average weighted repair level and the preset multiple is used as the repair confidence threshold; the maximum value between the repair confidence threshold and the preset minimum repair threshold is selected as the location repair benchmark threshold. If the weighted repair coefficient for each spatial location is greater than the location repair baseline threshold, the corresponding spatial location will be designated as the spatial location to be repaired.

10. The method for generating multi-style fusion images based on deep learning according to claim 8, characterized in that, The method for obtaining the output feature vector of the spatial location to be repaired includes: The average feature vector of the semantic style corresponding to the mask value of each spatial location to be repaired in the aligned two-dimensional semantic mask is used as the target vector of each spatial location to be repaired. The sum of the target vector and the geometric residual vector for each spatial location to be repaired is obtained and used as the output feature vector for each spatial location to be repaired.

Images