Complex landscape multi-state image generation method and device and program product
Through semantic segmentation and a style transfer model combined with Gated-MLP and CycleGAN, local style transfer is carried out to complex images, solving the problem of long transfer time and overfitting of complex images in the existing technology, and achieving efficient and good-quality styles Migration effect.
Patent Information
- Application Number
- CN202411284494.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-13
- Publication Date
- 2024-12-17
AI Technical Summary
The prior art requires a lot of computing resources and time when realizing complex image style migration, and is prone to overfitting problems, resulting in poor image quality.
Semantic segmentation technology is used to segment the original image into different categories of semantic objects, and a style transfer and fusion of each semantic object is used to generate a style transfer landscape image using a style transfer model combining Gated-MLP and CycleGAN.
The efficiency and quality of image style transfer are improved and overfitting problems are avoided. The structural similarity index can reach 80.27%, the peak signal-to-noise ratio can reach 28.4020, and the generation time of a single image is shortened to 23.352799ms.
Smart Images
Figure CN119151773A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image processing technology, and in particular relates to a method, device and program product for generating multi-state images of complex landscapes. Background Art
[0002] Style transfer plays an important role in the field of image processing and artistic creation. By transferring the style of one image to another, new images with artistic appeal can be created, and new works of art can be explored and created. By combining landscape images of different styles, unique and eye-catching visual effects can be created, which is of great significance in the fields of urban planning and landscape design.
[0003] The existing technical solutions for image style transfer can be mainly divided into neural network-based methods, optimization algorithm methods and statistics-based methods.
[0004] (1) Neural network-based methods
[0005] Image style transfer methods based on neural networks are one of the important research directions in the field of modern computer vision. This type of method mainly uses convolutional neural networks (CNNs) to achieve image style transfer. By learning high-level features of images (such as edges, textures, etc.), convolutional neural networks can apply the artistic style of the target style image to the content image while maintaining the structural features of the content image.
[0006] Neural network style transfer usually consists of two parts: content loss and style loss.
[0007] The content loss measures the similarity between the synthesized image and the content image in terms of high-level features, while the style loss measures the similarity between the synthesized image and the style image in terms of low-level features (such as color and texture distribution). By minimizing these two loss functions, a synthesized image can be generated that retains the structure of the content image and has the artistic effect of the style image.
[0008] In recent years, Generative Adversarial Networks (GANs) have also been widely used in image style transfer. Through adversarial training between a generator and a discriminator, GANs can learn complex style transfer processes and generate high-quality synthetic images.
[0009] However, neural network-based methods require a lot of computing resources and long training time, especially for complex style transfer tasks. In addition, due to the complexity of neural networks, their internal mechanisms are difficult to explain, and overfitting problems may occur during training, causing the model to perform poorly when processing new images. Therefore, it takes a long time to achieve complex image style transfer.
[0010] (2) Optimization algorithm method
[0011] The application of optimization algorithms in image style transfer is mainly reflected in the optimization process of the loss function. Usually, by defining content loss and style loss, and using numerical optimization methods (such as gradient descent, L-BFGS algorithm, etc.) to minimize these losses, the optimal synthetic image is generated. In traditional neural style transfer, the gradient descent method is often used to gradually adjust the pixel values of the synthetic image to meet the requirements of content and style. The gradient descent method calculates the gradient of the loss function and adjusts the image pixels in the opposite direction of the gradient, thereby gradually approaching the minimum value of the loss function. L-BFGS (Limited-memory Broyden-Fletcher-Goldfarb-Shanno) is a quasi-Newton optimization algorithm that can efficiently optimize under memory constraints. Compared with the standard gradient descent method, the L-BFGS algorithm usually has a faster convergence speed and better optimization effect.
[0012] However, optimization algorithms often face the problem of slow convergence in image style transfer, especially the standard gradient descent method, which performs poorly in high-dimensional space. At the same time, the optimization process is prone to fall into local optimal solutions, affecting the final image quality. In addition, these algorithms are sensitive to the initial synthetic images and hyperparameter adjustments, and require a lot of experiments for tuning. As a result, it takes a long time to achieve complex image style transfer.
[0013] (3) Statistical methods
[0014] Image style transfer methods based on statistics achieve style transfer by analyzing and modeling the statistical distribution of image features (such as color histograms, texture statistical features, etc.). These methods usually involve statistical analysis of image features and statistical concepts such as covariance matrix and spectral decomposition. This type of method achieves style transfer by extracting statistical features of the style image (such as color distribution, texture pattern, etc.) and applying these statistical features to the content image. For example, the Gram matrix of the style image and the content image can be calculated, and the Gram matrix of the content image can be adjusted to be close to the Gram matrix of the style image, thereby achieving style transfer.
[0015] However, statistical-based methods have limited capabilities in capturing image details and global consistency. Complex style transfer tasks may require a large amount of training data to accurately model statistical features, and have poor adaptability when processing different types of images or styles. Targeted adjustments to algorithms or parameters may be required, resulting in a longer time to achieve complex image style transfer.
[0016] Therefore, it is urgent to develop a complex landscape multi-state image generation method, device and program product to solve the problems in the prior art. Summary of the invention
[0017] The present invention aims to provide a method for generating multi-state images of complex landscapes, by using semantic segmentation and style transfer models to solve the problem that the existing image style transfer methods proposed in the above background technology take a long time to achieve complex image style transfer.
[0018] In order to solve the above technical problems, the specific technical solutions of the present invention are as follows:
[0019] A method for generating a complex landscape multi-state image comprises the following steps:
[0020] Obtain original complex landscape images;
[0021] Perform semantic segmentation on the original complex landscape image to obtain initial semantic objects of multiple categories;
[0022] Input the initial semantic object into the style transfer model to obtain style transfer objects with various transfer effects;
[0023] The style-transferred object is fused with the original landscape image to obtain a style-transferred landscape image.
[0024] Furthermore, the style transfer model includes two generators and two discriminators, the generator is used to perform style transfer on the initial semantic object to obtain a style transferred object, and the discriminator is used to determine whether the style transferred object is real.
[0025] Furthermore, the generator includes multiple feature extraction blocks, and the feature extraction blocks are Gated-MLP blocks.
[0026] Furthermore, the discriminator also includes a plurality of feature extraction blocks, and the feature extraction blocks are Gated-MLP blocks.
[0027] Further, after the image is input into the Gated-MLP block, the steps of the Gated-MLP block include the following:
[0028] A normalized linear perceptron is used to extract the abstract semantic features of each image block and output the feature map F of the image;
[0029] Through the multi-layer perceptron, the feature map is mapped to the channel to obtain the context information F of the feature map F in the channel direction. C ;
[0030] The context information F C The feature map U and feature map V are divided into two equal-sized feature maps in the channel direction. The multi-layer perceptron performs spatial mapping on the feature map U and multiplies it with the scalar product of the feature map V to obtain the context information F. C The context F in the spatial direction CS ;
[0031] The context F CS It is residually connected to the feature map F through scalar addition as the output of Gated-MLP.
[0032] Furthermore, the generator also includes an initial convolution layer, a downsampling layer, an upsampling layer and a final convolution layer, and the discriminator also includes an initial convolution layer, a downsampling layer and a fully connected layer.
[0033] Furthermore, the semantic segmentation of the original complex landscape image to obtain initial semantic objects of various categories is achieved through the PSPNet model.
[0034] Furthermore, when the style transfer object is fused with the original landscape image, the style transfer object is fused with the original image through a scalar product of an image matrix.
[0035] A computer device comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.
[0036] A computer program product comprises a computer program, which implements the steps of the method when executed by a processor.
[0037] The present invention has the following advantages:
[0038] This application performs semantic segmentation on the original complex landscape image, determines the semantic categories of each different area, and then uses the style transfer model to perform targeted style transfer on the initial semantic objects of different semantic categories. Finally, the style-transferred objects are merged according to the original image to achieve the style transfer of the entire image. It does not require a large amount of computing resources or long-term training, and there will be no overfitting problem during the training process. The image quality after style transfer is better.
[0039] Furthermore, the style transfer model of the present application combines the Gated-MLP and CycleGAN models to globally model the semantic information and contextual relationships of the input image, better capture the global dependencies of the image, and thus improve the ability to understand the overall image structure. Its structural similarity index can reach 80.27%, and its peak signal-to-noise ratio can reach 28.4020. At the same time, the generation time of a single image can be reduced to 23.352799ms.
[0040] Other features and advantages of the present invention will be disclosed in detail in the following specific embodiments and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 It is a schematic diagram of the overall process of the present invention;
[0042] Figure 2 This is a schematic diagram of the generator's workflow;
[0043] Figure 3 Schematic diagram of the discriminator's workflow;
[0044] Figure 4 It is a schematic diagram of the workflow of the present invention. DETAILED DESCRIPTION
[0045] In order to better understand the purpose, structure and function of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings. In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.
[0046] A method for generating multi-state images of complex landscapes, such as Figure 1 and Figure 4 As shown, the following steps are included:
[0047] S1: Acquire the original complex landscape image;
[0048] S2: Perform semantic segmentation on the original complex landscape image to obtain initial semantic objects of multiple categories;
[0049] S3: Input the initial semantic object into the style transfer model to obtain style transfer objects with various transfer effects;
[0050] S4: Fuse the style-transferred object with the original landscape image to obtain a style-transferred landscape image.
[0051] In this embodiment, the original complex landscape image in S1 refers to an image containing at least two semantic categories; optionally, the original complex landscape image may also be an image containing at least one semantic category.
[0052] The semantic segmentation of the original complex landscape image in S2 to obtain multiple categories of initial semantic objects can be performed by using an image semantic segmentation model. The image semantic segmentation model includes a PSPNet model, an FCN model, a DeepLab model, or a U-Net model.
[0053] After the original complex landscape image is input into the image semantic segmentation model, initial semantic objects with different categories can be obtained. This process is prior art and will not be described in detail in this application. The initial semantic objects have semantic categories to facilitate the fusion of the style transfer object with the original landscape image.
[0054] In this embodiment, the image semantic segmentation model predefines the training of a large number of images of landscape object semantic categories, and can determine the semantic category of each area in the original complex landscape image.
[0055] In this embodiment, the PSPNet model is used to perform semantic segmentation of the original complex landscape image, so as to effectively segment the original complex landscape image into different regions with semantic information, and provide a basis for subsequent style transfer and local style adjustment while determining the semantic category, so as to more accurately retain and adjust the landscape style characteristics of different regions during the style transfer process, better handle the changes in style characteristics of different landscape regions, improve the quality and realism of the generated image, and can be combined with the style transfer model of the present application to effectively improve the effect and performance of the present application in processing the original complex landscape image.
[0056] The uniqueness of the PSPNet model in the field of semantic segmentation lies in its use of the pyramid pooling module, which can effectively capture contextual information at different scales. The integration of this multi-scale information enables PSPNet to more comprehensively understand and segment complex scenes, thereby improving segmentation accuracy and the accuracy of boundary details. In addition, PSPNet has a simple and efficient structure, and can achieve relatively fast reasoning speed while maintaining good performance, making it highly competitive in practical applications and helping to determine the semantic categories of each object in the original complex landscape image.
[0057] The style transfer model in S3 can be a cycle generative adversarial network model. Optionally, it can be a CycleGAN model. CycleGAN (Cycle-Consistent Generative Adversarial Network) is a deep learning model for unsupervised image conversion tasks. It is able to learn mappings from one domain to another without paired training data. The core innovation of CycleGAN is the introduction of cycle consistency loss and the use of adversarial loss to achieve bidirectional image conversion. This makes CycleGAN perform well in applications in different fields, such as style transfer, image translation, and even speech conversion.
[0058] Gated-MLP is a model structure based on multi-layer perceptron (MLP). It enhances the ability to capture long-range spatial dependencies in images through its gating mechanism and the combination of multiple MLP blocks, and can more effectively understand complex patterns and relationships in images.
[0059] In this embodiment, a style transfer model Gated-CycleGAN is proposed by combining the CycleGAN model and the gated multi-layer perceptron (Gated-MLP), which can perform style transfer on the extracted landscape objects respectively. Through targeted category transfer, style transfer objects of different landscape categories can be obtained.
[0060] Specifically, in this embodiment, the style transfer model includes two generators G and F, and two discriminators D_A and D_B, the generator G is used to convert the domain A style image into the domain B style image, and the generator F converts the domain B style image into the domain A style image. The discriminator D_A is used to distinguish between the real domain A style image and the generated domain A style image, and the discriminator D_B is used to distinguish between the real domain B style image and the generated domain B style image.
[0061] Specifically, the generator includes an initial convolution layer, a downsampling layer, multiple feature extraction blocks, an upsampling layer and a final convolution layer, wherein the multiple feature extraction blocks are Gated-MLP blocks.
[0062] like Figure 2 As shown, after the image is input into the generator, the following steps are included:
[0063] Patch Embedding divides the image into multiple small patches, each of which contains local image information.
[0064] After a layer of Gated-MLP blocks, the input small blocks or path patches are deeply extracted and transformed to obtain feature matrices;
[0065] After three layers of transposed convolution operations, the feature map is gradually enlarged and restored to the migrated image.
[0066] This application replaces the traditional ResNet block in the generator with a Gated-MLP block to enhance the feature extraction capability. Compared with the ResNet block, the Gated-MLP block of this application can better capture the global dependencies of the image, thereby improving the generator's ability to understand the overall image structure.
[0067] In this embodiment, the discriminator includes an initial convolutional layer, multiple feature extraction blocks, a downsampling layer and a final fully connected layer, wherein the feature extraction block is also a Gated-MLP block.
[0068] like Figure 3 As shown, after the image is input into the discriminator, the following steps are included:
[0069] Patch Embedding is used to divide the image into multiple patches, each of which contains local image information.
[0070] After a layer of Gated-MLP blocks, deep feature extraction is performed on the input patch to learn the local features and global structure of the image to obtain a feature map;
[0071] The obtained feature map is input into the convolutional layer to further learn the spatial structure and combination of features. Finally, a probability value is output through the Discriminative to indicate the probability that the input image is a real image, thereby judging the authenticity of the input image.
[0072] This application changes the feature extraction block in the discriminator to a Gated-MLP block, and uses the global visual perception ability of the Gated-MLP block to more comprehensively understand the semantic information and structural features of the entire image. In contrast, the ResNet block is generally limited by the design of local connections and residual blocks, making it difficult to fully capture the global features of the entire image, while the Gated-MLP block is lighter and has higher perception capabilities.
[0073] Through the output of the discriminator, the generator can know the probability that the generated image is judged as "real". This feedback signal can help the generator adjust its own generation strategy, making the generated image closer to the real image, improving the performance of the generator and the quality of the generated image.
[0074] After the image is input into the Gated-MLP block, the Gated-MLP block includes the following steps:
[0075] S31: Take the image block Split(x) as input, where the forward Patch Embedding layer divides the image x into 4x4 image blocks Split(x); the normalized linear perceptron (NLP) extracts the abstract semantic features of each image block and outputs the feature map F of the image, as shown below:
[0076] F = LP(Split(x));
[0077] Among them, F is the feature map, x is the input image, Split means segmentation, Split(x) means the image block after PatchEmbedding processing, and LP means linear projection, that is, the normalized linear perceptron (NLP) is used to extract the abstract semantic features of each image block.
[0078] S32: Perform channel mapping on the feature map F through a two-layer multi-layer perceptron (MLP) to obtain the context information F of the feature map F in the channel direction C , as shown below:
[0079] F C =GELU(MLP(F));
[0080] Among them, F C is the context information of the feature map F in the channel direction, and GELU represents the non-linear activation function.
[0081] S33: The context information F of the feature map F in the channel direction C In the channel direction, it is divided into two equally sized feature maps U and V. U is spatially mapped by two layers of MLP to obtain U g and through U g The scalar product of F and V is C In the context of spatial directions, the formula is as follows:
[0082] U,V=Split(F C );
[0083] U g =MLP(U);
[0084] F CS =U g ×V;
[0085] Among them, F CS F C In the context of spatial directions, U g To update the feature map mapping, the feature map is obtained after spatial mapping of U through two layers of MLP, which is used to capture and update spatial information. MLP means spatial mapping through MLP.
[0086] S34: Final F CS The residual connection with F is made through scalar addition as the output F of Gated-MLP out , while retaining certain local features while extracting the channel space context, as shown below:
[0087] F out =F+MLP(F CS );
[0088] Among them, F out is the output of Gated-MLP.
[0089] This step is due to F CS The channel dimension of F is different from that of F, so the final CS Do channel mapping so that F CS The dimension is consistent with F.
[0090] In step S4, the style transfer object is merged with the original landscape image to obtain the style transfer landscape image, including the following steps:
[0091] The style-transferred object after style transfer is matched with the initial semantic object obtained by semantic segmentation and brought back to the original image. The matching process uses the scalar product of the image matrix. The basic formula is as follows:
[0092] X rec =(OM)×X+X trans ;
[0093] Where, X rec is a style-transferred landscape image; O is a matrix of all 1s, in this embodiment, it is a matrix of all 1s of 256×256; M is the label image after segmentation by PSPNet, X is the original complex landscape image, X trans It is the style transfer object.
[0094] The label map M segmented by PSPNet is inverted to obtain the binary image of each semantics, which is scalar-multiplied with the original complex landscape image X and then with the style transfer object X migrated by Gated-CycleGAN. trans Add them together to finally get the style migration landscape image X rec .
[0095] After performing style transfer on different categories of landscape objects in the original complex landscape image, this application combines the semantic categories of the original complex landscape image to obtain style transfer objects of different categories with different transfer effects, and fuses the style transfer objects obtained after style transfer with the original image through the scalar product of the matrix to complete local style transfer. The whole process preserves the overall semantics of the image while integrating the style of the specified area with the original image, thereby achieving a change in the local landscape style.
[0096] Table 1 shows the experimental data after experiments on real data sets, as follows:
[0097]
[0098]
[0099] Among them, Structural Similarity (SSIM) is an indicator to measure the similarity between two images. Peak Signal-to-Noise Ratio (PSNR) is an objective standard to measure the quality of images and videos.
[0100] According to the real experimental data of this embodiment, the structural similarity index of this application can reach 80.27%, the peak signal-to-noise ratio can reach 28.4020, and the single image generation time can be reduced to 23.352799ms. Therefore, this application can perform style transfer more efficiently and accurately; in addition, compared with other models such as CycleGAN and DualGAN, Gated-CycleGAN has higher accuracy in different data sets.
[0101] A computer device comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.
[0102] A computer program product comprises a computer program, which implements the steps of the method when executed by a processor.
[0103] A computer-readable storage medium stores a computer program, which implements the steps of the method when executed by a processor.
[0104] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.
[0105] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.
Claims
1. A method for generating a complex landscape multi-state image, characterized in that: The steps include: Obtain original complex landscape images; Perform semantic segmentation on the original complex landscape image to obtain initial semantic objects of multiple categories; Input the initial semantic object into the style transfer model to obtain style transfer objects with various transfer effects; The style-transferred object is fused with the original landscape image to obtain a style-transferred landscape image.
2. The complex landscape multi-state image generation method according to claim 1, characterized in that: The style transfer model includes two generators and two discriminators. The generator is used to perform style transfer on the initial semantic object to obtain a style transferred object, and the discriminator is used to determine whether the style transferred object is real.
3. The complex landscape multi-state image generation method according to claim 2, characterized in that: The generator includes a plurality of feature extraction blocks, and the feature extraction blocks are Gated-MLP blocks.
4. The complex landscape multi-state image generation method according to claim 3, characterized in that: The discriminator also includes a plurality of feature extraction blocks, and the feature extraction blocks are Gated-MLP blocks.
5. The complex landscape multi-state image generation method according to claim 3 or 4, characterized in that: After the image is input into the Gated-MLP block, the steps of the Gated-MLP block include the following: A normalized linear perceptron is used to extract the abstract semantic features of each image block and output the feature map F of the image; Through the multi-layer perceptron, the feature map is mapped to the channel to obtain the context information F of the feature map F in the channel direction. C ; The context information F C The feature map U and feature map V are divided into two equal-sized feature maps in the channel direction. The multi-layer perceptron performs spatial mapping on the feature map U and multiplies it with the scalar product of the feature map V to obtain the context information F. C The context F in the spatial direction CS ; The context F CS It is residually connected to the feature map F through scalar addition as the output of Gated-MLP.
6. The complex landscape multi-state image generation method according to claim 5, characterized in that: The generator also includes an initial convolution layer, a downsampling layer, an upsampling layer and a final convolution layer, and the discriminator also includes an initial convolution layer, a downsampling layer and a fully connected layer.
7. The complex landscape multi-state image generation method according to any one of claims 1 to 6, characterized in that: The semantic segmentation of the original complex landscape image to obtain initial semantic objects of multiple categories is achieved through the PSPNet model.
8. The complex landscape multi-state image generation method according to claim 7, characterized in that: When the style transfer object is fused with the original landscape image, the style transfer object is fused with the original image through a scalar product of an image matrix.
9. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.