Style migration method and device, storage medium and computer program product

Abstract

The embodiment of the invention provides a style migration method and device, a storage medium and a computer program product. The method comprises the steps that a first image and a second image are acquired; wherein the styles of the first image and the second image are different; inputting the first image and the second image into a first model to obtain a third image; wherein the content of the third image is similar to that of the first image, and the style of the third image is similar to that of the second image; the first model is obtained by training a first initial model based on a first loss function; wherein the first initial model comprises an initial generator, the first loss function comprises a second loss function, the second loss function is used for measuring the difference between the texture information of the output image of the initial generator and the texture information of the input image, and the style of the output result can be controlled by controlling the style of the second image. Therefore, a one-to-many style mapping relation is achieved, the cost is reduced, and the individual requirements of different customers are met.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a style transfer method, apparatus, storage medium, and computer program product. Background Technology

[0002] Style transfer is an image processing technique that aims to transfer the visual style of one image to another, thereby achieving a personalized expression in appearance while keeping the content unchanged. This technique is widely used in digital media, human-computer interaction interface design, and other fields, and is particularly valuable in UI / UX design, helping to enhance user emotional resonance and overall experience.

[0003] In related technologies, deep learning-based style transfer methods typically employ generative adversarial networks (GANs) or convolutional neural network (CNNs) to achieve style transfer effects. For example, a recurrent generative adversarial network (Cycle-GAN) can learn a one-to-one mapping relationship between two styles, thereby transforming the input image into an image of a specific style.

[0004] However, Cycle-GAN can only learn a one-to-one mapping relationship between two styles, meaning it can only transform an input image into an image of a specific style. It cannot control the transformation of an input image into an image of a specified style, thus failing to meet the personalized needs of customers. Furthermore, Cycle-GAN requires a large amount of training data and time to learn the mapping relationship between two different styles. Training a high-quality model may take several days or weeks, which greatly increases the training cost. At the same time, image transformation in complex scenes still suffers from distortion, affecting its promotion and use in scenarios with high real-time requirements, such as UI design. Summary of the Invention

[0005] This application provides a style transfer method, device, storage medium, and computer program product that can achieve a one-to-many style mapping relationship, meet the personalized needs of different customers, and greatly reduce training costs.

[0006] The technical solution of this application embodiment is implemented as follows: In a first aspect, embodiments of this application provide a style transfer method, the method comprising: Acquire a first image and a second image; wherein the first image and the second image have different styles; The first image and the second image are input into the first model to obtain a third image; wherein the third image is similar in content to the first image and in style to the second image; The first model is obtained by training a first initial model based on a first loss function; wherein the first initial model includes an initial generator, the first loss function includes a second loss function, and the second loss function is used to measure the difference between the texture information of the output image of the initial generator and the texture information of the input image.

[0007] Secondly, embodiments of this application provide a style transfer device, the style transfer device comprising: a processor and a memory; wherein, The memory is used to store computer programs that can run on the processor; The processor is configured to execute the style transfer method as described above when running the computer program.

[0008] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer program code, which, when executed by a computer, implements the style transfer method as described above.

[0009] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the style transfer method as described above.

[0010] This application provides a style transfer method, device, storage medium, and computer program product. The method includes: acquiring a first image and a second image; wherein the first image and the second image have different styles; inputting the first image and the second image into a first model to obtain a third image; wherein the third image is similar in content to the first image and similar in style to the second image; the first model is obtained by training a first initial model based on a first loss function; wherein the first initial model includes an initial generator, and the first loss function includes a second loss function, which is used to measure the difference between the texture information of the output image of the initial generator and the texture information of the input image. Therefore, by acquiring first and second images with different styles and inputting them into a first model trained with a specific loss function, a third image with content similar to the first image and style similar to the second image can be generated. This method introduces a second loss function to measure the texture difference between the output image and the input image, which improves the style consistency and content fidelity of the generated image, while reducing the dependence on a large amount of training data and improving training efficiency and generation quality. Furthermore, the embodiments of this application can control the style of the output result by controlling the style of the second image, thereby achieving a one-to-many style mapping relationship and realizing the requirement of generating multiple style class images using a single network model (i.e., the first model). This reduces costs and meets the personalized needs of different customers. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the style transfer method proposed in the embodiments of this application. Figure 1 ; Figure 2 This is a schematic diagram of the first encoder structure proposed in an embodiment of this application; Figure 3 This is a schematic diagram of the second encoder structure proposed in an embodiment of this application; Figure 4 This is a schematic diagram of the discriminator structure proposed in an embodiment of this application; Figure 5 This is a schematic diagram of the style transfer method proposed in the embodiments of this application. Figure 2 ; Figure 6 This is a schematic diagram of the UI web image database structure proposed in an embodiment of this application; Figure 7 This is a schematic diagram of the generator network structure proposed in an embodiment of this application; Figure 8 This is a schematic diagram of the composition and structure of the style transfer device proposed in the embodiments of this application. Figure 1 ; Figure 9 This is a schematic diagram of the composition and structure of the style transfer device proposed in the embodiments of this application. Figure 2 . Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the relevant application and not for limiting the application. Furthermore, it should be noted that, for ease of description, only the parts related to the relevant application are shown in the accompanying drawings.

[0013] Style transfer is an image processing technique that aims to transfer the visual style of one image to another, thereby achieving a personalized expression in appearance while keeping the content unchanged. This technique is widely used in digital media, human-computer interaction interface design, and other fields, and is particularly valuable in UI / UX design, helping to enhance user emotional resonance and overall experience.

[0014] In related technologies, deep learning-based style transfer methods typically employ generative adversarial networks (GANs) or convolutional neural network (CNNs) to achieve style transfer effects. For example, a recurrent generative adversarial network (Cycle-GAN) can learn a one-to-one mapping relationship between two styles, thereby transforming the input image into an image of a specific style.

[0015] However, Cycle-GAN can only learn a one-to-one mapping relationship between two styles, meaning it can only transform an input image into an image of a specific style. It cannot control the transformation of an input image into an image of a specified style, thus failing to meet the personalized needs of customers. Furthermore, Cycle-GAN requires a large amount of training data and time to learn the mapping relationship between two different styles. Training a high-quality model may take several days or weeks, which greatly increases the training cost. At the same time, image transformation in complex scenes still suffers from distortion, affecting its promotion and use in scenarios with high real-time requirements, such as UI design.

[0016] To address the current inability to control the conversion of input images into images of a specified style, thus failing to meet customers' personalized needs, this application provides a style transfer method, device, storage medium, and computer program product. The method includes: acquiring a first image and a second image; wherein the first image and the second image have different styles; inputting the first image and the second image into a first model to obtain a third image; wherein the third image is similar in content to the first image and similar in style to the second image; the first model is obtained by training a first initial model based on a first loss function; wherein the first initial model includes an initial generator, and the first loss function includes a second loss function, which measures the difference between the texture information of the output image of the initial generator and the texture information of the input image. Therefore, by acquiring first and second images with different styles and inputting them into a first model trained with a specific loss function, a third image with content similar to the first image and style similar to the second image can be generated. This method introduces a second loss function to measure the texture difference between the output image and the input image, which improves the style consistency and content fidelity of the generated image, while reducing the dependence on a large amount of training data and improving training efficiency and generation quality. Furthermore, the embodiments of this application can control the style of the output result by controlling the style of the second image, thereby achieving a one-to-many style mapping relationship and realizing the requirement of generating multiple style class images using a single network model (i.e., the first model). This reduces costs and meets the personalized needs of different customers.

[0017] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0018] This application provides a style transfer method. Figure 1 This is a schematic diagram of the style transfer method proposed in the embodiments of this application. Figure 1 ,like Figure 1 As shown, the style transfer method may include the following steps: Step 101: Obtain the first image and the second image; wherein the first image and the second image have different styles.

[0019] In embodiments of this application, the style transfer device can acquire a first image and a second image.

[0020] It should be noted that, in the embodiments of this application, the style transfer device can be any terminal or device with storage and communication functions. For example, the style transfer device can be a personal computer (PC). This application does not specifically limit the type of style transfer device.

[0021] It should be noted that, in the embodiments of this application, the first image may be a content image, used to provide basic structural information of the target image. For example, the first image may be a screenshot of a regular web page interface with standard layout and color scheme. This application does not specifically limit the type of the first image.

[0022] It should be noted that, in the embodiments of this application, the second image can be a style image, used to provide the artistic style characteristics of the target image. For example, the second image can be a UI design drawing with a unique artistic style uploaded by the user, such as a cartoon style or a retro style. This application does not specifically limit the type of the second image.

[0023] For example, in the embodiments of this application, the first image and the second image are from different databases: a general UI web database and a personalized UI web database. The personalized UI web database stores a large number of style elements with user preferences, while the general UI web database contains widely used standard interface styles. By selecting appropriate content images and style images from the general UI web database and the personalized UI web database, users can be provided with diverse style transfer options. This application does not specifically limit the acquisition methods of the first image and the second image.

[0024] Step 102: Input the first image and the second image into the first model to obtain the third image; wherein the third image is similar in content to the first image and similar in style to the second image; the first model is obtained by training the first initial model based on the first loss function; wherein the first initial model includes an initial generator, the first loss function includes a second loss function, and the second loss function is used to measure the difference between the texture information of the output image of the initial generator and the texture information of the input image.

[0025] In the embodiments of this application, after acquiring the first image and the second image, the style transfer device can input the first image and the second image into the first model to obtain the third image.

[0026] It should be noted that, in the embodiments of this application, the first model may use a Generative Adversarial Network (GAN) structure, including a generator and a discriminator; wherein, the generator is responsible for fusing the structural information of the first image with the visual features of the second image to output an image that both preserves the original content layout and possesses the target style. This application does not specifically limit the structural type of the first model.

[0027] It should be noted that, in the embodiments of this application, the specific structure of the first model obtained by training the first initial model based on the first loss function determines the detail representation and overall style effect of the generated image; by optimizing the parameters of the initial generator, a more natural and realistic style transfer result can be achieved, meeting the user's needs for personalized UI design.

[0028] It should be noted that, in the embodiments of this application, the first initial model refers to the initial network structure of the Generative Adversarial Network (GAN), which includes an initial generator and an initial discriminator. The first initial model has not been trained, and its parameters are in a randomly initialized state. The initial discriminator is a submodule in the first initial model, responsible for determining whether the input image comes from a real image database or is an image generated by the generator. The initial discriminator provides feedback to optimize the output quality of the generator.

[0029] It should be noted that, in the embodiments of this application, by introducing an initial discriminator, the ability of the first initial model to distinguish the authenticity of images can be enhanced, thereby enabling the generator to learn image representations that are closer to the target style. During subsequent training, the generator will more effectively adjust its output, making the output visually closer to the user's desired UI style.

[0030] For example, in an embodiment of this application, the style transfer device can train a first initial model based on a first loss function and a first training dataset to obtain a first model.

[0031] It should be noted that, in the embodiments of this application, the first training dataset may include a first image database and a second image database. The first image database includes one or more fourth images, and the second image patch includes one or more fifth images. The fourth images and the fifth images have different styles.

[0032] For example, in the embodiments of this application, the first image database can be a general UI web database, that is, a database that collects UI interface images that are common in daily life, and the fourth image can be an image selected from the general UI web database with a more common style. This application does not specifically limit the type of the fourth image.

[0033] For example, in an embodiment of this application, the second image database may be a personalized UI web database, that is, a database consisting of UI interface images with personalized styles that users prefer. The fifth image is an image selected from the personalized UI web database, which has the preferred style of a specific user. This application does not specifically limit the type of the fifth image.

[0034] It should be noted that, in the embodiments of this application, since the fourth and fifth images have different styles, this style difference helps the training model learn the mapping relationship between multiple styles, improving the model's generalization ability and adaptability. For example, during training, a common-style image can be input as the content image, and a personalized-style image can be input as the style image, thereby training a generator that can flexibly switch styles according to user needs.

[0035] It should be noted that, in the embodiments of this application, the first loss function refers to the total loss function, which may consist of three parts: a second loss function, a third loss function, and a fourth loss function. This application does not specifically limit the types and number of functions included in the first loss function.

[0036] It should be noted that, in the embodiments of this application, the second loss function may refer to the identity loss, which is used to measure the difference between the texture information of the initial generator output image and the texture information of the input image; the identity loss is used to ensure that the texture information of the generator's input content image and output image are consistent.

[0037] It should be noted that, in the embodiments of this application, the third loss function refers to adversarial loss. The role of adversarial loss is to ensure that the initial discriminator can distinguish between the real image and the output image of the initial generator, and to encourage the initial generator to generate a more realistic image. To improve image quality, the performance of the initial generator is optimized by minimizing the discriminative difference between the output image of the initial generator and the real image.

[0038] It should be noted that, in the embodiments of this application, the fourth loss function refers to Cycle Consistency Loss. The purpose of Cycle Consistency Loss is to ensure that the content information of the initial generator output image is consistent with the content information of the fourth image, preventing the loss of the original layout and structural information during style transfer. Cycle Consistency Loss helps the generated image achieve natural style transfer while retaining the original content.

[0039] For example, in an embodiment of this application, the formula for the first loss function is shown in the following formula (1).

[0040] (1) in, Denotes the first loss function. Indicating resistance to loss, This represents the loss of cycle consistency. Indicates an equal loss. , , They are respectively , , The parameters can have values ​​of 1, 10, or 10.

[0041] For example, in an embodiment of this application, the third loss function (i.e., adversarial loss) can be as shown in the following formula (2). The adversarial loss has the function of enabling the discriminator to distinguish between the real image and the output image of the generator, and the function of the generator to deceive the discriminator by generating an image that is close to the real image.

[0042] (2) in, This indicates the batch size. , Indicates the discriminator, , Represents a generator. This represents the content image (i.e., the fourth image). The style image (i.e., the fifth image).

[0043] For example, in an embodiment of this application, the fourth loss function (i.e., cycle consistency loss) can be as shown in the following formula (3), using cycle consistency loss to ensure that the output image of the generator has the same structure as the content image.

[0044] (3) For example, in an embodiment of this application, the second loss function (i.e., the identity loss) can be as shown in the following formula (4), using the identity loss to ensure that the texture information of the input content image and the output image of the generator are consistent.

[0045] (4) It should be noted that the style transfer method provided in this application not only reduces the need for training data and increases the diversity of generated images, but also effectively avoids the problem of image distortion by introducing identity loss and other methods, thereby significantly improving the personalized design capabilities and user experience of UI Web interfaces.

[0046] It should be noted that, in the embodiments of this application, the style transfer device can input the first image and the second image into the trained first model to obtain a third image that is similar in content to the first image and similar in style to the second image.

[0047] It should be noted that, in the embodiments of this application, the first model includes a generator, which includes a first encoder, a second encoder, and a decoder.

[0048] It should be noted that, in the embodiments of this application, the first encoder can be responsible for extracting the structural information of the content image (i.e., the first image), and the second encoder can be responsible for extracting the visual features of the style image (i.e., the second image).

[0049] Optionally, in embodiments of this application, when the style transfer device inputs the first image and the second image into the first model to obtain the third image, it can input the first image into the first encoder for content feature extraction to obtain the first image features; and input the second image into the second encoder for style feature extraction to obtain the second image features; and then the first image features and the second image features can be fused by the decoder to obtain the third image.

[0050] It should be noted that, in the embodiments of this application, content features may include feature vectors describing non-stylistic information such as object structure and layout in the image, such as borders, button positions, and text layout. This application does not specifically limit the type of content features.

[0051] It should be noted that, in the embodiments of this application, style features may include features describing the visual appearance of an image, such as color matching, texture style, font style, etc. This application does not specifically limit the type of style features.

[0052] It should be noted that in this embodiment, by extracting features from the content image and the style image separately, decoupling of content and style can be achieved. In the subsequent fusion process, the structural information of the original image is preserved, and a new style is assigned to it. Through this method, it can be ensured that the generated image maintains the integrity of the original content while possessing the style expression desired by the user, thereby enhancing the ability for personalized customization.

[0053] It should be noted that, in the embodiments of this application, the decoder is used to fuse content features (i.e., first image features) and style features (i.e., second image features), and to convert the fusion result of content features and style features into a visualized image, so as to ensure that the generated image meets expectations in terms of spatial resolution and visual effect.

[0054] In other words, in the embodiments of this application, through the fusion processing of the decoder, the structure of the content image and the visual features of the style image can be combined to generate a UI web interface image that meets user needs. This enables end-to-end image style transfer, significantly reducing training costs and computational resource consumption, while improving the quality and efficiency of image generation, thereby further optimizing the user's interactive experience and design satisfaction.

[0055] It should be noted that, in the embodiments of this application, Figure 2 This is a schematic diagram of the first encoder structure proposed in an embodiment of this application, as shown below. Figure 2 As shown, the first encoder may include N two-dimensional convolutional layers (Conv) and M residual blocks (ResBlk), where N and M are positive integers. For example, the first encoder may include 4 two-dimensional convolutional layers and 2 residual blocks. This application does not specifically limit the structure of the first encoder.

[0056] Optionally, in embodiments of this application, when the style transfer device inputs the first image to the first encoder for content feature extraction to obtain the first image features, it can sequentially input the first image to N two-dimensional convolutional layers and M residual blocks to output the first image features.

[0057] For example, in an embodiment of this application, the first image can be processed sequentially through N two-dimensional convolutional layers and M residual blocks. First, the image enters the first two-dimensional convolutional layer, where low-level features such as edges and corners are extracted using filters. Then, the image enters the next two-dimensional convolutional layer to extract higher-level features such as texture and shape. As the number of layers increases, the extracted features become increasingly abstract, ultimately forming an image feature vector with rich semantic information. After processing through all two-dimensional convolutional layers, the image enters the M residual blocks. Due to the skip connections within the residual blocks, the input signal can be directly transmitted to the output, avoiding information loss. Furthermore, the residual blocks can further optimize the image features through nonlinear transformations, which make the image features more stable and robust. Finally, after processing by the entire encoder, the first image is converted into a high-dimensional feature vector, i.e., the first image feature. The first image feature contains the main content information of the first image, providing a reliable foundation for the subsequent style transfer process.

[0058] It should be noted that, in the embodiments of this application, Figure 3 This is a schematic diagram of the second encoder structure proposed in an embodiment of this application, as shown below. Figure 3As shown, the second encoder includes L two-dimensional convolutional layers (Conv) and S average pooling layers (AvgPooling), where L and S are positive integers. For example, the second encoder may include 5 two-dimensional convolutional layers and 1 average pooling layer. This application does not specifically limit the structure of the second encoder.

[0059] Optionally, in embodiments of this application, when the style transfer device inputs the second image to the second encoder for style feature extraction to obtain the second image features, it can sequentially input the second image to L two-dimensional convolutional layers and S average pooling layers to output the second image features.

[0060] It should be noted that, in the embodiments of this application, an efficient image style feature extraction structure is constructed by setting L two-dimensional convolutional layers and S average pooling layers, which can extract multi-scale features of the image layer by layer, more accurately express the style attributes of the image, and thus achieve high-quality style transfer effect to meet the personalized needs of users.

[0061] It should be noted that, in the embodiments of this application, the decoder in the first model may include P normalized residual blocks, Q upsampled convolutional layers, and O fully connected layers, where P, Q, and O are positive integers; for example, it may include 2 normalized residual blocks, 4 upsampled convolutional layers, and 3 fully connected layers.

[0062] Optionally, in an embodiment of this application, when the style transfer device fuses the first image features and the second image features through a decoder to obtain a third image, it can input the second image features into O fully connected layers to obtain affine transformation features; then, it can input the affine transformation features and the first image features into P normalized residual blocks and Q upsampling convolutional layers to obtain the third image.

[0063] For example, in an embodiment of this application, after the second image features are processed by three fully connected layers, style-related global information can be extracted, thereby generating affine transformation features. Affine transformation features can refer to new features obtained by performing linear transformations (such as scaling or translation) on the original features. This transformation can preserve the structural information of the original features while enhancing the consistency between the original features and the target style.

[0064] It should be noted that, in the embodiments of this application, after receiving the affine transformation features and the first image features, the normalized residual block can first perform feature alignment and fusion operations, and enhance style expressiveness through multi-layer nonlinear transformations; subsequently, the fused features are passed to the upsampling convolutional layer to gradually enlarge the image size and restore image details. In the data processing between the normalized residual block and the upsampling convolutional layer, each layer relies on the output of the previous module in the normalized residual block and the upsampling convolutional layer as input, forming a continuous data processing flow.

[0065] In other words, in the embodiments of this application, the style transfer device can input the second image features into a multi-layer fully connected network, which converts the second image features into affine transformation features. Then, the affine transformation features and the first image features are input together into a normalized residual block and an upsampling convolutional layer, which fuse the affine transformation features and the first image features. The normalized residual block and the upsampling convolutional layer can improve the matching accuracy of content and style during style transfer, thereby enhancing the realism and personalized expressiveness of the output image, and thus meeting users' needs for diverse UI interface styles.

[0066] It should be noted that, in the embodiments of this application, the first model further includes a discriminator. Figure 4 This is a schematic diagram of the discriminator structure proposed in the embodiments of this application, as shown below. Figure 4 As shown, the discriminator may include E two-dimensional convolutional layers (Conv) and fully connected layers, where E is a positive integer; for example, the discriminator may include 5 two-dimensional convolutional layers and fully connected layers; where Output Image can represent a third image, and this application does not specifically limit the structure type of the discriminator.

[0067] Optionally, in embodiments of this application, after the style transfer device inputs the first image and the second image into the first model to obtain the third image, as follows: Figure 4 As shown, features can be extracted from the third image using E two-dimensional convolutional layers, and the extracted features are input into a fully connected layer to obtain a first feature vector, which is then used to identify whether the third image is a real image.

[0068] It should be noted that, in the embodiments of this application, a real image can refer to an original image that has not been processed or synthesized in any way. It usually has a high degree of visual realism and naturalness. By using a discriminator to determine whether the generated image is close to the real image, the quality of the generated image can be evaluated.

[0069] It should be noted that, in the embodiments of this application, the first feature vector can be a one-dimensional feature set output by a fully connected layer, which contains an abstract description of the overall characteristics of the image. The first feature vector can be used to measure the realism of the image, that is, it can be used to determine whether the third image is close to the distribution of a natural image.

[0070] It should be noted that, in the embodiments of this application, by sending the generated third image into a discriminator for feature extraction and authenticity determination, the problem of distorted or undesirable generated images can be effectively prevented, while ensuring that the image maintains the target style and also has a realistic visual feel. The above image processing flow not only improves the accuracy of image generation but also enhances the robustness and generalization ability of the model.

[0071] In summary, this application's embodiments design a few-shot learning network model (i.e., the first model). The few-shot learning employs a generative adversarial network (GAN) model, consisting of a generator and a discriminator. The generator takes a content image (i.e., the first image) and a style image (i.e., the second image) as input, and outputs a UI web image (i.e., the third image) that is structurally similar to the content image and stylistically similar to the style image. This allows control over the style of the output image by controlling the style of the input style image, achieving a one-to-many style mapping relationship and satisfying the requirement of generating multiple style images using a single network model. This reduces costs and meets the personalized needs of different clients. Furthermore, the training loss function of the few-shot learning network model uses a form that adds adversarial loss, cycle consistency loss, and identity loss according to different proportional coefficients. Compared to the loss function of Cycle-GAN, this application's embodiments introduce identity loss to ensure consistency in texture information between the generator's input content image and output image. Thus, training the network model (the first initial model) using this training method significantly improves the quality of the UI web generated by the generator model.

[0072] This application provides a style transfer method, which includes: acquiring a first image and a second image; wherein the first image and the second image have different styles; inputting the first image and the second image into a first model to obtain a third image; wherein the third image is similar in content to the first image and similar in style to the second image; the first model is obtained by training a first initial model based on a first loss function; wherein the first initial model includes an initial generator, the first loss function includes a second loss function, and the second loss function is used to measure the difference between the texture information of the output image of the initial generator and the texture information of the input image. Therefore, by acquiring first and second images with different styles and inputting them into a first model trained with a specific loss function, a third image with content similar to the first image and style similar to the second image can be generated. This method introduces a second loss function to measure the texture difference between the output image and the input image, which improves the style consistency and content fidelity of the generated image, while reducing the dependence on a large amount of training data and improving training efficiency and generation quality. Furthermore, the embodiments of this application can control the style of the output result by controlling the style of the second image, thereby achieving a one-to-many style mapping relationship and realizing the requirement of generating multiple style class images using a single network model (i.e., the first model). This reduces costs and meets the personalized needs of different customers.

[0073] Based on the above embodiments, another embodiment of this application provides a style transfer method, which uses the style of an input personalized UI web image (i.e., a second image) as a reference to perform personalized style transfer on an input ordinary UI web image (i.e., a first image). Figure 5 This is a schematic diagram of the style transfer method proposed in the embodiments of this application. Figure 2 ,like Figure 5 As shown, the following steps may be included: Step 201, constructing a UI web image database; Step 202, designing a FUNIT network (i.e., the first initial model); Step 203, training the FUNIT network (i.e., the first initial model) to transfer image styles through the trained first model.

[0074] It should be noted that, in the embodiments of this application, Figure 6 This is a schematic diagram of the UI web image database structure proposed in the embodiments of this application, such as... Figure 6As shown, various UI web resources that meet users' personalized needs are obtained through image libraries and resource websites, templates and marketplaces, open-source projects, etc., and are all included in the database. This database contains two types of image databases: a personalized UI web database (i.e., the second image database) and a general UI web database (i.e., the first image database). The personalized UI web database is built by collecting user preference UI web data, including color, style, and layout. The general database is constructed by collecting UI web images commonly seen in daily life. The UI web image database provides the dataset foundation for training few-shot learning networks. It is used to collect user preference data, including color, style, and layout. An image style library is established, containing images of various styles, such as abstract, cartoon, and sketch.

[0075] It should be noted that, in the embodiments of this application, few-shot learning employs a generative adversarial network model (i.e., the first model), which consists of a generator and a discriminator. The generator can take a content image (i.e., the first image) and a style image (i.e., the second image) as input, and the output is a UI web image (i.e., the third image) that is structurally similar to the content image and stylistically similar to the style image. Figure 7 This is a schematic diagram of the generator network structure proposed in the embodiments of this application, such as... Figure 7 As shown, the generator can consist of a content encoder (i.e., the first encoder), a style encoder (i.e., the second encoder), and a decoder. The content encoder, which extracts the structural information of the content image, can consist of four 2D convolutional layers (Conv) and two residual blocks (ResBlk); the style encoder, which extracts the style information of the image, consists of five 2D convolutional layers (Conv) and one average pooling layer (Avg Pooling); the decoder accepts two inputs: content code (i.e., the first image feature) and style code (i.e., the second image feature). The content code is processed through two adaptive instance normalization residual blocks (AdaIN ResBlk), followed by four upsampling convolutional layers (Conv). The style code is merged into the content code through a series of fully connected layers (FC-256) performing an affine transformation of the style features. The affine transformation operates globally on the content image, thus preserving its structure.

[0076] It should be noted that, in the embodiments of this application, the discriminator can consist of five two-dimensional convolutional layers and one fully connected layer. These five convolutional layers are used to extract features from the input image (i.e., the third image). The last fully connected layer converts the outputs of the five convolutional layers into a one-dimensional feature vector output, thus easily achieving the purpose of discrimination.

[0077] It should be noted that, in the embodiments of this application, the training of the few-shot learning network model (i.e., the first initial model) may include the following: The few-shot learning network model is trained using deep learning techniques, utilizing a pre-constructed personalized UI web image library and ordinary UI web images. First, the pre-constructed personalized UI web database is classified as class A, and the ordinary UI web database is classified as class B. Then, the few-shot learning network model is trained using these two types of UI web images. After training, the generator network model takes a style-class image and a content-class image as input, and outputs an image with a structure similar to the content image and a style similar to the style image. Therefore, two generator network models can be obtained after training: G(F)B2A and G(F)A2B. G(F)B2A can convert ordinary style UI web images into personalized UI web images with a style similar to the input style image, and G(F)A2B can convert personalized UI web images into ordinary UI web images with a style similar to the input style image. The network training process is the process of optimizing the connection parameters between layers in the network, including the weights and biases between adjacent layers. The Adaptive Moment Estimation Optimization Algorithm (Adam) optimizer was used as the main optimizer for optimizing network parameters. During training, the training epoch, batch size, and learning rate (LR) were set to 100, 16, and 0.0002, respectively. The network model could be trained using the PyTorch 1.5.1 framework and accelerated using a 2080Ti. The loss function improved the generation quality of the generator. The total loss of the generator (i.e., the first loss function) can be expressed as Equation (1) above. The adversarial loss (i.e., the third loss function) enables the discriminator to distinguish between the real image and the generator's output image, and allows the generator to deceive the discriminator by generating images that are close to the real image. The adversarial loss can be described as Equation (2) above. Cycle consistency loss (i.e., the fourth loss function) is used to ensure the integrity of the content image structure information, which will ensure that the generator's output image has the same structure as the content image. Cycle consistency loss can be described as Equation (3) above. The identity loss (i.e., the second loss function) is used to ensure that the texture information of the generator's input content image and output image is consistent. The identity loss can be described by the above formula (4).

[0078] It should be noted that, in the embodiments of this application, in order to obtain a high-quality UI web interface with the target style, the embodiments of this application propose a dual-input network architecture and a network model training method incorporating identity loss. The proposed new network model and training method greatly improve the quality and imaging efficiency of the generated style images, meeting the personalized needs of customers.

[0079] In summary, this application's embodiments design a few-shot learning network model (i.e., the first model). The few-shot learning employs a generative adversarial network (GAN) model, consisting of a generator and a discriminator. The generator takes a content image (i.e., the first image) and a style image (i.e., the second image) as input, and outputs a UI web image (i.e., the third image) that is structurally similar to the content image and stylistically similar to the style image. This allows control over the style of the output image by controlling the style of the input style image, achieving a one-to-many style mapping relationship and satisfying the requirement of generating multiple style images using a single network model. This reduces costs and meets the personalized needs of different clients. Furthermore, the training loss function of the few-shot learning network model uses a form that adds adversarial loss, cycle consistency loss, and identity loss according to different proportional coefficients. Compared to the loss function of Cycle-GAN, this application's embodiments introduce identity loss to ensure consistency in texture information between the generator's input content image and output image. Thus, training the network model (the first initial model) using this training method significantly improves the quality of the UI web generated by the generator model.

[0080] This application provides a style transfer method, which includes: acquiring a first image and a second image; wherein the first image and the second image have different styles; inputting the first image and the second image into a first model to obtain a third image; wherein the third image is similar in content to the first image and similar in style to the second image; the first model is obtained by training a first initial model based on a first loss function; wherein the first initial model includes an initial generator, the first loss function includes a second loss function, and the second loss function is used to measure the difference between the texture information of the output image of the initial generator and the texture information of the input image. Therefore, by acquiring first and second images with different styles and inputting them into a first model trained with a specific loss function, a third image with content similar to the first image and style similar to the second image can be generated. This method introduces a second loss function to measure the texture difference between the output image and the input image, which improves the style consistency and content fidelity of the generated image, while reducing the dependence on a large amount of training data and improving training efficiency and generation quality. Furthermore, the embodiments of this application can control the style of the output result by controlling the style of the second image, thereby achieving a one-to-many style mapping relationship and realizing the requirement of generating multiple style class images using a single network model (i.e., the first model). This reduces costs and meets the personalized needs of different customers.

[0081] Based on the above embodiments, this application provides a style transfer device. Figure 8A schematic diagram of the composition of a style transfer device. Figure 1 ,like Figure 8 As shown, the style transfer device 10 includes: an acquisition unit 11 and an input unit 12; wherein, The acquisition unit 11 is used to acquire a first image and a second image; wherein the first image and the second image have different styles; The input unit 12 is used to input the first image and the second image into the first model to obtain a third image; wherein the third image is similar in content to the first image and similar in style to the second image; the first model is obtained by training a first initial model based on a first loss function; wherein the first initial model includes an initial generator, the first loss function includes a second loss function, and the second loss function is used to measure the difference between the texture information of the output image of the initial generator and the texture information of the input image.

[0082] In the embodiments of this application, further, Figure 9 A schematic diagram of the composition of a style transfer device. Figure 2 ,like Figure 9 As shown, the style transfer device 10 proposed in this application embodiment may further include a processor 13, a memory 14 storing instructions executable by the processor 13, and further, the style transfer device 10 may further include a communication interface 15 and a bus 16 for connecting the processor 13, the memory 14 and the communication interface 15.

[0083] In the embodiments of this application, the processor 13 can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field-Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that for different devices, the electronic device used to implement the above-mentioned processor function can also be other types, and this application embodiment does not specifically limit this. The style transfer device 10 may also include a memory 14, which can be connected to the processor 13. The memory 14 is used to store executable program code, which includes computer operation instructions. The memory 14 may include high-speed RAM memory and may also include non-volatile memory, such as at least two disk drives.

[0084] In embodiments of this application, bus 16 is used to connect communication interface 15, processor 13 and memory 14 and the mutual communication between these devices.

[0085] In embodiments of this application, memory 14 is used to store instructions and data.

[0086] Further, in an embodiment of this application, the processor 13 is configured to acquire a first image and a second image; wherein the first image and the second image have different styles; input the first image and the second image into a first model to obtain a third image; wherein the third image is similar in content to the first image and similar in style to the second image; the first model is obtained by training a first initial model based on a first loss function; wherein the first initial model includes an initial generator, the first loss function includes a second loss function, and the second loss function is used to measure the difference between the texture information of the output image of the initial generator and the texture information of the input image.

[0087] In practical applications, the aforementioned memory 14 can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 13.

[0088] This application provides a style transfer device that acquires a first image and a second image, wherein the first image and the second image have different styles; the first image and the second image are input into a first model to obtain a third image; wherein the third image is similar in content to the first image and similar in style to the second image; the first model is obtained by training a first initial model based on a first loss function; wherein the first initial model includes an initial generator, the first loss function includes a second loss function, and the second loss function is used to measure the difference between the texture information of the output image of the initial generator and the texture information of the input image. Therefore, by acquiring first and second images with different styles and inputting them into a first model trained with a specific loss function, a third image with content similar to the first image and style similar to the second image can be generated. This method introduces a second loss function to measure the texture difference between the output image and the input image, which improves the style consistency and content fidelity of the generated image, while reducing the dependence on a large amount of training data and improving training efficiency and generation quality. Furthermore, the embodiments of this application can control the style of the output result by controlling the style of the second image, thereby achieving a one-to-many style mapping relationship and realizing the requirement of generating multiple style class images using a single network model (i.e., the first model). This reduces costs and meets the personalized needs of different customers.

[0089] This application provides a computer-readable storage medium storing a program that, when executed by a processor, implements the style transfer method described above.

[0090] Specifically, the program instructions corresponding to a style transfer method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the program instructions corresponding to a style transfer method in the storage media are read or executed by an electronic device, the following steps are included: Acquire a first image and a second image; wherein the first image and the second image have different styles; The first image and the second image are input into the first model to obtain a third image; wherein the third image is similar in content to the first image and in style to the second image; The first model is obtained by training a first initial model based on a first loss function; wherein the first initial model includes an initial generator, the first loss function includes a second loss function, and the second loss function is used to measure the difference between the texture information of the output image of the initial generator and the texture information of the input image.

[0091] This application also provides a computer program product, including a computer program that can be executed by the processor 13 of the style transfer device 10 to perform the steps described in any of the foregoing methods.

[0092] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0093] This application is described with reference to schematic and / or block diagrams of implementations of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the schematic and / or block diagrams can be implemented by computer program instructions, and combinations of blocks in the schematic and / or block diagrams can be implemented. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the schematic and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0094] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the implementation flow diagram. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0095] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0096] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A style transfer method, characterized in that, The method includes: Acquire a first image and a second image; wherein the first image and the second image have different styles; The first image and the second image are input into the first model to obtain a third image; wherein the third image is similar in content to the first image and in style to the second image; The first model is obtained by training a first initial model based on a first loss function; wherein the first initial model includes an initial generator, the first loss function includes a second loss function, and the second loss function is used to measure the difference between the texture information of the output image of the initial generator and the texture information of the input image.

2. The method according to claim 1, characterized in that, The first model includes a generator, which includes a first encoder, a second encoder, and a decoder; The step of inputting the first image and the second image into the first model to obtain the third image includes: The first image is input into the first encoder for content feature extraction to obtain the first image features; and the second image is input into the second encoder for style feature extraction to obtain the second image features. The decoder fuses the first image features and the second image features to obtain the third image.

3. The method according to claim 2, characterized in that, The first encoder includes N two-dimensional convolutional layers and M residual blocks, where N and M are positive integers; The step of inputting the first image into the first encoder for content feature extraction to obtain the first image features includes: The first image is sequentially input into the N two-dimensional convolutional layers and M residual blocks, and the first image features are output.

4. The method according to claim 2, characterized in that, The second encoder includes L two-dimensional convolutional layers and S average pooling layers, where L and S are positive integers; The step of inputting the second image into the second encoder for style feature extraction to obtain the second image features includes: The second image is sequentially input into the L two-dimensional convolutional layers and S average pooling layers to output the features of the second image.

5. The method according to claim 2 or 3, characterized in that, The decoder includes P normalized residual blocks, Q upsampled convolutional layers, and O fully connected layers, where P, Q, and O are positive integers; The step of fusing the first image features and the second image features through the decoder to obtain the third image includes: The second image feature is input into the O fully connected layers to obtain affine transformation features; The affine transformation features and the first image features are input into the P normalized residual blocks and the Q upsampled convolutional layers to obtain the third image.

6. The method according to claim 1, characterized in that, The first model also includes a discriminator, which comprises E two-dimensional convolutional layers and fully connected layers, where E is a positive integer; After inputting the first image and the second image into the first model to obtain the third image, the method further includes: The third image is subjected to feature extraction through the E two-dimensional convolutional layers, and the result of feature extraction is input into the fully connected layer to obtain a first feature vector, so as to identify whether the third image is a real image based on the first feature vector.

7. The method according to claim 1, characterized in that, The first model is obtained by training the first initial model based on the first loss function and the first training dataset; wherein, the first initial model further includes an initial discriminator; The first training dataset includes a first image database and a second image database. The first image database includes one or more fourth images, and the second image patch includes one or more fifth images. The fourth images and the fifth images have different styles. The first loss function further includes a third loss function and a fourth loss function. The third loss function is used at least by the initial discriminator to measure the difference between the real image and the output image of the initial generator. The fourth loss function is used to measure the similarity between the content of the output image of the initial generator and the content of the fourth image.

8. A style transfer device, characterized in that, The style transfer device includes: a processor and a memory; wherein... The memory is used to store computer programs that can run on the processor; The processor is configured to perform the method as described in any one of claims 1-7 when running the computer program.

9. A computer-readable storage medium, characterized in that, The storage medium stores computer program code, which, when executed by a computer, performs the method described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method according to any one of claims 1-7.

Images