Image processing method and related device

Abstract

The application provides an image processing method and related equipment, and relates to the technical field of artificial intelligence. The method obtains a style mapping parameter set output by a parameter neural network model based on an example image and a style classification label, and obtains input image content based on a content mapping parameter through an image generation model, and performs style conversion on the input image content based on the style mapping parameter to obtain an output image, so that the style of the output image after style conversion can conform to both a target style and a target style of the example image, thereby improving the style fineness of the output image after style conversion.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image processing method and related equipment. Background Technology

[0002] Different users have different preferences for image styles. To meet the needs of different users, image style transfer has become a research hotspot. When performing style transfer on an image, the content presented in the image should remain unchanged.

[0003] Currently, the general approach to image style transfer is as follows: extract the content features of the input image and the style features of the example image using a model, then input the style features of the example image and the content features of the input image into the model simultaneously, and the model outputs the style-transferred image.

[0004] Existing image style transfer methods can only roughly control the conversion results, and the style of the generated converted images is not very refined. Summary of the Invention

[0005] This application provides an image processing method and related equipment to improve the precision of image style transfer.

[0006] Firstly, an image processing method is provided, including:

[0007] Obtain the example image, the input image for style transfer, and the target style classification label;

[0008] The example image and the target style classification label are input into a parameter neural network model to obtain a style mapping parameter set output by the parameter neural network model; wherein, the style mapping parameter set is used to convert the style of the input image into a fused style, and the fused style represents the image style after fusing the style of the example image and the style indicated by the target style classification label;

[0009] An image generation model obtains the input image content based on content mapping parameters, and outputs an image after style conversion of the input image content based on the style mapping parameters. The parameters of the parameter neural network model and the content mapping parameters are obtained by training with sample input images, sample target style classification labels and sample example images.

[0010] Secondly, an image processing method is provided, including:

[0011] In response to a style conversion input operation on the style conversion interface, a style conversion request is sent to the server, so that the server processes the input image, the example image, and the target style classification label according to any of the methods described in the first aspect to obtain a style-converted output image; wherein, the style conversion request includes the input image, the example image, and the target style classification label;

[0012] Obtain and display the style-transformed output image returned by the server.

[0013] Thirdly, an image processing apparatus is provided, comprising:

[0014] The transceiver module is used to obtain the example image, the input image that needs to be style-transformed, and the target style classification label;

[0015] A processing module is configured to input the example image and the target style classification label into a parameter neural network model to obtain a style mapping parameter set output by the parameter neural network model; wherein, the style mapping parameter set is used to convert the style of the input image into a fused style, the fused style representing the image style after fusing the style of the example image and the style indicated by the target style classification label; and,

[0016] This is used to obtain an output image after the image generation model acquires the input image content based on the content mapping parameters and performs style conversion on the input image content based on the style mapping parameters. The parameters of the parameter neural network model and the content mapping parameters are obtained by training with sample input images, sample target style classification labels and sample example images.

[0017] In one possible embodiment, the parametric neural network model includes multiple convolutional units, multiple activation units, and fully connected units, wherein the number of convolutional units and activation units is the same, and the fully connected unit includes multiple sub-learning units. The processing module is specifically used for:

[0018] The example image and the target style classification label are sequentially convolved through each of the multiple convolutional units to obtain the feature map output by each convolutional unit, thus obtaining the first set of feature maps.

[0019] By sequentially passing through each of the multiple activation units, the feature maps in the first set of feature maps output by the corresponding connected convolutional units are activated to obtain the feature maps output by each activation unit, thus obtaining the second set of feature maps.

[0020] Determine the mean and standard deviation of all pixels in each feature map of the second set of feature maps, obtain multiple means and multiple standard deviations, and concatenate the multiple means and multiple standard deviations to obtain the style feature vector;

[0021] The style feature vector is post-processed through the multiple learning sub-units to obtain a style mapping parameter set; wherein the post-processing includes weighted averaging and activation processing.

[0022] In one possible embodiment, when there are multiple example images, or when there are multiple target style classification labels, the processing module is specifically used for:

[0023] For each example image and its corresponding style classification label, an intermediate style mapping parameter set is obtained for each combination. The intermediate style mapping parameter set is obtained by the parameter neural network model through convolution processing, activation processing, mean and standard deviation determination processing, concatenation of the determined mean and standard deviation, and post-processing. The post-processing includes weighted averaging and activation processing.

[0024] Based on preset weights, each intermediate style mapping parameter set is weighted to obtain the style mapping parameter set.

[0025] In one possible embodiment, when there are multiple example images, or when there are multiple target style classification labels, the parametric neural network model includes fully connected units, each of which includes multiple sub-learning units. The processing module is specifically used for:

[0026] For each example image and its corresponding style classification label, an intermediate style feature vector is obtained for each combination. The intermediate style feature vector is obtained by performing sequential convolution processing, activation processing, and mean and standard deviation on an example image and its corresponding target style classification label through the parametric neural network model, and then concatenating the determined mean and standard deviation.

[0027] The intermediate style feature vectors are weighted according to the preset weights;

[0028] The weighted intermediate style feature vectors are post-processed through multiple learning sub-units in the fully connected unit to obtain a style mapping parameter set; wherein the post-processing includes weighted averaging and activation processing.

[0029] In one possible embodiment, the image generation model includes an encoding unit, a residual unit, and a decoding unit. Both the residual unit and the encoding unit include a non-shared layer and a shared layer. The parameters of the encoding unit are set according to the content mapping parameter set, the parameters of the shared layer are set according to the content mapping parameter set, and the parameters of the non-shared layer are set according to the style mapping parameter set. The processing module is specifically used for:

[0030] The input image is convolved by the encoding unit to obtain the input image content;

[0031] The input image content is convolved using the residual unit to obtain a fused convolutional feature map;

[0032] The decoding unit performs upsampling on the fused convolutional feature map to obtain the style-converted output image.

[0033] In one possible embodiment, the non-shared layers and shared layers in the residual unit are spaced apart, and the non-shared layers and shared layers in the decoding unit are also spaced apart.

[0034] In one possible embodiment, the parametric neural network includes grouped fully connected layers, the number of which is the same as the total number of non-shared layers in the image generation model.

[0035] In one possible embodiment, the target style classification label is a one-hot encoding of the target style classification.

[0036] In one possible embodiment, the parameters of the parametric neural network module and the content mapping parameters of the image generation module are obtained by training with multiple sample input images, multiple sample style classification labels of sample target domains, and multiple sample example images until the image style transfer loss satisfies the target loss.

[0037] The image style transfer loss is a weighted sum of the content loss of the input image and the output image after style transfer, the generation loss between the style classification label of the input image and the target style classification label of the input image, and the style loss between the output image after style transfer and the sample image.

[0038] Fourthly, a terminal device is provided, comprising:

[0039] The transceiver module is used to respond to a style conversion input operation on the style conversion interface, send a style conversion request to the server, so that the server processes the input image, the example image, and the target style classification label according to any of the methods described in the first aspect to obtain a style-converted output image, and to obtain the style-converted output image returned by the server; wherein, the style conversion request includes the input image, the example image, and the target style classification label;

[0040] The display module is used to display the output image after the style conversion.

[0041] Fifthly, an image processing apparatus is provided, comprising:

[0042] At least one processor, and

[0043] A memory that is communicatively connected to the at least one processor;

[0044] The memory stores instructions executable by the at least one processor, which implements the method as described in any one of the first or second aspects by executing the instructions stored in the memory.

[0045] A sixth aspect is a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the method as described in any one of the first or second aspects.

[0046] Since the embodiments of this application adopt the above-described technical solution, they have at least the following technical effects:

[0047] This application embodiment uses a parametric neural network model to process example images and target style classification labels, extracting a style mapping parameter set. This style mapping parameter set is then used as part of the parameters of an image generation model. The image generation model uses content mapping parameters to extract the content of the input image, and then uses the style mapping parameters to map this content, obtaining a style-transformed output image. Because it combines example images and target style classification labels, it not only ensures that the style of the style-transformed output image matches the target style, but also allows for precise control of the input image's style using the example image, achieving more refined image style transformation. Furthermore, since the parametric neural network can process different example images and target style classification labels in real time, this method is applicable to style feature extraction from any example image and target style classification label, improving its versatility. Also, because some parameters are generated in real time, the amount of parameter training can be relatively reduced. Moreover, since corresponding style mapping parameter sets can be generated for different example images and different target styles, it is not necessary to train a separate model for each style; the model can perform transformation processing for all styles, thus relatively reducing the memory space occupied by the model. Furthermore, compared to using a single model to achieve image style transfer, the model in this embodiment includes two parts: a parametric neural network model and an image generation model. These two parts can run relatively independently, thereby improving the running speed of the model. Attached Figure Description

[0048] Figure 1 This is an example diagram of an existing image processing method;

[0049] Figure 2 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0050] Figure 3 This is a schematic diagram illustrating an application scenario of an image processing method provided in an embodiment of this application;

[0051] Figure 4 A schematic diagram illustrating the principle of an image processing method provided in an embodiment of this application;

[0052] Figure 5 A schematic diagram of the architecture of an image processing model provided in an embodiment of this application;

[0053] Figure 6 An interactive schematic diagram of an image processing method provided in an embodiment of this application;

[0054] Figure 7A A schematic diagram of a style switching interface provided in an embodiment of this application. Figure 1 ;

[0055] Figure 7B A schematic diagram of a style switching interface provided in an embodiment of this application. Figure 2 ;

[0056] Figure 8 A schematic diagram of the output image obtained by processing under different weights for the example images provided in the embodiments of this application;

[0057] Figure 9 Schematic diagrams of the output images obtained after processing under different example images provided in the embodiments of this application;

[0058] Figure 10 A schematic diagram of the interface for an output image after display style conversion, provided as an embodiment of this application;

[0059] Figure 11 A comparative diagram showing the output images obtained with and without a specified target style, provided for embodiments of this application;

[0060] Figure 12 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0061] Figure 13 A schematic diagram of the structure of a terminal device provided in this application embodiment. Figure 1 ;

[0062] Figure 14 A schematic diagram of the structure of a terminal device provided in this application embodiment. Figure 2 . Detailed Implementation

[0063] To better understand the technical solutions provided in the embodiments of this application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0064] To facilitate a better understanding of the embodiments of this application by those skilled in the art, the following is a description of the proprietary terms involved in this application.

[0065] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0066] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0067] Computer Vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0068] Style refers to the visual attributes of an image, including but not limited to colors, textures, brightness, and lines. For example, a real image obtained by an image acquisition device can be considered a style, while artworks such as oil paintings, sketches, and watercolors can be considered other different styles.

[0069] Input image: refers to the image that needs to be style-transformed.

[0070] Image style transfer refers to changing the style of one image to that of another. Generally, the content of the input image remains unchanged during image style transfer.

[0071] Target style: This refers to the style that the input image needs to be transformed into. For example, if we want to transform an artwork into an oil painting style, then the oil painting style can be considered the target style. The oil painting style can be understood as all images within the oil painting domain possessing the style of an oil painting.

[0072] A domain refers to a set of images that share a specific style. For example, the oil painting domain represents a set of images with an oil painting style. If image A has an oil painting style, then image A belongs to the oil painting domain. An image belonging to a style can also be understood as belonging to the domain corresponding to that style.

[0073] Target domain: refers to the domain corresponding to the target style.

[0074] Target style category label: refers to the style category identifier corresponding to the target style. For example, the target style can be represented by one-hot encoding.

[0075] Example image: This refers to the image to be converted from the input image to the style corresponding to the target style. The style of the example image may be the same as the target style or it may be different from the target style.

[0076] Domain-related content: also known as style features or domain-related features, is used to represent the style features of an image.

[0077] Domain-independent content: also known as content features or domain-independent features, is used to represent the content features of an image. Domain-independent content includes information such as the image's outline and texture.

[0078] Design Concept

[0079] For example, please refer to Figure 1 This represents an example diagram of image processing using existing technology. Figure 1 This refers to transforming a cat into a dog style, and vice versa. Cats and dogs can be considered two different styles. This indicates the extraction of stylistic features from cats. This indicates the extraction of content features from cats. This indicates the extraction of style features of the dog (also known as domain-related content). G represents the extraction of content features of the dog (also known as domain-independent content). x and G y This represents two decoding neural networks.

[0080] The following example illustrates the process of converting dog y into cat x. First, the content features of y and the style features of x are simultaneously input into G. x In the middle, then by G x The output is the transformation result u. The style features of transformation result u are the same as those of x, and the content features of transformation result u are the same as those of y. When the content features of transformation result u and the style features of x are then input into G... x x can be reconstructed from the middle.

[0081] The following example illustrates the process of converting cat x into dog y. First, the content features of x and the style features of y are simultaneously input into G. y In the middle, then by G y The output is the transformation result v. The style features of the transformation result v are the same as those of y, and the content features of the transformation result v are the same as those of x. When the content features of v and the style features of y are then input into G... x y can be reconstructed from the middle.

[0082] The inventors of this application have discovered that the aforementioned image style transfer methods can only perform coarse-grained style feature conversion, resulting in low conversion precision. Furthermore, a separate model needs to be trained for each style category.

[0083] Therefore, the inventors of this application have designed an image processing method. This method first processes the example image and the target style classification label using a parametric neural network to independently extract a style mapping parameter set. This style mapping parameter set is then used as part of the parameters of an image generation model. The image generation model first extracts the content features of the input image, and then uses the style mapping parameters to map these content features to obtain the style-transformed output image. This method not only ensures that the style of the style-transformed output image matches the target style, but also allows for precise control of the input image's style using the example image, achieving more refined image style transformation. Furthermore, since the parametric neural network can process different example images and target style classification labels in real time, it is applicable to style feature extraction from any example image and target style classification label, providing a more universal image style transformation method.

[0084] Application scenario examples

[0085] Please refer to Figure 2 This is a schematic diagram of the structure of an image processing device that performs the image processing method in the embodiments of this application. The image processing device 200 includes one or more input devices 201, one or more processors 202, one or more memories 203, and one or more output devices 204.

[0086] Input device 201 provides an input interface to acquire input images from external devices or users. After acquiring the input image, input device 201 sends it to processor 202. Processor 202 uses program instructions stored in memory 203 to perform a style conversion process on the input image, obtaining a style-converted output image. The style-converted output image is then output through output device 204.

[0087] Input device 201 may include, but is not limited to, one or more of the following: physical keyboard, function keys, trackball, mouse, touchscreen, joystick, etc. Processor 202 may be a central processing unit (CPU) or a digital processing unit, etc. Memory 203 may be volatile memory, such as random-access memory (RAM); memory 203 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 203 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures, accessible by a computer, but is not limited thereto. Memory 203 may be a combination of the above-mentioned memories. Output device 204 includes, for example, a monitor, speaker, and printer, etc.

[0088] In possible embodiments, the image processing device 200 can be a user-end device or a server-end device. The user-end device can be a mobile terminal, fixed terminal, or portable terminal, such as a mobile phone, site, unit, device, multimedia computer, multimedia tablet, internet node, communicator, desktop computer, laptop computer, notebook computer, netbook computer, tablet computer, personal communication system (PCS) device, personal navigation device, personal digital assistant (PDA), audio / video player, digital camera / camcorder, positioning device, television receiver, radio receiver, e-book device, gaming device, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It is also foreseeable that the image processing device 200 can support any type of user-facing interface (e.g., wearable devices). The server-end device can be a server providing various services, a large computing device, etc. A server can be one or more servers. A server can also be a physical server or a virtual server, etc.

[0089] In one possible application scenario, please refer to Figure 3This describes an example application scenario where the image processing device 200 is implemented via a server 320. A user selects a sample image, a target style classification label, and an input image through a client 311 in a terminal device 310. The client 311 generates an image style conversion request. The server 320 performs the aforementioned processing on the sample image, target style classification label, and input image to obtain a style-converted output image, which is then returned to the client 311. The client 311 generally refers to any client capable of image style conversion, such as a camera app or image processing app. This application does not limit the specific type of client 311.

[0090] Working principle

[0091] In this embodiment, it is assumed that the image c(·) and style s(·) are independent of each other. The main principle of style transfer is that for images belonging to different styles, the content features can be extracted by a general model, and the extracted features are mapped according to different styles (example image and target style).

[0092] The style transfer process in this application embodiment can be represented as follows:

[0093]

[0094] Among them, I y Represents an example image, I x Indicates the input image. This represents the output image after style transfer, without a clear example image I. y When the target style is corresponding, the example image I y The corresponding target style may be multiple, where p(s(Ixy)|Iy) represents example images I with multiple target styles. y The mixture, that is

[0095] In determining example image I y The associated target style is S i Then, the above formula (1) can be expressed as:

[0096]

[0097] Based on formula (2), it can be seen that the main process of this application includes the following:

[0098] Please refer to Figure 4 , Figure 4 This describes the processing principle of the image processing device 200. The image processing process includes: style feature extraction part 410 and style mapping part 420.

[0099] For example, Figure 4 The example image in the text contains a building, and the image style is line drawing. Figure 4 The input image contains architecture and has a photographic style. After processing, the output image still contains architecture, but its style has been changed to photographic.

[0100] In this embodiment, the style feature extraction part 410 is implemented through a parametric neural network model, and the style mapping part 420 is implemented through an image generation model. The model in this embodiment is described below.

[0101] Please refer to Figure 5 The network model includes a parametric neural network model 510 and an image generation model 520. The parametric neural network model 510 has a multi-layered structure, which fuses the style of the example image and the style of the target style classification label to obtain a style mapping parameter set, which can also be understood as style features. The image generation model 520 is used to map the input image using the style mapping parameter set to obtain a style-transformed output image.

[0102] Specifically, to ensure that this method is applicable to various style conversions, in this embodiment, the style mapping parameter set is used as the style mapping parameters in the image generation model 520. This allows the parametric neural network model 510 to generate the corresponding style mapping parameters in real time each time an image style conversion is performed. This enables different style features to process their corresponding input images and reduces the number of parameters that the image generation model 520 needs to train in advance. After obtaining the style mapping parameters, the image generation model 520 extracts the content features of the input image and maps these features according to the style mapping parameter set and the content mapping parameter set to obtain the style-converted output image. Even though different target styles share some content features, the image generation model 520 in this embodiment also includes a content mapping parameter set. This content mapping parameter set is shareable across different target styles and can therefore be pre-trained.

[0103] In one possible embodiment, the parametric neural network model 510 includes multiple convolutional units 511, multiple activation units 512, and fully connected layers 513. Each convolutional unit 511 may include one or more convolutional layers. The number of convolutional units 511 is the same as the number of activation units 512, and one convolutional unit 511 is connected to one activation unit 512. The convolutional units 511 perform convolution processing on the example image and the target classification label. Multiple convolutional units can output corresponding feature maps. Each activation unit 512 will sequentially activate the output feature maps. Activation processing is a type of non-linear processing, which can be expressed as relu(h) = max(0, h), where h represents the feature value corresponding to each pixel in the feature map. Then, the mean and standard deviation of each feature map in all the feature maps after activation processing are determined. Multiple standard deviations and multiple means are concatenated to form a style feature vector. Then, multiple sub-learning units (each sub-learning unit can be understood as a neuron) in the fully connected layer 513 perform multiple weighted averages, activation processing, etc., on the style feature vector to obtain style mapping parameters.

[0104] As one example, concatenating multiple means and multiple standard deviations can be done by arranging the multiple means first and the multiple standard deviations second, or by arranging the multiple standard deviations first and the multiple means second, or by arranging them one mean and one standard deviation at a time, thereby obtaining a style feature vector.

[0105] In one possible embodiment, the parametric neural network model 510 can be trained using a VGG19 neural network and a fully connected neural network, or using a VGG16 neural network and a fully connected neural network. For example, the format of a parametric neural network model 510 is shown in Table 1 below.

[0106] Table 1

[0107]

[0108]

[0109] Please refer to Table 1. In Table 1, Conv2d-ReLU and MaxPool2d represent the convolutional layer and max-pooling layer in convolutional unit 511, respectively. Conv2d-ReLU represents the activation layer in activation unit 512. Table 1 uses fully connected unit 513, which includes one fully connected layer and one grouped fully connected layer, as an example. Conv2d, IN, and ReLU represent the convolution, instance normalization, and ReLU activation functions for a two-dimensional image, respectively.

[0110] Specifically, Table 1 uses the VGG16 neural network implementation with convolutional unit 511 and activation unit 512 as an example, and Table 1 also uses the fully connected unit 513 implemented through a fully connected neural network as an example. As shown in Table 1, the parametric neural network model 510 consists of a VGG16 neural network pre-trained on ImageNet and a fully connected neural network. To output the parameters of the generated model based on the example image and the target style classification label, the example image and the target style classification label are first input into the VGG16 neural network. Then, the feature maps output from the relu1_2, relu_2_2, relu3_3, relu4_3 and relu5_1 layers in the VGG16 are fused together. The fused result is then input into the fully connected neural network, which outputs the parameters of the generated model. The feature map fusion method is as follows: calculate the mean and standard deviation (i.e., multiple style feature values) of each feature map in the above five layers, and arrange the calculated mean and standard deviation into a row vector to complete the feature map fusion. The number of elements in the row vector is the number of calculated mean and standard deviation, which is twice the number of feature maps in the above five layers. Because the mean and standard deviation of each feature map are calculated, the total number of mean and standard deviation obtained is twice the number of feature maps.

[0111] A fully connected neural network consists of one fully connected layer and one grouped fully connected layer. The number of groups in the grouped fully connected layer is the same as the number of non-shared layers 524 in the generative model. Each group of the grouped fully connected layer outputs an image that generates the parameters of the corresponding module (non-shared layer 524) in the generative model 520. Specific model details of the fully connected neural network are shown in Table 1. In Table 1, the number of channels, kernel size, stride, and padding represent the number of feature maps output by each convolutional layer of the convolutional neural network, the height and width of the convolutional kernel (the convolutional kernel is a square matrix), the stride of the convolutional kernel, and the number of rows and columns added to the resulting feature map, respectively. N in Table 1... Share and M c Let N represent the number of modules that need to generate parameters (i.e., the number of non-shared layers 524 in the image generation model 520) and the number of parameters in the c-th non-shared layer 524 of the image generation model, respectively, where 1 ≤ c ≤ N. Share .

[0112] In one possible embodiment, the number of non-shared layers 524 in the image generation model 520 is the same as the number of groups of fully connected layers in the parametric neural network model 510. That is, each group in the fully connected layers generates parameters corresponding to one non-shared layer 524. Of course, the parametric neural network model 510 may also not include grouped fully connected layers. In this case, the fully connected layers directly output style mapping parameter sets, which can be sequentially assigned to the corresponding non-shared layers 524 in a pre-specified order.

[0113] In one possible embodiment, please continue to refer to Figure 5 The image generation model 520 includes an encoding unit 521, a residual unit 522, and a decoding unit 523. The encoding unit 521 is used to extract the content features of the input image, the residual unit 522 is used to fuse the content features and style mapping parameters, and the decoding unit 523 is used to decode the output of the residual unit 522 to obtain the style-transformed output image.

[0114] In one possible embodiment, the image generation model 520 includes a non-shared layer 524 and a shared layer 525. The parameters of the non-shared layer 524 are set according to the style mapping parameter set discussed above. That is, the parameters of the non-shared layer 524 do not need to be pre-trained, while the parameters of the shared layer 525 are set through the content mapping parameter set discussed above. The image generation model 520 may include multiple non-shared layers 524. Therefore, some parameters in the style mapping parameter set can be used as parameters for a certain layer. The specific allocation method can be referred to the content discussed above, and will not be repeated here. Similarly, the content mapping parameter set refers to the total number of parameters of all shared layers 525. The parameters of any shared layer 525 may be the same or different. The specific parameters of each shared layer 525 are obtained based on model training.

[0115] In one possible embodiment, the residual unit 522 includes a non-shared layer 524 and a shared layer 525 spaced apart, and the decoding unit also includes a non-shared layer 524 and a shared layer 525 spaced apart. The spaced distribution can be understood as a shared layer 525 positioned between any two non-shared layers 524, and a non-shared layer 524 positioned between any two shared layers 525.

[0116] In this embodiment, the use of an interval distribution with non-shared layers 524 and shared layers 525 reduces network complexity. This is because if all parameters in the generative model needed to be generated by a parametric neural network, the complexity of the fully connected layers would be extremely high. Style mapping is essentially a process of selecting and recombinating features; therefore, some common operations should be performed on different target styles. This necessitates the inclusion of a certain number of shared layers 525, which improves the accuracy of subsequent style transfer results. Furthermore, this interval distribution method is highly effective in practical applications and can compress the storage space required by the parametric neural network.

[0117] The image generation model 520 is illustrated below.

[0118] Table 2

[0119]

[0120] Table 2 illustrates a specific structural diagram of an image generation model, but the specific structure of the image generation model is not actually limited. Between the encoding unit 521 and the decoding unit 523 of the image generation model 520, there are nine residual units 522. Each residual unit 522 contains two Cond2d-IN-ReLU subunits, used to process and compress features extracted from the input image. The decoding unit 523 in Table 2 includes three upsampling subunits, used to reconstruct a transformation result of the same size as the input image. In Table 2, the trainable field True indicates that the corresponding parameters can be trained together with the parametric neural network, and the trainable field False indicates that the corresponding parameters are generated by the parametric neural network. Conv2d, IN, and ReLU represent convolution, instance normalization, and ReLU activation functions for a two-dimensional image, respectively.

[0121] In one possible embodiment, a discriminant model 530 is introduced to facilitate training the parameter neural network model 510 and the image generation model 520. The discriminant model 530 can be a binary classification model used to determine whether the output image after style transfer belongs to the target style category. The discriminant model 530 can also employ a Patch GAN structure, which is beneficial for improving the quality and resolution of the generated images, for example, with a receptive field of 70×70. Unlike the binary classification model 530, which only outputs a binary classification result of 0 or 1 (0 representing a generated image, 1 representing a real image), the discriminant model using the Patch GAN structure outputs a matrix, where each position of the matrix takes a value of 0 or 1, indicating whether each patch of the image is real or generated.

[0122] The following example, using Table 3, illustrates the structure of a discrimination model 530.

[0123] Table 3

[0124]

[0125] As shown in Table 3, the discriminant model 530 includes four Conv2d-IN-LeakyReLU units, one zero-padding layer, and one convolutional layer. In Table 3, the output dimensions represent the length and width of the output feature map for each unit of the discriminant model. The padding size represents the number of rows and columns added to the resulting feature map after the convolution operation; the added rows and columns are all zeros.

[0126] After introducing the model structure involved in this application, the following section provides examples of how to obtain the parameters of the aforementioned parameterized neural network model 510 and the content mapping parameter set in the image generation model 520. The model training process can be divided into three parts: obtaining the sample dataset, constructing the image style transfer loss, and updating the parameters. Each part will be described in turn below.

[0127] Part 1: Obtaining the sample dataset.

[0128] Specifically, two sample target domains (which are equivalent to two different target styles) are selected. Sample example image sets are obtained by sampling from the first sample target domain, and sample input image sets are obtained by sampling from the second sample target domain. The sample style classification labels of the first sample target domain are obtained, and the sample style classification labels of the second sample target domain are generated.

[0129] As one example, the sample style classification label can be generated using one-hot encoding. Although one-hot encoding may not be able to represent the style features corresponding to the sample style classification label, the one-hot encodings corresponding to different sample style labels are different. Therefore, one-hot encoding can reflect the differences between different sample styles to a certain extent, so that when using the parametric neural network model 510, different sample styles can be distinguished based on one-hot encoding.

[0130] As one embodiment, the sample example image can be represented by a first matrix. For example, one or more of the pixel values, brightness, color, etc. of each pixel of the sample example image can be selected to form the first matrix. The one-hot encoding corresponding to the sample style can be used as a row or column of the first matrix, and the matrix after adding the one-hot encoding is input into the parameter neural network model 510.

[0131] Part 2: Constructing the image style transfer loss.

[0132] Image style transfer loss is used to represent the loss during the style transfer process of an image. Image style transfer loss can be obtained by weighting the generation loss, content loss, and style loss. The three types of loss are introduced below.

[0133] Generation loss refers to the generation loss between the style classification label of the input image and the target style classification label of the image.

[0134] Specifically, the generation loss can be calculated by the discriminant model 530. The image generation model 520 aims to generate an image belonging to the target style (which is also the target domain) to fool the discriminant model 530, while the discriminant model 530 aims to distinguish between the generated image and the image that truly belongs to the target domain. The image generation model 520 and the discriminant model 530 continuously improve their respective performance in the process of competing with each other. In the end, the image generation model 520 is able to generate an image that is "indistinguishable from the real one", while the discriminant model 530 cannot distinguish between the generated image and the image that truly belongs to the target domain.

[0135] A specific expression for the generation loss is as follows:

[0136] ζadv:=log D(I x,X)+log(1-D(G(I x ,I y ∈Y),Y))+log D(I y ,Y)+log(1-D(G(I y ,I x ∈X),X)) (3)

[0137] In the above formula (3), ":=" indicates assignment, D represents the discriminant model 530, G represents the image generation model 520, the discriminant model 530 is used to output the binary classification result, and the judgment I y Does it belong to the first sample style? x Whether it belongs to the style corresponding to the input image, Y represents the first target domain. Figure 5 d in y Y represents the style classification label of the first sample, and X represents the second target domain. Figure 5 d in x Let X represent the style classification label of the second sample. During training, the discriminant model 530 continuously adjusts its parameters to maximize the result of formula (3), while the image generation model 520 continuously adjusts its parameters to minimize the generation loss.

[0138] However, the inventors of this application considered that if only the generation loss is used to train the image generation model 520, then there may be no correlation between the style-transformed output image and the input image, and it is impossible to determine whether the content features of the input image are retained in the style-transformed output image. Therefore, the inventors of this application considered to add content loss to maintain the common features between the input image and the style-transformed output image. Thus, content loss is introduced, which is used to represent the content loss between the sample input image and the style-transformed sample output image.

[0139] One form of content loss can be represented using the cycle-consistent loss, which is specifically represented as follows:

[0140] ζcyc:=||Ix'-I x ||1+||Iy'-I y ||1 (4)

[0141] Image I is obtained by sampling from domains X and Y. x and I y I x’ Represented as I x As the sample input image, I y When used as a sample example image, the style-transformed sample output image generated by the image generation model 520 is then used as the input image, and I xAs an example image, the style-transformed sample output image obtained by image generation model 520, I y’ Represented as I y As the sample input image, I x When used as a sample example image, the style-transformed output image generated by the image generation model 520 is then used as the input image, and I y As an example image, the style-transformed sample output image is obtained by image generation model 520. Let I... x’ and I y’ G(G(I) x ,I y ∈Y),I x ∈X) and G(G(I) y ,I x ∈X),I y To ensure that the generated image and the input image have the same content, the cycle consistency loss needs to be calculated for I(∈Y). x with I x’ L1 distance and I y with I y’ The sum of L1 distances. ||·||1 represents the L1 distance.

[0142] As one example, the applicant considers that, in addition to using the cycle consistency loss, the content loss can be calculated using the following formula:

[0143]

[0144] Among them, VGG16 in formula (5) l This represents the output of the preset l-th layer in the parametric neural network model 510. Taking the parametric neural network model 510 as an example, which is the network result shown in Table 1 above, the preset layer can be the l-th layer, and ||·||2 represents the L2 distance.

[0145] The inventors of this application considered that even after introducing generation loss and content loss, the style of the input image after style transformation still cannot be controlled. Therefore, in order to enable users to control the transformation result using a sample image, it is also necessary to measure the style difference between the transformation result and the sample image during the training process. Thus, the inventors of this application also introduced style loss, which is used to represent the style loss between the style-transformed sample output image and the sample sample image.

[0146] In one possible embodiment, the formula for calculating style loss provided in this application is as follows:

[0147]

[0148] Where, NL M represents the number of network layers in a 510 neural network required to calculate style loss. l The parameter represents the number of feature maps output by the preset layer of the neural network 510, where μ and σ represent the mean and standard deviation of the feature maps, respectively.

[0149] One formula for calculating style loss is as follows:

[0150]

[0151] Based on formulas (3), (4) and (6), the image style transfer loss can be expressed as follows:

[0152]

[0153] Here, λ and η are coefficients used to balance different loss function terms.

[0154] As an example, the cycle consistency loss in formula (4) above can be replaced by the result calculated by formula (5). Formula (6) above can be calculated using the result calculated by formula (7).

[0155] As one example, λ can be set to 10, while the value of η can be set to the range of 0.01 to 0.1.

[0156] Part Three: Parameter Update Process.

[0157] The sample example image and the sample style classification label corresponding to the Y domain are input into the parameter neural network model 510. The parameter neural network model 510 performs mapping processing on the sample example image and the sample style classification label corresponding to the Y domain to obtain the first sample style mapping parameters. This mapping processing specifically includes: extracting the first set of sample feature maps of the sample example image and the sample style classification label of the Y domain; determining the mean and standard deviation of each sample feature map in the first set of sample feature maps; concatenating the mean and standard deviation into a style feature vector; and then performing a weighted average and activation processing on the style feature vector for a preset number of times to obtain the first style mapping parameter set. The first style mapping parameter set and the sample input image are input into the image generation module 520. After processing by the image generation module 520, the sample output image after image style conversion is obtained. The sample output image after image style conversion and the sample style classification label of the Y domain are then input into the judgment model 530 to obtain the result of whether the sample output image after image style conversion belongs to the sample style classification label of the X domain.

[0158] Similarly, the sample input image and the sample style classification label of the X target domain are input into the parametric neural network model. The parametric neural network model 510 maps the sample input image and the sample style classification label of the X domain to obtain the second sample style mapping parameters. The sample example image is then input into the generation module 520. After processing by the image generation module 520, the style-transformed sample example image is obtained. The style-transformed sample example image and the sample style classification label of the X domain are then input into the judgment model 530 to obtain the result of whether the style-transformed sample example image belongs to the sample style classification label of the X domain.

[0159] The specific representations of the parameters for each model are explained below:

[0160] The output of the parametric neural network model 510 (which can be understood as the parameters of all non-shared layers 424) is specifically represented as follows:

[0161] θ p :=G p (I y ∈S i ;ψ) (9)

[0162] Among them, I y S represents a sample image. i ψ represents the target domain to which the sample image belongs, and ψ represents the parameters of the parameterized neural network model 510.

[0163] The formal expression of image generation model 520 is as follows:

[0164]

[0165] Where θ represents the trainable parameters in the parametric neural network model 50 and the image generation model 520. I represents the sample output image after image style transfer. x This represents the sample input image.

[0166] Combining formulas (9) and (10) above, formula (10) can be specifically transformed into formula (11) below, which is specifically expressed as follows:

[0167]

[0168] Where, θ p θ represents the trainable parameters in the entire model. s This represents the parameters that are not trainable in the entire model, which are the parameters of the non-shared layer 524.

[0169] The parameter adjustment process is described below:

[0170] The gradients ▽Ψ, ζtotal, and ▽θsζtotal of the neural network and generative model are calculated based on the image style transfer loss.

[0171] The Adam algorithm can be used to perform a gradient descent update, that is:

[0172] Ψ=Ψ-η▽Ψζtotal (12)

[0173] θs=θs-η▽θsζtotal (13)

[0174] The gradient ▽Ψ, descent gradient θd, and ζtotal of the discriminant model 530 are calculated based on the generation loss.

[0175] Perform a gradient descent update, i.e., θd = θd - η▽Ψθdζtotal, until the image transfer style loss meets the preset conditions, and obtain θs, which are the parameters of the parameter neural network model 510, the content mapping parameter set in the image generation model 520, and the parameters in the discriminant model 530.

[0176] In one possible embodiment, the sample example image and the sample input image can be scaled and flipped to expand the sample dataset.

[0177] Example process

[0178] The following is based on Figure 3 As shown in the application scenario, this application provides an image processing method. Please refer to... Figure 6 The image processing method includes:

[0179] S610, client 311 responds to the style conversion input operation on the style conversion interface and generates a style conversion request.

[0180] Specifically, when a user needs to perform style conversion on an image, they can select a corresponding example image and target style information. The client 311 can pre-store example images of the corresponding style, or the user can define their own example images. The client 311 can also recommend example images to the user based on their previous preferences during image style conversions. Similarly, the client 311 can pre-store multiple target style information sets, or the user can define their own target style information. The client 311 can also recommend target style information to the user based on their previous preferences during image style conversions.

[0181] As one embodiment, the target style information can be a target style category or a target style category label, etc. If it is target style information, after obtaining the target style category, the client 311 can generate a target style category label corresponding to the target style category based on the target style category, for example, by using one-hot encoding to generate the target style category label corresponding to the target style.

[0182] As one embodiment, a user can select one or more example images, or one or more target style information. This application does not limit the number of example images or target style information. When a user selects multiple example images or multiple target styles, the user can set weights for the multiple example images or multiple target styles, so that the user can set the weight of each example image or each target style in the converted image. Alternatively, if the user does not set the weight of each example image or each target style in the converted image, the client 311 can set corresponding weights by default, such as equal weights for each example image or each target style.

[0183] After selecting a sample image and a target style, the user performs a style conversion input operation, such as manually clicking to confirm, or confirming via voice or gesture. Client 311 responds to this operation and generates a style conversion request based on the input image, sample image, and target style category label. This style conversion request requests server 320 to perform image style conversion and includes the input image, sample image, and target style category label.

[0184] As one embodiment, the example image included in the style transfer request can be a specific image or index information corresponding to the example image. For example, the server 320 may have various example images and index information pre-stored. The client 311 needs to send the index information of the example image to the server 320. In this way, the network transmission burden can be reduced.

[0185] For example, please refer to Figure 7A This represents a schematic diagram of a style conversion interface. The user can select an example image 1 on the style conversion interface. The image content of example image 1 includes mountains, water and sunset. The style of example image 1 is photo style. The user can also select a photo style (target style category). After selection, the user clicks the confirmation operation control 730, which is equivalent to the client 311 responding to the image style conversion input operation and generating a style conversion request. Figure 7A The input image style is artistic. Figure 7A The image elements of the input image include a path, blue sky and white clouds, the sea and mountains.

[0186] Continue with Figure 7A For example, after selecting example image 1, the user can click the "Continue Selection" control 720, displaying the interface shown in 7B. The user can then select example image 2 and target style information. The user can also click the "Set Weight" control 740 to set the weights of the selected example images 1 and 2. A higher weight indicates that the user wants the output image to have a style more similar to the selected example image. Only after the user clicks the "Confirm" control 730 is the client 311 considered to have received the image style conversion input information. Example image 2 contains images of the sea, blue sky, and rocks, and its image style is photographic.

[0187] S620, client 311 sends a style conversion request to server 320.

[0188] Specifically, after client 311 generates a style conversion request, it can send the style conversion request to server 320, which means that server 320 has received the style conversion request.

[0189] S630, server 320 obtains the example image, input image, and target style classification label.

[0190] Specifically, after receiving the image style conversion request, server 320 parses the style conversion request to obtain the example image, the target style classification label, and the input image.

[0191] S640, server 320 inputs the example image and target style classification label into parameter neural network model 510 to obtain style mapping parameter set.

[0192] Specifically, after obtaining the example image, the input image, and the target style classification label, the server 320 can input the example image and the target style classification label into the parametric neural network model 510 discussed above, and obtain the style mapping parameter set through the model 510.

[0193] Specifically, server 320 performs multiple convolutional processes on the example image and target style classification labels using parametric neural network model 510. This is achieved through multiple convolutional units 511 within the parametric neural network model 510, each of which may include one or more convolutional layers. Each convolutional unit 511 outputs a corresponding feature map. This set of feature maps can be considered the first set of feature maps, or a selection of feature maps output by some convolutional units can be used as the first set of feature maps. The first set of feature maps may include multiple feature maps.

[0194] The activation units 512 in the multiple residual units of the parametric neural network model 510 are used to activate the corresponding feature maps in the first set of feature maps to obtain the second set of feature maps.

[0195] Specifically, a residual unit is connected to a convolutional unit. The feature map output by the convolutional unit is activated by the residual unit, such as by max pooling, to obtain the feature map output by the residual unit. After each residual unit processes its corresponding feature map, a second set of feature maps is obtained.

[0196] The specific example of obtaining the style mapping parameter set based on the second set of feature maps is as follows:

[0197] Method 1:

[0198] After obtaining the second set of feature maps, the feature maps in the second set of feature maps can be post-processed by each sub-learning unit (which can also be understood as a neuron) in the fully connected unit of the parametric neural network model 510, that is, weighted averaging and activation processing, to obtain the style mapping parameter set.

[0199] Method 2:

[0200] After obtaining the second set of feature maps, we can first determine the mean and standard deviation of the pixel values ​​of all pixel units in each feature map of the second set of feature maps. Since there are multiple feature maps in the second set of feature maps, we can obtain the mean and standard deviation corresponding to each feature map, resulting in multiple means and standard deviations. We can then concatenate these multiple means and standard deviations to obtain the style feature vector. Next, through multiple learning sub-units in the fully connected unit, we can perform weighted averaging and activation processing on the style feature vector to obtain the style mapping parameter set. The concatenation order can be: all means first, then standard deviations; one mean followed by one standard deviation; or all standard deviations first, then all means.

[0201] The preceding text discussed how to obtain the style mapping parameter set when there is only one example image and one target style classification label. The following text provides examples of how to obtain the style mapping parameter set when there are multiple example images or multiple target style classification labels.

[0202] Method 3:

[0203] The intermediate style mapping parameter sets corresponding to each example image are weighted to obtain the style mapping parameter set.

[0204] Specifically, when a user selects multiple example images, each with the same target style category label, the server 320 can combine each example image and its corresponding target style category label to form one combination, thus obtaining multiple combinations. When a user selects one example image, each with a different target style category label, the server 320 can combine that example image with each target style category label to form one combination, thus obtaining multiple combinations. When a user selects multiple example images with different target styles, the server 320 can combine each example image with its corresponding target style label to form one combination, thus obtaining multiple combinations. In any case, the server 320 can obtain multiple combinations.

[0205] Server 320 can input each combination into the parametric neural network model 510, which performs convolution, activation, and post-processing on the combination to obtain the intermediate style mapping parameter set corresponding to that combination. After processing multiple combinations, multiple intermediate style mapping parameter sets can be obtained. Server 320 weights these multiple intermediate style mapping parameter sets according to preset weights to obtain the style mapping parameter set. The convolution, activation, and post-processing are described above and will not be repeated here.

[0206] Specifically, the intermediate style mapping parameter sets corresponding to each example image contain the same number of style mapping parameters and the same element positions in the corresponding sets. Therefore, when weighting multiple intermediate style mapping parameter sets, the style mapping parameters at the corresponding positions are weighted, so that the number of intermediate style mapping parameter sets and the final style mapping parameter set are the same.

[0207] In this embodiment of the application, when there are multiple example images, the style mapping parameter sets corresponding to each example image are directly weighted to determine the style mapping parameter sets of multiple example images in a simple and direct way.

[0208] Method 4:

[0209] Weight the intermediate style feature vectors corresponding to each example image;

[0210] By using multiple learning sub-units in the parametric neural network model 510, the weighted intermediate style feature vectors are post-processed to obtain the style mapping parameter set.

[0211] Specifically, the combination method of the target style classification label and the example image can be referred to the discussion of Method 3 above, and will not be repeated here. Server 320 can use multiple convolutional units 511 in the parametric neural network model 510 to perform convolution processing on an example image and its corresponding target style classification, obtaining the first set of feature maps corresponding to the example image. Then, multiple activation units 512 are used to activate each feature map to obtain the second set of feature maps corresponding to the example image. The mean and standard deviation of each feature map in the second set of feature maps are determined, obtaining multiple means and multiple standard deviations. These multiple means and multiple standard deviations are concatenated to obtain the intermediate style feature vector corresponding to the example image. This process is repeated to obtain the intermediate feature vectors corresponding to each combination (i.e., the combination of an example image and a target style classification label). The values ​​at corresponding positions in the intermediate feature vectors corresponding to each example image are post-processed by the fully connected unit 513 to obtain the style mapping parameter set.

[0212] The following examples illustrate the output images obtained from the processing steps of Methods 3 and 4. For example, please refer to... Figure 8 This represents the style transfer results of the input image (exemplar1) and the example image (exemplar2) under different weights. For example... Figure 8 As shown, the larger the weight corresponding to example image 1, the more the style of the generated output image tends to be similar to that of example image 1; similarly, the larger the weight corresponding to example image 2, the more the style of the generated output image tends to be similar to that of example image 2.

[0213] in, Figure 8 The first row of the input image contains city buildings, roads, and vehicles, and the input image style is photograph. Figure 8 The first example image in the first row contains a blue sky and rocks, and its style is artistic. Figure 8 Example in the first line Figure 2 The image content includes airplanes and the sky, as shown in the example in the first row. Figure 2 The image style is cartoonish. Figure 8 The second row of the input image contains a handbag, and the image style is a black and white line drawing. Figure 8 The first example image in the second row depicts a handbag. It has a photo-style image and a darker color tone. Figure 8 Example in the second line Figure 2 The image content is a handbag, the example in the second row. Figure 2 The image style is photorealistic, and the colors are yellowish.

[0214] S650: Obtain the input image content and output the image after style conversion of the input image content based on style mapping parameters.

[0215] Specifically, the image generation model 520 has already obtained content mapping parameters through the training process. Therefore, the input image content corresponding to the input image can be obtained through the encoding unit 521 in the image generation model 520. The parameters of the encoding unit 521 are determined based on the content mapping parameters. Then, the residual unit 522 and the decoding unit 523 in the image generation model 520 perform style transformation on the input image content through the style mapping parameters to obtain the style-transformed output image.

[0216] The input image is convolved by the encoding unit 521 to obtain the input image content. The input image content is then convolved to obtain a fused convolutional feature map. The fused convolutional feature map is upsampled by the decoding unit 524 to obtain the style-transformed output image.

[0217] For example, please refer to Figure 9 This represents a schematic diagram of different example images performing style conversion on different input images, and a schematic diagram of obtaining the style-converted output image. Figure 9 In B1, A1-A8 represent different input images. The small image in the lower left corner of B1 represents the example image. The portion of B1 excluding the lower left corner represents the transformation result of the example image shown in A1 based on the small image in the lower left corner of B1. Similarly, B2 represents the result of transforming A1 using the example image shown in the lower left corner of B2. C1 and C2 represent the results of style transformation of A2 using two different example images. D1 and D2 represent the results of style transformation of A3 using two different example images, and E1 and E2 represent the results of style transformation of A4 using two different example images.

[0218] Continue to refer to Figure 9 F1-F8 represent different example images, and G1-G8 represent different output images after style conversion. G1 represents the output image after style conversion of A5 using F1 as the example image, G2 represents the output image after style conversion of A5 using F2 as the example image, and so on. Figure 9 The target style classification label for each transformation uses the target style classification to which the example image belongs.

[0219] in, Figure 9Input image A1 includes the sea, blue sky and white clouds, and a path, with an artistic style. The example image in the lower left corner of B1 includes a sunset, mountains, and water, with colors leaning towards sunset hues. The style-transformed output image in B1 maintains the same content as input image A1, but its style is more similar to the example image in the lower left corner of B1. The example image in the lower left corner of B2 includes rocks and the sea. The style-transformed output image in B2 maintains the same content as input image A1, and its style is consistent with the example image.

[0220] Figure 9 The input image A2 contains a path, trees, and a house. Figure 9 The input image A2 has a retro style. Figure 9 The example image in the lower left corner of C1 includes the sky, sunset, and trees. The image style of this example image is dark green. The image content of the style-transformed output image in C1 is consistent with that in A2, and the image style is consistent with that of the example image in the lower left corner of C1.

[0221] Figure 9 The input image A3 contains a path, trees, mountains, and a sunset. Figure 9 The input image A3 has a sunset style. Figure 9 The example image in the lower left corner of D1 includes blue sky, white clouds, and trees. The image style of this example image is cool-toned. The image content of the style-transformed output image in D1 is consistent with the image content of A3, and the image style is consistent with the example image in the lower left corner of D1. Figure 9 The example image in the lower left corner of D2 includes an airplane and the sky. The image style of this example image is cartoonish. The image content of the style-transformed output image in D2 is consistent with the image content of A3, and the image style is consistent with the example image in the lower left corner of D2.

[0222] Figure 9 The input image A4 contains a bridge, a sunset, and the sky. Figure 9 The input image A4 has a twilight style. Figure 9 The example image in the lower left corner of E1 includes trees. The image style of this example image is fresh and clean. The image content of the style-transformed output image in E1 is consistent with the image content of A4, and the image style is consistent with the example image in the lower left corner of E1. Figure 9The example image in the lower left corner of E2 includes a person and a red background. The image style of this example image is cartoonish red. The image content of the style-transformed output image in E2 is consistent with the image content of A4, and the image style is consistent with the example image in the lower left corner of E2.

[0223] Figure 9 The input image A5 is a pair of sneakers with a black and white line drawing style. The example image F1 is a pair of boots with a yellow style. The output image after style conversion between the example image F1 and the input image A5 is G1. The image content of G1 is the same as A5, and the image style of G1 is consistent with that of A5. The image content of the example image F2 is a pair of mid-calf boots with a black style. The style conversion result of the input image A5 and the example image F2 is G2. The image content of G2 is consistent with that of A5, and the image style of G2 is consistent with that of F2.

[0224] Figure 9 The input image A6 is a handbag with a black and white line drawing style. Example image F3 also contains a handbag with a red and white background. The output image after style conversion between example image F3 and input image A6 is G3. G3 has the same image content as A6, and its style is consistent with A6. Example image F4 also contains a handbag with an ethnic style. The style conversion result between input image A6 and example image F4 is G4. G4 has the same image content as A6, and its style is consistent with F4.

[0225] Figure 9 The input image A7 is a building with a line drawing style. The example image F5 is a building with a photographic style. The output image after style conversion between the example image F5 and the input image A7 is G5. The image content of G5 is the same as A7, and the image style of G5 is consistent with that of A7. The image content of the example image F6 is a building with a dark tone style. The style conversion result of the input image A7 and the example image F6 is G6. The image content of G6 is consistent with that of A7, and the image style of G6 is consistent with that of F6.

[0226] Figure 9 The input image A8 is a building with a photographic style. The example image F7 is a building with a line art style. The output image after style conversion between the example image F7 and the input image A8 is G7. The image content of G7 is the same as A8, and the image style of G7 is consistent with that of A8. The image content of the example image F8 is a building with a line art style. The style conversion result of the input image A8 and the example image F8 is G8. The image content of G8 is consistent with that of A8, and the image style of G8 is consistent with that of F8.

[0227] As an example, the contents of the parametric neural network model 510 and the image generation model 520 can be referred to the above discussion, and will not be repeated here.

[0228] As one embodiment, the parametric neural network model 510 and the image generation model 520 can be obtained by the server 320 through the model training process described above, or they can be trained by other image processing devices and directly applied to the server 320.

[0229] S660, server 320 sends the style-transformed output image to client 311.

[0230] S670, Client 311 displays the output image after style conversion.

[0231] Specifically, after receiving the style-transformed output image from the server 320, the client 311 displays the style-transformed output image. To facilitate user viewing of the effect, the example image and the style-transformed output image can be compared when displaying the style-transformed output image.

[0232] As one example, the user can perform style conversion on the output image after style conversion. The client 311 and the server 320 can repeat the above process to continue style conversion on the output image after style conversion.

[0233] As one embodiment, the client 311 can save the original input image, and the user can cancel the style conversion operation. The client 311 will then display the style-converted output image as the input image.

[0234] For example, by user selection Figure 7A Using the example image 1 and the target style as shown, client 311 obtains the output image of the style conversion and displays it as shown. Figure 10 The image shown is the output image after style conversion. Users can click... Figure 10 The cancel operation control 1001 in the middle cancels the style conversion of the input image. You can also click it. Figure 10 Continue to use the conversion operation control 1002 to perform style conversion on the output image after style conversion. Figure 10 The small image in the bottom left corner is an example image. Figure 10 The style of the output image after style conversion is consistent with that of the example image in the lower left corner, and the image content is consistent with that of the input image.

[0235] In one possible embodiment, Figure 6The example provided illustrates the interaction between the client 311 and the server 320 in processing images. However, in reality, the image processing device 200 can perform the S630-S650 process on the input image independently. The specific execution of the S630-S650 process can be found in the previous discussion and will not be repeated here.

[0236] Figure 6 In the illustrated embodiment, the input image is style-transformed using example images and target style classification labels, so that the generated output image can simultaneously conform to the target style classification and the style corresponding to the example image, and can improve the style refinement of the output image.

[0237] For example, please refer to Figure 11 This represents a comparison between the output images after style transfer with a specified target style and without a specified target style. Figure 11 In the diagram, A1-A3 represent the input images, B1-B3 represent example images of style transfer performed on A1-A3, C1, C3, and C5 represent the style-transferred output images obtained by server 320 when no target style is specified, and C2, C4, and C6 represent the style-transferred output images obtained by server 320 when a target style is specified. This can be based on... Figure 11 It can be intuitively seen that specifying a target style in the embodiments of this application can result in an output image with higher style conversion accuracy compared to not specifying a target style.

[0238] Figure 11 The input image shown in A1 contains a building, and the example image shown in B1 contains a building with a line style. Style conversion is performed on B1 and A1 without specifying a target domain. The output image after style conversion is shown in C1. Style conversion is performed on B1 and A1 with a specified target domain. The output image after style conversion is shown in C2. It can be seen that compared with C1, the image style of C2 is closer to that of example image B1.

[0239] Figure 11 The input image shown in A2 contains a building, and the example image shown in B2 contains a building. Both images are in the style of a photograph. Style conversion is performed on B2 and A2 without specifying a target domain. The output image after style conversion is shown in C3. Style conversion is performed on B2 and A2 with a specified target domain. The output image after style conversion is shown in C4. It can be seen that the image style of C4 is closer to that of the example image B2 than that of C3.

[0240] Figure 11The input image shown in A3 contains shoes, and the example image shown in B3 contains shoes. The image style is a black and white line drawing. Style conversion is performed on B2 and A2 without specifying a target domain. The output image after style conversion is shown in C5. Style conversion is performed on B3 and A3 with a specified target domain. The output image after style conversion is shown in C6. It can be seen that the image style of C6 is closer to that of the example image B3 than that of C5.

[0241] Based on the same inventive concept, this application also provides an image processing apparatus 1200, which may be installed in the image processing device 200 described above, or it may be installed in the server 320 described above. Please refer to... Figure 12 The device 1200 includes: a transceiver module 1201 and a processing module 1202, wherein:

[0242] The transceiver module 1201 is used to obtain the example image, the input image that needs to be style-transformed, and the target style classification label;

[0243] Processing module 1202 is used to input the example image and the target style classification label into a parameter neural network model to obtain a style mapping parameter set output by the parameter neural network model; wherein, the style mapping parameter set is used to convert the style of the input image into a fused style, the fused style representing the image style after fusing the style of the example image and the style indicated by the target style classification label; and,

[0244] This is used to obtain the output image after the image generation model obtains the input image content based on the content mapping parameters and performs style conversion on the input image content based on the style mapping parameters. The parameters of the parametric neural network model and the content mapping parameters are obtained by training with sample input images, sample target style classification labels and sample example images.

[0245] In one possible embodiment, the parametric neural network model includes multiple convolutional units, multiple activation units, and fully connected units. The number of convolutional units and activation units is the same, and the fully connected units include multiple sub-learning units. The processing module 1202 is specifically used for:

[0246] The example image and the target style classification label are convolved sequentially through each of the multiple convolutional units to obtain the feature map output by each convolutional unit, thus obtaining the first set of feature maps.

[0247] By sequentially passing through each activation unit in multiple activation units, the feature maps in the first set of feature maps output by the corresponding connected convolutional units are activated to obtain the feature maps output by each activation unit, thus obtaining the second set of feature maps.

[0248] Determine the mean and standard deviation of all pixels in each feature map of the second set of feature maps, obtain multiple means and multiple standard deviations, and concatenate the multiple means and multiple standard deviations to obtain the style feature vector;

[0249] The style feature vectors are post-processed through multiple learning sub-units to obtain a style mapping parameter set; the post-processing includes weighted averaging and activation processing.

[0250] In one possible embodiment, when there are multiple example images, or when there are multiple target style classification labels, the processing module 1202 is specifically used for:

[0251] For each example image and its corresponding style classification label, an intermediate style mapping parameter set is obtained. The intermediate style mapping parameter set is obtained by using a parametric neural network model to sequentially process an example image and its corresponding target style classification label through convolution, activation, determination of mean and standard deviation, concatenation of the determined mean and standard deviation, and post-processing. The post-processing includes weighted averaging and activation.

[0252] Based on preset weights, each intermediate style mapping parameter set is weighted to obtain the style mapping parameter set.

[0253] In one possible embodiment, when there are multiple example images, or when there are multiple target style classification labels, the parametric neural network model includes fully connected units, each of which includes multiple sub-learning units. The processing module 1202 is specifically used for:

[0254] For each example image and its corresponding style classification label, an intermediate style feature vector is obtained. The intermediate style feature vector is obtained by performing convolution processing, activation processing, and determining the mean and standard deviation of an example image and its corresponding target style classification label in sequence through a parametric neural network model, and then concatenating the determined mean and standard deviation.

[0255] The intermediate style feature vectors are weighted according to the preset weights;

[0256] By using multiple learning sub-units in the fully connected unit, the weighted intermediate style feature vectors are post-processed to obtain the style mapping parameter set; the post-processing includes weighted averaging and activation processing.

[0257] In one possible embodiment, the image generation model includes an encoding unit, a residual unit, and a decoding unit. Both the residual unit and the encoding unit include a non-shared layer and a shared layer. The parameters of the encoding unit are set according to a content mapping parameter set, the parameters of the shared layer are set according to a content mapping parameter set, and the parameters of the non-shared layer are set according to a style mapping parameter set. The processing module 1202 is specifically used for:

[0258] The input image content is obtained by performing convolution processing on the input image through the encoding unit;

[0259] By using residual units, the input image content is convolved to obtain a fused convolutional feature map;

[0260] The decoding unit upsamples the fused convolutional feature map to obtain the style-converted output image.

[0261] In one possible embodiment, the non-shared layers and shared layers in the residual unit are spaced apart, and the non-shared layers and shared layers in the decoding unit are spaced apart.

[0262] In one possible embodiment, the parametric neural network includes grouped fully connected layers, the number of which is the same as the total number of non-shared layers in the image generation model.

[0263] In one possible embodiment, the target style classification label is a one-hot encoding of the target style classification.

[0264] In one possible embodiment, the parameters of the parametric neural network module and the content mapping parameters of the image generation module are obtained by training with multiple sample input images, multiple sample style classification labels of the target domain, and multiple sample example images until the image style transfer loss satisfies the target loss.

[0265] The image style transfer loss is a weighted sum of the content loss of the input image and the output image after style transfer, the generation loss between the style classification label of the input image and the target style classification label, and the style loss between the output image and the example image after style transfer.

[0266] Based on the same inventive concept, this application also provides a terminal device, please refer to... Figure 13 The terminal device 310 includes:

[0267] The transceiver module 1301 is used to respond to the style conversion input operation on the style conversion interface, send a style conversion request to the server, so that the server processes the input image, the example image and the target style classification label according to any of the methods in the first aspect to obtain the style-converted output image, and to obtain the style-converted output image returned by the server; wherein, the style conversion request includes the input image, the example image and the target style classification label;

[0268] Display module 1302 is used to display the output image after style conversion.

[0269] Figure 13 In addition to the transceiver module 1301 and the display module 1302, the terminal device 310 may also include other necessary components. In this embodiment, no specific restrictions are imposed on other components.

[0270] Based on the same inventive concept, this application also provides an image processing device 200. Please continue to refer to... Figure 2 The image processing device 200 includes a processor 202 and a memory 203. The processor 202 is used to call program instructions stored in the memory 203 to implement any of the image processing methods discussed above, or it can implement the functions of the server 320 discussed above. For more information about the image processing device 200, please refer to the preceding text. Figure 2 The content discussed therein will not be repeated here.

[0271] As one example, Figure 2 The processor 202 in the middle can achieve the above. Figure 12 The functions of the processing module 1202 and the transceiver module 1201.

[0272] Based on the same inventive concept, this application also provides a terminal device 310, please refer to... Figure 14 The terminal device 310 includes a processor 1401 and a memory 1402. The processor 1401 is used to call program instructions stored in the memory 1402 to implement any of the image processing methods discussed above, or to implement the aforementioned methods. Figure 6 The functions corresponding to client 311 discussed in the text.

[0273] As one example, Figure 14 The processor 1401 in the middle can achieve the above. Figure 13 The functions of the transceiver module 1301 and the display module 1302.

[0274] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the image processing method described above.

[0275] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0276] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An image processing method, characterized by, The method comprises the following steps: obtaining an example image, an input image needing to be converted in style, and a target style classification label; inputting the example image and the target style classification label into a parameter neural network model to obtain a style mapping parameter set output by the parameter neural network model; wherein the style mapping parameter set is used to convert the style of the input image into a fusion style, and the fusion style represents an image style after the style of the example image and the style indicated by the target style classification label are fused; obtaining input image content and an output image after the input image content is converted in style based on the style mapping parameter set, wherein the input image content is obtained by an image generation model based on a content mapping parameter set, and the parameters of the parameter neural network model and the content mapping parameter set are obtained by training sample input images, sample target style classification labels, and sample example images; the image generation model comprises an encoding unit, a residual unit, and a decoding unit, the residual unit and the decoding unit each comprise a non-shared layer and a shared layer, the parameters of the encoding unit are set according to the content mapping parameter set, the parameters of the shared layer are set according to the content mapping parameter set, and the parameters of the non-shared layer are set according to the style mapping parameter set; obtaining input image content and an output image after the input image content is converted in style based on the style mapping parameter set comprises: performing convolution processing on the input image by the encoding unit to obtain the input image content; performing convolution processing on the input image content by the residual unit to obtain a fusion convolution feature map; and performing up-sampling processing on the fusion convolution feature map by the decoding unit to obtain an output image after conversion in style.

2. The method of claim 1, wherein, The parameter neural network model comprises a plurality of convolution units, a plurality of activation units, and a fully connected unit, the number of the plurality of convolution units is the same as the number of the plurality of activation units, and the fully connected unit comprises a plurality of sub-learning units; inputting the example image and the style classification label into the parameter neural network model to obtain a style mapping parameter set output by the parameter neural network model comprises: sequentially performing convolution processing on the example image and the target style classification label by each convolution unit in the plurality of convolution units to obtain a feature map output by each convolution unit and obtain a first group of feature maps; sequentially performing activation processing on the feature maps in the first group of feature maps output by the respective corresponding connected convolution units by each activation unit in the plurality of activation units to obtain a feature map output by each activation unit and obtain a second group of feature maps; determining the mean and standard deviation of all pixels in each feature map in the second group of feature maps to obtain a plurality of means and a plurality of standard deviations, and splicing the plurality of means and the plurality of standard deviations to obtain a style feature vector; respectively performing post-processing on the style feature vector by a plurality of learning sub-units to obtain a style mapping parameter set; wherein the post-processing comprises weighted averaging and activation processing.

3. The method of claim 1, wherein, When the example images are multiple, or when the target style classification labels are multiple, the example images and the style classification labels are input into the parameter neural network model, and a style mapping parameter set output by the parameter neural network model is obtained, including: An intermediate style mapping parameter set of each combination is obtained respectively for a combination of an example image and a style classification label corresponding to the example image; wherein the intermediate style mapping parameter set is obtained by sequentially performing convolution processing, activation processing, determining mean and standard deviation processing, splicing the determined mean and standard deviation, and post-processing on an example image and a target style classification label corresponding to the example image by the parameter neural network model, wherein the post-processing includes weighted average and activation processing; The intermediate style mapping parameter sets are weighted according to a preset weight, and a style mapping parameter set is obtained.

4. The method of claim 1, wherein, When the example images are multiple, or when the target style classification labels are multiple, the parameter neural network model includes a full connection unit, the full connection unit includes a plurality of sub-learning units, the example images and the style classification labels are input into the parameter neural network model, and a style mapping parameter set output by the parameter neural network model is obtained, including: An intermediate style feature vector for each combination is obtained respectively for a combination of an example image and a style classification label corresponding to the example image; wherein the intermediate style feature vector is obtained by sequentially performing convolution processing, activation processing, and determining mean and standard deviation on an example image and a target style classification label corresponding to the example image by the parameter neural network model, and splicing the determined mean and standard deviation; The intermediate style feature vectors are weighted according to a preset weight; The weighted intermediate style feature vectors are respectively post-processed by the plurality of learning sub-units in the full connection unit, and a style mapping parameter set is obtained; wherein the post-processing includes weighted average and activation processing.

5. The method of claim 1, wherein, The non-shared layers and the shared layers in the residual unit are distributed at intervals, and the non-shared layers and the shared layers in the decoding unit are distributed at intervals.

6. The method of claim 1, wherein, The parameter neural network includes a grouped full connection layer, and the number of groups of the grouped full connection layer is the same as the total number of non-shared layers included in the image generation model.

7. The method of any one of claims 1-6, wherein, The target style classification label is one-hot encoded.

8. The method of any one of claims 1-6, wherein, The parameters of the parameter neural network module and the content mapping parameter set are obtained by training a plurality of sample input images, a plurality of sample target domain sample style classification labels, and a plurality of sample example images until an image style conversion loss meets a target loss; The image style conversion loss is a weighted result of a content loss of a sample input image and a style converted sample output image, a generation loss between a style classification label of the sample input image and a sample target style classification label, and a style loss between the style converted sample output image and the sample example image.

9. An image processing method characterized by, including: In response to a style conversion input operation on the style conversion interface, a style conversion request is sent to a server to cause the server to process an input image, an example image, and a target style classification label according to any one of the methods of claims 1-8 to obtain a style-converted output image; wherein the style conversion request includes the input image, the example image, and the target style classification label; The style-converted output image returned by the server is obtained and displayed.

10. An image processing apparatus characterized by comprising: Comprise: A transceiver module for obtaining an example image, an input image that needs to be style-converted, and a target style classification label; A processing module for inputting the example image and the target style classification label into a parameter neural network model to obtain a style mapping parameter set output by the parameter neural network model; wherein the style mapping parameter set is used to convert the style of the input image into a fusion style, and the fusion style represents the image style after the fusion of the style of the example image and the style indicated by the target style classification label; and An output image obtained by obtaining input image content and style-converting the input image content based on the style mapping parameter set, wherein the input image content is obtained by an image generation model based on a content mapping parameter set, and the parameters of the parameter neural network model and the content mapping parameter set are obtained by training sample input images, sample target style classification labels, and sample example images. The image generation model includes an encoding unit, a residual unit, and a decoding unit, the residual unit and the decoding unit each include a non-shared layer and a shared layer, the parameters of the encoding unit are set according to the content mapping parameter set, the parameters of the shared layer are set according to the content mapping parameter set, and the parameters of the non-shared layer are set according to the style mapping parameter set. Obtain input image content, and style-convert the input image content based on the style mapping parameter to obtain an output image, comprising: performing convolution processing on the input image through the encoding unit to obtain the input image content; performing convolution processing on the input image content through the residual unit to obtain a fusion convolution feature map; and performing up-sampling processing on the fusion convolution feature map through the decoding unit to obtain an output image converted in style.

11. The apparatus of claim 10, wherein, The parameter neural network model includes a plurality of convolution units, a plurality of activation units, and a fully connected unit, the number of the plurality of convolution units and the plurality of activation units is the same, the fully connected unit includes a plurality of sub-learning units, and the processing module is specifically configured to: Convolve the example image and the target style classification label through each convolution unit in the plurality of convolution units in sequence to obtain a feature map output by each convolution unit to obtain a first group of feature maps; Activate the feature maps in the first group of feature maps output by the respective connected convolution units through each activation unit in the plurality of activation units in sequence to obtain a feature map output by each activation unit to obtain a second group of feature maps; Determine the mean and standard deviation of the pixels of all pixel units in each feature map in the second group of feature maps, obtain a plurality of means and a plurality of standard deviations, and splice the plurality of means and the plurality of standard deviations to obtain a style feature vector; Respectively through a plurality of learning sub-units, post-processing the style feature vector to obtain a style mapping parameter set; wherein the post-processing includes weighted average and activation processing.

12. A terminal device, comprising: Comprise: The transceiver module is used for sending a style conversion request to the server in response to a style conversion input operation on the style conversion interface, so that the server processes the input image, the example image and the target style classification label according to the method in any one of claims 1-8 to obtain the output image after style conversion, and is used for obtaining the output image after style conversion returned by the server; wherein the style conversion request comprises the input image, the example image and the target style classification label; The display module is used for displaying the output image after style conversion.

13. An image processing apparatus characterized by comprising: Comprise: At least one processor, and The memory is connected in communication with the at least one processor; Wherein, the memory stores instructions executable by the at least one processor, and the at least one processor implements the method in any one of claims 1-8 or 9 by executing the instructions stored in the memory.

14. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, when the computer instructions run on the computer, make the computer execute the method in any one of 1-8 or 9. The computer readable storage medium stores computer instructions, when the computer instructions run on the computer, make the computer execute the method in any one of 1-8 or 9.

Images