Image processing method, device, electronic device and storage medium

Abstract

The present disclosure relates to an image processing method, device, electronic device and storage medium, the method comprising: obtaining an image to be processed; segmenting objects in the image to be processed to obtain segmented objects; performing image style conversion processing on the segmented objects according to a selected image style through an image conversion model to obtain a processed target object, wherein the image conversion model is a model obtained by iteratively training by first increasing the number of channels of a deep separable convolutional layer in a hypernetwork and then deleting the channels that do not meet preset conditions after the increase. The present disclosure increases and then decreases the number of channels of a deep separable convolutional layer in a hypernetwork, which not only increases or decreases the network's expressive power under visually acceptable conditions, but also reduces the amount of computational processing within the model, thereby improving the efficiency of image style conversion.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] With the rise of deep learning, deep learning architectures and algorithms have greatly developed computer vision, pattern recognition, and natural language processing. Among them, the application of convolutional neural networks (CNN) in image processing is the most significant.

[0003] In related technologies, deep learning networks are usually used to extract features to assist image style transfer. That is, the content of the input image and the features of the style image are extracted through the CNN network, and the Gram matrix is ​​constructed to define the content loss function and the style loss function. Then, the stylized target image is obtained through optimization. However, in this method, although a stylized target image with good authenticity can be obtained, because the optimization process takes a long time and the amount of calculation is large, it is impossible to achieve fast style transfer, which reduces the efficiency of style transfer and poor user experience. Summary of the invention

[0004] The present disclosure provides an image processing method, device, electronic device and storage medium to at least solve the technical problem in the related art that the image style transfer efficiency is low due to the large amount of image style transfer calculation and long time consumption. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including:

[0006] Get the image to be processed;

[0007] Segmenting the object in the image to be processed to obtain the segmented object;

[0008] The segmented object is subjected to image style conversion processing according to a selected image style through an image conversion model to obtain a processed target object, wherein the image conversion model is a model obtained by iterative training in a manner of first increasing the number of channels of a depth-separable convolutional layer in a hypernetwork and then deleting channels that do not meet preset conditions after the increase.

[0009] Optionally, before performing image style conversion processing on the segmented object according to the selected image style through the image conversion model, the method further includes:

[0010] Generating a hypernetwork based on a depthwise separable convolutional layer of a convolutional neural network, and determining parameters of the hypernetwork;

[0011] A convolutional neural network model is trained according to the determined parameters of the hypernetwork to obtain the image conversion model.

[0012] Optionally, the generating a hypernetwork based on the deep separable convolutional layer of the convolutional neural network and determining the parameters of the hypernetwork include:

[0013] Expanding the number of input channels C of the depthwise separable convolutional layer of the convolutional neural network to an integer multiple of 2 to obtain 2xNxC channels, where N is the number of convolution kernels;

[0014] Split the enlarged depth-separable convolutional layer into convolutional units i, and assign parameter g_i to each convolutional unit i;

[0015] Convolve each enlarged input channel with the corresponding convolution kernel to obtain the corresponding convolution result;

[0016] Multiply each convolution result by the parameter g_i of the corresponding convolution unit i to obtain the convolution parameter of the corresponding channel;

[0017] The convolution parameters of all channels are combined together as the parameters of the hypernetwork.

[0018] Optionally, the convolutional neural network model includes: a first generation network having a depth-separable convolutional layer, a first discriminant network, and a second generation network;

[0019] The step of training a convolutional neural network model according to the determined parameters of the hypernetwork to obtain the image conversion model comprises:

[0020] Selecting a first type of common image training sample and a second type of style image training sample;

[0021] Inputting the first type of common image training samples into the first generation network for generation processing, and outputting the second type of images;

[0022] Inputting the second type of images into the first discriminant network respectively, and obtaining the probability that the second type of images belong to the second type of style image training samples, wherein the probability is called the cross entropy loss;

[0023] Inputting the second type of images into the second generation network respectively, and obtaining an image loss when the second type of images pass through the second generation network is called consistency loss;

[0024] Calculating the sum of the consistency loss and the cross entropy loss, or performing weighted summation on the consistency loss and the cross entropy loss to obtain the sum of the losses;

[0025] Calculate a feature map on the gradient domain according to the sum of the losses;

[0026] Updating parameters of the hypernetwork according to the feature graph;

[0027] The channels corresponding to the updated parameters of the hypernetwork are iteratively trained until an image conversion model is generated.

[0028] Optionally, the iterative training of channels corresponding to the updated parameters of the hypernetwork until an image conversion model is generated includes:

[0029] In response to a parameter of the hypernetwork being less than a set parameter threshold, deleting a channel corresponding to the parameter less than the set parameter threshold;

[0030] In response to a parameter of the hypernetwork being not less than a set parameter threshold, retaining a channel corresponding to a parameter not less than the set parameter threshold;

[0031] The next iterative training is repeated until an image conversion model is generated.

[0032] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:

[0033] An acquisition module is configured to acquire an image to be processed;

[0034] A segmentation module is configured to segment the object in the image to be processed to obtain the segmented object;

[0035] The style conversion processing module is configured to perform image style conversion processing on the segmented object according to the selected image style through the image conversion model to obtain the processed target object, wherein the image conversion model is a model obtained by iterative training in a manner of first increasing the number of channels of the depth-separable convolutional layer in the hypernetwork and then deleting the channels that do not meet the preset conditions after the increase.

[0036] Optionally, the device further comprises:

[0037] A determination module is configured to generate a hypernetwork based on a deep separable convolutional layer of a convolutional neural network and determine parameters of the hypernetwork before the style conversion processing module performs image style conversion processing on the segmented object according to the selected image style through the image conversion model;

[0038] The training module is configured to execute training of a convolutional neural network model according to the determined parameters of the hypernetwork to obtain the image conversion model.

[0039] Optionally, the determining module includes:

[0040] A channel expansion module is configured to expand the number of input channels C of the depthwise separable convolutional layer of the hypernetwork to an integer multiple of 2 to obtain 2xNxC channels, where N is the number of convolution kernels;

[0041] A splitting module, configured to split the enlarged depth-separable convolutional layer into convolutional units i, and assign a parameter g_i to each convolutional unit i;

[0042] The convolution module is configured to perform convolution processing on each enlarged input channel with the corresponding convolution kernel to obtain a corresponding convolution result;

[0043] The product module is configured to perform multiplication of each convolution result by the parameter g_i of the corresponding convolution unit i to obtain the convolution parameter of the corresponding channel;

[0044] The combination module is configured to perform the convolutional parameters of all channels together as the parameters of the hypernetwork.

[0045] Optionally, the convolutional neural network model includes: a first generation network having a depth-separable convolutional layer, a first discriminant network, and a second generation network;

[0046] The training module includes:

[0047] A selection module is configured to select a first type of common image training sample and a second type of style image training sample;

[0048] A first image processing module is configured to input the first type of common image training samples into the first generation network for generation processing, and output a second type of image;

[0049] A first loss determination module is configured to input the second type of images into the first discriminant network for judgment, and obtain a probability that the second type of images belong to the second type of style image training samples, wherein the probability is called a cross entropy loss;

[0050] A second loss determination module is configured to input the second type of images into the second generation network respectively, and obtain an image loss when the second type of images pass through the second generation network, which is called consistency loss;

[0051] A calculation module is configured to calculate the sum of the consistency loss and the cross entropy loss, or to perform weighted summation on the consistency loss and the cross entropy loss to obtain the sum of the losses;

[0052] A feature map calculation module is configured to calculate a feature map in a gradient domain according to the sum of the losses;

[0053] An updating module, configured to update the parameters of the super network according to the feature graph;

[0054] The iteration module is configured to perform iterative training on the channels corresponding to the updated parameters of the hypernetwork until an image conversion model is generated.

[0055] Optionally, the iteration module includes:

[0056] A channel deletion module is configured to execute, in response to a parameter of the hypernetwork being less than a set parameter threshold, deleting a channel corresponding to a parameter less than the set parameter threshold;

[0057] A channel reservation module is configured to execute, in response to a parameter of the super network being not less than a set parameter threshold, retaining a channel corresponding to a parameter not less than the set parameter threshold;

[0058] The iterative training module is repeatedly configured to perform the next iterative training repeatedly until an image conversion model is generated.

[0059] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0060] processor;

[0061] a memory for storing instructions executable by the processor;

[0062] The processor is configured to execute the instructions to implement the image processing method as described above.

[0063] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the image processing method as described above.

[0064] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the image processing method as described above.

[0065] The technical solution provided by the embodiments of the present disclosure may at least include the following beneficial effects:

[0066] In the present disclosure, the image to be processed is segmented to obtain segmented objects; the segmented objects are subjected to image style conversion processing according to the selected image style through an image conversion model to obtain a processed target object, wherein the image conversion model is a model obtained by iterative training by first increasing the number of channels of the deep separable convolutional layer in the hypernetwork and then deleting the channels that do not meet the preset conditions after the increase. The present disclosure increases and then decreases the number of channels of the deep separable convolutional layer in the hypernetwork, which not only increases or decreases the expression ability of the network under visually acceptable conditions, but also reduces the amount of calculation for internal processing of the model, thereby improving the efficiency of image style conversion and user satisfaction with image conversion.

[0067] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0069] Figure 1 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0070] Figure 2 The figure is a schematic diagram of decomposing a depth-wise separable convolutional layer according to an exemplary embodiment.

[0071] Figure 3 It is a training schematic diagram provided by the present disclosure for converting a first type of image into a second type of image.

[0072] Figure 4 It is a training schematic diagram provided by the present disclosure for converting a second type of image into a first type of image.

[0073] Figure 5 is a block diagram of an image processing device according to an exemplary embodiment.

[0074] Figure 6 is another block diagram of an image processing apparatus according to an exemplary embodiment.

[0075] Figure 7 It is a block diagram of a determination module according to an exemplary embodiment.

[0076] Figure 8 It is a block diagram of a training module according to an exemplary embodiment.

[0077] Fig. 9It is a block diagram of an electronic device according to an exemplary embodiment.

[0078] Fig.10 The invention is a block diagram of a device for image processing according to an exemplary embodiment. DETAILED DESCRIPTION

[0079] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0080] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0081] In order to facilitate the understanding of the present disclosure, the following terms are explained before introducing the present disclosure.

[0082] The present disclosure divides and modifies the structure of the main body of the existing model, converts the channels of the existing model into multi-channel, and the model can be selected. Specifically, the existing convolutional neural network CNN generative models are all based on the traditional CONV2D operation, which is very computationally intensive. The present disclosure proposes to replace the traditional convolutional layer (Convolutional Layer) in the existing convolutional neural network production model with a depthwise separable convolutional layer (DSCL), wherein the replacement method can be a complete replacement, or certain specific convolutional layers in the convolutional neural network can be replaced according to actual needs.

[0083] Figure 1 is a flowchart of an image processing method according to an exemplary embodiment. Figure 1 As shown, the image processing method comprises the following steps:

[0084] In step 101, an image to be processed is obtained.

[0085] In step 102, the object in the image to be processed is segmented to obtain the segmented object;

[0086] In step 103, the segmented object is subjected to image style conversion processing according to the selected image style through an image conversion model to obtain a processed target object, wherein the image conversion model is a model obtained by iterative training in a manner of first increasing the number of channels of the depthwise separable convolutional layer in the hypernetwork and then deleting the channels that do not meet the preset conditions after the increase.

[0087] The image processing method disclosed in the present invention can be applied to terminals, servers, etc., without limitation herein, and its terminal implementation device can be a smart phone, a laptop computer, a tablet computer, and other electronic devices, without limitation herein.

[0088] Combine the following Figure 1 , the specific implementation steps of an image processing method provided by an embodiment of the present disclosure are described in detail.

[0089] First, step 101 is executed to obtain an image to be processed.

[0090] In this step, the image to be processed is the image that needs to be style-converted. The image can be an image selected from an album or a newly-shot image. Usually, the selected image includes objects such as portraits or faces, and of course, can also be animal portraits. This embodiment does not limit this.

[0091] Next, step 102 is executed to segment the object in the image to be processed to obtain the segmented object.

[0092] In this step, face recognition detection is first performed on the image to be processed. If a face image or a head portrait is detected, the detected face image or head portrait is segmented. Usually, the segmentation can be performed according to the size set now, for example, the face or head portrait is segmented into 512*512 image blocks, etc. The process of identifying an image by a face recognition algorithm is already a well-known technology for those skilled in the art, and will not be repeated here.

[0093] In the present disclosure, a variety of segmentation methods can be used for the image, such as an average segmentation method or a fixed segmentation method. Of course, image segmentation software can also be used for segmentation, which is not limited in this embodiment. The average segmentation method can be averagely split according to the average number of rows and columns, that is, the image is divided into several equal parts according to the number of rows and columns; and the fixed segmentation method is to divide the image into pixel sizes of specified image blocks. After the segmentation is completed, the file format of the required image block can be obtained.

[0094] Finally, step 103 is executed, and the segmented object is subjected to image style conversion processing according to the selected image style through the image conversion model to obtain the processed target object, wherein the image conversion model is a model obtained by iteratively training by first increasing the number of channels of the depth-separable convolutional layer in the hypernetwork and then deleting the channels that do not meet the preset conditions after the increase.

[0095] In this step, the segmented object is input into the image conversion model, and the image conversion model performs image style conversion processing according to the selected image style, and outputs a target object of the same size after processing, wherein the target object is a target object with the selected image style and is an image of a different type from the object; wherein the image conversion model is a model obtained by iteratively training the deep separable convolutional layer of the hypernetwork, that is, a model obtained by iteratively training by first increasing the number of channels of the deep separable convolutional layer of the hypernetwork and then deleting the channels that do not meet the preset conditions after the increase. wherein, not meeting the preset conditions may be not meeting the parameter threshold of the hypernetwork, or not meeting the channel parameter threshold of the channel of the deep separable convolutional layer, etc.

[0096] For example, if the current input object is a 512*512 face image and the selected image style is a 512*512 cartoon face, then after the face image is input into the image conversion model for image style conversion processing, the output is a 512*512 cartoon face style.

[0097] It should be noted that the image conversion model in this embodiment is a pre-trained model, and its training process is detailed as follows.

[0098] Furthermore, the method may also include replacing the corresponding object in the image to be processed with the target object obtained after processing to obtain a style image.

[0099] In this step, the target object obtained after the style conversion process in step 103 is replaced with the object in the image to be processed, thereby obtaining a complete style image. In other words, the obtained complete style image has an intrinsic connection with the original image to be processed, but belongs to a different type of style image, for example, a natural face is processed by the image conversion model to generate a baby face, or a natural face is processed by the image conversion model to generate a cartoon face, or a cartoon is processed by the image conversion model to change from a sketch style to an oil painting style, etc.

[0100] In the present disclosure, the acquired image to be processed is segmented to obtain segmented objects, and the obtained objects are subjected to image style conversion processing according to the selected image style through an image conversion model to obtain a processed target object, wherein the image conversion model is a model obtained by iteratively training by first increasing the number of channels of the deep separable convolutional layer in the hypernetwork and then deleting the channels that do not meet the preset conditions after the increase. Furthermore, the target object can also replace the corresponding object in the image to be processed to obtain a complete style image. In other words, the present disclosure not only increases or decreases the expression ability of the network under visually acceptable conditions by first increasing and then decreasing the number of channels of the deep separable convolutional layer in the hypernetwork, but also reduces the amount of calculation for internal processing of the model, thereby improving the efficiency of image style conversion and user satisfaction with image conversion.

[0101] Optionally, in another embodiment, based on the above embodiment, before performing image style conversion processing on the segmented object according to the selected image style through the image conversion model, the method may further include:

[0102] 1) generating a hypernetwork based on a deep separable convolutional layer of a convolutional neural network, and determining parameters of the hypernetwork;

[0103] Specifically, it includes: expanding the number of input channels C of the deep separable convolution layer of the hyperconvolutional neural network to an integer multiple of 2, obtaining 2xNxC channels, where N is the number of convolution kernels; splitting the enlarged deep separable convolution layer into convolution units (each convolution unit is composed of a filter) i, and assigning parameters g_i to each convolution unit i; convolving each enlarged input channel with the corresponding convolution kernel to obtain the corresponding convolution result; multiplying each convolution result by the parameter g_i of the corresponding convolution unit i to obtain the convolution parameter of the corresponding channel, and each convolution parameter is a tensor; combining the convolution parameters of all channels together as the parameters of the hypernetwork. That is, the above process determines the parameters of the hypernetwork as the initial parameters.

[0104] It should be noted that the channels of the depthwise separable convolutional layer in the present disclosure have two different meanings. The first is that for sample images (images are used as training samples), channels refer to color channels, and color channels are used to represent the channels of sample images. The second is the dimension of the output space, for example, the number of output channels in a convolution operation, or the number of convolution kernels in each convolution layer.

[0105] In other words, this step is to define the supernet structure based on the depthwise separable convolutional layer of the convolutional neural network. Specifically, first, expand the number of channels of the depthwise separable convolutional layer of the CNN network to an integer multiple of 2. That is, multiply the number of input and output channels by an integer multiple of 2. Assuming that the original number of channels of this layer is C, we expand it to 2xNxC, where N is the number of convolution kernels. Among them, each NxC uses (3x3 kernel, 5x5 kernel), or (3x3kernel 7x7 kernel). This is enough to create a supernet with strong enough expression ability.

[0106] Secondly, each depthwise separable convolutional layer is split into multiple small units (i.e., convolutional units), represented by i, and each convolutional unit i is assigned a parameter g_i. The splitting diagram is shown in the following figure. Figure 2 As shown, Figure 2 In the example, it is assumed that there are three convolution units i, that is, the parameters assigned to each convolution unit are g_1, g_2, and g_3, but in practical applications, it is not limited to this. The parameters of the small units of the traditional separable convolution layer separable conv are all 1, that is, gi = 1, and in the present disclosure, each convolution unit i after the split is assigned a corresponding parameter g_i, where i can be equal to 1, 2, 3, ... and so on. In the depthwise convolution, each channel is independently convolved with a 3x3 kernel. That is, C_out_i = C_in_i*(3x3kernel), and then a positive parameter g_i is multiplied on C_out_i, that is, each convolution result is multiplied by the assigned parameter g_i of the corresponding convolution unit to obtain the convolution parameter of the corresponding channel, and each convolution parameter is a tensor. Among them, a tensor consists of a set of primitive values ​​that form an array (of any dimension), the order of the tensor is its dimension, and its shape is an integer tuple that specifies the length of each dimension of the array; finally, the convolution parameters of all these channels are combined together as the output of the parameters of the hypernetwork. Through the above definition of the present disclosure, the parameters of the hypernetwork can not only be learned by themselves, but also g_i as the parameters of the corresponding kernel can be learned.

[0107] 2) Training a convolutional neural network model according to the determined hypernetwork parameters to obtain the image conversion model.

[0108] Wherein, in this step, the convolutional neural network model may include: a first generation network having a depth-separable convolutional layer, a first discriminant network, and a second generation network, and the convolutional neural network model is trained according to the determined parameters of the super network to obtain the image conversion model, including:

[0109] First, select a first type of ordinary image training samples and a second type of style image training samples; then input the first type of ordinary image training samples into the first generation network for generation processing, and output the second type of image; then, input the second type of image into the first discriminant network respectively, and obtain the probability that the second type of image belongs to the second type of style image training samples, and the probability is called cross entropy loss; and input the second type of image into the second generation network respectively, and obtain the image loss of the second type of image when passing through the second generation network, which is called consistency loss; and calculate the sum of the consistency loss and the cross entropy loss, or perform weighted summation of the consistency loss and the cross entropy loss to obtain the sum of the losses; finally, calculate the feature map on the gradient domain according to the sum of the losses; update the parameters of the super network according to the feature map; and iteratively train the channels corresponding to the updated parameters of the super network until an image conversion model is generated.

[0110] That is to say, the first type of ordinary image training samples are selected and input into the first generation network for generation processing, and the second type of image is output. The second type of image is respectively input into the first discriminant network and the second generation network, and the first discriminant network performs similarity judgment in the image field and outputs the probability that the generated second type of image belongs to the second type of style image training sample, which is called cross entropy loss. The second generation network performs style conversion processing and outputs a style image with the second type of image. At this time, the image loss of the second type of image when passing through the second generation network is called consistency loss. Then, the sum of the consistency loss and the cross entropy loss is calculated, or the weighted sum of the consistency loss and the cross entropy loss is performed to obtain the sum of the losses, and the feature map on the gradient domain is calculated according to the sum of the losses; the parameters of the super network are updated according to the feature map; the channels corresponding to the updated parameters of the super network are iteratively trained until an image conversion model is generated.

[0111] Similarly, the second type of style image training sample is input into the second generation network for generation processing, and the first type of style image is output. The first type of style image is input into the second discriminant network and the first generation network respectively, and the second discriminant network judges the image type similarity and outputs the probability that the generated first type of style image belongs to the first type of style image; the first generation network performs style conversion processing and outputs an image with the first type of image; the cross entropy loss during the generation and judgment of the second type of image is calculated, and the sum of the consistency loss during the generation and judgment of the first type of style image is calculated, or the consistency loss and the cross entropy loss are weighted summed to obtain the sum of the losses; the feature map on the gradient domain is calculated based on the sum of the losses; the determined super network parameters are updated based on the feature map, and the channels corresponding to the updated super network parameters are iteratively trained until an image conversion model is generated.

[0112] The iterative training of the channels corresponding to the updated parameters of the hypernetwork until the image conversion model is generated includes: in response to the parameters of the hypernetwork being less than a set parameter threshold, deleting the channels corresponding to the parameters less than the set parameter threshold; or, in response to the parameters of the hypernetwork being not less than the set parameter threshold, retaining the channels corresponding to the parameters not less than the set parameter threshold; and repeatedly performing the next iterative training until the image conversion model is generated.

[0113] In the present disclosure, the parameters of the hypernetwork may include the convolution parameters of all channels of the depthwise separable convolution layer. Of course, they may also include the position (weight) and the parameters g_i of the previously defined convolution unit, etc., which are not limited in this embodiment.

[0114] Among them, the parameter threshold, namely λ (Lambda), can be selected with different values ​​in each training. Its initial value is pre-set, and it can also be the set value in subsequent iterative training. Of course, the middle value of the corresponding parameters of all channels can also be selected. This embodiment does not limit it. Its purpose is to reduce channels and further reduce the super network. For example, by judging whether the convolution parameter of the channel is less than Lambda, and deleting the channel corresponding to the convolution parameter less than Lambda, the unimportant branches are subtracted, and then the next data forward transmission is performed, and the parameters of the super network are updated, and then the next iteration training is performed.

[0115] That is to say, it is determined whether the updated parameters of the hypernetwork are less than the set parameter threshold. If so, the channels corresponding to the parameters less than the set parameter threshold are deleted; if not, the channels corresponding to the parameters not less than the set parameter threshold are retained and the next iterative training is performed until the image conversion model is generated.

[0116] In the present disclosure, the convolutional neural network model includes at least four networks, which are two generating networks and two discriminant networks. Among them, the generating network included in the first type is G_a2b (Generator_a2b_super), which is called the first generating network in the present disclosure, and the discriminant network is: D_a2b (Discriminator_a2b_super), which is called the first discriminant network in the present disclosure; the generating network included in the second type is: G_b2a (Generator_b2a_super), which is called the second generating network in the present disclosure, and the discriminant network is D_b2a (Discriminator_b2a super), which is called the second discriminant in the present disclosure.

[0117] Among them, G_a2b is a network that converts the first type of image into the second type of image; D_a2b is a discriminant network that determines the probability that the second type of image obtained after the conversion of G_a2b belongs to the second type. Similarly, G_b2a is a network that converts the second type of image into the first type of image; D_b2a is a discriminant network that determines the probability that the first type of image obtained after the conversion of G_b2a belongs to the first type. That is to say, during training, the input is a pair of single-frame images of two types, one each, in order to generate another style image that is intrinsically related to the original image. It should be noted that the above-mentioned first generation network is also called the first generator, the first discriminant network is called the first discriminator, the second generation network is called the second generator, and the second discriminant network is called the second discriminator. The specific process of its training is as follows. Figure 3 and Figure 4 As shown, Figure 3 : is a training schematic diagram of converting a first type of image into a second type of image provided by the present disclosure: Figure 4 It is a training schematic diagram provided by the present disclosure for converting a second type of image into a first type of image.

[0118] like Figure 3 As shown: First, select one image of each type, that is, select an image of type A (domain A), called image A (Input image_A);

[0119] Then, the input image_A is input to the first generation network (G_a2b, Generator_a2b_super), the first generation network converts the image A and outputs the image A2B (image_A2B), and then the image A2B is input to the first discriminator (D_a2b, Discriminator_a2b_super) and the second generator (G_b2a, Generator_b2a_super) respectively. The first discriminator determines the probability of whether the image A2B belongs to the type B (domain B) picture and outputs the judgment probability. Among them, the first discriminator is a two-class classification network, and the output is the probability that the image belongs to domain B. During the judgment, a corresponding cross entropy loss is generated. The cross entropy loss is a commonly used loss function for classification problems.

[0120] At the same time, G_b2a converts the input image_A2B according to the selected style image to generate image A2B2A (image_A2B2A). The purpose is to make image_A and image_A2B2A very similar, but they belong to different types of styles. At this time, a loss loss is generated during the generation, that is, the consistency loss. Generally speaking, the loss is the L1 norm or the L2 norm, where the L1 norm is the loss in the matrix imageA-imageA2B2A, that is, the sum of the absolute values ​​of each element, such as the L1 norm of (1, -1) is: 1+1=2. After that, the total loss of the network is the sum of these two losses, or the weighted sum of the losses of the consistency loss and the cross entropy loss is obtained, and then the feature map on the gradient domain is calculated according to the sum of the losses, and the parameters of the determined super network are updated according to the feature map. Among them, the calculation method of the sum of the losses and the calculation of the feature map on the gradient domain according to the sum of the losses are well known to those skilled in the art, and will not be repeated here.

[0121] Similarly, if Figure 4As shown, first select an image of type B (domainB), called image B (Input image_B), input the second type of image B (image_B) into the second generator (G_b2a), the second generator converts the image B, and outputs image B2A (image_B2A), then input image B2A into the second discriminator (D_b2a) and the first generator (G_a2b), respectively, the second discriminator determines the probability of whether image B2A belongs to type A (domain A) and outputs the determined probability, wherein the second discriminator is also a two-class classification network, and the output is the probability that the image belongs to domain B. During the discrimination, a corresponding cross entropy loss is generated, which is a commonly used loss function for classification problems.

[0122] At the same time, G_b2a converts the input image_B2A according to the selected style image to generate image B2A2B (image_B2A2B). The purpose is to make image_B and image_B2A2B very similar, but they belong to different types of styles.

[0123] In other words, convolutional neural network models are called two types of generative models and two types of discriminative models. The generative model's generation method learns the joint probability distribution P(X,Y) of samples and labels through observed data. The trained model can generate new data that conforms to the sample distribution. It can be used for supervised learning and unsupervised learning. The discriminative model: regards the tracking problem as a binary classification problem, and then finds the decision boundary between the target and the background. It doesn't matter how the target is described, as long as it knows the difference between the target and the background, then for an input image, see which side of the boundary it is on, and classify it into which category, that is, which type of style it belongs to.

[0124] The present invention first segments the objects in the image to be processed, and then inputs the segmented objects into an image style transfer model trained according to the parameters of a hypernetwork to perform conversion processing according to a selected image style, thereby generating a target object of the selected style, that is, reducing the amount of computational processing within the model while being visually acceptable, which not only improves the efficiency of image style transfer, but also provides user satisfaction.

[0125] It should be noted that, for the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that this implementation disclosure is not limited by the order of the actions described, because according to this disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by this disclosure.

[0126] Figure 5 FIG. 1 is a block diagram of an image processing device according to an exemplary embodiment. Figure 5 The device includes an acquisition module 501, a segmentation module 502 and a style conversion processing module 503, wherein:

[0127] The acquisition module 501 is configured to acquire an image to be processed;

[0128] The segmentation module 502 is configured to segment the object in the image to be processed to obtain the segmented object;

[0129] The style conversion processing module 503 is configured to perform image style conversion processing on the segmented object according to the selected image style through an image conversion model to obtain a processed target object, wherein the image conversion model is a model obtained by iterative training in a manner of first increasing the number of channels of the depth-separable convolutional layer in the hypernetwork and then deleting the channels that do not meet the preset conditions after the increase.

[0130] Optionally, in another embodiment, based on the above embodiment, the device further includes: a determination module 601 and a training module 602, the structure of which is shown in FIG. Figure 6 As shown,

[0131] The determination module 601 is configured to generate a hypernetwork based on a deep separable convolutional layer of a convolutional neural network and determine the parameters of the hypernetwork before the style conversion processing module 503 performs image style conversion processing on the segmented object according to the selected image style through the image conversion model;

[0132] The training module 602 is configured to execute training of a convolutional neural network model according to the determined parameters of the hypernetwork to obtain the image conversion model.

[0133] Optionally, in another embodiment, based on the above embodiment, the determination module 601 includes: a channel expansion module 701, a splitting module 702, a convolution module 703, a product module 704 and a combination module 705, and its structural diagram is shown as follows: Figure 7 As shown,

[0134] The channel expansion module 701 is configured to expand the number of input channels C of the depthwise separable convolutional layer of the hypernetwork to an integer multiple of 2 to obtain 2xNxC channels, where N is the number of convolution kernels;

[0135] The splitting module 702 is configured to split the enlarged depth-separable convolutional layer into convolutional units i, and assign a parameter g_i to each convolutional unit i;

[0136] The convolution module 703 is configured to perform convolution processing on each enlarged input channel with a corresponding convolution kernel to obtain a corresponding convolution result;

[0137] The product module 704 is configured to perform multiplication of each convolution result by the parameter g_i of the corresponding convolution unit i to obtain the convolution parameter of the corresponding channel;

[0138] The combination module 705 is configured to combine the convolution parameters of all channels together as the parameters of the hypernetwork.

[0139] Optionally, in another embodiment, based on the above embodiment, the adversarial neural network model includes: a first generation network with a deep separable convolutional layer, a second generation network and a first discriminant network;

[0140] The training module package 602 includes: a selection module 801, a first image processing module 802, a first loss determination module 803, a second loss determination module 804, a calculation module 805, a feature map calculation module 806, an update module 807 and an iteration module 808. The structural diagram is shown in FIG. Figure 8 As shown,

[0141] The selection module 801 is configured to select a first type of common image training sample and a second type of style image training sample;

[0142] The first image processing module 802 is configured to input the first type of common image training samples into the first generation network for generation processing, and output a second type of image;

[0143] The first loss determination 803 is configured to input the second type of images into the first discriminant network for judgment, and obtain the probability that the second type of images belong to the second type of style image training samples, and the probability is called cross entropy loss;

[0144] The second loss determination module 804 is configured to input the second type of images into the second generation network respectively, and the image loss obtained when the second type of images pass through the second generation network is called consistency loss;

[0145] The calculation module 805 is configured to calculate the sum of the consistency loss and the cross entropy loss, or perform weighted summation of the consistency loss and the cross entropy loss to obtain the sum of the losses;

[0146] The feature map calculation 806 is configured to perform a feature map calculation on a gradient domain according to the sum of the losses;

[0147] The updating module 807 is configured to update the parameters of the super network according to the feature graph;

[0148] The iteration module 808 is configured to perform iterative training on the channels corresponding to the updated parameters of the super network until an image conversion model is generated.

[0149] Optionally, in another embodiment, based on the above embodiment, the iteration module includes: a channel deletion module and a channel retention module, wherein:

[0150] The channel deletion module is configured to execute, in response to a parameter of the hypernetwork being less than a set parameter threshold, deleting a channel corresponding to a parameter less than the set parameter threshold;

[0151] The channel reservation module is configured to execute, in response to a parameter of the hypernetwork being not less than a set parameter threshold, retaining a channel corresponding to a parameter not less than the set parameter threshold;

[0152] The iterative training module is repeatedly configured to perform the next iterative training repeatedly until an image conversion model is generated.

[0153] The present disclosure also provides an electronic device, comprising:

[0154] processor;

[0155] a memory for storing instructions executable by the processor;

[0156] The processor is configured to execute the instructions to implement the image processing method as described above.

[0157] The present disclosure also provides a storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the image processing method as described above.

[0158] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method. For relevant details, please refer to the partial description of the method embodiment, and will not be elaborated here.

[0159] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, and the instructions can be executed by a processor of an electronic device to complete the above method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0160] Fig. 9 1 is a block diagram of an electronic device 900 according to an exemplary embodiment. For example, the electronic device 900 may be a mobile terminal or a server, and the electronic device is a mobile terminal as an example for explanation in the embodiment of the disclosure. For example, the electronic device 900 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0161] Reference Fig. 9 , the electronic device 900 may include one or more of the following components: a processing component 902 , a memory 904 , a power component 906 , a multimedia component 908 , an audio component 910 , an input / output (I / O) interface 912 , a sensor component 914 , and a communication component 916 .

[0162] The processing component 902 generally controls the overall operation of the electronic device 900, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 902 may include one or more modules to facilitate the interaction between the processing component 902 and other components. For example, the processing component 902 may include a multimedia module to facilitate the interaction between the multimedia component 908 and the processing component 902.

[0163] The memory 904 is configured to store various types of data to support operations on the device 900. Examples of such data include instructions for any application or method operating on the electronic device 900, contact data, phone book data, messages, pictures, videos, etc. The memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0164] The power supply component 906 provides power to the various components of the electronic device 900. The power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 900.

[0165] The multimedia component 908 includes a screen that provides an output interface between the electronic device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 908 includes a front camera and / or a rear camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0166] The audio component 910 is configured to output and / or input audio signals. For example, the audio component 910 includes a microphone (MIC), and when the electronic device 900 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 904 or sent via the communication component 916. In some embodiments, the audio component 910 also includes a speaker for outputting audio signals.

[0167] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0168] The sensor assembly 914 includes one or more sensors for providing various aspects of status assessment for the electronic device 900. For example, the sensor assembly 914 can detect the open / closed state of the device 900, the relative positioning of components, such as the display and keypad of the electronic device 900, and the sensor assembly 914 can also detect the position change of the electronic device 900 or a component of the electronic device 900, the presence or absence of user contact with the electronic device 900, the orientation or acceleration / deceleration of the electronic device 900, and the temperature change of the electronic device 900. The sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 914 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 914 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0169] The communication component 916 is configured to facilitate wired or wireless communication between the electronic device 900 and other devices. The electronic device 900 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 916 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 916 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0170] In an exemplary embodiment, the electronic device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the image processing method shown above.

[0171] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, and the instructions can be executed by a processor 920 of an electronic device 900 to perform the image processing method shown above. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0172] In an exemplary embodiment, a computer program product is further provided. When instructions in the computer program product are executed by the processor 920 of the electronic device 900, the electronic device 900 executes the image processing method shown above.

[0173] Fig.10 1 is a block diagram of an apparatus 1000 for image processing according to an exemplary embodiment. For example, the apparatus 1000 may be provided as a server. Fig.10 The apparatus 1000 includes a processing component 1022, which further includes one or more processors, and a memory resource represented by a memory 1032 for storing instructions, such as an application, that can be executed by the processing component 1022. The application stored in the memory 1032 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1022 is configured to execute the instructions to perform the above-mentioned image processing method.

[0174] The device 1000 may also include a power supply component 1026 configured to perform power management of the device 1000, a wired or wireless network interface 1050 configured to connect the device 1000 to a network, and an input / output (I / O) interface 1058. The device 1000 may operate based on an operating system stored in the memory 1032, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™, or the like.

[0175] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0176] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Get the image to be processed; Segmenting the object in the image to be processed to obtain the segmented object; Performing image style conversion processing on the segmented object according to the selected image style through an image conversion model to obtain a processed target object, wherein the image conversion model is a model obtained by iterative training by first increasing the number of channels of the depthwise separable convolutional layer in the hypernetwork and then deleting the channels that do not meet the preset conditions after the increase; Before performing image style conversion processing on the segmented object according to the selected image style through the image conversion model, the method further includes: Generating a hypernetwork based on a depthwise separable convolutional layer of a convolutional neural network, and determining parameters of the hypernetwork; Training a convolutional neural network model according to the determined parameters of the hypernetwork to obtain the image conversion model; The generation of a hypernetwork based on the deep separable convolutional layer of the convolutional neural network and determining the parameters of the hypernetwork include: Expanding the number of input channels C of the depthwise separable convolutional layer of the convolutional neural network to an integer multiple of 2 to obtain 2xNxC channels, where N is the number of convolution kernels; Split the enlarged depth-separable convolutional layer into convolutional units i, and assign parameter g_ i ; Convolve each enlarged input channel with the corresponding convolution kernel to obtain the corresponding convolution result; Multiply each convolution result by the parameter g_ of the corresponding convolution unit i i , get the convolution parameters of the corresponding channel; The convolution parameters of all channels are combined together as the parameters of the hypernetwork.

2. The image processing method according to claim 1, characterized in that: The convolutional neural network model includes: a first generation network with a depth-separable convolutional layer, a first discriminant network, and a second generation network; The step of training a convolutional neural network model according to the determined parameters of the hypernetwork to obtain the image conversion model comprises: Selecting a first type of common image training sample and a second type of style image training sample; Inputting the first type of common image training samples into the first generation network for generation processing, and outputting the second type of images; Inputting the second type of images into the first discriminant network respectively, and obtaining the probability that the second type of images belong to the second type of style image training samples, wherein the probability is called the cross entropy loss; Inputting the second type of images into the second generation network respectively, and obtaining an image loss when the second type of images pass through the second generation network is called consistency loss; Calculating the sum of the consistency loss and the cross entropy loss, or performing weighted summation on the consistency loss and the cross entropy loss to obtain the sum of the losses; Calculate a feature map on the gradient domain according to the sum of the losses; Updating parameters of the hypernetwork according to the feature graph; The channels corresponding to the updated parameters of the hypernetwork are iteratively trained until an image conversion model is generated.

3. The image processing method according to claim 2, characterized in that: The iterative training of the channels corresponding to the updated parameters of the hypernetwork until an image conversion model is generated comprises: In response to a parameter of the hypernetwork being less than a set parameter threshold, deleting a channel corresponding to the parameter less than the set parameter threshold; In response to a parameter of the hypernetwork being not less than a set parameter threshold, retaining a channel corresponding to a parameter not less than the set parameter threshold; The next iterative training is repeated until an image conversion model is generated.

4. An image processing device, characterized in that: include: An acquisition module is configured to acquire an image to be processed; A segmentation module is configured to segment the object in the image to be processed to obtain the segmented object; A style conversion processing module is configured to perform image style conversion processing on the segmented object according to the selected image style through an image conversion model to obtain a processed target object, wherein the image conversion model is a model obtained by iterative training by first increasing the number of channels of the depthwise separable convolutional layer in the hypernetwork and then deleting the channels that do not meet the preset conditions after the increase; The device also includes: A determination module is configured to generate a hypernetwork based on a deep separable convolutional layer of a convolutional neural network and determine parameters of the hypernetwork before the style conversion processing module performs image style conversion processing on the segmented object according to the selected image style through the image conversion model; A training module is configured to train a convolutional neural network model according to the determined parameters of the hypernetwork to obtain the image conversion model; Wherein, the determination module includes: A channel expansion module is configured to expand the number of input channels C of the depthwise separable convolutional layer of the hypernetwork to an integer multiple of 2 to obtain 2xNxC channels, where N is the number of convolution kernels; A splitting module is configured to split the enlarged depth-separable convolutional layer into convolutional units i and assign a parameter g_ i ; The convolution module is configured to perform convolution processing on each enlarged input channel with the corresponding convolution kernel to obtain a corresponding convolution result; The product module is configured to perform each convolution result multiplied by the parameter g_ of the corresponding convolution unit i i , get the convolution parameters of the corresponding channel; The combination module is configured to perform the convolutional parameters of all channels together as the parameters of the hypernetwork.

5. The image processing device according to claim 4, characterized in that: The convolutional neural network model includes: a first generation network having a depth-separable convolutional layer, a first discriminant network, and a second generation network; The training module includes: A selection module is configured to select a first type of common image training sample and a second type of style image training sample; A first image processing module is configured to input the first type of common image training samples into the first generation network for generation processing, and output a second type of image; A first loss determination module is configured to input the second type of images into the first discriminant network for judgment, and obtain a probability that the second type of images belong to the second type of style image training samples, wherein the probability is called a cross entropy loss; A second loss determination module is configured to input the second type of images into the second generation network respectively, and the image loss obtained when the second type of images pass through the second generation network is called consistency loss; A calculation module is configured to calculate the sum of the consistency loss and the cross entropy loss, or to perform weighted summation on the consistency loss and the cross entropy loss to obtain the sum of the losses; A feature map calculation module is configured to calculate a feature map in a gradient domain according to the sum of the losses; An updating module, configured to update the parameters of the super network according to the feature graph; The iteration module is configured to perform iterative training on the channels corresponding to the updated parameters of the hypernetwork until an image conversion model is generated.

6. The image processing device according to claim 5, characterized in that: The iteration module includes: A channel deletion module is configured to execute, in response to a parameter of the hypernetwork being less than a set parameter threshold, deleting a channel corresponding to a parameter less than the set parameter threshold; A channel reservation module is configured to execute, in response to a parameter of the super network being not less than a set parameter threshold, retaining a channel corresponding to a parameter not less than the set parameter threshold; The iterative training module is repeatedly configured to perform the next iterative training repeatedly until an image conversion model is generated.

7. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image processing method according to any one of claims 1 to 3.

8. A storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image processing method as claimed in any one of claims 1 to 3.

Images