Image Processing Method, Apparatus, Electronic Device, and Computer-Readable Storage Medium

Abstract

An embodiment of the present invention discloses an image processing method, apparatus, electronic device, and computer-readable storage medium. After obtaining an image sample set, the embodiment of the present invention uses a preset image processing model to extract style features from a style image sample to obtain initial style features of the style image sample. Then, the initial style features are clustered to obtain a style feature pool corresponding to the image sample set. According to the style feature pool, the initial style features are adjusted to obtain style features of the style image sample. Then, based on the style features of the style image sample and the original image sample, the preset image processing model is converged to obtain a trained image processing model, and the trained image processing model is used to perform style conversion on a to-be-processed original image according to the style image. This solution can improve the accuracy of image processing.

Description

Technical Field

[0001] The present invention relates to the field of communication technologies, and in particular, to an image processing method, apparatus, electronic device, and computer-readable storage medium. Background Art

[0002] In recent years, with the rapid development of Internet technologies, the types and styles of images have become increasingly diverse. There are also more and more ways to process images. For example, a real photo taken can be converted into a style image with an artistic style. Existing image processing methods often directly perform image conversion or style transfer on the original image to generate a style image with a specific artistic style.

[0003] In the process of researching and practicing the existing technologies, the inventors of the present invention found that the method of image conversion cannot generate images of different styles, and the method of style transfer only models the style as a texture to achieve the transfer. For some artistic style images without texture information, it is often impossible to ensure the accuracy of style transfer. Therefore, the accuracy of image processing is reduced. Summary of the Invention

[0004] Embodiments of the present invention provide an image processing method, apparatus, electronic device, and computer-readable storage medium, which can improve the accuracy of image processing.

[0005] An image processing method includes:

[0006] Obtaining an image sample set, where the image sample set includes an original image sample and a style image sample;

[0007] Extracting style features of the style image sample by using a preset image processing model to obtain initial style features of the style image sample;

[0008] Clustering the initial style features to obtain a style feature pool corresponding to the image sample set;

[0009] Adjusting the initial style features according to the style feature pool to obtain style features of the style image sample;

[0010] Converging the preset image processing model based on the style features of the style image sample and the original image sample to obtain a trained image processing model, where the trained image processing model is used to perform style conversion on a to-be-processed original image according to a style image.

[0011] Correspondingly, an embodiment of the present invention provides an image processing apparatus, including:

[0012] An obtaining unit, configured to obtain an image sample set, where the image sample set includes an original image sample and a style image sample;

[0013] An extraction unit, configured to extract style features of the style image sample by using a preset image processing model, so as to obtain initial style features of the style image sample;

[0014] A clustering unit, configured to cluster the initial style features to obtain a style feature pool corresponding to the image sample set;

[0015] An adjustment unit, configured to adjust the initial style features according to the style feature pool to obtain style features of the style image sample;

[0016] A convergence unit, configured to converge the preset image processing model based on the style features of the style image sample and the original image sample to obtain a trained image processing model, where the trained image processing model is used to perform style conversion on a to-be-processed original image according to a style image.

[0017] Optionally, in some embodiments, the extraction unit may specifically be configured to perform data augmentation on the style image sample to obtain a plurality of homologous style image samples corresponding to the style image sample; and extract style features of the homologous style image samples by using a style feature extraction network of a preset image processing model to obtain initial style features of the style image sample.

[0018] Optionally, in some embodiments, the clustering unit may specifically be configured to cluster the initial style features to obtain a plurality of style feature clusters; calculate the mean value of the style features in each style feature cluster respectively to obtain a basic style feature corresponding to each style feature cluster; and fuse the basic style features to obtain a style feature pool corresponding to the image sample set.

[0019] Optionally, in some embodiments, the convergence unit may specifically be configured to converge the style feature extraction network based on the style features to obtain a trained style feature extraction network; extract target style features of the style image sample by using the trained style feature extraction network; train an image processing network according to the target style features and the original image sample to obtain a trained image processing network, and use the trained style feature extraction network and the trained image processing network as the trained image processing model.

[0020] Optionally, in some embodiments, the convergence unit may specifically be configured to classify the style features to obtain the homologous style features and non-homologous style features corresponding to the style image sample, where the homologous style features are the style features corresponding to the homologous style image samples of the style image sample; calculate the first feature distance between the homologous style features and the second feature distance between the homologous style features and the non-homologous style features respectively; determine the first style loss information of the style image sample based on the first feature distance and the second feature distance, and converge the style feature extraction network according to the first style loss information to obtain the trained style feature extraction network.

[0021] Optionally, in some embodiments, the convergence unit may specifically be configured to perform grayscale processing on the original push sample to obtain a grayscale image sample, perform image processing on the grayscale image sample according to the target style feature by using the image processing network to obtain the current style image corresponding to the grayscale image sample; perform reconstruction on the grayscale image sample and the current style image by using the image processing network to obtain the reconstructed grayscale image sample and the reconstructed style image; determine the image loss information of the image sample set based on the current style image, the grayscale image sample, the reconstructed grayscale image sample, and the reconstructed style image, and converge the image processing network based on the image loss information to obtain the trained image processing network.

[0022] Optionally, in some embodiments, the convergence unit may specifically be configured to use the image processing network to restore the style of the current style image to the style of the grayscale image sample to obtain a reconstructed grayscale image sample; screen out the first style image sample and the second style image sample for style image reconstruction from the style image samples, and use the image processing network to convert the first style image sample into a target grayscale image sample; perform processing on the target grayscale image sample and the second style image sample by using the image processing network to obtain a reconstructed style image.

[0023] Optionally, in some embodiments, the convergence unit may be specifically configured to determine the adversarial loss information of the image sample set according to the current style image and the target grayscale image sample; determine the grayscale image loss information of the image sample set based on the grayscale image sample and the reconstructed grayscale image sample; determine the style image loss information of the image sample set according to the first style image sample and the reconstructed style image; determine the second style loss information of the image sample set based on the target style feature, the current style image, the reconstructed style image, and the second style image sample; and fuse the second style loss information, the adversarial loss information, the grayscale image loss information, and the style image loss information to obtain the image loss information of the image sample set.

[0024] Optionally, in some embodiments, the convergence unit may be specifically configured to calculate the image similarity between the grayscale image sample and the reconstructed grayscale image sample, and determine the similarity loss information of the image sample set based on the image similarity; perform pixel truncation on the grayscale image sample and the reconstructed grayscale image sample, and compare the pixel-truncated grayscale image sample with the pixel-truncated reconstructed grayscale image sample to obtain the pixel loss information of the image sample set; and fuse the similarity loss information and the pixel loss information to obtain the grayscale image loss information of the image sample set.

[0025] Optionally, in some embodiments, the convergence unit may be specifically configured to separately extract style features from the first style image and the reconstructed style image, and calculate the feature distance between the extracted style features to obtain a first style feature distance; calculate the image similarity between the first style image and the reconstructed style image; and determine the style image loss information of the image sample set based on the style feature distance and the image similarity.

[0026] Optionally, in some embodiments, the convergence unit may be specifically configured to use a trained style feature extraction network to separately extract features from the current style image, the reconstructed style image, and the second style image sample, and adjust the extracted style features according to a style feature pool to obtain a first style feature of the current style image, a second style feature of the reconstructed style image, and a third style feature of the second style image sample; calculate the feature distance between the first style feature and the target style feature to obtain a second style feature distance, and calculate the feature distance between the second style feature and the third style feature to obtain a third style feature distance; and determine the second style loss information of the image sample set according to the second style feature distance and the third style feature distance.

[0027] Optionally, in some embodiments, the adjustment unit may be specifically configured to determine the mapping relationship of the initial style feature with respect to the style feature pool according to the style feature pool; based on the mapping relationship, calculate the mapping vector of the initial style feature to the style feature pool to obtain the style feature of the style image sample.

[0028] Optionally, in some embodiments, the image processing apparatus may further include a processing unit, and the processing unit may be specifically configured to obtain an original image to be processed and a style image, and perform feature extraction on the style image by using a trained image processing model to obtain the current style feature corresponding to the style image; perform grayscale processing on the original image to be processed to obtain a grayscale image to be processed, and perform feature extraction on the grayscale image to be processed; adjust the extracted image features based on the current style feature, and generate a target style image based on the adjusted image features.

[0029] In addition, an embodiment of the present invention further provides an electronic device, including a processor and a memory, where the memory stores an application program, and the processor is configured to run the application program in the memory to implement the image processing method provided by the embodiment of the present invention.

[0030] In addition, an embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores multiple instructions, and the instructions are suitable for being loaded by a processor to execute the steps in any one of the image processing methods provided by the embodiment of the present invention.

[0031] After obtaining the image sample set, the embodiment of the present invention performs style feature extraction on the style image sample by using a preset image processing model to obtain the initial style feature of the style image sample, and then clusters the initial style feature to obtain a style feature pool corresponding to the image sample set. According to the style feature pool, the initial style feature is adjusted to obtain the style feature of the style image sample. Then, based on the style feature of the style image sample and the original image sample, the preset image processing model is converged to obtain a trained image processing model, and the trained image processing model is used to perform style conversion on the original image to be processed according to the style image; since this solution clusters the extracted initial style features to obtain a style feature pool and uses the style feature pool to adjust the initial style features, the adjusted style features are less interfered by style-irrelevant information, and for artistic style images without texture information, it can also ensure the accuracy of the extracted style features, and can also generate style images of any style. Therefore, the accuracy of image processing can be improved. Description of the Drawings

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those skilled in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0033] Figure 1 is a schematic diagram of the scenario of image processing provided by the embodiments of the present invention;

[0034] Figure 2 is a schematic flowchart of the image processing method provided by the embodiments of the present invention;

[0035] Figure 3 is an image schematic diagram of the original image sample and the style image sample provided by the embodiments of the present invention;

[0036] Figure 4 is a schematic diagram of the training of the image processing model provided by the embodiments of the present invention;

[0037] Figure 5 is a schematic flowchart of extracting the style features of the image sample provided by the embodiments of the present invention;

[0038] Figure 6 is a schematic flowchart of reconstructing the original image sample and the current style image provided by the embodiments of the present invention;

[0039] Figure 7 is a schematic flowchart of performing image processing using the image processing model provided by the embodiments of the present invention;

[0040] Figure 8 is a schematic flowchart of the process of outputting a portrait painting according to the present solution provided by the embodiments of the present invention;

[0041] Figure 9 is another schematic flowchart of the image processing method provided by the embodiments of the present invention;

[0042] Figure 10 is a schematic diagram of the structure of the image processing device provided by the embodiments of the present invention;

[0043] Figure 11 is another schematic diagram of the structure of the image processing device provided by the embodiments of the present invention;

[0044] Figure 12 is a schematic diagram of the structure of the electronic device provided by the embodiments of the present invention. Detailed implementation manners

[0045] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts belong to the protection scope of the present invention.

[0046] The embodiments of the present invention provide an image processing method, apparatus, electronic device, and computer-readable storage medium. Among them, the image processing apparatus can be integrated in an electronic device, and the electronic device can be a server or a terminal device, etc.

[0047] Among them, the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication methods, and this application does not make any restrictions here.

[0048] For example, referring to Figure 1 , taking the case where the image processing apparatus is integrated in an electronic device as an example, after the electronic device obtains an image sample set, it uses a preset image processing model to extract the style features of the style image sample to obtain the initial style features of the style image sample. Then, it clusters the initial style features to obtain a style feature pool corresponding to the image sample set. According to the style feature pool, it adjusts the initial style features to obtain the style features of the style image sample. Then, based on the style features of the style image sample and the original image sample, it converges the preset image processing model to obtain a trained image processing model, and this trained image processing model is used to perform style conversion on the original image to be processed according to the style image.

[0049] Among them, image processing can be understood as converting a captured real photo into a style photo with a specific artistic style. For example, according to the style photo, a real face photo is converted into a portrait painting with a specific style, and the style of the portrait painting is the same as that of the style photo. The types of the style photo can be various. For example, it can include pre-set style photos, or it can include style photos drawn by the user himself, or it can also include new style photos generated by the user by fusing multiple pre-set or existing style photos, and so on. Therefore, compared with the existing way of pre-defining styles, there is no need to limit the style types in this solution.

[0050] The following will be described in detail respectively. It should be noted that the description order of the following embodiments does not limit the preferred order of the embodiments.

[0051] This embodiment will be described from the perspective of an image processing device. The image processing device can be specifically integrated in an electronic device, and the electronic device can be a server or a terminal device, etc.; among them, the terminal can include a tablet computer, a notebook computer, a personal computer (PC, Personal Computer), a wearable device, a virtual reality device, or other intelligent devices that can perform image processing, etc.

[0052] An image processing method includes:

[0053] Obtain an image sample set, where the image sample set includes an original image sample and a style image sample. Use a preset image processing model to extract style features from the style image sample to obtain the initial style features of the style image sample. Cluster the initial style features to obtain a style feature pool corresponding to the image sample set. According to the style feature pool, adjust the initial style features to obtain the style features of the style image sample. Converge the preset image processing model based on the style features of the style image sample and the original image sample to obtain a trained image processing model, and the trained image processing model is used to perform style conversion on the original image to be processed according to the style image.

[0054] As Figure 2 shown, the specific process of this image processing method is as follows:

[0055] 101. Obtain an image sample set.

[0056] Among them, the image sample set includes original image samples and style image samples. The so-called original image samples can be real images or video frame pictures collected by an image acquisition device. For example, it can include pictures of people, faces, animals, items, or other real pictures or images. The style image samples can be pictures with artistic styles generated by artistic means, which do not include directly shooting or collecting by the user's image acquisition device. The biggest difference between the style image samples and the original image samples is that the style image samples are not real images collected by the image acquisition device, but images with specific artistic styles generated by drawing or other image generation methods. Taking the original image sample as a real person's picture and the style image sample as a pencil drawing as an example, the original image sample and the style image sample can be as Figure 3 shown.

[0057] Among them, there are various ways to obtain the image sample set. For example, the image sample set can be directly obtained, or when the number of original image samples and style image samples in the image sample set is large or the memory is large, the image sample set can also be obtained indirectly. Specifically, it can be as follows:

[0058] (1) Directly obtain the image sample set;

[0059] For example, at least one original image sample and style image sample sent by the user through the terminal can be directly received to obtain the image sample set, or the original image samples and style image samples can also be directly obtained from the Internet or an image database to obtain the image sample set.

[0060] (2) Indirectly obtain the image sample set;

[0061] For example, an image processing request sent by the user through the terminal can be received. The storage address of the image sample set is carried in the image processing request. According to this storage address, the original image samples and style image samples are obtained in the memory, cache, or a third-party image database to obtain the image sample set.

[0062] Optionally, after obtaining the image sample set in the memory, cache, or a third-party image database, a prompt message can also be sent to the terminal, and this prompt message is used to prompt the terminal that the image sample set has been obtained.

[0063] 102. Use a preset image processing model to extract the style features of the style image sample to obtain the initial style features of the style image sample.

[0064] For example, data augmentation can be performed on the style image sample to obtain multiple homologous style image samples corresponding to the style image sample. The style feature extraction network of the preset image processing model is used to extract the style features of the homologous style image samples, and the initial style features of the style image sample are obtained.

[0065] Among them, there are various ways to perform data augmentation on the style image sample. For example, data augmentation methods such as random horizontal flipping, image blurring, or random graying can be performed on the style image sample, so as to obtain multiple homologous style image samples corresponding to the image sample. The so-called homologous style image samples can be understood as the style image samples after data augmentation from the same style image sample. The number of homologous style image samples can be set according to the actual application. For example, it can be 2 or any number.

[0066] After obtaining the homologous style image samples, the style features of the homologous style image samples can be extracted to obtain the initial style features of the style image sample. There are various ways to extract the style features. For example, the style feature extraction network of the preset image processing model can be used to perform operations such as convolution and pooling on the style image sample, and finally abstract into the initial style features of the preset dimension corresponding to each homologous style image sample, so as to obtain the initial style features of the style image sample.

[0067] 103. Cluster the initial style features to obtain the style feature pool corresponding to the image sample set.

[0068] Among them, the style feature pool is a set of style features used to represent the stable style information in the style image sample.

[0069] Among them, there are various ways to cluster the initial style features to construct the style feature pool corresponding to the image sample set. Specifically, it can be as follows:

[0070] For example, the initial style features can be clustered to obtain multiple style feature clusters, and the mean value of the style features in each style feature cluster is calculated respectively to obtain the basic style features corresponding to each style feature cluster. The basic style features are fused to obtain the style feature pool corresponding to the image sample set.

[0071] Among them, there are various ways to cluster the initial style features. For example, k-means (K-means) can be used to cluster the initial style features to obtain multiple style feature clusters. Or, other clustering algorithms can also be used to cluster the initial style features.

[0072] Among them, the style feature pool for constructing the style image samples can essentially be regarded as extracting style features strongly related to the style information of the style image samples from the extracted initial style features. For example, taking the number of homologous style image samples as 2, for all style picture samples X = {x1, x2, …, x n}, after data augmentation and style feature extraction, the initial style features H = {h′1, h″1, h′2, h″2, …, h′ n , h″ n} are obtained, where h′ i , h″ i correspond to two data-augmented homologous style image samples of the input style image sample. Perform K-means clustering on these initial style features to obtain T clustering clusters. Then, calculate the average value of all initial style features in each clustering cluster to obtain the final T representative features, and form a style feature pool with these T representative features

[0073] 104. According to the style feature pool, adjust the initial style features to obtain the style features of the style image samples.

[0074] For example, according to the style feature pool, determine the mapping relationship of the initial style features with respect to the style feature pool. Based on the mapping relationship, calculate the mapping vector of the initial style features to the style feature pool to obtain the style features of the style image samples.

[0075] Among them, there are various ways to calculate the mapping vector of the initial style features to the style feature pool. For example, by fusing the initial style features of two or more homologous style image samples corresponding to the style image sample with the style feature pool, the mapping vector of the initial style features to the style feature pool can be obtained, which can be specifically shown in formula (1):

[0076] s i = W(a i ) · B (1)

[0077] Among them, s i is the mapping vector, that is, the style feature, a ii is any style image sample, W(a i ) is the initial style feature of the style image sample a i , and B is the style feature pool.

[0078] Taking the case where the style image sample has two homologous style image samples, that is, the number of initial style features corresponding to the style image sample is 2 as an example, the calculation of the mapping vector of the initial style features to the style feature values can refer to formula (2) shown:[[]]

[0079] sf i ′ = h′ i B, sfi i ″ = h″ i B (2)

[0080] Among them, sf i ′ and sf i ″ are the mapping vectors corresponding to the homologous style images respectively, h′ i and h″ i are the initial style features of the homologous style images respectively, and B is the style feature pool.

[0081] The calculated mapping vector can be directly used as the style feature of the style image sample, or alternatively, the calculated mapping vectors can be fused to obtain the style feature of the style image sample.

[0082] Among them, it should be noted that the method of calculating the mapping vector to adjust the initial style features can, on the one hand, effectively reduce the dimension of the style features, and on the other hand, by introducing the style feature values composed of the representative features, it can effectively alleviate the interference caused by the information irrelevant to the style information in the initial style features.

[0083] 105. Converge the preset image processing model based on the style features of the style image sample and the original image sample to obtain the trained image processing model.

[0084] Among them, the trained image processing model is used to perform style conversion on the original image to be processed. The trained image processing model can include a trained style feature extraction network and a trained image processing network.

[0085] Among them, there are various ways to converge the preset image processing model. Specifically, it can be as follows:

[0086] For example, the style feature extraction network can be converged based on the style features to obtain a trained style feature extraction network. The trained style extraction network is used to extract the features of the style image sample to obtain the target style features of the style image sample. The image processing network is trained based on the target style features and the original image sample to obtain a trained image processing network. The trained style feature network and the trained image processing network are used as the trained image processing model. Specifically, it can be as follows:

[0087] S1. Converge the style feature extraction network based on the style features to obtain a trained style feature extraction network.

[0088] For example, classify the style features to obtain the homologous style features and non-homologous style features corresponding to the style image samples. Calculate the first feature distance between the homologous style features and the second feature distance between the homologous style features and the non-homologous style features respectively. Based on the first feature distance and the second feature distance, determine the first style loss information of the style image samples, and converge the style feature extraction network according to the first style loss information to obtain the trained style feature extraction network.

[0089] Among them, the homologous style features are the style features corresponding to the homologous style image samples in the style image samples. For example, the homologous style image samples of the style image sample x are x1 and x2, and the image style features h′1 and h″1 corresponding to the homologous style image samples x1 and x2 are the homologous style features, and the style feature h′2 corresponding to the homologous style image of the style image sample y is the non-homologous style feature.

[0090] After obtaining the homologous style features and non-homologous style features, the first feature distance between the homologous style features and the second feature distance between the homologous style features and the non-homologous style features can be calculated. There are various ways to calculate the feature distance. For example, the cosine similarity between the style features can be calculated, and then the feature distance between the style features can be determined according to the cosine similarity. The specific calculation formula of the cosine similarity can refer to formula (3) as follows:

[0091]

[0092] Among them, sim(a,b) is the cosine similarity between the style feature W(a i ) and the style feature W(a j ), a i , a j are any two style image samples respectively, W(a i ) is the style feature of the style image sample a i , and W(a j ) is the style feature of the style image sample a j .

[0093] Among them, when determining the feature distance between the style features based on the cosine similarity, the temperature parameter τ also needs to be considered. τ is a hyperparameter used to control the matching degree of the sample pairs, and can be specifically as shown in formula (4):

[0094]

[0095] Among them, ES(a i , a j ) is the feature distance, and sim(W(a i ), W(a j )) is the style feature W(ai ) and the cosine similarity between the style feature W(a j ), where τ is the temperature parameter.

[0096] Alternatively, the Euclidean distance or other types of distances between the style features can also be calculated as the first feature distance between the homologous style features and the second feature distance between the homologous style features and the non-homologous style features.

[0097] After calculating the first feature distance and the second feature distance, the first style loss information of the style image sample can be determined. The so-called first style loss information is used to constrain the distance between the style features of the homologous style image samples to be less than that of the non-homologous style image samples. There are various ways to determine the first style loss information. For example, the second feature distance can be fused to obtain the non-homologous feature distance, and the non-homologous feature distance is added to the first feature distance to obtain the target feature distance. The ratio of the first feature distance to the target feature distance is calculated respectively to obtain the feature distance ratio, and the first style loss information is determined according to the feature distance ratio. The specific formula is shown in formula (5):

[0098]

[0099] where L1 is the first style loss information, a i , is the homologous style feature, a i , is the non-homologous style feature, is the first feature distance, is the non-homologous feature distance.

[0100] Among them, when the number of homologous style image samples of the style image sample is two, the calculation of the first style loss information can be as shown in formula (6):

[0101]

[0102] where, is the first style loss information, h′ i , h′ j and h″ i , h″ j are non-homologous style features, h′ i , h″ i are homologous style features, is the first feature distance, and are the second feature distances, and τ is a hyperparameter used to control the matching degree of the sample pairs.

[0103] S2. Use the trained style feature extraction network to extract features from the style image samples to obtain the target style features of the style image samples.

[0104] For example, the trained style feature extraction network can be used to directly extract the style features from the style image samples to obtain the target style features of the style image samples. Alternatively, the trained style feature extraction network can be used to extract features from the style image samples to obtain the initial style features, and the style feature pool can be used to adjust the initial style features to obtain the target style features of the style image samples.

[0105] S3. Train the image processing network according to the target style features and the original image samples to obtain the trained image processing network, and use the trained style feature extraction network and the trained image processing network as the trained image processing model.

[0106] For example, the original image samples can be grayscale processed to obtain grayscale image samples. According to the target style features, the image processing network is used to perform image processing on the grayscale image samples to obtain the current style image corresponding to the grayscale image samples. The image processing network is used to reconstruct the grayscale image samples and the current style image to obtain the reconstructed grayscale image samples and the reconstructed style image. Based on the current style image, the grayscale image samples, the reconstructed grayscale image samples, and the reconstructed style image, the image loss information of the image sample set is determined, and the image processing network is converged based on the image loss information to obtain the trained image processing network.

[0107] Among them, the grayscale processing of the original image samples is mainly to get rid of the irrelevant information brought by the colors in the RGB (a color space) images as much as possible, so as to ensure that this solution can pay more attention to the parts directly related to the painting style of the target style image during the image processing process. Therefore, the accuracy of image processing can be improved. In addition, there are various ways to use the image processing network to perform image processing on the grayscale image samples. For example, the grayscale image samples can be feature extracted to obtain the sample image features of the grayscale image samples. Based on the target style features, the sample image features are adjusted, and based on the adjusted sample image features, the current style image corresponding to the grayscale image samples is generated.

[0108] After obtaining the current style image corresponding to the grayscale image sample, the grayscale image sample and the current style image can be reconstructed. There are various ways of reconstruction. For example, an image processing network can be used to restore the style of the current style image to the style of the grayscale image sample to obtain the reconstructed grayscale image sample. The first style image sample and the second style image sample for style image reconstruction are selected from the style image samples, and the image processing network is used to convert the first style image sample into the target grayscale image sample. The image processing network processes the target grayscale image sample and the second style image sample to obtain the reconstructed style image.

[0109] After reconstructing the grayscale image sample and the current style image, the image loss information of the image sample set can be determined based on the current style image, the grayscale image sample, the reconstructed grayscale image sample, and the reconstructed style image. There are various ways of determining the image loss information. For example, the adversarial loss information of the image sample set can be determined according to the current style image and the target grayscale image sample, the grayscale image loss information of the image sample set can be determined based on the grayscale image sample and the reconstructed grayscale image sample, the style image loss information of the image sample set can be determined according to the first style image sample and the reconstructed style image, and the second style loss information of the image sample set can be determined based on the target style feature, the current style image, the reconstructed style image, and the second style image sample. The second style loss information, the adversarial loss information, the original image loss information, and the style image loss information are fused to obtain the image loss information of the image sample set.

[0110] Among them, there are various ways of determining the adversarial loss information of the image sample set. For example, an adversarial network (GAN) can be used to compare the current style image and the target grayscale image sample output by the generator with the input style image sample and grayscale image sample, and a discriminator is used to classify the input image and the output image. Based on the comparison between the classification result and the real result, the adversarial loss information of the image sample set can be obtained. Specifically, it can be as shown in formula (7):

[0111]

[0112] Among them, L GAN is the adversarial loss information, G is the grayscale image sample, is the output target grayscale image sample, F is the style image sample, is the output current style image.

[0113] The algorithm for adversarial loss information constrains the detailed description of facial features and other parts in the image. Through the generator of GAN, it ensures that the output images (the current style image and the target grayscale image sample) are as close as possible to the input image in terms of real performance. The discriminator ensures that the algorithm can clearly and accurately classify the input and output images. Under the joint action of the two losses with opposite optimization directions, the network can finally reach a balance and generate images as real as possible.

[0114] Among them, there are various ways to determine the grayscale image loss information of the image sample set. For example, calculate the image similarity between the grayscale image sample and the reconstructed grayscale image sample, and based on the image similarity, determine the similarity loss information of the image sample set. Since the similarity loss is mainly used to constrain the lines of the original image sample and the reconstructed original image sample, the similarity loss information can also be called the relaxed reconstruction loss information, which can be specifically shown in formula (8):

[0115]

[0116] Among them, L realxed-cyc is the similarity loss information, G is the target style image generator, F is the target original image sample generator, p is the original image sample, H is the image feature, and S is the style feature.

[0117] Perform pixel truncation on the grayscale image sample and the reconstructed grayscale image sample, and compare the pixel-truncated grayscale image sample with the pixel-truncated reconstructed grayscale image sample to obtain the pixel loss information of the image sample set. Integrate the similarity loss information and the pixel loss information to obtain the grayscale image loss information of the image sample set.

[0118] Among them, the similarity loss information can be understood as the relaxed reconstruction loss information of the grayscale image sample, mainly to relax the similarity between the grayscale image sample and the reconstructed grayscale image sample, mainly used to constrain the lines of the grayscale image sample and the reconstructed grayscale image sample, while the pixel loss information is mainly used to constrain the consistency of the output after pixel truncation processing of the grayscale image sample and the reconstructed grayscale image sample, and can also be understood as constraining the unity at the pixel level.

[0119] Among them, there are various ways to determine the style image loss information of the image sample set. For example, the style features of the first style image sample and the reconstructed style image are respectively extracted, and the feature distance between the extracted style features is calculated to obtain the first style feature distance. The image similarity between the first style image sample and the reconstructed style image is calculated. Based on the first style feature distance and the image similarity, the style image loss information of the image sample set is determined. Since the style image loss information is used to constrain the unification of the reconstructed style image and the first style image at the pixel level, the style image loss information can also be called strict reconstruction loss information, specifically as shown in formula (9):

[0120]

[0121] Among them, d is the first style image sample, G is the style image sample generator, G(F(d), S(d)) is the reconstructed style image, F is the original image sample generator, and S is the style feature.

[0122] Among them, the adversarial loss information, the grayscale image loss information, and the style image loss information can all be regarded as guarantees for the generation quality of the target style image. Therefore, the adversarial loss information, the grayscale image loss information, and the style image loss information can also be called quality loss information.

[0123] Among them, the style loss information is mainly used to narrow the loss information of the styles of the input and output images. There are various ways to determine the style loss information. For example, the trained style feature extraction network can be used to extract the style features of the current style image, the reconstructed style image, and the second style image sample respectively, and the extracted style features are adjusted according to the style feature pool to obtain the first style feature of the current style image, the second style feature of the reconstructed style image, and the third style feature of the second style image sample. The feature distance between the first style feature and the target style feature is calculated to obtain the second style feature distance. The feature distance between the second style feature and the third style feature is calculated to obtain the third style feature distance. Based on the second style feature distance and the third style feature distance, the second style loss information of the image sample set is determined, specifically as shown in formula (10):

[0124]

[0125] Among them, L style is the second style loss information, D s (a) is the current style image, is the style image sample corresponding to the target style feature, is the reconstructed style image, For the second-style image sample, p is the original image sample, and G is the target-style image generator. The calculated second-style loss information controls the style mainly by narrowing the style gap between the input and output images. Especially, the style of the input image can be any style rather than some specific inherent style types. Moreover, it is difficult to uniformly classify image styles into several fixed categories, which makes the style of the generated target-style image more diverse and accurate. Therefore, the flexibility and accuracy of image processing can be improved.

[0126] After determining the second-style loss information, adversarial loss information, grayscale image loss information, and style image loss information, the second-style loss information, adversarial loss information, grayscale image loss information, and style image loss information can be fused to obtain the image loss information of the image sample set. There are various fusion methods. For example, the adversarial loss information, grayscale image loss information, and style image loss information can be used as quality loss information and added to the style loss information to obtain the image loss information of the image sample set, as shown in the reference formula (11):

[0127] L total =L quality +L style (11)

[0128] Among them, L total is the image loss information, L quality is the quality loss information, and L style is the style loss information.

[0129] Among them, the quality loss information can be obtained by fusing the adversarial loss information, grayscale image loss information, and style image loss information. There are various fusion methods. For example, the weighted parameters of the grayscale image loss information and the style image loss information can be obtained respectively, and based on these weighted parameters, the original image loss information and the style image loss information are weighted respectively. Then, the weighted grayscale image loss information and the style image loss information are fused with the adversarial loss information to obtain the quality loss information, as shown in formula (12) specifically:

[0130] L quality =L GAN +λ1L relaxed +λ2L strict +λ3L trunc (12)

[0131] Among them, L quality is the quality loss information, L GAN is the adversarial loss information, L relaxed is the similarity loss information (relaxed reconstruction loss), and L truncis pixel loss information, L strict is style image loss information (strict reconstruction loss).

[0132] After determining the image loss information of the image sample set, the image processing network can be converged. There are various ways of convergence. For example, the gradient descent algorithm can be used to update the network parameters of the image processing network according to the image loss information, or other update algorithms can be used to update the network parameters of the image processing network according to the image loss information, so as to obtain the trained image processing network. The trained style feature extraction network and the trained image processing network are used as the trained image processing model.

[0133] It should be noted that during the training process of the image processing model, it can be as Figure 4 shown. The style image samples are data-augmented through the style image sample data augmentation module, and then the initial style features of the homologous style image samples after data augmentation are extracted by the sample style feature extraction module. The initial style features are clustered by the sample style pool processing module to obtain the style feature pool. The sample style feature remapping module is used to calculate the mapping vector of the initial style features to the style feature pool, so as to obtain the style features of the style image samples, and the style feature extraction network of the image processing model is converged according to the generated style features. A style image is generated based on the style image (portrait painting) generation module. After the style image is generated, the style image and the grayscale image corresponding to the original image sample are reconstructed. After obtaining the reconstructed style image and the reconstructed grayscale image, the image loss information of the image sample set is determined, and the image processing network of the preset image processing model is converged based on this image loss information, so as to obtain the trained image processing model. Among them, the core in the training process of the style feature extraction network is to extract the style features of the image samples, specifically as Figure 5 shown. By constructing the style feature pool and mapping the initial style to the style feature pool, on the one hand, the dimension of the style features can be effectively reduced, and on the other hand, by introducing the style feature pool composed of representative features, the interference caused by the style-irrelevant information in the initial style features can be effectively alleviated. During the training process of the image processing network, a decoupled loop structure is adopted to simultaneously implement two reverse generation loops for reconstructing the grayscale image samples and the style images. Reconstructing the grayscale image samples is used to determine the relaxed reconstruction loss information of the image sample set, and reconstructing the style image is used to determine the strict reconstruction loss information of the image sample set. For the relaxed reconstruction loss information, reference can be made to Figure 6As shown in (a) therein, the original image sample is grayscale processed, the grayscale image sample and the style image sample are input, the current style image is generated, and then the style of the current style image is restored to the style of the grayscale image sample, so as to realize the reconstruction of the grayscale image sample. Then, using the image similarity between the reconstructed grayscale image sample and the grayscale image sample, the relaxation reconstruction loss information of the image sample set can be determined. For the strict reconstruction loss information, reference can be made to Figure 6 As shown in (b) therein, the first style image sample is selected from the style image samples as the input current style image, the input current style image is reconstructed into the target grayscale image sample, and then the second style image sample is selected from the style image samples as the style image sample of the target style, and the target original image sample and the second style image sample are processed to obtain the reconstructed style image. The whole process of reconstructing the grayscale image sample and the reconstructed style image can form a double loop of input and output. Two different recurrent networks adopt different sample pairs corresponding to the original image sample (grayscale image sample) - style image sample, and the richness of the training data pairing combination is more conducive to generating a more accurate target style image.

[0134] Optionally, after obtaining the trained image processing model, the trained image processing model can also be used to process the original image to be processed and the style image to obtain the target style image.

[0135] Among them, the target style image is an image obtained by adjusting the style of the original image to be processed to the target style of the style image.

[0136] Among them, there can be various processes for processing the original image to be processed and the style image. Specifically, it can be as follows:

[0137] For example, the original image to be processed and the style image are obtained, and the trained image processing model is used to extract the features of the style image to obtain the current style features corresponding to the style image. The original image to be processed is grayscale processed to obtain the grayscale image to be processed, and the features of the grayscale image to be processed are extracted. Based on the current style features, the extracted image features are adjusted, and based on the adjusted image features, the target style image is generated.

[0138] Among them, there can be various ways to extract the features of the style image. For example, the current style features of the style image can be directly extracted by using the style feature extraction network of the trained image processing model, or the style feature extraction network can be used to extract the features of the style image to obtain the initial style features, and the initial style features are adjusted based on the preset style feature pool to obtain the current style features of the style image.

[0139] Among them, the main function of grayscale processing the image to be processed is to get rid of the irrelevant information brought by colors in the RGB image as much as possible, ensuring that the image processing model focuses on the parts directly related to the painting style of the style image (portrait painting), such as line strokes and shadow usage.

[0140] Among them, there are various ways to adjust the extracted image features based on the current style features. For example, the original style features can be extracted from the image features and replaced with the current style features. Or, the current style features can also be added to the image features to obtain the adjusted image features.

[0141] After adjusting the image features, a target style image can be generated based on the adjusted image features. There are various ways to generate the target style image. For example, the adjusted image features can be converted into image parameters, and the target style image can be drawn based on these image parameters. Or, the adjusted image features can also be directly restored to a style image to obtain the target style image.

[0142] Among them, it should be noted that taking the target style image as a portrait painting as an example, two assumptions that conform to the actual situation are proposed in this solution: that is, it only contains unlabeled data and data that has not been paired with the original image - portrait image, which enables getting rid of the need for a large amount of labeled and paired data in actual operations and reasonably using a large amount of unlabeled data; contrastive self - supervised learning is introduced to depict style features, which also echoes the aforementioned assumption: obtaining a large amount of unlabeled training data at low cost. Narrow the distance between each portrait painting and its enhanced image, and widen the distance between different portrait paintings. In this way, the style feature extraction network can explore the potential relationships between the input data; a style pool is used to map the style features extracted by any of the above - mentioned style extractors, ensuring that they are less interfered by style - irrelevant information. At the same time, by reducing the dimension of the overall feature space, the stability of the training process is ensured; a decoupled two - stream cyclic structure is used to ensure the integrity, vividness of the generated portrait painting and its closeness to a specific style.

[0143] As can be seen from the above, after obtaining the image sample set in the embodiment of the present invention, a preset image processing model is used to extract the style features of the style image samples to obtain the initial style features of the style image samples. Then, the initial style features are clustered to obtain a style feature pool corresponding to the image sample set. According to the style feature pool, the initial style features are adjusted to obtain the style features of the style image samples. Then, based on the style features of the style image samples and the original image samples, the preset image processing model is converged to obtain a trained image processing model. The trained image processing model is used to perform style conversion on the original image to be processed according to the style image. Since this solution clusters the extracted initial style features to obtain a style feature pool and uses the style feature pool to adjust the initial style features, the adjusted style features are less interfered by style-irrelevant information. Moreover, for artistic style images without texture information, the accuracy of the extracted style features can be guaranteed, and style images of any style can be generated. Therefore, the accuracy of image processing can be improved.

[0144] According to the method described in the above embodiment, the following will give an example for further detailed description.

[0145] In this embodiment, it will be described by taking the case where the image processing device is integrated in an electronic device, the image processing model has been trained, and the trained image processing model is used to process the original image to be processed and the style image as an example.

[0146] For example, the original image to be processed and the style image can be obtained, the style features corresponding to the style image are extracted, the original image to be processed is grayscaled to obtain the grayscale image to be processed, the image features of the original image to be processed are extracted, the image features are adjusted based on the style features, and based on the adjusted image features, a target style image is generated. The target style image is an image in which the style of the original image to be processed is adjusted to the target style of the style image.

[0147] Among them, there are multiple ways to obtain the original image to be processed and the style image. For example, the original image to be processed and the style image directly uploaded by the user through the terminal can be received, or the original image to be processed uploaded by the user through the terminal and the style image selected by the user from the preset style image set for image processing can be received. Or, an image processing request sent by the user through the terminal can also be received. The image processing request carries the storage address or image identifier of the original image to be processed and the style image. Based on the storage address or image identifier, the original image to be processed and the style image are obtained.

[0148] After obtaining the original image to be processed and the style image, the style features of the style image can be extracted. There are various ways to extract the style features. For example, the style feature extraction network of the trained image processing model can be used to directly extract the style features of the style image. Or, the initial style features of the style image can be extracted using the style feature extraction network, and then, based on the style feature pool, the initial style features can be adjusted to obtain the style features of the style image.

[0149] Among them, there are various ways to adjust the image features of the original image to be processed based on the style features. For example, the original style features can be extracted from the image features and replaced with the style features of the style image. Or, the style features can be added to the image features of the original image to be processed to obtain the adjusted image features.

[0150] After obtaining the adjusted image features, the target style image can be generated. There are various ways to generate the target style image. For example, the image processing network of the trained image processing model can be used to convert the adjusted image features into image parameters, and based on these image parameters, the target style image can be drawn. Or, the image processing network of the trained image processing model can be directly used to restore the adjusted image features to the style image to obtain the target style image.

[0151] Among them, it should be noted that the process of using the trained image processing model to process the original image to be processed can be as Figure 7 shown. The original image module is used to obtain the original image to be processed, the style feature extraction module is used to obtain the style image and extract the style template of the style image, and the style image generation module is used to generate the target style image. Taking the target style image as a portrait as an example, a real-person image to be processed needs to be input, and then a style image needs to be input or selected. Then, the trained image processing model processes the input real-person image to be processed and the style image, thereby converting the real-person image to be processed into an image with the target style of the style image to obtain the target style image. Specifically, it can be as Figure 8 shown.

[0152] According to the method described in the above embodiments, the following will give further detailed examples.

[0153] In this embodiment, it will be described by taking the image processing device being specifically integrated in an electronic device, the electronic device being a server, the original image being a real-person image, and the target style image being a portrait as an example.

[0154] As Figure 9 shown, an image processing method has the following specific process:

[0155] 201. The server obtains an image sample set.

[0156] For example, the server can directly receive at least one real - person image sample and one style image sample sent by the user through the terminal, so as to obtain an image sample set. Or, it can also directly obtain real - person image samples and style image samples from the Internet or an image database, so as to obtain an image sample set.

[0157] The server can also receive an image - processing request sent by the user through the terminal. The storage address of the image sample set is carried in the image - processing request. According to this storage address, real - person image samples and style image samples are obtained from the memory, cache or a third - party image database, so as to obtain an image sample set.

[0158] Optionally, after obtaining the image sample set from the memory, cache or a third - party image database, the server can also send a prompt message to the terminal. This prompt message is used to prompt the terminal that the image sample set has been obtained.

[0159] 202. The server uses a preset image - processing model to extract style features from the style image sample, and obtains the initial style features of the style image sample.

[0160] For example, the server can perform data augmentation methods such as randomly horizontally flipping the style image sample, image blurring or randomly graying the image, so as to obtain 2 homologous style image samples corresponding to the image sample. Use the style - feature extraction network of the preset image - processing model to perform operations such as convolution and pooling on the style image sample, and finally abstract it into the initial style features of a preset dimension corresponding to each homologous style image sample, so as to obtain the initial style features of the style image sample.

[0161] 203. The server clusters the initial style features to obtain a style feature pool corresponding to the image sample set.

[0162] For example, for all style picture samples X = {x1, x2, …, x n}, after data augmentation and style - feature extraction, the initial style features H = {h′1, h″1, h′2, h″2, …, h′ n , h″ n} are obtained, where h′ i , h″ i correspond to two data - augmented homologous style image samples of the input style image sample. Perform K - means clustering on these initial style features to obtain T clustering clusters. Then, calculate the average value of all initial style features in each clustering cluster to obtain the final T representative features, and form a style feature pool with these T representative features.

[0163] 204. The server adjusts the initial style features according to the style feature pool to obtain the style features of the style image sample.

[0164] For example, the server can determine the mapping relationship of the initial style features with respect to the style feature pool according to the style feature pool, fuse the initial style features of two homologous style image samples corresponding to the style image sample with the style feature pool, and then obtain the mapping vector of the initial style features with respect to the style feature pool. The calculation of the mapping vector of the initial style features with respect to the style feature values can refer to the formula shown in (2). The calculated mapping vector can be directly used as the style features of the style image sample, or the calculated mapping vectors can be fused to obtain the style features of the style image sample.

[0165] 205. The server converges the style feature extraction network based on the style features to obtain the trained style feature extraction network.

[0166] For example, the server classifies the style features to obtain the homologous style features and non - homologous style features corresponding to the style image sample. The cosine similarity between the style features can be calculated. The specific calculation formula can refer to the formula shown in (3). Then, the feature distance between the style features can be determined according to the cosine similarity. The specific calculation formula can refer to the formula shown in (4). Thus, the first feature distance between the homologous style features and the second feature distance between the homologous style features and the non - homologous style features can be obtained. Or, the Euclidean distance or other types of distances between the style features can also be calculated as the first feature distance between the homologous style features and the second feature distance between the homologous style features and the non - homologous features.

[0167] The server can fuse the second feature distances to obtain the non - homologous feature distance, add the non - homologous feature distance to the first feature distance to obtain the target feature distance, calculate the ratio of the first feature distance to the target feature distance respectively to obtain the feature distance ratio, and determine the first style loss information according to the feature distance ratio. The specific formula is as shown in formula (6). The style feature extraction network is converged according to the first style loss information to obtain the trained style feature extraction network.

[0168] 206. The server uses the trained style feature extraction network to extract features from the style image sample to obtain the target style features of the style image sample.

[0169] For example, the server can directly extract the style features from the style image sample using the trained style feature extraction network to obtain the target style features of the style image sample, or the server can also use the trained style feature extraction network to extract features from the style image sample to obtain the initial style features, and then use the style feature pool to adjust the initial style features to obtain the target style features of the style image sample.

[0170] 207. The server trains the image processing network according to the target style features and the real human image samples to obtain the trained image processing network, and uses the trained style feature extraction network and the trained image processing network as the trained image processing model.

[0171] For example, the server can perform grayscale processing on the real human image samples to obtain real human grayscale image samples, extract features from the real human grayscale image samples, adjust the extracted sample image features based on the target style features, and generate the current portrait based on the adjusted sample image features.

[0172] The server can use the image processing network to restore the style of the current portrait to the style of the real human grayscale image sample to obtain the reconstructed real human grayscale image sample, select the first style image sample and the second style image sample for style image reconstruction from the style image samples, use the image processing network to convert the first style image sample into the target real human grayscale image sample, and use the image processing network to process the target real human grayscale image sample and the second style image sample to obtain the reconstructed portrait.

[0173] The server can use the adversarial network (GAN) to compare the current portrait output by the generator and the target real human grayscale image sample with the input style image sample and the real human grayscale image sample, and use the discriminator to classify the input image and the output image, and compare the classification result with the real result to obtain the adversarial loss information of the image sample set, which can be specifically shown in formula (7).

[0174] The server calculates the image similarity between the real human grayscale image sample and the reconstructed real human grayscale image sample, and determines the similarity loss information of the image sample set based on the image similarity. The similarity loss information can also be called the relaxation reconstruction loss information, which can be specifically shown in formula (8).

[0175] Perform pixel truncation on the real human grayscale image sample and the reconstructed real human grayscale image sample, and compare the pixel-truncated real human grayscale image sample and the pixel-truncated reconstructed real human grayscale image sample to obtain the pixel loss information of the image sample set, and fuse the similarity loss information and the pixel loss information to obtain the real human image loss information of the image sample set.

[0176] The server extracts style features from the first style image and the reconstructed style image respectively, calculates the feature distance between the extracted style features to obtain the first style feature distance, calculates the image similarity between the first style image and the reconstructed style image, and determines the style image loss information of the image sample set based on the first style feature distance and the image similarity. The style image loss information can also be called the strict reconstruction loss information, which can be specifically expressed by formula (9).

[0177] Among them, the adversarial loss information, the real person image loss information, and the style image loss information can also be called the quality loss information.

[0178] The server can use the trained style feature extraction network to extract style features from the current style image, the reconstructed style image, and the second style image sample respectively, and adjust the extracted style features according to the style feature pool to obtain the first style feature of the current style image, the second style feature of the reconstructed style image, and the third style feature of the second style image sample. Then, it calculates the feature distance between the first style feature and the target style feature to obtain the second style feature distance, calculates the distance between the second style feature and the third style feature to obtain the second style feature distance, and determines the second style loss information of the image sample set based on the second style feature distance and the third style feature distance, which can be specifically expressed by formula (10).

[0179] The server can obtain the weighted parameters of the real person image loss information and the style image loss information respectively, and based on these weighted parameters, weight the real person image loss information and the style image loss information respectively. Then, it fuses the weighted real person image loss information and style image loss information with the adversarial loss information to obtain the quality loss information, which can be specifically expressed by formula (12).

[0180] The server can add the quality loss information to the second style loss information to obtain the image loss information of the image sample set, as shown in reference formula (11).

[0181] The server can use the gradient descent algorithm to update the network parameters of the image processing network according to the image loss information, or use other update algorithms to update the network parameters of the image processing network according to the image loss information, so as to obtain the trained image processing network. The trained style feature extraction network and the trained image processing network are used as the trained image processing model.

[0182] 208. The server uses the trained image processing model to process the real person image and the style image to be processed, and obtains the target portrait.

[0183] For example, the server obtains a real-person image to be processed and a style image, and directly extracts the current style features of the style image by using the style feature extraction network of the trained image processing model. Alternatively, the style feature extraction network can also be used to extract features from the style image to obtain initial style features, and the initial style features are adjusted based on a preset style feature pool to obtain the current style features of the style image.

[0184] The server performs grayscale processing on the real-person image to be processed to obtain a grayscale image to be processed, and extracts features from the grayscale image to be processed. The original style features are extracted from the image features, and the original style features are replaced with the current style features. Alternatively, the current style features can also be added to the image features to obtain adjusted image features.

[0185] The server converts the adjusted image features into image parameters, and draws a target portrait based on the image parameters. Alternatively, the adjusted image features can also be directly restored to a portrait to obtain the target portrait.

[0186] As can be seen from the above, after the server in this embodiment obtains the image sample set, it uses a preset image processing model to extract style features from the style image samples to obtain the initial style features of the style image samples. Then, the initial style features are clustered to obtain a style feature pool corresponding to the image sample set. According to the style feature pool, the initial style features are adjusted to obtain the style features of the style image samples. Then, based on the style features of the style image samples and the original image samples, the preset image processing model is converged to obtain a trained image processing model. This trained image processing model is used to perform style conversion on the original image to be processed according to the style image. Since this solution clusters the extracted initial style features to obtain a style feature pool, and uses the style feature pool to adjust the initial style features, the adjusted style features are less interfered by style-irrelevant information, and for artistic style images without texture information, it can also ensure the accuracy of the extracted style features, and can also generate style images of any style. Therefore, the accuracy of image processing can be improved.

[0187] To better implement the above method, an embodiment of the present invention further provides an image processing device. This image processing device can be integrated in an electronic device, such as a server or a terminal, etc. The terminal can include a tablet computer, a notebook computer, and / or a personal computer, etc.

[0188] For example, as Figure 10 shown, the image processing device can include an acquisition unit 301, an extraction unit 302, a clustering unit 303, an adjustment unit 304, and a convergence unit 305, as follows:

[0189] (1) Acquisition unit 301;

[0190] An acquisition unit 301, configured to acquire a set of image samples, where the set of image samples includes original image samples and style image samples.

[0191] For example, the acquisition unit 301 may be specifically configured to directly acquire the set of image samples, or, when the number of original image samples and style image samples in the set of image samples is large or the memory is large, indirectly acquire the set of image samples.

[0192] (2) An extraction unit 302;

[0193] The extraction unit 302 is configured to extract style features from the style image samples by using a preset image processing model to obtain initial style features of the style image samples.

[0194] For example, the extraction unit 302 may be specifically configured to perform data augmentation on the style image samples to obtain a plurality of homologous style image samples corresponding to the style image samples, and extract style features from the homologous style image samples by using a style feature extraction network of the preset image processing model to obtain initial style features of the style image samples.

[0195] (3) A clustering unit 303;

[0196] The clustering unit 303 is configured to cluster the initial style features to obtain a style feature pool corresponding to the set of image samples.

[0197] For example, the clustering unit 303 may be specifically configured to cluster the initial style features to obtain a plurality of style feature clusters, calculate the mean of the style features in each style feature cluster respectively to obtain a basic style feature corresponding to each style feature cluster, and fuse the basic style features to obtain a style feature pool corresponding to the set of image samples.

[0198] (4) An adjustment unit 304;

[0199] The adjustment unit 304 is configured to adjust the initial style features according to the style feature pool to obtain style features of the style image samples.

[0200] For example, the adjustment unit 304 may be specifically configured to determine a mapping relationship of the initial style features with respect to the style feature pool according to the style feature pool, and calculate a mapping vector of the initial style features to the style feature pool based on the mapping relationship to obtain style features of the style image samples.

[0201] (5) A convergence unit 305;

[0202] A convergence unit 305 is configured to converge a preset image processing model based on the style features of the style image sample and the original image sample, so as to obtain a trained image processing model, and the trained image processing model is used to perform style conversion on the original image to be processed according to the style image.

[0203] For example, the convergence unit 305 may specifically be configured to converge a style feature extraction network based on the style features, so as to obtain a trained style feature extraction network, use the trained style extraction network to extract features from the style image sample, obtain the target style features of the style image sample, train an image processing network according to the target style features and the original image sample, obtain a trained image processing network, and use the trained style feature network and the trained image processing network as the trained image processing model.

[0204] Optionally, the image processing device may further include a processing unit 306, as Figure 11 shown, and specifically may be as follows:

[0205] The processing unit 306;

[0206] The processing unit 306 is configured to process the original image to be processed and the style image by using the trained image processing model, so as to obtain a target style image, where the target style image is an image obtained by adjusting the style of the original image to be processed to the target style of the style image.

[0207] For example, the processing unit 306 may specifically be configured to obtain the original image to be processed and the style image, extract features from the style image by using the trained image processing model, so as to obtain the current style features corresponding to the style image. Perform gray processing on the original image to be processed to obtain a grayscale image to be processed, extract features from the grayscale image to be processed, adjust the extracted image features based on the current style features, and generate a target style image based on the adjusted image features.

[0208] In specific implementation, each of the above units may be implemented as an independent entity, or may be arbitrarily combined and implemented as the same or several entities. For the specific implementation of each of the above units, reference may be made to the foregoing method embodiments, which will not be elaborated herein.

[0209] As can be seen from the above, after the acquisition unit 301 in this embodiment acquires the image sample set, the extraction unit 302 uses a preset image processing model to extract the style features of the style image sample to obtain the initial style features of the style image sample. Then, the clustering unit 303 clusters the initial style features to obtain a style feature pool corresponding to the image sample set. The adjustment unit 304 adjusts the initial style features according to the style feature pool to obtain the style features of the style image sample. Then, the convergence unit 305 converges the preset image processing model based on the style features of the style image sample and the original image sample to obtain a trained image processing model. The trained image processing model is used to perform style conversion on the raw image to be processed according to the style image. Since this solution clusters the extracted initial style features to obtain a style feature pool and uses the style feature pool to adjust the initial style features, the adjusted style features are less interfered by style-irrelevant information. Moreover, for artistic style images without texture information, the accuracy of the extracted style features can be guaranteed, and style images of any style can be generated. Therefore, the accuracy of image processing can be improved.

[0210] An embodiment of the present invention further provides an electronic device, as Figure 12 shown, which shows a schematic structural diagram of the electronic device involved in the embodiment of the present invention. Specifically:

[0211] The electronic device may include a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, an input unit 404, and other components. Those skilled in the art can understand that Figure 12 the structural diagram of the electronic device shown in

[0212] does not limit the electronic device, and it may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements. Among them:

[0213] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the electronic device. In addition, the memory 402 can include high-speed random access memory and can also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. Correspondingly, the memory 402 can also include a memory controller to provide the processor 401 with access to the memory 402.

[0214] The electronic device further includes a power supply 403 for supplying power to each component. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, a recharge system, a power failure detection circuit, a power converter or inverter, and a power status indicator.

[0215] The electronic device may further include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0216] Although not shown, the electronic device may further include a display unit, etc., which will not be elaborated here. Specifically, in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402 to implement various functions as follows:

[0217] Obtain an image sample set, which includes original image samples and style image samples. Use a preset image processing model to extract style features from the style image samples to obtain the initial style features of the style image samples. Cluster the initial style features to obtain a style feature pool corresponding to the image sample set. According to the style feature pool, adjust the initial style features to obtain the style features of the style image samples. Based on the style features of the style image samples and the original image samples, converge the preset image processing model to obtain a trained image processing model, and this trained image processing model is used to perform style conversion on the original image to be processed according to the style image.

[0218] For example, an electronic device can directly obtain a set of image samples. Alternatively, when the number of original image samples and style image samples in the set of image samples is large or the memory is large, the set of image samples can also be obtained indirectly. Data augmentation is performed on the style image samples to obtain multiple homologous style image samples corresponding to the style image samples. The style feature extraction network of a preset image processing model is used to extract style features from the homologous style image samples to obtain the initial style features of the style image samples. Clustering is performed on the initial style features to obtain multiple style feature clusters, and the mean value of the style features in each style feature cluster is calculated respectively to obtain the basic style feature corresponding to each style feature cluster. The basic style features are fused to obtain a style feature pool corresponding to the set of image samples. According to the style feature pool, the mapping relationship of the initial style features with respect to the style feature pool is determined. Based on the mapping relationship, the mapping vector of the initial style features to the style feature pool is calculated to obtain the style features of the style image samples. Based on the style features, the style feature extraction network is converged to obtain a trained style feature extraction network. The trained style extraction network is used to extract features from the style image samples to obtain the target style features of the style image samples. The image processing network is trained according to the target style features and the original image samples to obtain a trained image processing network, and the trained style feature network and the trained image processing network are used as a trained image processing model. This trained image processing model is used to perform style conversion on the original image to be processed according to the style image.

[0219] For the specific implementation of each of the above operations, reference can be made to the previous embodiments and will not be elaborated here.

[0220] As can be seen from the above, in the embodiment of the present invention, after obtaining the set of image samples, a preset image processing model is used to extract style features from the style image samples to obtain the initial style features of the style image samples. Then, clustering is performed on the initial style features to obtain a style feature pool corresponding to the set of image samples. According to the style feature pool, the initial style features are adjusted to obtain the style features of the style image samples. Then, based on the style features of the style image samples and the original image samples, the preset image processing model is converged to obtain a trained image processing model. This trained image processing model is used to perform style conversion on the original image to be processed according to the style image. Since this solution performs clustering on the extracted initial style features to obtain a style feature pool and uses the style feature pool to adjust the initial style features, the adjusted style features are less affected by style-irrelevant information, and for artistic style images without texture information, the accuracy of the extracted style features can also be guaranteed, and style images of any style can be generated. Therefore, the accuracy of image processing can be improved.

[0221] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions or by controlling related hardware through instructions. The instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0222] To this end, an embodiment of the present invention provides a computer-readable storage medium storing multiple instructions that can be loaded by a processor to execute the steps in any one of the image processing methods provided by the embodiments of the present invention. For example, the instructions can perform the following steps:

[0223] Obtain an image sample set, where the image sample set includes original image samples and style image samples. Use a preset image processing model to extract style features from the style image samples to obtain initial style features of the style image samples. Cluster the initial style features to obtain a style feature pool corresponding to the image sample set. According to the style feature pool, adjust the initial style features to obtain the style features of the style image samples. Converge the preset image processing model based on the style features of the style image samples and the original image samples to obtain a trained image processing model, and the trained image processing model is used to perform style conversion on the original image to be processed according to the style image.

[0224] For example, directly obtain the image sample set, or when the number of original image samples and style image samples in the image sample set is large or the memory is large, the image sample set can also be obtained indirectly. Perform data augmentation on the style image samples to obtain multiple homologous style image samples corresponding to the style image samples. Use the style feature extraction network of the preset image processing model to extract style features from the homologous style image samples to obtain the initial style features of the style image samples. Cluster the initial style features to obtain multiple style feature clusters, calculate the mean of the style features in each style feature cluster respectively to obtain the basic style features corresponding to each style feature cluster, and fuse the basic style features to obtain the style feature pool corresponding to the image sample set. According to the style feature pool, determine the mapping relationship of the initial style features with respect to the style feature pool, and based on the mapping relationship, calculate the mapping vector of the initial style features to the style feature pool to obtain the style features of the style image samples. Converge the style feature extraction network based on the style features to obtain a trained style feature extraction network. Use the trained style extraction network to extract features from the style image samples to obtain the target style features of the style image samples. Train the image processing network according to the target style features and the original image samples to obtain a trained image processing network, and use the trained style feature network and the trained image processing network as the trained image processing model, and the trained image processing model is used to perform style conversion on the original image to be processed according to the style image.

[0225] For the specific implementation of each of the above operations, reference may be made to the previous embodiments and will not be elaborated herein.

[0226] Among them, the computer-readable storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, etc.

[0227] Since the instructions stored in the computer-readable storage medium can execute the steps in any one of the image processing methods provided in the embodiments of the present invention, the beneficial effects achievable by any one of the image processing methods provided in the embodiments of the present invention can be realized. For details, reference may be made to the previous embodiments and will not be elaborated herein.

[0228] Among them, according to one aspect of the present application, a computer program product or a computer program is provided. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the methods provided in various optional implementation manners in the above image processing aspect or portrait generation aspect.

[0229] The above has introduced in detail an image processing method, apparatus, electronic device, and computer-readable storage medium provided by the embodiments of the present invention. Specific examples are used herein to elaborate the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. An image processing method, characterized in that, Including: Obtain a set of image samples, where the set of image samples includes original image samples and style image samples; Use a preset image processing model to extract style features from the style image samples to obtain the initial style features of the style image samples, where the preset image processing model includes a style feature extraction network and an image processing network; Cluster the initial style features to obtain a style feature pool corresponding to the set of image samples; Adjust the initial style features according to the style feature pool to obtain the style features of the style image samples; Converge the style feature extraction network based on the style features to obtain a trained style feature extraction network; Use the trained style feature extraction network to extract features from the style image samples to obtain the target style features of the style image samples; Perform grayscale processing on the original image samples to obtain grayscale image samples; According to the target style features, use the image processing network to perform image processing on the grayscale image samples to obtain the current style image corresponding to the grayscale image samples; Use the image processing network to restore the style of the current style image to the style of the grayscale image samples to obtain a reconstructed grayscale image sample; Select a first style image sample and a second style image sample for style image reconstruction from the style image samples, and use the image processing network to convert the first style image sample into a target grayscale image sample; Use the image processing network to process the target grayscale image sample and the second style image sample to obtain a reconstructed style image; Based on the current style image, the grayscale image sample, the reconstructed grayscale image sample, and the reconstructed style image, determine the image loss information of the set of image samples, and converge the image processing network based on the image loss information to obtain a trained image processing network; Use the trained style feature extraction network and the trained image processing network as a trained image processing model, and the trained image processing model is used to perform style conversion on the original image to be processed according to the style image.

2. The image processing method according to claim 1, wherein The step of using a preset image processing model to extract style features from the style image samples to obtain the initial style features of the style image samples includes: Perform data augmentation on the style image samples to obtain multiple homologous style image samples corresponding to the style image samples; Use the style feature extraction network of the preset image processing model to extract style features from the homologous style image samples to obtain the initial style features of the style image samples.

3. The image processing method according to claim 2, wherein The step of clustering the initial style features to obtain a style feature pool corresponding to the set of image samples includes: Cluster the initial style features to obtain multiple style feature clusters; Calculate the mean of the style features in each style feature cluster respectively to obtain the basic style features corresponding to each style feature cluster; Fuse the basic style features to obtain a style feature pool corresponding to the set of image samples.

4. The image processing method according to claim 1, characterized in that Converging the style feature extraction network based on the style features to obtain a trained style feature extraction network, including: Classifying the style features to obtain the homologous style features and non-homologous style features corresponding to the style image sample, where the homologous style features are the style features corresponding to the homologous style image samples of the style image sample; Calculating the first feature distance between the homologous style features and the second feature distance between the homologous style features and the non-homologous style features respectively; Determining the first style loss information of the style image sample based on the first feature distance and the second feature distance, and converging the style feature extraction network according to the first style loss information to obtain a trained style feature extraction network.

5. The image processing method according to claim 1, characterized in that, Determining the image loss information of the image sample set based on the current style image, grayscale image sample, reconstructed grayscale image sample, and reconstructed style image, including: Determining the adversarial loss information of the image sample set according to the current style image and the target grayscale image sample; Determining the grayscale image loss information of the image sample set based on the grayscale image sample and the reconstructed grayscale image sample; Determining the style image loss information of the image sample set according to the first style image sample and the reconstructed style image; Determining the second style loss information of the image sample set based on the target style feature, current style image, reconstructed style image, and second style image sample; Fusing the second style loss information, adversarial loss information, grayscale image loss information, and style image loss information to obtain the image loss information of the image sample set.

6. The image processing method according to claim 5, characterized in that Determining the grayscale image loss information of the image sample set based on the grayscale image sample and the reconstructed original image sample, including: Calculating the image similarity between the grayscale image sample and the reconstructed grayscale image sample, and determining the similarity loss information of the image sample set based on the image similarity; Performing pixel truncation on the grayscale image sample and the reconstructed grayscale image sample, and comparing the pixel-truncated grayscale image sample and the pixel-truncated reconstructed grayscale image sample to obtain the pixel loss information of the image sample set; Fusing the similarity loss information and the pixel loss information to obtain the grayscale image loss information of the image sample set.

7. The image processing method according to claim 5, wherein Determining the style image loss information of the image sample set according to the first style image sample and the reconstructed style image, including: Performing style feature extraction on the first style image and the reconstructed style image respectively, and calculating the feature distance between the extracted style features to obtain the first style feature distance; Calculating the image similarity between the first style image and the reconstructed style image; Determining the style image loss information of the image sample set based on the first style feature distance and the image similarity.

8. The image processing method according to claim 5, wherein, Determining the second style loss information of the image sample set based on the target style feature, current style image, reconstructed style image, and second style image sample, including: The trained style feature extraction network is used to extract features from the current style image, the reconstructed style image, and the second style image sample respectively, and the extracted style features are adjusted according to the style feature pool to obtain the first style feature of the current style image, the second style feature of the reconstructed style image, and the third style feature of the second style image sample; Calculate the feature distance between the first style feature and the target style feature to obtain the second style feature distance, and calculate the feature distance between the second style feature and the third style feature to obtain the third style feature distance; Determine the second style loss information of the image sample set according to the second style feature distance and the third style feature distance.

9. The image processing method according to any one of claims 1 to 8, characterized in that, The adjusting the initial style feature according to the style feature pool to obtain the style feature of the style image sample includes: Determine the mapping relationship of the initial style feature with respect to the style feature pool according to the style feature pool; Based on the mapping relationship, calculate the mapping vector of the initial style feature to the style feature pool to obtain the style feature of the style image sample.

10. The image processing method according to any one of claims 1 to 8, characterized in that After the preset image processing model is converged based on the style feature of the style image sample and the original image sample to obtain the trained image processing model, it further includes: Obtain the original image to be processed and the style image, and use the trained image processing model to extract features from the style image to obtain the current style feature corresponding to the style image; Perform gray processing on the original image to be processed to obtain a gray image to be processed, and extract features from the gray image to be processed; Adjust the extracted image features based on the current style feature, and generate a target style image based on the adjusted image features.

11. An image processing apparatus, characterized in that, It includes: An acquisition unit for acquiring an image sample set, where the image sample set includes an original image sample and a style image sample; An extraction unit for extracting style features of the style image sample by using a preset image processing model to obtain the initial style features of the style image sample, where the preset image processing model includes a style feature extraction network and an image processing network; A clustering unit for clustering the initial style features to obtain a style feature pool corresponding to the image sample set; An adjustment unit for adjusting the initial style features according to the style feature pool to obtain the style features of the style image sample; A convergence unit is used to converge the style feature extraction network based on the style features to obtain a trained style feature extraction network. The trained style feature extraction network is used to extract features from the style image samples to obtain the target style features of the style image samples. The original image samples are grayscale processed to obtain grayscale image samples. According to the target style features, the image processing network is used to perform image processing on the grayscale image samples to obtain the current style images corresponding to the grayscale image samples. The image processing network is used to restore the style of the current style images to the style of the grayscale image samples to obtain the reconstructed grayscale image samples. The first style image samples and the second style image samples for style image reconstruction are selected from the style image samples, and the image processing network is used to convert the first style image samples into target grayscale image samples. The image processing network is used to process the target grayscale image samples and the second style image samples to obtain the reconstructed style images. Based on the current style images, the grayscale image samples, the reconstructed grayscale image samples, and the reconstructed style images, the image loss information of the image sample set is determined, and the image processing network is converged based on the image loss information to obtain a trained image processing network. The trained style feature extraction network and the trained image processing network are used as a trained image processing model, and the trained image processing model is used to perform style conversion on the original image to be processed according to the style image.

12. An electronic device, characterized in that, It includes a processor and a memory. The memory stores an application program, and the processor is used to run the application program in the memory to execute the steps in the image processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores multiple instructions, and the instructions are suitable for being loaded by the processor to execute the steps in the image processing method according to any one of claims 1 to 10.

14. A computer program product comprising computer instructions, characterized in that, The computer instructions are stored in the computer-readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the steps in the image processing method according to any one of claims 1 to 10.

Images