Image processing model training method, image processing method and related equipment

Abstract

The invention relates to an image processing model training method, an image processing method and related equipment, and the method comprises the steps: inputting a sample image into a to-be-trained image processing model and a pre-trained image processing model of an image generator, and enabling the model parameter quantity of the pre-trained image processing model to be larger than the model parameter quantity of the to-be-trained image processing model; the method comprises the following steps: performing feature extraction through a to-be-trained image processing model, generating a sample-processed image based on the extracted target feature, extracting a global feature based on an attention mechanism through a pre-trained image processing model, aligning the target feature and the global feature, and determining the target feature and the global feature based on the difference between the aligned related features. And alternately training the generator and the discriminator according to a discrimination result of the image discriminator to obtain a trained image processing model, and fixing model parameters of the pre-trained image processing model in an alternate training process. According to the invention, the image processing model has the global feature extraction capability, and the image processing quality is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing model training method, an image processing method and related equipment. Background Art

[0002] With the development of artificial intelligence technology, skin beautification functions based on image processing have been widely used in a variety of applications. The skin beautification function mainly identifies and analyzes skin problems in the image to be processed, and applies corresponding skin beautification algorithms to modify it, so as to remove skin problems such as acne, blemishes, and spots, making the skin look smoother, more even and younger.

[0003] However, the skin texture processing method for images in related technologies will lead to the loss of details of the skin area. For example, personal features such as moles and dimples on the face are easily misprocessed, and facial obstructions such as hands and masks are also easily misprocessed. In addition, the three-dimensional sense of the skin area is lost after processing, so that the skin texture beautification effect in complex scenes cannot meet expectations, and there are defects in the consistency of the overall skin area effect. Summary of the invention

[0004] The present disclosure provides an image processing model training method, an image processing method and related equipment to at least solve the problem that the skin texture processing method for images in the related art makes the skin texture beautification effect in complex scenes unable to achieve the expected effect, and the consistency of the overall skin area effect is defective. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing model training method, comprising:

[0006] Acquire a sample image and an expected processed image corresponding to the sample image, wherein the sample image includes a target skin area of ​​the subject, and the expected processed image indicates the target skin area where a processing effect meets an expected effect;

[0007] The sample image is input into the image processing model to be trained and the pre-trained image processing model of the image generator respectively, and the sample image is subjected to feature extraction by the image processing model to be trained, and a sample processed image is generated based on the extracted target features; the global features of the sample image are extracted by the pre-trained image processing model based on the attention mechanism; wherein the model parameter amount of the pre-trained image processing model is greater than the model parameter amount of the image processing model to be trained;

[0008] Aligning the target feature with the global feature to obtain an aligned target feature and an aligned global feature;

[0009] Inputting the sample processed image and the expected processed image into an image discriminator for discrimination processing to obtain a discrimination result;

[0010] Based on the difference between the aligned target features and the aligned global features, and the discrimination results, the image generator and the image discriminator are alternately trained until a preset training end condition is met to obtain a trained image processing model; wherein, the model parameters of the pre-trained image processing model are fixed during the alternating training process.

[0011] In some exemplary embodiments, the encoding network in the image processing model to be trained performs multi-scale local encoding on the sample image to obtain encoded image features, and the decoding network in the image processing model to be trained performs multi-scale local decoding based on the encoded image features to obtain decoded image features and the sample processed image; wherein the encoding network and the decoding network are jump-connected at the same scale;

[0012] The extracting the global features of the sample image based on the attention mechanism by the pre-trained image processing model comprises: extracting the global visual features of the sample image based on the attention mechanism by the pre-trained image processing model, and decoding based on the global visual features to obtain global decoded visual features;

[0013] The step of aligning the target feature with the global feature to obtain an aligned target feature and an aligned global feature comprises: aligning the encoded image feature and the decoded image feature with the global visual feature and the global decoded visual feature, respectively, to obtain a target encoded image feature, a target global visual feature, a target decoded image feature, and a target global decoded visual feature.

[0014] In some exemplary embodiments, the alternately training the image generator and the image discriminator based on the difference between the target encoded image feature and the target global visual feature, the difference between the target decoded image feature and the target global decoded visual feature, and the discrimination result includes:

[0015] Determine an alignment loss using an alignment loss function based on a difference between the target encoded image feature and the target global visual feature, and a difference between the target decoded image feature and the target global decoded visual feature;

[0016] Based on the discrimination result, determining a discrimination loss;

[0017] determining an image processing loss based on a difference between the sample processed image and the desired processed image;

[0018] Based on the alignment loss, the discrimination loss and the image processing loss, the image generator and the image discriminator are alternately trained.

[0019] In some exemplary embodiments, aligning the decoded image feature with the global decoded visual feature to obtain a target decoded image feature and a target global decoded visual feature comprises:

[0020] Mapping the decoded image features to the decoding space of the pre-trained image processing model to obtain intermediate decoded image features;

[0021] The intermediate decoded image features are aligned with the global decoded visual features to obtain target decoded image features and target global decoded visual features.

[0022] In some exemplary embodiments, performing multi-scale local encoding on the sample image through the encoding network in the image processing model to be trained to obtain the encoded image features includes:

[0023] The sample image is encoded sequentially by a plurality of local encoding units of different scales in the encoding network to obtain intermediate encoded image features output by each local encoding unit; the encoding process includes a convolution operation and a downsampling operation;

[0024] Performing convolution processing on the intermediate coded image features output by the last local coding unit to obtain a first convolution result;

[0025] The first convolution result is processed based on the channel attention mechanism and the spatial attention mechanism in sequence to obtain the encoded image feature.

[0026] In some exemplary embodiments, the first convolution result is processed based on the channel attention mechanism and the spatial attention mechanism in sequence to obtain the encoded image feature, including:

[0027] Processing the first convolution result based on the channel attention mechanism to obtain channel attention; fusing the channel attention with the first convolution result to obtain a fused convolution result;

[0028] The fused convolution result is processed based on the spatial attention mechanism to obtain spatial attention; the spatial attention is fused with the fused convolution result to obtain the encoded image feature.

[0029] In some exemplary embodiments, performing multi-scale local decoding based on the encoded image features by a decoding network in the image processing model to be trained to obtain decoded image features and a sample processed image includes:

[0030] The encoded image features are sequentially decoded by a plurality of local decoding units of different scales in the decoding network; wherein each local decoding unit performs upsampling and convolution operations on the features input to the decoding unit based on the intermediate encoded image features output by the local encoding unit of the corresponding scale;

[0031] Obtaining the sample processed image based on the decoding result output by the last local decoding unit;

[0032] A decoding result output by the second to last local decoding unit is obtained as the decoded image feature.

[0033] In some exemplary embodiments, extracting the global visual features of the sample image based on the attention mechanism by the pre-trained image processing model includes:

[0034] Extracting shallow global visual features and deep global visual features of the sample image respectively based on the attention mechanism through the pre-trained image processing model;

[0035] The decoding based on the global visual feature to obtain the global decoded visual feature includes:

[0036] fusing the shallow global visual feature and the deep global visual feature to obtain a fused global visual feature;

[0037] The fused global visual feature is decoded to obtain the global decoded visual feature.

[0038] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing method, including:

[0039] Acquire an image to be processed, wherein the image to be processed includes a target skin area of ​​the object;

[0040] Inputting the image to be processed into an image processing model for image processing to obtain a target processed image;

[0041] Wherein, the image processing model is trained based on any image processing model training method described in the first aspect above.

[0042] According to a third aspect of an embodiment of the present disclosure, there is provided an image processing model training device, comprising:

[0043] A sample image acquisition unit is configured to acquire a sample image and an expected processed image corresponding to the sample image, wherein the sample image includes a target skin area of ​​the object, and the expected processed image indicates the target skin area where a processing effect meets an expected effect;

[0044] The sample image processing unit is configured to execute the steps of inputting the sample image into a to-be-trained image processing model and a pre-trained image processing model of an image generator respectively, extracting features of the sample image through the to-be-trained image processing model, and generating a sample processed image based on the extracted target features; extracting global features of the sample image based on an attention mechanism through the pre-trained image processing model; wherein the model parameter amount of the pre-trained image processing model is greater than the model parameter amount of the to-be-trained image processing model;

[0045] an alignment unit, configured to perform an alignment process on the target feature and the global feature to obtain an aligned target feature and an aligned global feature;

[0046] A discriminating unit is configured to input the sample processed image and the expected processed image into an image discriminator for discriminating processing to obtain a discriminating result;

[0047] A training unit is configured to perform alternating training on the image generator and the image discriminator based on the difference between the aligned target features and the aligned global features, and the discrimination results until a preset training end condition is met, thereby obtaining a trained image processing model; wherein the model parameters of the pre-trained image processing model are fixed during the alternating training process.

[0048] In some exemplary embodiments, the sample image processing unit includes:

[0049] A first sample image processing unit is configured to perform multi-scale local encoding of the sample image through the encoding network in the image processing model to be trained to obtain encoded image features, and perform multi-scale local decoding based on the encoded image features through the decoding network in the image processing model to be trained to obtain decoded image features and the sample processed image; wherein the encoding network and the decoding network are jump-connected at the same scale;

[0050] A second sample image processing unit is configured to extract global visual features of the sample image based on an attention mechanism through the pre-trained image processing model, and decode the global visual features to obtain global decoded visual features;

[0051] The alignment unit is specifically configured to perform alignment processing on the encoded image features and the decoded image features with the global visual features and the global decoded visual features respectively, to obtain target encoded image features, target global visual features, target decoded image features and target global decoded visual features.

[0052] In some exemplary embodiments, the training unit includes:

[0053] an alignment loss determining unit, configured to determine an alignment loss using an alignment loss function based on a difference between the target encoded image feature and the target global visual feature, and a difference between the target decoded image feature and the target global decoded visual feature;

[0054] a discrimination loss determining unit, configured to determine a discrimination loss based on the discrimination result;

[0055] an image processing loss determining unit, configured to determine an image processing loss based on a difference between the sample processed image and the expected processed image;

[0056] The alternating training unit is configured to perform alternating training on the image generator and the image discriminator based on the alignment loss, the discriminant loss and the image processing loss.

[0057] In some exemplary embodiments, the alignment unit includes:

[0058] A decoding feature mapping unit, configured to perform mapping of the decoded image features to a decoding space of the pre-trained image processing model to obtain intermediate decoded image features;

[0059] The decoded image feature alignment unit is configured to perform alignment processing on the intermediate decoded image feature and the global decoded visual feature to obtain a target decoded image feature and a target global decoded visual feature.

[0060] In some exemplary embodiments, the first sample image processing unit includes:

[0061] A multi-scale coding unit, configured to perform coding processing on the sample image in sequence through a plurality of local coding units of different scales in the coding network to obtain intermediate coded image features output by each local coding unit; the coding processing includes a convolution operation and a downsampling operation;

[0062] A convolution processing unit, configured to perform convolution processing on the intermediate coded image features output by the last local coding unit to obtain a first convolution result;

[0063] The attention fusion unit is configured to process the first convolution result based on the channel attention mechanism and the spatial attention mechanism in sequence to obtain the encoded image feature.

[0064] In some exemplary embodiments, the attention fusion unit includes:

[0065] A channel attention fusion unit is configured to process the first convolution result based on the channel attention mechanism to obtain channel attention; and fuse the channel attention with the first convolution result to obtain a fused convolution result;

[0066] The spatial attention fusion unit is configured to process the fused convolution result based on the spatial attention mechanism to obtain spatial attention; and fuse the spatial attention with the fused convolution result to obtain the encoded image feature.

[0067] In some exemplary embodiments, the first sample image processing unit further includes:

[0068] A multi-scale decoding unit, configured to perform decoding processing on the encoded image features in sequence through a plurality of local decoding units of different scales in the decoding network; wherein each of the local decoding units performs upsampling and convolution operations on the features input to the decoding unit based on the intermediate encoded image features output by the local encoding unit of the corresponding scale;

[0069] A sample processed image determining unit, configured to execute a decoding result based on the last local decoding unit output to obtain the sample processed image;

[0070] The decoded image feature determination unit is configured to obtain the decoding result output by the penultimate local decoding unit as the decoded image feature.

[0071] In some exemplary embodiments, when the second sample image processing unit extracts the global visual features of the sample image through the pre-trained image processing model based on the attention mechanism, it is specifically configured to perform: extracting shallow global visual features and deep global visual features of the sample image respectively through the pre-trained image processing model based on the attention mechanism;

[0072] When the second sample image processing unit decodes based on the global visual feature to obtain the global decoded visual feature, the second sample image processing unit is specifically configured to execute: fusing the shallow global visual feature and the deep global visual feature to obtain the fused global visual feature; and decoding the fused global visual feature to obtain the global decoded visual feature.

[0073] According to a fourth aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:

[0074] A to-be-processed image acquisition unit, configured to acquire an to-be-processed image, wherein the to-be-processed image includes a target skin area of ​​the subject;

[0075] An image processing unit is configured to input the image to be processed into an image processing model for image processing to obtain a target processed image;

[0076] Wherein, the image processing model is trained based on any image processing model training method described in the first aspect above.

[0077] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0078] processor;

[0079] a memory for storing instructions executable by the processor;

[0080] Wherein, the processor is configured to execute the instructions to implement the image processing model training method of the first aspect mentioned above, or the image processing method of the second aspect mentioned above.

[0081] According to the sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image processing model training method of the first aspect or the image processing method of the second aspect.

[0082] According to the seventh aspect of the embodiments of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the image processing model training method of the first aspect or the image processing method of the second aspect.

[0083] The disclosed embodiment inputs a sample image into a to-be-trained image processing model and a pre-trained image processing model of an image generator respectively, extracts features of the sample image through the to-be-trained image processing model, generates a sample processed image based on the extracted target features, and extracts global features of the sample image through the pre-trained image processing model based on an attention mechanism, wherein the model parameter amount of the pre-trained image processing model is greater than the model parameter amount of the to-be-trained image processing model, and then aligns the target features with the global features to obtain aligned target features and aligned global features, and inputs the sample processed image and the expected processed image corresponding to the sample image into an image discriminator for discrimination to obtain a discrimination result, and alternately trains the image generator and the image discriminator based on the difference between the aligned target features and the aligned global features and the discrimination result until a preset training end condition is met to obtain a trained image processing model, and the model parameters of the pre-trained image processing model are fixed during the alternating training process. In the above technical scheme, a pre-trained image processing model is used to guide the training of the image processing model to be trained, so that the trained image processing model can better extract structural features with a large receptive field, enhance the recognition ability of large structures such as hands, masks, and facial dimples in the skin area, improve the skin beautification effect in complex scenes and the consistency of the overall effect, while maintaining lightweight and fast image processing capabilities, high operating efficiency, and suitable for mobile use.

[0084] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0086] Figure 1 is a schematic diagram of an application environment according to an exemplary embodiment;

[0087] Figure 2 is a flowchart of an image processing model training method according to an exemplary embodiment;

[0088] Figure 3 is a flowchart of another image processing model training method according to an exemplary embodiment;

[0089] Figure 4 is a flowchart of another image processing model training method according to an exemplary embodiment;

[0090] Figure 5 is an example of an image processing model training process shown according to an exemplary embodiment;

[0091] Figure 6 is a structural block diagram of an image processing model training device according to an exemplary embodiment;

[0092] Figure 7 is a structural block diagram of an image processing device according to an exemplary embodiment;

[0093] Figure 8 The present invention is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0094] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0095] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0096] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0097] See also Figure 1 , which is a schematic diagram of an application environment according to an exemplary embodiment, the application environment may include a terminal 110 and a server 120, and the terminal 110 and the server 120 may be connected via a wired network or a wireless network.

[0098] The terminal 110 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The terminal 110 may be installed with client software such as an application (Application, referred to as App) that provides image processing functions, wherein the image processing function includes a skin beautification function, such as facial skin beautification, and the application may be an independent application or a subroutine in the application. Exemplarily, the application may include a video application, a live broadcast application, etc. The user of the terminal 110 may log in to the application using pre-registered user information, which may include an account number and a password.

[0099] The server 120 may be a server that provides background services for the application in the terminal 110, and the background services may be image processing services. The server 120 may store a trained image processing model, and the server 120 may call the image processing model to implement image processing of the image to be processed, and the image processing includes adjusting the skin quality of the target skin area of ​​the object in the image to be processed to the desired skin quality. Exemplarily, the trained image processing model may be a Unet model with global feature extraction capability.

[0100] In a specific application scenario, the terminal 110 may send the image to be processed to the server 120, and the server 120 calls the above-mentioned image processing model to perform image processing on the image to be processed to obtain a target processed image, and returns the target processed image to the terminal 110. In another specific application scenario, the terminal 110 may download the image processing model from the server 120 in advance and store it locally, and then call the locally stored image processing model to process the image to be processed to obtain the target processed image during image processing.

[0101] In the disclosed embodiments, server 120 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, an edge computing node, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0102] Combine the following Figures 2 to 4 To introduce the training process of the image processing model of the embodiment of the present disclosure.

[0103] See also Figure 2 , which is a flowchart of an image processing model training method according to an exemplary embodiment. Figure 2 As shown, the following steps are included:

[0104] In step S201, a sample image and an expected processed image corresponding to the sample image are acquired, wherein the sample image includes a target skin area of ​​the object, and the expected processed image indicates the target skin area where the processing effect meets the expected effect.

[0105] The object may be a complete human body or a part of a human body, such as a human face, and the target skin area of ​​the object may be any skin area of ​​the object that needs to beautify the skin quality. For example, if the object is a human face and there is an obstruction such as a hand on the face, the target skin area of ​​the object is the unobstructed human face skin area.

[0106] In the disclosed embodiment, the expected processed image corresponding to the sample image can be understood as the true value of the target skin area in the beautified sample image. Specifically, the expected processed image can be one or a combination of multiple skin beautification effects such as blemish removal, skin smoothing, age reduction, dark circle removal, nasolabial fold removal, etc. on the target skin area.

[0107] The sample image may be any image in an image set used to train the image processing model.

[0108] In step S203, the sample image is respectively input into the image processing model to be trained and the pre-trained image processing model of the image generator, and features of the sample image are extracted by the image processing model to be trained, a sample processed image is generated based on the extracted target features, and global features of the sample image are extracted by the pre-trained image processing model based on the attention mechanism.

[0109] Among them, the model parameter amount of the pre-trained image processing model is greater than the model parameter amount of the image processing model to be trained. That is to say, the pre-trained image processing model serves as a teacher model, and the image processing model to be trained serves as a student model. The pre-trained teacher model is used to guide the feature learning of the student model, so that the image processing model to be trained has a global feature extraction capability similar to that of the teacher model.

[0110] Specifically, the pre-trained image processing model is an image processing model obtained after using a large amount of training data to extract global features based on the attention mechanism to perform skin beautification task training, wherein the attention mechanism may include a self-attention mechanism and / or a cross-attention mechanism. In a specific implementation, the pre-trained image processing model is a model that uses a large amount of facial training images to perform skin beautification task training, wherein the skin beautification task may include one or a combination of multiple skin beautification effects such as blemish removal, skin smoothing, age reduction, dark circle removal, nasolabial fold removal, etc.

[0111] Exemplarily, the image processing model to be trained is a lightweight convolutional neural network model based on an encoding-decoding structure, such as a Unet model. The pre-trained image processing model may be a pre-trained image processing model based on a Transform structure.

[0112] In step S205, the target feature is aligned with the global feature to obtain an aligned target feature and an aligned global feature.

[0113] The alignment process may include channel alignment and size alignment, that is, the two aligned features have the same number of channels and size.

[0114] In step S207, the sample processed image and the expected processed image are input into an image discriminator for discrimination processing to obtain a discrimination result.

[0115] Among them, the discrimination result indicates the probability that the sample processed image is false, and the probability that the expected processed image is true.

[0116] In step S209, based on the difference between the aligned target features and the aligned global features, and the discrimination results, the image generator and the image discriminator are alternately trained until a preset training end condition is met, thereby obtaining a trained image processing model.

[0117] Wherein, the model parameters of the pre-trained image processing model are fixed during the alternating training process.

[0118] The disclosed embodiment performs adversarial learning based on a generative adversarial network framework, wherein the generative adversarial network framework includes an image generator and an image discriminator, wherein the image generator includes the aforementioned image processing model to be trained and a pre-trained image processing model, wherein the image processing model to be trained serves as a student model, and the pre-trained image processing model serves as a teacher model, and the pre-trained image processing model has a global feature extraction capability, and during the adversarial learning process, the image generator and the image discriminator are alternately trained, wherein only the model parameters of the image processing model to be trained are updated for the parameter update of the image generator, and the model parameters of the pre-trained image processing model are fixed, and the training is terminated when a preset training termination condition is met, and the image processing model to be trained at the end of the training is used as the trained image processing model. The preset training termination condition may include the number of iterations reaching a preset iteration number threshold, and the preset iteration number threshold may be set based on actual needs, and may also include other termination conditions indicating that the image generator has converged.

[0119] In the above-mentioned embodiment, the image generator in the training stage includes a pre-trained pre-trained image processing model as a teacher model and an image processing model to be trained as a student model. Through adversarial learning between the image generator and the image discriminator, the teacher model guides the feature learning of the student model, so that the trained image processing model has a global feature extraction capability similar to the pre-trained image processing model, so that the trained image processing model can better extract structural features with a large receptive field, enhance the recognition ability of large structures such as skin area occluders such as hands, masks, and facial dimples, improve the skin beautification effect in complex scenes and the consistency of the overall effect, while maintaining lightweight and fast image processing capabilities, high operating efficiency, and suitable for mobile use.

[0120] In some exemplary embodiments, Figure 3 As shown, the above step S203 may include:

[0121] In step S301, the sample image is input into the image processing model to be trained of the image generator, the encoding network in the image processing model to be trained performs multi-scale local encoding on the sample image to obtain encoded image features, and the decoding network in the image processing model to be trained performs multi-scale local decoding based on the encoded image features to obtain decoded image features and a sample processed image;

[0122] Among them, the encoding network and the decoding network in the image processing model to be trained are skipped at the same scale. Skip connection is a simple and effective operation to fuse deep features and shallow features. Skip connection can add shallow features to deep features of the same scale at the element level. Specifically, features of different scales contain different feature information. The smaller the scale, the greater the depth, which belongs to deep features; conversely, the larger the scale, the smaller the depth, which belongs to shallow features. The receptive field of deep features is relatively large, with strong semantic information representation ability and weak geometric detail information representation ability, while the receptive field of shallow features is relatively small, with weak semantic information representation ability and strong geometric detail information representation ability. Among them, the receptive field is the size of the area mapped on the input image by the pixel points on the feature map output by each layer of the convolutional neural network.

[0123] In the disclosed embodiment, the image processing model to be trained is a lightweight convolutional neural network model based on an encoding-decoding structure, for example, a Unet model.

[0124] In step S303, the sample image is input into a pre-trained image processing model of the image generator, the global visual features of the sample image are extracted by the pre-trained image processing model based on an attention mechanism, and the global decoded visual features are obtained by decoding based on the global visual features.

[0125] Specifically, the pre-trained image processing model may include a hierarchical Transform encoder and a global decoder, wherein the hierarchical Transform encoder may generate high-resolution shallow global visual features and low-resolution deep global visual features based on an attention mechanism. For example, the hierarchical Transform encoder may be a 4-level Transform encoder, and the high-resolution shallow global visual features may be multi-level global visual features of 1 / 4 or 1 / 8 of the original image resolution, and the low-resolution deep global visual features may be multi-level global visual features of 1 / 16 or 1 / 32 of the original image resolution. It should be noted that the hierarchical Transform encoder extracts global features based on an attention mechanism at each level of the Transform layer.

[0126] Among them, the global decoder in the pre-trained image processing model can upsample the global visual features of each level extracted by the hierarchical Transform encoder and merge them along the channel dimension, and then perform two convolutions on the merged features to obtain the global decoded visual features. The global decoded visual features and the original image are added to obtain the image processed by the pre-trained image processing model.

[0127] Based on this, the above step 303 may include, when extracting the global visual features of the sample image based on the attention mechanism through the pre-trained image processing model:

[0128] The pre-trained image processing model is used to extract shallow global visual features and deep global visual features of the sample image based on the attention mechanism.

[0129] The above step S303 may include:

[0130] fusing the shallow global visual feature and the deep global visual feature to obtain a fused global visual feature;

[0131] The fused global visual feature is decoded to obtain the global decoded visual feature.

[0132] The fusion of the shallow global visual features and the deep global visual features may be performed by first upsampling and then fusing along the channel dimension to obtain the fused global visual features.

[0133] The fused global visual feature may be decoded by sequentially performing convolutions through two serially connected convolutional layers, and the result of the convolution is used as the global decoded visual feature.

[0134] In the above implementation, the shallow global visual features and deep global visual features of the sample image are extracted respectively based on the attention mechanism through the pre-trained image processing model, and the shallow global visual features and the deep global visual features are first fused when decoding based on the global visual features, and then the fused global visual features are decoded to obtain the global decoded visual features, which can improve the representation accuracy of the global visual features and the global decoded visual features, and thus help to improve the training effect of the image processing model to be trained.

[0135] Based on this, the aforementioned step S205 may include: Figure 3 Step S305: Align the encoded image features and the decoded image features with the global visual features and the global decoded visual features respectively to obtain target encoded image features, target global visual features, target decoded image features and target global decoded visual features.

[0136] Specifically, the coded image features and the global visual features are aligned to obtain aligned coded image features and aligned global visual features, wherein the aligned coded image features are referred to as target coded image features, and the aligned global visual features are referred to as target global visual features. The decoded image features and the global decoded visual features are aligned to obtain aligned decoded image features and aligned global decoded visual features, wherein the aligned decoded image features are referred to as target decoded image features, and the aligned global decoded visual features are referred to as target global decoded visual features.

[0137] In some exemplary embodiments, considering that there are significant differences between the decoding space of the image processing model to be trained and the decoding space of the pre-trained image processing model, if the decoded image features obtained by decoding the image processing model to be trained and the global decoded visual features obtained by decoding the pre-trained image processing model are directly aligned, the training effect of the image processing model to be trained is limited. In order to further improve the training effect of the image processing model to be trained, the above step S305 may include the following when aligning the decoded image features with the global decoded visual features to obtain the target decoded image features and the target global decoded visual features:

[0138] Mapping the decoded image features to the decoding space of the pre-trained image processing model to obtain intermediate decoded image features;

[0139] The intermediate decoded image features are aligned with the global decoded visual features to obtain target decoded image features and target global decoded visual features.

[0140] In a specific implementation, the decoded image features can be mapped to the decoding input dimensions of the pre-trained image processing model through a mapping layer, and then the mapped results can be input into the decoder of the pre-trained image processing model for decoding to obtain output decoded features, which can be used as intermediate decoded image features and then aligned with the global decoded visual features obtained by decoding the global visual features of the pre-trained image processing model.

[0141] In the above embodiment, the decoded image features obtained by decoding the image processing model to be trained are mapped to the decoding space of the pre-trained image processing model to obtain intermediate decoded image features, and then the intermediate decoded image features and the global decoded visual features are aligned. This can improve the recognition ability of the subsequently trained image processing model for large structures on the target skin area (such as hands blocking the face, masks, etc.).

[0142] Based on this, during the specific implementation of the aforementioned step S209, the image generator and the image discriminator can be alternately trained based on the difference between the target encoded image features and the target global visual features, the difference between the target decoded image features and the target global decoded visual features, and the discrimination results until the preset training end conditions are met, thereby obtaining a trained image processing model.

[0143] In the above implementation, the encoding network in the image processing model to be trained performs multi-scale local encoding on the sample image to obtain the encoded image features, and the decoding network in the image processing model to be trained performs multi-scale local decoding based on the encoded image features to obtain the decoded image features, and the encoding network and the decoding network are jump-connected at the same scale, so that the encoded image features and the decoded image features both contain richer information. Then, after the global visual features of the sample image are extracted based on the attention mechanism by the pre-trained image processing model, and the global decoded visual features are decoded based on the global visual features to obtain the global decoded visual features, the encoded image features and the decoded image features are respectively compared with the global visual features. The target encoding image features, target global visual features, target decoding image features and target global decoding visual features are aligned to obtain target encoding image features, target global visual features, target decoding image features and target global decoding visual features, and subsequent alternating training can be based on the difference between more refined target encoding image features and target global visual features, and the difference between target decoding image features and target global decoding visual features. This is conducive to further improving the trained image processing model's ability to extract structural features of a larger receptive field, further improving the skin beautification effect in complex scenes and the consistency of the overall effect, while maintaining lightweight and fast image processing capabilities, high operating efficiency, and suitable for mobile use.

[0144] In some exemplary embodiments, see Figure 3 , when performing the alternate training of the image generator and the image discriminator based on the difference between the aligned target feature and the aligned global feature and the discrimination result, the above step S209 may include:

[0145] Determine an alignment loss using an alignment loss function based on a difference between the target encoded image feature and the target global visual feature, and a difference between the target decoded image feature and the target global decoded visual feature;

[0146] Based on the discrimination result, determining a discrimination loss;

[0147] determining an image processing loss based on a difference between the sample processed image and the desired processed image;

[0148] Based on the alignment loss, the discrimination loss and the image processing loss, the image generator and the image discriminator are alternately trained.

[0149] Specifically, the alignment loss function can be a cross entropy loss, a mean square error loss, or a distillation learning loss. For example, the alignment loss function L cwd It can be expressed as follows:

[0150]

[0151] Where C represents the number of channels, c = 1, 2, ..., C; i represents a specific pixel, H and W represent the height and width of the image; T is a hyperparameter; represents the activation function, x C Take x T and x S , where x T represents the features corresponding to the pre-trained image processing model, x S Represents the features corresponding to the image processing model to be trained.

[0152] In the specific implementation, the target encoded image features are used as x in the above alignment loss function S , and use the target global visual feature as x in the above alignment loss function T , the first alignment loss can be obtained by using the above alignment loss function; in addition, the x in the above alignment loss function of the target decoded image feature S , and the target global decoded visual features are used as x in the above alignment loss function T , the second alignment loss can be obtained by using the above alignment loss function, and then the alignment loss can be obtained by combining the sum of the first alignment loss and the second alignment loss.

[0153] Specifically, the discriminative loss Loss1 can be expressed using the following adversarial loss function:

[0154]

[0155] Where y represents the sample processed image; x represents the expected processed image; z represents random training noise data; G represents the image processing model to be trained in the image generator; D represents the image discriminator;

[0156] Specifically, the image processing loss may include image consistency loss and perceptual loss, wherein the image consistency loss may be expressed as Loss1=||P1-P2||1, wherein ||*||1 represents the 1-norm, P1 represents the sample processed image, and P2 represents the expected processed image; the perceptual loss may be expressed as Loss2==||F1-F2||2, wherein ||*||2 represents the 2-norm, F1 represents the first feature map corresponding to the sample processed image, and F2 represents the second feature map corresponding to the expected processed image.

[0157] Then, when the image generator and the image discriminator are alternately trained based on the alignment loss, the discrimination loss and the image processing loss, in one iteration, the parameters of the image discriminator can be adjusted based on the discrimination loss Loss1 until the image discriminator converges, and then the total loss is determined by the alignment loss, the discrimination loss and the image processing loss, and the model parameters of the image processing model to be trained in the image generator are adjusted based on the total loss, and the model parameters of the pre-trained image processing model in the image generator are fixed until the image generator converges, where the total loss can be obtained by weighted summation of the alignment loss, the discrimination loss and the image processing loss.

[0158] In the above implementation, the image generator and the image discriminator are alternately trained based on alignment loss, discrimination loss and image processing loss, which not only improves the image generation ability of the image processing model to be trained in the image generator, but also enables it to learn the global feature extraction ability under the guidance of the pre-trained image processing model as a teacher model, and has a global receptive field, thereby improving the processing effect of the trained image processing model.

[0159] In some exemplary embodiments, in order to further expand the receptive field of the trained image processing model, the trained image processing model has the same feature long-distance dependency capability and global receptive field as the pre-trained image processing model, such as Figure 4 As shown, the aforementioned step S301 may include:

[0160] In step S401, the sample image is encoded in sequence by a plurality of local encoding units of different scales in the encoding network to obtain intermediate encoded image features output by each local encoding unit.

[0161] The encoding process of each local coding unit includes a convolution operation and a downsampling operation.

[0162] Specifically, different scales of local coding units in the coding network may be set based on actual needs, for example, may include three different scales.

[0163] In step S403, convolution processing is performed on the intermediate coded image features output by the last local coding unit to obtain a first convolution result.

[0164] In step S405, the first convolution result is processed based on the channel attention mechanism and the spatial attention mechanism in turn to obtain the encoded image feature.

[0165] Among them, the channel attention mechanism is used to compress the spatial dimension while keeping the channel dimension unchanged, and the spatial attention mechanism is used to compress the channel dimension while keeping the spatial dimension unchanged.

[0166] In the above implementation, local coding units of multiple different scales in the coding network sequentially perform convolution operations and downsampling operations on the sample image to obtain intermediate coded image features output by each local coding unit, and perform convolution processing on the intermediate coded image features output by the last local coding unit to obtain a first convolution result, and then sequentially process the first convolution result based on the channel attention mechanism and the spatial attention mechanism to obtain the coded image features, thereby alleviating the semantic difference between the coded image features and the global visual features output by the pre-trained image processing model, so that the trained image processing model has the same feature long-distance dependency capability and global receptive field as the pre-trained image processing model, thereby improving the performance of the trained image processing model.

[0167] Specifically, the first convolution result is processed based on the channel attention mechanism and the spatial attention mechanism to obtain the encoded image features, which may include:

[0168] Processing the first convolution result based on the channel attention mechanism to obtain channel attention; fusing the channel attention with the first convolution result to obtain a fused convolution result;

[0169] The fused convolution result is processed based on the spatial attention mechanism to obtain spatial attention; the spatial attention is fused with the fused convolution result to obtain the encoded image feature.

[0170] Specifically, the first convolution result passes through two parallel maximum pooling layers and average pooling layers to change the first convolution result of C*H*W to the size of C*1*1, and then passes through a shared multi-layer perception module, in which it first compresses the number of channels and then expands the original number of channels. After the ReLU activation function, two activated results are obtained. The two activated results are added element by element and then passed through a sigmoid activation function to obtain the channel attention output. The channel attention is multiplied with the first convolution result for fusion and converted back to the fused convolution result of the size of C*H*W.

[0171] Specifically, the fused convolution result of size C*H*W is subjected to maximum pooling and average pooling to obtain two 1*H*W feature maps, and then the two 1*H*W feature maps are concatenated (Concat operation). The concatenated result is convolved into a 1-channel feature map, and then a sigmoid is passed to obtain the spatial attention output. The spatial attention is multiplied with the fused convolution result for fusion and converted back to the encoded image feature of size C*H*W.

[0172] In the above implementation, by performing attention operations in the channel dimension and the spatial dimension in sequence, the receptive field of the encoded image features is expanded, which is beneficial to improving the training effect of the image processing model to be trained.

[0173] Continue to see Figure 4 The aforementioned step S301 may include, when performing multi-scale local decoding based on the encoded image features by the decoding network in the image processing model to be trained to obtain the decoded image features and the sample processed image:

[0174] In step S407, the encoded image features are decoded in sequence by a plurality of local decoding units of different scales in the decoding network.

[0175] Each of the local decoding units performs upsampling and convolution operations on features input to the decoding unit based on features of an intermediate coded image output by a local coding unit of a corresponding scale.

[0176] Specifically, the decoding network is symmetrical to the encoding network, that is, a plurality of local decoding units of different scales in the decoding network correspond one-to-one to a plurality of local encoding units of different scales in the encoding network.

[0177] In step S409, the sample processed image is obtained based on the decoding result output by the last local decoding unit; and the decoding result output by the second to last local decoding unit is obtained as the decoded image feature.

[0178] In the above implementation, the encoded image features are decoded in sequence by a plurality of local decoding units of different scales in the decoding network, and a sample processed image is obtained based on the decoding result output by the last local decoding unit, and the decoding result output by the second to last local decoding unit is obtained as the decoded image feature. This can improve the matching of the decoded image feature with the global decoded visual feature output by the pre-trained image processing model, thereby improving the model training efficiency.

[0179] In order to facilitate understanding of the training process of the embodiment of the present disclosure, Figure 5 An exemplary illustration of the training process is shown below.

[0180] like Figure 5As shown, the network structure used in the training phase in the embodiment of the present disclosure adopts a GAN (Generative Adversarial Nets, Generative Adversarial Network) network, including an image generator and an image discriminator. Among them, the image generator is composed of an image processing model to be trained and a pre-trained image processing model. The image processing model to be trained in the image generator is a Unet model, and the pre-trained image processing model in the image generator is a Transform structure model. The Transform structure model is composed of an Encoder and a Decoder. The Transform structure model is pre-trained for the skin texture beautification task using training data (facial image-facial image after desired beautification), and then all weights are frozen and only used in the training phase of the embodiment of the present disclosure, and not used in the reasoning phase. The Encoder in the Transform structure model can be composed of 4 hierarchical Transformer Blocks in the semantic segmentation model SegFormer, and the outputs of the last two TransformerBlocks are taken as global visual features for subsequent alignment with the encoded image features of the Unet model; the Decoder in the Transform structure model is composed of the features extracted by the 4 Transformer Blocks of the Encoder, which are merged along the channel dimension after upsampling and then input into two convolutional layers for convolution processing in sequence to obtain the output global decoded visual features. The Unet model in the image generator is composed of multiple residual modules Res Block and two middle layers of attention fusion modules, where the output of the attention fusion module is used as the encoded image feature.

[0181] In order to make the trained Unet model obtain image generation capabilities similar to Transform, the Transform structure model is used as the teacher network in the training stage of the disclosed embodiment, and the Unet model to be trained is used as the student network, and the alignment module is used to make the Unet model have facial skin beautification capabilities similar to Transform. The alignment module includes a first alignment module and a second alignment module, the first alignment module is used to align the encoded image features and the global visual features, and the second alignment module is used to align the decoded image features and the global decoded visual features.

[0182] Specifically, the two features of the Unet model extracted by the attention fusion module and the features extracted by the 3rd and 4th Transformer Blocks in the Transform structure model are respectively input into the first alignment module for alignment, so that the two features of the Unet model are respectively obtained from H C xW C xC C Convert to HxWxCC The features extracted by the Transformer Block are also converted to HxWxX C , and then through the aforementioned alignment loss function L cwd Determine the first alignment loss.

[0183] The features obtained by the penultimate residual module Res Block of the Unet model are taken out, mapped to the decoding space of the Decoder in the Transform structure model through the mapping layer to obtain the decoded image features, and then the decoded image features are input into the second alignment module. At the same time, the global decoded visual features extracted by the Decoder in the Transform structure model for the sample image are also input into the second alignment module to align the decoded image features with the global decoded visual features, and the alignment loss function L is used as mentioned above. cwd Determine the second alignment loss.

[0184] Specifically, the attention fusion module in the Unet model is mainly composed of a convolutional layer and a convolutional block attention module (CBAM). The features extracted by the convolutional layer are input into the convolutional attention module to obtain the encoded image features output by the convolutional attention module. The convolutional attention module adopts a channel attention mechanism and a spatial attention mechanism, so that the output encoded image features have a global receptive field, which narrows the semantic difference between Transform and Unet, and is conducive to the subsequent use of the alignment module to extract the Unet model feature extraction and facial beautification capabilities. In addition, the attention fusion module is relatively lightweight and easy to deploy on mobile terminals.

[0185] In addition, the sample image input to the Unet model in the image generator is processed by the Unet model to output a sample processed image. The sample processed image and the corresponding expected processed image are used as the input of the image discriminator, which performs authenticity discrimination processing and outputs the discrimination result, and then determines the discrimination loss through the aforementioned adversarial loss function. In addition, the image processing loss and the perceptual loss are further determined based on the difference between the sample processed image and the corresponding expected processed image. Finally, the image generator and the image discriminator are alternately trained based on the alignment loss, the discrimination loss, the image processing loss and the perceptual loss. Specifically, in one iteration, the parameters of the image generator are first fixed, and the image discriminator is trained until it converges through the discrimination loss, and then the parameters of the image discriminator are fixed, and the image generator is trained until it converges through the alignment loss, the discrimination loss, the image processing loss and the perceptual loss. When training the image generator, only the parameters of the Unet model in the image generator are adjusted, and the parameters of the Transform structure model are fixed. After the training is completed, the trained Unet model in the image generator is used as the image processing model for online skin texture beautification.

[0186] In the above implementation, the Transform structure model used only in the training phase is introduced to assist in enhancing the feature extraction capability of the image processing model used for online skin beautification, and the attention fusion module is added to the image processing model to expand the receptive field of feature extraction, so that the trained image processing model can capture longer-distance dependencies, greatly improving the skin beautification quality and overall facial consistency of the trained image processing model in complex scenes, and can have a variety of skin beautification effects such as removing facial blemishes, smoothing skin and reducing age, removing dark circles, and removing nasolabial folds. In addition, the trained image processing model is very lightweight and requires only a small amount of computing resources (computational volume is less than 100 MB). On most mobile phones, it can achieve real-time operation at 30 frames per second, making it widely used in real-time scenarios such as live broadcasts.

[0187] The present disclosure also provides an image processing method, which specifically includes:

[0188] Acquire an image to be processed, wherein the image to be processed includes a target skin area of ​​the object;

[0189] Inputting the image to be processed into an image processing model for image processing to obtain a target processed image;

[0190] The image processing model is trained based on any of the image processing model training methods described in the embodiments of the present disclosure. Exemplarily, the image processing model may be a Unet model trained based on the image processing model training method of the embodiments of the present disclosure.

[0191] Specifically, the image to be processed is input into the encoding network of the image processing model, and the local encoding units of different scales in the encoding network sequentially encode the image to be processed (including convolution operations and downsampling operations), and the intermediate encoded image features output by each local encoding unit are obtained. The intermediate encoded image features output by the last local encoding unit are subjected to two attention fusions to obtain the output fusion result, and the fusion result is input into the decoding network of the image processing model, and the local decoding units of different scales in the decoding network sequentially decode the fusion result (including upsampling operations and convolution operations), and the target processed image output by the image processing model is obtained. Among them, the attention fusion includes convolution processing, and the results of the convolution processing are sequentially fused based on the channel attention mechanism and the spatial attention mechanism.

[0192] Exemplarily, the image to be processed may be a facial image, and the target skin area of ​​the object may be a facial skin area.

[0193] The image processing model in the above implementation has a global feature extraction capability similar to Transform, and thus has the ability to accurately recognize target skin areas such as facial obstructions such as hands, masks, and large structures such as facial dimples, thereby improving the skin beautification effect and the consistency of the overall facial effect in complex scenes. It also has lightweight and fast image processing capabilities, is suitable for mobile terminals, and can run smoothly in real time (30fps) on most mobile devices, which is very suitable for real-time scenes such as live broadcasts.

[0194] Figure 6 is a structural block diagram of an image processing model training device according to an exemplary embodiment. Figure 5 , the image processing model training device 600 comprises:

[0195] The sample image acquisition unit 610 is configured to acquire a sample image and an expected processed image corresponding to the sample image, wherein the sample image includes a target skin area of ​​the object, and the expected processed image indicates the target skin area where the processing effect meets the expected effect;

[0196] The sample image processing unit 620 is configured to execute the steps of inputting the sample image into the image processing model to be trained and the pre-trained image processing model of the image generator respectively, extracting features of the sample image through the image processing model to be trained, and generating a sample processed image based on the extracted target features; extracting global features of the sample image based on the attention mechanism through the pre-trained image processing model; wherein the model parameter amount of the pre-trained image processing model is greater than the model parameter amount of the image processing model to be trained;

[0197] An alignment unit 630 is configured to perform an alignment process on the target feature and the global feature to obtain an aligned target feature and an aligned global feature;

[0198] The discrimination unit 640 is configured to input the sample processed image and the expected processed image into an image discriminator for discrimination processing to obtain a discrimination result;

[0199] The training unit 650 is configured to perform alternating training on the image generator and the image discriminator based on the difference between the aligned target features and the aligned global features, and the discrimination results until a preset training end condition is met, so as to obtain a trained image processing model; wherein the model parameters of the pre-trained image processing model are fixed during the alternating training process.

[0200] In some exemplary embodiments, the sample image processing unit 620 includes:

[0201] A first sample image processing unit is configured to perform multi-scale local encoding of the sample image through the encoding network in the image processing model to be trained to obtain encoded image features, and perform multi-scale local decoding based on the encoded image features through the decoding network in the image processing model to be trained to obtain decoded image features and the sample processed image; wherein the encoding network and the decoding network are jump-connected at the same scale;

[0202] A second sample image processing unit is configured to extract global visual features of the sample image based on an attention mechanism through the pre-trained image processing model, and decode the global visual features to obtain global decoded visual features;

[0203] The alignment unit 630 is specifically configured to perform alignment processing on the encoded image features and the decoded image features with the global visual features and the global decoded visual features, respectively, to obtain target encoded image features, target global visual features, target decoded image features and target global decoded visual features.

[0204] In some exemplary embodiments, the training unit 650 includes:

[0205] an alignment loss determining unit, configured to determine an alignment loss using an alignment loss function based on a difference between the target encoded image feature and the target global visual feature, and a difference between the target decoded image feature and the target global decoded visual feature;

[0206] a discrimination loss determining unit, configured to determine a discrimination loss based on the discrimination result;

[0207] an image processing loss determining unit, configured to determine an image processing loss based on a difference between the sample processed image and the expected processed image;

[0208] The alternating training unit is configured to perform alternating training on the image generator and the image discriminator based on the alignment loss, the discriminant loss and the image processing loss.

[0209] In some exemplary embodiments, the alignment unit 630 includes:

[0210] A decoding feature mapping unit, configured to perform mapping of the decoded image features to a decoding space of the pre-trained image processing model to obtain intermediate decoded image features;

[0211] The decoded image feature alignment unit is configured to perform alignment processing on the intermediate decoded image feature and the global decoded visual feature to obtain a target decoded image feature and a target global decoded visual feature.

[0212] In some exemplary embodiments, the first sample image processing unit includes:

[0213] A multi-scale coding unit, configured to perform coding processing on the sample image in sequence through a plurality of local coding units of different scales in the coding network to obtain intermediate coded image features output by each local coding unit; the coding processing includes a convolution operation and a downsampling operation;

[0214] A convolution processing unit, configured to perform convolution processing on the intermediate coded image features output by the last local coding unit to obtain a first convolution result;

[0215] The attention fusion unit is configured to process the first convolution result based on the channel attention mechanism and the spatial attention mechanism in sequence to obtain the encoded image feature.

[0216] In some exemplary embodiments, the attention fusion unit includes:

[0217] A channel attention fusion unit is configured to process the first convolution result based on the channel attention mechanism to obtain channel attention; and fuse the channel attention with the first convolution result to obtain a fused convolution result;

[0218] The spatial attention fusion unit is configured to process the fused convolution result based on the spatial attention mechanism to obtain spatial attention; and fuse the spatial attention with the fused convolution result to obtain the encoded image feature.

[0219] In some exemplary embodiments, the first sample image processing unit further includes:

[0220] A multi-scale decoding unit, configured to perform decoding processing on the encoded image features in sequence through a plurality of local decoding units of different scales in the decoding network; wherein each of the local decoding units performs upsampling and convolution operations on the features input to the decoding unit based on the intermediate encoded image features output by the local encoding unit of the corresponding scale;

[0221] A sample processed image determining unit, configured to execute a decoding result based on the last local decoding unit output to obtain the sample processed image;

[0222] The decoded image feature determination unit is configured to obtain the decoding result output by the penultimate local decoding unit as the decoded image feature.

[0223] In some exemplary embodiments, when the second sample image processing unit extracts the global visual features of the sample image through the pre-trained image processing model based on the attention mechanism, it is specifically configured to perform: extracting shallow global visual features and deep global visual features of the sample image respectively through the pre-trained image processing model based on the attention mechanism;

[0224] When the second sample image processing unit decodes based on the global visual feature to obtain the global decoded visual feature, the second sample image processing unit is specifically configured to execute: fusing the shallow global visual feature and the deep global visual feature to obtain the fused global visual feature; and decoding the fused global visual feature to obtain the global decoded visual feature.

[0225] Figure 7 is a structural block diagram of an image processing device according to an exemplary embodiment. Figure 7 , the image processing device 700 comprises:

[0226] The image to be processed acquiring unit 710 is configured to acquire an image to be processed, wherein the image to be processed includes a target skin area of ​​the object;

[0227] The image processing unit 720 is configured to input the image to be processed into an image processing model for image processing to obtain a target processed image;

[0228] The image processing model is trained based on any one of the image processing model training methods of the embodiments of the present disclosure.

[0229] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0230] In an exemplary embodiment, an electronic device is also provided, including a processor; a memory for storing instructions executable by the processor; wherein, when the processor is configured to execute the instructions stored in the memory, it implements the image processing model training method or image processing method provided in any embodiment of the present disclosure.

[0231] The electronic device may be a terminal, a server or a similar computing device. For example, the electronic device is a server. Figure 8 is a block diagram of an electronic device for running an image processing model training method or an image processing method according to an exemplary embodiment, such as Figure 8 As shown, the server 800 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 810 (the processor 810 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 830 for storing data, and one or more storage media 820 (such as one or more mass storage devices) for storing application programs 823 or data 822. Among them, the memory 830 and the storage medium 820 can be short-term storage or permanent storage. The program stored in the storage medium 820 may include one or more modules, each of which may include a series of instruction operations on the server. Furthermore, the central processing unit 810 can be configured to communicate with the storage medium 820 and execute a series of instruction operations in the storage medium 820 on the server 800. The server 800 may also include one or more power supplies 860, one or more wired or wireless network interfaces 850, one or more input and output interfaces 840, and / or one or more operating systems 821, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0232] The input / output interface 840 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the server 800. In one example, the input / output interface 840 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices via a base station so as to communicate with the Internet. In one example, the input / output interface 840 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0233] It can be understood by those skilled in the art that Figure 8 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 8 More or fewer components as shown, or with Figure 8 Different configurations are shown.

[0234] In an exemplary embodiment, a computer-readable storage medium is also provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the image processing model training method or image processing method provided in any of the above embodiments.

[0235] In an exemplary embodiment, a computer program product is also provided, including a computer program, which, when executed by a processor, implements the image processing model training method or the image processing method provided in any of the above embodiments.

[0236] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0237] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for training an image processing model, characterized in that: include: Acquire a sample image and an expected processed image corresponding to the sample image, wherein the sample image includes a target skin area of ​​the subject, and the expected processed image indicates the target skin area where a processing effect meets an expected effect; The sample image is input into the image processing model to be trained and the pre-trained image processing model of the image generator respectively, and the feature of the sample image is extracted by the image processing model to be trained, and a sample processed image is generated based on the extracted target feature; the global feature of the sample image is extracted by the pre-trained image processing model based on the attention mechanism; wherein the model parameter amount of the pre-trained image processing model is greater than the model parameter amount of the image processing model to be trained; Aligning the target feature with the global feature to obtain an aligned target feature and an aligned global feature; Inputting the sample processed image and the expected processed image into an image discriminator for discrimination processing to obtain a discrimination result; Based on the difference between the aligned target features and the aligned global features, and the discrimination results, the image generator and the image discriminator are alternately trained until a preset training end condition is met to obtain a trained image processing model; wherein, the model parameters of the pre-trained image processing model are fixed during the alternating training process.

2. The method according to claim 1, characterized in that The extracting features of the sample image by using the image processing model to be trained and generating a sample processed image based on the extracted target features comprises: The encoding network in the image processing model to be trained performs multi-scale local encoding on the sample image to obtain encoded image features, and the decoding network in the image processing model to be trained performs multi-scale local decoding based on the encoded image features to obtain decoded image features and the sample processed image; wherein the encoding network and the decoding network are jump-connected at the same scale; The extracting the global features of the sample image based on the attention mechanism by the pre-trained image processing model comprises: extracting the global visual features of the sample image based on the attention mechanism by the pre-trained image processing model, and decoding based on the global visual features to obtain global decoded visual features; The step of aligning the target feature with the global feature to obtain an aligned target feature and an aligned global feature comprises: aligning the encoded image feature and the decoded image feature with the global visual feature and the global decoded visual feature, respectively, to obtain a target encoded image feature, a target global visual feature, a target decoded image feature, and a target global decoded visual feature.

3. The method according to claim 2, characterized in that The alternately training the image generator and the image discriminator based on the difference between the aligned target feature and the aligned global feature, and the discrimination result, comprises: Determine an alignment loss using an alignment loss function based on a difference between the target encoded image feature and the target global visual feature, and a difference between the target decoded image feature and the target global decoded visual feature; Based on the discrimination result, determining a discrimination loss; determining an image processing loss based on a difference between the sample processed image and the desired processed image; Based on the alignment loss, the discrimination loss and the image processing loss, the image generator and the image discriminator are alternately trained.

4. The method according to claim 2, characterized in that: Aligning the decoded image features with the global decoded visual features to obtain target decoded image features and target global decoded visual features includes: Mapping the decoded image features to the decoding space of the pre-trained image processing model to obtain intermediate decoded image features; The intermediate decoded image features are aligned with the global decoded visual features to obtain target decoded image features and target global decoded visual features.

5. The method according to claim 2, characterized in that: The step of performing multi-scale local encoding on the sample image through the encoding network in the image processing model to be trained to obtain the encoded image features includes: The sample image is encoded sequentially by a plurality of local encoding units of different scales in the encoding network to obtain intermediate encoded image features output by each local encoding unit; the encoding process includes a convolution operation and a downsampling operation; Performing convolution processing on the intermediate coded image features output by the last local coding unit to obtain a first convolution result; The first convolution result is processed based on the channel attention mechanism and the spatial attention mechanism in sequence to obtain the encoded image feature.

6. The method according to claim 5, characterized in that The first convolution result is processed based on the channel attention mechanism and the spatial attention mechanism in sequence to obtain the encoded image features, including: Processing the first convolution result based on the channel attention mechanism to obtain channel attention; fusing the channel attention with the first convolution result to obtain a fused convolution result; The fused convolution result is processed based on the spatial attention mechanism to obtain spatial attention; the spatial attention is fused with the fused convolution result to obtain the encoded image feature.

7. The method according to claim 5, characterized in that The method of performing multi-scale local decoding based on the encoded image features by the decoding network in the image processing model to be trained to obtain the decoded image features and the sample processed image comprises: The encoded image features are sequentially decoded by a plurality of local decoding units of different scales in the decoding network; wherein each local decoding unit performs upsampling and convolution operations on the features input to the decoding unit based on the intermediate encoded image features output by the local encoding unit of the corresponding scale; Obtaining the sample processed image based on the decoding result output by the last local decoding unit; A decoding result output by the second to last local decoding unit is obtained as the decoded image feature.

8. The method according to claim 5, characterized in that The extracting the global visual features of the sample image based on the attention mechanism by the pre-trained image processing model includes: Extracting shallow global visual features and deep global visual features of the sample image respectively based on the attention mechanism through the pre-trained image processing model; The decoding based on the global visual feature to obtain the global decoded visual feature includes: fusing the shallow global visual feature and the deep global visual feature to obtain a fused global visual feature; The fused global visual feature is decoded to obtain the global decoded visual feature.

9. An image processing method, characterized in that: The method comprises: Acquire an image to be processed, wherein the image to be processed includes a target skin area of ​​the object; Inputting the image to be processed into an image processing model for image processing to obtain a target processed image; The image processing model is trained based on the image processing model training method according to any one of claims 1 to 8.

10. An image processing model training device, characterized in that: include: A sample image acquisition unit is configured to acquire a sample image and an expected processed image corresponding to the sample image, wherein the sample image includes a target skin area of ​​the object, and the expected processed image indicates the target skin area where a processing effect meets an expected effect; The sample image processing unit is configured to execute the steps of inputting the sample image into a to-be-trained image processing model and a pre-trained image processing model of an image generator respectively, extracting features of the sample image through the to-be-trained image processing model, and generating a sample processed image based on the extracted target features; extracting global features of the sample image based on an attention mechanism through the pre-trained image processing model; wherein the model parameter amount of the pre-trained image processing model is greater than the model parameter amount of the to-be-trained image processing model; an alignment unit, configured to perform an alignment process on the target feature and the global feature to obtain an aligned target feature and an aligned global feature; A discriminating unit is configured to input the sample processed image and the expected processed image into an image discriminator for discriminating processing to obtain a discriminating result; A training unit is configured to perform alternating training on the image generator and the image discriminator based on the difference between the aligned target features and the aligned global features, and the discrimination results until a preset training end condition is met, thereby obtaining a trained image processing model; wherein the model parameters of the pre-trained image processing model are fixed during the alternating training process.

11. An image processing device, characterized in that: include: A to-be-processed image acquisition unit, configured to acquire an to-be-processed image, wherein the to-be-processed image includes a target skin area of ​​the subject; An image processing unit is configured to input the image to be processed into an image processing model for image processing to obtain a target processed image; The image processing model is trained based on the image processing model training method according to any one of claims 1 to 8.

12. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image processing model training method as described in any one of claims 1 to 8, or the image processing method described in claim 9.

13. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image processing model training method as described in any one of claims 1 to 8, or the image processing method described in claim 9.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the image processing model training method as described in any one of claims 1 to 8, or the image processing method described in claim 9.

Images