A method, apparatus, storage medium and electronic device for image processing

Abstract

The specification discloses a method, device, storage medium and electronic equipment for image processing. The method comprises: obtaining an original sample image and a beautified sample image corresponding to the original sample image; inputting the original sample image into a to-be-trained image processing model to obtain a processed image; determining a first loss value according to a deviation between an image quality value corresponding to the original sample image and an image quality value corresponding to the processed image; determining a second loss value according to the processed image and the beautified sample image; and training the image processing model according to the first loss value and the second loss value.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of image processing, and particularly relates to an image processing method and device, a storage medium and an electronic device. BACKGROUND

[0002] For images shown to users in a business platform, the quality of the images often affects the clicking situation and use experience of the users. After beautifying the images with poor quality, the images with better quality can usually make the users more interested, thereby guiding the users to click or browse the content related to the images. Therefore, beautifying the images is an important step before the images are shown to the users in the business platform.

[0003] However, there is no suitable method to automatically process the images to be shown to realize beautifying the images. Usually, professional retouching personnel are needed to manually process the images to make the images more beautiful, and then the beautified images can be shown. However, the processing efficiency of this method is extremely low, and a large amount of resources are wasted.

[0004] Therefore, how to improve the efficiency of beautifying the images is a problem to be solved. SUMMARY

[0005] The present specification provides an image processing method and device, a storage medium and an electronic device to partially solve the above problems in the prior art.

[0006] The present specification adopts the following technical solutions:

[0007] The present specification provides an image processing method, comprising:

[0008] obtaining an image to be shown;

[0009] inputting the image to be shown into a pre-trained image processing model to obtain a beautified image, and showing the beautified image; wherein

[0010] The image processing model is trained by the following steps:

[0011] obtaining an original sample image and a beautified sample image corresponding to the original sample image;

[0012] inputting the original sample image into a to-be-trained image processing model to obtain a processed image;

[0013] determine a first loss value according to a deviation between the image quality value corresponding to the original sample image and the determined image quality value corresponding to the processed image, and determine a second loss value according to the processed image and the beautified sample image;

[0014] train the image processing model according to the first loss value and the second loss value.

[0015] Optionally, the determining of the first loss value according to the deviation between the image quality value corresponding to the original sample image and the determined image quality value corresponding to the processed image specifically includes:

[0016] determining whether the image quality value corresponding to the processed image is greater than a preset quality value compared with the image quality value corresponding to the original sample image;

[0017] if yes, taking a first specified value as the first loss value;

[0018] if no, taking a second specified value as the first loss value, wherein the first specified value is less than the second specified value.

[0019] Optionally, the determining of the first loss value according to the deviation between the image quality value corresponding to the original sample image and the determined image quality value corresponding to the processed image specifically includes:

[0020] determining whether the image quality value corresponding to the processed image is greater than a preset quality value compared with the image quality value corresponding to the original sample image;

[0021] if yes, taking a third specified value as the first loss value;

[0022] if no, taking a sum value between the deviation between the image quality value corresponding to the original sample image and the image quality value corresponding to the processed image and the preset quality value as the first loss value.

[0023] Optionally, the determining of the second loss value according to the processed image and the beautified sample image specifically includes:

[0024] determining a second sub-loss value in each dimension between the processed image and the beautified sample image according to the processed image and the beautified sample image;

[0025] determining the second loss value according to at least one second sub-loss value.

[0026] Optionally, the image processing model includes a discriminative network.

[0027] According to the processed image and the beautification sample image, a second sub-loss value of the processed image and the beautification sample image in each dimension is determined, specifically including:

[0028] Through the discrimination network, a probability that the processed image is judged as the beautification sample image is determined.

[0029] According to the probability, a second sub-loss value of the processed image and the beautification sample image in the texture dimension is determined.

[0030] Optionally, according to the processed image and the beautification sample image, a second sub-loss value of the processed image and the beautification sample image in each dimension is determined, specifically including:

[0031] The processed image is subjected to smoothing processing to reduce color difference of different color regions at a region boundary in the processed image, to obtain a smoothed processed image, and the beautification sample image is subjected to smoothing processing to reduce color difference of different color regions at a region boundary in the beautification sample image, to obtain a smoothed beautification sample image.

[0032] According to a deviation between a contrast corresponding to the smoothed processed image and a contrast corresponding to the smoothed beautification sample image, a second sub-loss value of the processed image and the beautification sample image in the color dimension is determined.

[0033] Optionally, according to the processed image and the beautification sample image, a second sub-loss value of the processed image and the beautification sample image in each dimension is determined, specifically including:

[0034] Through a feature extraction layer of the image processing model, feature extraction is performed on the processed image and the beautification sample image to obtain a feature map corresponding to the processed image and a feature map corresponding to the beautification sample image.

[0035] According to a deviation between the feature map corresponding to the processed image and the feature map corresponding to the beautification sample image, a second sub-loss value of the processed image and the beautification sample image in the image content dimension is determined.

[0036] Optionally, according to the first loss value and the second loss value, the image processing model is trained, specifically including:

[0037] According to the first loss value and the second loss value, a comprehensive loss value corresponding to the image processing model is determined.

[0038] The image processing model is trained with a minimization of the comprehensive loss value as an optimization target.

[0039] Optionally, according to the first loss value and the second loss value, a comprehensive loss value corresponding to the image processing model is determined, specifically comprising:

[0040] The processed image is subjected to noise reduction processing to obtain a noise-reduced processed image;

[0041] According to the noise-reduced processed image, a third loss value is determined;

[0042] According to the first loss value, the second loss value and the third loss value, a comprehensive loss value corresponding to the image processing model is determined.

[0043] The present specification provides an image processing device, comprising:

[0044] A first acquisition module acquires a to-be-displayed image;

[0045] A first input module inputs the to-be-displayed image into a pre-trained image processing model to obtain a beautified image, and displays the beautified image;

[0046] The device further comprises a module for training the image processing model, wherein

[0047] A second acquisition module acquires an original sample image and a beautified sample image corresponding to the original sample image;

[0048] A second input module inputs the original sample image into a to-be-trained image processing model to obtain a processed image;

[0049] A determination module determines a first loss value according to the deviation between the image quality value corresponding to the original sample image and the determined image quality value corresponding to the processed image, and determines a second loss value according to the processed image and the beautified sample image;

[0050] A training module trains the image processing model according to the first loss value and the second loss value.

[0051] The present specification provides a computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above-mentioned image processing method.

[0052] The present specification provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the above-mentioned image processing method when executing the program.

[0053] The above-mentioned at least one technical solution adopted by the present specification can achieve the following beneficial effects:

[0054] In the method for image processing provided in the specification, the original sample image and the beautified sample image are first obtained, and then the original sample image is input into the image processing model to be trained to determine the processed image corresponding to the original sample image, and then the first loss value corresponding to the image processing model is determined according to the deviation between the quality value corresponding to the original sample image and the quality value corresponding to the processed image, and the second loss value corresponding to the image processing model is determined according to the processed image and the beautified sample image, and the image processing model is trained according to the first loss value and the second loss value.

[0055] As can be seen from the above method, when the image processing model is trained, the image processing model is trained according to the deviation between the quality value corresponding to the original sample image and the quality value corresponding to the processed image, and the original sample image and the beautified sample image with a higher quality value. In this way, the final generated beautified image can be closer to the beautified sample image while the quality value is also improved to a certain extent. Compared with the existing scheme which can only be processed by manual processing, the image processing model trained by the present scheme can process the image while improving the quality value of the image, thereby improving the efficiency of image processing. BRIEF DESCRIPTION OF DRAWINGS

[0056] The accompanying drawings used to provide further understanding of the specification, constitute a part of the specification, the illustrative embodiments of the specification and the explanation thereof are used to explain the specification, and do not constitute improper limitation on the specification. In the drawings:

[0057] Figure 1 It is a flowchart of the method for image processing provided in the specification;

[0058] Figure 2 It is a schematic diagram of the training method of the image processing model provided in the specification;

[0059] Figure 3 It is a schematic diagram of the device for image processing provided in the specification;

[0060] Figure 4 It is a schematic diagram of the electronic device corresponding to the image processing provided in the specification. Figure 1 DETAILED DESCRIPTION

[0061] ​For the purposes of the present description, the technical solutions and advantages, the technical solutions provided by the embodiments of the present description will be described in detail below in conjunction with the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present description, not all embodiments. Based on the embodiments in the present description, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present description.

[0062] The technical solutions provided by the embodiments of the present description will be described in detail below in conjunction with the drawings.

[0063] Figure 1 The flowchart of the image processing method provided in the present description is shown in the flowchart, wherein only the training step of the image processing model is shown, and the specific steps are as follows:

[0064] S101: Obtain the original sample image and the beautified sample image corresponding to the original sample image.

[0065] When the business platform obtains some images related to the business (such as various food images provided by the merchant in the take-out platform), and wants to show these images to the user in the client or webpage of the business platform, due to the parameter setting of the image acquisition device (such as mobile phone, camera, etc.), the acquisition environment or the operation of the image acquisition personnel, etc. Factors, it is possible to cause the quality of the image to be low, such as some images will appear dark brightness, low clarity, high exposure or low saturation, etc. Problems, resulting in these images do not look beautiful, therefore, if these images are directly shown to the user in the business platform, it is likely to affect the user's experience, further affecting the user's click and browsing of information.

[0066] Generally, the business client or server will first obtain the image to be displayed, and then input the image to be displayed into the pre-trained image processing model, process the image to be displayed through the image processing model, improve the quality of the image, and obtain the beautified image, and show the beautified image to the user in the business client or webpage.

[0067] In the image processing process, the training method of the image processing model often determines the processing effect of the image. Based on this, the present description provides an image processing method.

[0068] Among them, the original sample image and the beautified sample image corresponding to the original sample image need to be obtained, and each group of original sample image and its corresponding sample beautified image is composed of a sample image pair as an input of the image processing model to be trained.

[0069] In practical applications, the beautified sample images described above can be obtained by performing corresponding beautification processing on the original sample images by professional retouching personnel, and thus it can be considered that the quality values of these beautified sample images are improved compared to the quality values of the original sample images before beautification.

[0070] Of course, the beautified sample images described above can also be images with higher quality values obtained by beautifying the original sample images in other ways, and the present specification does not specifically limit this.

[0071] In addition, in the present specification, the execution subject of the method for implementing image processing can refer to a designated device such as a server provided in a business platform. In order to facilitate description, the present specification only takes the server as the execution subject as an example to describe a method for image processing provided by the present specification.

[0072] S102: inputting the original sample image into the image processing model to be trained to obtain a processed image.

[0073] After obtaining the sample image pair containing the original sample image and the beautified sample image, the server can input the sample image pair into the processing model to be trained, and generate the processed image corresponding to the original sample image through the image processing model to be trained.

[0074] Specifically, the image processing model can contain a corresponding generation network, wherein the generation network is a full convolutional network and contains 12 network layers. For each original sample image input into the model, the image is first preprocessed through a 9*9 network layer in the generation network, and then enhanced through 4 residual blocks (each convolutional block contains two convolutional layers) and 3 convolutional layers, so as to obtain the processed image.

[0075] S103: determining a first loss value according to the deviation between the image quality value corresponding to the original sample image and the image quality value corresponding to the processed image determined, and determining a second loss value according to the processed image and the beautified sample image.

[0076] After obtaining the processed image, the server can take the processed image and the beautified sample image as a new image pair for further training of the model.

[0077] In the process of training the image processing model, the image processing model can further include a corresponding discriminator network in addition to the generation network. The generation network is used to generate processed images, and the generated processed images become closer to the beautified sample images as the processing model is continuously trained, thereby "fooling" the discriminator network. The discriminator network gradually improves the ability to distinguish between processed images and beautified sample images during the training process, thereby "opposing" the generation network. In this way, the ability of the discriminator network to distinguish can be improved while making the processed images closer to the beautified sample images.

[0078] It should be noted that the image processing model after training can not include the above-mentioned discriminator network in the process of processing the image.

[0079] In actual application, due to different aesthetic and retouching habits of different retouching personnel, the beautified sample images corresponding to different original sample images will also be different. Therefore, the processed image generated each time can not be very close to the beautified sample image corresponding to the original sample image. Therefore, the server can determine the quality value loss corresponding to the image processing model according to the deviation between the image quality value corresponding to the original sample image and the image quality value corresponding to the processed image, and take the quality value loss as the first loss value of the image processing model.

[0080] The image quality value is used to represent the aesthetic degree of the image, and the image quality value can be determined by manual annotation. Users belonging to different user groups rate the quality values of the original sample image and the processed image, such as dividing the image quality values corresponding to the original sample image and the processed image into five levels of poor, relatively poor, general, good, and relatively good according to the aesthetic standards of different users, and each level corresponds to a different image quality value. In addition, the server can also input the original sample image and the processed image into a pre-trained image quality value determination model to determine the image quality values corresponding to the original sample image and the processed image, respectively. Of course, other ways can also be used to determine the image quality value, which is not limited in the specification.

[0081] Specifically, the server can determine whether the image quality value corresponding to the processed image is greater than the preset quality value compared with the image quality value corresponding to the original sample image. If yes, it means that the improvement of the quality value corresponding to the processed image has reached the target, so the model will have a smaller first loss value. If not, it means that the improvement of the quality value corresponding to the processed image has not reached the target, so the model will have a larger first loss value. The preset quality value can be set according to actual conditions, which is not limited in the specification.

[0082] Specifically, the server can determine whether the image quality value corresponding to the processed image is greater than the preset quality value compared with the image quality value corresponding to the original sample image, if yes, the first specified value is taken as the first loss value, if not, the second specified value is taken as the first loss value. Wherein, the first specified value is smaller than the second specified value. Since the first loss value is determined according to the deviation of the image in the image quality value, the first loss value can also be understood as the quality value loss of the image.

[0083] The quality value loss can be represented by the following formula:

[0084] L aesthetic =sign(sign(X s -Y s +α)+1)

[0085] Wherein, L aesthetic is the first loss value (i.e. quality value loss) corresponding to the image processing model, X s is the original sample image, Y s is the processed image, and α is the preset quality value, and the value of α is positive.

[0086] As can be seen from the formula, if the image quality value corresponding to the processed image is greater than the preset quality value compared with the image quality value corresponding to the original sample image, (X s -Y s +α)<0, therefore, the loss value of L aesthetic is 0 (i.e. the first specified value), and when the image quality value corresponding to the processed image is less than the preset quality value compared with the image quality value corresponding to the original sample image, (X s -Y s +α)>0, therefore, the loss value of L aesthetic is 1 (i.e. the second specified value).

[0087] Of course, if the image quality value corresponding to the processed image is greater than the preset quality value compared with the image quality value corresponding to the original sample image, the third specified value is taken as the first loss value, and if not, the sum value between the deviation between the image quality value corresponding to the original sample image and the image quality value corresponding to the processed image and the preset quality value is taken as the first loss value, and the sum value is certainly greater than the third specified value. The first loss value can also be determined by the following formula:

[0088] L aesthetic =max(X s -Y s +α,0)

[0089] It can be seen from the formula that if the image quality value corresponding to the processed image is greater than the preset quality value compared with the image quality value corresponding to the original sample image, (X s -Y s + a) < 0, so at this time, the loss value of L aesthetic is 0 (i.e., the third specified value), and when the image quality value corresponding to the processed image is less than the preset quality value compared with the image quality value corresponding to the original sample image, (X s -Y s + a) > 0, so at this time, the loss value of L aesthetic is X s -Y s + a.

[0090] The server can determine the first loss value of the image processing model by any one of the above two methods. It should be noted that in the above two methods, the former pays more attention to discrete value loss, that is, there is a corresponding first loss value (i.e., the loss value is 1) only when the quality value of the processed image is not improved enough, and there is no corresponding first loss value (i.e., the loss value is 0) when the quality value is improved to the target. The latter pays more attention to continuous value loss. When the quality value of the processed image is not improved enough, the lower the quality value is improved, the greater the first loss value is.

[0091] In addition, it should be emphasized that the above quality value deviation refers to the difference between the quality value corresponding to the original sample image and the quality value corresponding to the processed image. When the quality value corresponding to the processed image is higher than the quality value corresponding to the original sample image, the value of the quality value deviation is negative. When the quality value corresponding to the processed image is lower than the quality value corresponding to the original sample image, the value of the quality value deviation is positive. That is, when the image quality value corresponding to the processed image is less than the preset quality value compared with the image quality value corresponding to the original sample image, the image quality value deviation between the two is negative, and the smaller the absolute value of the deviation is than the preset quality value a, the greater the first loss value is.

[0092] In addition, the server can also determine the second loss value corresponding to the image processing model according to the processed image and the beautified sample image.

[0093] Further, the server can determine the second sub-loss value of the processed image and the beautified sample image in each dimension (such as color dimension, texture dimension, image content dimension, etc.) according to the processed image and the beautified sample image, and determine the second loss value according to at least one second sub-loss value.

[0094] Specifically, the server can determine the probability that the processed image is judged as the beautified sample image according to the discrimination network of the image processing model. Generally, the discrimination network determines the probability that the processed image is judged as the beautified sample image according to the difference between the texture information of the processed image and the beautified sample image. Therefore, the server can determine the second sub-loss value (i.e., the texture loss) of the image processing model in the texture dimension according to the probability.

[0095] The greater the probability that the discrimination network judges the processed image as the beautified sample image, the closer the processed image is to the beautified sample image. At this time, the second sub-loss value of the image processing model in the texture dimension is smaller. The smaller the probability that the discrimination network judges the processed image as the beautified sample image, the greater the difference between the processed image and the beautified sample image. At this time, the second sub-loss value of the image processing model in the texture dimension is greater.

[0096] The second sub-loss value (i.e., the texture loss) of the image processing model in the texture dimension can be represented by the following formula:

[0097]

[0098] wherein L texture is the texture loss corresponding to the image processing model, D is the discrimination network of the image processing model, F W is the generation network of the image processing model, I s is the original sample image, and F W (I s ) is the processed image generated by the generation network, I t is the beautified sample image. As can be seen from the formula, the greater the probability that the discrimination network judges I s as I t , the smaller the texture loss.

[0099] Of course, in this specification, the second sub-loss value in the texture dimension can also be determined in other ways. For example, the server can default the beautified sample image as “true” and the processed image as “false”, and then input the processed image and the beautified sample image into the discrimination network. When the discrimination network judges the processed image as “true” and the beautified sample image as “false”, it means that the discrimination network cannot distinguish the processed image from the beautified sample image at this time. Therefore, the image processing model obtains a smaller texture loss. When the discrimination network judges the processed image as “false” and the beautified sample image as “true”, it means that there is a greater difference between the generated processed image and the beautified sample image. Therefore, the image processing model obtains a greater texture loss.

[0100] In addition, the server performs smoothing processing (e.g., Gaussian blur) on the processed image to reduce color differences of different color regions at region boundaries in the processed image, to obtain a smoothed processed image, and performs smoothing processing on the beautified sample image to reduce color differences of different color regions at region boundaries in the beautified sample image, to obtain a smoothed beautified sample image. In this way, the second sub-loss value (i.e., color loss) of the image processing model in the color dimension can be determined according to the deviation between the contrast corresponding to the smoothed processed image and the contrast corresponding to the smoothed beautified sample image. The color loss can be represented by the following formula:

[0101]

[0102] wherein L color is the second sub-loss (color loss) of the image processing model in the color dimension, X b is the smoothed beautified sample image, Y b is the smoothed processed image. As can be seen from the formula, the greater the deviation between the contrast corresponding to the smoothed processed image and the contrast corresponding to the smoothed beautified sample image, the greater the color loss. The smaller the deviation between the contrast corresponding to the smoothed processed image and the contrast corresponding to the smoothed beautified sample image, the smaller the color loss.

[0103] In addition, the server can also perform feature extraction on the processed image and the beautified sample image through the feature extraction layer in the image processing model, so as to determine the second sub-loss (i.e., content loss) of the image processing model in the image content dimension according to the deviation between the feature map corresponding to the processed image and the feature map corresponding to the beautified sample image. In this specification, the server can determine the content loss corresponding to the image processing model according to the distance (e.g., Euclidean distance, Manhattan distance, etc.) between the above two feature maps (feature vectors). The content loss can be represented by the following formula:

[0104] L content =‖ψ j (F W (I s ))-ψ j (F t (I j )‖ / C j H j

[0105] wherein L content is the second sub-loss (content loss) of the image processing model in the image content dimension, ψ j (F W (Is )) is a generated network F W the generated processed image I s the corresponding feature map, ψ j (I t ) is the beautified sample image I t the corresponding feature map, C j H j W j are the number of channels of the feature map, the height size of the feature map, and the width size of the feature map. As can be seen from the formula, the greater the deviation between the feature map corresponding to the processed image and the feature map corresponding to the beautified sample image, the greater the content loss function. The smaller the deviation between the feature map corresponding to the processed image and the feature map corresponding to the beautified sample image, the smaller the content loss function.

[0106] In this specification, the server can also determine the second loss value of the image processing model according to part or all of the above color loss, content loss, and texture loss. For example, the server can take the sum of the color loss, content loss, and texture loss as the second loss value of the image processing model.

[0107] In addition, the server can also determine the second loss value according to the deviation between the quality value corresponding to the processed image and the quality value corresponding to the beautified sample image. The method of determining the loss value can be the same as that of determining the first loss value, which will not be described in detail here.

[0108] S104: training the image processing model according to the first loss value and the second loss value.

[0109] After determining the first loss value and the second loss value of the image processing model, the server can determine the comprehensive loss value of the image processing model according to the first loss value and the second loss value.

[0110] Since the processed image generated by the generation network of the image processing model usually contains some noise (such as salt and pepper noise), in order to eliminate this part of noise, the server can perform noise reduction processing on the processed image to obtain a noise-reduced processed image, and then determine the noise loss of the image processing model according to the noise-reduced processed image, and take the noise loss as the third loss value of the image processing model. The third loss value can be represented by the following formula:

[0111]

[0112] wherein, L tv is the third loss value (noise loss) of the image processing model, the noise determined after differentiating the horizontal direction of the processed image F W (I s ) in the vertical direction, the noise determined after differentiating the horizontal direction of the processed image F W (I s ) in the vertical direction.

[0113] Therefore, the server can also determine the comprehensive loss value of the model according to the first loss value, the second loss value, and the third loss value. The comprehensive loss value of the model can be represented by the following formula:

[0114] LOSS = L aesthetic + L texture + L color + L content + L tv

[0115] where LOSS is the comprehensive loss value corresponding to the image processing model, L texture + L color + L content is the second loss value corresponding to the image processing model.

[0116] For ease of understanding, the present specification also provides a schematic diagram of a training method of an image processing model, as shown in Figure 2 .

[0117] Figure 2 The present specification provides a schematic diagram of a training method of an image processing model.

[0118] After receiving the original sample image and the beautified sample image, the server first inputs the original sample image into the generation network of the image processing model, thereby generating a processed image corresponding to the original sample image through the generation network. Then, the server can determine the quality value loss (i.e., the first loss) corresponding to the image processing model according to the quality value deviation between the original sample image and the processed image, determine the color loss corresponding to the image processing model according to the smoothed processed image and the smoothed beautified sample image, determine the content loss corresponding to the image processing model according to the feature vector of the processed image extracted by the feature extraction layer and the feature vector of the beautified sample image, determine the texture loss corresponding to the image processing model according to the probability of the processed image being judged as a beautified image by the discriminator network, determine the noise loss corresponding to the image processing model according to the noise contained in the processed image, and then determine the comprehensive loss value corresponding to the image processing model according to the quality value loss, the color loss, the content loss, and the noise loss. The image processing model is trained with the minimum comprehensive loss value as the optimization target.

[0119] After determining the comprehensive loss value corresponding to the image processing model, the server can train the image processing model with the optimization goal of minimizing the comprehensive loss value until the training target is met.

[0120] The training target can be that the image processing model converges within a preset threshold range, or reaches a preset training number, to ensure that the improvement degree of the quality value of the generated processed image reaches the expected value and is closer to the image processed by the real retouching personnel. The preset threshold and the preset training number can be set according to actual needs, and the present specification does not make specific limitations.

[0121] After the model training is completed, the model can be deployed in a business client or a server on a terminal device such as a mobile phone, a tablet computer, a notebook computer, a desktop computer, and the like, so that the received image is processed by the business client, and then the beautified image obtained after processing is displayed to the user.

[0122] It should be noted that all actions of obtaining signals, information or data in the present specification are performed in accordance with the corresponding data protection regulations and policies of the country where the device is located, and with the authorization given by the owner of the corresponding device.

[0123] As can be seen from the above method, when the image processing model is trained, the image processing model is trained according to the deviation between the quality value corresponding to the original sample image and the quality value corresponding to the processed image, and the original sample image and the beautified sample image with a higher quality value. In this way, the final generated beautified image can be closer to the beautified sample image while the quality value is also improved to a certain extent. Compared with the existing scheme which can only be processed by artificial processing, the image processing model trained by the present scheme can process the image under the premise of improving the quality value of the image, thereby improving the efficiency of image processing.

[0124] The above is one or more embodiments of the method of the present specification for image processing. Based on the same idea, the present specification also provides a corresponding image processing device, as shown in Figure 3 .

[0125] Figure 3 A schematic diagram of an image processing device provided by the present specification includes:

[0126] The first acquisition module 301 is configured to acquire a to-be-displayed image.

[0127] The first input module 302 is configured to input the to-be-displayed image into a pre-trained image processing model to obtain a beautified image, and display the beautified image.

[0128] The apparatus further comprises a module for training the image processing model, wherein

[0129] A second obtaining module 303 is configured to obtain an original sample image and a beautified sample image corresponding to the original sample image.

[0130] A second input module 304 is configured to input the original sample image into an image processing model to be trained to obtain a processed image.

[0131] A determining module 305 is configured to determine a first loss value according to a deviation between an image quality value corresponding to the original sample image and an image quality value corresponding to the processed image, and determine a second loss value according to the processed image and the beautified sample image.

[0132] A training module 306 is configured to train the image processing model according to the first loss value and the second loss value.

[0133] Optionally, the determining module 305 is specifically configured to determine whether the image quality value corresponding to the processed image is greater than a preset quality value compared with the image quality value corresponding to the original sample image; if yes, a first specified value is taken as the first loss value; if no, a second specified value is taken as the first loss value, wherein the first specified value is less than the second specified value.

[0134] Optionally, the determining module 305 is specifically configured to determine whether the image quality value corresponding to the processed image is greater than a preset quality value compared with the image quality value corresponding to the original sample image; if yes, a third specified value is taken as the first loss value; if no, a sum value between a deviation between the image quality value corresponding to the original sample image and the image quality value corresponding to the processed image and the preset quality value is taken as the first loss value.

[0135] Optionally, the image processing model comprises a discriminator network.

[0136] The determining module 305 is specifically configured to determine, by the discriminator network, a probability that the processed image is judged as the beautified sample image, and determine a second sub-loss value of the processed image and the beautified sample image in a texture dimension according to the probability.

[0137] Optionally, the determining module 305 is specifically configured to: smooth the processed image to reduce the color difference between different color regions at the region boundaries in the processed image, to obtain a smoothed processed image; and smooth the beautified sample image to reduce the color difference between different color regions at the region boundaries in the beautified sample image, to obtain a smoothed beautified sample image; and determine a second sub-loss value in the color dimension between the processed image and the beautified sample image based on the deviation between the contrast corresponding to the smoothed processed image and the contrast corresponding to the smoothed beautified sample image.

[0138] Optionally, the determining module 305 is specifically used to: extract features from the processed image and the beautified sample image through the feature extraction layer of the image processing model to obtain the feature map corresponding to the processed image and the feature map corresponding to the beautified sample image; and determine the second sub-loss value of the processed image and the beautified sample image in the image content dimension based on the deviation between the feature map corresponding to the processed image and the feature map corresponding to the beautified sample image.

[0139] Optionally, the training module 306 is specifically used to determine the comprehensive loss value corresponding to the image processing model based on the first loss value and the second loss value; and to train the image processing model with minimizing the comprehensive loss value as the optimization objective.

[0140] Optionally, the training module 306 is specifically used to: perform noise reduction processing on the processed image to obtain a noise-reduced processed image; determine a third loss value based on the noise-reduced processed image; and determine a comprehensive loss value corresponding to the image processing model based on the first loss value, the second loss value, and the third loss value.

[0141] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 This provides an image processing method.

[0142] This instruction manual also provides Figure 4 One of the corresponding Figure 1 A schematic diagram of the structure of an electronic device. (e.g.) Figure 4 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1The method of image processing. Of course, in addition to the software implementation, the present specification does not exclude other implementations, such as logic devices or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0143] In the 1990s, it was quite obvious to distinguish whether an improvement in a technology was in hardware (e.g., improvement in circuit structures of diodes, transistors, switches, etc.) or in software (improvement in method flow). However, as technology has evolved, many improvements in method flow today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structures by programming the improved method flow into hardware circuits. Therefore, it cannot be said that an improvement in a method flow cannot be implemented by hardware entity modules. For example, a programmable logic device (PLD) such as a field programmable gate array (FPGA) is an integrated circuit whose logic function is determined by user programming of the device. A digital system is "integrated" on a piece of PLD by the designer programming it by himself, without having to ask a chip manufacturer to design and manufacture a special integrated circuit chip. Moreover, instead of manually fabricating an integrated circuit chip, this programming is now mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing a program, and the original code before compilation also has to be written in a specific programming language, which is called a hardware description language (HDL), and there are many types of HDL, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should be aware that, as long as the method flow is logically programmed in the above-mentioned hardware description languages and programmed into an integrated circuit, a hardware circuit that implements the logical method flow can be easily obtained.

[0144] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to being implemented in pure computer readable program code, the controller can equally well be implemented to perform the same functions using logic gates, switches, an application specific integrated circuit, a programmable logic controller and an embedded microcontroller, etc. by means of a logical programming of the method steps. The controller can thus be considered as a hardware component, and the means comprised therein for performing the various functions can be considered as structures within the hardware component. Alternatively, the means for performing the various functions can even be considered as both a software module implementing the method and a structure within the hardware component.

[0145] The systems, apparatuses, modules or units illustrated by the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0146] For the sake of description, the above apparatuses are described in various units by functions respectively. Of course, the functions of the units can be implemented in one or more software and / or hardware in the implementation of the present specification.

[0147] Those skilled in the art will understand that the embodiments of the present specification can be provided as a method, a system or a computer program product. Therefore, the present specification can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] The specification is presented with reference to flow diagrams and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the specification. It will be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing element or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks.

[0149] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks.

[0150] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks. ​ The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks.

[0151] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0152] The memory can include non-persistent memory and / or storage mechanisms such as, for example, random access memory (RAM), non-volatile memory (NVM), and / or a persistent memory such as, for example, read-only memory (ROM) or flash memory. The memory is an example of computer-readable media.

[0153] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0154] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0155] Those skilled in the art will appreciate that embodiments of the present specification can be provided as methods, systems or computer program products. Therefore, the present specification can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] The present specification can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The present specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including storage devices.

[0157] The various embodiments described in this specification are described using a numbering of embodiments approach: these are each individually integrated contributions pertaining to different but related aspects of the description. Each of the various embodiments can stand on its own, and each can be combined with the subject matter of other embodiments to produce further embodiments. Where appropriate, therefore, the contents of the specification can be regarded as being incorporated by reference, including the description, drawings, claims, abstract and the like.

[0158] The above description is embodied in the form of embodiments only and is not intended to limit the present specification. The present specification can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification should be included in the scope of the claims of the present specification.

Claims

1. A method of image processing, characterized by, The method comprises: obtaining an image to be displayed; inputting the image to be displayed into a pre-trained image processing model to obtain a beautified image, and displaying the beautified image; wherein the image processing model is trained by the following steps: obtaining an original sample image and a beautified sample image corresponding to the original sample image; inputting the original sample image into a to-be-trained image processing model to obtain a processed image; determining a first loss value according to the deviation between the image quality value corresponding to the original sample image and the determined image quality value corresponding to the processed image, and determining a second loss value according to the processed image and the beautified sample image; training the image processing model according to the first loss value and the second loss value; determining a first loss value according to the deviation between the image quality value corresponding to the original sample image and the determined image quality value corresponding to the processed image, specifically comprising: determining whether the image quality value corresponding to the processed image is greater than a preset quality value compared with the image quality value corresponding to the original sample image; if yes, a first specified value is taken as the first loss value; if no, a second specified value is taken as the first loss value, wherein the first specified value is less than the second specified value.

2. The method of claim 1, wherein, determining a first loss value according to the deviation between the image quality value corresponding to the original sample image and the determined image quality value corresponding to the processed image, specifically comprising: determining whether the image quality value corresponding to the processed image is greater than a preset quality value compared with the image quality value corresponding to the original sample image; if yes, a third specified value is taken as the first loss value; if no, the sum value between the deviation between the image quality value corresponding to the original sample image and the image quality value corresponding to the processed image and the preset quality value is taken as the first loss value.

3. The method of claim 1, wherein, determining a second loss value according to the processed image and the beautified sample image, specifically comprising: determining a second sub-loss value of the processed image and the beautified sample image in each dimension according to the processed image and the beautified sample image; determining the second loss value according to at least one second sub-loss value.

4. The method of claim 3, wherein, The image processing model comprises a discriminator network. determining a second sub-loss value of the processed image and the beautified sample image in each dimension according to the processed image and the beautified sample image, specifically comprising: determining the probability of judging the processed image as the beautified sample image through the discriminator network; determining the second sub-loss value of the processed image and the beautified sample image in the texture dimension according to the probability.

5. The method of claim 3, wherein, determining a second sub-loss value of the processed image and the beautified sample image in each dimension according to the processed image and the beautified sample image, specifically comprising: The processed image is smoothed to reduce color differences of different color regions at region boundaries in the processed image, to obtain a smoothed processed image, and the beautified sample image is smoothed to reduce color differences of different color regions at region boundaries in the beautified sample image, to obtain a smoothed beautified sample image; a second sub-loss value of the processed image and the beautified sample image in the color dimension is determined according to a deviation between a contrast corresponding to the smoothed processed image and a contrast corresponding to the smoothed beautified sample image.

6. The method of claim 3, wherein, According to the processed image and the beautified sample image, a second sub-loss value of the processed image and the beautified sample image in each dimension is determined, specifically including: The processed image and the beautified sample image are feature extracted through a feature extraction layer of the image processing model, to obtain a feature map corresponding to the processed image and a feature map corresponding to the beautified sample image; A second sub-loss value of the processed image and the beautified sample image in the image content dimension is determined according to a deviation between the feature map corresponding to the processed image and the feature map corresponding to the beautified sample image.

7. The method according to any one of claims 1 to 6, characterized in that, According to the first loss value and the second loss value, the image processing model is trained, specifically including: According to the first loss value and the second loss value, a comprehensive loss value corresponding to the image processing model is determined; The image processing model is trained with minimizing the comprehensive loss value as an optimization goal.

8. The method of claim 7, wherein, According to the first loss value and the second loss value, a comprehensive loss value corresponding to the image processing model is determined, specifically including: The processed image is denoised to obtain a denoised processed image; A third loss value is determined according to the denoised processed image; A comprehensive loss value corresponding to the image processing model is determined according to the first loss value, the second loss value and the third loss value.

9. An apparatus for image processing, characterized by Including: The first acquisition module acquires the image to be displayed; The first input module inputs the image to be displayed into the pre-trained image processing model to obtain a beautified image, and displays the beautified image; The device further includes a module for training the image processing model, wherein the second acquisition module acquires an original sample image and a beautified sample image corresponding to the original sample image; The second input module inputs the original sample image into the image processing model to be trained to obtain a processed image; The determination module determines a first loss value according to a deviation between an image quality value corresponding to the original sample image and an image quality value corresponding to the processed image determined, and determines a second loss value according to the processed image and the beautified sample image; The training module trains the image processing model according to the first loss value and the second loss value; The first loss value is determined according to a deviation between an image quality value corresponding to the original sample image and an image quality value corresponding to the processed image determined, specifically including: determining whether an image quality value corresponding to the processed image is greater than a preset quality value compared with an image quality value corresponding to the original sample image; if yes, taking a first specified value as the first loss value; if no, taking a second specified value as the first loss value, wherein the first specified value is less than the second specified value.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1-8.

11. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the method in any one of claims 1-8.

Images